WEBVTT 00:00:01.490 --> 00:00:05.730 It's Rust in Production, a podcast about companies who use Rust to shape the 00:00:05.730 --> 00:00:06.650 future of infrastructure. 00:00:07.010 --> 00:00:11.010 My name is Matthias Endler from Corode, and today I'm talking to Andrew Tinka 00:00:11.010 --> 00:00:14.490 from Scythe about grassroots robotics with Rust. 00:00:18.850 --> 00:00:23.450 Andrew, thanks so much for taking the time today for the interview. 00:00:23.990 --> 00:00:27.190 Can you please say a few words about yourself and about Scythe? 00:00:27.950 --> 00:00:33.370 Absolutely. My name is Andrew Tinka. I'm the Director of Software Engineering at Scythe Robotics. 00:00:33.830 --> 00:00:39.170 Scythe designs and manufactures professional-grade autonomous lawnmowers for 00:00:39.170 --> 00:00:40.670 the commercial landscaping industry. 00:00:40.910 --> 00:00:46.470 So when I say autonomous lawnmowers, many people think of a Roomba for your front lawn. 00:00:46.650 --> 00:00:50.790 What we build is larger and more heavy-duty. 00:00:50.970 --> 00:01:00.510 It is a 1,400-pound machine, roughly 650 kilograms. It is a fully capable stand-on lawnmower. 00:01:00.690 --> 00:01:07.290 So you can watch crews use this machine as a fully capable electric manual lawnmower, 00:01:07.310 --> 00:01:12.190 or they can put it to work to autonomously mow large areas. 00:01:12.390 --> 00:01:15.190 The value that Scythe brings is to... 00:01:16.120 --> 00:01:21.900 Increase the efficiency of taking care of green spaces, parks, 00:01:22.560 --> 00:01:28.580 athletic fields, corporate campuses, large areas of grass where mowing it all 00:01:28.580 --> 00:01:31.540 manually is not a great use of people's time. 00:01:32.100 --> 00:01:37.300 Ideally, a landscaping crew would get more done with the same number of people 00:01:37.300 --> 00:01:39.720 because while the lawn mowing is happening, 00:01:40.080 --> 00:01:44.180 the rest of the crew is handling the string trimming and the weeding and all 00:01:44.180 --> 00:01:48.640 of the other more manually involved jobs around the property, 00:01:48.900 --> 00:01:53.320 while the bulk of the just mowing the grass is getting done by a robot. 00:01:53.540 --> 00:01:58.260 That means it's very much a heavy-duty machine and it's for industry use. 00:01:58.800 --> 00:02:06.440 Absolutely. Our ideal customer is a commercial landscaper who is serving a lot of contracts. 00:02:06.620 --> 00:02:09.600 And so they're mowing 40 hours a week. They're loading these machines up and 00:02:09.600 --> 00:02:13.280 going from one property to another, taking care of a park in the morning, 00:02:13.480 --> 00:02:19.820 taking care of a big field in the afternoon, and doing this work every day throughout the week. 00:02:20.180 --> 00:02:24.980 Yeah. You get really good value out of it when you use that machine as often 00:02:24.980 --> 00:02:29.160 as possible, ideally throughout the workday. throughout the week. 00:02:30.600 --> 00:02:34.200 And for how long has Scythe been in existence? 00:02:34.780 --> 00:02:39.580 Scythe got started in 2018 as a classic garage startup. 00:02:39.780 --> 00:02:43.540 Just a few engineers in a small space putting together the first prototype. 00:02:44.260 --> 00:02:50.120 And so since then, we have gone through several successive generations of our 00:02:50.120 --> 00:02:54.020 hardware, and our fleet has grown every year, and the number of customers we've 00:02:54.020 --> 00:02:55.980 served has been growing every years since 2018. 00:02:56.540 --> 00:03:01.500 And was the idea always to have an automated robot? Was that the idea? 00:03:01.960 --> 00:03:08.860 Absolutely. The founding principle of the company was to use robotics to help 00:03:08.860 --> 00:03:11.060 take care of the earth better. 00:03:11.300 --> 00:03:15.560 And so there's a couple of ways in which that decision played out. 00:03:16.280 --> 00:03:22.540 The first one was to find the activities that people do to take care of green 00:03:22.540 --> 00:03:27.360 spaces and see which ones are a place where a robot can bring value. 00:03:27.640 --> 00:03:32.840 And the second aspect of that decision is our commitment to build all electric machines. 00:03:33.380 --> 00:03:36.440 Gas-powered landscaping equipment 00:03:36.440 --> 00:03:42.520 is often uniquely polluting even for internal combustion engines. 00:03:42.520 --> 00:03:49.920 And so bringing a valid replacement to the market to allow these kind of operations 00:03:49.920 --> 00:03:55.780 to be done with electric machines instead of internal combustion engines is 00:03:55.780 --> 00:03:57.620 a real positive contribution. 00:03:57.620 --> 00:04:02.860 But in my mind, it also sounds very daunting to start such an endeavor, 00:04:02.860 --> 00:04:07.540 because you deal with a lot of moving parts, 00:04:07.860 --> 00:04:13.780 you deal with potentially safety-critical systems, you want to make sure not 00:04:13.780 --> 00:04:18.300 to harm anyone, nor the environment. How do you navigate that space? 00:04:18.300 --> 00:04:21.900 Yeah, the design of the machine is a big part of it. 00:04:22.060 --> 00:04:26.080 I mentioned earlier that it's a fully capable manual machine. 00:04:26.220 --> 00:04:30.540 It has manual controls. You can jump on board and start mowing the grass yourself. 00:04:31.120 --> 00:04:37.760 And that was a really clever strategy to allow us to approach the autonomy problem incrementally. 00:04:38.240 --> 00:04:41.900 We could, in our early stages, as our autonomy stack was maturing, 00:04:42.480 --> 00:04:47.240 essentially see it as a mixed autonomy problem where the robot would handle 00:04:47.240 --> 00:04:51.340 the big, bulky, easy parts of the job, just the center of the field where you 00:04:51.340 --> 00:04:53.040 just have to go back and forth a bunch of times. 00:04:53.890 --> 00:04:59.570 And the parts of the mowing job that are more difficult, the edges, 00:05:00.050 --> 00:05:04.450 the obstacles, crinkly bits around the corners, we could say, 00:05:04.570 --> 00:05:07.150 all right, well, this is a place where humans do a better job. 00:05:07.250 --> 00:05:10.410 And so humans can take over and mow those areas. 00:05:10.710 --> 00:05:14.370 And that gives us kind of a continuous space to refine our autonomy. 00:05:14.830 --> 00:05:19.110 We start out saying, okay, we take the easy middle. And as we got better every 00:05:19.110 --> 00:05:22.710 year, we start saying, okay, actually, we can take over more and more of these 00:05:22.710 --> 00:05:23.950 tricky bits around the edges. 00:05:24.410 --> 00:05:29.150 The company is based in Colorado, which means there's a lot of skiers on the team. 00:05:29.310 --> 00:05:33.330 And so we have a skiing-based metaphor for difficulty. 00:05:33.550 --> 00:05:36.910 We can look at a field and say, oh, this is a green circle, or this is a blue 00:05:36.910 --> 00:05:38.370 square, or this is a black diamond. 00:05:38.370 --> 00:05:41.530 So we we talk about our 00:05:41.530 --> 00:05:47.430 our progress on autonomy and saying okay this year we moved from doing 80 of 00:05:47.430 --> 00:05:52.430 the the blue squares to 95 of the blue squares that's that's a big step forward 00:05:52.430 --> 00:05:56.890 in our autonomy and so we're chipping away at the gradient of of autonomous 00:05:56.890 --> 00:06:01.430 difficulty one step at a time would. 00:06:01.430 --> 00:06:06.310 You describe scythe as a software company a hardware company a robotics company 00:06:06.310 --> 00:06:07.610 or something else entirely. 00:06:07.610 --> 00:06:10.670 I think it is a robotics company in 00:06:10.670 --> 00:06:14.710 that the hardware and the software teams both 00:06:14.710 --> 00:06:18.170 have an essential contribution and they have to pull together the 00:06:18.170 --> 00:06:21.990 the designs are are closely coupled the 00:06:21.990 --> 00:06:24.970 software we build relies on the information that 00:06:24.970 --> 00:06:28.250 we gather from the sensors that the hardware team chose 00:06:28.250 --> 00:06:31.850 to to install on the machine and so 00:06:31.850 --> 00:06:35.310 when the hardware team designs the next 00:06:35.310 --> 00:06:39.990 generation of a robot the software team gets consulted to say okay well what 00:06:39.990 --> 00:06:45.790 what areas are are most difficult for this generation how can the next generation 00:06:45.790 --> 00:06:50.230 make the problem different the software problem different so that it is more 00:06:50.230 --> 00:06:53.330 amenable to to be solved the. 00:06:54.810 --> 00:07:00.330 The nature of design cycles does give it this back and forth character. 00:07:01.170 --> 00:07:06.470 Hardware designs on annual cycle, if you're lucky, if you're lucky, 00:07:06.470 --> 00:07:09.350 you can manage a one-year turnaround on your hardware designs. 00:07:09.550 --> 00:07:14.610 We work very, very hard to essentially get that one-year turnaround on our hardware designs. 00:07:14.970 --> 00:07:18.530 Whereas software is so much more protean. 00:07:18.730 --> 00:07:27.790 It moves so much faster. We can release software on timescales of weeks as opposed to years. 00:07:28.090 --> 00:07:35.610 And so that makes the software design more adaptable and more flexible and allows 00:07:35.610 --> 00:07:44.450 us to be closer to an agile kind of methodology for our work than the hardware team has to. 00:07:44.450 --> 00:07:52.390 The hardware team must follow a more solid and heavyweight design process because 00:07:52.390 --> 00:07:56.890 the nature of their design commitments are so much higher impact. 00:07:57.350 --> 00:08:01.990 Yeah. And you said it yourself, that hardware is in the field, 00:08:02.410 --> 00:08:07.470 like literally in the field for a very long time, if you allow me the pun. 00:08:07.790 --> 00:08:11.430 That means you need to select that hardware really carefully. 00:08:11.670 --> 00:08:14.870 What's inside of these machines? What keeps them running? 00:08:15.250 --> 00:08:18.530 Speaking of the CPU, the sensors, everything that makes it tick. 00:08:20.210 --> 00:08:26.450 Most of our computation happens on embedded NVIDIA module, a Jetson module, 00:08:26.810 --> 00:08:31.630 which is a ARM CPU with a GPU. 00:08:31.930 --> 00:08:36.390 They share the same memory space, which is a very convenient architectural feature 00:08:36.390 --> 00:08:40.590 for handing data over between the GPU and the CPU. 00:08:41.820 --> 00:08:50.340 The rest of the embedded system are in-house designed PCBs, so different PCBs 00:08:50.340 --> 00:08:56.280 to control the motors, to control the various sensors and other peripherals. 00:08:56.500 --> 00:09:04.840 And each one has a microcontroller of some sort to govern a small selection of the hardware. 00:09:04.840 --> 00:09:07.840 The the battery is 00:09:07.840 --> 00:09:12.400 probably one of our definitely our heaviest component and and one of our one 00:09:12.400 --> 00:09:18.120 of our most expensive and carefully designed components because the choosing 00:09:18.120 --> 00:09:21.700 the battery defines the the runtime that you get in the field and that's extremely 00:09:21.700 --> 00:09:25.500 important to professional landscapers they want to be able to use this machine 00:09:25.500 --> 00:09:27.800 for many many hours during the day, 00:09:28.520 --> 00:09:31.540 the the motors the the chassis the the 00:09:31.540 --> 00:09:34.320 motors are are are sourced components that uh 00:09:34.320 --> 00:09:37.980 that that we we choose from carefully selected vendors the 00:09:37.980 --> 00:09:42.540 chassis and all of the the the physical instantiation of the machine that is 00:09:42.540 --> 00:09:49.920 all scythe designed and small run manufactured steel components that are that 00:09:49.920 --> 00:09:54.820 are produced on the on the production runs that we do wow. 00:09:54.820 --> 00:10:00.800 That's a lot of moving parts Yeah. About the Jetson component, 00:10:01.240 --> 00:10:07.560 is that a thing that even has Rust support, or would that be driven by, 00:10:07.840 --> 00:10:12.260 say, a C or C++ component, or is there some other way to drive it? 00:10:12.260 --> 00:10:19.860 Definitely the ecosystem's center of gravity is definitely C++. 00:10:20.200 --> 00:10:26.800 By default, you expect that an embedded component was, but the vendor provides... 00:10:27.780 --> 00:10:31.160 All of the vendor-supported files, all of the board support packages, 00:10:31.460 --> 00:10:34.540 all of the drivers are in C++ by default. 00:10:34.780 --> 00:10:40.220 And these are the boundaries where we have to do C++ to Rust interop. 00:10:40.400 --> 00:10:46.660 We basically expect that there's going to be a C++ driver kind of making the 00:10:46.660 --> 00:10:55.600 last connection to our peripherals or the video system or any other component on the board. 00:10:55.600 --> 00:11:03.640 And that the data will cross the border over from C++ into Rust where we do our business logic. 00:11:04.460 --> 00:11:10.720 Yeah, but to take a step back, just for context, did you start with a hybrid 00:11:10.720 --> 00:11:17.480 C++ and Rust codebase or did you start with, say, an existing C++ codebase and gradually oxidized it? 00:11:17.800 --> 00:11:21.760 Yes, it was a sudden and sharp decision. 00:11:21.840 --> 00:11:27.560 So it's a fun story. I think it actually is a story that really illustrates 00:11:27.560 --> 00:11:34.060 how your architecture choices are driven by what's most important to you at the time. 00:11:34.360 --> 00:11:39.060 So in 2018, when the company came together as a small number of engineers in 00:11:39.060 --> 00:11:43.660 a garage, the priority was to get 00:11:43.660 --> 00:11:47.700 something working to show to potential investors as quickly as possible. 00:11:48.200 --> 00:11:52.440 And the robotics C++ ecosystem is very deep. 00:11:52.440 --> 00:12:00.580 And so the right choice in early 2018 was to put together a demo-worthy prototype 00:12:00.580 --> 00:12:05.460 in all C++, reusing as much open source code as possible, 00:12:06.140 --> 00:12:09.420 going quick, dirty, and scrappy, and making a machine that moved. 00:12:10.780 --> 00:12:16.180 The company got its first seed funding in late 2018, and that was the moment 00:12:16.180 --> 00:12:23.940 when their design priorities changed, and with it, their software ecosystem changed all at once. 00:12:24.120 --> 00:12:26.900 They essentially threw everything out, everything that they had built, 00:12:27.160 --> 00:12:30.600 just in the garbage, blank page, starting from scratch. 00:12:31.040 --> 00:12:35.780 And their most important decision was to build something that would last, 00:12:35.960 --> 00:12:38.500 that would serve the company throughout its entire lifetime. 00:12:39.320 --> 00:12:44.360 And that was the point where they said that Rust was the best choice to base 00:12:44.360 --> 00:12:46.600 as much of the robot logic as possible. 00:12:47.400 --> 00:12:52.360 That Rust provided the kind of code quality that they were looking for, 00:12:52.780 --> 00:12:57.540 and along with the kind of performance and efficiency that they needed. 00:12:57.940 --> 00:13:02.780 And even at that point, so late 2018, early 2019, I'm saying they, 00:13:02.880 --> 00:13:06.200 of course, because I wasn't with the company at the time. These are the founders 00:13:06.200 --> 00:13:09.780 and the first engineers who were building something from scratch. 00:13:10.780 --> 00:13:20.700 They saw that the state of Rust to C++ interoperability was sufficiently mature and reliable, 00:13:20.740 --> 00:13:24.660 that they were confident that they could use whatever C++ component they needed 00:13:24.660 --> 00:13:26.600 on the periphery of the system, 00:13:26.880 --> 00:13:31.420 literally for peripherals, or to engage with the rest of the robot operating 00:13:31.420 --> 00:13:33.040 system, the Rust ecosystem. 00:13:33.680 --> 00:13:37.280 And that they were confident that they could make any Rust code they needed 00:13:37.280 --> 00:13:39.560 work inside that ecosystem. 00:13:40.640 --> 00:13:47.680 I find that so cool and I find it so inspiring because as soon as the developers got funding, 00:13:47.980 --> 00:13:53.780 they moved over or they looked for greener pastures on the Rust side, 00:13:54.020 --> 00:13:56.320 maybe, to build the thing right. 00:13:56.720 --> 00:14:01.440 But immediately someone might say, well, couldn't you do the same thing with C++? 00:14:01.760 --> 00:14:06.740 Because there are also long-term projects written in C++ and we use them every 00:14:06.740 --> 00:14:13.160 day. couldn't you just keep on building on top of c++ and not throw the code 00:14:13.160 --> 00:14:15.960 away that would have saved you time at least initially. 00:14:15.960 --> 00:14:18.500 Absolutely there are there 00:14:18.500 --> 00:14:21.820 are many successful robotics companies out there that are 00:14:21.820 --> 00:14:29.560 almost exclusively c++ i think that many of the the the those first engineers 00:14:29.560 --> 00:14:34.720 many of those first engineers had spent a lot of time writing c++ and had spent 00:14:34.720 --> 00:14:38.740 a lot of time getting burned by the same problems over and over. 00:14:39.020 --> 00:14:48.200 The same relatively simple mistakes that turn into an incredibly difficult to 00:14:48.200 --> 00:14:50.520 diagnose bug that takes a long, 00:14:50.660 --> 00:14:55.140 long time to root cause and find the original cause. 00:14:56.120 --> 00:15:01.980 And everyone who was in those decisions, All of the early engineers, 00:15:02.220 --> 00:15:07.200 all of the founders talk about a desire for code quality, 00:15:07.340 --> 00:15:13.300 a desire to be able to write confidently, knowing that what they were writing 00:15:13.300 --> 00:15:19.200 would not burn them in one of these almost stereotypical ways, 00:15:19.400 --> 00:15:23.420 one of these classic mistakes which leads to trouble down the road. 00:15:24.660 --> 00:15:30.540 Okay that means they were veterans c++ developers who had a background in the 00:15:30.540 --> 00:15:35.440 language they knew what they were doing they were able to cobble together a 00:15:35.440 --> 00:15:40.680 prototype to convince venture capitalists to invest so that's a really positive 00:15:40.680 --> 00:15:44.540 sign and yet those people even with their c++ experience, 00:15:46.160 --> 00:15:49.080 preferred to write newer components in rust. 00:15:49.080 --> 00:15:55.040 Absolutely and they would each of them would would would defend that decision 00:15:55.040 --> 00:15:59.660 passionately that and by staying with the company and by it by continuing to 00:15:59.660 --> 00:16:08.280 develop in this uh in in in this mixed style of of rust inside a a world of c plus they basically uh. 00:16:09.160 --> 00:16:14.000 Voted with their feet for the next year, for all the years they stayed with 00:16:14.000 --> 00:16:15.720 the company and continued to develop in that way. 00:16:16.320 --> 00:16:19.880 From my perspective, I joined the company two and a half years ago. 00:16:20.020 --> 00:16:22.100 And I joined not knowing Rust. 00:16:22.280 --> 00:16:25.260 I joined thinking that one day I would like to learn Rust. 00:16:25.540 --> 00:16:30.460 And so I was open to the idea. And when I was hired by Scythe, 00:16:30.680 --> 00:16:32.100 they essentially said, welcome to Scythe. 00:16:32.200 --> 00:16:35.100 We understand you don't know Rust. Very few people who join our company do. 00:16:35.800 --> 00:16:40.940 Here's your desk. Here's a handful of web links for resources where you can learn. 00:16:41.140 --> 00:16:45.740 We look forward to your first merge request. And that's the pattern that most 00:16:45.740 --> 00:16:47.960 software engineers who join Scythe follow. 00:16:48.460 --> 00:16:51.520 Almost none of our software engineers know about what they join. 00:16:51.740 --> 00:16:57.300 They almost all are robotics domain experts with experience in the robotics 00:16:57.300 --> 00:17:00.640 industry, typically in C++, although sometimes in other languages. 00:17:01.000 --> 00:17:07.860 And they join us with a willingness to learn. And there's an almost stereotypical 00:17:07.860 --> 00:17:13.100 pattern of joining the company, banging your head against the borrow checker, 00:17:13.660 --> 00:17:14.940 learning how to get past that, 00:17:15.640 --> 00:17:21.580 writing your first MR, your first merge request, and then going through the 00:17:21.580 --> 00:17:26.300 same sort of rust idiom conversations that everyone needs to go through when 00:17:26.300 --> 00:17:27.000 they're getting started. 00:17:27.000 --> 00:17:33.840 It's almost a predictable pattern of what your first MR will look like and the 00:17:33.840 --> 00:17:35.820 first style conversations we'll have. 00:17:36.760 --> 00:17:44.140 And then after a few iterations of feedback, people have made the transition. 00:17:44.380 --> 00:17:48.240 They're Rust developers now and they don't look back. 00:17:50.350 --> 00:17:55.910 I can relate to that. What were your main programming languages before you joined Scythe? 00:17:56.210 --> 00:18:00.970 I had done C and C++, particularly in my distant past in grad school. 00:18:01.170 --> 00:18:06.030 But a slightly idiosyncratic part of my journey is that my most recent job had 00:18:06.030 --> 00:18:09.730 been 10 years at a company that used Java for robotics. 00:18:10.190 --> 00:18:15.050 And so I was already coming from a slightly non-standard language. 00:18:15.410 --> 00:18:21.030 And that maybe contributed to my willingness to pull up stakes and head it over to Rust instead. 00:18:22.330 --> 00:18:28.230 And what were some of those stylistic, eureka moments that you had, given your background? 00:18:28.470 --> 00:18:33.030 One of the design patterns that happens in Java a lot, and it happens in other 00:18:33.030 --> 00:18:35.070 languages too, but Java particularly, 00:18:35.350 --> 00:18:40.010 is using an immutable data structure 00:18:40.010 --> 00:18:44.450 in order to signal that there is some need to keep data coherent. 00:18:44.750 --> 00:18:50.190 So a structure has a collection of fields, and it's important for those fields 00:18:50.190 --> 00:18:51.530 to stay consistent with each other. 00:18:51.750 --> 00:18:57.410 Messing with just one of them is likely to cause some sort of inconsistency or bug. 00:18:57.830 --> 00:19:02.790 And so in Java, you declare everything final, you don't give setters, 00:19:02.950 --> 00:19:07.810 and that's a clear signal that this data should be treated as a package. 00:19:08.270 --> 00:19:11.670 Unfortunately, when you do need to make changes, you often are making a lot 00:19:11.670 --> 00:19:16.630 of copies of this same data just so that you're allowed to make the legitimate 00:19:16.630 --> 00:19:20.270 changes you need to make through construction of a new copy. 00:19:21.550 --> 00:19:27.650 But I came to Scythe sort of with this pattern well ingrained in me saying to 00:19:27.650 --> 00:19:30.810 myself, when in doubt, a data structure should be immutable. 00:19:30.950 --> 00:19:33.230 You should have a good reason to be able to mess with it. 00:19:33.830 --> 00:19:39.370 And that was one of the early features of Rust that drew me in, 00:19:39.490 --> 00:19:44.850 because clearly these ideas of when you're allowed to change data versus when 00:19:44.850 --> 00:19:49.890 you have a shared reference and you basically can't mutate it, it's, 00:19:50.550 --> 00:19:54.350 That's the first thing that people notice when they start writing Rust. 00:19:54.970 --> 00:19:59.650 And then I noticed, oh, well, this language, this expressivity of saying, 00:19:59.810 --> 00:20:04.010 oh, I have a shared reference or an exclusive reference to this structure. 00:20:04.150 --> 00:20:06.290 I'm allowed to mutate it. Now I'm not. 00:20:06.710 --> 00:20:12.350 I'm allowed to take a shared reference and mutate this data without making a copy of it. 00:20:12.970 --> 00:20:18.370 And when I lose scope, when my exclusive reference goes out of scope, 00:20:18.550 --> 00:20:21.930 I don't have mutable access to this structure anymore. 00:20:22.850 --> 00:20:28.210 Suddenly, it was like the next level of this mutability-immutability concept 00:20:28.210 --> 00:20:29.610 that I've been working with. 00:20:30.610 --> 00:20:35.750 You could efficiently make changes when you need to in a way that very clearly 00:20:35.750 --> 00:20:38.450 signaled that you were the only one allowed to make these changes. 00:20:38.450 --> 00:20:42.890 And then give that access back and be confident that the data wouldn't change 00:20:42.890 --> 00:20:44.030 when you weren't looking at it. 00:20:45.090 --> 00:20:51.270 Okay. What I hear from you is that you value immutability now and explicitness. 00:20:52.070 --> 00:20:58.130 The notion that if you return a thing that is immutable, it will stay immutable. 00:20:58.370 --> 00:21:02.290 And also you do that explicitly. So you pass it back and then someone else can 00:21:02.290 --> 00:21:04.150 work with the data, but not really mutate it. 00:21:04.490 --> 00:21:08.790 I guess- This might be unique to robotics or not unique to robotics, 00:21:08.910 --> 00:21:14.990 but might be particular to robotics, that quite often you are working with some 00:21:14.990 --> 00:21:17.530 fact about the world, which is multidimensional. 00:21:17.810 --> 00:21:20.730 All of these things are true right now at this instant. 00:21:20.910 --> 00:21:25.510 And it's not really valid to talk about keeping 80% of these facts, 00:21:25.730 --> 00:21:26.930 but not 100% of the facts. 00:21:27.070 --> 00:21:29.190 I've made a package describing the world. 00:21:29.350 --> 00:21:33.350 You need to make your decision based on this package of information, this struct. 00:21:33.690 --> 00:21:39.450 Please do not take this struct apart and change any of the pieces because it's 00:21:39.450 --> 00:21:45.610 only true when it's all together that that kind of multi-dimensionality truth 00:21:45.610 --> 00:21:49.070 is i think a robotic specific feature. 00:21:50.020 --> 00:21:56.100 It reminds me of a talk that I saw the other day by Jon Gjengset about a type-safe 00:21:56.100 --> 00:22:00.620 spatial math library in Rust called Sguaba. We will link to it in the show notes. 00:22:00.820 --> 00:22:05.780 It is for locations of objects in space. 00:22:06.400 --> 00:22:12.040 And it feels like a lot of people use Rust for that specific purpose because 00:22:12.040 --> 00:22:16.220 of its safety guarantees, because of its expressive type system. 00:22:16.760 --> 00:22:22.240 It feels like you're kind of agreeing with this and you're also working towards that. 00:22:22.880 --> 00:22:26.700 Absolutely. And I think when you say the expressive type system, 00:22:27.000 --> 00:22:36.740 that is the door that opened up to the Rust way of seeing the world that I didn't 00:22:36.740 --> 00:22:37.980 know about when I joined Scythe. 00:22:38.100 --> 00:22:45.120 As a complete Rust novice, I did not see the value of an expressive type system. 00:22:45.480 --> 00:22:51.040 Now, I'm not a Java developer, but to me it feels like Java also has a very 00:22:51.040 --> 00:22:52.120 expressive type system. 00:22:52.320 --> 00:22:55.100 Couldn't you encode the same invariance with Java? 00:22:55.880 --> 00:23:02.420 It's absolutely true that Java has an expressive type system that can be used 00:23:02.420 --> 00:23:03.840 in a lot of different ways. 00:23:04.560 --> 00:23:13.700 I think Rust's decision to essentially not do inheritance or not make inheritance easy. 00:23:14.830 --> 00:23:24.370 Leads to the Rust-type system being used in different ways to much greater effect. 00:23:24.750 --> 00:23:36.290 The Java capability of classes inheriting from each other leads to a lot of 00:23:36.290 --> 00:23:39.390 elaboration and customization of a class. 00:23:39.470 --> 00:23:42.610 Oh, I want this, but I want it to act a little bit differently, 00:23:42.610 --> 00:23:45.030 So I'm going to inherit from it and make a few changes. 00:23:45.410 --> 00:23:50.530 For me, the first thing that I think about when I think about the Rust expressive 00:23:50.530 --> 00:24:00.250 type system is the expressive enums and data carrying variants of enums. 00:24:00.250 --> 00:24:07.010 And so Rust encourages you not to build deep, 00:24:07.210 --> 00:24:11.610 deep hierarchies of classes inheriting from each other and making everything 00:24:11.610 --> 00:24:15.570 more complicated as you go, until sometimes you don't quite understand what's 00:24:15.570 --> 00:24:17.850 happening at the bottom of this long chain of inheritance. 00:24:17.850 --> 00:24:24.170 Instead, Rust encourages you to make a single layer of hierarchy, 00:24:24.350 --> 00:24:30.190 a single enum with all of the possibilities enumerated alongside each other in parallel. 00:24:30.650 --> 00:24:36.230 And so you have a lot of expressivity and you can use it for a lot of things. 00:24:37.510 --> 00:24:41.030 But it doesn't get so deep so as to be obscure. 00:24:41.390 --> 00:24:45.410 You can usually trust that you only need to look one place in your Rust code 00:24:45.410 --> 00:24:48.470 base to understand the range of options which is available to you. 00:24:49.610 --> 00:24:55.550 Yeah. It almost feels like the Java ecosystem encourages inheritance. 00:24:56.190 --> 00:24:57.090 Oh, absolutely. 00:24:58.370 --> 00:25:01.090 And I think it's fair to 00:25:01.090 --> 00:25:09.390 say that Java developed to take maximum advantage of inheritance before there 00:25:09.390 --> 00:25:18.490 was a language design pushback to say maybe inheritance is making things more 00:25:18.490 --> 00:25:19.710 complicated than it's worth. 00:25:19.710 --> 00:25:28.090 And I think you can see that those design ideas coming out a little bit after 00:25:28.090 --> 00:25:31.230 Java's maximum flourishing and growth. 00:25:31.370 --> 00:25:37.770 When Java settled down and became a stable sort of industry background choice, 00:25:38.150 --> 00:25:45.270 after that there kind of was a counterreaction to inheritance being used as 00:25:45.270 --> 00:25:47.590 the design feature to solve all design problems. 00:25:49.230 --> 00:25:51.970 You came to scythe with all of 00:25:51.970 --> 00:25:58.530 that background and knowing that there might be some work on on the rust side 00:25:58.530 --> 00:26:05.390 maybe with hardware that maybe is not familiar to you sensors and so on it sounds 00:26:05.390 --> 00:26:10.910 like a very daunting challenge and you still signed with them, 00:26:11.760 --> 00:26:15.940 Knowing all of that, what was your thought process back then? 00:26:16.100 --> 00:26:19.620 Were you curious about what it was all about to work on that level? 00:26:19.860 --> 00:26:23.160 Were you curious about the project? What was the main driver for you? 00:26:23.480 --> 00:26:29.520 I was excited to learn. It was absolutely a daunting challenge for all the reasons you're describing. 00:26:29.740 --> 00:26:34.300 I knew that I was signing up to learn some new skills. 00:26:34.780 --> 00:26:38.300 I had been working with Java for 10 years. That's a point in your career where 00:26:38.300 --> 00:26:42.140 you say, okay, maybe this is where I've settled. Maybe this is who I am and 00:26:42.140 --> 00:26:43.340 this is my core competency. 00:26:44.460 --> 00:26:48.400 It, it felt adventurous. It felt like a bit like, a bit like jumping off a cliff, 00:26:48.440 --> 00:26:53.220 but I'm glad to say that I was jumping off the cliff into, into a wonderful, warm swimming pool. 00:26:53.360 --> 00:26:56.360 And it was, and the water was fine. And I greatly enjoyed the transition. 00:26:57.040 --> 00:26:59.260 Um, the, the. 00:27:00.350 --> 00:27:09.350 It was a chance to learn new things and to pick up old problems with new tools. 00:27:09.790 --> 00:27:18.750 The robotic algorithmic challenges that Scythe is facing is the same as the 00:27:18.750 --> 00:27:24.310 algorithmic challenges faced by many mobile robotics applications. You need to plan paths. 00:27:24.450 --> 00:27:27.890 You need to verify that your trajectories have no collisions. 00:27:27.890 --> 00:27:34.070 You need to sequence a collection of work in a schedule that makes sense. 00:27:34.330 --> 00:27:40.510 These are all problems that many robotics companies need to solve. 00:27:40.650 --> 00:27:46.990 And they always have to solve them using custom approaches. It's rare to get 00:27:46.990 --> 00:27:53.910 a clean solution that solves the entire class of path planning, for example. 00:27:54.970 --> 00:27:59.890 Domain specificity is almost always a feature of the problems faced in robotics. 00:28:00.110 --> 00:28:04.230 You really need to grapple with the very specific features of your application. 00:28:04.370 --> 00:28:08.390 Oh, I have a lawnmower. That means I'm trying to cover all the grass. 00:28:08.570 --> 00:28:11.550 I'm not interested in finding the shortest path from point A to point B. 00:28:11.730 --> 00:28:15.330 I'm interested in finding the path from point A to point B that covers the lawn, 00:28:15.510 --> 00:28:17.430 that gets all of the grass mowed. 00:28:19.470 --> 00:28:26.390 These kind of domain-specific features mean that a lot of implementations are 00:28:26.390 --> 00:28:31.470 always going to be in-house and whether you've chosen the right tool for the 00:28:31.470 --> 00:28:34.810 job or not changes your quality of life as a developer. 00:28:35.350 --> 00:28:42.490 I'll bring it back and I'll say that Rust has allowed us to solve many of these 00:28:42.490 --> 00:28:51.370 classic problems in our own way using code that that we trust that we got right the first time mm-hmm. 00:28:52.190 --> 00:28:59.730 That means you write it, you build it, and it runs more or less indefinitely without any problems? 00:29:00.210 --> 00:29:03.490 The joke is, if it compiles, it works. And I don't believe that. 00:29:03.670 --> 00:29:10.870 You can always make mistakes. But your mistakes will be like logic errors, not safety errors. 00:29:11.450 --> 00:29:13.570 Testing is a huge investment for us. 00:29:14.730 --> 00:29:19.710 Every feature we write, every piece of code we write, needs to be tested thoroughly 00:29:19.710 --> 00:29:22.370 in order to be trusted to work in the autonomous space. 00:29:22.570 --> 00:29:29.150 To move a 1,400-pound machine with spinning blades through space is a daunting challenge. 00:29:29.270 --> 00:29:35.450 We take it very seriously to verify that the code we write does the job. 00:29:36.090 --> 00:29:40.890 Every test is an investment. Every test has a cost. 00:29:41.330 --> 00:29:47.690 It is incredibly powerful to say that an entire class of bugs has been excluded 00:29:47.690 --> 00:29:51.230 through what the compiler does for you. 00:29:51.390 --> 00:29:56.510 You don't have to worry about running it for hours and hours and hours just 00:29:56.510 --> 00:29:59.370 to make sure you don't leak memory. The compiler did that. 00:29:59.990 --> 00:30:03.130 We need to worry about whether the robot turns right instead of turning left. 00:30:03.370 --> 00:30:06.270 That's the kind of mistake that the compiler won't catch for us. 00:30:06.270 --> 00:30:15.230 But fundamentals of the stability of the the embedded compute those are largely 00:30:15.230 --> 00:30:21.690 handled and we don't need to invest we don't need to focus our testing attention there. 00:30:22.890 --> 00:30:27.790 That's incredible to see tests as an investment this is what it should be because 00:30:27.790 --> 00:30:31.370 an investment has a potential payoff in the future it's. 00:30:31.370 --> 00:30:33.570 Absolutely and there's a there's 00:30:33.570 --> 00:30:40.790 a maturity process as your technology evolves, that in the early days, 00:30:41.720 --> 00:30:45.980 Testing is easy and a little bit discouraging because your robot runs successfully 00:30:45.980 --> 00:30:48.740 for five minutes and then encounters a fatal error. 00:30:49.080 --> 00:30:51.420 And so your cycles can be very, very fast. 00:30:51.780 --> 00:30:55.840 And relatively speaking, your testing investment is relatively light. 00:30:56.200 --> 00:30:59.840 Five minutes of work gets you a new bug and off you go and you've got something to work on that day. 00:31:00.300 --> 00:31:08.060 As your technology matures, as you dial your system in, suddenly you have to 00:31:08.060 --> 00:31:11.620 test for long, long periods of time before you find something actionable. 00:31:11.720 --> 00:31:17.100 And so you're putting in hours and hours and hours in the field to find one 00:31:17.100 --> 00:31:20.700 piece of information which you can take back. And maybe it's no longer a fatal error. 00:31:20.860 --> 00:31:23.320 Maybe we're not even talking about the robot stopping. 00:31:23.540 --> 00:31:27.240 Maybe we're just talking about the robot doing something that we would prefer it not do. 00:31:27.240 --> 00:31:33.780 And so the amount of time you need to put in to find, oh, statistically, 00:31:34.440 --> 00:31:39.900 we're not quite hitting all of these cases exactly the way we would like. 00:31:39.900 --> 00:31:45.520 We would rather lift this percentage from 40% up to 70%, please. 00:31:45.660 --> 00:31:48.980 But it took us 200 hours of testing to gather that information. 00:31:49.720 --> 00:31:53.960 You know you're doing your job right when your testing budget starts getting 00:31:53.960 --> 00:31:58.480 very, very large. because you need so much time to gain statistical power on 00:31:58.480 --> 00:32:00.060 the features you're trying to investigate. 00:32:00.820 --> 00:32:07.240 Yeah. There's always this point in every project where you make a change and 00:32:07.240 --> 00:32:12.040 it breaks and then you take a step back and you realize, no, in fact, 00:32:12.280 --> 00:32:19.120 the system prevented a bug here and the system is more robust than I thought it was already. 00:32:19.600 --> 00:32:22.440 It's always very enlightening. 00:32:22.440 --> 00:32:26.080 And that is somewhere where Rust shines. 00:32:26.440 --> 00:32:35.580 The general Rust pattern of non-exclusive coverage being a compile time error 00:32:35.580 --> 00:32:39.500 saves so much time and effort. 00:32:40.620 --> 00:32:46.140 I think this is another one of those patterns, which are particular to robotics, 00:32:46.780 --> 00:32:50.120 is highly coupled state across components. 00:32:50.480 --> 00:32:56.320