Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: My name is Matthias Endler from Corode, and today I'm talking to Andrew Tinka
Matthias: from Scythe about grassroots robotics with Rust.
Matthias: Andrew, thanks so much for taking the time today for the interview.
Matthias: Can you please say a few words about yourself and about Scythe?
Andrew: Absolutely. My name is Andrew Tinka. I'm the Director of Software Engineering at Scythe Robotics.
Andrew: Scythe designs and manufactures professional-grade autonomous lawnmowers for
Andrew: the commercial landscaping industry.
Andrew: So when I say autonomous lawnmowers, many people think of a Roomba for your front lawn.
Andrew: What we build is larger and more heavy-duty.
Andrew: It is a 1,400-pound machine, roughly 650 kilograms. It is a fully capable stand-on lawnmower.
Andrew: So you can watch crews use this machine as a fully capable electric manual lawnmower,
Andrew: or they can put it to work to autonomously mow large areas.
Andrew: The value that Scythe brings is to...
Andrew: Increase the efficiency of taking care of green spaces, parks,
Andrew: athletic fields, corporate campuses, large areas of grass where mowing it all
Andrew: manually is not a great use of people's time.
Andrew: Ideally, a landscaping crew would get more done with the same number of people
Andrew: because while the lawn mowing is happening,
Andrew: the rest of the crew is handling the string trimming and the weeding and all
Andrew: of the other more manually involved jobs around the property,
Andrew: while the bulk of the just mowing the grass is getting done by a robot.
Matthias: That means it's very much a heavy-duty machine and it's for industry use.
Andrew: Absolutely. Our ideal customer is a commercial landscaper who is serving a lot of contracts.
Andrew: And so they're mowing 40 hours a week. They're loading these machines up and
Andrew: going from one property to another, taking care of a park in the morning,
Andrew: taking care of a big field in the afternoon, and doing this work every day throughout the week.
Matthias: Yeah. You get really good value out of it when you use that machine as often
Matthias: as possible, ideally throughout the workday. throughout the week.
Matthias: And for how long has Scythe been in existence?
Andrew: Scythe got started in 2018 as a classic garage startup.
Andrew: Just a few engineers in a small space putting together the first prototype.
Andrew: And so since then, we have gone through several successive generations of our
Andrew: hardware, and our fleet has grown every year, and the number of customers we've
Andrew: served has been growing every years since 2018.
Matthias: And was the idea always to have an automated robot? Was that the idea?
Andrew: Absolutely. The founding principle of the company was to use robotics to help
Andrew: take care of the earth better.
Andrew: And so there's a couple of ways in which that decision played out.
Andrew: The first one was to find the activities that people do to take care of green
Andrew: spaces and see which ones are a place where a robot can bring value.
Andrew: And the second aspect of that decision is our commitment to build all electric machines.
Andrew: Gas-powered landscaping equipment
Andrew: is often uniquely polluting even for internal combustion engines.
Andrew: And so bringing a valid replacement to the market to allow these kind of operations
Andrew: to be done with electric machines instead of internal combustion engines is
Andrew: a real positive contribution.
Matthias: But in my mind, it also sounds very daunting to start such an endeavor,
Matthias: because you deal with a lot of moving parts,
Matthias: you deal with potentially safety-critical systems, you want to make sure not
Matthias: to harm anyone, nor the environment. How do you navigate that space?
Andrew: Yeah, the design of the machine is a big part of it.
Andrew: I mentioned earlier that it's a fully capable manual machine.
Andrew: It has manual controls. You can jump on board and start mowing the grass yourself.
Andrew: And that was a really clever strategy to allow us to approach the autonomy problem incrementally.
Andrew: We could, in our early stages, as our autonomy stack was maturing,
Andrew: essentially see it as a mixed autonomy problem where the robot would handle
Andrew: the big, bulky, easy parts of the job, just the center of the field where you
Andrew: just have to go back and forth a bunch of times.
Andrew: And the parts of the mowing job that are more difficult, the edges,
Andrew: the obstacles, crinkly bits around the corners, we could say,
Andrew: all right, well, this is a place where humans do a better job.
Andrew: And so humans can take over and mow those areas.
Andrew: And that gives us kind of a continuous space to refine our autonomy.
Andrew: We start out saying, okay, we take the easy middle. And as we got better every
Andrew: year, we start saying, okay, actually, we can take over more and more of these
Andrew: tricky bits around the edges.
Andrew: The company is based in Colorado, which means there's a lot of skiers on the team.
Andrew: And so we have a skiing-based metaphor for difficulty.
Andrew: We can look at a field and say, oh, this is a green circle, or this is a blue
Andrew: square, or this is a black diamond.
Andrew: So we we talk about our
Andrew: our progress on autonomy and saying okay this year we moved from doing 80 of
Andrew: the the blue squares to 95 of the blue squares that's that's a big step forward
Andrew: in our autonomy and so we're chipping away at the gradient of of autonomous
Andrew: difficulty one step at a time would.
Matthias: You describe scythe as a software company a hardware company a robotics company
Matthias: or something else entirely.
Andrew: I think it is a robotics company in
Andrew: that the hardware and the software teams both
Andrew: have an essential contribution and they have to pull together the
Andrew: the designs are are closely coupled the
Andrew: software we build relies on the information that
Andrew: we gather from the sensors that the hardware team chose
Andrew: to to install on the machine and so
Andrew: when the hardware team designs the next
Andrew: generation of a robot the software team gets consulted to say okay well what
Andrew: what areas are are most difficult for this generation how can the next generation
Andrew: make the problem different the software problem different so that it is more
Andrew: amenable to to be solved the.
Andrew: The nature of design cycles does give it this back and forth character.
Andrew: Hardware designs on annual cycle, if you're lucky, if you're lucky,
Andrew: you can manage a one-year turnaround on your hardware designs.
Andrew: We work very, very hard to essentially get that one-year turnaround on our hardware designs.
Andrew: Whereas software is so much more protean.
Andrew: It moves so much faster. We can release software on timescales of weeks as opposed to years.
Andrew: And so that makes the software design more adaptable and more flexible and allows
Andrew: us to be closer to an agile kind of methodology for our work than the hardware team has to.
Andrew: The hardware team must follow a more solid and heavyweight design process because
Andrew: the nature of their design commitments are so much higher impact.
Matthias: Yeah. And you said it yourself, that hardware is in the field,
Matthias: like literally in the field for a very long time, if you allow me the pun.
Matthias: That means you need to select that hardware really carefully.
Matthias: What's inside of these machines? What keeps them running?
Matthias: Speaking of the CPU, the sensors, everything that makes it tick.
Andrew: Most of our computation happens on embedded NVIDIA module, a Jetson module,
Andrew: which is a ARM CPU with a GPU.
Andrew: They share the same memory space, which is a very convenient architectural feature
Andrew: for handing data over between the GPU and the CPU.
Andrew: The rest of the embedded system are in-house designed PCBs, so different PCBs
Andrew: to control the motors, to control the various sensors and other peripherals.
Andrew: And each one has a microcontroller of some sort to govern a small selection of the hardware.
Andrew: The the battery is
Andrew: probably one of our definitely our heaviest component and and one of our one
Andrew: of our most expensive and carefully designed components because the choosing
Andrew: the battery defines the the runtime that you get in the field and that's extremely
Andrew: important to professional landscapers they want to be able to use this machine
Andrew: for many many hours during the day,
Andrew: the the motors the the chassis the the
Andrew: motors are are are sourced components that uh
Andrew: that that we we choose from carefully selected vendors the
Andrew: chassis and all of the the the physical instantiation of the machine that is
Andrew: all scythe designed and small run manufactured steel components that are that
Andrew: are produced on the on the production runs that we do wow.
Matthias: That's a lot of moving parts Yeah. About the Jetson component,
Matthias: is that a thing that even has Rust support, or would that be driven by,
Matthias: say, a C or C++ component, or is there some other way to drive it?
Andrew: Definitely the ecosystem's center of gravity is definitely C++.
Andrew: By default, you expect that an embedded component was, but the vendor provides...
Andrew: All of the vendor-supported files, all of the board support packages,
Andrew: all of the drivers are in C++ by default.
Andrew: And these are the boundaries where we have to do C++ to Rust interop.
Andrew: We basically expect that there's going to be a C++ driver kind of making the
Andrew: last connection to our peripherals or the video system or any other component on the board.
Andrew: And that the data will cross the border over from C++ into Rust where we do our business logic.
Matthias: Yeah, but to take a step back, just for context, did you start with a hybrid
Matthias: C++ and Rust codebase or did you start with, say, an existing C++ codebase and gradually oxidized it?
Andrew: Yes, it was a sudden and sharp decision.
Andrew: So it's a fun story. I think it actually is a story that really illustrates
Andrew: how your architecture choices are driven by what's most important to you at the time.
Andrew: So in 2018, when the company came together as a small number of engineers in
Andrew: a garage, the priority was to get
Andrew: something working to show to potential investors as quickly as possible.
Andrew: And the robotics C++ ecosystem is very deep.
Andrew: And so the right choice in early 2018 was to put together a demo-worthy prototype
Andrew: in all C++, reusing as much open source code as possible,
Andrew: going quick, dirty, and scrappy, and making a machine that moved.
Andrew: The company got its first seed funding in late 2018, and that was the moment
Andrew: when their design priorities changed, and with it, their software ecosystem changed all at once.
Andrew: They essentially threw everything out, everything that they had built,
Andrew: just in the garbage, blank page, starting from scratch.
Andrew: And their most important decision was to build something that would last,
Andrew: that would serve the company throughout its entire lifetime.
Andrew: And that was the point where they said that Rust was the best choice to base
Andrew: as much of the robot logic as possible.
Andrew: That Rust provided the kind of code quality that they were looking for,
Andrew: and along with the kind of performance and efficiency that they needed.
Andrew: And even at that point, so late 2018, early 2019, I'm saying they,
Andrew: of course, because I wasn't with the company at the time. These are the founders
Andrew: and the first engineers who were building something from scratch.
Andrew: They saw that the state of Rust to C++ interoperability was sufficiently mature and reliable,
Andrew: that they were confident that they could use whatever C++ component they needed
Andrew: on the periphery of the system,
Andrew: literally for peripherals, or to engage with the rest of the robot operating
Andrew: system, the Rust ecosystem.
Andrew: And that they were confident that they could make any Rust code they needed
Andrew: work inside that ecosystem.
Matthias: I find that so cool and I find it so inspiring because as soon as the developers got funding,
Matthias: they moved over or they looked for greener pastures on the Rust side,
Matthias: maybe, to build the thing right.
Matthias: But immediately someone might say, well, couldn't you do the same thing with C++?
Matthias: Because there are also long-term projects written in C++ and we use them every
Matthias: day. couldn't you just keep on building on top of c++ and not throw the code
Matthias: away that would have saved you time at least initially.
Andrew: Absolutely there are there
Andrew: are many successful robotics companies out there that are
Andrew: almost exclusively c++ i think that many of the the the those first engineers
Andrew: many of those first engineers had spent a lot of time writing c++ and had spent
Andrew: a lot of time getting burned by the same problems over and over.
Andrew: The same relatively simple mistakes that turn into an incredibly difficult to
Andrew: diagnose bug that takes a long,
Andrew: long time to root cause and find the original cause.
Andrew: And everyone who was in those decisions, All of the early engineers,
Andrew: all of the founders talk about a desire for code quality,
Andrew: a desire to be able to write confidently, knowing that what they were writing
Andrew: would not burn them in one of these almost stereotypical ways,
Andrew: one of these classic mistakes which leads to trouble down the road.
Matthias: Okay that means they were veterans c++ developers who had a background in the
Matthias: language they knew what they were doing they were able to cobble together a
Matthias: prototype to convince venture capitalists to invest so that's a really positive
Matthias: sign and yet those people even with their c++ experience,
Matthias: preferred to write newer components in rust.
Andrew: Absolutely and they would each of them would would would defend that decision
Andrew: passionately that and by staying with the company and by it by continuing to
Andrew: develop in this uh in in in this mixed style of of rust inside a a world of c plus they basically uh.
Andrew: Voted with their feet for the next year, for all the years they stayed with
Andrew: the company and continued to develop in that way.
Andrew: From my perspective, I joined the company two and a half years ago.
Andrew: And I joined not knowing Rust.
Andrew: I joined thinking that one day I would like to learn Rust.
Andrew: And so I was open to the idea. And when I was hired by Scythe,
Andrew: they essentially said, welcome to Scythe.
Andrew: We understand you don't know Rust. Very few people who join our company do.
Andrew: Here's your desk. Here's a handful of web links for resources where you can learn.
Andrew: We look forward to your first merge request. And that's the pattern that most
Andrew: software engineers who join Scythe follow.
Andrew: Almost none of our software engineers know about what they join.
Andrew: They almost all are robotics domain experts with experience in the robotics
Andrew: industry, typically in C++, although sometimes in other languages.
Andrew: And they join us with a willingness to learn. And there's an almost stereotypical
Andrew: pattern of joining the company, banging your head against the borrow checker,
Andrew: learning how to get past that,
Andrew: writing your first MR, your first merge request, and then going through the
Andrew: same sort of rust idiom conversations that everyone needs to go through when
Andrew: they're getting started.
Andrew: It's almost a predictable pattern of what your first MR will look like and the
Andrew: first style conversations we'll have.
Andrew: And then after a few iterations of feedback, people have made the transition.
Andrew: They're Rust developers now and they don't look back.
Matthias: I can relate to that. What were your main programming languages before you joined Scythe?
Andrew: I had done C and C++, particularly in my distant past in grad school.
Andrew: But a slightly idiosyncratic part of my journey is that my most recent job had
Andrew: been 10 years at a company that used Java for robotics.
Andrew: And so I was already coming from a slightly non-standard language.
Andrew: And that maybe contributed to my willingness to pull up stakes and head it over to Rust instead.
Matthias: And what were some of those stylistic, eureka moments that you had, given your background?
Andrew: One of the design patterns that happens in Java a lot, and it happens in other
Andrew: languages too, but Java particularly,
Andrew: is using an immutable data structure
Andrew: in order to signal that there is some need to keep data coherent.
Andrew: So a structure has a collection of fields, and it's important for those fields
Andrew: to stay consistent with each other.
Andrew: Messing with just one of them is likely to cause some sort of inconsistency or bug.
Andrew: And so in Java, you declare everything final, you don't give setters,
Andrew: and that's a clear signal that this data should be treated as a package.
Andrew: Unfortunately, when you do need to make changes, you often are making a lot
Andrew: of copies of this same data just so that you're allowed to make the legitimate
Andrew: changes you need to make through construction of a new copy.
Andrew: But I came to Scythe sort of with this pattern well ingrained in me saying to
Andrew: myself, when in doubt, a data structure should be immutable.
Andrew: You should have a good reason to be able to mess with it.
Andrew: And that was one of the early features of Rust that drew me in,
Andrew: because clearly these ideas of when you're allowed to change data versus when
Andrew: you have a shared reference and you basically can't mutate it, it's,
Andrew: That's the first thing that people notice when they start writing Rust.
Andrew: And then I noticed, oh, well, this language, this expressivity of saying,
Andrew: oh, I have a shared reference or an exclusive reference to this structure.
Andrew: I'm allowed to mutate it. Now I'm not.
Andrew: I'm allowed to take a shared reference and mutate this data without making a copy of it.
Andrew: And when I lose scope, when my exclusive reference goes out of scope,
Andrew: I don't have mutable access to this structure anymore.
Andrew: Suddenly, it was like the next level of this mutability-immutability concept
Andrew: that I've been working with.
Andrew: You could efficiently make changes when you need to in a way that very clearly
Andrew: signaled that you were the only one allowed to make these changes.
Andrew: And then give that access back and be confident that the data wouldn't change
Andrew: when you weren't looking at it.
Matthias: Okay. What I hear from you is that you value immutability now and explicitness.
Matthias: The notion that if you return a thing that is immutable, it will stay immutable.
Matthias: And also you do that explicitly. So you pass it back and then someone else can
Matthias: work with the data, but not really mutate it.
Andrew: I guess- This might be unique to robotics or not unique to robotics,
Andrew: but might be particular to robotics, that quite often you are working with some
Andrew: fact about the world, which is multidimensional.
Andrew: All of these things are true right now at this instant.
Andrew: And it's not really valid to talk about keeping 80% of these facts,
Andrew: but not 100% of the facts.
Andrew: I've made a package describing the world.
Andrew: You need to make your decision based on this package of information, this struct.
Andrew: Please do not take this struct apart and change any of the pieces because it's
Andrew: only true when it's all together that that kind of multi-dimensionality truth
Andrew: is i think a robotic specific feature.
Matthias: It reminds me of a talk that I saw the other day by Jon Gjengset about a type-safe
Matthias: spatial math library in Rust called Sguaba. We will link to it in the show notes.
Matthias: It is for locations of objects in space.
Matthias: And it feels like a lot of people use Rust for that specific purpose because
Matthias: of its safety guarantees, because of its expressive type system.
Matthias: It feels like you're kind of agreeing with this and you're also working towards that.
Andrew: Absolutely. And I think when you say the expressive type system,
Andrew: that is the door that opened up to the Rust way of seeing the world that I didn't
Andrew: know about when I joined Scythe.
Andrew: As a complete Rust novice, I did not see the value of an expressive type system.
Matthias: Now, I'm not a Java developer, but to me it feels like Java also has a very
Matthias: expressive type system.
Matthias: Couldn't you encode the same invariance with Java?
Andrew: It's absolutely true that Java has an expressive type system that can be used
Andrew: in a lot of different ways.
Andrew: I think Rust's decision to essentially not do inheritance or not make inheritance easy.
Andrew: Leads to the Rust-type system being used in different ways to much greater effect.
Andrew: The Java capability of classes inheriting from each other leads to a lot of
Andrew: elaboration and customization of a class.
Andrew: Oh, I want this, but I want it to act a little bit differently,
Andrew: So I'm going to inherit from it and make a few changes.
Andrew: For me, the first thing that I think about when I think about the Rust expressive
Andrew: type system is the expressive enums and data carrying variants of enums.
Andrew: And so Rust encourages you not to build deep,
Andrew: deep hierarchies of classes inheriting from each other and making everything
Andrew: more complicated as you go, until sometimes you don't quite understand what's
Andrew: happening at the bottom of this long chain of inheritance.
Andrew: Instead, Rust encourages you to make a single layer of hierarchy,
Andrew: a single enum with all of the possibilities enumerated alongside each other in parallel.
Andrew: And so you have a lot of expressivity and you can use it for a lot of things.
Andrew: But it doesn't get so deep so as to be obscure.
Andrew: You can usually trust that you only need to look one place in your Rust code
Andrew: base to understand the range of options which is available to you.
Matthias: Yeah. It almost feels like the Java ecosystem encourages inheritance.
Andrew: Oh, absolutely.
Andrew: And I think it's fair to
Andrew: say that Java developed to take maximum advantage of inheritance before there
Andrew: was a language design pushback to say maybe inheritance is making things more
Andrew: complicated than it's worth.
Andrew: And I think you can see that those design ideas coming out a little bit after
Andrew: Java's maximum flourishing and growth.
Andrew: When Java settled down and became a stable sort of industry background choice,
Andrew: after that there kind of was a counterreaction to inheritance being used as
Andrew: the design feature to solve all design problems.
Matthias: You came to scythe with all of
Matthias: that background and knowing that there might be some work on on the rust side
Matthias: maybe with hardware that maybe is not familiar to you sensors and so on it sounds
Matthias: like a very daunting challenge and you still signed with them,
Matthias: Knowing all of that, what was your thought process back then?
Matthias: Were you curious about what it was all about to work on that level?
Matthias: Were you curious about the project? What was the main driver for you?
Andrew: I was excited to learn. It was absolutely a daunting challenge for all the reasons you're describing.
Andrew: I knew that I was signing up to learn some new skills.
Andrew: I had been working with Java for 10 years. That's a point in your career where
Andrew: you say, okay, maybe this is where I've settled. Maybe this is who I am and
Andrew: this is my core competency.
Andrew: It, it felt adventurous. It felt like a bit like, a bit like jumping off a cliff,
Andrew: but I'm glad to say that I was jumping off the cliff into, into a wonderful, warm swimming pool.
Andrew: And it was, and the water was fine. And I greatly enjoyed the transition.
Andrew: Um, the, the.
Andrew: It was a chance to learn new things and to pick up old problems with new tools.
Andrew: The robotic algorithmic challenges that Scythe is facing is the same as the
Andrew: algorithmic challenges faced by many mobile robotics applications. You need to plan paths.
Andrew: You need to verify that your trajectories have no collisions.
Andrew: You need to sequence a collection of work in a schedule that makes sense.
Andrew: These are all problems that many robotics companies need to solve.
Andrew: And they always have to solve them using custom approaches. It's rare to get
Andrew: a clean solution that solves the entire class of path planning, for example.
Andrew: Domain specificity is almost always a feature of the problems faced in robotics.
Andrew: You really need to grapple with the very specific features of your application.
Andrew: Oh, I have a lawnmower. That means I'm trying to cover all the grass.
Andrew: I'm not interested in finding the shortest path from point A to point B.
Andrew: I'm interested in finding the path from point A to point B that covers the lawn,
Andrew: that gets all of the grass mowed.
Andrew: These kind of domain-specific features mean that a lot of implementations are
Andrew: always going to be in-house and whether you've chosen the right tool for the
Andrew: job or not changes your quality of life as a developer.
Andrew: I'll bring it back and I'll say that Rust has allowed us to solve many of these
Andrew: classic problems in our own way using code that that we trust that we got right the first time mm-hmm.
Matthias: That means you write it, you build it, and it runs more or less indefinitely without any problems?
Andrew: The joke is, if it compiles, it works. And I don't believe that.
Andrew: You can always make mistakes. But your mistakes will be like logic errors, not safety errors.
Andrew: Testing is a huge investment for us.
Andrew: Every feature we write, every piece of code we write, needs to be tested thoroughly
Andrew: in order to be trusted to work in the autonomous space.
Andrew: To move a 1,400-pound machine with spinning blades through space is a daunting challenge.
Andrew: We take it very seriously to verify that the code we write does the job.
Andrew: Every test is an investment. Every test has a cost.
Andrew: It is incredibly powerful to say that an entire class of bugs has been excluded
Andrew: through what the compiler does for you.
Andrew: You don't have to worry about running it for hours and hours and hours just
Andrew: to make sure you don't leak memory. The compiler did that.
Andrew: We need to worry about whether the robot turns right instead of turning left.
Andrew: That's the kind of mistake that the compiler won't catch for us.
Andrew: But fundamentals of the stability of the the embedded compute those are largely
Andrew: handled and we don't need to invest we don't need to focus our testing attention there.
Matthias: That's incredible to see tests as an investment this is what it should be because
Matthias: an investment has a potential payoff in the future it's.
Andrew: Absolutely and there's a there's
Andrew: a maturity process as your technology evolves, that in the early days,
Andrew: Testing is easy and a little bit discouraging because your robot runs successfully
Andrew: for five minutes and then encounters a fatal error.
Andrew: And so your cycles can be very, very fast.
Andrew: And relatively speaking, your testing investment is relatively light.
Andrew: Five minutes of work gets you a new bug and off you go and you've got something to work on that day.
Andrew: As your technology matures, as you dial your system in, suddenly you have to
Andrew: test for long, long periods of time before you find something actionable.
Andrew: And so you're putting in hours and hours and hours in the field to find one
Andrew: piece of information which you can take back. And maybe it's no longer a fatal error.
Andrew: Maybe we're not even talking about the robot stopping.
Andrew: Maybe we're just talking about the robot doing something that we would prefer it not do.
Andrew: And so the amount of time you need to put in to find, oh, statistically,
Andrew: we're not quite hitting all of these cases exactly the way we would like.
Andrew: We would rather lift this percentage from 40% up to 70%, please.
Andrew: But it took us 200 hours of testing to gather that information.
Andrew: You know you're doing your job right when your testing budget starts getting
Andrew: very, very large. because you need so much time to gain statistical power on
Andrew: the features you're trying to investigate.
Matthias: Yeah. There's always this point in every project where you make a change and
Matthias: it breaks and then you take a step back and you realize, no, in fact,
Matthias: the system prevented a bug here and the system is more robust than I thought it was already.
Matthias: It's always very enlightening.
Andrew: And that is somewhere where Rust shines.
Andrew: The general Rust pattern of non-exclusive coverage being a compile time error
Andrew: saves so much time and effort.
Andrew: I think this is another one of those patterns, which are particular to robotics,
Andrew: is highly coupled state across components.
Andrew: We try to take our complete problem, make this robot drive autonomously.
Andrew: When we try to break it down, we try to decompose it into components in a stack.
Andrew: Each component is responsible for an aspect of the decision making.
Andrew: And ideally we would hope that those those
Andrew: components were very cleanly separated and the information
Andrew: they they share between them is extremely limited the reality is that you're
Andrew: always fighting against the tendency to to couple your data across your components
Andrew: severely that data is useful it's very important information to know what kind
Andrew: of mission you're doing so a task planning level concern.
Andrew: It's very useful to have that information down when you're deciding how the
Andrew: robot should move through space when you're making a trajectory decision.
Andrew: And so a large robotics code base tends to have long range coupling across components,
Andrew: even though architecturally we're fighting against that as hard as we can.
Andrew: But those couplings still exist.
Andrew: It's incredibly valuable that when you make a change in a data structure in one part of the system,
Andrew: the compiler catches the 10 errors you didn't think of because you had those
Andrew: long range couplings and now your trajectory planner needs to be rewritten to
Andrew: handle the change that you made to the task planner's data structure.
Andrew: If your data structure has changed and you haven't covered all the cases,
Andrew: it's an error, catches so many of these long range dependencies that otherwise
Andrew: would be difficult, that would be runtime errors instead of compile time.
Matthias: Now, wouldn't that be a property of a static type system, though?
Andrew: Honestly, I think it's more about idiom and design philosophy.
Andrew: So, for example, if you define a struct in Rust and you add a field,
Andrew: If somebody tries to construct that struct and they haven't provided that last
Andrew: new field because they didn't know about it, because this is one of those long
Andrew: range dependencies we were talking about.
Andrew: That is a compile time error unless you used the default annotation and unless
Andrew: you said that it's possible for fields to be filled in by default.
Andrew: So in Rust, it's possible to kind of remove this safety feature that if you
Andrew: haven't specified every field of your struct, you can't build it.
Andrew: But as the idiom if you say to yourself i would
Andrew: rather not use default when i when i don't have to because
Andrew: i kind of want to keep this feature of of like if
Andrew: i if i've forgotten one of my fields i want to know about it
Andrew: then uh then then you have that property it's
Andrew: this this example of of
Andrew: all fields being necessary is is just one of the of these kind of non-exhaustive
Andrew: non-exhaustive coverage means error patterns another one is is match statements
Andrew: or or pattern matching in general if your pattern matching isn't exhaustive
Andrew: rust will tell you about it that isn't true in other languages.
Matthias: Yeah yeah you could still use an underscore for a match
Matthias: case. But
Matthias: If i understand you correctly it's sort of an anti-pattern in a larger application
Matthias: that favors correctness.
Andrew: Absolutely and so it is
Andrew: fair to say that that that Rust doesn't doesn't
Andrew: do all of the work here for you that that like like you said there are there
Andrew: are there are ways to to
Andrew: have defaults or to to have the underscore match and they are it almost feels
Andrew: like like it's it's context whether it's appropriate or inappropriate there's
Andrew: plenty of times when it's fine to use underscore or to catch all the remaining
Andrew: cases in the match. That's fine.
Andrew: That becomes kind of company style or company culture almost where you are encouraged
Andrew: or discouraged from using these kind of patterns.
Matthias: Yeah. Do you have a coding guideline at Scythe?
Matthias: And how does the review process look like? Do you look at such patterns,
Matthias: tell people about it and tell them why it's a bad thing?
Andrew: That would be an aspect of our growth that I'd love to see. We tend to hold
Andrew: these kinds of styles as more of a culture than an explicit style guide.
Andrew: Clippy gets us halfway there, by the way. Clippy has so many requirements,
Andrew: so many good requirements, so many solid requirements that are enforced in our tool chain.
Andrew: So it's not true that it's completely a free-for-all.
Andrew: It is more about our code review practices that we say to ourselves,
Andrew: okay, when we review code, we all know that we prefer to see things written this way.
Andrew: We never use any of the functions which can plausibly panic.
Andrew: Don't unwrap. Unwrap or else. Those sort of choices.
Andrew: There are code bases, there are circumstances where panicking under unforeseen
Andrew: conditions is entirely the appropriate thing to do.
Andrew: We prefer not to do it because we would rather not panic our robot process as
Andrew: it is trying to decide whether to go left or right.
Matthias: Earlier, you said that the times to find a bug tend to become longer and longer,
Matthias: the more robust the system becomes.
Matthias: What are some of the problems that you find, if you allow me to put in the field with that robot?
Matthias: It's probably, is it more of a business logic problem or a systems problem or
Matthias: a domain problem? What is it that you find?
Andrew: We are always striving to to improve
Andrew: the reliability of the of the
Andrew: machine expressed as the the
Andrew: number of interventions that that you need to make when the the the robot is
Andrew: in the field mowing and that's a function of the the difficulty of the the the
Andrew: job that you're you're doing i mentioned before about the progression from up
Andrew: the ski slope of of difficulty as we get better,
Andrew: our customers trust us and put our robot in more and more difficult situations.
Andrew: So that's the heart of our defect discovery through testing.
Andrew: We say to ourselves, we know that our customers this year are putting our robot
Andrew: through more difficult scenarios.
Andrew: Let's say, for example, with respect to slopes. Sloped fields are more difficult
Andrew: to do robot control in than nice flat green fields.
Andrew: And so we'll say to ourselves, okay, we are going to ensure that our selection
Andrew: of testing fields, the grass we mow in order to test, we're going to make sure
Andrew: it includes a lot of slopes.
Andrew: And so we're going to make our testing problem more difficult for ourselves.
Andrew: Okay, now we measure the number of times when we.
Andrew: When our robot trajectory goes too far away from
Andrew: nominal as you're trying to to turn on a
Andrew: slope and we say to ourselves well we want to bring that we we
Andrew: want to bring these number of defects down they may not even be defects that
Andrew: require any intervention and they certainly aren't aren't defects that lead
Andrew: to any kind of of safety incident it's just we want the robot to to track more
Andrew: solidly on on hills than it is already and we call it we call it a defect when
Andrew: when it exceeds these bounds.
Andrew: And okay, so we're now having one of these kind of trajectory defects every
Andrew: five hours of operation.
Andrew: Let's get that to one every 10 hours of operation.
Andrew: That's the kind of defect that we're chasing when we're on a mature autonomous robotic system.
Andrew: And it doesn't quite even fit into the rubric of code correctness anymore.
Andrew: Now this is more like capabilities and tuning. So we say, okay, we are going to.
Andrew: We are going to increase this parameter in order to emphasize tracking to the
Andrew: desired trajectory more solidly.
Andrew: Well, that has consequences. That leads to a probabilistic defect in other parts
Andrew: of the system because the robot is now tracking more or is more solidly tracking
Andrew: its desired trajectory.
Andrew: That means in a different scenario than the robots on a slope,
Andrew: when the robot is navigating around an unexpected obstacle, something different
Andrew: happens with its performance. And now we're in a trade-off space.
Andrew: Do we want to change our defects on slopes or do we want to change our defects
Andrew: when navigating around obstacles?
Andrew: And now it's a whole conversation about what's more important to us and what
Andrew: the customer will value more in terms of a reliable machine.
Matthias: I watched a talk from the Oxidize conference
Matthias: from 2024 recently and in
Matthias: there there was a company which described a navigation
Matthias: or a planning system for autonomous robots
Matthias: in a warehouse scenario where they
Matthias: had multiple small robots and they would find a common trajectory for all of
Matthias: them they used a system for deterministic testing they had an event log on each
Matthias: machine and And then they could replay the exact scenario that happened,
Matthias: which caused a deadlock.
Matthias: Is that a thing that you can apply to your domain as well?
Andrew: I love determinism so much. It is such a precious jewel to gain it when you have it.
Andrew: And it is so hard to achieve.
Andrew: That is an incredible investment which yields incredible rewards in robotics.
Andrew: I'm sorry to say that most of our systems do not have that fundamental determinism.
Andrew: First of all, the real world doesn't bring that kind of determinism.
Andrew: The variations in timing of events in the real world,
Andrew: the variations of how a wheeled robot on a slope behaves means that every real
Andrew: world scenario is a unique and unreproducible event.
Andrew: Even in purely simulation-based testing, determinism and reproducibility is
Andrew: very, very hard to achieve.
Andrew: We haven't achieved that in general on all of our simulators.
Andrew: There are many smaller components, unit testing and integration testing of isolated
Andrew: components, where we require and where we achieve that kind of determinism.
Andrew: Because it is an enormous asset in root-causing your problems.
Andrew: When you have a fully reproducible example that you are confident you can get back anytime you need.
Andrew: Eventually, you can take that scenario and make it one of your integration tests
Andrew: and come back to it to confirm that you've fixed the bug and it's stayed fixed
Andrew: for as long as your system exists. So I respect it.
Andrew: It's the kind of system feature which is very valuable. It takes a huge investment to get there.
Matthias: Yeah and and even then you might not be 100% sure if it's worth it to pay to
Matthias: pay that price because you might as well work on features at the same time or
Matthias: maybe absolutely find another way how to test it in in the real world yeah.
Andrew: The the reality of the of a startup with a short funding timeframe and a lean
Andrew: team of engineers is that you really have to make careful trade-off choices
Andrew: about which features you're going to invest in.
Andrew: I've sung the praises of deterministic simulation, and I really believe that we may get there one day.
Andrew: It's not the best thing for us to build right now.
Matthias: Yeah. And even before you do that, you probably want to invest in a good failover
Matthias: system, which you already have in place.
Matthias: You briefly mentioned the disengagement scenario where for example you might see that there's some,
Matthias: fundamental condition which is incorrect in the system so like an invariant
Matthias: and then you you want someone to take over so there's probably a way for you
Matthias: to say okay we can't handle the situation right now stop the engine and wait
Matthias: for manual intervention this.
Andrew: Is one of the the application specific advantages of the lawn mowing application in particular.
Andrew: You don't have that freedom if you are building an autonomous car that's moving
Andrew: down the freeway at 100 kilometers an hour.
Andrew: You don't have the freedom to say, oh, this situation is off nominal.
Andrew: I will simply disengage.
Andrew: Your path to safety is complicated.
Andrew: When you are mowing a lawn, your path to safety is short and assured. You just have to stop.
Andrew: If your system is, if your conditions are sufficiently off nominal,
Andrew: bring the drive motors to a stop, bring the blades to a stop and ask for help.
Andrew: Everything will be okay.
Andrew: Nobody is depending on you to get to the side of the road or to bring other
Andrew: hazardous components to a safe state.
Andrew: It's an application where safety is still critical, but there is a simple story
Andrew: to how to make the system safe.
Andrew: So, yeah, just to summarize, a deterministic system in simulation is incredibly
Andrew: valuable, but it's not the most valuable thing for us to build right away.
Andrew: The startup world means making hard implementation trade-offs as to what will
Andrew: be the most valuable thing for your customers to build next.
Andrew: And so our investments have gone to other places.
Matthias: Which brings us to today. What is the current state of the code base?
Matthias: How much of it is in Rust? How much of it is in C++?
Matthias: Do you still use ROS and to what extent?
Andrew: Yes. So the ROS is the middleware that is the backbone of the system.
Andrew: ROS stands for robot operating system, But it's best understood as inter-process
Andrew: communication, along with an ecosystem of tools for simulation,
Andrew: for observability, for injection, and other capabilities.
Andrew: So ROS moves the data around our system. Most of our perception stack is in C++.
Andrew: The ability to work with GPUs and CUDA in Rust is maturing.
Andrew: And there are exciting projects right now. But over the history of Scythe's
Andrew: development, that has been similar where C++ has stayed very strong.
Andrew: So most of our perception stack is in C++.
Andrew: That information gets moved from the perception processes over to the autonomy
Andrew: processes using ROS as the middleware.
Andrew: Almost all of our autonomy code is in Rust. so that
Andrew: is every decision we make once we have
Andrew: assembled our perception of the world our our
Andrew: task planning we decide what we're deciding we're doing next at
Andrew: the high level our navigation are deciding
Andrew: where we're going to move and how we're going to get there and our
Andrew: trajectory and motor control so all the way down to deciding what
Andrew: torques should be applied to to the motors is is
Andrew: a decision being made in rust we favor an actor-based framework inside our Rust
Andrew: system to decompose the problem down into components that exchange messages
Andrew: as a way of prompting each other to make decisions or sharing information with each other.
Andrew: And so I've mentioned before that most of our software engineers join our company not knowing Rust.
Andrew: And they develop into application-level developers who write Rust to solve robot
Andrew: decision-making problems.
Andrew: There are also some engineers at our company who are deep Rust experts who showed
Andrew: up with very deep knowledge.
Andrew: And one of the parts of the system that they build,
Andrew: is the framework that supports these robot logic components.
Andrew: So the actor-based framework that I described.
Andrew: And so the actor-based system,
Andrew: its overall framework is written by our Rust systems experts and the actors themselves,
Andrew: the components that make the decisions are built by our robotics domain experts
Andrew: who are solid application-level Rust developers.
Matthias: Then that actor framework is not open source. That's probably in-house.
Andrew: It's an in-house actor framework. There are several open source actor frameworks
Andrew: in the Rust ecosystem right now.
Andrew: I think the choices that were available to us in 2019 when we were starting
Andrew: down our Rust journey were such that an in-house actor framework was the right choice for us.
Andrew: If we if scythe were being founded in a garage today in 2025 it's quite possible
Andrew: that many of these exclusively in-house components would instead be the open
Andrew: source rust crates because the ecosystem has has expanded and matured during
Andrew: the during the last six years.
Matthias: Yeah and if someone's listening
Matthias: and wondering what a modern active
Matthias: framework might look like in rust the one that i
Matthias: like a lot is Ractor we will link to it in the show notes it's i guess it leans
Matthias: into the Erlang model a bit and it has a lot of components in there that are
Matthias: really helpful for a production actor system for example uh,
Matthias: factory so being able to have multiple actors
Matthias: in a queuing system and supervision
Matthias: and things that you need once you run that thing at a larger scale but your
Matthias: actor framework specifically is that synchronous does it mean you you mostly
Matthias: write synchronous code where possible or do you also use tokio in combination with it.
Andrew: It's an asynchronous actor system but most of the time can we write synchronous code?
Andrew: So this is maybe one of those distinguishing characteristics between the deep
Andrew: rust expert and the robotics expert who's writing in rust today,
Andrew: is the comfort with asynchronous rust code.
Andrew: At the top level, our actor framework uses tokio to marshal all the actors together
Andrew: as asynchronous actors.
Andrew: But any asynchronous code can call synchronous code inside it.
Andrew: So the framework exposed to our robotics developers is handle message function
Andrew: call, which looks like a synchronous function. It is a synchronous function.
Andrew: It is being called from asynchronous code, but you don't need to worry about that.
Andrew: This is most often the right division of responsibilities.
Andrew: Asynchronous Rust code has some pitfalls. You need to be aware of cancel safety
Andrew: and many related issues in order to get the job done right.
Andrew: Many of the decision-making components in our collection of actors don't need
Andrew: to leverage the features that asynchronous Rust provides.
Andrew: Most of them can be thought of as a message handling system that gives quick
Andrew: answers to each incoming message.
Andrew: And you might as well write that as a piece of synchronous code and not concern
Andrew: yourself with the possible pitfalls of asynchronous Rust.
Andrew: I'm very glad that the folks on our team who built the actor framework are well-versed
Andrew: in all the pitfalls of asynchronous code.
Andrew: But it's a convenient decoupling of responsibilities to say that the asynchronous
Andrew: code exists only at the actor framework.
Andrew: And most individual actors are just synchronous.
Matthias: Mm-hmm.
Matthias: Gotta say, that was great foresight from the people that worked on the framework,
Matthias: because you don't have to deal with the complexities that you mentioned in the async Rust ecosystem.
Matthias: You mentioned cancellation specifically. Do you have an example for when that
Matthias: becomes relevant in your domain?
Andrew: For this i think i really have to just defer to
Andrew: an excellent talk at RustConf in August
Andrew: on cancel safety given by Rain where they
Andrew: ran through so many examples of
Andrew: how cancel safety is important and how it can
Andrew: go wrong i guess the the easiest pitfall
Andrew: that i can just cite off the top is in an
Andrew: actor-based system where control and
Andrew: and essential information is arriving in messages
Andrew: probably the easiest way to mess up a cancel
Andrew: safety issue is to cancel
Andrew: and have the message disappear forever or
Andrew: the message that you were handling disappear forever if that that message was
Andrew: a critical piece of information which will only arrive once well then you probably
Andrew: have caused something a serious problem by by never looking at that message again so So...
Andrew: If your actors are receiving unique, irreplaceable, must-be-handled messages,
Andrew: then you better get your cancel safety right when you dequeue those messages and
Andrew: hand them off to your message handler. That's the easiest example I've got.
Matthias: Yeah. And that helps with determinism, too.
Andrew: Absolutely. Yes. For sure. So yes, to expand on that a little bit,
Andrew: one of our most common testing strategies is to take an actor and feed it a
Andrew: test benched set of messages in a particular order at particular times,
Andrew: and then make the test dependent on whether the actor gives the right answers back.
Andrew: So an actor can be fully deterministic.
Andrew: Given the identical sequence of incoming messages, the actor will always give
Andrew: you the correct output, or you strive for that.
Andrew: It's only in the glorious complex composition of all of your actors against
Andrew: a real-world system that is triggering events at uncertain times that you gain
Andrew: the kind of indeterminism that makes it a big complicated problem.
Matthias: We're getting close to the end and the final question in this podcast is always
Matthias: your message to the rust community.
Andrew: Absolutely stage is yours when i
Andrew: when i think when i think about rust and when i
Andrew: think about the open source community that that built
Andrew: it and maintains it and and moves it forward today
Andrew: it's my feeling is one of of of gratitude this is an incredible thing that a
Andrew: large diverse community of people have built and rust as a language sits at
Andrew: a very valuable point in the language design space,
Andrew: really is something special in terms of the guarantees it gives,
Andrew: the confidence you can have when building complicated systems with it,
Andrew: knowing that the language has struck such a valuable trade-off between competing
Andrew: concerns of safety and efficiency and expressiveness.
Andrew: It's a technology i'm glad to use every day that i use it and i hope that its
Andrew: development toward these goals continues as long as it can.
Matthias: Couldn't have said it better. Andrew thank you so much for taking the time today
Matthias: to do the interview thank.
Andrew: You very much Matthias. It's been a pleasure.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me Matthias Endler
Matthias: and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.