Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we talk to Carter Schultz from
Matthias: AMP about recycling billions of items per year with Rust.
Matthias: Okay, welcome to the show, Carter. Can you quickly introduce yourself and the
Matthias: company AMP Robotics that you work for?
Carter: Yeah, so my name is Carter Schultz. I am the robotics architect at AMP Robotics,
Carter: and we are trying to solve the world's recycling problem as best as we possible
Carter: can by just throwing a lot of AI at it.
Carter: For the last 10 years, I've been writing an enormous amount of C++ code to make robots sort recycling.
Carter: And for the last three years, we've started writing a lot of Rust code,
Carter: and I'm very excited about it.
Matthias: When you say you try to solve the world's recycling problem like what is the scope that
Matthias: you target i just did some research and i learned that every american produces
Matthias: five pounds of garbage every single day or do you mostly tackle the american market right now yeah.
Carter: We actually have robots in europe asia and north america i think we have like
Carter: 13 countries or something.
Carter: So at least for our robotics business, we've kind of gone global with that.
Carter: But really, the mission of the company was to solve waste in general.
Carter: We wanted to fundamentally change humanity's relationship to waste.
Carter: And so we're going wherever we can and wherever we can find have leverage to
Carter: just make less waste be created and to get more and more recycled,
Carter: reused, repurposed effectively.
Carter: We have the corporate slides that we present to investors, but the The waste problem is real.
Carter: Humanity is throwing away hundreds of billions of dollars worth of value of
Carter: material that we invested in and spent resources to get to quality and purity,
Carter: and then we just throw it away.
Carter: And you want to talk about carbon sequestration options or ways to reduce energy
Carter: usage or just make the world better, there's a huge opportunity in waste.
Carter: And because it's not sexy and glamorous, not enough people are trying to solve it.
Matthias: I do think it's very sexy. I do think it's very important too.
Matthias: And a lot of people right now, engineers mostly, they look for meaning in their work.
Matthias: And I think you would be able to reach a lot of people that are interested in
Matthias: hard technical problems, because I assume that's what it is.
Matthias: How many pieces of material are we talking about?
Matthias: Pieces of recycling material that you try to sort with your systems?
Carter: So the stat we throw around a lot is in 2022, we sorted 5 billion items,
Carter: which is, you know, it sounds very impressive and I'm very proud of it.
Carter: It's still a very small percentage of the waste shed.
Carter: Like we're trying to get to trillions of items as quickly as we can.
Carter: And, you know, when we talk about like a recycling facility,
Carter: they're typically rated in tons per hour.
Carter: So tons of material that they take in at a moment. So we're building facilities
Carter: that are in the, you know, like 20 tons an hour range of input material that they can take in.
Carter: And that just means truckload after truckload after truckload in and out constantly.
Carter: So the scale kind of boggles the mind when you really start to deal with it.
Matthias: Where's the bottleneck? Is it a technical bottleneck? Do the robots not move fast enough?
Matthias: Do you just need to build bigger plants?
Carter: There's bottlenecks everywhere. And I think like there's no there's no part
Carter: of the process that's easy and there's no part of the process that's cheap.
Carter: One of the fundamental realities right now is that an enormous amount of the
Carter: money spent to make recycling happen is actually being spent on diesel fuel.
Carter: Recycling commodities have a very low bulk density, so proportional to the value
Carter: of the commodity, it's very expensive to ship it around.
Carter: And when you talk about the fuel it takes to send recycling trucks down every
Carter: street in America and pick up recycling bins and then bring that recycling to a facility to sort it,
Carter: and then trucks that take it from that facility to other facilities that repurpose
Carter: it, the amount of diesel that you can burn moving the material can often exceed
Carter: the value of the material.
Carter: Which is why a lot of people aren't the biggest fans of recycling and don't believe in it.
Carter: But when you can move the sortation facilities closer to the producers,
Carter: to the people producing the recycling, when you can move the repurposing facilities
Carter: closer and have more of them,
Carter: you cut that travel down substantially and can make the entire process a lot cheaper.
Carter: One of the big things that the industry has basically fought with for most of
Carter: the last century is that generally the processing plants that repurpose the
Carter: material and the sortation facilities that sort it have really good economies
Carter: of scale where when you build those facilities giant, they're much more efficient.
Carter: But when you build them giant, you have to truck the material a bigger distance to get them there.
Carter: And that's kind of been one of
Carter: the double-edged swords of the recycling industry for a really long time.
Matthias: Did you spend any engineering resources on making these facilities easier
Matthias: to build and maybe making them easier to ramp up locally where the garbage is located?
Carter: That's kind of the dream of our new technology and what we've been working on.
Carter: We just recently did a big unveiling, I think last week of this AMP ONE,
Carter: which is the design of our facilities.
Carter: We're actually like showing it out publicly to the world.
Carter: Up until now, it's kind of been our our trade secret as to how these facilities operate internally.
Carter: But we've come up with what we think is a very scalable, simple piece of technology
Carter: that can sort material that you can then compose into modular facilities of
Carter: various sizes and various effectiveness.
Carter: So we're doing everything we can to take the technology unlocks that we have
Carter: and figure out how to leverage them the most to make the biggest difference in the waste shed.
Matthias: When you explained that, I was wondering how much of it is a software and how
Matthias: much of it is a hardware problem.
Matthias: So do you see AMP Robotics as more of a software or more of a hardware company?
Carter: You know, at the start of the company, it was a software company.
Carter: What made AMP Robotics possible and how AMP Robotics got going was that our
Carter: CEO and founder realized that neural networks ability to identify materials
Carter: and classify them correctly was on an insane exponential curve of performance.
Carter: And so, you know, six years ago, we had a neural network that exceeded human
Carter: sorter accuracy at identifying recycling materials.
Carter: And that kind of turns the industry on its head a little bit where previously,
Carter: you know, the best technology for identifying recycling materials were hyperspectral
Carter: cameras that cost, you know, $100,000 for one camera.
Carter: And now with a cheap little GPU and, you know, effectively a webcam,
Carter: we can identify recycling better than that.
Carter: And with that added ability and ability to see things, a lot of things were unlocked.
Carter: So with that sort of core camera system, our first like real product we got
Carter: traction with was the pick and place robots that we put behind it that manually
Carter: did sorting. And those are good robots that do a good product and purpose in the world.
Carter: But when you have that core camera system that lets you just see everything
Carter: that's on the conveyor belt, you can actually do a lot of things else with that camera system.
Carter: So as time's gone on, that camera system's gotten to a very matured state where
Carter: we really understand how to make the neural networks see well.
Carter: And more and more of the company's investment has gone into the mechanical side
Carter: and in developing new hardware and new capabilities to extract value from that neural network unlock.
Matthias: Okay, so you move into hardware a lot.
Matthias: And is it also true that all of these pieces of recycling are different to handle
Matthias: or do you have a very scalable or maybe a very commoditized system to handle
Matthias: all of the different pieces of garbage?
Carter: So the best system that, you know, kind of existed before us and we're now just
Carter: repurposing in a new way are actually air jet sorters.
Carter: So most of what's sorting in our facilities are what we just call jets,
Carter: which is, you know, the material goes off the end of a conveyor belt.
Carter: It goes ballistic and it's free falling in air. And while it's falling in the
Carter: air at the end of the conveyor belt, you hit it with a puff of air.
Carter: That puff of air is incredibly good at like 99% of the materials that end up
Carter: in the recycling stream.
Carter: There's a small subset of stuff like phone books and frying pans that a puff
Carter: of air can't really handle.
Carter: But the vast majority of the sorting that's happening in one of our facilities
Carter: is being done actually by these air jet sorters.
Matthias: It's kind of funny because you started with this idea of building a software solution.
Matthias: What turned out to be mostly a hardware problem. So you went from building software
Matthias: for, you know, vision or object recognition, I guess, and then you transition to sorting robots.
Matthias: And now you think about sorting facilities, like AMP ONE, for example.
Matthias: Is that correct, first and foremost? And second, what's the next step?
Carter: Tip? Yeah, no, I think that's correct.
Carter: And it's a really good lesson, I think, in how to be successful as a startup,
Carter: which is you can't eat the whole sandwich in one go.
Carter: You have to take bite-sized chunks at it. And when you're a tiny little startup
Carter: just getting off the ground, hardware is really expensive to build.
Carter: It's really expensive to invest in.
Carter: So when you can focus on the software first, prove that you have some value
Carter: there and get a toehold in the market, that's a really nice way to grow.
Carter: And then when you have enough money to invest in building a good robotics order, you build that.
Carter: And that robotics order becomes part of the flywheel where now sales of those
Carter: robots fund investments into more technology that let you build more and more things.
Carter: So there's been this kind of turning flywheel of getting AMP going by just taking
Carter: off the next bite-sized chunk of the problem that we have the ability to actually tackle and solve.
Carter: The latest version of that has been building our own individual recycling facilities.
Carter: The next step honestly comes in the problem of scale.
Carter: We have a few facilities right now and it's like, how quickly can we get to
Carter: dozens and hundreds of facilities?
Carter: And when you actually look at everything it takes to build a facility,
Carter: boy, that's a hard thing to scale.
Carter: It takes a lot of work and a lot of technical know-how and a lot of people and
Carter: skills to figure out how to get buildings and permits and electrical runs and
Carter: cables and conveyor belts and trucks and logistics.
Carter: And, you know, the tech now, you know, the software that made everything possible
Carter: isn't the hard problem anymore.
Carter: It's the problem of building factories to build factories like that's that's really hard.
Matthias: Couldn't you license out a lot of that technology to other vendors?
Matthias: Say, I want to use your technology, but I'm not an engineer or maybe I'm not
Matthias: well-versed in building these systems.
Matthias: I would probably go to you and approach you and ask you, hey,
Matthias: can you just license your technology to me and I build one facility myself? Yeah.
Carter: Yeah, it's actually, you know, it's definitely something that's on the table.
Carter: And it's something we've experimented with in the past.
Carter: At the same time that AMP built its own robotics order, we actually did license
Carter: our vision system to another company that built a robotic system off of our vision system.
Carter: So like, we've done that in the past, and we've had some success with it.
Carter: The biggest challenge that we've really had is that the pace of innovation of
Carter: the vision technology has been so fast that it's really hard for people to keep up to date with it.
Carter: Like our neural nets are improving by 10 or 20 percent every year.
Carter: And the ways that they're improving are not obvious ways that fit in the existing
Carter: API of like the data that the neural net used to give you. It gives you new
Carter: and different data that you have to then process and think about in different
Carter: ways to extract the value from it.
Carter: So right now, when we're building these facilities, we are intending in general
Carter: to build and operate them ourselves.
Carter: And we believe that like our staff actually running the facility that's built
Carter: is the best way to get the value out of it for the time being,
Carter: because this technology is still really, really new.
Carter: Then we need people who are experts in it and are able to adapt to it changing
Carter: quickly at the helm of actually operating it.
Carter: Probably like five, 10 years from now when the technology hopefully settles
Carter: down a little bit and I still have my hair and haven't ripped it all out,
Carter: then it really becomes a commodity technology that everybody can have and everybody can use.
Carter: And we have standard manuals and part sheets, and it's just an off-the-shelf standard thing.
Carter: As it stands right now, every two years, it's evolving rapidly and dynamically.
Carter: One of the things that AMP struggles with a lot is six years ago when we started
Carter: doing vision-based neural networks, we were at the bleeding edge of it.
Carter: And it was almost impossible to get the neural networks that we were using working
Carter: in the ways we were getting them to work.
Carter: But every year that gets easier and that gets a lot easier and Google and NVIDIA
Carter: are spending billions of dollars to make it easier and easier and easier for
Carter: other companies to catch up to us.
Carter: So we can't just stop and say, oh yeah, no, the vision system's good enough and done.
Carter: We keep iterating on it as hard as we possibly can.
Matthias: Right. In order to keep your competitive edge.
Matthias: And I guess that ties into what I had in mind when I thought about the drawbacks
Matthias: of licensing, because one thing you can do with this very tight knit environment
Matthias: is that you have vertical integration.
Matthias: Integration that means you can iterate very quickly if
Matthias: you have an improvement in your neural network maybe you can tweak your system
Matthias: such that it sorts more or you have your robotic arm and you have better software
Matthias: for it you can update it right away and then maybe it's faster so like not arms
Matthias: but chats i mean but the entire thing kind of,
Matthias: it improves from within yeah it's there's a positive feedback cycle yeah.
Carter: And in particular Our vision system has unlocked that positive feedback cycle.
Carter: The vast majority of recycling facilities that exist right now,
Carter: how do they optimize how well that facility is working?
Carter: Some guy goes out on the factory floor and looks at the conveyor belts and says,
Carter: oh, well, they look pretty good today, I guess, doing a good job.
Carter: We don't have that. We have data. We have cameras on
Carter: every single conveyor belt in the facility simultaneously measuring how well
Carter: every piece of equipment is working and how the facility overall is performing
Carter: that allows us to live while we're running continuously adjust parameters in
Carter: the facility and in real time measure the effect of those changes on the sortation performance.
Carter: And like that is the secret sauce like that unlocks better recycling than anybody
Carter: has ever had and is the you know, the new amazing tech differentiator that we're really excited about.
Carter: And like while it's really easy for people to figure out, you know,
Carter: how to build a neural net to identify recycling. Well, not really easy,
Carter: but it's getting easier.
Carter: Figuring out how to auto-tune a recycling facility to self-optimize on the material it's processing.
Carter: There's no other way of doing that than by building a bunch of recycling facilities
Carter: and trying to do it, which is really expensive.
Carter: So it's building a technological moat that makes it harder for people to really
Carter: come contest our space in the market.
Matthias: What I hear from you is one word, disruption and
Matthias: radical integration and also maybe
Matthias: tackling problems from a completely different angle
Matthias: from a data from a data science angle maybe from
Matthias: an engineering angle and then i wonder
Matthias: you have so many things going on and maybe only
Matthias: so many innovation tokens to spend and now we're talking about rust which is
Matthias: yet another thing in this entire tool chain something that is maybe also So
Matthias: still on the verge of getting adapted or getting adopted on a broader sense.
Matthias: And do you think that it's too much innovation and maybe you overstretched a bit?
Matthias: Or would you say, no, actually, it's been pretty smooth sailing so far?
Carter: You know, we've definitely overreached in a few areas. Like not every product
Carter: that AMP has tried to build has kind of found a market fit and sold like gangbusters.
Carter: So like no path is perfectly smooth and easy. There have been plenty of bumps along the way.
Carter: I would not say Rust is one of those bumps.
Carter: I think my view here is very colored, but I'm incredibly satisfied with our choice of Rust.
Carter: And I'm, you know, we're doing everything we can to double down on that choice
Carter: and continue to invest more in it because we really did see immediate positive dividends from it.
Matthias: Did people immediately embrace Rust or did you have to do any convincing to
Matthias: move the team towards Rust?
Carter: You know, I got incredibly lucky here. And this is part of the advantage of
Carter: being a startup that's growing quickly.
Carter: So like, put yourselves in my shoes, wind back, I think three-ish years at this point.
Carter: And we decide we're going to build our first real big production facility.
Carter: We have a little tiny test facility that we built right by our headquarters
Carter: so that we could work out some of the kinks.
Carter: But this is like the first facility that's going to go straight into 24-7 operation.
Carter: I'm responsible for figuring out the control system for that facility.
Carter: What is actually going to turn the conveyor belts on and off?
Carter: What is going to coordinate all of the different camera systems together?
Carter: What is going to be the overall SCADA system, like the supervisory control?
Carter: For that facility? And how do we get that done? Oh, and by the way,
Carter: we want the facility to start running six months from now.
Carter: That was like a hard problem that I was given. On the flip side,
Carter: I was given, you know, kind of carte blanche, like any way I could figure out
Carter: to make that work was probably good enough.
Carter: I was hiring a new team to be the facility software controls teams.
Carter: Like I brought one engineer over from our robotic software side,
Carter: but I was able to hire a new team in the model of what I thought this new problem
Carter: needed. and it was completely greenfield.
Carter: Like we had a Python application that was actually running our little test facility
Carter: that we'd worked out some of the drivers and communication protocols.
Carter: And we had a toy example of a factory control system up, but we already knew
Carter: Python was not the right choice for it and was causing us problems in a few different ways.
Carter: And so from that model, like we already had a beta, it was just,
Carter: okay, build a facility control system as quickly as we can.
Carter: That is the right thing for us going forward. And I, at this point,
Carter: I'd been playing with Rust on the side for probably a year and a half or two years.
Carter: I was pretty convinced that Rust was a good contender for building something like this.
Carter: And I was also pretty convinced that the right thing for AMP as a company was
Carter: to invest in building this control system from scratch ourselves.
Carter: That like the ability to control the facility overall in really,
Carter: really fine detail with our own software that we completely own was going to
Carter: be a major technological advantage for us going forward.
Carter: And it was worth doing it ourselves, doing it exactly the way we wanted,
Carter: building an application that was bespoke for our needs and not cobbled together
Carter: with kind of the off the shelf industrial solutions that exist.
Matthias: You had six months you had to
Matthias: build this from scratch greenfield project i wonder when you hired people what
Matthias: background did you look for what sort of specialities did you need how did you
Matthias: try to integrate the team what were your metrics your kpis at that time.
Carter: Good questions. And it's a little bit of chaos.
Carter: I think we have to acknowledge the way that startups build things is not always
Carter: with the best plan, and it's a lot more firefighting.
Carter: So the requirements for what we needed to be built were not in any way written down at the start.
Carter: They were being discovered as we tried to make things work.
Carter: For backgrounds, I went for programmers that I thought were malleable,
Carter: that were willing to learn things, that were hungry for new experiences.
Carter: I looked a lot for programmers that had some kind of real-time-ish experience,
Carter: had worked on video games or worked on embedded applications where some connection
Carter: to a tight timing loop was pretty important.
Carter: And I looked for programmers that were willing to get their hands dirty with hardware.
Carter: We couldn't have... Coming from a robotics background,
Carter: programmers that just sit behind the keyboard all day are way less valuables
Carter: than programmers that are willing to go out there and debug the wiring and debug
Carter: the motor that's burning up and see the physical process that's happening and
Carter: engage with both the physical and software side of it well.
Carter: So we pulled from a pretty diverse background of people, like some from robotics,
Carter: some from like the video game industry, some from just general like systems programming stuff.
Carter: But everybody was eager to learn Rust.
Carter: I mean, you know, people were coming with different levels of experience in
Carter: different language from beforehand. So there wasn't any resistance to the language.
Matthias: Walk us through the early tech stack at the time. You said Python.
Matthias: Did you use OpenCV? Did those machines run on Linux?
Matthias: How low level was it? was it?
Carter: Yeah. So each of the individual camera systems, that is one application that
Carter: at that point AMP had been working on that application for four or five years.
Carter: And it's a C++-based application built around ROS, which is the robot operating system.
Carter: It's not an operating system, but it's a framework for building robot software
Carter: in a mixture of C++ and Python.
Carter: And we'd pretty heavily invested in the ROS ecosystem.
Carter: So most of the non-cloud, non-machine learning programmers at the company at
Carter: the time were C++ developers used to working in this ROS framework.
Carter: And the ROS framework is really microservices for robots.
Carter: You just write a bunch of individual processes that publish messages to each
Carter: other on a message bus to represent your robotics application.
Carter: So an example of what one of the robotic applications looks like,
Carter: there's a camera node that is pulling images off of the camera and publishing them on an image topic.
Carter: There's then like a node that's running the neural network that's receiving
Carter: those images, processing them
Carter: with the neural network, outputting the results of the neural network.
Carter: And you build this kind of compute graph of processes connected together,
Carter: each one receiving some inputs and producing some outputs that kind of flows
Carter: nicely from one side to the other side.
Carter: And the core C++ application that that is, is responsible for both getting the
Carter: images off the camera, doing the neural network work inference on them,
Carter: and then also control of the immediate sorting system that's attached to that camera.
Carter: So that whole system runs in a very, very tight time-critical loop of getting
Carter: the frame off, doing the processing, executing the actual sorting,
Carter: whether that's turning on specific air jets or planning a path for the robot to go pick.
Carter: So that's kind of the core existing robot application we have that runs on equivalent
Carter: to basically a regular desktop that's sitting at your desk.
Carter: We had i7 processors, NVIDIA GPUs, nothing crazy or exceptional.
Carter: We actually run that application on a really wide range of hardware because
Carter: we've been experimenting with different hardware over time.
Carter: So for our facilities, it typically runs on a server, in a server rack, in a server room.
Carter: And we have one server per kind of camera as our hardware matchup right now.
Matthias: And what time frame are we talking about for each frame is it comparable to
Matthias: a real-time game engine that maybe runs at 30 FPS or 60 FPS so you have a couple
Matthias: milliseconds per frame yeah.
Carter: It's it's a slightly different than that because we're not,
Carter: we're not doing everything per frame. There's no like one global clock that
Carter: kind of runs that loop instead.
Carter: Everything's asynchronously happening at the maximum frame rate that it can really handle.
Carter: So like we can pull images off the camera at like up to 40 frames a second,
Carter: but the neural network can't run that fast.
Carter: It might be running at some, you know, frame rate of 10, 10 frames a second.
Carter: It's actually all over the place depending on application.
Carter: And then the control loop that's actually doing the sorting might be running
Carter: behind that at like a thousand Hertz for the, for like the jet valves.
Carter: And so different parts of the application are running at different speeds,
Carter: and they're making the best decision they can at any given point based off of
Carter: the most recent information they've received from the other parts of the application.
Carter: And there's different timing requirements at every different part of that,
Carter: that are hard to achieve, but you throw enough hardware at it, it's not too bad.
Matthias: I can see how moving from Python to C++ gives you really great great performance benefits.
Matthias: But I wonder, Rust is also, of course, very, very fast.
Matthias: It has a reputation for performance and safety, but at the same time,
Matthias: it also has a reputation for a very steep learning curve.
Matthias: And I wonder, once you hired this new team, and I assume some of them learned
Matthias: Rust on the job, how did that impact your team's productivity,
Matthias: especially under those tight deadlines?
Carter: You know, I...
Carter: I have thoughts on Rust's learning curve. I don't think Rust has a very steep
Carter: learning curve, especially when you compare it to the alternative we had in hand, which is C++.
Carter: Like if you honestly ask me how hard is C++ to learn versus how hard is Rust
Carter: to learn, I firmly, firmly believe C++ is harder to learn.
Carter: And it's not, like the nice thing about C++ is you don't have to eat the whole sandwich at once.
Carter: Like you can learn a part of C++, but you don't know templates.
Carter: And then you do dumb programming in C++ plus because you
Carter: don't know templates and then you try to use templates in c plus plus and
Carter: you do it wrong and you get horrible impossible to debug bugs
Carter: that drive you crazy because you don't know templates like templates
Carter: is a turing complete language in and of itself buried inside of c plus plus
Carter: that's brutal and the other thing that's brutal about c plus plus is like there's
Carter: not one way of doing anything there's nine ways of doing anything and i'll fire
Carter: you if you do seven out of the nine of them because they're dangerous and safe
Carter: and bad practices and not things that should be done any well.
Carter: So if you ask me, is it harder to teach someone to write Rust well or to write C++ well?
Carter: I think it's harder to teach someone to write C++ well.
Carter: And for people who are pretty experienced C++ devs, I think going from C++ to
Carter: Rust feels like a breath of fresh air.
Carter: There tends to be a pretty nice one-to-one pairing of concepts that you were using in C++.
Carter: Well, here's the Rust version of them, and they're a little easier to use,
Carter: and they're a little more cleaned up.
Carter: And so most of the people who I taught Rust to had a very smooth path to it,
Carter: having come from a decent level of C++ experience before.
Carter: I think Rust is pretty brutal to learn if you've only been in Python and JavaScript
Carter: and you've never dealt with memory management and mutexes and atomic reference counters.
Carter: Those are hard concepts to learn. But if you're already struggling with all
Carter: those concepts and already needing to do them in C++, in Rust,
Carter: I think it's It's presented better, it's packaged better, and there's less rough
Carter: edges on the actual learning process.
Carter: Like the documentation for Rust is so much better than the documentation for
Carter: C++ for the equivalent levels of complexity that they're dealing with.
Matthias: When you said you taught people how to use Rust,
Matthias: were there any specifics that you noticed that people got wrong initially or
Matthias: some repetitive patterns that
Matthias: over time, a lot of people struggled with this specific thing in Rust?
Matthias: Maybe besides the obvious, like fighting with the borrow check in the beginning.
Carter: Yeah, I think the two biggest things that we ran across, the first one is like
Carter: general code architecture and layout. out.
Carter: We had a lot of people who were just used to writing classes and they thought
Carter: in classes and classes were how they wanted to organize their code.
Carter: And so they would write a Rust class that had all of the data they wanted in it.
Carter: And then they would try to write a bunch of functions on that class,
Carter: but they don't realize that by putting everything in one big pile where now
Carter: you're taking a reference to self,
Carter: you have all these splitting borrow problems where you actually want to like
Carter: mutably borrow part of self and not mutably borrow the other part of it.
Carter: And so they started writing these, you know, Mondo structs with,
Carter: you know, hundreds of impls that are individual functions on that struct,
Carter: because that's just the C++ way to organize code.
Carter: And kind of breaking people of those habits and having them write,
Carter: you know, smaller structs, leave the data a little more broken apart,
Carter: write fewer functions that are, you know, member functions of those structs,
Carter: but write more free functions that are just floating around.
Carter: Like in C++, it's dangerous to just have a function declared floating around
Carter: somewhere in Rust, I actually think it's better in general to write a free function
Carter: that just takes in two things than to write, you know, functions on self all the time.
Carter: Like, unless you really need all of the data of self, why are you making that,
Carter: you know, a member function of that struct instead of a free function that's
Carter: just operating on only the data it needs?
Carter: So first one, sorry, that was part one answer was just general code style.
Carter: The other thing that we kind of most struggled with was just async in general.
Carter: And this was something that even I struggled with in this project.
Carter: You know, the previous architecture that we were coming from was writing this robotic application.
Carter: And ROS's idea of microservices is essential to writing robust software in C++.
Carter: Because, you know, undefined behavior and memory corruption can come from any
Carter: part of your code in your application anywhere and theoretically corrupt anything
Carter: else nearby. by, a way that we build robust systems in C++ is we break them
Carter: into multiple processes.
Carter: And by breaking them into multiple processes, if the camera driver that we're
Carter: using from the vendor has a bug in it that causes memory corruption, well, what does it do?
Carter: It takes down the camera driver, the camera driver restarts,
Carter: and the whole system keeps on trucking.
Carter: The camera driver crashing doesn't take down the robot sorting because they're
Carter: actually isolated in different system processes.
Carter: And we're trusting Linux's process isolation to keep one process from taking down another process.
Carter: That turns out to be really good, but we naturally had this writing style of,
Carter: okay, we write everything in individual microservices.
Carter: And in C++, we had strongly erred towards the style of one thread per microservice,
Carter: where we did not internally multi-thread any of those microservices because
Carter: we've continuously found multi-threading is the source of the vast majority of bugs in C++.
Carter: And so if you don't do internal multi-threading to a process,
Carter: but instead push everything over the message bus between the nodes,
Carter: you can avoid a lot of classes of bugs.
Carter: So we were coming from that style into Rust.
Carter: And we started writing our application in that same way, where everything was
Carter: a standalone, non-async node.
Carter: It was receiving stuff synchronously. It was publishing synchronously.
Carter: Every node had one control loop around it.
Carter: And we We were just publishing and subscribing on a message bus,
Carter: and that worked pretty well, but then we started wanting to play with async,
Carter: and we were like, oh, well, async should be better, and it should work well.
Carter: And in fact, it really, really does.
Carter: What got really messy is when we started to mix async with that message bus
Carter: pattern, where now we have one node that's subscribing to the message bus,
Carter: but internal to that node, it's now spawning a bunch of separate actors that
Carter: are internally forming a little message bus out of channels inside of that node
Carter: to wrangle data between it, and it just got confusing and undebugable.
Carter: And I would say, like, I think one of the detriments of async rust is that when
Carter: you start writing complex things in async rust, debugging them becomes really, really, really hard.
Carter: Like, understanding the flow through your kind of async graph that you've made
Carter: of where data is being passed and where things are getting deadlocked and where
Carter: things are getting hung up is really, really hard. So, like, of course...
Carter: Of actual bugs that we wrote in Rust that we had to debug when we were writing
Carter: it, I think 90% of them were deadlocks in one form or another,
Carter: where we had code that was waiting on code that was waiting on other code that was stuck in a loop.
Carter: But when you write a lot of async Rust, seeing a loop in kind of the data graph
Carter: that can create a deadlock, not the easiest thing to spot.
Carter: Like in the way that just async code is written, those are non-obvious when
Carter: they appear. And we struggle with that a lot.
Matthias: On one side, you have ROS, the robot operating system with its processes.
Matthias: I almost said actor model, a way to separate the concerns through individual threads.
Matthias: And on the other side, you have the tokio, or more in general,
Matthias: ROS async model, which has work stealing and doesn't necessarily have thread per core.
Matthias: It's just a different way of modeling your application.
Matthias: It seems like it's an architectural problem as well were you fighting
Matthias: those different models those different contexts when you try to integrate them
Matthias: you mentioned deadlocks right that was kind of a symptom already of that we.
Carter: Were definitely fighting it and i think you know at the beginning we wrote everything
Carter: in the rust in the ROS model of it i hate how similar those two words.
Matthias: Hehe
Carter: But we wrote everything as individual processes to start with.
Carter: And I think that was a fine architecture.
Carter: We then started playing around with async and mixing async in with that architecture.
Carter: And that was frankly a bad idea to just like mix it in a little bit like that.
Carter: The two really don't match super well with each other.
Carter: But what we found is, is that async rust is actually better.
Carter: It's more ergonomic. It's easier to use when you it's way, way lower, performance overhead.
Carter: And gradually over time, we've started re-architecting more and more towards
Carter: that async model and away from the separate process model. But,
Carter: It's been slow and gradual, and it's had to be done with some intention because
Carter: our early experiments into it led us a little bit astray.
Matthias: Why did you start with AsyncRust in the first place? Is it an I.O.
Matthias: Bound problem that you're faced with?
Matthias: I thought that you mostly dealt with image recognition, which I would assume would be CPU bound.
Matthias: What does AsyncRust give you?
Carter: Yeah, so thing to remember. So we have the individual robot application,
Carter: which is responsible for like gets a frame in from a camera,
Carter: runs that data, does the motion planning. That's all still in C++ today.
Carter: So like the core robotics application is still in C++. We're starting to add
Carter: some rust in there, but that's going slower.
Carter: For the facility control system, what is the facility control system doing?
Carter: It's talking to something like 450 Ethernet devices that include all of those
Carter: C++ applications that are doing the data, but also include IO devices and VFD devices.
Carter: And it's doing frankly, very, very minimal compute on all of them.
Carter: But what it needs to do is collect fresh data from all of them,
Carter: ensure that it has connection to all of them, ensure that data is fresh and
Carter: valid and everything is still healthy, and then keep the facility ticking with
Carter: small, minor adjustments.
Carter: So the main control loop of the Rust application, we don't need it to run faster than one hertz.
Carter: Like one update a second is actually really fast for a facility control system.
Carter: The vast majority of what we're doing is just dealing with talking to hundreds
Carter: of other devices that are all of the things in the facility,
Carter: getting all of that data marshaled into a single central location that we can then process on.
Matthias: Yeah, then it sounds like a perfect use case for asyncrust.
Carter: Yeah. And like, you know, we were having some bottlenecks in the beginning because,
Carter: you know, VFDs are the variable frequency drives.
Carter: They control each conveyor. We might have a hundred of those in a facility.
Carter: We would spawn a separate process for each one of those VFDs at the very beginning.
Carter: And we actually ran into bottlenecks in like, hey, when we need to bring the
Carter: facility down and bring it up,
Carter: spawning each one of those VFD processes was taking a really long time and causing
Carter: facility startup to be slow because of weird annoyances in the drivers that
Carter: we were using to talk to the VFDs,
Carter: turning that from 100 separate processes to one node that is the VFD reader
Carter: that has 100 actors inside of it, each one handling an individual VFD is where we're at today.
Carter: And that's working very well for us.
Matthias: So the VFDs control the conveyor belt.
Matthias: Is that something that you have under control, the software part of it?
Matthias: Or is that a proprietary driver that you maybe cannot change?
Matthias: And if it is controlled by you, is it in C++ or Rust?
Carter: It's in Rust right now. And so the VFDs provide communication interfaces.
Carter: Like basically all industrial devices nowadays provide an API for you to control
Carter: them. Most of the APIs are in some asinine industrial protocol that there's
Carter: not great representation for.
Carter: And unfortunately, the most common one, the like most common industrial language
Carter: that the most devices speak is a protocol called Modbus, which is from like
Carter: the 1970s, which is literally like read and write raw addresses in memory.
Carter: There's like five commands and it's, hey, at this magic number address,
Carter: tell me the 16 bits that are present there or write those 16 bits to it.
Carter: So instead, you get a manual from the manufacturer that's about a thousand pages
Carter: long that has just tables of this memory address controls this function.
Carter: And here's what the bits do in it. It's very similar if you're doing like embedded
Carter: programming where like on the microcontroller, you get all the special addresses
Carter: for that microcontroller.
Carter: We're basically doing the same thing, but it's over an Ethernet cable,
Carter: which is a little bit crappy.
Carter: But, you know, the actual driver, you know, there are Rust Modbus libraries
Carter: that very happily produce and send Modbus packets. So like those aren't hard for us to use.
Carter: But then we have to wrap them in, OK, we're using this specific VFD for this
Carter: specific manufacturer. So here's all the addresses that we care about.
Carter: And then here's how to convert, you know, 16-bit Modbus types to actually like
Carter: useful types on our end to send the data back and forth.
Carter: So there's a lot of just converting down to some dumb industrial protocol and
Carter: then converting back up to, you know, regular types that are useful to work with.
Matthias: It sounds fun when he explains it like that. It sounds like a lot of fun.
Matthias: And it also reminds me of Oxide Computer, which also takes what was there before and then they...
Matthias: You know reimagine it and bring it back into
Matthias: the future and they do something similar for server
Matthias: hardware but i imagine that it is much of
Matthias: the same work you try to make sense of what is out there and you read those
Matthias: data sheets and then you build your own abstractions and you probably do that
Matthias: at the very edge of computation so whenever possible you work on a high level
Matthias: and only at the last possible moment you convert it or serialize it to something
Matthias: that is just a byte stream that the controller can understand.
Carter: Yeah. You know, it's important for us to not be locked to a particular VFD manufacturers.
Carter: Like we had to come up with a generic representation of a VFD that we could
Carter: work with inside of our application.
Carter: And then how do you convert that to different manufacturers representations
Carter: under the hood and deal with, well, this VFD does or doesn't have this capability or not.
Carter: And, you know, like enums and option are so useful and so good for representing
Carter: things like that, of like this VFD does or does not have that capability.
Carter: And making an application like a facility controller, you're really dealing
Carter: with a lot of being generic over different hardware.
Carter: That's a lot of the problem that we're dealing with.
Carter: And Rust has just some great tools for that, that made that process very pleasurable to deal with.
Carter: I'm going to take us back for a second because earlier you asked the question,
Carter: okay, you were a software company and then you transitioned into a hardware company.
Carter: And I kind of agreed with that and answered that. But honestly,
Carter: I'm going to put a twist on it.
Carter: And I tell this to people a lot. The value of AMP is not in its software or
Carter: in its hardware. It's in its systems integration.
Carter: It's in the ability to put everything together in one package and have that
Carter: package cohesively work together.
Carter: The weaknesses of the hardware have to be compensated by
Carter: the software the weaknesses of the software have to be compensated by
Carter: the hardware there's not like a bit of AMP that
Carter: you can pull away and say like oh well they have really good vfd drivers that's
Carter: great no the vfd drivers don't matter the vfd drivers
Carter: are one small tiny part in getting
Carter: a machine with 10 million parts to work
Carter: together you know we have a server room with dozens of computers in
Carter: it hundreds of ethernet cables switches network diverters converters
Carter: there's serial devices with different communication protocols like there's just
Carter: an enormous machine and most of the value isn't in either the hardware or the
Carter: software it's in the capability to put it all together and actually have it
Carter: work at the end of the day yes that's that's like a beautiful thing.
Matthias: I Thought about products today and i
Matthias: realized that if you build a great product people will
Matthias: look at it and think it's obvious it's it's simple
Matthias: it does exactly what it is supposed to
Matthias: do there are no extra parts that you don't
Matthias: need and getting to a point where you
Matthias: build a product like this is very hard engineering
Matthias: work and it feels like this is what you literally do you make a very very hard
Matthias: technical challenge consisting out of different levels of expertise and different
Matthias: disciplines you combine it into a very nice vertically integrated product,
Matthias: on the outside, it looks like, yeah, pretty much magic.
Matthias: And then you go in and you realize that all of those things need to be addressed
Matthias: and the problems need to be worked around.
Matthias: And you end up with something that just works. It's polished.
Carter: Yeah. And honestly, you know, the only advice I can really give to people who
Carter: are trying to do the same thing is to not try to polish particularly upfront.
Carter: The biggest lesson I've taken away from AMPand the biggest thing that i would
Carter: say has made us successful is you know commitment to the fact that the fastest
Carter: way to build the right thing is to build the wrong thing twice first,
Carter: we always always always try to
Carter: build the simplest dumbest thing like what
Carter: could possibly solve this problem well yeah
Carter: it might not work but let's build it and try it see what's
Carter: wrong with it and then make adjustments from there we like
Carter: we try to avoid over engineering as
Carter: much as humanly possible everywhere like what is
Carter: the simplest possible thing we could do do that thing and you know part of part
Carter: of like why we ended up with a rust control system you might say like that flies
Carter: in the face of it but at the beginning we wanted the simplest possible thing
Carter: that would turn all the vfds on well what could i do that would get me that
Carter: the fastest well i can walk in the back and I can write it myself and it will take me a day.
Carter: Okay, I go write that, I've done that. Now we have something that can turn the VFDs on.
Carter: Well, what's the next thing we have to build? And then just one at a time,
Carter: laying the bricks on top of each other, not worrying about what the cathedral
Carter: is going to look like at the end of the day,
Carter: and being ready to knock down a few walls and rebuild some stuff as you go,
Carter: as you learn what really matters in the problem that you're working on.
Matthias: Okay, but coming back to Rust for a second, I had two questions on my mind.
Matthias: Why is it bad to have a free function in C++?
Carter: I would say it's just generally viewed as bad style because of like poor namespace
Carter: control and poor module controls.
Carter: If you aren't putting that function under a class where I'm calling it with
Carter: like dot through the class, it becomes easy to have namespace collisions with free function names.
Carter: If you are not being pedantic about namespaces and the other libraries that
Carter: you're using aren't being pedantic about namespaces.
Carter: So like Rust's module system with like tighter includes just makes that a little
Carter: bit easier to deal with in general.
Carter: That's probably my biggest answer to it. The other answer I think really comes
Carter: from like problems of RAII and data management where like when I have free functions
Carter: in Rust that want to free things or allocate things or transfer memory ownership,
Carter: you can't really express those well.
Carter: And so in C++, I think the pattern that most people have kind of come to is,
Carter: well, we have these functions that are operating on data and they're the member
Carter: functions of our class and the class is concerned with the lifetime of the data
Carter: and the member functions are concerned with the processing of it,
Carter: but they've bifurcated it where there's this constructor function and there's
Carter: this destructor function on the class that are going to deal with the memory management for them.
Carter: And then the functions on the class can just be regular functions that they
Carter: don't think about that much.
Carter: But when you just have free functions floating in space, they really don't have
Carter: any ability to control the memory that's being passed in and out of them in
Carter: effective ways it's a little bit better nowadays with like how move has worked
Carter: out but it's hard like most people get it wrong when they try to get it wrong
Carter: when they try to do it.
Matthias: Okay, if i understand correctly then it's easier for the programmer to reason about
Matthias: memory usage if you talk about it on an object level because all the memory
Matthias: that is associated with this very object is encapsulated in the object model or the class itself.
Matthias: Whereas if you had a free function, it's not as easy to understand whether you
Matthias: can free memory or what even the parts are that this function might allocate
Matthias: or forget to deallocate.
Matthias: And that makes, let's say, programming hygiene a little harder.
Matthias: And it puts the burden on the programmer. I see. And the second question was,
Matthias: how much of the Rust ecosystem can you actually leverage to solve some of your problems.
Matthias: Do you use any crates or do you mostly write your own stuff?
Matthias: I mentioned that a lot of the things that you build are very purpose built,
Matthias: but maybe there are things like the bitflags crate or working with byte streams
Matthias: that you can still use and leverage.
Carter: Yeah, honestly, we ended up using a ton. And it's one of the main reasons that
Carter: we ended up choosing Rust was that when we went looking for the crates that
Carter: we would need, that we had C++ equivalents of, we're like, C++ can do this for this, whatever.
Carter: We found Rust equivalents for the things that we were looking for.
Carter: The vast majority of like the the rust crates
Carter: that we're using are communication related both for the
Carter: message bus that we're using between the nodes and for talking to the industrial
Carter: devices so like tokio_tungstenite that does MQTTcommunication we use that an
Carter: enormous amount wait i think that's i got that crate name right i'm gonna be
Carter: embarrassed if i got that crate name wrong.
Matthias: There is a crate name called tokio_tungstenite.
Carter: Think tungstenite's tungstenite's either the WebSocket one, which we use,
Carter: or it's the MQTT one. I think it might be the WebSocket one.
Carter: But we use that. We use a bunch of the communication infrastructure to talk
Carter: all kinds of different protocols, whether it's Modbus, MQTT,
Carter: OPC UA, like JSON WebSocket stuff.
Carter: All of that is being used at different points. And Rust's networking crates
Carter: in particular are of exceptional quality and modular enough that you can adapt
Carter: them to the kind of needs you want.
Carter: I'll also call out like serde or serde as different people pronounce it is such a gem.
Carter: Like the ability to just serialize and deserialize whatever we're working on
Carter: into a bunch of different formats to send to different locations.
Carter: Like that has been, oh my God, the amount of boilerplate that serde has replaced
Carter: in our application is absurd.
Matthias: What serde formats are we talking about?
Carter: JSON is definitely the heaviest used one. There is serde ROS message,
Carter: which actually lets us convert to the ROS byte level protocol that like ROS uses.
Carter: There's also a few oddball ones for like random systems that we'll talk to.
Carter: Another, you know, another great crate to just kind of mention and call out
Carter: if we're just shouting out crates, our user interfaces are all Angular applications
Carter: that are written TypeScript.
Carter: That's kind of legacy from like, that's how our company's been writing UIs the
Carter: entire time I've worked there.
Carter: So we have a lot of TypeScript knowledge on building UIs. And okay,
Carter: now you want to talk between a Rust backend and a TypeScript frontend.
Carter: There's an amazing crate ts-rs that for any Rust types you define will define
Carter: TypeScript types for them,
Carter: so that you can have closed loop type safety between them
Carter: and it doesn't matter really on what the serialization protocol like
Carter: we end up using serde json to define a rust type serialize it to json send
Carter: it to the front end the front end uses the ts-rs crate to get typescript types
Carter: for it but we have you know closed loop type safety across both applications
Carter: for practically free like that was so easy to set up and.
Matthias: Soon the front end will also be written in rust because there's leptos and tauri have you heard of.
Carter: Yeah and we we We keep, you know, I keep poking around with those.
Carter: I think convincing our front-end developers to switch to Rust is a much harder
Carter: pill to swallow than the C++ devs. Like the C++ devs, there's not a single one
Carter: of them who's like, yeah, I'll defend C++.
Carter: It's a great language. Every single one of them is like, yeah,
Carter: it's an old beat-up truck.
Carter: It still runs, so it gets the job done. But boy, howdy, would it be nice to
Carter: drive something a little newer?
Carter: So they're hungry for Rust. The front end devs, like, yeah, you can make some
Carter: arguments for Rust there, but, you know, they've got a really nice flow in TypeScript
Carter: and Angular and like performance is so much less critical and reliability is so much less critical.
Carter: And, you know, we're not delivering these UIs over the Internet to people publicly
Carter: like their internal HMIs within a closed loop network on our facility.
Carter: So even some of like the security things are a little less interesting, right?
Matthias: When you wrapped up the project you wrote
Matthias: a blog post and in that blog post you said
Matthias: something which struck a chord with me which was the project went shockingly
Matthias: well and everyone that likes rust probably would not wonder that it worked shocking
Matthias: well we well but that you were kind of shocked by it.
Matthias: What were your presumptions going in?
Matthias: What did you anticipate?
Matthias: What might be the problems that you would have likely have to face, the setbacks and so on?
Carter: Yeah. So I come from a lineage of having stood up a lot of complex systems.
Carter: And particularly when you're building robots, software often gets the really
Carter: short end of the stick, where you're told to write all of the software,
Carter: why mechanical and electrical are designing and building it mechanical
Carter: and electrical run behind schedule so the physical hardware is ready you know
Carter: 24 hours before the demo to the board and you have to test the software and
Carter: integrate everything at midnight like i have done the like 1 a.m get the software
Carter: on the hardware for the first time just before the demo schtick probably 10 times,
Carter: This is one of the worst possible scenarios of that.
Carter: Like this is an entire factory. It's hundreds of devices, millions of points
Carter: where things could go wrong.
Carter: At this point, the Rust application had only been tested against maybe like
Carter: 10 devices sitting on a benchtop in our lab.
Carter: Like it had never been plugged into the full facility scale,
Carter: the full facility control.
Carter: At this point, all of the code has been written in, you know,
Carter: against simulated targets that We also wrote the simulated devices in Rust to
Carter: try to end-to-end test the equipment.
Carter: And the scale of the whole facility is just massive. There's a lot,
Carter: a lot, a lot of things to go wrong.
Carter: Considering how short the project timeline was, considering this was the first
Carter: major project that anyone on the team had done in Rust, considering most of
Carter: the team were new to the company,
Carter: the odds of that software just working the first time and there not being massive
Carter: glaring critical bug that prevents the facility running should have been zero.
Carter: That application had no goddamn right just turning on and running correctly
Carter: the first time. And I'm not going to say it was perfect.
Carter: We probably fixed like two or three bugs on site in the first couple of days
Carter: of running that caused random problem here and there.
Carter: But the reality is from like the first time we tried to power up the facility,
Carter: the rust application was running 99% smoothly.
Carter: No major hiccups was not the blocker. So, you know, at the end of this six month
Carter: build of an entire facility and it's time to turn on the facility and run software
Carter: did not cost us a single day of schedule.
Carter: Like we were able to test and develop the software in parallel to the hardware getting stood up.
Carter: And at the end of the day, when we plugged them in together and it was like,
Carter: okay, time to go integrate. It just worked.
Carter: That never happens. That should not happen. I feel like an asshole getting up
Carter: here and telling people that that happened because I won the lottery ticket.
Carter: That's not a reproducible thing that people should be like, oh,
Carter: well, that's what I should expect out of Rust. No, we got lucky.
Carter: I will fully admit that. But I do think Rust helped make that possible.
Carter: Just given the volume of C++ code I've written in the past, I know that if you
Carter: write 150,000 lines of C++ code,
Carter: there will probably be 15 or 20 gnarly hard to debug just insane things going
Carter: on that are undefined behavior and segmentation faults and weird crap like that
Carter: we had zero of those at time of powering up the facility like none of them appeared
Carter: and that's that was just cool that's nice and.
Matthias: Why would you say rust allowed you to have less bugs in production production.
Carter: A few things. The first I would say, you know, in these types of extremely async
Carter: programs that we're writing, like these applications where you're doing a ton
Carter: of things in parallel and mixing the data in together,
Carter: the majority of the bugs that we have historically encountered in C++ are memory safety bugs.
Carter: They are data races. They are use after freeze. They are, you know,
Carter: shared pointers getting freed or not freed when they should be.
Carter: That is just kind of one of the penalties that you pay when you write those
Carter: kinds of applications. in C++.
Carter: With Rust's memory safety, with the borrow checker, there was fundamentally
Carter: an entire class of bug that I am used to dealing with that we solve that compile
Carter: time instead of solving at runtime.
Carter: I'll also say this, I have used an enormous number of C++ libraries and you
Carter: can't build these kinds of applications purely in Scratch yourself.
Carter: A lot of people try to because it gives them control over the code quality,
Carter: but we pull in dozens of libraries in our C++ applications, and we find dozens
Carter: and dozens of bugs in them.
Carter: In all of the Rust crates that we pulled off the shelf to kind of integrate
Carter: into our application, we have never filed a bug against a single one of the
Carter: Rust crates that we've pulled up.
Carter: We started to file one against Paho crate, which I think is the MQTT one.
Carter: I'm getting that correct now.
Carter: And working with the authors of that crate, we traced that bug into they were
Carter: wrapping the C library, and we found a bug in the C library that the Rust library was wrapping there.
Carter: Like that's the only bug we found in a Rust crate we depend on in like three
Carter: years of using the ecosystem.
Carter: The quality just feels exceptional.
Carter: It feels unbelievably high.
Matthias: That says a lot about supply chain security. That's a really nice benefit on
Matthias: top of all the other guarantees of Rust.
Matthias: But a lot of those crates that maybe some people use in production,
Matthias: I'm not sure they are still not 1.0. So they could technically be,
Matthias: have breaking changes in the
Matthias: future. How do you see the long-term sustainability of your facilities?
Matthias: How do you see ensuring that the factory runs as planned and that you can evolve with Rust over time?
Carter: Yeah, so this is at least kind of fundamental to the idea of writing our own
Carter: facility control system.
Carter: And if we were going to do this as a one-off for one facility,
Carter: it would never have been worth it. The maintenance burden would be too high.
Carter: The support would be, you know, it's too much investment for one facility.
Carter: Our plan here is that we're writing a single application that we're going to
Carter: deploy across eventually hundreds of facilities.
Carter: All of the facilities are going to run the same application.
Carter: Simply configured to be different facilities.
Carter: That's already true right now. We are running, I think, four or five facilities
Carter: at the moment off of one Rust application.
Carter: And that means that having a sustaining team whose job it is to keep that Rust
Carter: application live and running and fresh is well worth the investment.
Carter: Because when we develop a new feature for one of our facilities,
Carter: it's immediately available at all of them.
Carter: And all of them are sharing the benefit of continuing to evolve and keep that ecosystem.
Carter: So it was at least acknowledged and planned upfront that having resources into
Carter: the sustaining aspects of this would be desired.
Carter: And definitely, we're using a ton of crates that haven't hit 1.0 yet.
Carter: We are fairly regularly updating all of our crate dependencies.
Carter: And my God, the library authors are fantastic about correctly obeying SemVer,
Carter: about having good release notes.
Carter: I wish release notes and change logs got a little bit more standardized in how
Carter: you document them, but we find the critical information we need.
Carter: And also Rust makes writing good tests substantially easier than C++ does.
Carter: So we're able to update a bunch of crate dependencies, run our test suite and
Carter: have pretty damn high confidence, it's going to then work on the actual factory afterwards.
Carter: So, you know, we've gone through, you know.
Carter: 10 releases of the core language plus hundreds of
Carter: crate updates and it's been pretty seamless and smooth and i know like that's
Carter: not going to be everybody's mileage and we are still careful about which crate
Carter: dependencies we do pull in and use and try to use you know trusted well supported
Carter: ones that look like they're gonna stick it stick it out and continue to be there
Carter: but so far you know no problems.
Matthias: And it
Matthias: makes sense to think about this as a platform now
Matthias: and have a team that makes sure that it
Matthias: runs smoothly and operations are predictable do
Matthias: you sometimes miss a tool for running
Matthias: rust applications at scale or in production maybe something
Matthias: like a supervisor that would tell you the state of dependencies or a thing that
Matthias: would tell you about upcoming breaking changes or that sort of thing or would
Matthias: you say in fact no it's pretty much smooth sailing from now on?
Carter: You know, I could imagine a tool like that being useful. I certainly think,
Carter: you know, as the application continues to grow and there's more and more dependencies
Carter: and more and more time goes on, the needs for that will be higher and higher.
Carter: We're, you know, two, three years into this being a real application at this
Carter: point and have not felt the need for that yet.
Matthias: That's a good verdict. Yeah.
Matthias: Now, if you reflect on the entire project, and if you were to start over,
Matthias: what would you do differently in terms of choosing and implementing Rust at AMP Robotics?
Carter: Yeah. You know, if I came with all of the Rust knowledge I had today,
Carter: I would probably first and foremost say that I'm just not afraid of async in
Carter: the way that I was, that I should have leaned even harder into Rust's ability
Carter: to do things in parallel safely and not have memory corruption bugs.
Carter: And I probably would have not done the full microservices architecture that
Carter: we started with doing it over from scratch.
Carter: I probably would have written it entirely as one single process with one tokio
Carter: runtime at the core of it and everything as kind of actors within a,
Carter: you know, a tokio runtime representing still the same compute graph,
Carter: the same chain of things happening, but relying Relying way more on tracing
Carter: and way more on the Rust ecosystem's tools for dealing with the communication
Carter: between those different actors than what we did, which is rely on a message bus.
Carter: One of the big things that we decided is, okay, we're going to have these different.
Carter: Processes communicating with each other. We need to use some message bus there.
Carter: There are a lot of options for those message buses.
Carter: We ended up picking MQTT, which is not what people traditionally think of as
Carter: for inter-process process communication on one system.
Carter: But one of the major advantages of MQTT is that the broker that's kind of brokering
Carter: that communication can be set up to forward those messages to the cloud.
Carter: And we immediately for free got historical cloud metrics and databases for any
Carter: topic that we wanted to mirror to the cloud.
Carter: We just changed a config flag and suddenly that message, that topic field is being backed up.
Carter: Into Google and we have databases on it. That was really nice and handy.
Carter: But at the end of the day, it turned out MQTT ended up being quite a bit of a bottleneck for us.
Carter: We had a few bugs with MQTT communication.
Carter: We had to switch to async because there were some variable latency problems.
Carter: The broker that we were using, there are many MQTT brokers available for running on your system.
Carter: We had broker crashes, which would take the whole stack down.
Carter: So one of the core of core pieces of technology that we ended up relying on
Carter: at the heart of our application was this message bus that we decided to build from,
Carter: you know, off the shelf tools that weren't Rust tools and then ended up being
Carter: one of the biggest problems in the whole application at the end of the day.
Carter: So certainly rethinking that message bus and instead of doing it as like MQTT
Carter: topics, doing it just with straight up channels inside of the code in kind of
Carter: the Rust more idiomatic way.
Carter: And instead of breaking everything into multiple processes where we need to
Carter: do IPC, doing it as actors that can have zero overhead in those message passes,
Carter: yeah, I think that would have set us off on a better foundation.
Carter: Frankly, everything I just said there is stuff that we are likely to change
Carter: about this application in the next couple years as we continue to invest in it.
Matthias: The one thing that comes to mind is if you start using normal in-memory channels,
Matthias: then you might have a hard time rebooting the system if you have state.
Matthias: But I wonder if you even have state in the system or if you start from a clean
Matthias: slate pretty much every time.
Carter: There is one microservice with state. And so the state is as isolated as it
Carter: is humanly possible to be.
Carter: And every one of those actors tries to be a pure function. You know,
Carter: actually winding back and saying, you know, a slightly different tact.
Carter: One of the things that I absolutely loved about our microservice architecture
Carter: and something that I would try to figure out how to make possible as we move
Carter: that way is when we're developing on this application,
Carter: our compile times were incredibly low because we're not compiling the entire application.
Carter: We're compiling one of those processes, right?
Carter: There's some common crates that they all depend on, but each one of those processes
Carter: that is a node is in its own crate. So I can compile that node independently.
Carter: I can also then stand up on my system, a live running simulation of the whole
Carter: facility with all of the nodes in place.
Carter: And I can iteratively compile and run new versions of that one node that then
Carter: just drops into all of the running existing traffic so that I have a really
Carter: short, quick iteration loop in being able to develop on one part of the application.
Carter: You know, compile times for the whole application might be several minutes.
Carter: But if we're talking about just iterating on one node, that's what you're currently
Carter: developing on from like timed finish hitting line of code to seeing the results
Carter: of that node running, we often keep under a second.
Carter: Like you can recompile that one node, have it stood up and running and in like
Carter: messages now traveling through it instantaneously and giving your developers
Carter: really tight feedback loops like that is critical.
Carter: And I think one of the things that's a challenge in a lot of bigger Rust monolithic applications.
Matthias: Do you have a shared types crate where you define the messages that go over the wire?
Carter: Yep, we have we have kind of one crate that that crate has nothing but a bunch
Carter: of structs with a bunch of drive macros on it.
Carter: That one crate generates the struct definitions that then all of the other crates
Carter: use as well as that crate generates the TypeScript definitions that the front end uses.
Carter: It also generates like JSON schema definitions that we use in other places.
Carter: So one centralized types crate kind of untangles the dependency tree there.
Carter: And it does mean that like we have a bottleneck in our compilation graph,
Carter: but that crate is very quick to compile and doesn't change horrendously often.
Matthias: And finally, do you have a microservice called WALL-E? Because if not,
Matthias: then you should introduce one.
Carter: Why do we need one called? Oh, because he does the trash. So our conference
Carter: rooms at our headquarters are all named after robots.
Carter: So like WALL-Eis the conference room next to my desk.
Matthias: Amazing.
Carter: We have a lot of WALL-E references. We don't have microservices.
Carter: I will say, you know, like three hard problems in programming,
Carter: off by one errors and naming.
Carter: We deal a lot with naming things. Boy, it never gets easier, does it?
Matthias: It has become sort of a tradition around here to ask the guest at the end whether
Matthias: they had a message to the entire Rust community. Is there something that just comes to mind right now?
Carter: Yeah, and it's a really positive message. Like the Rust community has built
Carter: something truly incredible.
Carter: And I think, you know, sitting behind our keyboards and staring at our screens
Carter: all day, it's easy to lose sight of the impact of better software in the real world.
Carter: Like it's easy to forget that, you know, like a slightly better programming
Carter: language isn't just a nice to have thing that makes programmers lives better.
Carter: It's a thing that ends up having true real world impact.
Carter: And so at least in a little bit of a way, every single person who's contributed to the Rust ecosystem,
Carter: whether it's the languages, whether it's the packages, whether it's the conferences
Carter: or the documentation, or like incredible YouTube videos that make it really
Carter: easy to teach people Rust, all of those things are helping achieve AMP's mission right now.
Carter: Like without the Rust community, it would have been harder for this recycling
Carter: company to build better recycling facilities and less material would have been
Carter: recycled last year and the year before, if not for their efforts.
Carter: So I just want to give them a huge shout out and a huge thank you to the entire community.
Carter: You guys are making a positive impact in the world by making it easier to build
Carter: better software that does have real world impacts, that does make a real world
Carter: difference, that does make the world a better place.
Carter: And I feel bad that,
Carter: I get so much of that benefit and get to see it all.
Carter: And I'm doing everything I can to try to figure out how to give back to that
Carter: ecosystem and continue the virtuous positive cycle.
Carter: But, you know, it's incredible what's been built here. And it really does matter. And thank you.
Matthias: Where can people learn more about AMP Robotics if they got interested in the
Matthias: topic? Can you share some resources?
Carter: Yeah, so we changed our name, I think, literally last week. And we dropped the
Carter: word robotics off of it, because we're trying to be more about the facilities
Carter: now. That's our new goal.
Carter: So the company is just called AMP now, which is fine.
Carter: Yeah, not the hugest change, but we did just launch a brand new website.
Carter: So if you go to ampsortation.com now, which is our new website,
Carter: it really shows all of the different technologies we've built,
Carter: and it shows our facilities.
Carter: We just put up a couple of incredibly well-produced YouTube videos
Carter: that give you tours through our facilities and show
Carter: you what these air jet sorters look like and show you what our camera systems look
Carter: like and it's the first time that we've been really able to
Carter: like publicly talk about what's inside these facilities and it's incredible
Carter: to see so really go check out ampsortation.com the website looks slick it's
Carter: not like it's got a crazy like scroll thing where like as you scroll through
Carter: it you like zoom through the 3d model which is a little gimmicky but it's also
Carter: really really well done. And I'm just excited about it.
Matthias: Yes, check it out. Carter, thanks a lot for being a guest on the show.
Matthias: It was really incredible.
Matthias: I learned a lot. And thanks.
Carter: Thank you.
Matthias: Rust in Production is a podcast by Corot and hosted by me, Matthias Endler.
Matthias: For show notes, transcripts, and to learn more about how I can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.