It's time for another episode of Rust in Production. I'm Matthias Endler from corrode, and today we're wrapping up Season 3 and our first year of producing this podcast. In December 23, we launched this podcast to uncover stories about Rust in production. Since then, we've watched Rust transform from a fringe language choice into a powerhouse that drives critical production workloads. This season, we had very different guests from all sorts of industries. From developer tools with Zed and Gitbutler, over infrastructure with Fluvio and Oxide, to browser technology with Brave and C8E software with Su, all the way to automotive with Volvo. The conversation about Rust has shifted dramatically. Companies don't pitch Rust as an innovation anymore, they pick it as their go-to language when replacing legacy systems. I've even seen competing companies build multiple parallel implementations of the same thing in Rust. The well-known ThoughtWorks tech radar now ranks Rust as well-established in systems programming. Rust has carved out strongholds in safety-critical sectors, especially automotive, and dominates the developer tooling space. Let's hear Steve Klabnik from Oxide. A lot of people don't realize how much production Rust is out there and how many companies truly depend on it. And if the idea if rust like were to suddenly implode how people wouldn't step up to fix it like just meta alone has i think like on the order of 10 million lines of rust it's like i don't think it's it's a hundred million lines but i think it's more than single digit millions it's like between one and 10 million lines of rust you know amazon like rust is now involved in like s3 aws like aws ec2 uh tons of different aws servers are all like have rust at really key points i mean rust is in the windows kernel already of. All the stories we heard the one from volvo caught me completely off guard in the best way possible i never expected traditional car manufacturers to jump on rust so quickly yet here we are, And alongside Renault and others, they bet big on Rust. That shift tells us something important. Rust has evolved beyond its system programming roots into a solution for building reliable software at scale, regardless of the industry. Web development's Rust revolution surprised me too. Of course, the benefits seem obvious now. Who doesn't want better performance and reliability, I guess? But the sheer speed and scale of adoption? Well, that blew me away. I personally like that more smaller companies are coming forward with their Rust usage now. Okay, now let's dig into the last eight episodes and unpack how different companies tackle Rust adoption. Their stories maybe can teach us valuable lessons about bringing Rust into production. Chapter 2. Enterprise Adoption Patterns The way enterprises adopt Rust has matured significantly. For example, the Chrome team invests heavily into Rust by integrating it into the browser's build system, which is far from a trivial task and requires careful alignment. Anton Lazarev from Brave shares some insights. Chrome took a very measured approach, in particular, with adopting Rust. I think they've been interested in it for a while, but they wanted to really stress test it and make sure that they have good integration with their build process and that everything works and doesn't cause unexpected issues out in the wild. Because they have a huge user base. Things can go very wrong in ways that you have no idea about. So they were prototyping a way to include Rust code in their code base for a while. And we were watching that with a lot of interest because obviously we had our own build system integrating with Rust for a while. And we had ours pretty early on and we were pretty confident about it. And we just said, we'll try it. We'll see what happens. And as a result, we had it working pretty well. And I'm not sure if the Chrome team looked at us as a success story, but we were certainly available at the time. This integration process is quite complex. It might make sense to get external support for guidance and to save time. Here is Julius Gustafsson from Volvo. Yeah, so we brought on these experts. They were the foremost experts in Sweden at the time. And they also, they helped us, get started and and helped us with a lot of the driver development and and things like that so so i would say that is also a big piece of the puzzle to get good people to help you get started either either from within the company or externally depending on what options do you have in. General we see adoption patterns shift across the industry previously greenfield projects were the norm. However, SLAs and longer lead times are now part of the conversation. Support for legacy systems is a must-have. Partial migrations are now more common. That's because Rust just plays very nicely with other ecosystems. Teams actively compare Rust with alternatives, like Java with Quarkus. And maybe a full rewrite isn't necessary after all. In general, it's a more conservative, measured pace of adoption. And the numbers back it up. Teams I consult for consistently build 100,000 line code bases within six months. Legacy code ports are more straightforward than commonly expected, especially with one or two Rust champions leading the team. Google's director of engineering, Lars Bergström, even reported that Rust teams are twice as productive as teams using C++. They've seen Android memory vulnerabilities drop by 52% after switching to Rust. But what fascinates me the most is how teams grow into Rust. Their journey typically spans two distinct phases. Year one focuses on fundamentals. Onboarding developers, proving feasibility, making time for adoption, and so on. Year two tackles bigger challenges. Establishing design patterns, maintaining code quality, managing compile times at scale, building systems for knowledge transfer, improving developer experience and so on. I've watched teams repeatedly succeed with this methodical approach. The key is to keep things simple. Focus on the developer experience and use each language where it makes the most sense. Hype-driven development sucks and backwards compatibility is a big win. Here is Scott Shakin, co-founder of GitHub and now founder of GitButler on the same topic. I don't like clever code, right? I like dumb code that is easy to understand and easy to modify. And so I always kind of go away from super clever stuff because even though it makes you feel smart, it may not. And that's why it makes you feel smart is because somebody else has a harder time understanding it or figuring it out. Like I don't like magic. Conrad Irwin from Zed agrees. I've been doing Rust for about a year now and I just about feel like a beginner. So if we can make it simpler for people joining in, I think that will help with everything. One thing I learned, we did a rewrite of GPUI, the graphics framework, that was fairly heavy on use of generics and ended up exporting a bunch of things that were instantiated multiple times with multiple different types. And the way that Rust compiles generics is to kind of copy-paste the code each time for each type. And so we had some kind of big compile-time regressions at that point, just because of the way the code was structured. Chapter three, not all that glitters. Not every Rust adoption story ends in success. It's worth looking at the other side of the coin. For example, the game dev space hasn't picked up Rust as quickly as I expected. No AAA title in sight. There's a thoughtful article called Leaving Rust for Game Dev that explains why. Perhaps Rust isn't the best fit for game development after all. Of course, there are systems like Bevi, but its entity component system and trade-based approach takes a while to get used to. Paired with the tendency to work on mutable state in gamedev, many developers initially hit a wall when they tried to use Rust for their first big gamedev project. Well, at least we have TinyGlade, which is a game fully written in Rust that recently launched on Steam. Even a Linux kernel adoption does not go without drama. Who would have guessed? The message is clear. Use the right tool for the job. Sometimes that's Rust. Sometimes it's Zig. And sometimes it's something else entirely. Let's hear Jessie Frazelle from Zoo. Our CAD kernel is also sadly not written in Rust, it's written in C++. And that was because Vulkan and a lot of the CUDA libraries, like all of the good graphics libraries and like NURBS libraries are in C++. And you could generate bindings for them, but it will never be the same. I've used the CUDA Rust bindings and like it leaves a lot to be desired. I actually think hopefully there will be better ones soon. And here's Deb Roy Chowdhury from Infineon. The co-founders, they actually were initially debating on whether to use Golang or Rust to build Fluvio five years ago. And it was a fairly contentious decision and the CTO won that debate and Rust was selected. They had done a bunch of Golang work with server side and Golang is pretty fast, right? But the things that are most difficult to learn in Rust for beginners, like borrower checker, like async traits, are the ones that guarantee safety and security and guarantee that you get your data the way you want it. And this is why we made, or the CTO and the CEO, when they were making the decision, made the choice to write the Fluvia project in Rust. In Season 2, we had Daniel Stenberg talk about Rust's adoption in CURL. As of recently, Hyper, the HTTP backend written in Rust, was removed again from the CURL codebase. He mentioned outstanding issues with hyper and other crates, which no one had the time or patience to tackle. That said, Rust-TLS, which stands for Transport Layer Security Protocol, is still part of Curl. Heck, even ThePrimeagen, a prominent YouTuber and content creator, moved away from Rust. But more seriously, there's a major trend pushing Rust adoption, which is practicality. Companies increasingly choose Rust for non-technical reasons. Every year it's getting harder to find good Perl or Scala developers. Even great C and C++ engineers are hard to find. And as a consequence, new projects rarely start in C++ anymore. I recently heard that even some conservative companies, like my insurance provider, are slowly moving to Rust simply because that's where the talent is. These practical considerations often outweigh technical arguments. In a contracting economy, especially when we are facing a recession, cost becomes a primary driver. Production bugs are the biggest cost factor next to salaries after all. And Rust's promise of shifting complexity to the left, which means to development and away from production, catching issues at compile time rather than later on where it's more expensive, resonates strongly with business leaders. Chapter 4. Community Evolution The Rust community hit some impressive milestones this year. We crossed 100 billion crate downloads. A number that seemed impossible just a few years ago. The Rust subreddit now also has more subscribers than C++. But raw numbers only tell part of the story. The nature of discussions has shifted dramatically as well. People no longer ask, is Rust used in production? Instead, they ask, who uses Rust in my industry? That's a major shift forward in the public debate about Rust adoption. The ecosystem shows its maturity. Core crates like serde and clap are rock solid. The ecosystem feels complete for 90% of the core use cases. Here's Kiril Videlov from GitButler. The median quality of a library or crate that you can find is greater than the median quality of other ecosystems. And I have been exploring languages for many years now, and I genuinely feel this way. Yet, this growth brings new challenges. Creates.io shows its limitations. For example, it still doesn't support two-factor authentication, and name squatting is a constant annoyance. Personally, I would also like to see the addition of namespaces, but I know that's a controversial topic. Some teams have faced challenges with cargo at scale. You know, it's nice and it's got its problems too. Like, you know, I mentioned cargo being really useful before. It is true that cargo is wonderful and it's definitely very useful, but also we basically had to write a build system on top of cargo to make this work because cargo has several deficiencies with dealing with this kind of problem. Is actually one of the first things that i worked on at oxide was sort of rewriting the build system yet again on top of cargo you know definitely like it is it is a very different thing than developing a web app or developing something that's a much higher level and you know it's definitely some people enjoy it and some people don't and you know i used to work on that stuff and i found that it was a pleasure but you know obviously everybody has their different levels of expertise tell me a little bit about that what were some of the limitations that you ran into with cargo and also why did you have to build your own build system how does that look like yeah so a very simple and straightforward one is that cargo kind of very deliberately does not include any post-processing of stuff and we need to build an operating system image so like on some level there is a step after cargo runs you know cargo produces a program or a couple programs and then we need to like assemble that into a thing that you would actually you know load onto to the microprocessor. However, this approach remains the exception. For the most part, Cargo works just fine. And where it lacks in functionality, other ecosystems like ProbeOS improve developer experience for embedded environments, for example. The community faces different problems than it did in the past. How do we handle knowledge transfer in larger teams? How do we maintain consistency across massive code bases? How do we handle the scale that comes with success? I've noticed a shift in the people too. The early enthusiasts who shaped Rust's direction are still around, but they are now joined by practical engineers who just want to get stuff done. Some might even say Rust is the new Java. And while that might sound a bit negative, it actually signals that Rust becomes a mainstream language. I personally use Rust for everything from prototyping to production, and it feels great to have a single stack across all programming tasks. Julius Gustafsson from Volvo agrees. We quite quickly figured that having a 100% Rust-based solution would actually be a much better choice. Why? Yeah, because you don't have to worry about all these FFI boundaries between C and also when you're building it on bare metal embedded, it kind of depends on what compiler you're using, what linkers you're using, because the C libraries, they expect certain initialization to be done in a certain way. And so there may be incompatibilities there that, yeah, so even though Rust and C can interoperate seamlessly, there are. There can be some issues when you're doing it on a bare metal target like this. The fact that you can actually, you know, take different components from the community and you can use them with confidence that you don't really have another. I mean, both hassle-free, almost always. It just builds and works and you can just use it. And also, you know, all these things that you always have to take care of. First, you need to, okay, can you get it to build in my build system? You know, and first time, probably not. You probably need to spend a lot of time to get that to work. But then when it actually builds and you start to use it, does it use memory in the same way as you do? Does it make the same assumptions? Like who is allocating this buffer? Who is freeing it? Who has the responsibility to do what? And so on and so forth. Those are not standardized in C and C++, meaning that, yeah, so even if you can build it, it's no guarantee that you can actually use it in any productive way. But in Rust, on the other hand, everything is, maybe it's not true to say everything, But I mean, you're pretty much guaranteed that it will work. And also, like you said, I don't think there's any language that is as scalable, like from the lowest, smallest microcontrollers to like backend systems or whatever biggest server farms that you can imagine and everything in between. Big tech's influence on Rust grows stronger. Google, Microsoft and AWS shape Rust's development through the Rust Foundation as Platinum members. For better or worse, they control a large part of the narrative now. But Rust's heart still beats through individual contributors. We need to support them more. Companies need to step up their sponsoring game. The number of conferences has exploded. I can barely keep track of all the Rust events anymore. That's why I created a dedicated calendar with all events. Check out the show notes for the link if you're interested. But there's still room for more. I wish for more US-based conferences in the future and more efforts like Rust Asia. We also need a conference or two in Africa. Does anyone want to rise up to the occasion? It's normal now to run into Rustations at any tech company I talk to. Sometimes they're quietly using Rust for internal tooling. sometimes they're building entire platforms with it. It makes me happy that they think using Rust is the obvious choice for their problem. We've come a long way. But with this growth comes great responsibility. Crypto companies are still some of the biggest employers for Rust Talent, although recently they struggled to fill positions as the hype moves on to AI. We're seeing more people put Rust on their CVs, more or less to impress, without the deep understanding that made early Rust developers stand out. It's a natural part of growth, but something we need to be mindful of. Chapter 5. The technical evolution. The technical landscape of Rust looks very different now compared to just a few years ago. Async Rust has matured significantly. The paper cuts that bothered early adopters have mostly been addressed. Tokyo won the runtime wars. Async STD is effectively dead. That's actually good for the ecosystem. Although I personally think structured concurrency and small async runtimes deserve more attention. Many companies now maintain official crates for their use cases. For example, Microsoft maintains an official crate for the Windows API. FFI with Python has improved dramatically. We've gained async closures and async traits in Rust. It has more const capabilities and better compile times. The foundations feel pretty solid. Here's steve again. Microsoft is rewriting some legacy windows stuff in rust like gdi the graphics drawing interface they have like a port of that to rust stuff and so you know there's just there are millions upon millions of rust lines of rust out there and being used for real important things, if something were to happen to the rust team like it's also worth thinking about what would it mean for rust to die okay that would mean that the rust team would somehow not exist anymore but the code base would still exist of rust itself and so like at that point those companies it would be an existential threat to their business for that technology to implode and so they would have to come up with some alternate way of making that work because like you know it can't just stop at this point like rust has reached escape velocity and is a language that like will survive now is it will survive in the sense of like vb.net where like it technically exists but not a lot of people use it but like or is it a COBOL where like it's used for some things but like not for anything else i have no idea if it's that kind of legacy or is it like a c and c plus plus where it's used for smaller and smaller amounts of things but still for very important things i don't know but it's definitely past the point where it will just like disappear at some point because it is just, used in too many things that are too meaningful the. Tooling story has some bright spots as well in my opinion ross drover shows promise many of the teams i work for use it as their main ide and it's a fantastic choice because it comes with awesome quality of life features like refactoring capabilities and automatically moving code into modules while fixing the imports. It's always a good sign if other languages copy our ideas as well. For example, result types and pattern matching are commonplace now. But of course we still face challenges. Compile times for large projects remain a constant pain point. Many teams spend significant money on beefy build machines just to keep developer productivity high. And the LLM revolution has changed how people learn Rust. Sure, it helps with the initial hurdles, but I see way too many developers letting LLMs take the wheel and write all their code. This way they miss out on understanding the core patterns that make Rust so unique. My advice is to learn the fundamentals first without LLM assistance. And once you build up that muscle memory, you can use code generation to do the boring part. In the Python ecosystem, Polar's and UV show Rust's disruption potential. Especially Astral's UV is pretty much the standard in the Python ecosystem now, at least in the teams that I talk with. I hope to share more about that in the upcoming seasons. For the first time in a decade, we see some innovation on the VCS front. GitButler and JJ, both written in Rust, challenge the status quo on how we think about source control. and I love how Zed is a snappy new editor written in Rust. Yeah, Rust is known as a performant language but I think that fact is overemphasized outside the Rust community. Rust is just performant by default. The community doesn't really have to try too hard to make Rust code fast. Here's Anton Lazareff from Brave. I think what's standing out here is the really nice thing is that I don't even have to think about performance when I'm writing the Rust code versus when I look at, say, the uBlockOrigin codebase. There's so much kind of convolution that happens as a way to squeeze out extra performance. And in Rust, I really just don't even have to think about that. And it still is competitive. I get firsthand feedback from clients that they want reliable services first and fast services second. Performance almost becomes a side effect of writing reliable software. From what I see, reliability trumps raw speed for most teams. And the Rust compiler makes teams move forward with confidence, as Scott Chacon affirms. From the Ruby world, it couldn't be a more different language, right? And so I do like the compiler stuff. It is nice to be like, most of it is dealing with compiler issues rather than runtime issues. So if it compiles and it actually runs something, then it probably runs the way that I meant it to. Whereas in Ruby, you don't know until it runs into an issue and there's no good debugger and things like that. The language lends itself really well for maintainability and really exactness. There is no time that I'm like, oh, is this going to fail with some random input? It's like, no, you directly define what the input is going to be. And then as long as it's that, you're fine. And it will fail well. I mean, it will air well in the other cases. And so there isn't just this ambiguous, kind of vague definition of things. It's very, very exact. And in the beginning, when you learn the language, that can be off-putting. But I do feel like it's the hugest pro of the language is the exactness of it. That was Jessie Frazelle from Zoo. Chapter 6. Looking Forward Rust 1.85, also known as the Rust 2024 edition, is around the corner. It is set to release on February 20th, 2025. I love that the language gives strong guarantees around backward compatibility. Companies can finally plan long-term migrations. Being able to plan is very important for companies. Enterprise support is important as well, and we're still lagging behind there. But with projects like Ferrocene, we're starting to see a glimpse of what the future might hold. On the language side, I'm excited about some upcoming features. Async trade has finally landed, and generators start to take shape. Letchains will make complex conditional handling more ergonomic. And adding future to the prelude is a clear sign that the sync and async world are gradually coming together. But beyond specific features, I see Rust expanding into new territories. The aviation industry is taking notice. Space tech companies are adopting Rust for mission-critical systems. Healthcare and other highly regulated industries are exploring Rust's safety guarantees. Car manufacturers are building their next-gen user interfaces in Rust. These aren't just plans I know that well-known companies work on such projects internally and some of them have already reached out about sharing their stories in season 4. Keeps me excited after all these years of using Rust. Well, for one, seeing production systems just work is a reward in and of itself. And learning more about how computers work internally and building on top of these solid foundations is just great in and of itself. To me, it's also just using Rust across the entire stack, from low-level systems to web frontends. It's clear to me that we're entering a time of stability, and that's exactly what enterprises need right now. Before we close, we would like to make a few announcements. First off, we'll be at Rust Week. Come say hi if you're there. We'd love to chat about your thoughts on Rust and your current projects. Tickets are still available, and the lineup looks fantastic already. And by the way, we got Rust in production stickers. I want to thank Simon Brüggen for his incredible work editing this show. It's thanks to him that this podcast even exists. His attention to detail and dedication to quality make every episode much, much more enjoyable. We're coming back in April for Season 4 with another fantastic roaster of guests. If your company uses Rust, whether you're a startup or a Fortune 500, reach out. We'd love to hear your story. I rarely ask for this, but if you enjoyed this season, please take the time to rate the podcast on Apple, Spotify, or wherever you listen to your podcasts. Even better, share your favorite episode with a friend or a co-worker. Word of mouth helps us reach more people and bring you more great Rust stories, and that helps the Rust ecosystem grow and prosper. And that's all for today. we hope to see you back in season four rusting production is a podcast by corrode it is hosted by me matthias endler and produced by simon bruggen for show notes transcripts and to learn more about how we can help your company make the most of rust visit corrode.dev thanks for listening, to rusting production.