Hello everyone, it's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. I'm Matthias from corrode, and in this very special live episode from Rust Week 2025, we talk to Niko Matsiakis, Rust Core Team Member and Principal Engineer at AWS, about building foundational software with Rust. Niko, thanks for taking the time. This is a bit of a special episode today because we are at Rust Week in Utrecht, sponsored by RustNL. So thanks for the organizers. Can you talk a little bit about yourself and about your work? Yeah, so I have been working on Rust since 2011. I think I've been present and part of every Rust release. So I wasn't here from the very beginning. Only Graydon can say that. but I came in before 0.1 and I'm still here now and I mostly focus on the compiler and the language type system but I've also put a lot of time into thinking about how we run the project the kind of open source organization, and various other things over the years. What did you work on before Rust? So before I started working on Rust the most immediate thing I was doing was a PhD on, surprisingly, programming language design, which some of those ideas, I think, kind of made their way into Rust. But before that, I worked at a company writing a compiler. The company was called DataPower. We built an XSLT compiler, which is a functional programming language for transforming XML documents to other XML documents. And in the early aughts, you know, that XML was very hot. JSON didn't exist. Nobody cared. XML was going to be everything. And there it was. So it was kind of fun because this is a purely functional language. And so it had a lot of inefficiencies. Like, for example, if you want to have unique elements, you had to do something like for each element that doesn't appear earlier in the list. So that's like an on squared traversal. But our compiler would recognize that you were doing this. And we would, behind the scenes, insert a hash map, for example, so that it could be 01. So that way we could be like arbitrarily faster than everybody else's compilers. Just depending on how big I made the input essentially. You want a 10x win? No problem. 100x? No problem. I'll give you what you want. Did you have any influence on the language design? No, no. Okay, so was it also your first experience with building a compiler or working on the compiler? Yeah, I was fresh out of college. I think I took a compiler course and thought, this is the best. So then I found a job doing that. I thought that I knew everything, pretty sure. Pretty sure I thought I was the best programmer that had ever walked the earth at that time. But I met in this job, I met some really smart compiler devs, people who have more experience than me. And I came to realize, oh, there are some things I haven't yet learned. And so that was actually why I wanted to do a PhD, was I thought, oh, maybe this would be a good way to get deeper and really understand more stuff. What was the compiler written in? In C++. Okay. And how did that feel on a day-to-day? I love C++. I had a great time. I mean, I definitely, I remember debugging some data races that were pretty fun. I remember watching The Lord of the Rings and it would be like, because it took like two hours to reproduce it. So I could watch like half of a movie while the thing would run and then I'd get one extra thing and I'd go, okay, that doesn't help. Let me add one more printline and then restart the test and go back to the movie. So it was good times. You know, C++ is a cool language that let you get super control and do all the things we wanted to do. So that was great. But yeah, definitely the debugging and in general correctness costs were significant. So you studied in Zurich at ETH, and then you started working at Mozilla. Yes. And by the time you arrived at Mozilla, was Rust already on your radar? Did you hear about it? Yeah, I came to work on Rust. I'd heard of it. I found the idea of a new systems programming language pretty exciting. And it had just transitioned. So I never used OCaml boot, which was the OCaml-based compiler for Rust. It had just gone self-hosting around when I started. But I was excited to see what it could be. I thought it was completely hopeless. I mean, it was like totally Quixotic. I don't know how you pronounce that word, but Don Quixote-esque. Essentially, like, but it would be fun while it lasted. That was sort of my take. Maybe we'll find some good ideas. Steve Klabnik once gave a talk at FOSDEM where he called that time the great in years. And not many people know a lot about Graydon. He's a bit of a mysterious figure. How was working with them back then and what was your experience back in the day? Can you talk a little bit about that? Yeah. Well, when I first got there, Graydon was on a bit of a temporary leave, but he came back and I really enjoyed working with him. I mean, he's a really smart guy. He's got an incredible knowledge of programming languages, like a encyclopedic one which you can see if you go back to the old language it has all these esoteric keywords that he chose out of like an homage to older languages i remember at some point he wrote a message that was like don't take my esoteric keywords away or something like this but like leave me one weird keyword but you know like it used to be that instead of enum we wrote tag and instead of match we wrote alt and instead of return we wrote ret instead of continue we wrote cont. I was like, I think that might get, I think we should pick another word because that might get mispronounced and it's going to sound bad to some English speakers. And so then we went to like loop and again, I always thought loop was kind of clever. That was one of his, it was like, how do you go back around the loop? You write loop. But anyway, eventually we decided, no, we should just, we should just pick more normal keywords and landed there. But by we, you mean you? No, this was, well, this was after Graydon left. It was really me and Brian Anderson and Patrick Walton, who at the time were kind of the core team, the first core team of Rust before we had any other teams. And back then there were also quite a few pointer types that don't exist anymore. Yes. Well, they do exist. They've just gotten different names. You mean like the ~T and the @T. Yeah. Now, I mean, we had, I think I would say in general, what Rust's big evolution went from, Everything, we took a bunch of patterns, and they were all built into the language in the beginning. And then over time, they got moved out into more libraries as we realized the power of the basic type system concepts we were adding, right? Like of unique types that you can move from place to place, of borrowing. And that eventually got us to where we are now, where I'd say the actual core language of Rust, I mean, Rust is a pretty complex language. I think that is true. it's got a lot of stuff to know and understand but there's a core that it all kind of de-sugars to that is actually very very simple and that took us a while to find and get to, so in the beginning you would have for example there was a green threading library and you would launch tasks and the counting between tasks was all built in and we had these built in pointer types that kind of mirrored RC and box what is today RC and box, and all that stuff moved out right now the threads are in the standard library, Now RC and ARC are in the standard library as is Box. Box is still a little special. It's secretly the compiler knows things about Box it really shouldn't to make some stuff work. But it feels like it's in the library. And I think what that enabled is two things. It felt simpler, even though it's just a change of name. The fact that it's not a built-in character made it feel like applying a concept you already knew rather than a whole new thing. feels simpler than @T. But also it meant we could secretly explode the number of pointer types. So I would say Rust has a lot more pointer types now than it used to, because now we have, I mean, if nothing else, we have RC and ARC. But we have pointer types out in the libraries. We have PyO3 and C++ Interop. I think each have their own sort of Py wrapper type and stuff like that, right? And so we've been able to... Adapt Rust into a lot more scenarios by focusing more on the core capabilities that you can use to build abstractions and not only to build abstractions, but to build safe abstractions and letting people in the ecosystem build their own libraries that create those abstractions. I would say that's a pretty C++-inspired thing in my view. I think C++ is like one of the original library languages. They may even have coined that term. Probably they did. Where they really aim to make you able to write everything in a library that you could do in the language. They just don't have the safety part of it, which I think is pretty important. So back then Rust looked very different than it does now. I can still remember the pre-1.0 days. People used the nightly Rust compiler. Things would change on a more or less regular basis, maybe weekly basis even. There were breaking changes. You touched on that when you said we had a green threading ecosystem or runtime and we had a garbage collector and that sort of thing. It was a totally different language by then, but there was a simple core. And if you had to explain that simple core to someone who's technical but doesn't know Rust yet, what would be the simple core of Rust? You set that one up for yourself. I did. That was dumb. The simple core of Rust, in my view, is you have ownership and borrowing, and you have trait dispatch, as well as regular functions. Ownership and borrowing, that's the main way we do our memory management. The idea that a given value has a single owner, and when it goes out of scope, it's going to run a destructor. That simple concept, on top of that you can build Rc, you know, ref counting is just decrementing and incrementing the ref count at the right times and so on. Borrowing is just getting temporary access to something and knowing that it won't be dropped or mutated while you're accessing it, which then turns out to translate to kind of, that's the key property you need for an awful lot of things where you want to say, I need to give access to my internals but I need to make sure that that access as long as you're using my internals. I don't break you, essentially. And other languages don't have a way to do that, at least not to enforce it. So we have the ownership and the borrowing. The trait-based dispatch is the way to say, for a given, here's a given operation that can be implemented differently for a lot of different types. And so to have the same concept, like adding a number or adding things that can be applied. You can add numbers, you can add strings, you can add whatever. That's a key concept. And we use that then as a building block for a whole bunch of stuff, including things like marker traits, where there isn't even a method. It's really just like a set of types. So send and sync saying the set of types that are thread safe and that can be safely shared across threads. Those just are built on that simple core of types that implement traits. And then the last part is just kind of the standard C procedural programming. You know, we have structs and enums and functions and all that stuff. Yeah. But when you cooked it down to these essentials, and that happened, I guess, after the Graydon years when you were preparing for Rust 1.0, you had to remove certain things. You had to cut down the scope of the language. Was there a time when you realized, wow, there's really something novel there that hasn't been done before? Was there a moment when you realized, wow, this is something, all of these concepts that you mentioned, ownership, borrowing, trades. These form this really robust system? No. I think it came gradually over time. And of course, me being me, I'm the more thinking of it like, oh, all these concepts, they're so limited. There's still stuff we can't do. But that is to say, I think there really are a really robust system and they can do an awful lot. But I'm also aware that there's some areas where I wish we could do more. Like we can enforce that a reference to, we can enforce that as long as you hold a reference, you won't drop or you won't mutate the data inside. But sometimes you need more rich predicates. And I hope we get there someday of maybe even, in other words, I think there's still another generalization down. And I have secret wishes that we can find a way to get there and give just a little more power to build safe abstractions. But the question will be, can we do that while keeping the language lightweight and easy to use? I don't know. But yeah, I think it's something that we came at very gradually. And I don't know exactly when it, I can't recall exactly when it became clear that we had kind of landed on it. But I think it was, I guess I would say, at the point where we started seeing people come in with their own libraries doing surprisingly cool things based on the same set of type system concepts is when it became clear that, oh, this actually has legs. You can scale this up past the original stuff that we wanted to do and off into stuff we never even thought about. Why did it take so long? Because you couldn't have enforced these concepts in other languages that came before Rust. We have a long history of systems programming languages and none of them introduced ownership and borrowing as like a compile time guarantee enforced by the compiler. Why did that take so long? I mean, I think we were building on a pretty long history, both in industry and in research. It took a long time. Like when I wrote, the C++ that I wrote in the early aughts is not the C++ people right now. We used lots of pointers. Well, okay, it's debatable. It's not what they would call modern C++. We used lots of pointers. We used new and delete and all that stuff. And it took a while to develop and recognize the best practices and start to encapsulate them on the C++ side. But meanwhile, in research, there was ongoing work developing out linear types, which is to say unique types, borrowing. Those kind of concepts were also being explored, both in the context of systems programming and in terms of higher-level languages like Java, where they're still valuable. And so I kind of feel like the answer is because it wasn't obvious. And it took a while to get, you know, all the ingredients in the right place that you could kind of assemble the right stew out of them. Man, I'm not sure if that's the best metaphor, but in my view, like research and new things are always, almost always a combination of stuff that came before with a few tweaks and fit together in a new way. And I think that's the same here in Rust. We found a pretty good mix. But a lot of the stuff that we built on wasn't available like people hadn't thought of it five years before, ten years before. It was kind of nice when you said people can build really interesting things as libraries and they don't have to touch the core of the language and I guess that's also thanks in part to the trade ecosystem and enabling composition over inheritance for example. So can you maybe talk about some of the libraries that you saw that really astonished you that fit that description? I mean... Well, I don't know if I'd say that the regex crate astonished me, but it impressed me with the fact that it was able to be so complete and also so fast. It was like one of the very first crates where they came out for Rust, where suddenly you had this robust capability you didn't have before. And it was like, oh, it feels like using a real language. And if you look at it, it uses all these properties, like the ability to, when you match a regular expression against a string. You get back these match objects that are holding references into the string so they make sure that the original data doesn't go away while you're using the results of the regular expression match. And then you can cast them into ampersand stirs so you get out just a regular string data type that you can pass anywhere but it's tied all the way to your original string. And so actually even that simple interface winds up leveraging those concepts. I think that was one of the first where I looked at it and thought people are building like real stuff that's like i said feels like a real language and a competitive one faster than most of their competitors than the other regex implementations out there i mean rayon was another moment now that's one of my crates that's a little like to my own horn but nonetheless it was the result of a lot of iteration to get to the rayon interface and once we realized once it was like oh actually this this just works you know you can just make any iterator not any but most iterators parallel and they'll hold all the references they need and keep the stack frame alive and that was pretty cool that was i felt like oh we finally got it and and actually if you go back i guess maybe that was the moment to your earlier question at least as at least a moment because i remember i have notes somewhere i have to go back and find them but you know if different i had in mind something like rayon right and over it was evolving as the language evolved and it was much messier before and there was a point where I remember realizing oh I can take away like all that stuff that I used to have around at types and how to deal with them in the compiler and so on none of that's necessary because it all it's actually all just send and sync traits now that capture everything you need, I'm trying to think of others I mean. Tokio crates there's just so many good crates out there now it's like actually really hard to pick. But the Tokio grade is a great example honestly because for me it wasn't clear that you could build an entire async ecosystem as a library. Yeah and that i mean and a really performant one also right because. No other language has pulled it off before and i'm not even sure if any has pulled it up until now, You don't have to write the library itself, but also the trade system that goes with it, the futurist trade and so on. Were you involved in some of the discussions? Did people ask you for advice in that area, or did they completely build it independently? I wasn't very involved in the design of the future trade. I mostly followed along. Where I got more involved was really the last mile. Basically, should we write a weight X or X dot a weight? The hardest of all decisions that had to be made in that discussion, at least based on amount of time and a strong feeling that went into it. But I wasn't involved in the actual design of the async system itself. I have since become much more involved, more building on the foundations that I would say Alex Crichton, cramertj, withoutboats, Carl Lerch, Those are the names I mainly remember going around at the time. I'm sure there are a bunch I've left out. It's due to my ignorance, not their lack of contribution. Well, Aaron Turon, of course. I shouldn't overlook him. Correct. Do you think we made the right call? Dot await? Oh, definitely. Why do you think so? I mean, well, reason number one, every time I use a language that goes the other way, I'm like, oh, my God, this is so annoying. But also it's just it fits in with everything else it's you you want to be able to put a question mark after it you want to be able to do awaits in a row to say hey i take this thing which gives me a future i dot await it i do something else which gives me a future i dot await that basically the more we can shift towards towards post fix the more smoothly the language operates. I feel like it's kind of a thing we inherited from, I don't know, ALGOL or whomever's, I mean, we inherited from C, but people usually say ALGOL-like for some reason. Because I looked at ALGOL and it does not look like C. So I don't know what they're talking about. But anyway, it's this sort of mix of prefix and postfix operators, and they just kind of make everything a mess. So overall, I think, yes, we definitely made the right call. And I wouldn't mind if we moved a few other things post-fix as well. For example? Oh, man. I walked right into that one. Too. Let's put it this way. I think the arguments in favor of postfix deref, so foo.star, even postfix borrowing, foo.ampersand, I mean, no one's proposed it, but I think it makes more sense. But the one one thing i would really like to see is match like being able to say something something dot match and the pattern so that you can say call a function and then match on the thing that it gives back to you without having to move the match in front i mean usually that's fine but it's not always especially like we've like with a try catch kind of scenario where you might which we don't yet have as a language feature, but we've talked about try-blocks, which would take the question mark and sort of intercept it. So you could write a try-block and have a bunch of things in there which may have results and have question marks on them. And then that entire contents of that would have type result. And if you do that, that's great. That's super useful. But now maybe you want to match on that result to decide if, hey, if the result was okay, I'm going to do one thing. And if there was an error somewhere in that computation, I'm going to do something else. And right now you have two choices. You can either write like match, try, open thing, lots of code, close, curly, open curly, which is very awkward. And it just seems weird. You've got this match that by the time you see the arms of the match, you kind of forgot the match was even happening. Or you can store it into a variable, which just looks dorky. So I don't know. It feels like a clear case where it would be useful. I guess one main argument against the postfix notation was not necessarily about the postfix notation in and of itself, but the fact that await was not a method, but it looked a bit like a method without the parentheses. But then it wasn't clear if that was confusing or not. So there was some pushback there. Can you remember what was your standpoint on this? I mean, people were concerned that it would look like a field or look like a method. I think it doesn't. I think everyone knows what it means. I think the most interesting part of that discussion, there were two interesting parts. One was or one is a thing that comes up a lot which is how important is it that you have like should we be looking at the experience when you're in an ide where you have syntax coloring and syntax highlighting available to you or should we be looking at the experience where you don't and it's all plain text because it's pretty different than if the thing is bright red versus gray like that's very noticeable but the other thing the other related thing is can we can we put ourselves in the shoes of how we're going to feel about this feature two years from now three years from now versus how we feel about it now and that's often pretty different and i think a lot of times we over rotate on how we feel about it now so a lot of times i do ask myself when we're facing a tough question like basically just think of think of what's going to feel the best when you're used to it and and optimize for that more than what makes the easiest transition from what i already know. And it's not to say I don't think we should worry about how you learn things, but in the long run, I spend a lot more time using the language over time. So you want it all to feel nice when it's done. The other interesting thing about that debate was I remember there was so many threads with so many comments. And at some point, I opened another thread and said, okay, only comment on this if you've actually used the feature. And that was like crickets. There was like three people commented there. And I thought, okay, that's kind of what I thought. It's very easy to jump in and give an opinion, but it's so often not really informed by real experience. And I found like just using a thing, usually within just even a minute of using it, kind of clarifies a lot of these things. You thought you had a whole bunch of questions and they get wrapped up very quickly as soon as you've used it, just for a few minutes. Yeah. A lot of things are obvious in hindsight. Hindsight is always 20-20. I do remember that I also found it a bit weird, but now I see it as a keyword. I see await as a keyword. I don't even read it as anything else by now. But we are in between 2015 where we had Rust 1.0 and 2018 where people started to use Rust for real. And in the meantime, a lot of things happened because a lot of embedded developers and people that wanted to play around with async rust were still on nightly so things were not as smooth yet and it's very hard to make such decisions under pressure because 2018 was a bit of a crunch to get everything ready take us back to that time how was that making decisions on the pressure and keeping the team together. So well first thing await wasn't await wasn't technically gated on the rust 2018 edition so we decided that kind of independently which is good there was enough pressure on that edition as it was i think that was a bit of a tough time i mean i think it was the right well i'm happy that we did the 2018 edition and i think it was a watershed moment for rust where we really it wasn't just the work we did on the ergonomics initiative which for those of you who weren't paying attention at the time was an effort to like reduce basically reduce foot guns reduce paper cuts and in general make rust simpler to use right and that was controversial like the decision to do that was fairly controversial because in a in a way it's going against the rule that i said of focus on what you're gonna want at the end because a lot of the more experienced users were saying hey i've learned that stuff already like you know we should be doing whatever we should go deeper into this area or do that and and we were saying no right and we pivoted back and i think that was the right move both because yes it was still important to learn rust that was not good enough yet but also because actually i think even for those users and i hope many of them would agree now they still had a lot of rust left to write and like that paper cuts are still been adding up all this time right like it just made rust much nicer to use overall And that was important for experienced users more than they may have realized. I think sometimes you don't realize how much of a toll the little ergonomic hits take on you as your writing code. So we were working on that. And I think the things we did were important. But the fact that we chose to focus on that was also really important. And I'm really glad we did it. And I feel like for me it helped. Because... I remember coming out of 1.0. 1.0 was a really clarifying moment, right? Because you could say, we have to reach stability. Here's a feature list. Let's work down our list of stuff we got to do. And all right, maybe you could punt something to make it unstable. But we could really focus our effort and really know, what is the right next step to take at any moment? It was fairly easy to answer. And after 1.0, it got more complicated again. Because now all those questions that you kicked down the road, You said, well, we don't really need to stabilize this in order to have a sensible, usable product. Now, all of a sudden, you look over this huge field of priorities and you say, which ones am I going to do? And that becomes hard again. And I think the ergonomics initiative, I think, was our first effort to sort of actually point in a direction and try to walk there. Saying, yes, we know there's a ton of important things to do. We're going to pick this one because to us, it seems like the right place to focus our effort. And we're going to go there. and then we'll come in and do some of those other things and that i think that's something that. Is really important to do as a leader in a community to be able to put yourself on the line to some extent to be brave enough to say this is what i think we should do and stand behind it versus trying to embrace every possible route so that nobody gets irritated and if you go back and look at like i have this blog post that Aaron Turon like liked or likes to make fun of a blog post an internals post but you'll see one where i summarized all the things that we could do after 1.0 and it was basically like a laundry list it was like 6 000 features thing here's our next set of priorities we're probably still working on half of those i think being able to focus on, a small number is really important. And especially the ergonomics part because those small little wrinkles they make people trip up over time and since this is a live podcast i'm curious is how many of you, raise your hand, even the one sleeping, Know about non-lexical lifetimes. Who has heard of non-lexical lifetimes? Raise your hand. It's like half the room, I would say. I guess that's a win because the other half doesn't have to deal with that. That was back in the day when Santiago Pastorino worked with you on non-lexical lifetimes. That was a really good lesson for me. For your listeners who don't know, non-lexical lifetimes was a change in how we did borrowing in Rust that made it not have to be tied to the lexical structure of your program. So it used to be that many times we would consider things borrowed for way too long because we were very simplistic about, like, you couldn't borrow for just the if. If you had an if and you used the pointer in the then part of the if but not the else, it would sort of expand to the entire if-then-else. It couldn't be just one half. And to be honest, that was by design because I thought, oh, it'll be easier to learn. Like, yes, you'll have more errors, but it'll be easier to learn if the formalism is simpler. You know, because I was a PhD student coming into language design or whatever, like I was looking at the AST structure and thinking this is good. I wasn't thinking about it from the point of view of a programmer using Rust who has not authored compilers or studied type theory and so on. And so it was kind of an aha moment for me to realize that what seems simpler to the compiler author doesn't necessarily feel simpler to the language user. And... That's shifted my thinking in several cases where I'm really asking myself, all right, I think there's a balancing act. You want them ideally both to be simple. I think you can go too far on either side. If you just say it doesn't matter at all how complex the compiler is, then it can be really hard. That ends up leaking into complexity for the user in the end because they really can't wrap their head around the algorithm. But on the other side, you really have to think about how are they going to experience it and feel it and try to put yourself in those shoes. And that was also right around the time when this first notion of epochs was introduced, which are now called editions. This promise of robustness and the guarantee to the industry to say we won't break things so you can depend on it. It was, I think, another very striking moment in Rost's history. Can you talk a little bit about that and what was the reasoning behind this and how Rust is tested as well. Well, the reason behind the stability promise, which was the 2015 Rust 1.0 release, was very simple. We had been changing Rust on a, as you say, weekly, daily basis. And we knew that we'd been working on it for a long time. To be frank, we knew we had to show some results if we were going to continue funding this project. And we also knew, we had a sense that, like, the world was being patient with us there was a lot of enthusiasm about rust people liked hearing about it rust news would go to the top of hacker news but how long could that stay we had to start seeing some production results and really getting out there and seeing it be in use before it becomes kind of vaporware so it seemed like the biggest blocking thing was stability and so that's what we focused on but yeah i think around 2017 i don't remember exactly when there were definitely some decisions where we started looking and saying, you know, in hindsight. That may not have been the best call, right? And I think there are lots of, particularly the module system was one where syntactically, the older module system was really tripping a lot of people up. And it was like, well, we made a promise for stability, but we know that the future is way bigger than the past. And these changes are sort of, yes, everybody has to make changes, but they're so, so programmatic. Like the compiler could literally change it for you. It seems like a real shame to get stuck in this suboptimal design that we see causing problems just because we said we don't want to break people's code, oh, maybe there's a way we could do neither, right? We could make the change and not break people's code. And in fact, I did not do this on purpose, but now that you mention it, the inspiration for additions came from XSLT. I forgot that. Because XSLT, that XML transformation language, if you go look at it, the very first thing you do in every XSLT program is you write version equals one. And they have an XSLT 2.0, which can run XSLT 1.0 programs. And it knows that they're not the same because it sees the version equals one. It doesn't quite work like our additions. If I recall, it runs them almost the same, but not quite the same as they used to run. It has like a compatibility semantics. But still, I remember thinking that that was a really clever idea to just a little bit of foresight to just declare the version number in your program. Thanks, XSLT, the contribution to this language. Yeah, I think we could talk about the history of the Rust compiler for a very long time, but I also wanted to change gears and maybe talk about some of the things that changed during that time where people started to use Rust for things that were not necessarily systems programming tasks. So suddenly people started to use Rust for backends and other endeavors. Was that a surprise to you? I'll tell you it was the biggest surprise. The biggest surprise to me was realizing was not so much the tasks that people were using Rust for, it was who was using Rust. Because I thought that Rust was targeting C++ users. But actually, I think that's more true now. But I think at the time, where the biggest initial adoption came from was people who would have liked to use C++, but for the lack of memory safety right folks who were using ruby were using java were using python, because they needed the productivity that it had to offer but they had these hot spots that were a pain in the ass to optimize and they couldn't they either couldn't get the performance they want or they had to work really hard to get the performance they want and this gave them a third option and so that that's that i think that was a great thing for rust also because they were coming from modern ecosystems and they were less forgiving like i remember i was like we don't need cargo make is fine that's true confessions because i don't know i just mostly wrote c and c++ code and make seemed okay kind of did the job but luckily there were people there who had used npm and they were like no Niko you are you're an old you're showing your age we need cargo and they were right and. So so i think it's it brought us like a whole audience of people who had very different who experienced modern tooling and were not willing to accept what was cool in the 70s so that was good in terms of systems programming i mean back-end systems i guess it's kind of debatable if it's a systems program or not but i think i don't remember thinking actively about it one way or the other to be honest because i'm a compiler person and networking is like a little over my head but but it doesn't surprise me now but what does surprise me is actually how good rust is at like scripting like i've had python scripts that i port to rust and it's like oh it's, either 20 longer or not even longer at all it actually feels better in some ways and worse and i don't know it's it's competitive at least if you know rust well and you know what how to use it right i think if you came in fresh you would not think that but so the fact that rust is versatile enough to be optimized for those foundational low-level applications and yet be usable all the way up is pretty important. And a little surprising. I think a cool outcome. And then during that time, let's say... Fundamentals of Rust also changed because Mozilla was struggling a bit with providing funding for Rust on a broader scale, just like many other companies. And then eventually some of the core contributors moved to other ventures. Some of them moved to big tech. How did that change the language? And that was also around the time when the Rust Foundation was founded. Yeah you know i think that mozilla gave rust a parting gift actually in deciding not to fund it anymore and i know that sounds maybe a little odd but like the truth was they had it didn't make that much sense for them and they had been slowly you know reckoning with their own financial difficulties for some time and like the team had been getting smaller but it was like they didn't want to i think i guess it's an interesting lesson right we're not admitting when the time has come can sometimes be worse than just admitting it and cutting off and i think the fact that they said we're not going to be able to fund this anymore was both obvious to everyone but also. Well not to everyone it was obvious to everyone who was close in but it wasn't obvious to all the companies that had been building on rust including my current employer at amazon and so it was a it was helpful for them to get a real like get a reality check of oh hey we're building all of this kind of foundational infrastructure in this programming language who's going to support it well i guess we have to and so i think that it it gave rise to a lot of rust teams across the industry that might never have come to be and that's meant a lot more people working on rust and a lot more investment in rust right and of course the most like visible sign of that was kicking off the rust foundation but i think in some ways i mean that's been really impactful but also the hiring of developers and the investment in the project side and the open source building maintenance side has been very important too did anything changed when you started working at aws or did you do more or less the same as you did? No it's been the same. I remember like i started at amazon and there's like this two week long like set of videos you're supposed to watch or whatever for onboarding and i was like i i can't watch these guys i have like four bus meetings like tomorrow i i don't know i already have a job so so yeah i pretty much have I just have had no real difference between working at Mozilla and Amazon, except for one thing, which I think is good, which is that Amazon's a lot bigger, doing a lot more things. And so I have a lot more contact with teams using Rust at Amazon than I did at Mozilla, if for no other reason than there are many more of them. But also in many different domains. So we have people using it in consumer devices, people using it in embedded, people using it in web services. And that lets me get a lot more hands-on knowledge about how those teams, like what their problems are. And there's something about being at the same company that gives you, like I can go deeper. I can get more and I can push them a little harder. I can be like, just stop being whiny. You can deal with that problem. That's not a real one. I'm like, let's go to the next thing on your wish list and talk that one through because I think you can handle that. And so I've found it personally that it's helped me get more perspective. But it hasn't really impacted my ability to work in the open in any way. Are you involved into any architecture discussions at AWS regarding Rust or is it completely running by itself? Well, I don't know what you mean by architecture. Do you mean like systems using Rust? Do people book meetings with you to talk through systems design problems? No, I wouldn't take that meeting anyway. No, they don't. Maybe a little bit, but not really. And the truth is, I'm not a domain expert in what they're doing. So it wouldn't really make sense to get my opinion on how to build a distributed system. I would rather go listen and learn from them. They will occasionally come to me with like here we have a it's usually around FFI or interop or some very narrow problem where we're seeing some overhead do you have any thoughts on how we might, optimize this case and usually I'll say yeah go talk to the much more qualified people on my team who can help you with that but. A sort of naive take on this would be what's in it for Amazon to hire you. Well I think that's a good question actually it's one i should have a good answer to because uh performance assessment time i think i don't i don't you know i don't provide that sort of leverage right what i do do is well to a certain extent help shape overall language usage strategy but more importantly find the places where people are getting stuck and figure out the best place to fix them right and that could be internal to amazon sometimes it's like we're missing a library but a lot of times it's like no, that's an indication of a problem in the language and drive that fix. That's the sort of short-term benefit. But I think the long-term, what Amazon really is interested in is knowing that. Investing heavily in a language which they know will continue to succeed in the industry. No one wants to write a bunch of code and then find out that there's nobody to hire who knows the language or there's no ecosystem around it so nothing's being developed. You want to be kind of sharing in, this isn't a competitive advantage, so to speak, to use Rust. That's not the way I want to say it. It is a competitive advantage, in my opinion, to use Rust, but it's not a thing that you want to keep to yourself. It's not the thing you're selling to users and customers, right? And so really what you're mostly interested in is having it be as good as possible. And that means benefiting from all the rest of the work that the rest of the industry is doing while contributing in and helping do your share, right? And I think these both are really important. And it's kind of the blend. Like I wouldn't want, I think if everyone who worked on Rust came from Amazon, even if we had the same number of people, you wouldn't get the same quality work out. because we would be overly rotating on the constraints and the concerns of Amazon builders. And so as happy as I am to go talk to those teams and get that detailed look, I think that on its own, it's not enough. What's really great is that I can do that and then hear from like Tyler Mandry and who works at Google and hear from Josh Triplett and all these other people in the Lang team, but also around the community who have insights, similar insights, but to different groups. And often they have pretty different problems even when doing a similar thing. And so when you cut them together, you wind up with a really good overall solution. So that's kind of, I think Amazon recognizes that. And I certainly like to tell them that as well whenever I get a chance, I advocate for this point of view, that the goal is to pick foundational technologies that you can really invest in and really get involved in and be a sort of well-respected participant. That's the goal. And yet AWS builds a lot of really critical pieces of infrastructure with Rust nowadays. I'm not sure if you can talk about a project that you particularly liked or maybe something that shows how Rust adoption can work successfully. Yeah. Most people I talk to are not aware of how much Rust is in production at Amazon. Like every S3 request, every EC2 VPC, you can tell my ignorance of even our own product line, every Lambda invocation these things all go through Rust they also go through other languages I'm not saying that they're exclusively written in Rust by any means but more and more the hot path of a lot of these big services that you know is being rewritten in Rust or re-architected in Rust and there are some services that are 100% so the new, S3 Express One Zone which is like a really low latency add-on to S3 that was added for various super low latency use cases, that's 100% REST, for example. And the new serverless SQL is also 100% REST, which came out at reInvent and is going to be hopefully GA very soon. So there are things coming up that are newer services or newer side add-ons to existing services that are 100% REST, but also we're seeing a lot of REST just here and there, scattered in and out the data flow, the hot path usually of other things. Were these services always established to be in Rust from the get-go or were they written in different languages and then migrated to Rust? Do you mean the existing ones? Like, for example, the SQL, the serverless SQL part. Oh, this one. Yeah, that's an interesting one. That one was not originally, that's been a long time in the making. And it started out, I think, in Kotlin. And they found that for various reasons they weren't able to get the latency that they needed in certain parts of the system. And so they migrated over just those parts into Rust initially as a kind of experiment. And when that went really well, what they realized is, oh, actually when we take these Rust parts and we combine them with the stuff that we wrote in Kotlin, we're having a lot of duplicated logic and we're getting a lot of integration failures just because we haven't been able to test them together. Like the Rust stuff was tested with Turmoil, which is that, this is a really cool Tokyo library. Some of you may have seen that. They can't use that library on the testing they were doing in the other service. In their Kotlin, they had a different thing that did different patterns. And so these kind of integrations were causing more failures. And in the meantime, they realized, actually, they originally thought using Rust was going to be kind of slower but worth it for the results. But what they found is, no, once you get past the learning curve, it's really quite productive. It's just the initial learning curve bump is the biggest problem. And so they ended up rewriting the whole thing in Rust so that it all works together smoothly. And I thought that was a really cool and interesting story in a number of ways. I'm happy with the outcome. I wish that Rust integrated better. I think it would be really cool if you could go back and forth between Kotlin and Rust more smoothly. But also the biggest takeaway I had is this point that I made earlier. It's really important for Rust. If Rust is to succeed, it has to be able to scale, not just with the hardest problems, but with the simpler and all the way up and be productive also in the easy problems. And I think this brings us sort of to the end. I wanted to get back to this notion of systems programming, maybe not capturing what Rust is today. And I guess you proposed a different term for it, which is foundational software. What is foundational software to you and why is Rust a great fit for this? Yeah, so to me, foundational software, it's the software that everybody else builds on. And that could be many different things, right? I think sometimes that's like the kernel. That's very literally the thing that everyone else runs on. But other times it's a CLI tool that's super widely used and it's part of the scripting setup you use. Or I don't know, your WordPress blogging platform could be seen in a certain way as foundational software. And I think. That's what I think Rust is optimized for, right? Is this kind of software that is making the world run, in some sense, our digital world. And that kind of software has particular characteristics. It needs to be really reliable because if it breaks, everybody else's stuff breaks. It needs to be fast because everyone's building on it. And so whatever performance hits you take there, everybody has to work with. And it has this long-term maintenance story. So you sort of stitch in time saves nine. is this English expression that often comes to my mind where it's like you would rather take the time to make sure that your match statement covers every case even if some of them just say panic, than not because you know you really don't want to miss a case. You're going to be maintaining this over the long term or you know that when new match when new variants are added to your enum if you have an exhaustive match you'll be able to find where they might need to be handled and you will be more likely to have a reliable outcome. So in those cases where you really would prefer to take the care, to check for the exception, to handle those things. That's what Rust is optimized for, and I think that's what shows up in foundational software. So I don't mean that all foundational software should be written in Rust. Like probably WordPress. Well, I don't know. I've never read WordPress's source code, but probably would be great in Rust. But I guess also it's working pretty okay in whatever it's PHP or whatever they wrote it in now. But I think that it probably all should be writable in Rust. And it probably would be a nice experience. That should be our goal. And also that similarly, I don't mean the rush should only be used for that kind of software, but that is what we optimize for. That's our sweet spot. Everything else should be good. We don't have to be, it may not be the best. It's a very profound insight to say the software that a business depends on, that is the foundational part. And then you can add, let's say, TypeScript or Kotlin or Python code on top of it. but seeing it as the runtime of the business, the core of the business. That's also one of the first users of Rust in production did. Dropbox. That's true. That's a good point. Yeah. Our very first production. Now, the very first production customer, I think, was Tilda, which is a smaller company. But Dropbox was like our first name band. Something I can say to my parents and they know what I'm talking about. That was cool. So what's next? what is mission of enabling foundational software in Rust for the next three years? Well, alright. What I'm, I have a lot of answers to that question, but the one that I'm most focused on right now, especially here at this conference, is laying out this vision doc project. And the idea is to take a look at where we are and say, hey, we're 10 years old. What's going to happen in our teenage years? And in particular, I think with all the growth of different companies investing in Rust, the number of Rust maintainers has gotten up. I don't know what the number is off the top of my head, but I'm sure it's in the hundreds, if not, well, probably not thousands, so hundreds somewhere in there. That's a lot of people to coordinate, right? That's a lot of work going on constantly. And I feel like we, I would like us to all be, if not on the same page, at least all sharing the same understanding of what Rust's users are experiencing, which things, where Rust is working really well and where it could use improvement and so on. And then using that to make decisions about what the next big initiative should be. So really in some sort of meta way, what I see as our top priority is building that shared understanding of what problems we want to solve more than the individual problems themselves. And as a final question, what's your statement to the Rust community? Yeah, be excellent to each other. I think Bill and Ted put it best and I can't do better than that. Niko, thanks so much for taking the time and thanks for all your work on the Rust compiler. Thank you for having me. This was a very special episode recorded at Rust Week in Utrecht. Thanks to Eric Jornkas, Hugo van der Poel and the other organizers and volunteers at Rust in L for providing the space and the opportunity to record this live episode. We had a blast at the event and we hope to be back next year for a full week of talks and events around Rust. If you want to learn more, head over to rustweek.org. Rust in Production is a podcast by corrode. It's hosted by me, Matthias Endler, and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.