Matthias: Hello everyone, it's Rust in Production, a podcast about companies who use Rust
Matthias: to shape the future of infrastructure.
Matthias: I'm Matthias from corrode, and in this very special live episode from Rust Week 2025,
Matthias: we talk to Niko Matsiakis, Rust Core Team Member and Principal Engineer at AWS,
Matthias: about building foundational software with Rust.
Matthias: Niko, thanks for taking the time. This is a bit of a special episode today because
Matthias: we are at Rust Week in Utrecht, sponsored by RustNL.
Matthias: So thanks for the organizers. Can you talk a little bit about yourself and about your work?
Niko: Yeah, so I have been working on Rust since 2011.
Niko: I think I've been present and part of every Rust release. So I wasn't here from the very beginning.
Niko: Only Graydon can say that. but I came in before 0.1 and I'm still here now and
Niko: I mostly focus on the compiler and the language type system but I've also put
Niko: a lot of time into thinking about how we run the project the kind of open source organization,
Niko: and various other things over the years.
Matthias: What did you work on before Rust?
Niko: So before I started working on Rust the most immediate thing I was doing was
Niko: a PhD on, surprisingly, programming language design, which some of those ideas,
Niko: I think, kind of made their way into Rust.
Niko: But before that, I worked at a company writing a compiler.
Niko: The company was called DataPower. We built an XSLT compiler,
Niko: which is a functional programming language for transforming XML documents to other XML documents.
Niko: And in the early aughts, you know, that XML was very hot.
Niko: JSON didn't exist. Nobody cared. XML was going to be everything.
Niko: And there it was. So it was kind of fun because this is a purely functional
Niko: language. And so it had a lot of inefficiencies.
Niko: Like, for example, if you want to have unique elements, you had to do something
Niko: like for each element that doesn't appear earlier in the list.
Niko: So that's like an on squared traversal.
Niko: But our compiler would recognize that you were doing this. And we would,
Niko: behind the scenes, insert a hash map, for example, so that it could be 01.
Niko: So that way we could be like arbitrarily faster than everybody else's compilers.
Niko: Just depending on how big I made the input essentially. You want a 10x win? No problem.
Niko: 100x? No problem. I'll give you what you want.
Matthias: Did you have any influence on the language design?
Niko: No, no.
Matthias: Okay, so was it also your first experience with building a compiler or working on the compiler?
Niko: Yeah, I was fresh out of college. I think I took a compiler course and thought,
Niko: this is the best. So then I found a job doing that.
Niko: I thought that I knew everything, pretty sure. Pretty sure I thought I was the
Niko: best programmer that had ever walked the earth at that time.
Niko: But I met in this job, I met some really smart compiler devs,
Niko: people who have more experience than me.
Niko: And I came to realize, oh, there are some things I haven't yet learned.
Niko: And so that was actually why I wanted to do a PhD, was I thought,
Niko: oh, maybe this would be a good way to get deeper and really understand more stuff.
Matthias: What was the compiler written in?
Niko: In C++.
Matthias: Okay. And how did that feel on a day-to-day?
Niko: I love C++. I had a great time. I mean, I definitely, I remember debugging some
Niko: data races that were pretty fun.
Niko: I remember watching The Lord of the Rings and it would be like,
Niko: because it took like two hours to reproduce it.
Niko: So I could watch like half of a movie while the thing would run and then I'd
Niko: get one extra thing and I'd go, okay, that doesn't help.
Niko: Let me add one more printline and then restart the test and go back to the movie. So it was good times.
Niko: You know, C++ is a cool language that let you get super control and do all the
Niko: things we wanted to do. So that was great.
Niko: But yeah, definitely the debugging and in general correctness costs were significant.
Matthias: So you studied in Zurich at ETH, and then you started working at Mozilla.
Niko: Yes.
Matthias: And by the time you arrived at Mozilla, was Rust already on your radar? Did you hear about it?
Niko: Yeah, I came to work on Rust. I'd heard of it. I found the idea of a new systems
Niko: programming language pretty exciting.
Niko: And it had just transitioned. So I never used OCaml boot, which was the OCaml-based
Niko: compiler for Rust. It had just gone self-hosting around when I started.
Niko: But I was excited to see what it could be. I thought it was completely hopeless.
Niko: I mean, it was like totally Quixotic. I don't know how you pronounce that word, but Don Quixote-esque.
Niko: Essentially, like, but it would be fun while it lasted. That was sort of my take.
Niko: Maybe we'll find some good ideas.
Matthias: Steve Klabnik once gave a talk at FOSDEM where he called that time the great in years.
Matthias: And not many people know a lot about Graydon.
Matthias: He's a bit of a mysterious figure. How was working with them back then and what
Matthias: was your experience back in the day? Can you talk a little bit about that?
Niko: Yeah. Well, when I first got there, Graydon was on a bit of a temporary leave,
Niko: but he came back and I really enjoyed working with him. I mean, he's a really smart guy.
Niko: He's got an incredible knowledge of programming languages,
Niko: like a encyclopedic one which you
Niko: can see if you go back to the old language it has all
Niko: these esoteric keywords that he
Niko: chose out of like an homage to older languages i
Niko: remember at some point he wrote a message that was like don't take my esoteric
Niko: keywords away or something like this but like leave me one weird keyword but
Niko: you know like it used to be that instead of enum we wrote tag and instead of
Niko: match we wrote alt and instead of return we wrote ret instead of continue we wrote cont.
Niko: I was like, I think that might get, I think we should pick another word because
Niko: that might get mispronounced and it's going to sound bad to some English speakers.
Niko: And so then we went to like loop and again, I always thought loop was kind of
Niko: clever. That was one of his, it was like, how do you go back around the loop? You write loop.
Niko: But anyway, eventually we decided, no, we should just, we should just pick more
Niko: normal keywords and landed there.
Matthias: But by we, you mean you?
Niko: No, this was, well, this was after Graydon left. It was really me and Brian
Niko: Anderson and Patrick Walton, who at the time were kind of the core team,
Niko: the first core team of Rust before we had any other teams.
Matthias: And back then there were also quite a few pointer types that don't exist anymore.
Niko: Yes. Well, they do exist. They've just gotten different names.
Niko: You mean like the ~T and the @T. Yeah.
Niko: Now, I mean, we had, I think I would say in general, what Rust's big evolution went from,
Niko: Everything, we took a bunch of patterns, and they were all built into the language in the beginning.
Niko: And then over time, they got moved out into more libraries as we realized the
Niko: power of the basic type system concepts we were adding, right?
Niko: Like of unique types that you can move from place to place, of borrowing.
Niko: And that eventually got us to where we are now, where I'd say the actual core
Niko: language of Rust, I mean, Rust is a pretty complex language.
Niko: I think that is true. it's got a lot of stuff to know and understand but there's
Niko: a core that it all kind of de-sugars to that is actually very very simple and
Niko: that took us a while to find and get to,
Niko: so in the beginning you would have for example there was a green threading library
Niko: and you would launch tasks and the counting between tasks was all built in and
Niko: we had these built in pointer types that kind of mirrored RC and box what is today RC and box,
Niko: and all that stuff moved out right now the threads are in the standard library,
Niko: Now RC and ARC are in the standard library as is Box.
Niko: Box is still a little special. It's secretly the compiler knows things about
Niko: Box it really shouldn't to make some stuff work. But it feels like it's in the library.
Niko: And I think what that enabled is two things.
Niko: It felt simpler, even though it's just a change of name.
Niko: The fact that it's not a built-in character made it feel like applying a concept
Niko: you already knew rather than a whole new thing.
Niko: So Rcfeels simpler than @T.
Niko: But also it meant we could secretly explode the number of pointer types.
Niko: So I would say Rust has a lot more pointer types now than it used to,
Niko: because now we have, I mean, if nothing else, we have RC and ARC.
Niko: But we have pointer types out in the libraries. We have PyO3 and C++ Interop.
Niko: I think each have their own sort of Py wrapper type and stuff like that,
Niko: right? And so we've been able to...
Niko: Adapt Rust into a lot more scenarios by focusing more on the core capabilities
Niko: that you can use to build abstractions and not only to build abstractions,
Niko: but to build safe abstractions and letting people in the ecosystem build their
Niko: own libraries that create those abstractions.
Niko: I would say that's a pretty C++-inspired thing in my view.
Niko: I think C++ is like one of the original library languages. They may even have
Niko: coined that term. Probably they did.
Niko: Where they really aim to make you able to write everything in a library that
Niko: you could do in the language.
Niko: They just don't have the safety part of it, which I think is pretty important.
Matthias: So back then Rust looked very different than it does now.
Matthias: I can still remember the pre-1.0 days. People used the nightly Rust compiler.
Matthias: Things would change on a more or less regular basis, maybe weekly basis even.
Matthias: There were breaking changes.
Matthias: You touched on that when you said we had a green threading ecosystem or runtime
Matthias: and we had a garbage collector and that sort of thing.
Matthias: It was a totally different language by then, but there was a simple core.
Matthias: And if you had to explain that simple core to someone who's technical but doesn't
Matthias: know Rust yet, what would be the simple core of Rust?
Matthias: You set that one up for yourself.
Niko: I did. That was dumb. The simple core of Rust, in my view, is you have ownership
Niko: and borrowing, and you have trait dispatch, as well as regular functions.
Niko: Ownership and borrowing, that's the main way we do our memory management.
Niko: The idea that a given value has a single owner, and when it goes out of scope,
Niko: it's going to run a destructor.
Niko: That simple concept, on top of that you can build Rc, you know,
Niko: ref counting is just decrementing and incrementing the ref count at the right times and so on.
Niko: Borrowing is just getting temporary access to something and knowing that it
Niko: won't be dropped or mutated while you're accessing it,
Niko: which then turns out to translate to kind of, that's the key property you need
Niko: for an awful lot of things where you want to say, I need to give access to my
Niko: internals but I need to make sure that that access as long as you're using my internals.
Niko: I don't break you, essentially. And other languages don't have a way to do that,
Niko: at least not to enforce it. So we have the ownership and the borrowing.
Niko: The trait-based dispatch is the way to say, for a given, here's a given operation
Niko: that can be implemented differently for a lot of different types.
Niko: And so to have the same concept, like adding a number or adding things that can be applied.
Niko: You can add numbers, you can add strings, you can add whatever.
Niko: That's a key concept. And we use that then as a building block for a whole bunch
Niko: of stuff, including things like marker traits, where there isn't even a method.
Niko: It's really just like a set of types.
Niko: So send and sync saying the set of types that are thread safe and that can be
Niko: safely shared across threads.
Niko: Those just are built on that simple core of types that implement traits.
Niko: And then the last part is just kind of the standard C procedural programming.
Niko: You know, we have structs and enums and functions and all that stuff.
Matthias: Yeah. But when you cooked it down to these essentials, and that happened,
Matthias: I guess, after the Graydon years when you were preparing for Rust 1.0,
Matthias: you had to remove certain things.
Matthias: You had to cut down the scope of the language. Was there a time when you realized,
Matthias: wow, there's really something novel there that hasn't been done before?
Matthias: Was there a moment when you realized, wow, this is something,
Matthias: all of these concepts that you mentioned, ownership, borrowing, trades.
Matthias: These form this really robust system?
Niko: No. I think it came gradually over time.
Niko: And of course, me being me, I'm the more thinking of it like,
Niko: oh, all these concepts, they're so limited. There's still stuff we can't do.
Niko: But that is to say, I think there really are a really robust system and they
Niko: can do an awful lot. But I'm also aware that there's some areas where I wish we could do more.
Niko: Like we can enforce that a reference to, we can enforce that as long as you
Niko: hold a reference, you won't drop or you won't mutate the data inside.
Niko: But sometimes you need more rich predicates. And I hope we get there someday
Niko: of maybe even, in other words, I think there's still another generalization down.
Niko: And I have secret wishes that we can find a way to get there and give just a
Niko: little more power to build safe abstractions.
Niko: But the question will be, can we do that while keeping the language lightweight
Niko: and easy to use? I don't know.
Niko: But yeah, I think it's something that we came at very gradually.
Niko: And I don't know exactly when it, I can't recall exactly when it became clear
Niko: that we had kind of landed on it.
Niko: But I think it was, I guess I would say, at the point where we started seeing
Niko: people come in with their own libraries doing surprisingly cool things based
Niko: on the same set of type system concepts is when it became clear that,
Niko: oh, this actually has legs.
Niko: You can scale this up past the original stuff that we wanted to do and off into
Niko: stuff we never even thought about.
Matthias: Why did it take so long? Because you couldn't have enforced these concepts in
Matthias: other languages that came before Rust.
Matthias: We have a long history of systems programming languages and none of them introduced
Matthias: ownership and borrowing as like a compile time guarantee enforced by the compiler.
Matthias: Why did that take so long?
Niko: I mean, I think we were building on a pretty long history, both in industry and in research.
Niko: It took a long time. Like when I wrote, the C++ that I wrote in the early aughts
Niko: is not the C++ people right now.
Niko: We used lots of pointers. Well, okay, it's debatable. It's not what they would call modern C++.
Niko: We used lots of pointers. We used new and delete and all that stuff.
Niko: And it took a while to develop and recognize the best practices and start to
Niko: encapsulate them on the C++ side.
Niko: But meanwhile, in research, there was ongoing work developing out linear types,
Niko: which is to say unique types, borrowing.
Niko: Those kind of concepts were also being explored, both in the context of systems
Niko: programming and in terms of higher-level languages like Java,
Niko: where they're still valuable.
Niko: And so I kind of feel like the answer is because it wasn't obvious.
Niko: And it took a while to get, you know, all the ingredients in the right place
Niko: that you could kind of assemble the right stew out of them.
Niko: Man, I'm not sure if that's the best metaphor, but in my view,
Niko: like research and new things are always, almost always a combination of stuff
Niko: that came before with a few tweaks and fit together in a new way.
Niko: And I think that's the same here in Rust. We found a pretty good mix.
Niko: But a lot of the stuff that we built on wasn't available like people hadn't
Niko: thought of it five years before, ten years before.
Matthias: It was kind of nice when you said people can build really interesting things
Matthias: as libraries and they don't have to touch the core of the language and I guess
Matthias: that's also thanks in part to the trade ecosystem and enabling composition over
Matthias: inheritance for example.
Matthias: So can you maybe talk about some of the libraries that you saw that really astonished
Matthias: you that fit that description?
Niko: I mean...
Niko: Well, I don't know if I'd say that the regex crate astonished me,
Niko: but it impressed me with the fact that it was able to be so complete and also so fast.
Niko: It was like one of the very first crates where they came out for Rust,
Niko: where suddenly you had this robust capability you didn't have before.
Niko: And it was like, oh, it feels like using a real language.
Niko: And if you look at it, it uses all these properties, like the ability to,
Niko: when you match a regular expression against a string.
Niko: You get back these match objects that are holding references into the string
Niko: so they make sure that the original data doesn't go away while you're using
Niko: the results of the regular expression match.
Niko: And then you can cast them into ampersand stirs so you get out just a regular
Niko: string data type that you can pass anywhere but it's tied all the way to your original string.
Niko: And so actually even that simple interface winds up leveraging those concepts.
Niko: I think that was one of the first where I looked at it and thought
Niko: people are building like real stuff that's like
Niko: i said feels like a real language and a competitive one faster
Niko: than most of their competitors than the other regex implementations
Niko: out there i mean rayon was another
Niko: moment now that's one of my crates that's a little like to
Niko: my own horn but nonetheless it was the
Niko: result of a lot of iteration to get to the rayon interface
Niko: and once we realized once it was like oh
Niko: actually this this just works you know you can
Niko: just make any iterator not any but most iterators parallel
Niko: and they'll hold all the
Niko: references they need and keep the stack frame alive and that was
Niko: pretty cool that was i felt like oh we
Niko: finally got it and and actually if you go back i
Niko: guess maybe that was the moment to your earlier question at least
Niko: as at least a moment because i remember i have
Niko: notes somewhere i have to go back and find them but you know
Niko: if different i had in mind something like rayon
Niko: right and over it was evolving as the language evolved and it was much messier
Niko: before and there was a point where I remember realizing oh I can take away like
Niko: all that stuff that I used to have around at types and how to deal with them
Niko: in the compiler and so on none of that's necessary because it all it's actually
Niko: all just send and sync traits now that capture everything you need,
Niko: I'm trying to think of others I mean.
Niko: Tokio crates there's just so many good crates out there now it's like actually really hard to pick.
Matthias: But the Tokio grade is a great example honestly because for me it wasn't clear
Matthias: that you could build an entire async ecosystem as a library.
Niko: Yeah and that i mean and a really performant one also right because.
Matthias: No other language has pulled it off before and i'm not even sure if any has pulled it up until now,
Matthias: You don't have to write the library itself, but also the trade system that goes
Matthias: with it, the futurist trade and so on.
Matthias: Were you involved in some of the discussions? Did people ask you for advice
Matthias: in that area, or did they completely build it independently?
Niko: I wasn't very involved in the design of the future trade. I mostly followed along.
Niko: Where I got more involved was really the last mile.
Niko: Basically, should we write a weight X or X dot a weight?
Niko: The hardest of all decisions that had to be made in that discussion,
Niko: at least based on amount of time and a strong feeling that went into it.
Niko: But I wasn't involved in the actual design of the async system itself.
Niko: I have since become much more involved, more building on the foundations that
Niko: I would say Alex Crichton, cramertj,
Niko: withoutboats, Carl Lerch, Those are the names I mainly remember going around
Niko: at the time. I'm sure there are a bunch I've left out.
Niko: It's due to my ignorance, not their lack of contribution.
Niko: Well, Aaron Turon, of course. I shouldn't overlook him.
Matthias: Correct. Do you think we made the right call? Dot await?
Niko: Oh,
Niko: definitely.
Matthias: Why do you think so?
Niko: I mean, well, reason number one, every time I use a language that goes the other
Niko: way, I'm like, oh, my God, this is so annoying.
Niko: But also it's just it fits
Niko: in with everything else it's you you want
Niko: to be able to put a question mark after it you want to be able to do awaits
Niko: in a row to say hey i take this thing which gives me a future i dot await it
Niko: i do something else which gives me a future i dot await that basically the more
Niko: we can shift towards towards post fix the more smoothly the language operates.
Niko: I feel like it's kind of a thing we inherited from, I don't know,
Niko: ALGOL or whomever's, I mean, we inherited from C, but people usually say ALGOL-like for some reason.
Niko: Because I looked at ALGOL and it does not look like C. So I don't know what they're talking about.
Niko: But anyway, it's this sort of mix of prefix and postfix operators,
Niko: and they just kind of make everything a mess.
Niko: So overall, I think, yes, we definitely made the right call.
Niko: And I wouldn't mind if we moved a few other things post-fix as well.
Matthias: For example?
Niko: Oh, man. I walked right into that one
Niko: too.
Niko: Let's put it this way. I think the arguments in favor of postfix deref,
Niko: so foo.star, even postfix borrowing, foo.ampersand, I mean, no one's proposed
Niko: it, but I think it makes more sense.
Niko: But the one one thing i would really like to see is match like
Niko: being able to say something something dot match
Niko: and the pattern so that you can say call a function and then
Niko: match on the thing that it gives back to you without having to move the match
Niko: in front i mean usually that's fine but it's not always especially like we've
Niko: like with a try catch kind of scenario where you might which we don't yet have
Niko: as a language feature, but we've talked about try-blocks,
Niko: which would take the question mark and sort of intercept it.
Niko: So you could write a try-block and have a bunch of things in there which may
Niko: have results and have question marks on them.
Niko: And then that entire contents of that would have type result.
Niko: And if you do that, that's great. That's super useful.
Niko: But now maybe you want to match on that result to decide if,
Niko: hey, if the result was okay, I'm going to do one thing. And if there was an
Niko: error somewhere in that computation, I'm going to do something else.
Niko: And right now you have two choices. You can either write like match,
Niko: try, open thing, lots of code, close, curly, open curly, which is very awkward.
Niko: And it just seems weird. You've got this match that by the time you see the
Niko: arms of the match, you kind of forgot the match was even happening.
Niko: Or you can store it into a variable, which just looks dorky.
Niko: So I don't know. It feels like a clear case where it would be useful.
Matthias: I guess one main argument against the postfix notation was not necessarily about
Matthias: the postfix notation in and of itself, but the fact that await was not a method,
Matthias: but it looked a bit like a method without the parentheses.
Matthias: But then it wasn't clear if that was confusing or not. So there was some pushback there.
Matthias: Can you remember what was your standpoint on this?
Niko: I mean, people were concerned that it would look like a field or look like a method.
Niko: I think it doesn't. I think everyone knows what it means.
Niko: I think the most interesting part of that discussion, there were two interesting parts.
Niko: One was or one is a thing that
Niko: comes up a lot which is how important is it that
Niko: you have like should we be looking at the experience when
Niko: you're in an ide where you have syntax coloring and syntax
Niko: highlighting available to you or should we be looking at
Niko: the experience where you don't and it's all plain text because it's pretty different
Niko: than if the thing is bright red versus gray like that's very noticeable but
Niko: the other thing the other related thing is can we can we put ourselves in the
Niko: shoes of how we're going to feel about this feature two years from now three years from now
Niko: versus how we feel about it now and that's
Niko: often pretty different and i think a lot of times we over rotate
Niko: on how we feel about it now so a lot of times i do ask myself when we're facing
Niko: a tough question like basically just think of think of what's going to feel
Niko: the best when you're used to it and and optimize for that more than what makes
Niko: the easiest transition from what i already know.
Niko: And it's not to say I don't think we should worry about how you learn things,
Niko: but in the long run, I spend a lot more time using the language over time.
Niko: So you want it all to feel nice when it's done.
Niko: The other interesting thing about that debate was I remember there was so many
Niko: threads with so many comments.
Niko: And at some point, I opened another thread and said, okay, only comment on this
Niko: if you've actually used the feature.
Niko: And that was like crickets. There was like three people commented there.
Niko: And I thought, okay, that's kind of what I thought. It's very easy to jump in
Niko: and give an opinion, but it's so often not really informed by real experience.
Niko: And I found like just using a thing, usually within just even a minute of using
Niko: it, kind of clarifies a lot of these things.
Niko: You thought you had a whole bunch of questions and they get wrapped up very
Niko: quickly as soon as you've used it, just for a few minutes.
Matthias: Yeah.
Matthias: A lot of things are obvious in hindsight. Hindsight is always 20-20.
Matthias: I do remember that I also found it a bit weird, but now I see it as a keyword.
Matthias: I see await as a keyword. I don't even read it as anything else by now.
Matthias: But we are in between 2015 where we had Rust 1.0 and 2018 where people started to use Rust for real.
Matthias: And in the meantime, a lot of things happened because a lot
Matthias: of embedded developers and people that wanted to play around
Matthias: with async rust were still on nightly so things were not as smooth yet and it's
Matthias: very hard to make such decisions under pressure because 2018 was a bit of a
Matthias: crunch to get everything ready take us back to that time how was that making
Matthias: decisions on the pressure and keeping the team together.
Niko: So well
Niko: first thing await wasn't await wasn't technically gated
Niko: on the rust 2018 edition so we
Niko: decided that kind of independently which is good there was
Niko: enough pressure on that edition as it was i think that was
Niko: a bit of a tough time i mean i think it was the right well i'm
Niko: happy that we did the 2018 edition and i think it was a watershed
Niko: moment for rust where we really it wasn't
Niko: just the work we did on the ergonomics initiative which for
Niko: those of you who weren't paying attention at the time was an effort to like
Niko: reduce basically reduce foot
Niko: guns reduce paper cuts and in
Niko: general make rust simpler to use right and that was controversial
Niko: like the decision to do that was fairly controversial
Niko: because in a in a way it's going
Niko: against the rule that i said of focus on
Niko: what you're gonna want at the end because a
Niko: lot of the more experienced users were saying hey i've learned
Niko: that stuff already like you know we should be
Niko: doing whatever we should go deeper into this area or
Niko: do that and and we were saying no
Niko: right and we pivoted back and i think that was the right move
Niko: both because yes it was still important to learn rust
Niko: that was not good enough yet but also because actually i think even for those
Niko: users and i hope many of them would agree now they still had a lot of rust left
Niko: to write and like that paper cuts are still been adding up all this time right
Niko: like it just made rust much nicer to use overall And that was important for
Niko: experienced users more than they may have realized.
Niko: I think sometimes you don't realize how much of a toll the little ergonomic
Niko: hits take on you as your writing code. So we were working on that.
Niko: And I think the things we did were important. But the fact that we chose to
Niko: focus on that was also really important. And I'm really glad we did it.
Niko: And I feel like for me it helped. Because...
Niko: I remember coming out of 1.0. 1.0 was a really clarifying moment,
Niko: right? Because you could say, we have to reach stability.
Niko: Here's a feature list. Let's work down our list of stuff we got to do.
Niko: And all right, maybe you could punt something to make it unstable.
Niko: But we could really focus our effort and really know, what is the right next
Niko: step to take at any moment? It was fairly easy to answer.
Niko: And after 1.0, it got more complicated again. Because now all those questions
Niko: that you kicked down the road, You said, well, we don't really need to stabilize
Niko: this in order to have a sensible, usable product.
Niko: Now, all of a sudden, you look over this huge field of priorities and you say,
Niko: which ones am I going to do?
Niko: And that becomes hard again. And I think the ergonomics initiative,
Niko: I think, was our first effort to sort of actually point in a direction and try to walk there.
Niko: Saying, yes, we know there's a ton of important things to do.
Niko: We're going to pick this one because to us, it seems like the right place to focus our effort.
Niko: And we're going to go there. and then we'll come in and do some of those other
Niko: things and that i think that's something that.
Niko: Is really important to do as a leader
Niko: in a community to be able to put yourself
Niko: on the line to some extent to be brave enough to say this is
Niko: what i think we should do and stand behind it versus trying
Niko: to embrace every possible route so that
Niko: nobody gets irritated and if you go back
Niko: and look at like i have this blog post that Aaron Turon like liked or likes to
Niko: make fun of a blog post an internals post but you'll see one where i summarized
Niko: all the things that we could do after 1.0 and it was basically like a laundry
Niko: list it was like 6 000 features thing here's our next set of priorities we're
Niko: probably still working on half of those i think being able to focus on,
Niko: a small number is really important.
Matthias: And especially the ergonomics part because those small little wrinkles they
Matthias: make people trip up over time and since this is a live podcast i'm curious is
Matthias: how many of you, raise your hand, even the one sleeping,
Matthias: Know about non-lexical lifetimes. Who has heard of non-lexical lifetimes?
Matthias: Raise your hand. It's like half the room, I would say.
Matthias: I guess that's a win because the other half doesn't have to deal with that.
Matthias: That was back in the day when Santiago Pastorino worked with you on non-lexical lifetimes.
Niko: That was a really good lesson for me. For your listeners who don't know,
Niko: non-lexical lifetimes was a change in how we did borrowing in Rust that made
Niko: it not have to be tied to the lexical structure of your program.
Niko: So it used to be that many times we would consider things borrowed for way too
Niko: long because we were very simplistic about, like, you couldn't borrow for just the if.
Niko: If you had an if and you used the pointer in the then part of the if but not
Niko: the else, it would sort of expand to the entire if-then-else.
Niko: It couldn't be just one half.
Niko: And to be honest, that was by design because I thought, oh, it'll be easier
Niko: to learn. Like, yes, you'll have more errors, but it'll be easier to learn if
Niko: the formalism is simpler.
Niko: You know, because I was a PhD student coming into language design or whatever,
Niko: like I was looking at the AST structure and thinking this is good.
Niko: I wasn't thinking about it from the point of view of a programmer using Rust
Niko: who has not authored compilers or studied type theory and so on.
Niko: And so it was kind of an aha moment for me to realize that what seems simpler
Niko: to the compiler author doesn't necessarily feel simpler to the language user. And...
Niko: That's shifted my thinking in several cases where I'm really asking myself,
Niko: all right, I think there's a balancing act.
Niko: You want them ideally both to be simple. I think you can go too far on either side.
Niko: If you just say it doesn't matter at all how complex the compiler is,
Niko: then it can be really hard.
Niko: That ends up leaking into complexity for the user in the end because they really
Niko: can't wrap their head around the algorithm.
Niko: But on the other side, you really have to think about how are they going to
Niko: experience it and feel it and try to put yourself in those shoes.
Matthias: And that was also right around the time when this first notion of epochs was
Matthias: introduced, which are now called editions.
Matthias: This promise of robustness and the guarantee to the industry to say we won't
Matthias: break things so you can depend on it.
Matthias: It was, I think, another very striking moment in Rost's history.
Matthias: Can you talk a little bit about that and what was the reasoning behind this
Matthias: and how Rust is tested as well.
Niko: Well, the reason behind the stability promise, which was the 2015 Rust 1.0 release, was very simple.
Niko: We had been changing Rust on a, as you say, weekly, daily basis.
Niko: And we knew that we'd been working on it for a long time.
Niko: To be frank, we knew we had to show some results if we were going to continue funding this project.
Niko: And we also knew, we had a sense that, like,
Niko: the world was being patient with us there was a lot of enthusiasm
Niko: about rust people liked hearing about it rust news
Niko: would go to the top of hacker news but how long could
Niko: that stay we had to start seeing some production results and
Niko: really getting out there and seeing it be in use before it becomes kind of vaporware
Niko: so it seemed like the biggest blocking thing was stability and so that's what
Niko: we focused on but yeah i think around 2017 i don't remember exactly when there
Niko: were definitely some decisions where we started looking and saying, you know, in hindsight.
Niko: That may not have been the best call, right? And I think there are lots of,
Niko: particularly the module system was one where syntactically, the older module
Niko: system was really tripping a lot of people up.
Niko: And it was like, well, we made a promise for stability, but we know that the
Niko: future is way bigger than the past.
Niko: And these changes are sort of, yes, everybody has to make changes,
Niko: but they're so, so programmatic.
Niko: Like the compiler could literally change it for you. It seems like a real shame
Niko: to get stuck in this suboptimal design that we see causing problems just because
Niko: we said we don't want to break people's code,
Niko: oh, maybe there's a way we could do neither, right?
Niko: We could make the change and not break people's code.
Niko: And in fact, I did not do this on purpose, but now that you mention it,
Niko: the inspiration for additions came from XSLT. I forgot that.
Niko: Because XSLT, that XML transformation language, if you go look at it,
Niko: the very first thing you do in every XSLT program is you write version equals one.
Niko: And they have an XSLT 2.0, which can run XSLT 1.0 programs.
Niko: And it knows that they're not the same because it sees the version equals one.
Niko: It doesn't quite work like our additions. If I recall, it runs them almost the
Niko: same, but not quite the same as they used to run.
Niko: It has like a compatibility semantics.
Niko: But still, I remember thinking that that was a really clever idea to just a
Niko: little bit of foresight to just declare the version number in your program.
Matthias: Thanks, XSLT, the contribution to this language.
Matthias: Yeah, I think we could talk about the history of the Rust compiler for a very long time,
Matthias: but I also wanted to change gears and maybe talk about some of the things that
Matthias: changed during that time where people started to use Rust for things that were
Matthias: not necessarily systems programming tasks.
Matthias: So suddenly people started to use Rust for backends and other endeavors.
Matthias: Was that a surprise to you?
Niko: I'll tell you it was the biggest surprise. The biggest surprise to me was realizing
Niko: was not so much the tasks that people were using Rust for, it was who was using Rust.
Niko: Because I thought that Rust was targeting C++ users.
Niko: But actually, I think that's more true now.
Niko: But I think at the time, where the biggest initial adoption came from was people
Niko: who would have liked to use C++, but for the lack of memory safety right folks
Niko: who were using ruby were using java were using python,
Niko: because they needed the productivity that it had to offer but they
Niko: had these hot spots that were a pain in the ass to optimize and they
Niko: couldn't they either couldn't get the performance they want or
Niko: they had to work really hard to get the performance they want and this
Niko: gave them a third option and so that that's that i think
Niko: that was a great thing for rust also because they were coming from
Niko: modern ecosystems and they were less forgiving like i
Niko: remember i was like we don't need cargo make is fine that's
Niko: true confessions because i don't know i just mostly wrote c and c++ code and
Niko: make seemed okay kind of did the job but luckily there were people there who
Niko: had used npm and they were like no Niko you are you're an old you're showing
Niko: your age we need cargo and they were right and.
Niko: So so i think it's it brought us like a whole audience of
Niko: people who had very different who experienced modern tooling
Niko: and were not willing to accept what was
Niko: cool in the 70s so that was
Niko: good in terms of systems programming
Niko: i mean back-end systems i guess it's kind of debatable if it's a systems
Niko: program or not but i think i don't
Niko: remember thinking actively about it one way or the other to be honest because
Niko: i'm a compiler person and networking is like a little over my head but but it
Niko: doesn't surprise me now but what does surprise me is actually how good rust
Niko: is at like scripting like i've had python scripts that i port to rust and it's like oh it's,
Niko: either 20 longer or not even longer at all it actually
Niko: feels better in some ways and worse and i don't know it's it's competitive at
Niko: least if you know rust well and you know what how to use it right i think if
Niko: you came in fresh you would not think that but so the fact that rust is versatile
Niko: enough to be optimized for those foundational low-level applications and yet
Niko: be usable all the way up is pretty important.
Niko: And a little surprising. I think a cool outcome.
Matthias: And then during that time, let's say...
Matthias: Fundamentals of Rust also changed because Mozilla was struggling a bit with
Matthias: providing funding for Rust on a broader scale, just like many other companies.
Matthias: And then eventually some of the core contributors moved to other ventures.
Matthias: Some of them moved to big tech.
Matthias: How did that change the language?
Matthias: And that was also around the time when the Rust Foundation was founded.
Niko: Yeah you know i think that mozilla gave rust
Niko: a parting gift actually in deciding
Niko: not to fund it anymore and i know
Niko: that sounds maybe a little odd but like the truth
Niko: was they had it didn't make that
Niko: much sense for them and they had been slowly you know
Niko: reckoning with their own financial difficulties for some time and like the team
Niko: had been getting smaller but it was like they didn't want to i think i guess
Niko: it's an interesting lesson right we're not admitting when the time has come
Niko: can sometimes be worse than just admitting it and cutting off and i think the
Niko: fact that they said we're not going to be able to fund this anymore was both
Niko: obvious to everyone but also.
Niko: Well not to everyone it was obvious to everyone who was close in but it
Niko: wasn't obvious to all the companies that had been building on rust
Niko: including my current employer at amazon and so
Niko: it was a it was helpful for them to get a real like get
Niko: a reality check of oh hey we're building all of this
Niko: kind of foundational infrastructure in this
Niko: programming language who's going to support it well i
Niko: guess we have to and so i think that it it gave rise to
Niko: a lot of rust teams across
Niko: the industry that might never have come to be
Niko: and that's meant a lot more people working on rust and a lot more
Niko: investment in rust right and of course the most like
Niko: visible sign of that was kicking off the rust
Niko: foundation but i think in some
Niko: ways i mean that's been really impactful but also the hiring of developers and
Niko: the investment in the project side and the open source building maintenance
Niko: side has been very important too
Matthias: did anything changed when you started working
Matthias: at aws or did you do more or less the same as you did?
Niko: No it's been the same.
Niko: I remember like i started at amazon and
Niko: there's like this two week long like set of videos you're supposed to watch
Niko: or whatever for onboarding and i was like i i can't watch these guys i have
Niko: like four bus meetings like tomorrow i i don't know i already have a job so
Niko: so yeah i pretty much have I just have had no real difference between working
Niko: at Mozilla and Amazon, except for one thing,
Niko: which I think is good, which is that Amazon's a lot bigger, doing a lot more things.
Niko: And so I have a lot more contact with teams using Rust at Amazon than I did
Niko: at Mozilla, if for no other reason than there are many more of them.
Niko: But also in many different domains. So we have people using it in consumer devices,
Niko: people using it in embedded, people using it in web services.
Niko: And that lets me get a lot more hands-on knowledge about how those teams,
Niko: like what their problems are.
Niko: And there's something about being at the same company that gives you, like I can go deeper.
Niko: I can get more and I can push them a little harder.
Niko: I can be like, just stop being whiny.
Niko: You can deal with that problem. That's not a real one. I'm like,
Niko: let's go to the next thing on your wish list and talk that one through because
Niko: I think you can handle that.
Niko: And so I've found it personally that it's helped me get more perspective.
Niko: But it hasn't really impacted my ability to work in the open in any way.
Matthias: Are you involved into any architecture discussions at AWS regarding Rust or
Matthias: is it completely running by itself?
Niko: Well, I don't know what you mean by architecture. Do you mean like systems using Rust?
Matthias: Do people book meetings with you to talk through systems design problems?
Niko: No, I wouldn't take that meeting anyway. No, they don't.
Niko: Maybe a little bit, but not really.
Niko: And the truth is, I'm not a domain expert in what they're doing.
Niko: So it wouldn't really make sense to get my opinion on how to build a distributed system.
Niko: I would rather go listen and learn from them.
Niko: They will occasionally come to me with like here we have a it's usually around
Niko: FFI or interop or some very narrow problem where we're seeing some overhead
Niko: do you have any thoughts on how we might,
Niko: optimize this case and usually I'll say yeah go talk to the much more qualified
Niko: people on my team who can help you with that but.
Matthias: A sort of naive take on this would be what's in it for Amazon to hire you.
Niko: Well I think that's a
Niko: good question actually it's one i should have a good answer to because uh
Niko: performance assessment time i think i
Niko: don't i don't you know i don't provide that sort of leverage right what
Niko: i do do is well to a certain extent help shape overall language usage strategy
Niko: but more importantly find the places where people are getting stuck and figure
Niko: out the best place to fix them right and that could be internal to amazon sometimes
Niko: it's like we're missing a library but a lot of times it's like no,
Niko: that's an indication of a problem in the language and drive that fix.
Niko: That's the sort of short-term benefit. But I think the long-term,
Niko: what Amazon really is interested in is knowing that.
Niko: Investing heavily in a language which they know will continue to succeed in the industry.
Niko: No one wants to write a bunch of code and then find out that there's nobody
Niko: to hire who knows the language or there's no ecosystem around it so nothing's being developed.
Niko: You want to be kind of sharing in, this isn't a competitive advantage, so to speak, to use Rust.
Niko: That's not the way I want to say it. It is a competitive advantage,
Niko: in my opinion, to use Rust, but it's not a thing that you want to keep to yourself.
Niko: It's not the thing you're selling to users and customers, right?
Niko: And so really what you're mostly interested in is having it be as good as possible.
Niko: And that means benefiting from all the rest of the work that the rest of the
Niko: industry is doing while contributing in and helping do your share, right?
Niko: And I think these both are really important. And it's kind of the blend.
Niko: Like I wouldn't want, I think if everyone who worked on Rust came from Amazon,
Niko: even if we had the same number of people, you wouldn't get the same quality
Niko: work out. because we would be overly rotating on the constraints and the concerns of Amazon builders.
Niko: And so as happy as I am to go talk to those teams and get that detailed look,
Niko: I think that on its own, it's not enough.
Niko: What's really great is that I can do that and then hear from like Tyler Mandry
Niko: and who works at Google and hear from Josh Triplett and all these other people in the Lang team,
Niko: but also around the community who have insights, similar insights,
Niko: but to different groups.
Niko: And often they have pretty different problems even when doing a similar thing.
Niko: And so when you cut them together, you wind up with a really good overall solution.
Niko: So that's kind of, I think Amazon recognizes that.
Niko: And I certainly like to tell them that as well whenever I get a chance,
Niko: I advocate for this point of view, that the goal is to pick foundational technologies
Niko: that you can really invest in and really get involved in and be a sort of well-respected participant.
Niko: That's the goal.
Matthias: And yet AWS builds a lot of really critical pieces of infrastructure with Rust nowadays.
Matthias: I'm not sure if you can talk about a project that you particularly liked or
Matthias: maybe something that shows how Rust adoption can work successfully.
Niko: Yeah. Most people I talk to are not aware of how much Rust is in production at Amazon.
Niko: Like every S3 request, every EC2 VPC, you can tell my ignorance of even our own product line,
Niko: every Lambda invocation these things
Niko: all go through Rust they also go through other languages I'm not saying that
Niko: they're exclusively written in Rust by any means but more and more the hot path
Niko: of a lot of these big services that you know is being rewritten in Rust or re-architected
Niko: in Rust and there are some services that are 100% so the new,
Niko: S3 Express One Zone which is like a really low latency add-on to S3 that was
Niko: added for various super low latency use cases, that's 100% Rust, for example.
Niko: And the new serverless SQL is also 100% Rust, which came out at reInvent and
Niko: is going to be hopefully GA very soon.
Niko: So there are things coming up that are newer services or newer side add-ons
Niko: to existing services that are 100% Rust, but also we're seeing a lot of Rust just here and there,
Niko: scattered in and out the data flow, the hot path usually of other things.
Matthias: Were these services always established to be in Rust from the get-go or were
Matthias: they written in different languages and then migrated to Rust?
Niko: Do you mean the existing ones?
Matthias: Like, for example, the SQL, the serverless SQL part.
Niko: Oh, this one. Yeah, that's an interesting one. That one was not originally,
Niko: that's been a long time in the making.
Niko: And it started out, I think, in Kotlin.
Niko: And they found that for various reasons they weren't able to get the latency
Niko: that they needed in certain parts of the system.
Niko: And so they migrated over just those parts into Rust initially as a kind of experiment.
Niko: And when that went really well, what they realized is, oh,
Niko: actually when we take these Rust parts and we combine them with the stuff that
Niko: we wrote in Kotlin, we're having a lot of duplicated logic and we're getting
Niko: a lot of integration failures just because we haven't been able to test them together.
Niko: Like the Rust stuff was tested with Turmoil, which is that, this is a really
Niko: cool Tokyo library. Some of you may have seen that.
Niko: They can't use that library on the testing they were doing in the other service.
Niko: In their Kotlin, they had a different thing that did different patterns.
Niko: And so these kind of integrations were causing more failures.
Niko: And in the meantime, they realized, actually, they originally thought using
Niko: Rust was going to be kind of slower but worth it for the results.
Niko: But what they found is, no, once you get past the learning curve,
Niko: it's really quite productive.
Niko: It's just the initial learning curve bump is the biggest problem.
Niko: And so they ended up rewriting the whole thing in Rust so that it all works together smoothly.
Niko: And I thought that was a really cool and interesting story in a number of ways.
Niko: I'm happy with the outcome. I wish that Rust integrated better.
Niko: I think it would be really cool if you could go back and forth between Kotlin and Rust more smoothly.
Niko: But also the biggest takeaway I had is this point that I made earlier.
Niko: It's really important for Rust. If Rust is to succeed, it has to be able to
Niko: scale, not just with the hardest problems, but with the simpler and all the
Niko: way up and be productive also in the easy problems.
Matthias: And I think this brings us sort of to the end.
Matthias: I wanted to get back to this notion of systems programming, maybe not capturing what Rust is today.
Matthias: And I guess you proposed a different term for it, which is foundational software.
Matthias: What is foundational software to you and why is Rust a great fit for this?
Niko: Yeah, so to me, foundational software, it's the software that everybody else builds on.
Niko: And that could be many different things, right? I think sometimes that's like the kernel.
Niko: That's very literally the thing that everyone else runs on. But other times
Niko: it's a CLI tool that's super widely used and it's part of the scripting setup you use.
Niko: Or I don't know, your WordPress blogging platform could be seen in a certain
Niko: way as foundational software. And I think.
Niko: That's what I think Rust is optimized for, right?
Niko: Is this kind of software that is making the world run, in some sense, our digital world.
Niko: And that kind of software has particular characteristics. It needs to be really
Niko: reliable because if it breaks, everybody else's stuff breaks.
Niko: It needs to be fast because everyone's building on it. And so whatever performance
Niko: hits you take there, everybody has to work with.
Niko: And it has this long-term maintenance story.
Niko: So you sort of stitch in time saves nine. is this English expression that often
Niko: comes to my mind where it's like you would rather take the time to make sure
Niko: that your match statement covers every case even if some of them just say panic,
Niko: than not because you know you really don't want to miss a case.
Niko: You're going to be maintaining this over the long term or you know that when
Niko: new match when new variants are added to your enum if you have an exhaustive
Niko: match you'll be able to find where they might need to be handled and you will
Niko: be more likely to have a reliable outcome.
Niko: So in those cases where you really would prefer to take the care,
Niko: to check for the exception, to handle those things.
Niko: That's what Rust is optimized for, and I think that's what shows up in foundational software.
Niko: So I don't mean that all foundational software should be written in Rust.
Niko: Like probably WordPress. Well, I don't know. I've never read WordPress's source
Niko: code, but probably would be great in Rust.
Niko: But I guess also it's working pretty okay in whatever it's PHP or whatever they wrote it in now.
Niko: But I think that it probably all should be writable in Rust.
Niko: And it probably would be a nice experience. That should be our goal.
Niko: And also that similarly, I don't mean the rush should only be used for that
Niko: kind of software, but that is what we optimize for. That's our sweet spot.
Niko: Everything else should be good. We don't have to be, it may not be the best.
Matthias: It's a very profound insight to say the software that a business depends on,
Matthias: that is the foundational part.
Matthias: And then you can add, let's say, TypeScript or Kotlin or Python code on top
Matthias: of it. but seeing it as the runtime of the business, the core of the business.
Matthias: That's also one of the first users of Rust in production did. Dropbox.
Niko: That's true. That's a good point. Yeah. Our very first production.
Niko: Now, the very first production customer, I think, was Tilda,
Niko: which is a smaller company. But Dropbox was like our first name band.
Niko: Something I can say to my parents and they know what I'm talking about. That was cool.
Matthias: So what's next? what is mission of enabling foundational software in Rust for the next three years?
Niko: Well, alright. What I'm,
Niko: I have a lot of answers to that question, but the one that I'm most focused
Niko: on right now, especially here at this conference, is laying out this vision doc project.
Niko: And the idea is to take a look at where we are and say, hey, we're 10 years old.
Niko: What's going to happen in our teenage years? And in particular,
Niko: I think with all the growth of different companies investing in Rust,
Niko: the number of Rust maintainers has gotten up.
Niko: I don't know what the number is off the top of my head, but I'm sure it's in
Niko: the hundreds, if not, well, probably not thousands, so hundreds somewhere in there.
Niko: That's a lot of people to coordinate, right? That's a lot of work going on constantly.
Niko: And I feel like we, I would like us to all be, if not on the same page,
Niko: at least all sharing the same understanding of what Rust's users are experiencing,
Niko: which things, where Rust is working really well and where it could use improvement and so on.
Niko: And then using that to make decisions about what the next big initiative should be.
Niko: So really in some sort of meta way, what I see as our top priority is building
Niko: that shared understanding of what
Niko: problems we want to solve more than the individual problems themselves.
Matthias: And as a final question, what's your statement to the Rust community?
Niko: Yeah, be excellent to each other. I think Bill and Ted put it best and I can't do better than that.
Matthias: Niko, thanks so much for taking the time and thanks for all your work on the Rust compiler.
Niko: Thank you for having me.
Matthias: This was a very special episode recorded at Rust Week in Utrecht.
Matthias: Thanks to Eric Jonkers, Hugo van de Pol and the other organizers and volunteers
Matthias: at Rust NL for providing the space and the opportunity to record this live episode.
Matthias: We had a blast at the event and we hope to be back next year for a full week
Matthias: of talks and events around Rust.
Matthias: If you want to learn more, head over to rustweek.org.
Matthias: Rust in Production is a podcast by corrode. It's hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.