Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we're talking to Steve Klabnik
Matthias: from Oxide Computer, about working on the hardware-software interface with Rust.
Matthias: Steve, I don't think you need a lot of introduction. A lot of Rust sessions
Matthias: know you, but maybe there's someone out there who doesn't yet.
Matthias: Can you quickly introduce yourself and oxide the company you work for.
Steve: Absolutely hi everybody i'm Steve Klabnik
Steve: i am most known in the rust community for having co-authored the
Steve: rust programming language which is the book that most people learn the language
Steve: from or at least many people learn the language from i was on the rust core
Steve: team for almost 10 years and you know i've been doing a bunch of rust stuff
Steve: for a very long time i started using rust in december 2012 so i'm one of the
Steve: few people that can say i have 10 years of rust experience so.
Matthias: You can finally apply to these job adverts which.
Steve: Yeah exactly oh yeah and oxide so oxide
Steve: computer company is the startup that i work for we're basically building servers
Steve: that you can buy and then it's like a very old-fashioned business you give us
Steve: money we give you a server you install it your data center you run your jobs
Steve: on it but the key is like you get a cloud-like deployment experience while it's
Steve: actually on-prem So you're not renting a cloud, you're buying a cloud.
Steve: And it's partially named Oxide because we use Rust for 99% of the code that
Steve: gets written at the company.
Matthias: And that's what we're here for. And when you explain it like that,
Matthias: it sounds so simple. But in reality, it's very complex to build servers, I guess.
Matthias: We get to that in a second. But I just want to thank you personally,
Matthias: because you introduced me to Rust many, many years ago when you gave a talk at Fostam.
Matthias: And I was there and we happened to have a very, very brief chat after the talk.
Matthias: I don't even know if you remember, but that was kind of the starting point for my Ross journey.
Matthias: And I think a lot of people got inspired by what he did. So thanks for that.
Steve: That's awesome. That talk is really meaningful to me. And I was actually very
Steve: sad because I only intended to give that talk one time at FOSDEM.
Steve: So, you know, as you remember, but like other people probably don't,
Steve: I kind of was chronicling like how Rust changed up before 1.0 because that was
Steve: around the time of the Rust 1.0 release.
Steve: And so I thought it'd be cool at FOSDEM to kind of like recap all of the development
Steve: history. But then they ended up losing the recording.
Steve: Something happened with the recording and it didn't work. So luckily
Steve: the ACM had me give it one more time and so we got a
Steve: recording then and so that was like a future thing but that
Steve: was also a very special conference for me because that was when larry wall announced
Steve: that pearl 6 was being like released finally or whatever and so i got to like
Steve: speak after him and i i did ruby before rust but i did pearl before ruby and
Steve: so that's always had a small soft spot in my heart for pearl stuff so that was
Steve: also kind of fun about that conference specifically.
Matthias: For the people who haven't been there it was a really magical
Matthias: moment the audience was cheering you on
Matthias: they you could feel the atmosphere you could feel the energy
Matthias: in the room and it was a very special moment so it's sad
Matthias: to hear that they lost the tape but in any case you
Matthias: managed to inspire a lot of people also because you have
Matthias: a way with words and you always start
Matthias: from first principles and i think
Matthias: that's unique because many engineers they
Matthias: either are very technical or they are
Matthias: you know very very high level but i think you can switch between
Matthias: those two two levels of abstraction you can talk
Matthias: to everyone and and i wanted to capture some of
Matthias: that today now thanks on that premise
Matthias: you had a blog post on
Matthias: your website once where you said that you were not planning on quitting cloud
Matthias: where you worked before but you just waited for a good opportunity and it seemed
Matthias: to have presented itself in oxide and i do wonder what specifically about oxide
Matthias: and their mission attracted you to make this move yeah.
Steve: So it's kind of a combination of two things the first
Steve: is i was born in 1986 i started programming when i was like seven years old
Steve: so like 93 ish roughly about and i also like a Mac was the first computer I
Steve: ever used and I was a really big like little Mac fanboy back in the day and so there's like a ton of.
Steve: No, I guess maybe I'm getting old. So I don't know if this is still necessarily true.
Steve: But like, at least in the early 2000s, late 90s, there was a lot of like,
Steve: I want to say like sort of reverence held for the sort of like 80s in computers,
Steve: like that's when computing stuff really took off.
Steve: I mean, obviously, there were computers, you know, many years before,
Steve: but like the 80s is when the sort of PC revolution started happening.
Steve: And when people started having computers in their home, instead of just having
Steve: them in universities and stuff like that.
Steve: And so i was really really big on learning kind of about
Steve: apple's history and a lot
Steve: of the stories that are on say like folklore.org where you
Steve: can like read a lot of early apple history were stories that i kind of grew
Steve: up with and were really like meaningful and impactful to me and there was like
Steve: one in particular i mean there's a lot of them i could recount but like a one
Steve: that specifically kind of draws to oxide in in specifics is that like there's
Steve: the story about how, as they were putting the first Mac together,
Steve: Steve Jobs got the entire team to sign the inside of the case of the original
Steve: Mac. And they ended up putting that in all the computers.
Steve: And when they asked why that's the case, he was like, real artists sign their work.
Steve: You get a painting and there's always a signature in the bottom corner.
Steve: And that's what we're doing here.
Steve: There was also a book that I used to read every year, and I haven't read in
Steve: a couple years now, but it's called The Soul of a New Machine by Tracy Kidder.
Steve: And I think it really sort of captures a lot of that.
Steve: It was about a company called Data General in the 80s and their journey to go
Steve: from like the 16 bit microcomputer to the 32 bit mini computer and them building
Steve: this computer and how like much effort the team had to put in and their struggles
Steve: and trials and tribulations while they're making all this work.
Steve: And so I've kind of like always sort of been like I wish I was 10 years older
Steve: so I could have like experienced that era of the industry you know because like
Steve: by the time I was graduating you know high school it was 2000 and by the time I was supposed to be well.
Steve: 2004 i started always high school in 2000 but like
Steve: i was graduating high school in 2004 and i was graduating college
Steve: in 2008 2009 and so by then
Steve: it was a good 30 years after all that stuff had happened and so people weren't
Steve: like starting computer companies anymore and so
Steve: i'd always kind of thought that i wouldn't really get to like
Steve: experience that sort of thing and so
Steve: then lo and behold oxide starts up
Steve: and it's a new computer company and so
Steve: that was really cool they also talked about the soul
Steve: of a new machine a lot and i was like oh like
Steve: that's you know there's a very big like cultural share
Steve: you know similarity here and they were using rust
Steve: which is a tool i loved and had worked on and so i was like oh that makes sense
Steve: and i had known two of the three co-founders Bryane and Jessie before oxide and
Steve: i had like respected their technical opinions and like you know knew them and
Steve: so that also seemed good and so just like all those things kind of aligning
Steve: at the same time. I was employee 17.
Steve: So you know, I didn't start the company, but I kind of got in on relatively on the ground floor.
Steve: And, you know, it was all of that kind of stuff that really sort of appealed
Steve: to me personally, it was, it's very, very rare that you get the opportunity
Steve: to like, get it on the ground of building a new kind of computer.
Steve: And so that was just like, too good to pass up.
Matthias: It sounds like a very lofty dream. But at the same time, it feels like building
Matthias: a new server is such a hard problem so what does it feel like on a day-to-day to work on this.
Steve: Yeah the thing is is it's not just a
Steve: hard problem it's like a thousand hard problems like we keep joking that like
Steve: oxide is actually a thousand startups in a trench coat like there are so many
Steve: things that we are doing that like could conceivably be their own company or
Steve: like if we were trying to do something for more than just us could be their own company.
Steve: So like, for example, you know, we want to give people that own the rack,
Steve: like very good way of monitoring how the system is doing. And so we need observability and metrics.
Steve: And because we're on prem, like that doesn't mean, you know,
Steve: you're not just going to hook up some sort of cloud metric service and ship
Steve: information off to them, right?
Steve: So we have like, are currently working on like internal metrics for the rack.
Steve: And that's like a thing that whole companies do separately. And we kind of have to do on our own.
Steve: You know, like there's just tons and tons of stuff that we're doing.
Steve: So, so yeah, so it's not just one problem, it's a ton of problems.
Steve: And so that also means, you know, it is a group effort.
Steve: I think a really big difference between the eighties and now is like part of
Steve: the reason why it was so easy to start a computer company in the eighties and
Steve: why so many people did is because it was feasible for one person to design and
Steve: code like the whole thing from top to bottom,
Steve: all the hardware and all the software could be meaningfully understood by a single person.
Steve: And we are far, far beyond that level complexity now.
Steve: So, you know, I said I was employee 17, because it's now around 60, 70 people.
Steve: And, you know, all of them are super necessary to get this done.
Steve: And so, you know, it's, it's very much, in some ways, it's very much like working
Steve: in a normal company in the sense of like, you know, you have the thing you're
Steve: working on, and other people have the thing they're working on.
Steve: And obviously, there's like a lot of cross pollination.
Steve: But you know, it's also like tons of people are working on really cool stuff.
Steve: You know, I'm not a hardware guy, personally, I'm a software person, mostly
Steve: and so it's been really neat to like learn about
Steve: how the hardware folks do their job and you know vice versa like
Steve: you know they'll ask some software people for help with software stuff
Steve: and so like there's a lot of cool collaboration that
Steve: occurs since the product is so broad but you know at the end of the day you're
Steve: still like working on you know your little part of the thing and oftentimes
Steve: many little parts of the thing like another thing with a startup life is that
Steve: you kind of have to wear many hats and so you know people move around and work
Steve: on a variety of different things too but yeah i don't know that's definitely a big part of it.
Matthias: A naive person might say shouldn't it be easier today to build such a computer
Matthias: company than just thinking about the 80s because now we have standardization
Matthias: and we have standardized hardware components and all of that stuff but the complexity of course,
Matthias: is still very much existent. And I guess the rabbit hole runs very deep.
Matthias: What are some of the other hard problems that you have to solve to build such a computer?
Steve: Yeah, so I mean, I think the biggest, most straightforward one is like,
Steve: it is fundamentally a distributed system.
Steve: So like, to be clear, like when you buy an oxide rack and you plug it in and
Steve: turn it on, you're not like managing individual servers yourself.
Steve: You get an API that gives you the ability to, say, spin up a virtual machine,
Steve: but it's not like you're like, OK, I have 10 VMs running on this sled and I
Steve: have 15 VMs running on this sled.
Steve: You're just presented as one whole stack of compute.
Steve: And what that means is that our control plane software is the one that's multiplexing
Steve: those VMs across all the hardware in the rack.
Steve: And so that means at a high level, it's a distributed system in a box.
Steve: And distributed systems are never easy in the first place.
Steve: You know and then you're talking about you know virtual machines that
Steve: have to manage hardware and do all that kind of stuff and so you know
Steve: i think that's like another example of a really hard problem to sort
Steve: of get to the like people think it's easy because the stuff is standardized
Steve: that's sort of true but the standardization is also
Steve: in some ways what we're like rejecting in many
Steve: ways and so like we're not using a
Steve: lot of those standard interfaces and we're writing our own
Steve: firmware instead and so like that is also
Steve: a really big problem and problem i mean problem to
Steve: be solved like so for example Bryan gave
Steve: a talk about called like i've i've come
Steve: to bury the bios i'm forgetting a couple words in that title but
Steve: it's like i haven't come to save the bios but to bury it or something like
Steve: that and so you know we've like thrown out the concept of a bios and the the
Steve: operating system boots up the hardware just like in the old days in order to
Steve: do that that means we had to write our own firmware and that's a really really
Steve: hard problem and so you know there's a lot of like stuff like that so yeah it's.
Matthias: A bit like is it a bit like with rust where
Matthias: some of the old knowledge was rediscovered with rust and suddenly you were able
Matthias: to use those concepts in the real production systems level language is it similar
Matthias: with oxide where you uncover some of the old truths that you know,
Matthias: OGs, the original creators of computing knew about and then we forgot about them over time?
Steve: I think that's true in a certain sense but also not entirely because like a
Steve: lot of these things aren't,
Steve: necessarily truths that are lost but it's more
Steve: about the way that the computer industry
Steve: evolved over time so like standardized interfaces
Steve: and parts being swappable is like what made the
Steve: pc platform succeed right like the whole
Steve: the standardization is what allows a giant
Steve: ecosystem of companies to be able to work together productively and
Steve: you know be able to like ship things and so that was like you
Steve: know again being like a mac fanboy in the 90s people be
Steve: like oh yeah i need to buy a new hard drive i have 15 different options
Steve: for buying new hard drives but you have one you
Steve: know from apple and so you take what you can get with them and
Steve: that's it you know oh you want to update your graphics card like
Steve: cool you know that's not really as viable as it is in
Steve: sort of the pc platform and so i and i
Steve: think that like that made sense for the economics of
Steve: the time and i mean it still makes sense in many contexts
Steve: it's just it doesn't necessarily make sense in servers anymore you know i mean
Steve: like my pc is still a very standard pc built from you know it's running all
Steve: that firmware that those manufacturers have built pulled together from 20 different
Steve: companies that are you know creating that together but i think there's sort of like.
Steve: The wisdom is not so much like lost as so much as it is that like as computers
Steve: at home like the industry changed a lot whenever it went from like a computer
Steve: is a thing a university has that you get to timeshare on sometimes,
Steve: to a computer is something that companies have, or maybe they have a couple
Steve: of them, to a computer is something that every individual person has. And so...
Steve: You know, it's the same sort of thing where with especially like the web and
Steve: the internet being such a big thing.
Steve: Now it's not like my company has a single server in a rack of servers in a data center.
Steve: It's now my company has a whole rack. And then it became my company has a data center.
Steve: And then it became my company rents some servers from, you know,
Steve: another company that runs the data center. And so those kind of like economic
Steve: changes happened over time and made different configurations of these things work.
Steve: Now, part of the additional kind of evolution that happened there is if you'd
Steve: read a lot of stuff about Google as they were sort of like growing back in the early days,
Steve: there was a lot of talk about how they use like commodity servers instead of buying big iron.
Steve: So instead of buying big old giant mainframes or servers, they were just using
Steve: regular PCs and hooking up a lot of them together.
Steve: You know, back in the Slashdot days, we'd made jokes about Beowulf clusters,
Steve: but like that's kind of a thing that they supposedly, well, they didn't supposedly, they did do that.
Steve: In fact, for a while they ran on commodity hardware, but they kind of learned
Steve: that like the style and architecture of a computer that works well for people
Steve: at home is not really well suited for running when you need like hundreds of
Steve: thousands of computers,
Steve: because it's sort of designed for the,
Steve: you know, running on your desk, like my tower is like right here outside of the frame.
Steve: But like, you know, the thing that's running on my desk has very different needs
Steve: than when I have an entire data center full of computers. And so they started
Steve: designing their own hardware.
Steve: And so did all the other, we call them the hyperscalers.
Steve: So like the AWSs of the world and all the other people that are running big clouds are.
Steve: They sort of moved away from the sort of PC model because that just like doesn't
Steve: really make sense economically speaking for a number of different reasons.
Steve: And so, but like Amazon is not in the business of selling computers.
Steve: They're in the business of renting them. And so you can't like say you're a
Steve: Fortune 100 company because here in the States, companies are people.
Steve: No, I kid. But like, you know, imagine you're a company, right?
Steve: And you need some servers.
Steve: So you go to try to buy some. you like can't
Steve: go to amazon and be like hey i need
Steve: one rack of the same servers that you've built custom built
Steve: to live in an aws data center right like that's
Steve: just not an option that's available to you uh these big clouds
Steve: aren't in this selling computer business they're in the renting computer business
Steve: and so you can rent them but you can't buy them and so there are other companies
Steve: that are selling servers but they're not really in the custom hardware business
Steve: they're still fundamentally selling you a big pc and so kind of what oxide is
Steve: doing is i sort of liken it to.
Steve: Prometheus you know the legend of prometheus who you
Steve: know went and stole the fire from the gods and brought it back down to humans we'll
Steve: ignore the part where prometheus was then tortured for all of eternity for
Steve: you know doing what he did but like we're sort of like
Steve: taking the concept we're not
Steve: literally selling the same designs as amazon's like we're like sneaking
Steve: into the data centers and you know like whatever but like we are
Steve: we are selling those style of servers to
Steve: companies that you know aren't necessarily going to be building their own
Steve: there's tons of you know big companies and smaller
Steve: ones necessarily but when you're talking about a whole rack of servers you
Steve: kind of inherently are going after larger organizations so you
Steve: know big companies that aren't going to spin up their own server design
Steve: division even though they may have the money they just don't have the expertise
Steve: of the culture to be able to do that right and they want to be able to buy computers
Steve: instead and so that's kind of like what we're doing is taking these sort of
Steve: hardware and software concepts from the hyperscalers and then selling them to
Steve: companies that could use that amount of compute but just literally can't buy
Steve: the same amount of computer anywhere else.
Matthias: I think this analogy with apple is really fitting for two reasons first they
Matthias: democratized computing for a lot more people but second they also focused on
Matthias: vertical integration In fact,
Matthias: when Steve Jobs moved away from Apple to build Next,
Matthias: he also built some sort of workstation,
Matthias: which was a different type of computer with different requirements.
Matthias: And he chose Objective-C for various reasons to build it, I guess.
Matthias: But of course, Oxide did not choose Objective-C, he chose Rust.
Matthias: What are some advantages that Rust offers for Oxide's unique hardware software
Matthias: integration challenges? Yeah.
Steve: Totally. So I should also really briefly say that, like, I always make the Apple
Steve: analogy, because it makes sense to me.
Steve: But like, Bryan literally worked at Sun forever. And a lot of the folks that
Steve: are also from Sun, and so Sun and Apple are kind of very similar and very different in different ways.
Steve: And so, you know, they might also pitch it as kind of like a,
Steve: you know, next version of Sun in some ways, too.
Steve: So anyway, in terms of like the why Rust, so I mean, you know, like fundamentally,
Steve: the, you know, okay, so first you decide, like,
Steve: you're going to get into this business so you know the the folks
Steve: who started it were previously at joyant before and they
Steve: ran a public cloud and so they kind of knew like what
Steve: it's like to be on the customer side of that relationship and so they were like okay
Steve: this sort of you know business needs to exist well you
Steve: know how are we going to like enable it and so you know
Steve: for a very long time a lot of the early folks at oxide were
Steve: you know very big c people obviously like you
Steve: know like i said Bryan started at sun working on the kernel
Steve: in like the late 90s or whatever at the same time i was learning
Steve: programming basically sorry Bryan you're a little older than me but like
Steve: you know so they've been using C for a really really long time and so you know
Steve: naturally kind of I would say we're initially a little skeptical of Rust conceptually
Steve: but like Bryan has a great series of blog posts on him personally coming to
Steve: realize like why he ended up liking you know Rust but.
Steve: I think that like sort of an
Steve: interesting thing about Rust positioning in this domain overall is that Rust
Steve: is kind of the first time in a very long time that there have actually been
Steve: a new language that can meaningfully replace C and C++ for the use cases where they're still in.
Steve: And what I mean by this, it's like a little complicated, but so like when I
Steve: started programming, C and C++ were sort of the like default choice for everything.
Steve: Even then I'm being, I'm waving my hands a little bit. Like the Mac OS I started
Steve: with used Pascal as its calling invention, right?
Steve: Like there's more options in many ways in the early nineties,
Steve: but like C and C++ were very dominant in many domains, you know,
Steve: not just network programming and operating systems programming,
Steve: but also application development. You would be writing stuff in that way.
Steve: And then, you know, Java comes along and it takes out a large chunk of the application use case.
Steve: And then scripting languages come along and they take out a lot of the sort
Steve: of like web development use cases. And so the overall scope of like where in
Steve: C and C++ have been the sort of dominant language has actually been declining for a really long time.
Steve: But the sort of like operating system and systems case is one where they are
Steve: truly has not been a challenger for a very, very long time.
Steve: I mean, some people wanted to make Java OS happen. It didn't happen.
Steve: And like there were some other people that came along and tried to do various things.
Steve: So, for example, I used to write the D programming language with I didn't write
Steve: the D programming language. I used the deprogramming language in college back
Steve: in that 2000, 2004 kind of like era or 2004 to 2010 kind of era,
Steve: but it had a garbage collector.
Steve: And that meant that it wasn't really, you know, even though me and my friends
Steve: in college were writing an operating system in it, although we did not get very
Steve: far in the end, but like we were able to get the beginnings of that going.
Steve: And while it worked, it still like felt like you were kind of fighting with
Steve: the language. And so a lot of like sort of dedicated C fans over the last 30
Steve: or 40 years have kind of seen people show up and be like.
Steve: We have a programming language that can do all the stuff C can do.
Steve: And that's like mostly been like, well, actually Java's too high level.
Steve: And like that sort of pattern has repeated itself over the years.
Steve: And so I think a lot of them have become very like kind of jaded against the
Steve: idea that there's a language that can like truly replace a lot of the low level use cases of C and C++.
Steve: And in fact, even early Rust had a garbage collector.
Steve: Like it was not necessarily actually a
Steve: truly good fit for cnc plus use
Steve: cases and so i bring this up partially because
Steve: it was in late 2014 that
Steve: rust made the decision so rust used to have a pretty big runtime and there was
Steve: the option to use native threads like operating system threads or green threads
Steve: and in sort of late 2014 there was the decision to pull out green threads as
Steve: a concept and move purely to native threads.
Steve: And that's kind of when Rust truly came into, it's like, you can really use
Steve: Rust for low-level tasks because we don't need this runtime anymore and all that kind of stuff.
Steve: And so it's funny because a lot
Steve: of people now think about Rust as kind of always been in that use case,
Steve: but it was honestly like really less than a year before 1.0 that it truly like
Steve: actually really kind of decided to truly go after that space. And so-
Steve: I bring that up partially because that is when Bryan specifically,
Steve: so the CTO, I've mentioned him several times with Bryan Cantrell,
Steve: the CTO and one of the co-founders of Oxide, saw that decision being made.
Steve: And he has said before, that is the moment where he decided to truly look into
Steve: Rust because he's a big anti-green threads person, or at least for the purposes of system stuff.
Steve: He's like, it's always been ripped out of every system they've ever tried. and so
Steve: the like that moment was when he was
Steve: like oh this is a language i need to take seriously because they
Steve: are actually finally deciding to truly do the low-level stuff
Steve: that i like care about and so so that
Steve: was i know when rust like truly first caught his eye
Steve: or at least like he had seen it before then but that was
Steve: the moment where he was like i need to investigate this truly and so it's
Steve: a very very long-winded answer but like part of
Steve: that is then okay so once you can do the
Steve: low level stuff like cool that like makes sense but
Steve: like sort of what value does it bring and a lot
Steve: of that is in the all the classic
Steve: standard rust things of you know we can have
Steve: a strong type system and we can eliminate certain values of
Steve: errors at compile time but i think a lot of people
Steve: only focus on stuff that's unique and not
Steve: the holistic package so like for example the the
Steve: tooling like cargo is like really really important for a lot of people
Steve: and stuff like enums we know once
Steve: you use enums you're like why is this not in every programming language
Steve: basically like we have structs or like you know product types
Steve: to get a little fancy with it and so you need enums or
Steve: the sum type to sort of complement that and it's so interesting that
Steve: many many languages have like one half of this equation but not both halves
Steve: and so there's a lot of kind of stuff like that holistically that makes rust
Steve: you know kind of very attractive but i think another one that's like really
Steve: interesting is that like i.
Steve: Don't necessarily like to think of the stack as being purely up and down.
Steve: Like it's not always exactly one way or the other.
Steve: But the important part is that like Rust has, because of its commitment to being
Steve: able to solve low level use cases, it also can make its way up the stack a surprising amount.
Steve: And so like a lot of times, you know, if you don't start with a language that
Steve: can address the lowest level needs that you have, you end up at least investing
Steve: in two different languages.
Steve: And I'll say that, you know, at Oxide, we definitely only don't only just use
Steve: rust for a hundred percent of the things, but like for 90% of the things.
Steve: And it's cool that we can even get to 90 because like.
Steve: A lot of times it's like, okay, we're going to write our web app in,
Steve: it's going to have JavaScript on the front end, and then Python or Ruby on the
Steve: back end, and then maybe we'll need to do some low-level stuff in C.
Steve: And you kind of have this at least three languages that are sort of necessary
Steve: to fill in this sort of stuff that you need to be able to do.
Steve: And so Rust enables us to get away with using one language for a lot more of
Steve: a chunk of that stuff than other things.
Steve: And so most importantly at the lowest level itself but
Steve: then also like if you look at my comments from
Steve: a long time ago back in 2014 era i would have told you i would
Steve: never write a web app in rust but like we have and
Steve: do many web apps actually at oxide and i think
Steve: it works totally fine for those use cases and not
Steve: necessarily for everyone but for us it does work really well and
Steve: so like that versatility is super
Steve: super useful and kind of like
Steve: you know enables us to do you know all
Steve: sorts of stuff so anyway i don't know if that like
Steve: fully covers it necessarily but i think it's just like important that like
Steve: c if we decided to go with c for
Steve: the full lowest levels of the oxide like stack we
Steve: would need some we would need to introduce another language much sooner
Steve: than we do with rust like rust gives us the ability to address
Steve: things at the high level of the stack as well as at the
Steve: low level of the stack and so i think that's like a very valuable thing when
Steve: you're talking about the scope of like you know we
Steve: do everything from the firmware up to the front end of website running in the
Steve: browser you know like we are a lot of people say full stack is like i can write
Steve: some front end and back end web code but we're like full full stack and so being
Steve: able to address all those use cases with fewer technologies is a legitimately valuable thing yeah.
Matthias: But some listeners who have a c background might say you folks just make it
Matthias: very hard on yourself there are things that you could have gotten for free if
Matthias: you chose c what would you say to them.
Steve: I don't really know what we would get for free necessarily.
Steve: Like, I mean, maybe not needing to like write some code, but also like sort
Steve: of the fundamental premise of the company in many ways is, so Alan Key has said in the past or NK,
Steve: like you can't meaningfully build your own software without making your own hardware.
Steve: Or I forget exactly how I phrased it. Maybe it's the other way around,
Steve: but like we sort of need to do both.
Steve: And like what that means on many levels is that like we're willing to write
Steve: our own stuff because, you know, sometimes just like if you're building something
Steve: custom, you need a lot of custom stuff.
Steve: And so it's true that maybe we get some stuff for free, sort of,
Steve: kind of, but then also means, you know, when it comes to things like support.
Steve: So we really care a lot about making sure that everything works and works well together.
Steve: And it's much, much harder to support a giant pile of other code that other
Steve: people have written than it is code that you've written that's custom for purpose.
Steve: And like you know bugs can appear sometimes because
Steve: something is solving a problem for someone else's use case
Steve: and not yours and a custom piece of software that only
Steve: does what you needed to do and nothing else ends up being easier
Steve: to understand and easier to take care of and so you know
Steve: i mean i definitely don't think it would be impossible without rust
Steve: like someone could make an oxide where c is used more
Steve: but like i also think that just like you know
Steve: at least for a seasoned rust developer like
Steve: rust overall saves me time implementing stuff
Steve: because i have to do so much less checking up
Steve: after the fact that what i'm doing is reasonably correct and
Steve: there's so many things that rust kind of gives you for free that you don't get
Steve: with c so you know i definitely i don't think it's possible but you know someone
Steve: is welcome to try let's put it that way like we we are we are demonstrating
Steve: it can be done with rust sure maybe it could have been done with c but that's
Steve: for somebody else to prove out, you know.
Matthias: This Alan Kay quote was really nice, where you say software informs the hardware
Matthias: and hardware informs the software.
Matthias: You can't build one without the other. How does it look at oxide?
Matthias: How does the hardware part of oxide and the software part of oxide,
Matthias: which is written in Rust, how does that inform each other?
Steve: Yeah, so one of my favorite stories, and this is not any work that I personally
Steve: did, but I love talking about my coworkers doing cool things.
Steve: So an example of this is, we talked a little bit earlier about the sort of the
Steve: standardization layers.
Steve: And so here's an example of how they can kind of like get in the way.
Steve: When you're, say you have an entire room full of servers, right?
Steve: And you, you know, maybe, you know, you're running tons and tons of jobs every
Steve: day. What that means is that small failure rates happen.
Steve: Like, I used to say this about living in New York City. It's like people be
Steve: like, oh, there's crime in New York City. And it's like, sure.
Steve: But that's also because there's millions and millions of people.
Steve: So a one tenth of a percent chance that something happens means that it happens
Steve: hundreds of times a day in New York City or whatever.
Steve: I probably have the order of magnitudes off. But like the point is,
Steve: is at scale, things that don't happen very often start to happen and happen often.
Steve: And what that means is say that your hard drive controllers firmware has a bug
Steve: and that bug only manifests 1% of the time.
Steve: Well, if you have 100,000 servers, that 1% of the time is going to be happening all the time.
Steve: Now, obviously, 1% is a very high rate for a bug in firmware.
Steve: Like, I'm not saying that firmware bugs happen literally 1% of the time,
Steve: but just like you will run across obscure edge cases in the software and hardware
Steve: that you use, and those problems will occur.
Steve: And so one of the things that standardization enables, which is great,
Steve: is having, as we said earlier, tons of organizations come together and build
Steve: all the stuff that goes into a computer.
Steve: But the problem with that is that like, so say you buy a server from Dell,
Steve: and you come across a firmware bug in the motherboard.
Steve: Well, Dell didn't write the firmware that's running on that motherboard.
Steve: It's some other company, you know, that ended up writing that firmware.
Steve: And so if they need to like fix that problem, they will like file a ticket with their vendor.
Steve: But then it's like, good luck getting that prioritized. You know what I mean?
Steve: As a customer, you are now dealing not just with the company you bought it from,
Steve: but the companies that they bought their stuff from.
Steve: And so you're not the customer of that motherboard vendor.
Steve: And so why should they care about you over someone else or whatever?
Steve: And so one of the things that we've sort of done is thrown out a lot of those
Steve: layers because we don't need...
Steve: The fundamental purpose of stuff like BIOS and UEFI, for example,
Steve: is to enable the operating system people to sort of right against the UEFI or
Steve: BIOS spec, and then for all the hardware vendors to sort of like produce the
Steve: APIs in their hardware that fit into that BIOS or UEFI spec.
Steve: But like, we don't need to support 75 different manufacturers of RAM,
Steve: you know, or five different manufacturers of hard drives or whatever.
Steve: Like we have at the moment, since we have the first revision,
Steve: or we're working on the second revision of the rack now, but like,
Steve: you know, we know physically what hardware is in the machine.
Steve: And so all of that extra sort of standardization interface is written for a
Steve: benefit that we don't actually see.
Steve: Like we're not we don't need to
Steve: be able to you know build all of these
Steve: different variants of all this different stuff and so what that
Steve: means is all that code is being written to serve a purpose that we don't need
Steve: anymore and so we've actually completely thrown out that layer in the oxide
Steve: rack there is no bios there is no uefi the the operating system boots the hardware
Steve: just sort of like in the very old days before that stuff even existed and so
Steve: what that means is that we
Steve: to throw out that firmware and so specifically like for amd there's a thing
Steve: called a gisa that's like part of this firmware package that you get that you
Steve: know you use to boot up the amd's cpu.
Steve: And we said no we're not going to use that and we wrote our own and
Steve: you know at first amd was kind of asking us like why are
Steve: you guys asking questions about this like that's just like in you know
Steve: the firmware and we're like yeah we're writing our own firmware they're like we
Steve: don't really believe you or like you know they were like kind of okay guys
Steve: sure whatever you say and eventually once we got it to boot we're
Steve: like hey by the way like you know here's here's an example of this booting and
Steve: they were like oh that's like really cool you know because we didn't expect they
Steve: were to do that because literally no one else does this and
Steve: so you know what that means on some
Steve: level is like stuff boots really quickly but like
Steve: how much does that matter because you're not really booting a server necessarily all
Steve: the time but more importantly like by throwing
Steve: away all of that stuff you know we've eliminated
Steve: a ton of possibilities for things
Steve: to go wrong you know we have eliminated security
Steve: issues like you know talk about another thing that we
Steve: got rid of is like the bmc or the baseboard management controllers that
Steve: you know server grade hardware has a ton of other computers running inside it
Steve: to make sure that the computer is running correctly so like for example you
Steve: really want your main cpu to be running you know the jobs that you're running
Steve: on the server you don't want it to be running stuff to manage the server itself right and so like.
Steve: Server-grade hardware has this additional CPU and other stuff in it called a BMC.
Steve: And that's usually, you know, making sure that like, oh, if something crashed
Steve: on the main CPU, we can reboot it.
Steve: Or, you know, we're able to like log into and monitor stuff from behind.
Steve: But like a lot of, you know, server vendors, those BMCs have like full operating
Steve: systems running on them.
Steve: And those full operating systems have bugs and they can have problems and there's,
Steve: you know, security issues. And it's like, it's really, really hard when you're
Steve: buying a server today to even know what code is running on it at all.
Steve: And so we have an equivalent thing, which I joke is the totally not a BMC,
Steve: we call it a service processor, but it serves the same general idea of like,
Steve: okay, it is a good thing to have a little mini extra computer monitoring the
Steve: main big computer to make sure that all that stuff works.
Steve: But like instead of just accepting
Steve: the one that would come with the like motherboard manufacturer that
Steve: we would buy something for since we designed our own motherboard we designed
Steve: our own replacement system and it is much much
Steve: smaller and it runs an os called hubris that we
Steve: wrote from scratch instead and so you know we know that you're not running a
Steve: full linux inside of the mini computer inside of the thing that's running your
Steve: big computer and that means we can be more efficient means we can have less
Steve: attack surface area it means we can audit everything because we're not just
Steve: like accepting whatever the external manufacturer is giving us.
Steve: And so that provides like a ton, a ton of benefit for, you know,
Steve: doing this sort of stuff.
Steve: And so anyway, that's like an example of like, we need to write the software
Steve: that manages the hardware in that way.
Steve: And it's only because we're designing our own hardware that we're able to write
Steve: the software in that way, because, you know, we sort of know what we're putting in the rack.
Steve: So if we didn't know, we would need a lot more of those standardized interfaces, you know?
Steve: And so that's kind of like an example of those two things informing
Steve: each other very deeply now the downside is you know
Steve: we're releasing a new version of the rack new versions of
Steve: cpus means we need to write more firmware right whereas
Steve: before we would just be able to like update it to whatever
Steve: and someone else wrote in that code for us so i'm not going to say that there's
Steve: no trade-off there there obviously is a very large trade-off but that's one
Steve: that we're willing to make and we think is like the only way to deliver on a
Steve: lot of the sort of like quality promises and support promises that we want to
Steve: give customers and reliability promises.
Matthias: So I knew that you had your own operating system called Hubris,
Matthias: and I wanted to talk a little bit about that because some people might wonder,
Matthias: why did you start your own operating system if something like TockOS already
Matthias: existed even back in the day?
Steve: Yeah.
Matthias: What's so special about Hubris that you needed to write that thing yourself?
Steve: So I've actually known the Tock people for a very long time.
Steve: They're great, and I really like their project in general. oxide before
Steve: i joined had investigated using Tock instead of hubris and
Steve: decided not to and basically what that boils down to
Steve: is a thing about embedded use
Steve: cases is that diversity is the rule not
Steve: the exception and like what i mean by that
Steve: is every application tends to be different and because you're
Steve: like literally often running on different hardware you have different
Steve: needs and so Tock is very focused on sort
Steve: of this use case where you kind of load a
Steve: variety of different programs at runtime and they also care a
Steve: lot about supporting programs that are written in c they have
Steve: other goals too but at the time i should at least frame this like you know it's
Steve: been a couple years since the decision was made so i don't necessarily want
Steve: to speak fully to what toxic goals are right now but at the time ox i was looking
Steve: into it it was like okay they're they're interested in dynamic program stuff
Steve: and they're interested in supporting c programs and we didn't really have the same needs and so.
Steve: The way that Hubris is different and sort of like kind of the reason we ended up like writing our own.
Steve: First of all is like saying you write your own OS sounds like a massive undertaking,
Steve: but the kernel and Hubris is like 4000 lines of Rust code. Like it's not very big, actually.
Steve: And I mean, a lot of the drivers and the stuff that you need to actually make
Steve: it useful and meaningful is a little bit more.
Steve: But like it is feasible and was largely designed by one person, just Cliff Biffle.
Steve: Many other people have also helped significantly but like you know
Steve: he was kind of the person who did a lot of the initial sort of design
Steve: work and what what makes
Steve: hubris special is that it is aggressively static
Steve: and by that what i mean is like when
Steve: you sort of like so when you think about an
Steve: operating system you're like the operating system's on my computer and then i
Steve: install programs then i run programs right like i click on
Steve: discord it boots up and it runs i click on firefox or
Steve: chrome or whatever and it boots up and it runs hubris is
Steve: kind of like when you make the image to even
Steve: install on the hardware at all you say up
Steve: front here are the programs that i am running and those programs run all of
Steve: the time like there is no sort of like dynamic list of here's how many programs
Steve: are currently running there's no like he let me load a program at runtime that
Steve: wasn't running initially.
Steve: It boots up, it starts all the programs, they're all running,
Steve: and there's one instance of each program.
Steve: It's not even like, okay, maybe I have three Firefox windows running or whatever,
Steve: and obviously those share some code, whatever, blah, blah, blah.
Steve: But the point is, on a regular computer, I can be running five instances of
Steve: Bash all at the same time.
Steve: And so in Hubris, you have one instance of every program, and they're running
Steve: all the time. And that's something that, for our purposes...
Steve: Works really well, but does not necessarily work for other people's use cases.
Steve: And what's cool about stuff being sort of static is it means that you can make,
Steve: I don't even say shortcuts exactly, but you can like skip some design decisions
Steve: that other people have to deal with.
Steve: So, for example, because we know every program that's running and that it's always running.
Steve: Like Hubris doesn't really use like virtual memory, for example.
Steve: Because we can actually like pre allocate, you know, at build time.
Steve: OK, in the image, these are where these are the where all the programs are running individually.
Steve: Like and we know the memory map statically of like where everything lives and
Steve: we could tell, you know, OK, this will fit.
Steve: Like we do have to worry about programs crashing at runtime to some
Steve: degree but i mean like something like you know and in
Steve: the days of virtual memory now on desktop machines this doesn't really happen but
Steve: like back in the day you know when i was using this first max right
Steve: you'd be like i have this much ram and my
Steve: programs are currently using this much ram and if i start up a new program it's
Steve: going to literally run out of ram and not work right and hubris is closer to
Steve: those systems since we don't use virtual memory and so like we don't
Steve: have to worry about the the problem of i start
Steve: up a program and there's not enough RAM to run it because like we know
Steve: at build time there is enough RAM to run the programs that
Steve: you're trying to run you know for the most part and so like
Steve: those kinds of aggressive design simplifications mean that
Steve: we can get away with doing a lot less so another great example is like there's
Steve: no global memory allocator or global heap in hubris because we don't need dynamically
Steve: length lists because we know at build time how many programs are running So
Steve: the OS does not need to keep track of a dynamic list of like,
Steve: here's how many programs are currently executing.
Steve: You know what I mean? And so these kinds of like design decisions all build
Steve: on top of each other and, you know, sort of enable us to do something a little different.
Steve: And that also means likewise, like I would never suggest that the Tock people
Steve: stop building Tock and use hubris for everything because they're just literally
Steve: trying to do something different than we're trying to do.
Steve: And so, yeah, that's, that's kind of, I would say that like at the highest level,
Steve: that's what hubris is, is trying to do.
Steve: Everything is aggressively static. It's all at compile time. It's all at build time.
Steve: Another thing is that hubris is very like a message passing OS,
Steve: but is also fully synchronous.
Steve: So we do use async rust a lot higher level in the stack.
Steve: And we don't necessarily think that async rust is bad in the lower levels either,
Steve: but for hubris specifically synchronous is much, much simpler than asynchronous.
Steve: And so when we're talking about the firmware that runs at the lowest levels
Steve: of our entire thing, we're really, really going for a simple system that could
Steve: be understood and reasoned about very easily.
Steve: And so Hubris is also aggressively synchronous.
Steve: And so that's a thing that works for us, but does not necessarily work for everybody else. when.
Matthias: You explained that you have one instance of one program at a given time i thought
Matthias: oh that would make scheduling extremely easy because essentially do you even
Matthias: context switch between those processes do you have some time sharing mechanism how does that work.
Steve: Yeah i mean we do have to still do some of that but like the
Steve: scheduler can be really really simple because you know
Steve: like it doesn't we're talking you know i
Steve: i forget exactly how many are running on like the real we're talking on
Steve: the order of like 10s not on the order of hundreds of processes right
Steve: so like you know the scheduler itself can
Steve: be really really simple like if i remember correctly it's basically just
Steve: around robin scheduler but there's a little bit of interesting bits around you
Steve: know if a program is waiting on an interrupt to come in then we don't need to
Steve: start it up again because we know that it's waiting on something like that but
Steve: like you know it can be aggressively simple because of the fact that you know
Steve: you're not talking about we don't need to handle you know what happens if someone
Steve: spins up a thousand instances of something and how do we deal with that. Right.
Steve: And so it's all like very, very straightforward and very, very simple,
Steve: but that ends up being also reliable and easy to debug and, you know, stuff like that.
Steve: So yeah, we have a little bit of like process priority stuff and there's some
Steve: important things to like.
Steve: A problem once you have process priority and scheduling is like a problem called
Steve: priority inversion, where a program with a low priority ends up waiting on a
Steve: program with a high priority that's blocked and or maybe vice versa.
Steve: I forget if I said that backwards, but the point is that like you get stuck
Steve: in loops, you know, and so we can actually like validate that that doesn't happen
Steve: because of the fact that everything is so straightforward.
Steve: And the fact that we don't do a lot of dynamic stuff is we can actually sort
Steve: of like lint against like, hey, you may have a program that's like going to
Steve: block execution of this other program and so stuff like that.
Steve: Yeah, it's much more straightforward than you would see in a commercial desktop
Steve: operating system solely because we're able to pare down those requirements so
Steve: much because of the limited context of like what this is actually trying to do.
Steve: And it turns out that like, you know, aggressively scoping down requirements
Steve: is what allows you to have simplicity in a system. Sometimes if you have complicated
Steve: requirements, you're going to have a complicated system.
Steve: I don't think simplicity is always better than complexity.
Steve: It's about how simple can you make something within a given set of requirements.
Steve: And sometimes you can over-scope requirements down too simple,
Steve: and then the thing doesn't work correctly.
Steve: But it turns out for embedded devices or use cases like this,
Steve: you can be really, really aggressive with how you scope stuff.
Steve: And then that can make a system simpler and easier to understand and therefore more reliable.
Matthias: Must be a bliss to develop such an operating system.
Steve: You know, it's nice and it's got its problems too. Like, you know,
Steve: I mentioned cargo being really useful before.
Steve: It is true that cargo is wonderful and it's definitely very useful,
Steve: but also we basically had to write a build system on top of cargo to make this
Steve: work because cargo has several deficiencies with dealing with this kind of problem.
Steve: Is actually one of the first things that i worked on at oxide was sort
Steve: of rewriting the build system yet again on top
Steve: of cargo you know definitely like it is it is a very different thing than developing
Steve: a web app or developing something that's a much higher level and you know it's
Steve: definitely some people enjoy it and some people don't and you know i used to
Steve: work on that stuff and i found that it was a pleasure but you know obviously
Steve: everybody has their different levels of expertise tell me a little.
Matthias: Bit about that what were some of the limitations that you ran into with cargo
Matthias: and also why did you have to build your own build system how does that look like.
Steve: Yeah so a very simple and straightforward one
Steve: is that cargo kind of very deliberately does not include any post-processing
Steve: of stuff and we need to build an operating system image so like on some level
Steve: there is a step after cargo runs you know cargo produces a program or a couple
Steve: programs and then we need to like assemble that into a thing that you would
Steve: actually you know load onto to the microprocessor.
Steve: So just even in those pure cases, we need something more than cargo because
Steve: cargo deliberately leaves that to some sort of other process.
Steve: So that's one example of sort of a lot of code in the Hubris build system is
Steve: like, okay, now that we've gotten cargo to spit out all the final programs,
Steve: how do we actually assemble that into an image for the OS to boot?
Matthias: How did you do that?
Steve: So there's a pattern called the X task pattern. And that's like the started
Steve: by Matt clad is kind of this idea that you sort of have a program in your workspace
Steve: called X task and you sort of write sort of scripts in Rust that sort of work that way.
Steve: And so we kind of have a very extensive set of X tasks that then like you basically
Steve: rarely invoke cargo build itself.
Steve: You like invoke used to be, I'm not sure if this has changed lately,
Steve: but like cargo X task dist to generate a distribution and that will go.
Steve: Okay cool i need to build you know
Steve: these five programs so let me invoke cargo five times to
Steve: build those five programs and then take the final binaries they produce
Steve: out and like run some code to figure out like okay you know how do i need to
Steve: tweak this stuff and whatever and so it's all it's all in rust code but it's
Steve: sort of written as this kind of like build system kind of like on top of things
Steve: another sort of area where cargo is like kind of a little weak in this area is like.
Steve: When you're doing embedded use cases you often need to build
Steve: some things for the host but some things for the target so like
Steve: if i want to build a thing where i'm building some
Steve: code that's supposed to be running on my local machine but then i also want
Steve: to build some programs to like be cross compiled cargo
Steve: is not very good at that it kind of understands you know you get one target flag
Steve: and so it'll try to build everything for the thing that you say you know
Steve: target for and so there's a lot of sort of like kind of like nitty-gritty issues
Steve: that become annoying when like say you know you want to be able to say like
Steve: okay these three programs are meant to be built on 32-bit arm v8 and 32-bit
Steve: arm v7 and then this is built on ar64 and then this is built in x86.
Steve: And it kind of is like not as good at those kinds
Steve: of use cases this comes up in a higher level use case if
Steve: you ever try to build like a project with wasm where you want to
Steve: build like say a server for your desktop you
Steve: also want to build wasm for your front end it's like a little awkward at
Steve: some of that kind of stuff and and yeah just like a
Steve: lot of sort of like nitty-gritty things like that like cargo cargo
Steve: is fantastic when you're sort of in the
Steve: normal use case but the more you sort of diverge from like weird cases like
Steve: you know sometimes we need to pass in interesting configuration stuff and it
Steve: can get a little gnarly sometimes to do things like that so yeah you know it's
Steve: definitely not like we're not going so far and you know obviously said we built
Steve: this on top of cargo we're not going so far as to throw cargo away although
Steve: i am very interested in the
Steve: buck build system but i have not tried to actually port hubris's build system
Steve: over to it although i've joked about trying it a number of times but you know
Steve: there's sort of at some point you kind of end up outgrowing cargo to some degree
Steve: but a lot of times it comes down to these sort of things where cargo is deliberately
Steve: saying that's out of scope so it's not like.
Steve: Cargo is inherently bad at those things and we're choosing not to do
Steve: it it's more like like i said right earlier about if you can
Steve: aggressively scope down your requirements it makes things easier cargo sort
Steve: of determined that like pre or post processing build stuff
Steve: is just not something cargo wants to do they want you to lay another tool on
Steve: top of it and so that's just something we've like had come up in another a number
Steve: of instances where we sort of need either pre or post processing of stuff that
Steve: cargo is going to do and so we kind of have to like put something over over
Steve: top of it to make it work and you know i don't think that's something that the
Steve: cargo folks would find to be objectionable i think that's what they would like
Steve: advise us to do in those cases because it's just not trying to do that it.
Matthias: Makes sense to keep the scope of cargo sort of
Matthias: small and and focused and there are also a couple of escape hatches that people
Matthias: can use if they don't want to go forward and build their own tooling for example
Matthias: you can have your own little build rs for pre-processing i guess and then you
Matthias: have workspaces which is very nice feature that a lot of enterprise customers use.
Matthias: I guess once you reach a certain level of maturity with your project and a certain
Matthias: scale, then you end up bumping against those limitations from time to time.
Matthias: For example, I also know that you have your own little CI service.
Matthias: I don't know if that is what you
Matthias: talked about, but the thing is called Build-O-MAT. And what is that about?
Steve: So, okay. So we talked about us writing our own operating system in the sense
Steve: of hubris, but that's used for the sort of embedded use cases.
Steve: The actual like control plane, the thing that's like scheduling your VMs to
Steve: run on the hardware, that's a repository we call Omicron, which was started
Steve: before COVID, kind of became an awkward name.
Steve: Oh, well, we're kind of past Omicron in terms of COVID stuff now.
Steve: We were actually just, you know, I've joked occasionally, like I wonder if we're
Steve: ever going to accidentally name something after another COVID variant. That'd be unfortunate.
Steve: But anyway, that needs to run something called like the host OS.
Steve: And so we do have like a more fully featured OS that's running on top of the
Steve: embedded stuff that is the thing that's actually scheduling your VMs.
Steve: And so for that, you know, many places use Linux with a KVM to like do those kinds of things.
Steve: But we decided to choose to use Illumos and bhyve.
Steve: And Illumos, for those of you who do have not heard of it, which is probably
Steve: most of you, to be honest, is a descendant of like Solaris and SunOS,
Steve: which is a descendant of the BSDs. So if you go back to Unix history,
Steve: they're kind of like cousins to Linux.
Steve: There is a common ancestor way back there, but Linux came as a separate re-implementation
Steve: of Unix, and Illumos is kind of like one of the many descendants of the tree
Steve: of actual BSD, actual Unix. Yeah.
Steve: So, yeah, so Illumos and bhyve is kind of like Linux and KVM,
Steve: if you want to think about making analogies to that stuff.
Steve: And so, yes, so like at those levels of the stack, we are running a full OS
Steve: that like is largely written in C.
Steve: Illumos has been around for, you know, it is, again, literally descended to that code.
Steve: So there's a very old code in there. we do run you
Steve: know right like we maintain the illumos port of rust for
Steve: example to make sure that rust programs are right that work well
Steve: on that and we do write some rust code and there is some rust
Steve: being put in illumos a little bit here and there but there's a
Steve: lot of reasons why the decision is a good one that makes sense for us but sort
Steve: of again kind of like a classic thing that comes up in oxide related
Steve: discussions is if you go your own way and build something custom
Steve: it also means you need to go your own way and build something
Steve: custom so there is no built-in github
Steve: actions runner that works with
Steve: illumos you know you can say please give me linux please give
Steve: me mac please give me windows but like you can't say please give me illumos and
Steve: github does have self-hosted runners they call it but that kind of requires
Steve: you know you to have a certain kinds of setup for some things and i don't fully
Steve: work on this so i can't speak to like all the details super specifically,
Steve: but like, basically just like at that point, you're already doing a bunch of
Steve: work to make this work out.
Steve: And so effectively, Build-A-MAT is used for Illumos native jobs.
Steve: Where we, because some of the stuff does, you know, because we're doing systems
Steve: work, you know, a lot of stuff you can test on Mac, Linux and Windows and make sure that it works well.
Steve: You know, a lot of our developer tooling, for example, runs on regular old GitHub
Steve: actions with a bunch of that stuff.
Steve: But occasionally we need to test on an honest to goodness Illumos system that
Steve: the thing actually works with Illumos because we're making Illumos-specific
Steve: system calls and dealing with Illumos-specific functionality.
Steve: And so Build-A-MAT was kind of created as sort of a way that we can plug in.
Steve: Illumos-specific jobs into these CI systems and make sure that that works.
Steve: And so if you look at some of our repositories, you'll see GitHub Actions,
Steve: sometimes there'll be normal Actions that'll also be like Build-A-MAT,
Steve: and that's basically for those Illumos things.
Steve: But sort of another reason why it kind of makes sense for us,
Steve: you know so github actions is running on some servers that github rent somewhere
Steve: but like we also you know we make servers we got a bunch of servers you know
Steve: not just because it's fun and because dogfooding is good but like also because
Steve: we do want to make sure our stuff works like.
Steve: You know, having build a mat run on an oxide rack in the office that's testing
Steve: the code we are using to build oxide is, you know, like a thing that makes more
Steve: sense for us than it would for other companies. You know what I mean?
Steve: And so like, you know, we spend money to get CI servers, we already have servers,
Steve: why not use our own servers for own CI?
Steve: You know what I mean? So being able to do that is also, you know,
Steve: really kind of useful and helpful.
Steve: But the core of it, you know, or the reason why we started doing that really does boil down to,
Steve: you know we have very specific needs for our ci it
Steve: is not reasonable for an upstream provider to be able
Steve: to offer or offer in a way that like totally makes sense for us
Steve: and so you know on some level that means needing to dig in and write our own
Steve: stuff and so oxide is also you know a lot of people say rewriting software is
Steve: bad and those people are not necessarily wrong but rules of thumb are only rules
Steve: of thumb they don't mean that they're laws of truth you know what i mean And
Steve: so Oxide is a place where we very,
Steve: very often find legitimate needs to rewrite some software.
Steve: And I'm not going to say that it's perfect or that it never introduces problems,
Steve: but it works far better than the people who say never, ever rewrite your own
Steve: software ever would lead you to believe, basically.
Steve: Because it turns out that like writing a basic
Steve: ci system is not an impossible task you know
Steve: it's it's a project that like one or two people work on and
Steve: in you know along with their other responsibilities and it serves
Steve: us well and so you know you may think that like that's an impossible thing to
Steve: do but some people think writing your own operating system is an impossible
Steve: thing to do and we did that too so you know it's just like it's how it goes
Steve: it does mean you know this is part of the reason why it took us four years to
Steve: get from starting the company to shipping servers you know I'm not going to
Steve: say that it's not easy necessarily,
Steve: but not everything that's worth doing is easy.
Steve: So what we're doing is very ambitious and hard. And sometimes that means you
Steve: just have to commit to doing the work.
Matthias: You say it as if it was easy, but you have some of the best people in the world.
Matthias: It's like an excellent team that you assembled there.
Matthias: And what does it take to write Rust at that level?
Matthias: Do you have any coding guidelines internally that you stuck with?
Matthias: Is there anything internal where you say, hey, other companies could also profit
Matthias: from that knowledge too?
Matthias: Things that we might avoid even, for example, overusing generics or mixing sync
Matthias: and async. what are some of the patterns that have evolved when using Rust at that level.
Steve: Yeah i mean i also want
Steve: to like i love my team everyone
Steve: who works docs at is great but also like we are while we are very senior engineers
Steve: almost universally you know there's not something super magic that necessarily
Steve: makes us hands and shoulders better than everyone else i also want to like you
Steve: know i appreciate the kind words and i don't want to say my co-workers are bad
Steve: at their jobs because they're not. They're definitely very accomplished.
Steve: But also, it's engineering, it's not magic.
Steve: So there are also many, many good Rust engineers in other places as well.
Steve: What I will say is that when I started four and a half years ago,
Steve: there was a lot less experienced Rust engineers in general, period, just in the industry.
Steve: And so a lot of the folks that we hired at the beginning were not necessarily
Steve: fantastic Rust programmers.
Steve: They were experienced and fantastic engineers in general, but they sort of,
Steve: not necessarily picked up Rust on the job entirely, but let's just say we didn't
Steve: have the luxury of demanding that people were good at Rust before joining Oxide in the earlier days.
Steve: And so part of me coming on relatively early on was kind of replicating the...
Steve: If you commented on Rust and Hacker News in the last 10 years,
Steve: you probably got an answer from me personally at some point because just like
Steve: I spent my time doing that.
Steve: And so in the early days of me working at Oxide, I would spend a lot more time
Steve: answering rust questions, helping people with the rust stuff,
Steve: giving advice and choosing packages.
Steve: But we also did have a lot of people who had a lot of deep rust experience when I started as well.
Steve: But like part of me coming on initially was to sort of help make sure that if
Steve: people had rust issues that like I was able to help, you know,
Steve: sort of guide them with stuff like that.
Steve: By now, you know, years later, there are many more experienced rust engineers out there.
Steve: And so I would say that like, one of the factors that sort of happened
Steve: is the overall quality of like the rust hiring
Steve: pool has like gone up and so at this point you know
Steve: we are much more likely to hire someone that knows rust than not know rust simply
Steve: because there are enough people that know rust at a high enough level that we're
Steve: able to find and hire those people so so that's changed over time to some degree
Steve: but also some things that we do that sort of any company that's doing rust can
Steve: kind of like replicate to help out with these sort of things.
Steve: We have a dedicated channel in our chat system that's just purely for Rust questions.
Steve: And so people will ask Rust specific questions. And I and many others who have
Steve: a lot of expertise in Rust will make sure to pay attention to that question
Steve: and answer questions as they come up. I think that's really important.
Steve: Having space for people to ask things is really a big deal. We have a biweekly
Steve: meeting that's optional.
Steve: Other than one-on-ones, all employees can go to all meetings.
Steve: So I want to say it's available to everyone, but that's just kind of true of
Steve: stuff at Oxide in general.
Steve: But if your company doesn't do that, I would still recommend a sort of like,
Steve: open to everyone, sort of rust, we call it the rust study group.
Steve: And basically, it's a meeting where, you know, every week, me and a couple other people who,
Steve: you know, love rust stuff, have always blocked off on our calendar to make space
Steve: for if somebody has a question that's maybe not hyper urgent,
Steve: you know, or, you know, sometimes people do like, oh, I was working on this hobby project.
Steve: And I came across, you know, I didn't want to bring it up at work in work hours,
Steve: because it's not really about work.
Steve: But I do, you know, like, I've been learning this thing about
Steve: rust and i don't really know this detail or whatever you know
Steve: that maybe helps them you know with their job and some other you know
Steve: maybe it's not on a project that's job related but like leveling up
Steve: a rust is still going to be good you know for your job so people will
Steve: come and bring questions about like you know oh hey i saw this thing go by earlier
Steve: in the week where people were talking about this new feature and i was curious
Steve: what people thought or how it works like i don't really understand why people
Steve: care about this or you know maybe they're like i have this tricky bit of code
Steve: that i can't really figure out why the bar checker's mad at me or like you know, whatever.
Steve: And so, you know, we kind of have both of those forms of sort of explicit time
Steve: and space to help people with Rust related problems that they kind of have.
Steve: In terms of specific things that have popped up, I think the biggest thing that's
Steve: kind of, we've talked about it a little bit, but we haven't really done as great
Steve: of a job of like getting our perspective on the conversation out there.
Steve: But a really big thing that's popped up with us that some people are talking
Steve: about for sure is async cancellation.
Steve: And so for those of you who aren't familiar, async Rust has what I think is
Steve: a really great cancellation model conceptually, which is like futures don't
Steve: do anything until they're pulled.
Steve: And to cancel a future all you got to do is just drop it and
Steve: then it won't ever be pulled again and so it's effectively canceled and that's
Steve: like cool but that also
Steve: means that there are subtle issues where you don't realize that
Steve: a future can be canceled because like dropping something is
Steve: not obvious in rust all the time and so
Steve: there are some patterns you can get into where something
Steve: will cancel and you don't realize it and that will lead to sort of like
Steve: logic bugs and so this is
Steve: sort of an area where we've kind of like we we
Steve: it it's a thing that surprises people because you
Steve: know if it compiles it works is not literally true but it
Steve: definitely can feel like it's true sometimes and so
Steve: a lot of people's experience with rust early on is like
Steve: oh my god it catches all these problems at compile time and
Steve: i don't have these bugs because i used to have to always worry about
Steve: you know like them happening and now the compiler catches them
Steve: for me and that's great and they go into async and they
Steve: hit their first time where something gets canceled where they don't expect it to and it
Steve: feels it feels like walking that back they're
Steve: like back in the land of like oh my god the compiler doesn't
Steve: catch this problem for me anymore and now i feel
Steve: like you know these like promises of rust catching all these things are like
Steve: a little not as true as they used to be and i think some of that is just a like
Steve: a natural counter reaction to how much rust handles for you in most cases but
Steve: it is true that like async cancellation issues are like not generally statically find outable.
Steve: And they also, you know, can be surprising. And that becomes itself surprising
Steve: when you're sort of so used to Rust catching things aggressively.
Steve: And so we've had to try to find some places where like, we can figure out like,
Steve: okay, you know, here are some patterns to be avoided, or here are some libraries
Steve: of like trying to be a little more explicit about cancellation.
Steve: I'm totally drawing a blank off the top of my head but there's been some experience experiments,
Steve: that some of my co-workers have tried to do to sort of like figure out some patterns
Steve: like that and we've wanted to kind of like talk a
Steve: little more about these problems publicly but have not actually had the
Steve: time because it's always super busy when you're doing a zillion things to
Steve: like necessarily get out there with all their experiences but definitely it's
Steve: a thing that the async folks are familiar with
Steve: being an issue overall and i think that anyone that does rust async at scale
Steve: has eventually either found out about these bugs or maybe you've had some mystery
Steve: bugs you can't track down that eventually you'll discover cancellation related
Steve: but that's definitely an area where we've like found stuff to be a little surprising
Steve: and have like started to get out there a little bit and talk about sometimes you.
Matthias: Said the drops are sometimes not obvious in rust and that can lead to subtle
Matthias: bugs with phasing execution can you give me an example for that.
Steve: Yeah so i'm gonna be very vague because to
Steve: be honest this is not an area of the product i work on personally and when i
Steve: was in these conversations was more like six months ago so i'm like a little
Steve: teeny bit out of the loop but let's just say like a big thing is like a concept
Steve: in this area is like if something is cancel safe and what that means is sometimes
Steve: you have some code that is like,
Steve: If you drop it in the middle of its operation, you need some sort of cleanup,
Steve: or you need to do some sort of thing or notify someone or something.
Steve: And so that won't necessarily happen.
Steve: If an operation is canceled in the middle of it occurring. So for example,
Steve: like, say that you're doing a select between some sort of like receiving on
Steve: a channel, and then you're also like printing, you know, you're sort of selecting
Steve: between some sort of timeout,
Steve: and some sort of receive on the channel, like that if,
Steve: if like the if the, if the timeout
Steve: finishes before you know you're calculating some
Steve: sort of value while waiting on the like receiver then
Steve: like that means the other future and that select gets dropped like
Steve: you know like if you're doing a select it kind of inherently means that
Steve: you're going to drop the other things that did not finish early and
Steve: so depending on how you've written those futures like obviously
Steve: waiting on a channel and you drop it that's fine you're just no longer listening
Steve: on the channel and say you're just sleeping and you drop that it's fine
Steve: you're no longer sleeping but say that the thing that happens first is
Steve: the timeout maybe that calculation had some sort of thing where it
Steve: needed some sort of cleanup step and now because it's timed
Steve: out that future just gets dropped well if you didn't do something
Steve: to ensure that the cleanup would happen properly then you
Steve: know maybe that doesn't necessarily like work and so this is like code that
Steve: is totally fine if all the futures are said like cancel safe if they have some
Steve: sort of ability to do you know that kind of thing but you know if they don't
Steve: then that can kind of like be an issue and so you know that's like yeah like
Steve: Like you can lose data sometimes.
Steve: So for example, say, you know, in one future, you're kind of like writing some
Steve: data inside of a loop and then that gets canceled in the middle of it.
Steve: Well, that maybe means you only wrote part of the data, but you never wrote
Steve: the final part of the data.
Steve: And so that's like an example of like a problem that can occur if something
Steve: is not necessarily like fully cancel safe.
Steve: And so, you know, this is where a lot of the discussion about async drop comes
Steve: in and like a lot of that stuff. If you've seen that discussion in sort of the
Steve: Rust async world. And so, yeah, I don't know. I guess that's like a high level
Steve: example of what's going on.
Matthias: Can you quickly explain what async drop means.
Steve: Yeah so there's a drop trade in
Steve: rust and when something goes out of scope for the final time
Steve: then the drop tree gets called and that runs some code this is like into a destructor
Steve: in many other languages like especially op languages and definitely conceptually
Steve: sort of similar but in the async world if a future doesn't get pulled anymore
Steve: there's no equivalent like hook right like the drop trait kind of gives you
Steve: the ability to They'll sort of hook into,
Steve: hey, this object is about to be destructed or destroyed. So like do something then.
Steve: So like a classic example is box. You know, you allocate memory up front and
Steve: then when drop runs, you know, it would deallocate the memory.
Steve: So a future doesn't have any sort of similar kind of mechanism.
Steve: Like if you would allocate some memory manually in a future and then it no longer
Steve: gets pulled, now that memory is leaked because there's no equivalent hook and
Steve: point to sort of say destroy that memory that was called.
Steve: And so there's been a lot of different proposals for what an async drop trait
Steve: could look like and how that might happen or how we might want to solve the
Steve: problem in a different way.
Steve: There's been sort of a discussion in the async working group for a couple of years now.
Matthias: Why can't you just implement drop for the future?
Steve: Because drop is a synchronous, it is a function, not an async function.
Steve: And therefore you're sort of like calling something synchronous inside of the,
Steve: and plus the other reason is like the definition of dropping it,
Steve: like it's no longer being called anymore, like kind of is a fuzzy definition.
Steve: Like if I don't call, if I don't call poll for an hour and then I call it again
Steve: later, you know, like technically I've called it again, but during that hour,
Steve: I may never know if I'm ever going to be getting it called again.
Steve: You know what I mean? And so there's also a little bit of like fuzziness there in the definition.
Steve: And it's much more straightforward with synchronous code because there's no like,
Steve: I mean, you might have a sleep call or something, but just like the point is
Steve: you can always statically know,
Steve: like okay eventually this point is going to happen where it's never going to
Steve: be used again and in async stuff that can be a little more tricky and or like
Steve: dynamic but you know also this is an area i work on specifically so i can only
Steve: really give a higher higher level answer on that yeah.
Matthias: I think the slippery slope here is that when you go forward with that proposal
Matthias: or you go forward with that idea of introducing async equivalents of sync traits,
Matthias: then you end up maybe replicating parts of the standard library.
Matthias: We see something similar with read and async read or write and async write.
Steve: Yeah i mean the thing is
Steve: is that programmers love like dry
Steve: and they love trying to unify concepts that are
Steve: that seem similar but are different and i think
Steve: that's true but i also think that sometimes similar operations
Steve: are just fundamentally different and you can't
Steve: really abstract over them in the same way read async
Steve: read and async write are great examples where we have the sync write
Steve: and sync read apis but the
Steve: thing is is that like for synchronous read and write those apis
Steve: are very straightforward and have existed for a very long time and they like make
Steve: sense and there's only one implementation of them but one of the reasons why
Steve: async read and async write have not been stabilized yet in rust is because there
Steve: are at least two meaningfully different proposals on how to implement those
Steve: apis and so yeah well conceptually it's like just read but async i think that
Steve: but is really load-bearing and like you know like,
Steve: This is maybe stretching an analogy a little bit too far, but like addition
Steve: and subtraction are both conceptually the same thing at some high level.
Steve: Like they're both a binary operation with an operation in the middle.
Steve: And you want to be able to say like, well, you know, like maybe those are the same thing.
Steve: And so we should abstract over both of them. And that's how you get monoids.
Steve: But like that doesn't always mean that abstraction is useful necessarily,
Steve: because sometimes it can paper over the details that matter.
Steve: You know, like at a high level, the idea that like I'm applying a binary operation
Steve: to two things and it does something is like, sure, sometimes working at that
Steve: level of abstraction makes sense.
Steve: But sometimes you really care if it's actually addition or actually subtraction.
Steve: You know what I mean? And so like, I think that a lot of people want to reach
Steve: for the idea that we should inherently be able to abstract over sync and async.
Steve: But i think that there are different enough things with different enough
Steve: semantics that doing so at least
Steve: let's put it this way for rust specifically a
Steve: language that cares about the low-level implications of what
Steve: you're doing that you need to be able to integrate with an underlying system that
Steve: you like the details matter to you i think
Steve: that over abstracting these things is a mistake i
Steve: think that in a language like haskell as a reason arranged for mono a
Steve: second ago like in haskell you extract over sync and
Steve: async because conceptually they're the
Steve: same thing but haskell does not have the same performance requirements
Steve: and like low level requirements and little commitments that rust does
Steve: and so it can it can afford that abstraction because that abstraction costs
Steve: you something but in rust i think that the the details are significant enough
Steve: and the processes are significantly different enough that it is important and
Steve: meaningful to keep them separate and so i don't think it's inherently a bad
Steve: thing that you're sort of redoing some.
Steve: Async stuff or some sync stuff that's in the standard library because like also.
Steve: Drop conceptually makes sense i'm not sure if i fully think that async
Steve: drop specifically is a good idea but some sort of analog
Steve: or way to solve that problem makes sense i think that read and write are there
Steve: true but like that's like 95 of it there's like not a whole lot more you know
Steve: like Like we're not necessarily like the other stuff that we may like make a sync is like still like,
Steve: I don't know, it's not a brand new concept or like, I don't think it's the,
Steve: I don't think it's the end of the world. Let's put it that way.
Steve: Some duplication is totally fine and meaningful.
Steve: And sometimes, you know, abstractions are good because they let you work with
Steve: stuff at a high level, but you also need to be able to do stuff at a low level.
Steve: And I just like think that sometimes that trade-off is that you don't get to
Steve: use the high level abstractions. So, but this is definitely like a really big
Steve: debate in the Rust world right now.
Matthias: But at the same time, I couldn't agree more. You really summarized it super
Matthias: well, because this is one thing that I really love about Rust.
Matthias: It's explicitness and not having these leaky abstractions, because you make
Matthias: it explicit that there is a difference between those abstractions.
Matthias: And I really like that part.
Matthias: But at the same time, the async ecosystem is pretty new.
Matthias: The rest of Rust, the sync part, has matured in the last 10 years.
Matthias: And since I have you, I might as well just ask you, because you have a lot of
Matthias: experience, you've been in this community for a long time, what would you say
Matthias: is one big mistake that the Rust language made?
Matthias: Something in the standard library or anything regarding the syntax or its semantics
Matthias: that you would probably see as a historical mistake and you would want to change.
Steve: Yeah so i have a joke
Steve: answer that's funny and then i have a time
Steve: where i thought that happened but i was wrong and then i
Steve: definitely have a good answer for one that's real but i'm gonna tell the first
Steve: two anyway to like let me think about what it actually is because like
Steve: you know i want to make sure i'm getting a good answer so the joke answer i
Steve: always say is that string should have been named stir buff and like
Steve: that's and that's just like because like
Steve: we path and path buff like i think that naming and
Steve: the fact that it's like capitalist string like i feel like there'd
Steve: be a lot less like rust has 36 different string types if
Steve: we like acknowledged a little more that like it's
Steve: a buffer like it's a mutable growable string as opposed to some
Steve: other kind of string or whatever and so that's that's what i've always joked
Steve: is like i want rust 2.0 i don't want anything changed
Steve: except for i definitely like want stir buff
Steve: instead of string and so that's kind of like a silly silly answer there
Steve: was a time where i thought rust was making a huge mistake and i
Steve: was definitely wrong and that is the post fic post
Steve: fix await syntax so that was
Steve: like for people who weren't around when we were doing async awaits
Steve: design like there was a point in
Steve: time where it was a hugely contentious topic about how to write a wait like
Steve: should it be a prefix thing like javascript should it you know be some sort
Steve: of other thing or like should it be what it is today which is like the dot await
Steve: and i am really conservative when it comes to programming language design, actually.
Steve: And I kind of like, at the time, I was like, we have a lot of people who are
Steve: coming from JavaScript.
Steve: They, and JavaScript and C Sharp both do this prefix 08 shenanigans.
Steve: And I think that like, when it's not clear which way you should go,
Steve: you should choose to be conservative when it comes to language design.
Steve: In many other way of things in life, I do not believe this, but like when it
Steve: comes to programming language stuff, I think being conservative is generally pretty good.
Steve: And so I thought it was a really big mistake to add this weird syntax for async await.
Steve: However, after having had to write a whole ton of async code.
Steve: That would have been a huge mistake. And I'm really glad they did not listen
Steve: to me personally and went with the post fix away anyway, because it's just clearly
Steve: superior in every possible way.
Steve: And brilliantly enough, people figured out how to address that major concern
Steve: that I had, which is like what happens when JavaScript people come and write the wrong thing.
Steve: And that is diagnostics in the compiler.
Steve: So a really cool thing that Rust does that doesn't have to do is sometimes the
Steve: Rust compiler will parse code that's wrong to give you a great error message.
Steve: And so if you write, you know, instead of foo.bar.await, if you write await foo.bar,
Steve: the Rust compiler knows how to parse that, even though it's not write Rust,
Steve: just so it can deliberately say, hey, this is not how you write await,
Steve: you would write it like this, foo.bar.await, do that instead.
Steve: And that is like a really trivial way that like anyone who is like writing their
Steve: old style of, you know, of coming from another language, they won't get confused,
Steve: they will get helped into writing the correct thing.
Steve: And so that's an area where I definitely was like, at the time,
Steve: I was like on the wrong side.
Steve: And I think I've like thoroughly admitted that that's like a mistake.
Steve: I think my biggest things and started in sort of like where rust made mistakes
Steve: that I think is a little more serious or real is like there was a couple things that sort of like.
Steve: I don't want to say that no one cared about them at 1.0, but there's some things
Steve: that were sort of designed in a certain way that there was so much work to do
Steve: that they never really got fully, completely, totally thought out.
Steve: And I think the biggest one of those is the module system.
Steve: I really like Rust module system. I think a lot of stuff that it does is good,
Steve: but it is a common problem for people coming to Rust.
Steve: And it has been. And in Rust's 2018, we did some changes to some things that
Steve: made it a little bit easier to understand.
Steve: But it is the number one thing that when people read the book and they say,
Steve: I was confused by something, they say the module system. And I don't know what that is.
Steve: I don't have a constructive necessarily answer for how we should fix things
Steve: instead. But like, it wasn't, it was something that was kind of built a very,
Steve: very long time ago. I don't even remember by who initially.
Steve: And then, like, there was just so much other stuff to do leading up to Rust 1.0.
Steve: There was never a moment where it was like, okay, we need to really make sure
Steve: to think through, is this how we want the module system to work?
Steve: And then even in 2018 when we did there was
Steve: a bunch of like legacy constraints of like okay we want to
Steve: change how some of this stuff works but we have to make sure
Steve: that it's not too different for all the people that currently know rust because
Steve: that would be like very very bad if there was two completely totally
Steve: different systems and so some of that was kind of like tied
Steve: up that way and i think in general it's not
Steve: really just the module system but also like name resolution in general
Steve: which is like when you type an identifier how does rust
Steve: determine what identifier you mean the module system is like part of that whole
Steve: situation and a lot of that is kind of just like was made and was never really
Steve: it never really had the time to be like fully thought through and designed by
Steve: someone before 1.0 came out and i think there's a way that you could simplify,
Steve: a lot of that stuff like.
Steve: This gets into stuff that like most Rust programmers never even really think
Steve: about, but like types and modules are two separate like namespaces.
Steve: And so you could have a module with the same name as a type and the Rust language
Steve: will disambiguate between the two and it's totally fine.
Steve: And there's like, I think there's another one too that I'm totally drawing a
Steve: blank on that does like three different sort of like versions of namespaces
Steve: and they can all be like overloaded.
Steve: And this like leads to like very confusing outcomes if people name things in
Steve: sort of strange ways. and like that's all complexity that the compiler has to
Steve: deal with when trying to look up and figure out a name is like looking through all those things.
Steve: And I think all of that probably could have been pretty radically simplified
Steve: and is something that's like basically kind of impossible to sort of like deal
Steve: with at this point. Like it's just kind of like baked in.
Steve: Macros also are kind of like Nick Cameron really wanted to do like a rebuild of macros.
Steve: And so like that's why before Rust 1.0 We renamed the macro keyword into
Steve: macro rules and reserved the macro keyword because the idea would be that someday
Steve: there would be a new macro system that would be built.
Steve: And that's just never happened.
Steve: That all disappeared at some point. And so that's definitely a thing that's kind of similar.
Steve: Although I don't think most macros are inherently bad, but there definitely
Steve: are other options that could have been considered a little more.
Steve: And like proc macros are like sort of wonderful
Steve: but they are also really complicated
Steve: and have some like really weird technical angles
Steve: like you have to make a proc macro its own crate why do
Steve: you have to do that well because proc macros are kind
Steve: of like an out growing of compiler
Steve: extensions which existed a long time ago and like why
Steve: were compiler extensions the way they were well back in the
Steve: day it was basically like you can ask the compiler to just like open up
Steve: a library you write and mess with a compiler internal data structures
Steve: and produce something and like proc macros are a crazy
Steve: strong aspect of the rust ecosystem but
Steve: that doesn't mean the feature had to be implemented the way that it's implemented and
Steve: now that's like kind of just there forever and also it's good
Steve: enough that there's nobody who's like truly invested in
Steve: making a better one and so you know like
Steve: that whole area yeah again is like sort of a similar like
Steve: you know is it the worst no could it have been better yes
Steve: and i think those are kind of some things that like a rust plus plus could like
Steve: meaningfully address i don't think they're necessarily enough to justify having
Steve: a whole separate new language but they're definitely areas that i think that
Steve: are big warts that you know maybe could have like been fixed but just like aren't
Steve: really possible to do now.
Matthias: All right, that was a really nice side quest into the Rust specifics.
Matthias: Before we close, I also wanted to quickly come back to Oxide because we covered
Matthias: the hardware, the firmware, we covered Hubris, we went up the stack to illumos and bhyve.
Matthias: But I wonder what's above that, the user-facing things, the interface with the
Matthias: system. Can you talk a little bit about that?
Steve: Yeah. So we're a super big believer in the OpenAPI spec.
Steve: Is it perfect? No. Is it good enough? Yes.
Steve: And so the way that you actually like interact with the Oxide API.
Steve: Like the Oxide rack is via an API.
Steve: So the same system that's running the sort of like control plane that's determining,
Steve: you know, what VMs get scheduled where and things, that's exposing an HTTP API to users.
Steve: And so what that means is, not only can you use client libraries to say,
Steve: you know, like spin up a VM, like you can use the Oxide CLI and be like,
Steve: give me, you know, a VM that looks like this, and it will like make one does
Steve: that by making HTTP request.
Steve: But also means we have a web console, which is just like the AWS console or
Steve: anything else where it's a website you can load up, and you can click buttons
Steve: and, you know, manage your rack that way.
Steve: And all that is built off of this idea of like the rack exposing an open API
Steve: definition, and then be able to generate clients on top of it.
Steve: And so we are using TypeScript for the sort of like front end of the website of the console.
Steve: And we're not using like Rust and Wasm or any of the sort of front end Rust technologies.
Steve: And that's basically like TypeScript gives us like 85% of what we would want
Steve: out of Rust over JavaScript.
Steve: Like having strong type stuff is like really, really important to us.
Steve: And, you know, all those kinds of things.
Steve: And the Rust front-end web ecosystem is even younger than many other parts of
Steve: the Rust ecosystems. And so, you know, I definitely don't want to say it's like
Steve: not usable at all, but we didn't necessarily want to, you know,
Steve: bet on something that was like that young while we're already doing so many other things.
Steve: And plus, you know, a lot of the people that do front-end web stuff are already
Steve: familiar with TypeScript. And so it's easier for them to kind of like pick up
Steve: that sort of stuff necessarily, although many of them also do a little bit of
Steve: Rust on the side because, you know, they interact with other parts of the stack too. Yeah.
Steve: This has kind of become a very common sort of pattern for building applications
Steve: at Oxide in general is like a Rust server backend, and then a TypeScript front
Steve: end with an API layer expressly in the middle.
Steve: And so we actually wrote a server framework called Dropshot that,
Steve: you know, at the time, like, you know, a lot of people use, you know,
Steve: Axum or, you know, what other other sort of web framework in Rust.
Steve: And we kind of wrote our own specifically because of the time that we were building
Steve: it, there was not a lot of stuff that had deep OpenAPI integration.
Steve: And so what this means is a part that stinks about OpenAPI is trying to write
Steve: the definition by hand. So we just don't do that.
Steve: So for example, our drop shot, our server framework, you write the endpoints
Steve: yourself, and then you can pass a command line flag to the server that says,
Steve: hey, please generate an OpenAPI document for me.
Steve: And it will look at all the code that you wrote to write all your endpoints,
Steve: and it will generate the full OpenAPI specification document for you.
Steve: So you don't need to write it by hand.
Steve: And then we have a TypeScript generator that's able to read in an OpenAPI document
Steve: and spit out a client library, not just for TypeScript, but on the web context is the most important.
Steve: And spit out a TypeScript library that's able to know how to interface with
Steve: things because of that OpenAPI document.
Steve: And that means that I get, in practice when I'm building web stuff at Oxide,
Steve: I write my server-side definition.
Steve: I say hey please generate stuff and
Steve: regenerate the client in typescript and when i switch back over
Steve: to my typescript file it'll give me a type error that says hey
Steve: you're not passing this correctly or whatever and so i get full type
Steve: safety the whole way up through the stack which is
Steve: really really cool and useful and so yeah so we've been very happy with you
Steve: know using typescript and that's just like one area you know of the product
Steve: where we're not fully using rust for everything but that's like based on the
Steve: pragmatic decision to like you know engage with that ecosystem deeply too so
Steve: yeah it's been very very nice people.
Matthias: Who are not familiar with open api might think oh that's just a lot of extra
Matthias: work that you do on top of what you already do which is an infinite number of
Matthias: other yaks to shave so i wonder what are the practical benefits of having an
Matthias: open api spec what can you do with that.
Steve: Yeah, so some of it is just like, very straightforward is like type safety is
Steve: cool. Like we like type safety, it helps it doesn't solve every problem.
Steve: But you know, also some of it is like we, people need to, you know,
Steve: write applications against our API, like everybody and using a bunch of different libraries.
Steve: And so I think for the non TypeScript side, I think that there's sort of like
Steve: the TypeScript one is interesting for its own reasons.
Steve: And the non TypeScript ones are interesting for their own reasons.
Steve: So I'm gonna talk about those ones first, because there's like simpler.
Steve: So, you know, in general, you know, you're going to want people that
Steve: definitely want go they definitely want rust and
Steve: then there's maybe some other stuff but like those are the two big ones but like
Steve: we do need to support a bunch of different languages that people are going to want to
Steve: write applications against and you know every company has
Steve: their own stack and so we can't necessarily guarantee that like you know maybe
Steve: they're a java shop and so they really want a java library okay so one of the
Steve: things that the benefit gets us to be able to like meet customers where they
Steve: are but like a lot of the i'm gonna say non-type script APIs are not very interesting
Steve: because it's just a general REST client, right? You're making HTTP calls.
Steve: There's nothing kind of like super novel there.
Steve: The benefit is in being able to not have to handwrite every single one in every
Steve: single language and have something reasonable kind of pop out in each language.
Steve: I mean, obviously, handwriting ones can be nicer than sort of the more generic
Steve: ones. But we're also not using the generic opening API tooling.
Steve: We wrote our own generator to sort of generate ones that only need the features that we use.
Steve: And so therefore, produce something that's a little bit nicer,
Steve: but like, okay, it's a slightly nicer rest API is not that interesting.
Steve: One of the things that I think is really cool on the TypeScript and like front-end side.
Steve: Is if you go to the console repo, which is what we call the front end to this
Steve: thing, the web console, if you go to that on GitHub, there'll actually be a
Steve: link to a little Vercel thing that lets you play with the console in your browser.
Steve: And the reason that that works is that we can use the OpenAPI definition to
Steve: also generate a mock server and then run it in a web worker in the browser.
Steve: And so you're able to play around and say like, spin up a server.
Steve: And then it will like pretend to spin up a server by running that in the mock worker in your browser.
Steve: And then when you go to list all servers, it'll have the one that you spun up.
Steve: And when you say spin up another one, even though there's fake page loads in
Steve: there, basically, it's able to remember all that stuff and you're able to get
Steve: some very basic logic like that kind of working.
Steve: And I just think that's such a cool demo to be able to actually play around
Steve: and you can see what is going on or what it feels like to do it without needing
Steve: to spin up a backend at all.
Steve: And it's only possible, it's kind of funny way earlier, I was talking about
Steve: how in the firmware level,
Steve: having interface layers is like no good but like
Steve: here you know on the very front end at the very highest levels
Steve: that is we're actually interfacing with the external world is
Steve: not something we can control and something where other people want
Steve: to use other technologies to interact with us and so at that level it
Steve: is worth it to actually do the work to sort of put
Steve: in that kind of you know universal interface as opposed
Steve: to something that's not and so you know open api
Steve: like i said before is it the best api description language that
Steve: ever existed no but it is one that a lot of people use and know
Steve: and it works well enough so you know that's like a good example of i think us
Steve: making a totally different trade-off at a different part of the stack where
Steve: you know internally speaking we don't need to collaborate with folks and so
Steve: those layers don't make sense but on the highest levels externally focusing
Steve: we do need to do that and so it's worth putting the time and effort in at that layer that's.
Matthias: So incredible to even think about having
Matthias: such a use case because i don't think you had that in mind from the very beginning
Matthias: it just evolved over time and then you had the ability to put that test server
Matthias: on the web and it must have been a really magical moment when you as a company
Matthias: realized that you could do that.
Steve: Yeah i think that happened before i showed up personally but definitely when
Steve: i was introduced to it when i got shown it i was like this is so cool i want
Steve: to talk about this all the time because i think it's just a really really neat
Steve: way of doing things for true and.
Matthias: No one else does it these are some of the things that are so exceptional,
Matthias: you rarely hear about them.
Matthias: It's another thing that I think you do, which not many other companies do necessarily,
Matthias: is to be very open about discussions and,
Matthias: wiring in the community you have this rfd process i think it stands for request for discussion.
Steve: Yeah and.
Matthias: It reminded me of ross rfc process is it modeled after that.
Steve: So it definitely takes some influence from that but also it's my understanding
Steve: that joyant used rfds to talk about stuff internally as well and so like i mean
Steve: both things were definitely inspired by the ietf rfc process Like, I think just in general,
Steve: the idea that there is like a written document that you come to consensus around
Steve: and then use that to move forward is like the high order bit.
Steve: And so, you know, I definitely think it takes some inspiration from a bunch of different places.
Steve: But, you know, at its core, all those processes are about the same sort of thing
Steve: of like when you need to get a lot of people on the same page to do something,
Steve: you know, you need some way to achieve consensus.
Steve: And we really value the written word at Oxide because that's the thing that truly scales.
Steve: You know if you have a meeting you know you can only have a meeting with so
Steve: many people before it really falls apart and doesn't work but a bunch of different
Steve: people commenting on some text.
Steve: Is you know able to not be at the same time it's
Steve: asynchronous instead of synchronous so you know we're a distributed
Steve: company and we have people that live all over the place and so you know
Steve: that like matters a lot and you know it lets
Steve: people work at their own pace you know like in a
Steve: synchronous meeting if i want to think about something you know
Steve: give it a half an hour's worth a thought that's not really feasible you
Steve: know in a meeting with 10 people i can't just be like well i want to sit here
Steve: and think about this for an hour before i say what i want to say you know it
Steve: doesn't work because you're wasting everyone's time whereas with an rfd you're
Steve: able to like say okay i'm going to sit with this and like really think about
Steve: it and you know you're not blocking anyone else and so i think there's a lot
Steve: of advantages to doing stuff that way for sure but yeah i.
Matthias: Think we covered a lot of ground going from hardware all the way up to the software
Matthias: interface and web applications it has been a crazy tour the one thing that i
Matthias: wonder about and maybe some listeners might also be curious about that if oxide started in 2024 and,
Matthias: the programming language landscape has changed a bit would you write oxide in zig now.
Steve: No and there's
Steve: a couple reasons for that i really like zig conceptually i've
Steve: known andrew for many years i consider him a friend he himself
Steve: says don't use zig for production yet like
Steve: it is still changing massively all the
Steve: time there is like you have like tigerbeetle and
Steve: you have bun and i think there's maybe one other company but
Steve: that's like it and at this point you
Steve: know rust is still the like safe choice that
Steve: has been used in you know it's like millions and millions
Steve: of lines of rust in production that you know everyone touches
Steve: like every day like even just cloud flare 10% of
Steve: the internet hits rust code all the time and so
Steve: you know i think also like there's lots
Steve: of cool things about zig that i wish rust would steal honestly but
Steve: i think the for me personally the
Steve: ironclad memory safety guarantee that rust has versus the
Steve: like we probably fix most memory safety bugs being
Steve: what zig has is just like a a deep
Steve: philosophical difference that i think for me personally 80% of the problem being
Steve: solved is not worth it but 100% of the problem solved is and obviously rust
Steve: is more like 99.99% of the time but like i still think it's a very significant
Steve: advantage and there Whereas more and more,
Steve: we are finding that 100% memory safe by default with an escape patch is the
Steve: correct choice for languages.
Steve: And so obviously, I would say that because I'm a Rust person and a Zig person,
Steve: but I do think that's meaningful.
Steve: And so I do think that the exact same choices would still be made today.
Steve: Because while it is true that there is Zig and there's a couple other languages
Steve: that are people working on upcoming, they're still very young and they're still
Steve: very early. and they are not attempting to solve memory safety fully in the same way that Rust is.
Matthias: I think a lot of people are really excited about Oxide because you do so many things right.
Matthias: You have vertical integration, you have top-notch branding, you have your own
Matthias: podcast, you wire in the community, the RFD process.
Matthias: Everything is done well and it's done with purpose. and this other thing that you,
Matthias: always do, and I think that's amazing, is to contribute back to the open source community.
Matthias: A lot of the tools that we talked about today are open source,
Matthias: so people could check out the source code.
Matthias: Can you maybe list a few projects that are open source?
Matthias: And in general, what is the methodology at Oxide to decide how and when something can be open sourced?
Steve: Yeah so hilariously i co-wrote
Steve: an rfd with Bryan on open source policy at
Steve: oxide and hilariously i've forgotten to like
Steve: mark that as completed and i think it's one of the ones that's not public so
Steve: i should also get on making that public at some point because it's
Steve: kind of funny that it itself is not but essentially the default
Steve: position of oxide is that everything should be
Steve: open sourced to the extent that we can possibly open source it
Steve: and there's a couple different reasons for that so because
Steve: we also have a little bit of an interesting relationship with open source stuff sometimes
Steve: like like so because we
Steve: are a company and we're doing so much we often
Steve: don't have time to like community build around
Steve: our projects so i mentioned drop shot before like
Steve: it is open source it is on github we do accept pull
Steve: requests but we're not like trying to make it the
Steve: rails of rust because like we don't really have the time to
Steve: run a full community managed product it's more like this
Steve: is a thing that we built that's useful for us and if it's useful for you
Steve: too then that's great like hubris for example like we
Steve: basically we we take pull requests in the
Steve: sense they are open and we'll accept them sometimes but like we also can't really
Steve: even have the time to review like if someone are to refactor a major component
Steve: or something like that's just like not a thing that we could accept because
Steve: we really need to stay focused on like what's good for us and so there's this
Steve: interesting balance between like we want to make stuff available.
Steve: And sometimes libraries are easier to share with people in a way that would
Steve: make sense conceptually.
Steve: But like a lot of our open source is kind of like a, it's not a source dump.
Steve: It's not like we're just throwing it over the wall, but we just like really
Steve: can't accept a ton of external contributions.
Steve: But it's really important to us that it is open sourced, because like,
Steve: one of the big problems, as I mentioned a long time ago, in sort of the firmware
Steve: part of this discussion is that like, you may not even know what is running on your computer.
Steve: If you buy a computer from another vendor, there's like whole operating systems
Steve: hiding in the nooks and crannies in your computer.
Steve: And we think it's really, really important that when you buy a server from us,
Steve: you are buying the server, like from us, and that like, it is yours.
Steve: And therefore you get to know what is running on it because it's
Steve: your hardware like we shouldn't have secret computer stuff
Steve: running on the hardware that you purchased from us and
Steve: that also includes like we don't do software licensing fees
Steve: like if you if you buy a server from dell you're going
Steve: to buy the hardware but you're also going to license the software and that
Steve: means it's an ongoing cost so did you really buy the
Steve: computer or are you just like paying a big
Steve: up front and you're renting some of it you know what i mean and so that
Steve: means that like so a thing that happened recently we
Steve: use cockroach db in the control plane and cockroach
Steve: announced that they're moving from the bsl the business source license to a
Steve: proprietary license they're like going sort of closed source again source and
Steve: yeah source available and so like until a lot of people said oh what that mean
Steve: you're going to do and we're like well we're sticking with the last version
Steve: that was apache licensed and we're just going to keep doing that.
Steve: And people were like, oh, why didn't you negotiate at a license with Upstream?
Steve: And it's like, maybe, you know, you could have paid for it, like,
Steve: you know, get a discount or whatever.
Steve: And it's like, well, you know, I mean, first of all, it's not like we didn't
Steve: talk to them at all, but like the idea that we would be paying a per rack license
Steve: fee for that doesn't make sense because our customers own the hardware after we sell it to them.
Steve: So what, are they going to pay Cockroach, you know, a licensing fee?
Steve: Like that wouldn't be fair on their behalf.
Steve: And so like, so that means we really need to like,
Steve: like open sourcing the software in general
Steve: means that our interests are aligned with our customers interests which is
Steve: that like we're trying to like sell you a computer i
Steve: keep always going back to that because it's just so funny because like on some level
Steve: access business is like so straightforward in a world where
Steve: so many tech companies businesses models are really complicated like we
Steve: really want to sell you a computer and then it's your computer obviously you
Steve: know support contracts are an ongoing thing that we you know
Steve: do and like stuff like that but like it's just true it's really
Steve: important that you should know what's running on the computer that you get and
Steve: that's that's true in like a an ethical sense but it's
Steve: also just true in like a security sense it's
Steve: important to us we feel like that if you you know want to make sure that you
Steve: know nobody's we you know if you install this this rack in your data center
Steve: it's important that nobody's trying to steal your stuff you know what i mean
Steve: like privacy is probably pretty important to you if you're buying your own computers
Steve: and racking them in your own data center,
Steve: you care about making sure that you're the only one that gets to know what's running on your stuff.
Steve: And so being auditable is a really important part of that. And so being open
Steve: source is not necessarily a precondition to being audible, but it certainly
Steve: makes being auditable a lot more easy because it means you can literally see the code. Now.
Steve: There still are a couple of binary blobs that we have to do.
Steve: Like while we did rewrite a lot of the firmware, there still are occasionally
Steve: bits of the firmware that we can't actually fully open source.
Steve: And occasionally there's stuff that, you know, maybe has an NDA or whatever.
Steve: So I'm not going to say it's fully 100% there, but in general,
Steve: like we try to open source everything that we do to the greatest extent possible.
Steve: And like, we've also found that there's a benefit there because sometimes it's
Steve: just more annoying for stuff to be closed source.
Steve: Like if you've ever had to deal with cargo trying to depend on Git dependencies
Steve: because you depend on some sort of closed source library that you have to deal
Steve: with, like it's kind of a pain in the butt, right?
Steve: And so like, it's just easier, even if you never intend for anyone else to actually
Steve: read the code, it's just easier to be like, hey, this is actually open source.
Steve: And so we use the MPL by default, which is a really interesting license.
Steve: It's not, you know, most stuff in the Rust world is Apache 2 is less MIT licensed.
Steve: And the MPL is a Mozilla public license.
Steve: And it basically is like a weird hybrid between the MIT and the GPL.
Steve: And so it basically is just like, you need to be still open source,
Steve: but like when you use MPL based code, it's sort of a still open source,
Steve: but it's the file level instead of the like project level.
Steve: And so we feel like there's a really nice trade off between copyleft and totally,
Steve: you know, the more Libre licenses.
Steve: And so that's sort of our default choice. But if you're integrating with a code
Steve: base that already has made some sort of choice there, then we need to stick
Steve: with the same license. So we don't like mandate that everything has the same
Steve: exact license, but we do feel like it's really important in many ways.
Steve: And so that's something that we try to do, even if we're not necessarily on
Steve: the big community building ends of open source stuff, because we're doing so much.
Steve: We just don't have the time, but we do think it's like, it's like important
Steve: to align our business interests with our customers' interests.
Steve: And that's open source is like a great way to do that.
Matthias: If I had to play the cynic for a moment, I would ask, couldn't you also change
Matthias: the license like Cockroach did and lock in your users?
Steve: In theory but like we also don't really do
Steve: copyright assignments at all so like you
Steve: know we do change some of that stuff like but or
Steve: like we don't have the ability if we've accepted stuff from other people
Steve: or if we're building on another project like we couldn't necessarily change it
Steve: but i mean you know i think a
Steve: more cynical question would be like well why does
Steve: it matter because it's custom hardware anyway so who's going to be running that
Steve: on a different computer you know and hilariously a lot of it is actually pretty
Steve: easy to run on other stuff but but yeah no i mean we could do it after anyone
Steve: could make any decision at any time
Steve: but you know currently you know that's that's where we're at so that's.
Matthias: Very fair and you never know where this approach leads you
Matthias: because maybe someone finds a very
Matthias: interesting way to use that source code in a completely different context at
Matthias: some point maybe 10 20 years down the road and also this is another thing that
Matthias: i was wondering about what makes you confident that rust will be around in 10
Matthias: years or might even be relevant or,
Matthias: I don't know. Let's think about 20 or 30 years down the road.
Matthias: A lot of languages, they go away over time.
Steve: Yeah. I think that a lot of people don't realize how much production Rust is
Steve: out there and how many companies truly depend on it.
Steve: And if the idea, if Rust were to suddenly implode, how people wouldn't step up to fix it.
Steve: Just meta alone has, I think,
Steve: like on the order of 10 million lines of rust it's like i don't
Steve: think it's it's not 100 million lines but i think it's more than single digit
Steve: millions it's like between one and 10 million
Steve: lines of rust you know amazon like rust
Steve: is now involved in like s3 aws like aws ec2 uh tons of different aws servers
Steve: are all like have rust at really key points i mean rust is in the windows kernel
Steve: already well people were talking about like oh is rust mature enough for the
Steve: linux kernel which, I mean,
Steve: Asahi Linux has already built graphics drivers on it.
Steve: And while it's not in the upstream kernel fully yet, and while there's been
Steve: a little bit of some discussion about that, people are like,
Steve: is it actually mature there?
Steve: Like it's in the Windows kernel. Like I'm talking to you on a Windows machine.
Steve: There is Rust code running in my kernel right now. It's just a little bit,
Steve: but like it's expanding.
Steve: Microsoft is rewriting some legacy Windows stuff in Rust, like GDI,
Steve: the graphics drawing interface.
Steve: They have like a port of that to Rust stuff. And so,
Steve: you know there's just there are millions upon millions of
Steve: rust lines of rust out there and being used for real important
Steve: things if something were to happen to the
Steve: rust team like it's also worth thinking about what would it
Steve: mean for rust to die okay that would mean that the rust team
Steve: would somehow not exist anymore but the
Steve: code base would still exist of rust itself and so
Steve: like at that point those companies it would be an existential
Steve: threat to their business for that technology to implode and so they
Steve: would have to come up with some alternate way of making that
Steve: work because like you know it can't just
Steve: stop at this point like rust has reached escape velocity
Steve: and is a language that like will survive now is
Steve: it will survive in the sense of like vb.net where
Steve: like it technically exists but not a lot of people use it but like
Steve: or is it a cobalt where like it's used for some things but
Steve: like not for anything else i have no idea if it's that kind of legacy or is
Steve: it like a c/c++ where it's used for smaller and smaller amounts of things
Steve: but still for very important things i don't know but it's definitely past the
Steve: point where it will just like disappear at some point because it is just used
Steve: in too many things that are too meaningful like like the united states government is talking about.
Steve: You know, preferring Rust over other languages when procuring things like,
Steve: you know, and is that a thing I feel great about?
Steve: I don't know. I feel complicated about it, but just like, it's at that level of maturity.
Steve: And so I think a lot of the people who sort of think Rust is a fad are just
Steve: like uninformed about the degree to which Rust is used in industry for real,
Steve: meaningful production applications.
Steve: It is absolutely going to live on.
Steve: I don't know if it's going to be 30 years or 100 years, but it's definitely going to be 30 years.
Steve: And maybe in 30 years there'll be the rust plus plus and cool kids will be using
Steve: that or whatever but there still will be jobs for rust programmers in the same
Steve: way that you know python super super hot now started in like you know 20 30
Steve: years ago itself so you know similar i think that's like kind of where rust
Steve: will find itself in the future yeah.
Matthias: That was in 91 or 92 somewhere around that time.
Steve: Yeah i think so yeah before.
Matthias: We close now you have the opportunity to mention any of the tools that you want
Matthias: people to check out can be tools from oxide can be external open source tools
Matthias: that people might or might not know and things that you find interesting in the rust ecosystem.
Steve: Yeah the two things that i'm most interested in
Steve: at the moment are not oxide projects but they are both written
Steve: and rust the first one is buck2 which is
Steve: a build system from facebook slash meta so buck
Steve: one is kind of like without getting the full history
Steve: it's kind of similar to basil or blaze if you're familiar with those from google
Steve: but at one point because i was doing as i mentioned earlier the like build system
Steve: stuff for hubris i got in my head like okay if i was gonna make a build system
Steve: from scratch what would i do and i read a bunch of papers and i learned about
Steve: the space. And I said, okay, cool.
Steve: I would want this general design. And then I was like, okay,
Steve: well, maybe I'll start writing that in Rust.
Steve: And so I started toying around with the idea of doing it. And then I learned that Buck 2 existed.
Steve: And what I found was, is it was all the people from all the papers that I read
Steve: doing the design I wanted to build and writing it in Rust.
Steve: So now what I'll say is that it's a little hard to use Buck if you're not familiar
Steve: with those tools already.
Steve: I think the developer documentation, the introductory documentation is like not very great.
Steve: And I'm not going to say that it's a flawless tool, but it is one that I'm very
Steve: interested in and learning more about.
Steve: I still have not used it a whole ton. And like I said, I haven't ported any
Steve: of my work projects over to it yet.
Steve: We are using it at Oxide on a project for FPGA shenanigans, but it's definitely
Steve: like a tool that I'm very interested in because.
Steve: Tools like Cargo and NPM and RubyGems are kind of build tools that are good
Steve: for the small, easy cases, as we talked about.
Steve: Like if you're on the straightforward path, they work really,
Steve: really well. And tools like,
Steve: Buck and Bazel are really good for the monorepo, Google style,
Steve: like everything in your whole company, hundreds of millions of lines of source
Steve: code are being built with these kind of things, but they're hard to use.
Steve: And so I'm really interested in how can we bridge these two worlds?
Steve: Is there a way for a build system to be good in the small and in the large?
Steve: And i think it's i think it's going to be better to bring the
Steve: like conceptually correct large build systems down to
Steve: be easier to use for the smaller cases than is
Steve: to scale the small easy to use build systems up
Steve: to the big cases but i don't think we're there yet so i want to i want to shout
Steve: it out as a tool i'm interested in i don't think it's perfect i'm not saying
Steve: you should switch to it tomorrow or anything but i do think it's interesting
Steve: sort of space to watch and the second tool is called jj which is from google
Steve: it is written in rust it is a new source control system that's Git compatible.
Steve: I've actually written a tutorial on it that I haven't published a super ton
Steve: yet, but is actually going to become the upstream tutorial sometime soon.
Steve: We've been talking about it with the team, but it is like a version control
Steve: system that is Git on the back end, but it's not Git on the front end.
Steve: And, you know, I love Git.
Steve: I've used Git for a very long time. I've loved Git CLI. Whenever people in the
Steve: past said Git CLI is bad, I would be like, I understand that you struggle with
Steve: it, but I do not. I think it's totally fine. I like it actually.
Steve: And JJ is the first time I've ever, like I haven't used Git in months at this
Steve: point because I just use JJ instead and it is both simpler and more powerful
Steve: than Git at the same time.
Steve: And like the C people, I think the Git people have been used to people saying
Steve: that to them for a long time.
Steve: And so I was very skeptical when I first heard about it, but JJ has a lot of
Steve: really interesting things going for it and it ends up being a smaller set of
Steve: primitives that are more orthogonal and that's why it ends up being like easier
Steve: and more powerful than Git at the same time.
Steve: So definitely check that out. And if you stock my GitHub, you can find my tutorial
Steve: or maybe someday if you're watching this, it will be the upstream tutorial.
Steve: But I definitely think it's really super powerful and as a tool I use every
Steve: day and love, even though it's a pre-release tool, you know,
Steve: it's got a lot going for it. I'm very, very excited about it.
Matthias: What I liked about JJ was the fact that you can avoid naming branches and you have a blog post on that.
Steve: Yeah, just wrote a blog post about that. Absolutely. That's another one of those
Steve: things where I never really understood. I was like, what do you mean?
Steve: How do you work with branches if you don't name them?
Steve: And then at some point you're like, oh, I haven't named a branch in like a long
Steve: time. I actually don't need to do that. It's cool. So yeah, absolutely.
Matthias: It was a very beautiful thought because you bridged the gap between version
Matthias: control, front-end and back-end technology with TypeScript and Rust. That was impressive.
Steve: Thank you.
Matthias: So many people asked me, and I almost forgot, one important thing.
Matthias: When will you start hiring in Europe?
Steve: So the funny thing I want to say is that, first of all, we would have been hiring
Steve: in Europe except for Britain left Europe. So we have some people in the UK.
Steve: But like we do, we are willing to hire people in Europe.
Steve: The only thing is that like, because the team is still so small,
Steve: we try to have working hours that overlaps to San Francisco.
Steve: And that's really hard for most people in Europe.
Steve: There are some people who definitely, you know, stay up late or wake up early
Steve: and like can make that work. But it's definitely a little bit of a challenge.
Steve: You know, as, as the company grows, we kind of like broaden and like make that
Steve: requirement a little less and less.
Steve: So you know it's definitely i would say that like europe is definitely not a
Steve: guaranteed no right now but you know it'll just become more and more common
Steve: as time goes by i don't have a great timeline for when exactly it is but that's
Steve: kind of like the overall philosophy is it's not it's not that europe is a no
Steve: it's the time zones are hard as programmers know so we'll get there in.
Matthias: Closing the traditional question do you have a message to the wider rust community
Matthias: anything that you want to share the stage is yours.
Steve: Yeah, I think that, you know, when I started using Rust, it was like 40 people
Steve: in an IRC room. And now it is, you know, millions of developers literally all across the globe.
Steve: And so growing is hard. And there's lots of changes that have happened.
Steve: If you've been around Rust for a long time, you've seen a lot of change.
Steve: I think change will continue to happen in the future.
Steve: You know the best thing that we can do is like continue to
Steve: write good software continue to try to treat each other well
Steve: and you know continue to like build cool
Steve: stuff in rust and share it with the world and you know not everybody's gonna
Steve: love rust and that's totally okay but i still think there's a lot of growth
Steve: in the rust world and i still think there's a lot of work to do and so i'm excited
Steve: for us to keep continuing to just like build cool stuff in rust and you know
Steve: just keep chugging along so yeah
Steve: i don't know that's that's i think what i have to say right now Steve.
Matthias: That was amazing. I have to thank you so much.
Matthias: If just a single listener starts learning Rust because of this,
Matthias: I think I've achieved my goal and you did the same a thousandfold.
Matthias: And I'm so happy to have you as an ambassador of the language I love.
Steve: Awesome. Thank you so much. That's very kind.
Matthias: Rust in Production is a podcast by Corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.