This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from corrode and today we are talking to Jeremy Soller from System76 about writing production-grade operating systems with Rust. Jeremy, thank you very much for being on the show today. Can you maybe introduce yourself to the audience and talk a little bit about what you do for work? Sure, thanks for having me. So I'm principle engineer at system 76 and I'm also the BDFL at Redox OS. So what I do for work at system 76 is I manage the team that writes both Pop!_OS and the cosmic desktop environment. And I also manage the firmware development. So we do our own firmware, core boot base firmware for our laptops. and I also take part in some other development tasks for other products that we sell. Like the launch keyboard, I've done firmware work for that and the desktops that we sell, I also do driver development. And at Redox OS, I created the project in 2015 and I have been maintaining it since as the lead maintainer. How did you get started at System76? What was your first contact point? Sure, so right around the end of 2016, I was working at my own company and we were having trouble finding investors and finding customers for a computer vision product that we were developing. And I was tired of doing sales work. I was really tired of it. And I started looking around for companies. that were hiring and I was especially happy to find one that was local and was interested in Linux because all of the development I had been doing had been based on Linux. System76 was the only company that was doing that in the Denver area, so I saw they had a job for kernel engineer. Now the requirements for this job were quite steep. This was for somebody who had an understanding of laptops and the hardware that they were selling and an understanding of Linux and those two things often do not go hand in hand. Most laptop development is targeting Windows as an operating system and most Linux developers hardly think about hardware. So finding somebody at the intersection and who was local I had been looking at this posting over the course of maybe a month or two and realized it was still there so nobody had gotten it so I sent an email and very soon after Carl, the CEO and owner of System76 sent me an email back with a code sample that they wanted me to create a simple project and so I finished it in about an hour and sent it back and I had passed that test and he had an interview with me and then on the phone and then I went to see the headquarters and I was very impressed with all the people there working there and it just felt like the exact kind of place where I wanted to work. So very quickly I was hired and I went in to start on January 17 in 2017. that was seven years ago, and it does not feel like seven years, but it really has been. from the outside at least it looks like you deal with everything from the hardware all the way up to the software and customer support. Yeah. So it's all from one vendor and this is really great if you deal with such a complex system like a computer. If you have someone that can customize these things you can see it with other vendors too and it's kind of a successful model. But I do wonder how does the integration work between the hardware and the software? It is a very dynamic process because our goal is to provide the latest possible hardware and there's stuff that is defined in stone. We have to offer the latest NVIDIA GPUs in our desktop line. That's just something that customers want. and it's for all kinds of purposes from machine learning to gaming. And we have to tailor our software around those needs. So first we take in those requirements. If there's a new Intel CPU, a new AMD or NVIDIA GPU, we have to offer it as soon as possible. That's how our model has been. Not every Linux company is targeting brand new hardware and I think there's a very big model for targeting older hardware as well because there's often fewer issues with Linux supporting such hardware. mhm. But the bleeding edge is a very important place because that is the place where most of the Windows market is focused. And we want to offer a pure competitor to Windows marketed laptops and desktops. and especially coming from vendors that offer support services and offer full installation. We're not talking about trying to attack markets where the chassis and components come from different vendors. We are, as you said before, trying to go all the way from the top down and providing a turnkey solution, a completely installed Linux computer. and so we have to choose components that customers want and those are new components and then we have to evaluate how they're working with the with how the length ecosystem is. And before Pop!_OS this usually involved taking Ubuntu and the choice for Ubuntu is something Carl has gone into detail before but basically during the start of the company in 2005. or earlier, he was experimenting with which operating system to use and ended up with Ubuntu as the one to offer. We have to evaluate how it works with the hardware and often there are issues, especially with brand new hardware. For instance, we recently had a new version of the Lemur. It has Meteor Lake Intel CPUs, and those CPUs don't have graphical support at all on kernels older than the 6.8 kernel. Well, a lot of operating systems, a lot of length distributions, are not offering the 6.8 kernel. If you want that, your options are maybe Fedora, Arch, and open SUSE Tumbleweed, ones that are targeting bleeding edge software. but a lot of customers are asking for something compatible with packages they're built for Ubuntu LTS. So there's a very big divide here between what customers want and what the market is offering. The customers especially in machine learning are really targeting Ubuntu LTS but the hardware The upstream hardware development is not targeting those distributions, and the distribution maintainers themselves are not targeting that hardware. It's a very strange environment. So we have to backport things from upstream. We backport the Linux kernel, so we always have a very new Linux kernel on Pop!_OS. We backport Mesa. We backport the NVIDIA drivers, and those are critical components to ensure that that base of being able to sell the latest hardware is accomplished. But beyond that, there's still a lot more of other things that we have to decide on. We pick and choose, for example, Wi-Fi components and storage disks, although that hasn't been an issue for a while, based on how well they perform with an up-to-date Linux distribution. So we won't offer certain Wi-Fi chips because they simply don't work well with Linux. That often ends up with us selling Intel Wi-Fi for most of our products. What was that? Part of your job responsibility when you started at System76 in 2017. Maybe you had to write custom integrations or custom software for these things or how did that work? Yes. We would often have things that were not working properly with Linux and we'd have to create patches for them. Audio was a particularly difficult one because audio, the chipsets themselves are often very, very It's motherboard specific, how they're hooked up. So you could have the same codec, the same audio chipset being used on two motherboards, but hooked up completely differently. And the Linux kernel has no way to know this. So there is a very large set of quirks for those kinds of devices. The very first project I worked on at System 76 was trying to fix up a laptop's SPEDIF output. So the digital output wasn't working properly and I had to go through a very large set of issues and go through the length's kernel and go through a whole bunch of things I never knew existed until that time. to figure out how to fix it. And in the end we had to have both a fix inside the kernel and another fix outside the kernel. The kernel fix was to add some quirks for that audio codec with that specific motherboard. And then the system76 driver project which is a large user space piece of Python basically. would have to do some things with ALSA externally to the kernel to make sure that hot plug of digital audio devices was working properly. And the thing was you don't have a lot of choices when you're selecting from motherboards that are manufactured by other people and you're using firmware that's manufactured by other people. We're trying to find a laptop that is a white box option and what that means is that it doesn't have a logo on it. It comes from a third party manufacturer and we are going to produce the drivers for that device. Now we have a lot more flexibility. Having got into the firmware level, we're able to fix these things at a lower level and having schematics for all the devices and also starting our own project of doing in-house development of motherboards. These are things that have allowed us to reduce that complexity and solve more of these issues at a lower level. I would assume that a lot of these problems nowadays are also tackled with Rust, but I wonder when was the first time that System76 used Rust for any sort of development? Yeah, so I started in 2017 in January, and I was using Rust for Redox OS, and I had started using Rust there in 2015, and when I came in, most of the projects were written in Python. and I have a particular disdain for Paison. It's not a bad language but it is an interpreted language and it is dynamically typed and those two things are quite painful for me. And I wanted to have something, anything statically typed, and I noticed that there was a pretty big gap in what we were offering. The firmware updates that we were offering were done manually so we would provide a zip file to customers and they would have to put the zip file on a USB drive and boot from it. And so my first Rust project was started about two months after I started at System76 I said, I'm going to automate this. I'm going to take our firmware updates that are all over the board and messy, just all over the place, and I'm going to consolidate them into a single project. And I'm going to use Rust. So I started the System76 firmware update project. and there's two sides of this, there's the user space code that's written in Rust that manages downloading the firmware from our servers and there's the firmware update GUI that's written in Rust and actually compiles for UEFI and this was a particularly new thing at that time. I based it on another project that was written in Rust and was MIT license. So they had gotten up to the point where they could show some text on a screen if I remember correctly from a UEFI executable that was compiled in Rust. The difficulty that I had at that time was figuring out how to link stuff in Rust. And after that, I was able to take it from there because I had a pretty good understanding of the rest of doing things in Rust. And so I wrote a GOP that's a graphic object protocol or something like that, a piece of code that would let me write to the frame buffer. And I ported over actually the orbital client library. For Redox, I ported it to UEFI, so it could write nice font and other items to the screen. So the firmware updater was able to load off of disk in UEFI mode and then run some firmware executables and generate nice output to the frame buffer. And it was a big sharing moment too with Redox because the Redox bootloader at the same time was sharing the same code. So I was able to do UEFI work which was very new in Rust. and share code between them. And the result of that was the creation of three different UEFI crates for Redox that are now used in a number of places, both in System76 and in Redox. So we touched on redox quite a few times already but maybe now is a good time to introduce it properly. Maybe give it a good go. So what is redox? What's the big idea behind it? Maybe let's start from the beginning. Sure. Well, the beginning is I was working on computer vision software, and it was in 2014. I was using C++, a lot of C++, and I was writing code for Qt, and I was running Kubuntu. I was very invested in C++, and OpenCV pushes the use of C++. That's a computer vision toolkit. and so I was a C++ programmer and I was writing things that were very strange to write in C++. I wrote something called a lock pointer and what it was was a shared pointer that tried to wrap a mutex and C++ is very good for having RAII which is resource acquisition is initialization. I really like that feature. and what it means is that you can define a constructor and a deconstructor for a C++ object. and so you can be assured that it will release its resources or it will run some code to do that after it falls out of scope and so I wrote this lock pointer abstraction for use in a very concurrent project because doing computer vision I was trying to spread out as many tasks on different cores as possible and I was sharing images between them and running different detectors on the same image and it was quite a complex concurrent project. I kept running into segfaults and I'm like, what the hell is going on? I wrote this abstraction that is supposed to verifiably tell me if I acquire this object, RAII, then I have this lock because I wrote the code to do that in the constructor. And if I release this object because it goes out of scope, then I release the lock. At no point should I be able to access the object when it's not locked. and so I started writing a description of changes I would make to C++ or links that I would create and I called it safe object language which happens to be an acronym that matches the first three letters of my last name. I was pretty all about doing things with my last name at the time. and it was basically I would drop all of the things about C++ that I thought could be causing this problem and one of the very big ones was that nothing would be mutable by default, everything would be const by default and it was during this moment of wasteful wasteful idealism that my friend sent me a message, hey, have you seen this new programming language? It's called Rust. and I looked at it like, wow, that syntax is really nasty. Because at the time, and this is coming from C++, so if the C++ guy thought that the Rust syntax was nasty, you know that it must have been. This was at a pre-1.0 phase, this was in 2014, and the syntax was changing a lot. and it was nasty. And I said, well, I'll think about this later. And maybe a couple months later, I looked at it again and I'm like, wait, this is what I was trying to do. Everything is constant by default. Everything gets ownership over its scope and they already have an abstraction for the thing that I'm trying to make. It's called a ARK mutex. It's already there. It's a thing. And it works. It doesn't, it doesn't not work. The thing was, because OpenCV is C++, I couldn't use it in the project that I was working on. I did eventually figure out the bug and fix it, but I wasn't able to use Rust for the project I was working on, so I thought, what will I work on? This is a constant theme over the development of Rust. You come in and it's too early for a certain thing. During different points in time, it has been too early to do system development, too early to do GUI development, too early to do web server development. But as time has gone on, every one of those two earlys has turned into, now is the right time. at the time I was working on an operating system and it was written all completely in x86 assembler and I was like well this might be a good thing to try and write and rust and the only problem was Although I had figured out how to write an operating system in Assembler, I could not figure out how to link Rust properly to run as a kernel and as a bootloader. But I found somebody who did, who did figure it out, and they had a project called Rust Boot. and the project may still be around but it basically just got up to running some code and outputting some stuff to a VGA address and so I think it was just text only. and I said, well, they know how to do the linking. I know how to do everything else. I'm going to just fork this project and dump in, convert all of my assembler. And so I started and it progressed very rapidly, much more rapidly than I thought it would. I had all the code for setting Visa modes. And this is basically, if you have a BIOS system, Visa BIOS extensions is an API you can use to set the graphical mode and get a frame buffer. and that's been one of my interests for a long time is frame buffers and even in redox we're still using a lot of the same code. So I get a 32-bit frame buffer using code that I wrote a long time ago for an assembly operating system. And then I start dumping in tons of Rust code for drawing to the screen. And all of a sudden I have a mouse, I have keyboard, I have all these things growing around it. I have a file system that I ported over from the assembly thing. And I have a monolithic, not just a monolithic kernel, but really It's everything is inside of the kernel. And at that point, somebody posted it, his name, his nickname is Tiki, and he posted it to Reddit. And I'm like, whoa, I'm not ready. I'm not ready to post it. At that time, I was quite shy, and things have changed quite dramatically. Now I can't get enough attention. But back then, I had no idea what this meant. And people commented on it, this is so cool. and they started asking about the direction and I started thinking about it because at the time everything was built into a single kernel program so anything that was running was part of the same process in kernel space and I thought, well, if I'm going to start doing user space, if I'm going to create user space programs, I'm going to move everything I can to user space. I'm going to do a microkernel. and at that point, very serious design changes started happening. Where drivers started moving out to user space, I moved out the graphics stack, I moved out the shell, a whole bunch of things moved out and some of those still exist today. The kernel was rewritten later on. It was rewritten because we had some very serious issues with memory management. and so I did a complete rewrite from scratch to fix those issues and over time it has grown and grown to the point where now it's very unlikely that we make changes to the kernel. We are primarily focusing on building out the user space and porting programs. and that kind of ties into where it connects to system 76 because at this time they're sharing a lot of code. They're sharing code both in UEFI space because the bootloader for Redox uses some of the same libraries that are being used by the firmware updater, firmware setup menu, and other programs that we make at System76. And they're also sharing GUI code. We both have, like, cosmic has been ported over to Redox. A lot of the applications for cosmic crates for cosmic such as iced, soft buffer, win it. Those crates have been, have been ported over to, to Redox. And a driving concern of cosmic is to ensure that it's portable to systems like Redox. Before we get into this, and I will come back to that in a second, I do have a couple of follow-up questions because these are all very low-level concepts. Not everyone might be familiar with them. Sure. Yeah. Maybe we can take a step back and establish what the Rust boot does for you. So is it similar to Coreboot? Is it a bootloader, something that starts the machine and then You take over from there, you're in quote unquote, user space already, or is it something that only provides something low level that you need for your operating system to link and you still need a bootloader? So at the time I was just targeting a BIOS bootloader and kernel and basically the system starts up and it runs from a ROM chip on the motherboard, it runs code that initializes the CPU and memory. Then it tries to find a disk that has a bootloader. and it loads the bootloader in this case, it would be a custom bootloader that I wrote in assembler and that code was still used with the Rust Boot project because the Rust Boot project was really a kernel that you could load in with some other bootloader like Grub and what the Rust Boot project brought was a properly linked Rust code that would run in kernel mode. mhm. Very little of that has survived. Probably none of it has survived past to the current present day Redox kernel. The bootloader has now been rewritten in Rust, so now the bootloader for Redox is mostly Rust. and it even encodes in the redox file system and then it loads the kernel and boots it. And what a bootloader is trying to do is to fill in the gap between the firmware implementation on ROM and what the kernel needs to run. And usually that's file system access or something like that. In a UEFI system, there's actually very little that a bootloader needs to do. It often just tries to find an executable file and tells the firmware to run that executable file with the right arguments. Like system deboot is really just forwarding arguments to a kernel, and the kernel is UEFI executable. In Redox, there's a little more that bootloader is doing because it has file system capabilities. and when you started with this project, what were some of the challenges with using Rust in such environments? I can imagine that maybe you had issues with, I guess so. There were a lot. Maybe you can touch on that. When I started, the syntax was changing rapidly. We were approaching the 1.0 release, but still nowhere close to it. Things were changing all over the place, the way modules were included changed, the way that things were boxed were changed. There were low-level changes to the syntax, too. And this can all be expressed by looking at the original code for Redox OS. The very first commit does not compile anymore. And I don't even know if you can find a Rust compiler that would be able to compile it because I'm not sure that they keep them around before the 1.0 release. Mm. Then there were things that you could not do at all. In-line assembler was something that was still very much in flux and was a nightly feature until maybe a year and a half ago or two years ago around the 1.50 release. So we had to use a nightly compiler and we had to deal with syntax changes. I say we, but it was really only me until Tikki published it and even then very few people working on this project. The person who wrote the original tiny piece of kernel called Rust Boot, I don't think there have been any changes to that project. It's gone unmaintained and I've never been able to contact them. But what that piece was was basically to say, here is some Rust that can run in kernel mode and write to the VGA frame buffer. And that was already extremely impressive because of the amount of jumping through hoops you had to do to get Rust to compile for a kernel. Now a lot of that is built into the Rust compiler itself. They have a special target for building kernel code. They didn't have that back then. So you had to have your own tool chain and you had to confuse the Rust compiler into outputting the right stuff. It was quite a lot of code to get it to compile stuff that would work in bare metal. What is the target that you use to really target kernel stuff nowadays? Is it no STD? So, say things that don't rely on an allocator or is it something else? As a crate, if you want to be included in a project compiled for a kernel, you should probably use no standard, so the no STD flag. But as a binary that you want to compile together and link together as a finished product like a kernel, you have to use the x86 underscore 64 dash unknown dash none. target and that target is for bare metal and then you still have to add linker flags to specify how to link it so you probably need a custom linking script. and even then the environment is complex. Like it doesn't set up uninitialized variables are probably not going to be set up properly. Thread locals are not possible at all. So there are features you can use in Rust that may end up generating garbage if you don't have some initial code run that sets those up. It has to run, it has to initialize all of the uninitialized memory to zeroes, it has to set up thread locals, things like that. and the redox kernel actually uses thread locals in a special way that they become CPU locals. So each CPU instance gets its own thread local variable that identifies certain aspects of that kernel instance running on that CPU. I wonder if Rust didn't exist in an alternative universe, would you have used C++ or would you have stick to assembly or would Redox even exist in such a form? No. I would have Redox would not exist. I was not interested in writing a kernel in C or C++. Those were things that had already been done and C++ is also a complete mess to get it to run in kernel space. Rust has continued to get much and much better in this regard. And now there are tons of Rust operating system projects that are out there. with all kinds of different qualities. And Redox is just one of them that happens to be one of the early ones. But for C++, I would never have done that because C++ never felt to me like it brought enough to the table for a kernel. It was, for me, it felt like a minor extension to see. And the nicest thing that I had from it was references that were tracked properly. You didn't have to pass pointers around, you could pass a reference. And RAII, the ability to have things deallocate when they fell out of scope, instead of having to manually do that like you do in C. but neither C nor C++ really gave me the good feelings. As assembler is extremely fun to write. I still write assembly code. Right now I'm working on Z80 assembler code for a very old computer called the Osborne One, which was the first portable computer. and it was running CPM, which is the precursor to DOS and might have been the operating system that the IBM PC would have used if the company that made it hadn't basically refused the offer. So I love assembler, it's so fun. C and C++, I would not qualify them as fun to me. The thing that Rust brought I could work so quickly and build things out in a way that I never could with those other languages either C++ or C, or especially assembly. Assembly may be fun, but it takes a really long time to do anything. mhm. and building an operating system in assembler was like, it makes a lot of sense because when you get into, if you're a hobbyist trying to make your own operating system, you're going to be exposed to assembly very early. So, why not just keep doing assembly? It's like, it follows very easily. If I'm going to have to write the bootloader in assembly, why don't I just write the kernel in assembly and everything else in assembly and and there are tons of hobbyist OS's that are written in assembly. Once you inject C, you're already adding a lot of extra stuff you have to deal with that you might as well use whatever is the best possible language that gets compiled and that just ended up feeling to me like Rust. Even at that early time, there were so many advantages to it in a kernel with the ability to track references and to have them all be const by default and if they're mutable, you can track them across context and make sure that the kernel isn't doing bad stuff that if I was writing it and see, I'm sure I would run into that all the time. When I think of my time with C and especially C++, the feeling that comes to mind is fear. I was constantly afraid of messing up or forgetting things and then even if I tried my very best, Yeah. things would sometimes fail and there would be very nasty Sacfals in production and the Sacfall in production is not something that you want to wish on anyone. So I'm going to go on a bit of a limp here but if I had to pick one thing that I really like about Rust it would be refactoring in Rust because I feel confident when I do it and mm-hmm. Sometimes I would never do that in other projects with other languages because I would be way too scared and be afraid of making mistakes, but in Rust I feel in control. And I do wonder if that is the same for you, especially in the context of Redox riding an operating system with it. Absolutely. There's one thing I've missed and that is new type. And this is something that I think you can get across very easily to a newcomer how type systems can be used to improve the safety of a system. And this is something that has allowed refactoring to be extremely easy. If I change the order of arguments in a C function, I have no way of knowing if the compiler is actually going to tell me anything bad happens if the arguments are passed the wrong way. I used to write a lot of PHP and PHP has a few very nice examples of argument swapping. There are places where you search for a certain key and the haystack and the needle, as they're called, are in a different order depending on the function. But any value can be passed as the haystack and any value can be passed as the needle. And you're searching for the needle and the haystack, but obviously if you switch them up, you'll get no results. Because you're never going to find the haystack in the needle, unless they're exactly equal to each other. So this is a downfall of dynamically typed systems is that when you do refactoring, you have to be very careful in what you do to make sure that you don't swap arguments or do something similar to like that. And C is very prone to the same thing. Even though it's a typed language, it really isn't. So many different types are just integers. or just longs. Type death something long. Type death something int. These are interchangeable types. That means that if you mix up the type of a process ID and the type of a file ID, nothing anywhere will ever tell you anything is wrong until it runs. It has to run to the point where it fails. In Rust, you can wrap something in a type. Even if you have a plain integer, You can wrap it in a type. You create a new type. And this type is not interchangeable. Even if it has the exact same layout in memory, even if it is laid out like an integer, even if it's a 4-byte number, you cannot mix it up with another 4-byte number that has a different type. The compiler will complain. This is not the case in CNC++. It is very easy to have types be interchangeable. Couldn't you do the same thing in C++ though? Couldn't you define your own new type? It is, it is, you can define a new type in C++, yes. But you're working with a, with a standard library that often doesn't, that often is taking integers and taking things like that. And, and it's, it's, and pointers are easy to cast too in C and C++. In Rust, you have to drop down to unsafe. and I think this is another thing. It becomes difficult to make mistakes that are easy. How many times in C++ code, where you could use types, do people cast void stars? Do people send a void star as a user argument to a callback, something like that? Because that is the way that it's done in C and the way that people have done it for years and years. So you have a callback function, it takes a void star, and then it has to cast that. And it doesn't know if it was given a const or a mut. It doesn't know if it was given something mutable or not. It just knows I have a pointer. Now I have to interpret that pointer. And the interpretation of that pointer is the important part. C++ has lambdas. And those lambdas can be typed. but the amount of C code and C++ code that is being built on that doesn't use that concept is where the problem comes in. And this is where I was going with the safe object language. As I was trying to say, how can I get rid of these things in C++? I don't want things to be mutable by default. I think one of the primary things was I never want to ever send a pointer anywhere. Everything should be a reference. because a pointer means it could be null, it could be invalid. And C++ will happily take a null pointer and send it anywhere. And the only way to be sure you don't have a null pointer is to check it manually. And if you don't check it manually, it will be checked at runtime. And it will be checked by crashing the application. and maybe an invalid pointer doesn't have to crash the application, right? It can inject vulnerabilities. You have an invalid pointer, now you have a way for someone to latch in from the outside and say, what kind of invalid input do I have to give to this program? There's nothing in C++ that forbids the use of raw pointers, so they are often used. And you can try to lint them out from your own code, but you're going to build on top of other libraries. And those libraries will use them. Now there might be some listeners who are old-school maybe systems engineers or kernel hackers and they might say this is mostly a skill issue and you can avoid all of these problems with CNC++ if you just try good enough and there are tools out there like static analysis tools and linters nowadays which can prevent some of the issues. Yeah, I love that. Okay, I have so many examples. What would you say? I have so many examples of skilled programmers using the absolute top of the line. I mean, just recently there was a G-lib C issue and it can be exposed remotely through PHP code. A G-lib C issue can allow remote code execution. There is no library in the Linux ecosystem that is as heavily used as glbic. If there was a smart person who could apply some kind of technique to glbic, they have. They did. And yet, for 24 years, there was a vulnerability present. This is a fact of the language itself. You can say it's a skill issue and I've heard this many, many, many times. I've written operating systems in assembler. A lot of people are concerned about their own skills. I've never been concerned about my own skills. It's always been a concern of the human nature. Human beings are simply incapable of keeping an entire system in their minds. It is not possible, it's not feasible, no matter how much experience you have, no matter how long you've spent programming, no programmer alive is able to keep an entire system in their mind. Because you're having to deal with, if you're writing a program, you're having to deal with not only the program code, you write, sure, printf hello world. Okay, great. Can you keep in all in your mind all of the code that is running just for a print F because to have a print off you need to first invent the universe and then you need silicon and you need to melt the silicon and make it purified and you need to put it together into a chip and chips have vulnerabilities, right? As we've learned chips have hardware vulnerabilities like Spectre and Meltdown and then you need to have an operating system running on that chip and then you need to have a C library and finally in the C library you have a printf function and finally you have code that can run the printf function but then you need a compiler to output a binary that the operating system can run. You have billions of lines of code. And it's not just talking about the Lanx kernel, it's talking about the C libraries, it's talking about the whole entire ecosystem, the compilers you're using, it's talking about what was used to generate the hardware itself. Because human beings aren't keeping in their mind the CPU design either. Nobody can keep that in mind. The Intel passed a hundred billion transistors per chip with Alder Lake CPU designs. So with the Alder Lake desktop CPU, they finally passed a hundred billion transistors. Wow No human being alive can keep that in mind. Even if you write perfect code on a perfect CPU, something somewhere can be imperfect and it is impossible. And people will say, well, this is why we need AI to write code. Well, from what I've seen of AI writing code, right now what they're doing is replicating the way that humans write code. If humans can't do it perfectly, AI's trained on humans can't do it perfectly. And I've seen imperfect AI code. So the only thing you can do is inject at a lower level constraints that will salt resolve this. And Rust goes a little bit of the way there. Do you ever feel like you're spending more time managing your data pipelines than actually using the data? Data streaming is critical for modern software development, but most platforms struggle to keep up. Pipelines are often a nightmare, slow, flaky, and a pain to maintain. We end up cobbling together Kafka, Flink, Spark, and Lambdas just to get data where it needs to be. Today's sponsor, InfinyOn, the creators of Fluvio, is aiming to fix that. built from the ground up in Rust, Fluvio is lightning fast and rocks on it. It allows you to build composable event-driven data pipelines in minutes, not months. Fluvio compiles to a single 37 megabyte binary running stream processors on the edge. With smart modules, you can transform data on the fly. Whether you're dealing with IoT sensor data, financial transactions or custom events, Fluvio can handle it. It's built by devs, for devs with a clean API and an experience similar to Ruby on Rails. Visit infinyon.com/rustinprod to explore how you can streamline your data infrastructure. That's infinyon.com/rustinprod. Okay, no one can keep it in the head, maybe except Chuck Norris, no offense to him, but like very few people can, of course, but still you have these millions, you said maybe even billions of lines of code out there that are written in C and C++, maybe still maintain, maybe no longer maintain, you have them in all major operating systems, that stuff is not going to go away. Linux is 30 years old and Windows is older. So I do wonder how long will this stuff still be around and also where do you see the future of critical infrastructure going now that we have a tool like Rust? Is that a slow oxidation process? Is that happening or do you see challenges ahead? Well, I think it depends on the system because some systems you can completely transition to Rust. For example, if you have an embedded controller and it's one of the supported architectures and you have a very small project that you're working on and it's running raw on bare metal, you can convert to Rust pretty easily. But for an operating system at the operating system level, it is a very, very, very glacial process of replacing things with Rust. And this is where I've always been interested. I'm not too interested in taking on those small items because I think there's very little benefit because the major reason for Rust in my mind is to handle security vulnerabilities that exist only when you have a large scale software ecosystem like an operating system and the applications that run on top of it. And these large scale ecosystems have been built over dozens and dozens of years. As you noted, Linux is 30 years old. We have We have code older than Linux that still runs in Linux. Like X11 was from the 1980s, the late 80s, and X11 is, there's still code that is around from that time that's running on modern computers. Even if they've transitioned to Wayland, as a window manager, X Wayland is still running. And that's basically a containerized version of the X11 server. So this is a ecosystem in history that I don't think is ever really going to go away. But what we need to do is think about how to resolve certain aspects of the Unsafety that has been brought in by having a programming language that's been very loose and not very strongly typed and also doesn't have the baro checker like Rust does. These are aspects that you don't have to resolve them across the entire stack. What you have to do is think about how important security is at each level of the stack and how to address it in small incremental ways. and one incremental way is to implement sandboxing. So to ensure that critical components are sandboxed in a way that if they do experience failures, like if you have a piece of C code that you know is ripe for vulnerabilities, Then you sandbox it away so that it doesn't have access to other components on the system. This is part of the reason why Redox is a microkernel is because even though I believe in Rust being an improvement to security, I don't think it's the be-all-end-all. I think there are plenty of ways to write insecure Rust code, and that's why I want to segment off each individual driver, each individual file system, each individual service in its own process space. It's a sandboxing technique to ensure if each one of those gets compromised they won't be able to access any of the others. The interesting part of this is that even if a certain driver gets compromised, In Redox, almost all of the drivers drop all of their capabilities before they start operating. So they acquire their file system kind of namespace. In Redox, every driver exposes itself as a piece of the file system. and then they drop their ability to open files. They enter what's called the null namespace. And in doing so, if it's compromised, even if it's Rust code, I don't believe it's uncompromizable. I think there have been security vulnerabilities in the past. Even the standard library has had security issues. Is that a part of the runtime or is it something that you kind of have as a convention in drivers, so the fact that they have to drop all their permissions? The fact that they drop their permissions is it is a collaboration between the driver itself and the kernel. They don't have to drop their permissions, but all the drivers we have available on the regular system will. And they do this by entering into a namespace that doesn't have any file systems mapped except for some very minimal ones, like for allocating memory. So they can't access the network, they can't access the file system, they can't access any other drivers. and this sandboxing technique is also pretty common in the Lynx desktop. We have things like flatpack and we need to really lean into this and try to improve it where we nail down what kind of permissions certain things need and really reduce it. And in the server world, of course, we have Docker and things like that where we're able to run things in containerized instances. This is already proven to reduce the security vulnerability significantly and Absolutely, do you see it trickle down to the kernel level even at some point? Because companies like System76, they still, and for a very long time, they will depend on a Linux kernel. And fundamentally, it's written in C, you have driver support in Rust now, but do you also see the reverse where the C code runs in the sandbox and Rust is the major runtime in the kernel? I don't think the kernel will ever be compartmentalized this way and this is a fundamental disagreement in software is the design of the kernel is it a monolithic item running in one process space where most of the drivers are running in the same process space or is it a microkernel where where certain aspects of the system, what people think of as kernel services are spread out among different processes. There's not really an easy way to take the Linux kernel and say, I'm going to take the file system driver and run it in a safe, contained space. and I think Rust is improving along with other things like improving static analysis of C and fuzzing and things like that are improving the security of Linux kernel but it is always possible and I would put heavy money on it that there are vulnerabilities remaining in the Linux kernel. It's just too hard to say it's safe. I mean, if you look back in time and you look at, I mean, very recently we've had to have a security update to the kernel. We had to push kernel 6.8 because there was a security vulnerability. And it could be nothing that's the fault of the kernel. It could be another thing like Spectre or Meltdown where researchers find that there is an issue with the hardware itself and then the kernel has to adapt. and that also would apply to rust kernels as well. But the point is that there is no way to containerize the Linux kernel. The best that you can do is throw every technique you have at it and try to reduce the amount of things that you're doing in the kernel. So I do wonder about the advancements in tooling for that sort of work. Do you think that Rust also caused some sort of evolution there, some sort of renaissance of new and of older ideas that suddenly became popular again in systems-level development, thanks to Rust? ... Absolutely, I think so. I think that there has been so much focus on C as a language. I mean even Rust is built in some way or some fashion around C. The syntax is very similar to C, it compiles to very similar objects to C. You can easily interface with C through extern C functions or you can export your own functions as C functions that have a C compatible ABI. A lot of crates are just wrappers around C libraries. It's important to recognize how much of the ecosystem, even for other languages, is C. When you look at Python, you look at PHP, these are things that are really C ecosystems that are expressed through an interpreted language. Every Rust program that uses a standard library is usually using a C library. and the point is that C grew to be such an important thing that I think that now what we're seeing is that Rust introduces some new concepts but even for Rust itself to advance to the point of acquiring these security benefits that it has brought to the table. We need to apply them to large sets of C code. And the only way to do that is incremental changes to what is a massive set of legacy. You're talking about code that has been around for many decades and is not ready for change. It wasn't written with the concepts Rust has in mind. C code was never written to assume that you would have ownership over an object. Most C code does not have const sprinkled around where it should be. It uses casts to change objects into different types, liberally. There are just There are things that you would have to inject at a compiler level. You would have to say, now we're going to say, all of the C code for this platform is going to have to be compiled with this new flag. And this new flag is going to throw an error anytime anyone uses an invalid cast. and there have been cases where that has been done even at a Linux distribution level. There have been hardening things that have been done. Debian has added flags that are required that introduce hardening and those can improve security. They can reduce the likelihood of certain attacks. For example, having stack having what are called stack canaries that check if a stack value has been changed, to check for stack overflows. It doesn't require a change to most C code, but it allows for checking if the C code had a buffer overflow. It's not going to be completely 100%, though it doesn't count for heap overflows, and it doesn't account for specially crafted stack overflows that may be able to get around the canary value. being important. And there's things that are being done, but none of them are as radical as Rust itself. And that's the thing that concerns me is that if you're trying to apply the same techniques, especially the borrow checker, if you're trying to apply something similar like the ownership model of Rust to another language, I think you pretty much have to rewrite the language in any code that was written for it. and that brings up a very good question is why would you do that instead of using the language that that concept came from? What will be your answer? My answer, well, I'm not taking part in that at all. I have very, if a project is a C project, I usually don't touch it. There have been very few cases where I've started a new project and it hasn't been rust. and I just don't find there to be much life in trying to keep very, very old code around forever. I think in a thousand years, what kind of code will be around? Probably nothing that anybody has written today. Meanwhile, we'll still have all the great classics. the things that were written, the music, the books, these pieces of human culture that were written by people, we already have those from thousands of years ago before. but the code, this is another cultural artifact but we're treating it as though it will have long lasting status. And I don't think it will. I think this is something to be replaced. I think this is something where we have to move forward and we have to reconsider all of what a computer is over time. It can't just be a static thing like, oh, we finished. We're done. We already have operating systems. We already have everything working. We really don't need anymore. There is a degree of truth to that. I think humanity would be just fine living with the hardware and software we have right now. Nobody needs to write anything else. That's a bold statement. I think it is, I mean, most of what you need to do, you can do it in a spreadsheet, right? And just all you need is Excel, and then you're done. This is how a lot of office jobs are accomplished. True. The reality is that there will be long-standing issues. These projects, these very large projects, Windows, Linux, the entire stack on top of Linux have been written in C and these large projects will suffer from the problems that are not fixed by C and they will suffer for as long as they exist. There will be no mechanism that you can do to take pre-existing C code and prevent, for example, buffer overflows. It's impossible. It is literally impossible. The language itself allows for arbitrary access to memory. If it allows for arbitrary access to memory, lots of libraries and applications written in the language will have already used it, and they won't compile with the compiler that adds those constraints. so the language would be a different language. I hear you and I'm on your side, we are both huge Rust advocates, but let's assume that there was someone listening right now that is a bit of a lead engineer or decision maker at a company and they say, we hear you, we understand the benefits of Rust, but the reality is that we have to convince our team to give it a chance and every time they bring up this topic people say, oh yeah, Rust has a weird syntax and it has such a steep learning curve and how would you empower these people? What would you tell them to say to their lead or even to their fellow engineers maybe to make some progress here? I'd say it is a really hard sell to try and change pre-existing projects. and I don't think it's always valuable to try and do that. I would say the best place to start with Rust in production is with a new project. Come up with something new that you can do, no preexisting code, something new, and then write that in Rust. It can be something small. If you look at the very important projects in the Rust ecosystem that were spawned off by large companies, they often start with something small. They start with some piece of the back end that, oh, we need to write this better for performance reasons. We have a reason to write something new. Let's try it in Rust. And then you have a kernel, a tiny kernel of possibility, and from that you can grow. You can grow out an ecosystem. And I think it's important not to just, I mean, this is almost an evangelism thing. If an evangelist says what you're doing is wrong and you should be ashamed for it, That's a very bad way to start a conversation to try and convert someone to something new. If they say, hey, why don't you just try this for something small? Try to do something different. And I approach it the same way as trying to get my kids to eat vegetables. Rust can feel like eating vegetables for a child. You don't want to start. You know it's good for you. You know it's good. Okay, everybody says it's good for you. Maybe you're not so sure. But it doesn't taste good, leaves a bad taste in your mouth, it's really hard, and it's really hard to get interested in it. But if you cook it, just write. If you cook the rust, just write. then the kids will like it. I can get my five-year-old to eat asparagus, if it's cooked just right. And I can get a big company to experiment with Rust. We took System76 and we started. We had a little project. Me and my fellow engineers, we had the firmware updater, started in Rust, finished in Rust, and then we go to other projects. Then it was Pop!_OS. and it was the distribution installer. We wrote that in Rust. It's called Distanced. We didn't write the GUI in Rust though, so that was the next step. Now we're writing the GUI in Rust. If you start with a very big, well, we have to rewrite this project in Rust. It's going to get a bad reaction and it deserves a bad reaction because I think even though there's massive benefits to writing a project in Rust, as I've talked about before, are also massive costs. and you can be sure if you rewrite a kernel in Rust you're going to have vulnerabilities too because new code brings with it new ideas and new ways for vulnerabilities to attach and there are things that Rust still doesn't handle quite well like like integer overflows that I think you have to think about when you're writing new code and there's still a degree of skill that even writing Rust I know the people who say, well, you should write kernels in C because if you're not skilled enough to write it in C, then you shouldn't be writing kernels at all. Well, you still have to be skilled to write in Rust. It's not like all of a sudden you get rid of a lot of errors because there are still logical errors that can be made in Rust and there are still integer overflows and things that I'm honestly quite irritated about because if I had known about these earlier, I would have added so many links. to the Rust code. But guess what? Now I have a set of Rust code that is so large that applying those links takes days to work through the project and actually resolve all of them. It's almost exactly the same as starting with C. If you were to add links to the C project to say, Hey, you can't pass this pointer around because it could be made invalid or it's owned by another function. Then you would have to rewrite a whole bunch of code. Well, a lot of those gotchas still exist in Rust code too. So I would say start with something small. Don't try to rewrite the whole universe. I really like your analogy between vegetables and rust because my girlfriend is a nursery school teacher and she always says that if you chop vegetables and fruit up into very small pieces then it will be gone in no time because kids really like it and it's manageable but someone would listen to it and say well in reality most of what I deal with on a daily basis are these big monolithic projects, these huge code bases and I work with legacy all the time and I don't have time for this right now and it's one of the things that are important but not urgent so usually it gets pushed on the back burner so I do wonder Yes. What would you tell these people if they cannot start from a green field? Well, having Or is there always a green field project? They're almost always is, even if you can, I do realize that a lot of software out there is built in huge monolithic projects. And the concern is often not how to change these things into smaller bite-sized projects. The concern is how do I keep this old code running forever? And I would tell them, even if you don't figure out how to rewrite the old code. Some other company will and they will do it better and they will do it faster and they will compete with your company and they will replace your company. It is a reality that I've seen time and time again that having had to maintain legacy code bases but myself It ends with stagnation. You're not able to make changes to it even if you like the language it was written in because it is too large to attack and too large to come up with ways to improve it. So I think rewrites are healthy and even if it's not a rewrite in Rust, I think it's very hard to sell it though. and so I would encourage that if you're stuck in a position where you have a large piece of code and your company is reliant on it and it has to work forever and it has to not change, you have to consider what is the likelihood that another company will replace the company that you're currently working at. What is the likelihood that someone else will be able to do what you do better? And if it's very low, I would say suck it up. But if it's very high, I would say you have got to approach this as, can you do something to modernize this code base to bring it to the future so that you have something that remains marketable? Okay, then let me play the devil's advocate for a second. Yeah. A cynical person might answer Yeah, but Rust will not be around in 10 years anyway because there will be a new, hot thing and we will rewrite everything in sick in no time. So it's very hard for me to find people and hire people even five years down the road and then we are left with this big old Rust code base which then becomes legacy and we want to maintain that for a long time. Yeah. Well, this is the problem from the company, the company mindset of wanting to build everything out as a single very large big project, isn't it? And I understand that most projects are still done that way. And I, I, I try to, to dissuade all of the people working here from ever doing those kinds of projects because we have two of those kinds of projects already. And they have been massive failures. They have been disastrous for the company. They have caused significant amount of engineering time that could have gone to literally anything else. We have a very large project called Joshua, this written in PHP, and it has been around since probably 2013. And this project has not gone anywhere. There have been two or three different attempts to replace this project. So even the rewrites of the project have failed. And at the end of that process, probably we've had full-time engineers working on it, multiple full-time engineers for the entirety of that time since 2013. Not a single change since the original code was written. Not a single valuable change, I would say. not something that the user would see and would say, wow, that's a nice new feature. Because it's not even for selling to users, it's just for maintaining our information about sales. So it's our customer support and sales system. And this system is It is, since it's not a direct value thing for our company, we've decided to start outsourcing. So now we are going to replace it entirely with very minimal code that interfaces with a third party system. And the realization has been over those multiple rewrite processes. And some of them have been for language reasons too. The original code base is PHP. And Erlang. was elixir, elixir was a really hot topic and maybe still is. I'm not too wise about back-end stuff. But elixir, we had numerous engineers that were very excited about elixir and they said, we're going to rewrite this project in elixir. and they rewrite parts of it, it's true. But all that ended up doing was turning into a secondary monolithic project that had to exist alongside the original monolithic project because now they were intertwined in a way kind of halfway between transition. where neither could exist without the other. And then the elixir people were replaced with more elixir people, and then they were replaced with more people, and the turnover of the team over the past nine years, or 10 years, yeah, 10 years. has meant that now we have two monolithic systems where none of the original creators of those systems are present. And so these are legacy systems by definition. And one of them is written in a really cool language, but because it's monolithic and because it's intertwined with the other systems, it can't be easily extended or replaced. Right, so if I understand you correctly, the way to avoid this trap of having to maintain some big old legacy system in the future is to split it up and keep things simple and maintainable and not monolithic, but rather I don't want to use the term microservice, but something that is a bit more manageable and has very clear bounded context, so to say. Yes, and it has to start, this kind of methodology has to start at the beginning. Because if you go into a project saying, I'm going to split it up later, it's most likely it's going to make it to production. True. That's the title of this podcast. It's going to be in production. And then you won't be able to change it. Because once it's in production, you're making money off of it. Your salary is based on it. You have to make it stay in production. And then, and I just, I have to point this out. When you're doing software and your business is software, you're making money from software. Well, the great thing about software is anybody can write any software anytime. Someone can write your software better. And this is one reason why System76 has not been in the business of software directly. Because we felt it's been a very difficult business to be good at and to actually stay on top of things. And it hasn't been our primary focus. Selling hardware is something that's much more concrete and more predictable. And selling software has always been this way. And a lot of these Rust-based startups have been big replacements for other projects. When you look at the big companies that are using Rust now, they have been replacing these large monolithic projects in other languages. Can you give me an example? that there are now companies hiring for Rust that are high frequency traders that did not exist before Rust and are direct competitors. And their existence is because of Rust's capability in doing extremely, extremely concurrent things safely. There was no way to write this kind of code in C or C++ without considerable time planning beforehand. Now you can pump out concurrent Rust code faster than anyone could write concurrent Rust code before. Anyone could write concurrent code period before. I think it's a major advantage to Rust. You mentioned earlier that production was part of this podcast title and that is true but there's a different notion too in the title which is Rust in production. And so I wonder, some people, potential clients, they sometimes ask me, is Rust here to stay? Yes. I will make a big investment in this language now. Will I be able to hire people further down the road? And also, is this just a hype or is it here to stay? I would say it's here to stay and the reason is that it took this long just to get the Rust ecosystem up to spec with everyone being enamored with it. And there are naysayers of course, but has any language had this kind of, you can look at the stack overflow stats. They do a survey, a developer survey. What are your most loved languages and what are your most used languages? And the most used languages They tilt towards things like C, C++,Java,Python, and those kinds of languages in JavaScript and so on and PHP. Your most loved languages, those tend towards things like Rust. And I think that this kind of enthusiasm has not really existed for other languages in a long time, and has already built an ecosystem that is extremely large, and when you mention other languages like Zig, I mean, I'm fine with whatever ends up being the memory-safe compiled language. I don't know what that's going to end up being in 10 years, but if I had to bet on it, it would be Rust. I have not seen a big movement, a large amount of developers moving over to any other languages. I haven't seen them move to Zig. So if I had to bet on it, I would bet on Rust being here in 10 years and being the dominant memory safe compiled language. Maybe to expand on what we discussed earlier, because this is a great segue into what I promised we would come back to, which is cosmic at some point. I guess one other thing that I really like about Rust next to refactoring is the fact that you can use it across the stack from systems, level code, all the way up to WebAssembly and front-end stuff. mhm. and cosmic is something that is completely detached from the kernel, it's something that is in user space I guess and you kind of control things on the screen so it is very much a thing that people interact with on a regular basis. How does that feel when you have a language that you can use across the stack? How is that for such a big project? It's quite amazing because not only are languages used across the stack, but libraries are used across the stack. I can't think of many examples where firmware is using the same libraries as a GUI application. or a kernel. I mean, these are libraries being used, whether it's the Rust Core library and ALEC library being used across the stack or text rendering. I'm using the same orb client library. I've mentioned this before, it's the Orbital Client Library, it's for establishing graphics support on Redox, and it can communicate with the Orbital server to put Windows on the screen and to draw certain shapes onto those Windows. It is concurrently being used in both in redox for the orbital server but also in cosmic applications for interfacing with the orbital server but also in System76 firmware applications for drawing text and shapes on the screen. and just being able to mark a library no standard means that the library can exist anywhere from embedded Rust code running on a tiny ARM microcontroller to running in a GUI application and so that's been quite something and using cargo in all places so everywhere the same package manager in the same set of packages and it's more than just using the same language. It's using the same ecosystem across completely different boundaries and that's been crazy. And then you can use the same code in WASM. So you can end up putting stuff in a web browser or your backend code can use the same stuff. That's crazy to me that you can have a library defining some simple data types and then you can import it in your Rust code for your back end for your web server, import it in the client code in Wasm, you can import it in a GUI application that replaces that web browser version You can import it in a microcontroller. So you can have all of these things using the exact same definitions of objects of code shared across such wide boundaries that previously it wasn't really possible. Even if all these things could compile C code, C is a very wide spectrum. There is a very big difference between compiling C code for an embedded controller, which is something that I still work on at System 76. It's not Rust, it's C code. So we have an 8051 embedded controller, that's the architecture, Intel 8051. It's a very strange architecture. There's two different architectures that they teach in computer science, von Neumann and Harvard. and one of them has program and data space integrated. That's the one that most people use. That's the one that most computers use and most people are familiar with as programmers. But then this embed controller has a split space where you have program and data in separate spaces. and so it's a very unique architecture. It can't compile regular C code. You have to specify extra things to get C code to compile properly for it. So in order to actually have something spread the stack so widely, it is quite interesting. do you sometimes dare to touch it and oxidize that part? Well, unfortunately, there's no LLVM support for it. And so it's kind of dead in the water until that. But there is a mechanism. And the mechanism is to compile your code to risk five and then write a risk five emulator for the target architecture, which someone already wrote in 8051 to risk five emulator. So if I were to run that, then the firmware could be rust. But again, The embedded controllers, although there is a lot of interest in Rust, I don't think the security advantages are really there as much because you're not often dealing with third-party input. You're usually dealing with things that are very structured and have fixed sizes and even C code can deal with those quite well. And very rarely are we passing around things that are dynamically typed or doing things that aren't I would say you can define a subset of C that is safe. And that subset of C doesn't take any pointers. For example, if you have a function int add and int a comma int b. And all it does is add two integers. It's completely memory safe. There is nothing possible that could be interjected in that except for an incorrect compiler implementation. That will never ever have a memory safety vulnerability because it doesn't have any memory pointers. True. There was a joke in there. You could have said that subset of C is empty. And yeah, that's a missed opportunity right there. Well, a lot of embedded controller code is like that, where there's very little need for memory safety, so I'm not too concerned about it. Yeah. But on the other side, we have the launch keyboard, which right now it uses an ARM microcontroller. Yeah. It uses the Raspberry Pi RP2040. and so there has been definitely some interest in converting that to Rust because Rust compiles to ARM very easily and for ARM embedded controllers and there's already a big ecosystem for RP2040 for that specific ARM microcontroller. If someone wanted to get started with operating system projects or low level projects in Rust, what resources would you tell them to use? How could they even start it? And also, if you need some help with Redox, how can they learn more and what would you need help with? Yeah, so the resources I used when I was getting started were mostly on the OS dev wiki, which has a ton of great information about how to get something booting in an emulator. Real hardware is not so easy these days because most of it is transferred away from BIOS to UEFI and most of the tutorials are for BIOS. So getting past that hurdle is quite difficult. There is a Rust OS dev organization and I forget his full name if it's Philip. I know his nickname is Phil Op, but he has a set of tutorials that are pretty useful. I didn't use those because I think they came after the time when I first wrote redox, but I have looked at them since. and then as for what Redox needs, mostly it's not low level stuff anymore. It's higher level stuff like porting programs over and writing new code in Rust that can be utilized in an operating system project. The stuff that is needed that's specific to Redox is really low level like writing drivers. but if you can contribute to writing things like cosmic applications, those are going to be ported over to Redox and those expand our capabilities significantly. We will link this rust-osdev resource in the show notes. Shout out to Philipp Oppermann, who's a fellow German. He's in Karlsruhe. And yeah, and probably you also have some open source repositories on System76, written in Rust. Is that correct? that's correct. If you go to, I think the best starting place for that is probably with libcosmic because most of our work is centered around GUI and that's where I would also drive people too because I feel like this is an ecosystem that needs to be filled out and it's pretty nascent. It's pretty new. There is a lot of system code out there and I applaud anyone who's willing to take that on. But it is a lot easier to get started in gooey stuff and you get instant gratification. You don't get that kind of stuff from system coding where someone's like, you fix the Cisco implementation so it's 5% faster. Okay. But does the user see that? With a GUI application, it's very instant gratification. You have something that you can show people that they directly understand. So the libcosmic repository is where I would start. And there's examples there, and we have all the cosmic applications. If you are interested in going into deeper level, low level stuff, we have firmware applications that are written in Rust. And they're sharing code with Redox. and so we have the firmware setup application that we run on all of our core boot laptops and we also have the firmware update application and we also have low-level services that are written in Rust like Dist Inst which is the installer back in for Pop!_OS and we have things like system76 power that manages power profiles for Pop!_OS and we have system76 firmware which is the firmware downloading Library for Pop!_OS. It's pretty incredible. I don't know any other hardware vendor or Linux computer vendor that is so heavily invested in Rust. That's really great to hear. And I think it's something worth checking out if someone is looking for a new device or wants to learn more about systems level programming. As you said, there are other things too, which people can check out. Lastly, it has become a bit of a tradition around here to ask this final question, what would be your message to the Rust community? Could be anything that you have in mind, something technical or non-technical, completely up to you? Yeah, if there was one thing that I wish would change, this is because often I would give up something positive. But there's so much positivity with Rust that I feel like I need to give something negative. I haven't said anything negative about Rust yet. Please stick with your projects. Please stop abandoning your Rust projects. It has been somewhat of a trend that a very nice new Rust crate will be created and the developer will spend a couple days to fill it out and then it will be left forever dead but still available on the crates.io website. So I would encourage, if you, if you, if it sounds like you have a project like that, take a look at it. Make sure you're answering the issues. Make sure you're looking at pull requests. And if you're not able to maintain it, please consider passing over maintainer ship to someone who's interested in it. I've done this several times with projects where I've offered to maintain them. One, for example, is soft buffer, which is a winnet project now. It's part of the Rust windowing organization. It's now being utilized by the ICE toolkit for software rendering. But several years ago, it was filled out by its excellent creator who then left it. And nobody looked at it, nobody maintained it for years. And I found the project and I realized this would fit in really well with how ICE was going to approach software rendering because at the time ICE was only GPU rendered. and I utilize the project to write a special renderer for ICE and then the ICE maintainer, Hector, he rewrote the whole thing much better than what I had written and we were able to take this library that have gone unmaintained I talked to the WinIt maintainers and we were able to get access to the original maintainer and he agreed to transition it over. So this in my mind is a triumph. Somebody wrote amazing code years ago that filled in an important need in the Rust ecosystem to be able to software render easily on multiple operating systems. And it went on maintained and then after this need was re kind of a fire was lit under it by cosmic because we needed to have some sort of software rendering. We went out searching for a library and that's what I found. And then I brought it up to the Rust windowing people and they were able to bring it into that project and now it's maintained by multiple maintainers for each platform back end. There are so many projects out there that are like this though and I keep finding them. I keep bringing them into my code and then realizing later. This project is not maintained anymore, the original maintainer is not responding to pull requests or issues, and so I have to replace it. And it's a really difficult thing because I really want the Rust ecosystem to grow, but it feels like it grew so fast during some early years. that people got sidelined by other things. And so I encourage you, please keep your Rust code maintained. And if you're not able to maintain it, please put out a notice so that someone else can step in to maintain it. I'm all for it. I totally hear you been on both sides. I had libraries that I wanted to use and found to be unmaintained, but also sometimes deprecated other things. If you have to deprecated Yeah, a good advice would be to be open about it, communicate, and maybe archive the project so that people can see that it's so long maintained. Also, it sounds like there's an opportunity for a project where people could mark things that they want to have maintained somehow and maybe there will be an opportunity for people to find work in the open source space and get started. Right. Yeah, there is money out there too. I mean, there are things where often a maintainer will lose interest and they'll go quiet. Yeah. They won't ask for help, and there is help. There is money out in the Rust ecosystem. There are companies, there are other individuals who are willing to help out, so just saying something is so important. I've seen so many Rust projects go completely dark because they were created by a single maintainer, and it's a very important piece of the Rust ecosystem that needed to be filled in, and it just needs somebody to take action. Absolutely! Jeremy, it has been fantastic to have you on a show today. Thank you so much for your time. Thanks for having me. Rust in Production is a podcast by corrode. It is hosted by me Matthias Endler and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.