Matthias: Welcome to another episode of Rust in Production, a podcast about companies
Matthias: who use Rust to shape the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today I'm joined by Danilo Krummrich
Matthias: from the Linux Kernel Project to talk about bringing Rust into the kernel.
Matthias: Danilo, thanks so much for taking the time for the interview today.
Matthias: Can you quickly say a few words about yourself and about Red Hat,
Matthias: the company you work for?
Danilo: Of course, thanks for the invitation. Yeah, so my name is Danilo and I'm a Linux
Danilo: kernel engineer working at Red Hat in the accelerators and GPU group.
Danilo: And I'm also a maintainer in the Linux kernel of various subsystems,
Danilo: components and drivers and member of the Rust4Linux team.
Matthias: It's almost intimidating to think that you work on such an important project
Matthias: and also on such a low level.
Matthias: I guess a lot of people that are listening in either use Linux or get in contact
Matthias: with Linux on a daily basis because, you know, we use a ton of servers nowadays
Matthias: and a lot of these run on Linux.
Matthias: What does it feel like to work on that level and how did you even get started
Matthias: with kernel development?
Danilo: I don't think it necessarily feels different than working on any other software
Danilo: project, I would guess. I cannot really tell.
Danilo: I worked on a few other projects in the past 10 years ago. I worked a little
Danilo: bit on the AOSP, so the Android open source project.
Danilo: I worked there at kernel level as well, but also some user space portions.
Danilo: So i was always interested in the kernel since i finished my exams at university
Danilo: and when i joined the first company that i also worked at when i was a student
Danilo: i i just told people hey i'm interested in the kernel i want to work at that
Danilo: just get me to the just get me to the base layer team,
Danilo: and then i just started working there and and especially i started learning
Danilo: a lot the learning curve was very, very high.
Matthias: When was the first time you personally heard of the Rust4Linux project?
Matthias: When was the first time you heard about, I mean, discussions,
Matthias: adding Rust to the Linux kernel?
Danilo: I would say almost from the beginning, because I'm following what's going on
Danilo: in the project, what's discussed in mailing lists.
Danilo: But this was more reading along. It was not that I I was immediately,
Danilo: okay, this is definitely something we need to, I need to get involved in. But I was aware.
Matthias: Did you know Rust back then?
Matthias: So this was all new for you, but how did it feel like?
Matthias: Can you remember the time, the feeling that maybe thinking about the adoption
Matthias: of Rust and was that the time when you also looked at the language for the first time?
Matthias: Can you talk about your first impressions of the language coming from a mostly
Matthias: C background, I'm guessing?
Danilo: It was a bit intimidating, I would say. The C language is very, very simple.
Danilo: And in the kernel, we are doing very, very complicated things.
Danilo: With a very, very simple language.
Danilo: With Rust, it's different. Rust is a very difficult language, I would say.
Danilo: But at the same time, it makes some of the problems we have to solve much easier to solve.
Danilo: So I guess that's a good trade-off.
Matthias: Did you ever second-guess the adoption of Rust in a Linux kernel,
Matthias: or was it always clear to you that there were very obvious benefits?
Danilo: So after I got used to the concepts of Rust and understood what they are about
Danilo: and understood how I can utilize them, I don't have any doubt.
Danilo: This is really i'm very or let it say the other way around i'm very very convinced
Danilo: that rust definitely has a big future in the kernel how.
Matthias: Long did it take to build out the entire infrastructure infrastructure that
Matthias: is needed for such a project inside the kernel and outside the kernel from the rust side i mean.
Danilo: So i think a lot
Danilo: of initial so a lot of the initial work was
Danilo: and i cannot say a lot about that because that was not the time where i joined
Danilo: the project this was before i joined the project but i think and if i'm forgetting
Danilo: someone i'm very sorry about it but i i think it was mainly Miguel and and Wedson
Danilo: so Miguel Ojeda and Wedson Almeida who,
Danilo: started the Rust4Linux project and in a downstream branch worked out a lot
Danilo: of infrastructure,
Danilo: And the initial support in the Linux kernel, build system integration, and so on.
Danilo: And I think the initial lift there
Danilo: was to convince people that it's worth adding it to the Linux kernel.
Danilo: There were approaches adding a second language into the kernel in the past.
Danilo: I think C++ was tried a couple of times, and it failed badly.
Danilo: So adding a new language to the Linux kernel is definitely not an easy thing to attempt.
Danilo: And they were successful. So I think this milestone was really one of the big ones.
Matthias: Why has Rust succeeded, quote unquote, to be adopted in the Linux kernel and C++ not?
Danilo: So I think that a subset of C++ would have been a great addition to the kernel.
Danilo: Just to name two examples, the first one being inheritance.
Danilo: The C kernel code relies a lot on the inheritance pattern. So getting some language
Danilo: support for that would definitely been a great improvement.
Danilo: And the second one that I think is very, very useful is as simple as field visibilities of structures.
Danilo: So we have the case in the kernel that generic components obviously have fields that are,
Danilo: or should be accessed by design by
Danilo: the users but there is also private fields and what we often end up is that
Danilo: drivers instead of representing their needs by contributions to the common component
Danilo: in the kernel instead abuse private fields of the structure and basically peek into component.
Danilo: Internals, which obviously is not considered by the component maintainers in
Danilo: the end and hence causes problems in maintainability and can actually lead to bugs in the kernel.
Danilo: So those are problems that are solved by Rust as well.
Danilo: And I think overall, Rust is just the much better fit for the kernel because
Danilo: we have much less things that are not a good fit for the kernel.
Danilo: Plus, it has all the additional features that help with memory safety that C++ does not have.
Danilo: However, it also brings some disadvantages with it, which is the common practice
Danilo: in Rust to just unwrap results or unwrap options. So basically operations that panic the program.
Danilo: And in user space, that's the same option in a lot of cases.
Danilo: But in the kernel, it almost never is a sane option because a panic in the kernel
Danilo: is really just the last resort when the system is in a state where it's basically
Danilo: non-recoverable and otherwise you would corrupt memory or in the worst case
Danilo: even corrupt file systems.
Danilo: So this is something that we have to take care of when using Rust in the kernel
Danilo: that we're making it hard for people to accidentally trigger panics in the kernel.
Matthias: So what was your first task when starting with Rust4Linux?
Matthias: Was it a thing that you voluntarily picked up or was there a need at work where
Matthias: you needed to start the Rust, where you needed to touch the Rust part?
Danilo: I think it really started with me taking maintainership of the Nouveau driver.
Danilo: And it's also a result out of the way forward with Nouveau and the introduction of Nova.
Danilo: So with the Nouveau driver, we have a lot of issues which are...
Danilo: So back then when I started at Red Hat, I started working on a component that's
Danilo: named GPUVM for the DRM subsystem.
Danilo: I mean, it's basically managing the GPU's virtual address space and doing some
Danilo: more things in the context of providing
Danilo: helpers for drivers to implement Vulkan-compatible user space APIs.
Danilo: So APIs that are given out to user space drivers, for instance, Mesa.
Danilo: So the work of GPU VM I did in the context of adding a new UAPI to Nouveau,
Danilo: which is Vulkan-compatible.
Danilo: And in order to get that going in Nouveau, we needed to do quite some changes
Danilo: on how the page table management of the GPU itself works.
Danilo: But it was never possible to do it to the full and correct extent.
Danilo: We still have problems there.
Danilo: And that's because the design of how Nouveau is written just doesn't really
Danilo: work out well for this kind of requirements that we get from a Vulkan-compatible UAPI.
Danilo: So those design reworks would be a lot of effort,
Danilo: especially considering that Nouveau
Danilo: supports a huge range of GPUs and GPU architectures and generations.
Danilo: So you can basically split them up into the generations before NVIDIA introduced
Danilo: GSP and after NVIDIA introduced GSP.
Danilo: So GSP means GPU system processor, and it's a firmware processor that lives within the GPU.
Danilo: And instead of programming the hardware directly, if you have a GSP GPU,
Danilo: you can just talk to the firmware through a ring buffer and a firmware-specific
Danilo: protocol and instruct the firmware
Danilo: to do the things that you would usually program through registers.
Danilo: And by reworking all those layers to get a Vulkan compatible UAPI we would need
Danilo: to consider the whole range of GPUs, the ones pre-GSP and the ones since GSP introduction,
Danilo: so that would be a lot of effort just to rework something we also have other
Danilo: problems with and the other problems we have in Nouveau are.
Danilo: How well it's documented and how
Danilo: it's designed so Nouveau consists out of lots
Danilo: and lots of v tables with callbacks that
Danilo: are not really documented in terms of ownership
Danilo: and lifetimes of the objects that are allocated returned and
Danilo: and handled which is which
Danilo: is one of the huge problems and one problem
Danilo: that is really important to me personally to
Danilo: address is that Nouveau
Danilo: is just not really accessible to users we don't
Danilo: have a lot of contributors and i
Danilo: think the reason for that is because just no one really understands how it works
Danilo: and doesn't get comfortable working with that driver and for an open source
Danilo: project that's not really what you want so what we,
Danilo: there ended up with was to decide at a certain point we should do a new driver
Danilo: we should do a driver that is gsp only and if we do a new driver now we saw
Danilo: that Rust4Linux is progressing.
Danilo: We thought okay if we do a new driver now we have the choice do we do another c driver,
Danilo: or do we do a Rust driver where the language solves a lot of problems for us that DRM drivers have.
Danilo: So DRM drivers, GPU drivers are usually very, very complicated,
Danilo: suffer from race conditions, suffer from memory issues and just because they're
Danilo: complicated and it's hard to get it right.
Danilo: And Rust helps a lot with that. So we decided to go for a Rust driver and or
Danilo: actually first step evaluate if we want to do a rust drive and this is where
Danilo: i got in touch with rust in the kernel the first time because i started doing this
Danilo: evaluation of is a rust driver,
Danilo: for the successor of Nouveau doable is it is it reasonable and what are the
Danilo: things that we need to consider what are the things that we need to do to get it going.
Matthias: And so at this point in time you're
Matthias: thinking is rust a better choice for writing a completely new driver for nvidia
Matthias: graphics cards for the linux kernel and moving away from the legacy codebase which
Matthias: was Nouveau back in the day i think Nouveau not even sure when it started but
Matthias: it's been around for a while right.
Danilo: Yeah i think it's it's around for 15 20 years.
Matthias: Yeah okay so that
Matthias: must mean the code base is relatively evolved it has evolved and it's probably
Matthias: complicated to maintain i'm not sure about the status but i'm just guessing
Matthias: that with 15 years or 20 years of development time it might not be the easiest
Matthias: codebase to work with especially for beginners yeah.
Danilo: It absolutely grew over a long period of
Danilo: time and yeah so the lack of documentation and the fact that Nouveau was maintained
Danilo: for a long time by a single person and all the knowledge about how things are
Danilo: meant to work within the driver only really exists in the head of this person
Danilo: doesn't really help either.
Matthias: Okay, but when you describe it like that, it sounds really daunting to start
Matthias: a new driver because you kind of need to get that business logic,
Matthias: port it over first to a new language,
Matthias: second from an older code base, which maybe you are not 100% familiar with.
Matthias: I'm not sure. Maybe you were. Yeah.
Matthias: There might be a lot of internals or assumptions or implications in the code,
Matthias: which will make that port harder as well.
Matthias: And on top of it, you also have to understand the structure of the Linux kernel
Matthias: itself and how subsystems and drivers interact and so on.
Matthias: So it sounds like a very challenging task.
Danilo: Oh yeah, it is challenging, no question. But it's not like we're potting over the logic.
Danilo: The good thing is that we're starting from GSP only,
Danilo: which means that not only, but to a large extent, we really have to understand
Danilo: how the firmware interface works.
Danilo: And this is absolutely doable and doesn't really require to understand every
Danilo: single bit of what the Nouveau code does.
Matthias: And so you can start to port over that part alone in isolation.
Danilo: Yeah, so what we went for is we basically split the Nova driver into two separate drivers.
Danilo: One of them basically provides, which we call NovaCore,
Danilo: provides the hardware and firmware abstraction layer, which is needed not only
Danilo: by Nova DRM, which is the second Nova driver, basically, and implements all the DRM parts.
Danilo: So where DRM is direct rendering manager, it's the subsystem in the kernel that
Danilo: handles GPUs and accelerators.
Danilo: So this is then the second driver and it sits on top of NovaCore and it's,
Danilo: utilizes the hardware through this abstraction layer this
Danilo: is necessary for two reasons the first is that
Danilo: there is also the nvidia vgpu
Danilo: driver which is currently making its way upstream which
Danilo: is a virtualization layer or actually it's
Danilo: only really the manager of the virtualization layer because
Danilo: it manages the pci virtual
Danilo: functions so the gpu has has one
Danilo: physical function that is exposed through
Danilo: to the system to the host kernel which is
Danilo: where a normal driver would sit on top of but additionally
Danilo: there are protocols in the pci layer that allow
Danilo: the gpu to expose also
Danilo: virtual functions which looks like as if
Danilo: they were additional gpus but actually they
Danilo: are virtualized within the graphics card and vgpu
Danilo: manages those through nova core by utilizing nova core to expose those virtual
Danilo: functions to virtual machines and then virtual machines can run nova core again
Danilo: and then Nova DRM on top to actually expose the full graphics stack.
Danilo: So that's one reason why we have this driver split, because we have basically
Danilo: multiple clients for the firmware and hardware abstraction layer that lives within NovaCore.
Danilo: The second reason is the firmware API that is exposed by the GPU,
Danilo: is not stable, or at least we cannot rely on this API to be stable.
Danilo: And that's one of the things where Rust helps a lot as well,
Danilo: because abstracting the firmware API is much easier with the powerful Rust type
Danilo: system and things like PROC macros compared to C where we,
Danilo: in the end, would need to build endless VTables to differentiate between the
Danilo: different firmware versions And then you can build more VTables to gather the
Danilo: common code that's maybe similar between certain versions,
Danilo: which becomes a huge mess and is one of the reasons why Nouveau is also not sustainable.
Matthias: And in Rust, that would be a proc macro, which could be expanded,
Matthias: which might end up being the same Vtable, but you don't have to write it yourself.
Danilo: Yes. Yes.
Matthias: Now, a bit of a silly question, but bear with me for a second.
Matthias: How do you even start with developing such a driver in the kernel?
Matthias: You can't just possibly say cargo new.
Matthias: How do you initiate a new driver?
Danilo: From the technical side of things, it's very easy because the major work of
Danilo: integrating the Rust compiler into the Linux kernel has already been done before
Danilo: I joined the Rust4Linux project.
Danilo: And this is also what the Rust tree in the kernel is actually about.
Danilo: It's about compiler and build system support.
Danilo: It carries a lot of other things and carries a lot of other base infrastructure
Danilo: that we have in the kernel as well.
Danilo: But its main focus is compiler and build system.
Danilo: So you basically create a
Danilo: new driver the exact same way as you would create
Danilo: a new driver in c in the kernel which
Danilo: is you create a kconfig file create a
Danilo: new config option for the kernel to include this new
Danilo: component which is i don't know 10
Danilo: 15 20 lines of of code most of
Danilo: that being just module description and then
Danilo: you have a make file where you add the the
Danilo: core rs file which then pulls in all other modules to the global variable that
Danilo: grabs all the all the source files for the cabled build system and that's it
Danilo: and then it builds so this part is is actually pretty easy We're not using Cargo.
Danilo: So we're integrating the Rust compiler into kbuild.
Matthias: Okay, but then you have a driver which does essentially nothing.
Matthias: You still have to communicate with the outside world. How does that look like?
Danilo: Exactly. So then you basically have a driver stub.
Danilo: Actually, you don't even have a driver stub. The only thing you really have
Danilo: is a so-called kernel module, which has an entry point and an exit point,
Danilo: which is the init function and the exit function.
Danilo: That's it. In the Rust abstraction, it's basically an init function and the
Danilo: exit function is basically just drop off the module structure.
Danilo: But that's all you got. Now you need some driver infrastructure.
Danilo: And that's where the first challenge was for the Nova project because a lot
Danilo: of work has been done before I joined the Rust4Linux project.
Danilo: Like the initial integration of the Rust compiler into the build system,
Danilo: module support, lots of generic infrastructure that you need.
Danilo: So abstractions for handling C reference counted structures on the Rust side, for instance.
Danilo: Some locking stuff was there. So really just the absolute fundamental basics.
Danilo: But there was no driver infrastructure. So you weren't able to write drivers.
Danilo: So the first thing that we actually needed to come up with was core driver infrastructure,
Danilo: which is the representation of a device, representation of a driver,
Danilo: representation of a bus, and all the glue code to connect things together.
Danilo: But i think before i talk more about that we would
Danilo: need to go one step back and
Danilo: talk about what we call abstractions in
Danilo: Rust4Linux right yeah so the
Danilo: approach we we take and what rust focuses on
Danilo: in the kernel is really drivers this is
Danilo: where we can get most of the value of the language in
Danilo: the short term i think also because
Danilo: of the architecture of Linux. Linux is a monolithic kernel
Danilo: and therefore all drivers also
Danilo: run in kernel space and if a driver messes things
Danilo: up it breaks the whole machine right
Danilo: so if we are able to provide support for writing drivers in rust a huge part
Danilo: for safety and and reliability of the kernel is is is achieved also if we consider that.
Danilo: Subsystem code, so really kernel core code, is reviewed and tested much more
Danilo: thoroughly than driver code is.
Danilo: So the most value in the shortest time really get from drivers.
Danilo: And the mechanism that people have worked out to interface with C infrastructure
Danilo: is to write so-called abstractions. So we're not directly calling C functions
Danilo: from Rust because that would be fundamentally unsafe.
Danilo: Because then we would need to deal with raw pointers all the time.
Danilo: And then we wouldn't really get a lot of the benefits from Rust.
Danilo: So what we're doing instead is we're building abstractions, which means that
Danilo: for a C structure and its corresponding functions to Rust.
Danilo: Achieve the functionality that's intended by the structure.
Danilo: We write a Rust structure that in some way embeds the C structure or embeds
Danilo: a pointer to the C structure. It really depends on the actual case.
Danilo: And then we write some Rust code around that API. So you can only use that API in a way that is safe.
Danilo: So, for instance, if you obtain a device, a representation of a device in the
Danilo: system, you would not get a pointer, but rather a reference to the device.
Danilo: And instead of now dealing with only the reference of the device,
Danilo: you also have certain device states. So, in Rust, that's just type states.
Danilo: And in the kernel, a device can be in multiple states,
Danilo: and we represent that by type states, and then only allow the functions that
Danilo: are applicable for a certain state if the user can or if the user calls it on
Danilo: the abstraction type with the corresponding type state.
Matthias: Okay, that sounds super cool because type state is a very nice way to model state machines,
Matthias: obviously, but I don't know if you had an equivalent on the C side before or
Matthias: if that was a new integration or if that was a new abstraction that you added in that process.
Danilo: So on the C side, if we stay at the device example, that's not existent on the C side.
Danilo: On the C side, it's really just a raw struct device pointer,
Danilo: and it's on the user to use it in the correct way.
Matthias: Okay. Well, I made it sound like it was an obviously good idea,
Matthias: but maybe we can also take a step back here and maybe describe what is the benefit
Matthias: of the type state pattern, and maybe even what is it in the first place.
Danilo: Yeah. So let me explain real quick how device, driver, and bus fits into the bigger picture.
Danilo: So if you want to write a driver in the kernel, you have to implement the driver
Danilo: structure of the particular subsystem. Let's say PCI.
Danilo: So you implement the PCI structure, which requires you to implement a couple of callbacks.
Danilo: Lots of them are optional. Some of them are mandatory.
Danilo: The two most interesting callbacks are probe and remove.
Danilo: The probe callback of the driver is called once the bus, in this case the PCI
Danilo: bus, detects a device in the system that matches the driver.
Danilo: So the bus basically detects there is a new device and then it checks what is
Danilo: the vendor ID of the PCI device, What is the device ID of the PCI device?
Danilo: It checks a couple more things, of course. This is a bit oversimplified.
Danilo: But then it looks for a driver in the system that is registered that matches those attributes.
Danilo: And if it finds a driver in the system that matches the attributes,
Danilo: it calls the probe function that the driver registered and passes in the PCI
Danilo: device structure in the probe function.
Danilo: And then the driver can start operating the device.
Danilo: It can obtain device resources such as I.O. memory or interrupt handlers and
Danilo: just do its business logic.
Matthias: It gets a mutable pointer to that device, or what would be the input?
Danilo: So in C, it's really just a pointer. If we're in Rust, it's a reference to a
Danilo: PCI device structure with a certain type state.
Danilo: So for probe, so actually for all the callbacks that come from a bus, so like probe, remove.
Danilo: So remove is obviously when the driver is unbound from the device,
Danilo: when the device is unbound from the driver this way around.
Danilo: Um, but in the bus callbacks, it gets the type state core.
Danilo: So then the device has the core type state, and then you have access to certain
Danilo: functions of the PCI device that you would otherwise not have.
Danilo: And that's important because in the bus callbacks, a global bus lock is taken,
Danilo: and that allows you to modify certain fields within the device structure that
Danilo: are needed for instance to,
Danilo: enable i o memory in the first place or bus mastering and,
Danilo: So by having this type state, we can ensure that those functions that are only
Danilo: allowed to be called from bus callbacks are not called from anywhere else.
Danilo: And the way they could be called from anywhere else is because device structures
Danilo: are fundamentally reference counted in the kernel.
Danilo: So everyone can hold on to a reference of that thing.
Danilo: So it is theoretically possible, but in Rust, it's not.
Matthias: Yeah so a simplified version of that would be you
Matthias: get a device you are allowed to call a
Matthias: couple of methods on it which give you a
Matthias: new state of that device a thing that you
Matthias: can hold on to but then crucially you cannot call
Matthias: methods that would be invalid or you try
Matthias: to make invalid state irrepresentable this way
Matthias: yes that's pretty cool it feels
Matthias: like the type state pattern is a very core part of the abstractions that you
Matthias: needed for kernel development from rust are there any other components that
Matthias: you needed to build out which come
Matthias: to mind anything that other systems programmers could also learn from so.
Danilo: I think one example may be how we deal with reference counted structures on the C side.
Danilo: So in C, reference counting works that structures embed a struct kref,
Danilo: which is in the end an atomic counter that gives you a release callback once
Danilo: it drops to zero, and then you have to implement a release callback in a certain
Danilo: way to clean up that structure.
Danilo: And this is a pattern that we have very often. Things in the kernel are very
Danilo: often reference counted.
Danilo: And in C abstractions, we had to deal with that.
Danilo: So what people did were writing a trait that is called always ref counted,
Danilo: which implemented by the corresponding Rust structure representation of the
Danilo: reference counted C struct,
Danilo: just provides common helpers for reference counting and a common interface for reference counting,
Danilo: which is the a ref type with
Danilo: with a generic so you basically
Danilo: have your rust abstraction structure let's say device and since
Danilo: the device is fundamentally reference counted you then end up with an a ref
Danilo: device and then this a ref knows that there's always reference counted implemented
Danilo: for the device because it's a trade bound and then we can we can make those parts common.
Matthias: Well typically in a pure rust
Matthias: world you would probably lean into ownership a little more like a lot of things
Matthias: just move between different components in rust but i'm assuming that that will
Matthias: be very hard to do in an inux kernel because the rest of the code expects references
Matthias: and passes references around or ref counted structures to be more precise.
Danilo: We try to avoid reference counting if it's not absolutely necessary.
Danilo: But if the C structure already does it, there's usually a very good reason for it.
Danilo: Um, so this is something we didn't just have to adapt to.
Danilo: I may have one other good example that is worth mentioning in this context,
Danilo: which is you mentioned in Rust, you usually use move semantics and try to not
Danilo: reference count things or put things on the heap, um, for that.
Danilo: And this is another implication of writing abstractions to existing C code,
Danilo: which is a lot of the C structures and a lot of C code is actually self-referential.
Danilo: Linked lists in the kernel are self-referential.
Danilo: Lots of logs are subsequently self-referential. So this is why one of the Rust4Linux
Danilo: team members came up with a solution for that, which is called pin-init.
Danilo: And it basically does what the name implies,
Danilo: which is in place initialization and
Danilo: pinning so i don't
Danilo: know if you heard about pin-init but there is also a user space crate so so
Danilo: it's actually his name is Benno and he maintains pin in it in a user space crate
Danilo: that can be used there but also in the kernel but i think it originated from
Danilo: the kernel so moving things around is oftentimes not a possibility because of that,
Danilo: So pin in it is a great example of one of those common problems that had to be solved.
Matthias: From what we discussed so far, it feels like you built very solid abstractions,
Matthias: but mostly just to bridge the
Matthias: world from Rust to sea and also bridging the safe and the unsafe world.
Matthias: But I did wonder, was it worth it?
Matthias: Are there any improvements now in working with kernel drivers now that we have Rust support?
Matthias: And was all of the investment into Rust worth it? Is it more than just safety?
Danilo: Absolutely. So maybe to go one step back to the abstractions first.
Danilo: So writing those abstractions is really the absolute crucial part.
Danilo: And it's very, very difficult.
Danilo: And I see people who know the kernel very well, who start getting a good idea
Danilo: of how Rust works, really having trouble to get those abstractions right.
Danilo: Because this translation layer is really the hard part. now
Danilo: for drivers it's still very very
Danilo: worth because you get from the point where your
Danilo: driver so multiple things the first is
Danilo: where usually your development cycle
Danilo: would look like i write a piece of new code in my driver i compile it i deploy
Danilo: it to some test machine i boot the machine up and i get a kernel panic with
Danilo: a random page fault and then i have to take the stack trace and try to figure out what happened.
Danilo: That's what usually happens. Usually you don't write Z code that just works in a new driver.
Danilo: This now went to, once it compiles...
Danilo: Works in 99% of the cases to a point that it at least doesn't fold.
Danilo: Maybe it doesn't work semantically as you intended it, but it doesn't give you
Danilo: all the hard debug problems that you have when you do it and see in the first
Danilo: place because the compiler would complain first.
Danilo: So this is really the goal of the abstractions to get that away and to get the
Danilo: memory safety here, which I mean, this is kind of the memory safety part,
Danilo: but it's an implication of the memory safety part, Another aspect,
Danilo: which is, I think, the second one that's very important is we can use the powerful
Danilo: Rust type system not only to ensure additional safety and correctness to a certain kind,
Danilo: but due to the limitations of an API we can also impose on users,
Danilo: we can guide the user into the direction of using the API in the correct way.
Danilo: Where in C, you're basically free to do whatever you want. You allocate a structure
Danilo: and then you call random functions on it that you just find in a header file.
Danilo: You can design things in Rust in a way that they are used in the right way,
Danilo: that are used in a semantic way that makes sense. So you can encourage good
Danilo: code just by designing your abstraction in that way.
Danilo: And this is another big advantage that I really see from Rust.
Danilo: There are a lot of other advantages, I think, as well, which makes everyday
Danilo: work a little bit easier.
Danilo: We have the Rust format tool, so you don't have to think about how to format
Danilo: code anymore or run a tool that tells you how to do it.
Danilo: It depends on your IDE. But having Rust format is really nice,
Danilo: since a lot of kernel developers are just using Vim and nothing else,
Danilo: and not a fancy IDE that already formats code in a way that you want it.
Danilo: But it's just a minor thing. Another problem.
Danilo: Less minor thing, I would say, is the possibility of writing doc tests.
Danilo: So we have a unit test framework in the kernel that's called kUnit, which is pretty nice.
Danilo: Yet, a lot of code that goes into the kernel does not come with unit tests.
Danilo: Actually, I would say it's not common to send new kernel code with unit tests already.
Danilo: But in rust we have those doc tests and we compile those doc tests that are really just,
Danilo: small snippets of code right we compile them into
Danilo: k-unit tests and then you can enable them with a kernel config option
Danilo: when you boot the kernel and then they are running on boot and you get immediate
Danilo: feedback if something broke your api which was previously only possible if you
Danilo: really write a real k-unit test and insert the k-unit test module and run it
Danilo: so it it is much more overhead than running those doc tests,
Danilo: which give you some immediate sanity check of if your API still does the right thing.
Danilo: So that's another advantage as well.
Matthias: Now, the Rust standard library is split up into multiple subgrades,
Matthias: obviously, and one of them is a no-std part.
Matthias: Can you use the no-std module?
Matthias: Can you make use of the Rust standard library and to what extent or do you really
Matthias: have to use all of the kernel abstractions and you have to bring everything
Matthias: like data structures yourself.
Danilo: So we're using what Rust exposes as the core crate.
Danilo: This is what we use. We don't use any other crates.
Danilo: We had for a while, we had the alloc crate as well in the kernel.
Danilo: But we removed that because it just didn't fulfill the requirements that we
Danilo: needed for the kernel allocators.
Danilo: So one and a half years ago, I wrote the kernel allocator trade and the allocator abstractions,
Danilo: including all the stuff that you need as a basic allocation primitives like
Danilo: you have in Rust's alloc crate.
Danilo: Which is box and vec and those kind of things.
Danilo: So in the kernel, we have kbox and kvec, vbox, kvbox, which is basically just
Danilo: the approbations for the corresponding kernel allocators like kmalloc, vmalloc, kvmalloc.
Danilo: And that was necessary because kernel allocators have more arguments than the
Danilo: allocator API of the Rust allocate crate allows us to use.
Danilo: We have specific allocation flags we have
Danilo: to consider numa nodes and the upstream
Danilo: trade just didn't fit so we had to had to do our
Danilo: own thing so back then
Danilo: when we only or when we had the alloc
Danilo: crate used we only supported the kmalloc allocator
Danilo: so just a symbol so just one allocator and
Danilo: and had a flex extension for it no numa node support but long time it that's
Danilo: we just need to support the other ones as well so that's why i i did this work
Danilo: also in in preparation for for drivers because drivers absolutely need that okay.
Matthias: It's not really a question, but an observation. It feels like you learned Rust
Matthias: in hardcore mode by implementing all of these basic data structures and the allocator.
Danilo: Yeah, that's actually what happened. So the way I learned Rust is probably a
Danilo: way I wouldn't recommend to other people.
Danilo: So what I did is I knew the kernel very well because I'm a kernel engineer for
Danilo: a long time. So I knew a lot of parts.
Danilo: In the past, I've worked at all different kinds of subsystems in the kernel.
Danilo: So I'm more a generalist here than I was an expert for a very specific subsystem.
Danilo: In fact, I'm only working for about three years in DRM and on GPU drivers.
Danilo: All the time before, I was working on various other subsystems in the kernel.
Danilo: But that was to my advantage in this
Danilo: case because i i knew obviously from
Danilo: that a lot about various different areas
Danilo: in the kernel so what i was doing is i took advantage of that knowledge looked
Danilo: at existing rust code in the kernel and knowing what it is supposed to do semantically
Danilo: I basically reverse engineered for myself what the Rust code must be doing here.
Danilo: So this is how I learned Rust and how I approached it.
Danilo: Pretty painful way but it worked out in the end well for me.
Matthias: Yeah true now the
Matthias: community perception of some of the
Matthias: conflicts on the mailing lists and people maybe getting a little angry and maybe
Matthias: some people even leaving the project at some point what's your perspective on
Matthias: this what's your take on this what's the inside perception versus the outside perception here yeah.
Danilo: I i think that hits a very, very good point that is also very important to me.
Danilo: So I saw how those news about some controversies or some discussions that were
Danilo: a little bit controversial on the mailing list went through the news and sounded
Danilo: like if it would have been the biggest drama on earth.
Danilo: But in the end, we have thousands of contributors in the kernel.
Danilo: So you have thousands of different opinions, I think, for such fundamental addition
Danilo: to the kernel, like a new language.
Danilo: It's absolutely normal that people have different opinions and some people have
Danilo: stronger opinions than other people.
Danilo: Some people express their opinions stronger than other people, right?
Danilo: So I haven't seen anything that is dramatic or is unexpected to a certain extent and I think.
Danilo: I feel like it appears to be made a bigger deal out of it as it actually is.
Danilo: Another aspect of this is everyone talked about it and lots of news sites picked it up.
Danilo: But in the end, it was one interaction from thousands of other interactions
Danilo: that went in the absolute other direction.
Danilo: So just to name a few, and I know the terms may not make a lot of sense to a
Danilo: lot of people who are not super familiar with the kernel,
Danilo: but I just want to list a few of them just to give an impression of where things go very well, actually.
Danilo: And we have new contributors that have not been contributing to the kernel ever
Danilo: before who stepped up doing kernel work because of Rust and because they were interested in Rust.
Danilo: We have people that were a long time around in the kernel and are now doing Rust work as well.
Danilo: And between those people who are doing this Rust work,
Danilo: so those Rust contributors and C-maintainers,
Danilo: we have so many interactions where great relationships has been established
Danilo: and where actually ideas or implementations of Rust abstractions brought back
Danilo: improvements to the seaside as well,
Danilo: which are not going through the news pages, right?
Danilo: And just to name a few of them, this is DriverCore, this is Memory Management,
Danilo: PCI, OpenFirmware, ACPI, DRM, Networking, Timekeeping.
Danilo: MISC Device, I2C, the clock framework, PWM regulators...
Danilo: Primitives like maple tree, xarray, work queues, firmware api and and a lot more to be
Danilo: honest where people have established relationships with the maintainers and
Danilo: are working together and we have great interactions get great results and i
Danilo: i think it's it's only fair to mention those as well.
Matthias: Wow i had no idea and you
Matthias: know as someone who does not know anything about
Matthias: the linux kernel but a few things about rust
Matthias: to me it feels enabling as well
Matthias: like as just someone who thinks i might
Matthias: potentially at some point want to
Matthias: dabble with this don't have any plans but it would
Matthias: be more likely now than before because there's the Rust4Linux project you
Matthias: have a website and i checked it out it's very approachable and i'm guessing
Matthias: that the people behind it are so as well and so i feel more welcome than i was before,
Matthias: That's kind of a great achievement as well, besides the code.
Danilo: Absolutely agree.
Matthias: Now, let's say you look forward and you say a year or two from now,
Matthias: we have another chat and you're enthusiastic about what was achieved.
Matthias: What would you be proud of?
Matthias: What's next for the Nova driver and for your work?
Danilo: So as for the nova project one thing
Danilo: that i want to say here is my role
Danilo: so far has been to do all the enablement
Danilo: work i implemented the driver core infrastructure
Danilo: in rust piece the pci bus and and
Danilo: generic io stuff and a lot of other
Danilo: things that i now also maintain and i
Danilo: will keep doing this work because i think it's
Danilo: important it's important for nova but it's
Danilo: also important for Rust4Linux
Danilo: and i also think it's important for the kernel overall so
Danilo: a lot of my time goes into
Danilo: that when it comes to nova one could ask who's doing nova now and i have a very
Danilo: very good answer to that which i'm very happy of and it turned out that a lot
Danilo: of nvidia folks actually stepped up writing code for Nova.
Danilo: Actually, the majority of the code that goes into NovaCore is written by NVIDIA people.
Danilo: And we just got a co-maintainer for NovaCore, which is Alexandre Courbot,
Danilo: who is helping me a lot in maintaining the project and moving the project forward.
Danilo: And so I'm very happy about the NVIDIA folks stepping up.
Danilo: So what I'm,
Danilo: hopefully proud of in a year from now is that
Danilo: Nova actually developed in the direction of solving the problems that we intended
Danilo: to solve when we decided doing a whole new driver from scratch and doing it
Danilo: in Rust is the correct way to go,
Danilo: which specifically means it solves the design problems in Nouveau that we had in Nouveau.
Danilo: It solves the problem of abstracting the firmware.
Danilo: It solves the problem of building a modular driver stack that supports virtualization
Danilo: and a solid compute and graphics driver on the host, as well as in VMs, of course.
Danilo: I'm i'm looking forward to people saying
Danilo: who run an nvidia gpu hey i'm
Danilo: running this out of the box it comes with my kernel installation
Danilo: from my distribution and it's not something that i
Danilo: have to install afterwards from from an out-of-tree source and i mean we have
Danilo: that with Nouveau but but honestly oftentimes it's a bit sad because i see all
Danilo: the effort that went into Nouveau and there were brilliant minds working on Nouveau.
Danilo: So I really want to give my appreciation for the work here.
Danilo: But there is also the reality that oftentimes Nouveau is used to install the proprietary driver.
Danilo: And that's also a bit sad. So I'm looking forward to change that.
Matthias: At least to me and potentially to a lot of listeners.
Matthias: The day-to-day work of an actual Linux kernel developer is completely unknown,
Matthias: and it sounds almost a bit daunting to work on that.
Matthias: What's it really like? What's your day-to-day? How does working in the Linux
Matthias: kernel really look like?
Danilo: Yeah, so there are many different aspects to that.
Danilo: One of it is obviously the maintainer work, which means that I have to review
Danilo: a lot of patches. have to give feedback, but also guide people into the right direction,
Danilo: design-wise for the subsystem or for the component, and ideally serve as a multiplier here.
Danilo: Make sure to get people interested in doing that work in the first place,
Danilo: get more contributors involved, and also scale myself,
Danilo: which means find people to work with who may want to take responsibility themselves,
Danilo: which is, I think, a part that oftentimes is forgotten by maintainers.
Matthias: And how much of that time do you spend in email, in your IDE,
Matthias: and during code reviews?
Danilo: So writing emails and doing code reviews, kind of the same thing.
Danilo: We're doing code review by mail. Patches are sent through email.
Danilo: I would say so roughly about half of the time goes definitely into writing mails
Danilo: and having discussions about code, reviewing patches.
Danilo: A significant amount of time also goes into dealing with patches.
Danilo: It's not only about reviewing them as a maintainer, but you also have to handle
Danilo: them, which means you have to apply them to your tree.
Danilo: You have to sanity check if everything about the patch is correct in terms of process and form.
Danilo: And ideally, you also do a build check, at least.
Danilo: And then you take it in your tree, then it goes into the next tree,
Danilo: which is a kernel tree that is basically capturing the trees of all kernel maintainers
Danilo: who want to be part of Linux Next.
Danilo: And it regularly, I think even nightly, merges all the maintainer trees into
Danilo: one single branch and reports conflicts.
Danilo: Most of the time, the conflicts are already solved by the Linux Next maintainer himself.
Danilo: Sometimes it's not possible that he just drops the branch for the night and
Danilo: comes back to the maintainers asking, hey, what I should do here?
Danilo: What's the right solution?
Danilo: So part of the job as a maintainer is also to deal with that.
Danilo: And of course, you have to send the pull requests either to Linux himself or
Danilo: to the next maintainer in the hierarchy at the end of the cycle.
Danilo: So a development cycle is roughly about three months.
Danilo: Then you send the changes that you accumulated over the time to Linus or the
Danilo: next maintainer in the hierarchy.
Danilo: And then also per release cycle, which is not release cycle,
Danilo: but release candidate cycle.
Danilo: You send the fixes for the last release or after the last merge window for the
Danilo: next release candidates to Linus or the maintainers. as well.
Danilo: And this goes on for the three months and then you have the next release and
Danilo: then this cycle starts from the beginning.
Danilo: Otherwise, yeah, I spend a lot of time in my editor, of course, writing actual code,
Danilo: But there's also a lot of time that just goes into coordinating,
Danilo: coordinating with other developers, coordinating with companies we're partnered
Danilo: with or coordinating with people from the community.
Matthias: And who's next in line for you? Do you send a patch to Greg Kroah-Hartman
Matthias: or someone else in the hierarchy?
Danilo: So that depends. I'm maintaining a couple of trees and a couple of different
Danilo: components or subsystems and drivers.
Danilo: So I'm sending patches to Dave Airlie, who's maintainer of DRM.
Danilo: I do that previously. I did that for the Nova tree.
Danilo: And now I'm doing it with the DRM Rust tree. We have an own DRM tree for Rust components.
Danilo: And what we do different than other subsystems with that.
Danilo: Then I submit patches to, or actually submit pull requests to Miguel,
Danilo: who's maintaining the Rust tree.
Danilo: This is for the allocator stuff that i mentioned previously and i'm also sending
Danilo: pull requests to linus himself for the driver core tree that i co-maintain nice.
Matthias: And how does the tooling look like then you mentioned that yeah well you need
Matthias: to use an email tool and you need to use at least an ide so what's the weapon of choice.
Danilo: Right yeah so this is where really
Danilo: my past of being a
Danilo: C kernel engineer from the get-go kind of shines through
Danilo: which is i'm really just using vim not
Danilo: even new vim just vim and mutt and that's that's basically about it so i i recently
Danilo: switched to aerc for email which is another console client but that's really
Danilo: all i use no no fancy ide no language server nothing and.
Matthias: What about the rust tooling,
Matthias: Do you use any tooling other than cargo format?
Matthias: Is there any kernel-specific tooling as well that maybe you've built or someone
Matthias: else builds which does various random tasks, administrative tasks?
Danilo: No, I mean, the formatting and all the Rust-specific things are kind of built
Danilo: into the cable build system, so you're not really running cargo Rust format.
Danilo: I don't even know if that's an actual command of cargo, but it's integrated
Danilo: into the kernel build system.
Danilo: We have in DRM, we have some maintainer tools which are called DIMM.
Danilo: It's a helper to apply patches, do back merges from other trees, send pull requests.
Danilo: So basically make a couple of maintainer and committer tasks a bit more easy. But other than that.
Matthias: No, just... I find it fascinating how much you can get done,
Matthias: with these tools and then just sticking to
Matthias: them and putting in the hours and putting in the work that's fascinating right
Matthias: danilo if someone is now interested and intrigued to learn more and perhaps
Matthias: even wants to contribute to the linux kernel at some point especially the rust
Matthias: part where would they learn how would they how could they learn more.
Danilo: So I think the best entry points here are the Zulip we have,
Danilo: where the Rust4Linux community is usually around and also the Rust for Rust4Linux
Danilo: team is around and can also give hints on where to start if people have a specific
Danilo: topic of interest they want to work on in the kernel or may be interested in.
Danilo: Otherwise, if you just want to get started doing your first steps,
Danilo: the Rust4Linux project in the GitHub repository maintains an issue list with good first issues.
Danilo: So we try to keep the list huge enough so that everyone has something interesting
Danilo: to find, to pick out of it, where we keep just things around where we know,
Danilo: okay, this would be a nice rework.
Danilo: It's not super pressuring to get it done, but it would be something we would
Danilo: like to see happening and it's a good thing for someone new to start contributing to.
Danilo: Those are the things we keep around there and I can really recommend to have a look at that.
Matthias: That was amazing. Everyone who's interested, please check out these resources.
Matthias: We will also link to them in the show notes.
Matthias: Finally, what's your message to the Rust community?
Danilo: Oh, I don't know if I have the message to the Rust community.
Danilo: I probably want to say thank you because I,
Danilo: Learning Rust and writing Rust code made me a better engineer also for the C
Danilo: parts in the kernel that I maintain.
Danilo: It made me think different about certain problems and just gave me a different
Danilo: perspective of approaching problems, which is very helpful.
Danilo: So my message is probably be keep up the great work.
Danilo: Otherwise, when it comes to the things that the kernel needs from the Rust project,
Danilo: and there are definitely things that are needed,
Danilo: I also want to say thank you for the great collaboration.
Danilo: I'm not that involved myself.
Danilo: I definitely want to make that clear. We have other members from the Rust4Linux
Danilo: team who are regularly talking to people from the Rust core team and from
Danilo: the Rust project in general.
Danilo: And there is great collaboration already ongoing.
Danilo: So, yeah, I'm very happy to see how things go.
Matthias: Amazing. Danilo, thanks so much for taking the time for the interview today.
Danilo: Thanks.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.