Rust for Linux with Alice Ryhl and Greg Kroah-Hartman
About saving the Linux kernel with Rust
2026-05-21 49 min
Description & Show Notes
Hot off the press: this episode is a live recording from Rust Week in Utrecht, just two days ago. On stage with me are two people who hardly need an introduction in the Linux world: Greg Kroah-Hartman, Linux Foundation Fellow, stable kernel maintainer and an ambassador for the kernel, and Alice Ryhl, core maintainer of Tokio and one of the driving forces behind Rust for Linux at Google.
I have to admit a bit of personal history here: I first wrote about Greg more than 20 years ago for the German online newspaper Pro-Linux. Getting to sit down with him, and with Alice, in front of a live audience to talk about how Rust is reshaping the most important piece of infrastructure on the planet, was a genuine career highlight.
We get into the big questions: Why does Alice believe that interop, not rewrites, is how Rust wins inside Linux? How do you carefully weave in Rust while maintaining a 35-million-line C codebase? And what does it actually feel like, day to day, to write kernel code in Rust?
“Rust is gonna save the Linux kernel.” — Greg Kroah-Hartman
About Rust for Linux
Rust for Linux is the project bringing the Rust programming language into the Linux kernel. After years of patches, proposals, and heated mailing list threads, Rust is now an officially supported language inside the kernel tree, no longer an experiment. The work spans everything from the build system and the kernel crate to drivers, abstractions over core subsystems and brand-new pieces of infrastructure written entirely in Rust.
About Greg Kroah-Hartman
Greg Kroah-Hartman is a Linux Foundation Fellow, the maintainer of the stable Linux kernel branch, and the maintainer of, among many other things, the USB subsystem, the driver core, sysfs, debugfs, kobject, TTY layer and staging tree. He has been a central figure in Linux for over two decades, has written several books about kernel development, and is convinced Rust belongs in the kernel.
About Alice Ryhl
Alice Ryhl is a software engineer at Google working on Android and Rust for Linux, and a core maintainer of Tokio, the asynchronous runtime that over 50% of all crates on crates.io directly depends on. Inside the kernel she works on Binder, on async abstractions, and on the bindings that allow Rust drivers to talk safely to the rest of the kernel.
About Rust Week
Rust Week is an annual conference organized by RustNL. The 2026 edition took place in Utrecht, the Netherlands, from May 18 to May 23. It features talks, workshops, the Rust All Hands, and expert sessions on a wide variety of topics revolving around Rust. This episode was recorded live on stage during the conference. Thanks to the Rust Week team who made this recording possible! Learn more about Rust Week on their website.
Links From The Episode
- Linux Docs: USB Subsystem Maintainer - Greg’s first contribution led to him maintaining the USB subsystem, and much more
- The Register: Happy birthday, Linux: From a bedroom project to billions of devices in 30 years - An interview with Greg celebrating the 30 year anniversary of the Linux kernel
- Tokio - Another big project maintained by Alice
- RustWeek: Untrusted data in Linux — How Rust is going to save us - Greg’s talk at RustWeek; Rust is gonna save Linux?!
- Rust in Production: Rust for Linux - With Danilo, one of the co-maintainers with Greg on the Driver Core subsystem and others
- Phoronix: New Linux Patch Confirms: Rust Experiment Is Done, Rust Is Here To Stay - The official end of experimental Rust
- Linux Plumbers Conference - A big conference for all levels of kernel developers
- std::boxed - The most basic kind of pointer in Rust
- kernel::list::List - Linux’ linked list Rust binding
- core lib - The most fundamental parts of the Rust libraries
- alloc lib - All things in the standard library that only require an allocator, not used by the kernel anymore
- std lib - The thing most people think of as the standard library, containing things like file access which requires running on a kernel
- QR code generator for kernel crashes - First Rust code added to the kernel
- Linux Rust Architecture support - Missing some big platforms like S390 (IBM Mainframes) and MIPS (a lot of consumer networking hardware)
- sched_ext Schedulers written in Rust - sched_lavd shows promise for video game performance, and servers?
- Aya - Build eBPF programs with nothing more than Rust and the Linux kernel
- RustWeek: Completion-based IO - Alice’s talk at RustWeek about IO
- WE DO NOT BREAK USERSPACE - An e-mail from Linus explaining the mantra in typical Linus fashion
- Linux clippy config - It’s not pedantic!
- Rust code style - Coding guidelines of the Linux project for Rust code
- rustfmt config - Almost vanilla with some ideas for the future
- clang-format config - Added in 2018 and tabs won!
- Coccinelle - Semantic code transformation and the reason Greg lives in Europe
- klint - Custom kernel specific lints, basically a repository of clippy lints for kernel code
Official Links
Transcript
It's Rust in Production, a podcast about companies who use Rust to shape the
future of infrastructure.
My name is Matthias Endler from corrode, and in this very special live episode
at RustWeek, we welcome Alice Ryhl and Greg Kroah-Hartman from the Rust for
Linux project to talk about saving the Linux kernel with Rust.
Welcome everyone to a very special live recording here at RustWeek in Utrecht.
This one has been a long time in the making at
least for me two of my personal heroes
are here today and we will talk about the
rust for linux project rust in the linux kernel
now if some of you have
listened to the podcast before we had danilo krummrich
from red hat on the podcast already
and we talked about the nouveau driver and
yeah we are back with one more episode for rust in production my name is matthias
and i have to thank Alice and Greg for coming over the first question is always
can you introduce yourself and perhaps the organization you're working for Alice
maybe if you want to start.
So i'm Alice i work at google on the android rust team where I work on Rust,
you know, in the Linux kernel, primarily.
Rust on Android more generally. And I also maintain Tokio, the asynchronous runtime for Rust.
And you are known as @darksonn on the internet, if some people remember that.
Exactly, yeah.
I'm Greg Kroah-Hartman linux kernel developer and maintainer i work for the links
foundation for the past many years i maintain parts of the links kernel like the driver core usb,
tty and lots of little things and i also maintain the stable kernels so we do
the stable kernel releases once or twice a week and i'm part of the security
team and part of the cna team for the CVEs and i've been getting involved in the rust stuff as well.
Now the kernel is getting close to i think 30 years of age nowadays which is
incredible and of course it has started as a c project and it has been a c project
for a very long time when was the time when you got first involved in linux in general.
Me i'm late in the late 1990s so only 25 years almost a newbie yeah a lot of
the so no i got involved late 90s i had been using it in work for other things
but i wrote firmware for usb devices and I worked on the USB specification.
And I made devices like barcode scanners and I had to hook my,
write my firmware to work on all other operating systems. So I tested on different ones.
Linux did not have good USB support at the time.
My device was supposed to show up as a keyboard, but if I plugged into Linux,
it showed up as a 102 key mouse.
That was my firmware's bug, but funnily enough, Windows worked fine.
So I fixed the firmware and Linux worked fine. but then I started contributing
to Linux kernel that way because I saw little bugs I could fix and things like
that and got involved and I realized I could get a job doing Linux full time
in the late 90s and haven't stopped since.
Can you say you started contributing to linux because you got curious and you
wanted to tinker with your own hardware and and you started to take on more
and more responsibility in the project.
Yeah i wanted my drivers and it was fun it was a fun feedback loop i wrote my
first driver my wife went away for the weekend said work on that driver that
you want to do i was like okay and i submitted it and i it was for a usb to
serial converter and it was like instantly people wrote back this is wrong this
is wrong this is wrong and oh have you heard of smp multi-processors i'm like
what's that two processors at once?
Locking? I never heard of that. But it was great. It was this feedback loop.
And I learned that the engineering culture is, from an engineer's point of view,
you learn so much from people who are much, much smarter than you.
And I could become a much better programmer.
I'd never learned about multiprocessors and four and 16-wave machines and things like that.
And it teaches you to be a better engineer over time. And I just love that feedback loop.
And it got really, really good after time. Funnily enough, one of my first messages
to the mailing list was like, how do I make a patch?
Cause it didn't, the documentation wasn't there. And the person who responded
to me was very helpful and nice and whatnot.
Like 10 years later, he ended up being my boss at SUSE.
So it's a small world at times.
Wow.
It's pretty incredible. Did you get into any segfaults? Did you have any segfaults?
Oh, machines crash all the time. But that's, I mean, you can crash your own
machine and keep on going and whatnot. So it's not a big deal.
And Alice, when was the first time you got involved in the Linux project or
in the Rust for Linux project?
Oh, I've been there for a long time, like three, maybe four years.
But you worked on Rust as well, way prior to that. So when was the time you started with Tokio?
That's right. So I've been playing around with Rust for maybe 10 years,
but for Tokio, the Async space, I got involved in around 2020, I would say.
And when was the time when you took over the project? Was that pretty much immediately
or was there a transition period?
Well, so, I mean, the way it all started was I showed up in the chat room.
People were asking questions. How do I do this with Tokio?
At some point I was like, I've seen this question a few times, let me go fix the docs.
So that was my first pull request to Tokio that was a docs fix.
And it kind of just kept going and after I don't actually know how long time
it took, but after a while I,
started you know becoming part of the maintainer team and you know it all went from there.
What i find interesting in both of your stories is that
you started taking on responsibility for extremely important technical projects
and both of you came from a position of curiosity you started because you wanted
to learn about that ecosystem and it wasn't a big bang moment or so,
or you didn't graduate from university to become maintainers of these projects,
but it was rather a very gradual process. And I really liked it.
Over 50% of the Rust crates depend on Tokio, and I think everyone in this room has used Linux.
Raise your hand who has never used Linux before.
Because that would be a lie.
Tons of Linux devices in this room alone. The projector. Sorry?
The projector. The projector, even.
And probably the technology for microphones and everything else.
Now, when we talk about Rust for Linux specifically, was that a social challenge
for the most part, or was it a technical challenge to introduce Rust?
I mean...
It's a trick question.
Let's go with... I mean, that was both.
It is both. It is both.
But probably the biggest part was social.
Why is that?
Well, I mean, if you are going to introduce a new language to a project, it's a big investment.
You have to do a lot of work to, you know, you start to use it in one place
and then, you know, and then people have to...
People who have been coding for C in C code all of their lives now have to suddenly
care about this Rust code that's becoming part of their code base.
And so there's a lot of stuff that you have to get buy-in from a lot of people.
It's almost a bit like a leap of faith of sorts. You have to build trust first.
Totally, yeah. Trust is the key here. I mean, technically it was proven five,
eight years ago that Rust could work in the kernel, right? They had examples
out of tree, it could work.
But the kernel works on trust of individuals and we trust people to maintain
the code that they're going to contribute and maintain it.
And we don't necessarily trust that you're going to get it right because we all get it wrong.
We trust that you're going to be there to fix it when you get it wrong,
because we all get it wrong. And that's the trust model.
We need to have the trust built up in the community that we have a big enough
body of people both that can deal with this infrastructure and deal with this
language and that benefits are there. We all agree that the benefits are there finally.
And that we trust the people who are maintaining and working on this that they
will maintain and work on it continue.
What made Rust unique in that sense? Because there were other languages which
were attempting to be adopted in the Linux kernel, which were not adopted before.
I think if you're going to go to the big investment of introducing a new language,
I think it has to be memory safe.
There's no way it makes sense to spend that much time and effort if you're not
going to go to a memory safe language these days.
And it also has to be a language that you can use in the kernel.
I mean, I'm sure Java would be great.
But, you know, there's a lot of languages out there that
are memory safe with a garbage collector or stuff
like that but i don't think any of those would work in the kernel and on the
other side you have all the memory and safe languages and i think rust is almost
the only language that fits it's the combination of things right it has all
the things you need i think.
It was the right idea at the right time.
I think people proved that it could work. I mean, before we always had these
ideas, wouldn't it be great if we could use C++? And then all of us were like, no, please no.
And for specific reasons. So nothing came along.
Nobody spent the time and effort and energy to prove, hey, here is a solution
of some way we can evolve forward. And the Rust developers did that.
And I have to hand it to them for doing that real work. It took them many,
many years, but they were the ones that did the work and proved that it could
be done. And previously, nobody had ever done that.
It's as simple as that. They did the work.
But you could also do the work with another language.
Nobody did, though. Yes, I'm not disagreeing with you, but see,
these people did the work and proved to us that this is a good idea,
and here are the benefits, and they're going to stick around.
This is going to save—I mean, I'll talk in my talk tomorrow about why I think
Rust is going to save Linux in the end.
And I think that's—we now understand and believe that. and they did the work to prove that to us.
It's a big statement to say rust is gonna safe Linux
Yeah,
I agree yeah i gotta talk tomorrow all about that.
Now of course when when you say rust was the first
to stick around and really tackle the hard
problems i agree but at the same time i think rust
is great because it can take a backseat in comparison to other languages like
it can be still c for the majority of the kernel and you can take smaller bits
and rewrite them in rust more or less gradually you don't have to do a big bang rewrite
Interop
Is the new rewrite.
Can you elaborate so.
I mean if you have your massive of code base in one language,
like as you said, you can't rewrite the entire thing.
There's no reason to do that. It doesn't make any sense. So what can you do instead?
Well, what if you add a little bit of the new language, but then you have to
talk, and that's interop is when one language talks to another.
How can you do that from a technical perspective? How do you facilitate the
interop story in the Linux kernel? Where do you even start?
It's hard, and that was my biggest fear. My biggest worry was, you know,
see in the kernel, we have these ideas of how we
do objects how we do memory how we handle this stuff and Rust
says it's very strong ideas of how it does memory and language and objects
and references and stuff and those bindings it's called
a binding is the magic glue right and getting that right is very very hard and
we have very smart people this room and Alice to do that work and without that
and that was my big fear that i don't think this can actually happen until we
prove that those bindings will work and a lot of people spent the time and energy
to prove that it could work.
And we have working systems today as proof of that.
But then you can take those bindings and just say from this point forward,
hey, maybe we only write all the new graphics drivers in Rust,
which the graphics driver maintainers have said.
So then over time, new stuff coming in will evolve different language and we can evolve over time.
The kernel evolves. We don't just do massive rewrites. Every little step of
the way, every tiny change that goes in work on its own. You can't break anything.
So it's slow evolution over time that change.
Will work, and we'll do that.
If you get the interop wrong, developer ergonomics will suffer.
And we'll fix it. I mean, that's the best part of this stuff.
There's nothing in these interoperability... I mean, we get it wrong all the
time. We're human, right?
We can fix it. There's nothing magic here. It's just code.
Are there strategies to get it right the first time?
Sure, we'd love to.
Does it live from a lot of tension between the old world and the new world?
Does it live from a lot of, let's say, inertia to say,
okay, this is how we think about this abstraction in the C world,
and then now we need to build a safer wrapper around this or an interrupt layer
for this to communicate from the Rust side?
Like, on the technical side, how do you do that? Where does the rubber hit the road?
Maybe I can give one example. Let's say you have a C function that returns a
pointer. Now the question is, what are the semantics of this pointer?
So in Rust, there are many different pointer types. We have the reference,
we have the box, we have the arc, we have a bunch of different pointer types.
And so when we write a Rust version of this C function, or like a Rust wrapper
that calls the C function, we have to think, okay, this pointer, what are the semantics?
And then in the Rust version, we have to pick the right pointer type.
And so in this sense, we're adding more details to the signature.
We're saying this C function, which didn't say in the signature,
what kind of pointer is it? Is it an owned pointer? Is it a borrowed pointer?
We have to add that. And so in some sense, we're adding a lot more semantics
in the code that were not there in the C version. And that's the tricky part.
But also along those ways, we've seen APIs and interfaces that we have in C
that are just simple and easy to use in C.
And it turns out it would be very complex in Rust. Like famously in the driver
core, there was some stuff that the driver, people writing the driver core bindings
were going through hundreds and hundreds of lines of Rust code when you could do it in one line of C.
Because, and I said, well, we can change the C code.
And they're like, oh, and that was the big moment that, hey,
if we work together, I can make C code simpler, do different,
and you get to delete hundreds of Rust code.
Great, let's do that. And a lot of the work we've done on the C side,
if Rust were to disappear today, hey, we've made our C code better because Rust has showed up.
We've had to think about how these pointers work. We've had to think about,
is this a mutable pointer or not? Is it const or is it not? How do we work on this?
And we changed the C code in the APIs because we can.
And that's been a huge benefit for everybody. Even if they don't touch the Rust
code, the C code has benefited so much from that.
Hinges a lot on the collaboration of a maintainer of a subproject sure.
But that's fine i mean yes and.
There might be subprojects which are more open to the idea and some which are more resistant.
I know where you're going with it so the big so yes there's a lot of maintainers
that don't want to deal with another language right for various reasons and
that's fine so in the kernel we have the rule that if you bring some code in
and maintainer doesn't want to maintain it you have to be the maintainer of it.
It's as simple as that. So then we have to find a responsible person that we
trust to maintain this code.
So for some subsystems, the maintainer that comes in for the bindings between
the C and the Rust code is a new developer and they will work alongside the C maintainer, right?
For other subsystems, I'll say like the driver core with me,
Danilo came on and he said, I will be the, just maintain the driver core bindings,
right? I said, no, no, no, no. You're going to be the maintainer of all this.
So we share it. Me and Rafael and Danilo share it. We do the binder,
we do the stress side, we do the seaside and we work together.
So it depends on the subsystem. It's just, it's working with people.
It's all just people and interactions.
And so far it's worked out really well. We have some new bindings that are coming
in for one subsystem and a maintainer that is like, I don't even like maintaining
the subsystem in the seaside. You want to maintain them both?
I mean, and maybe the person who's doing the Rust side will say,
okay, great, I'll do that. And you take over and go from there.
So it's just, we'll work through it as each individual subsystem.
And we have a good, I want to say, a very solid mass, critical mass in the kernel
of these bindings from the C to the Rust side to prove you can write real drivers.
And Alice has proven you can write real drivers in code in Rust for a device that works today.
So going forward, it should be much easier, much simpler.
Writing the Rust driver code is much simpler than the bindings.
At least from an outsider's perspective, there used to be some resistance from
some subproject maintainers to give Rust a chance in their ecosystem.
Has that changed to some extent? Have they changed their mind?
Did you see any change in the way Rust is perceived in the Linux kernel by people
who are, let's say, C veterans, very experienced C developers?
I mean, I'm a C veteran. I've been doing this for C for 35 years every day.
People always disagree. We have 5,500 kernel developers every year.
We can only agree on one thing.
And that only one thing is that we want Linux to succeed. After that,
we disagree about everything.
We're people, right? We don't have managers. We have the direction.
But we now have a critical mass of core maintainers. We made a public statement
last year saying, the Rust experiment is over. It's not an experiment.
We're going to do this for real. we accept this as a project,
we're going to go forward with this.
Maintainers don't have to work
with it and they don't have to interact with it if they don't want to.
Just like any individual maintainer doesn't have to accept changes from anybody
else, but everybody can be routed around.
It's ostensibly this very pretty triangle of pyramid of hierarchy,
but if you graph it, code flows in from everywhere, which is good.
We can all modify other people's code.
We can maintain other people's code. We can fix bugs in other people's code
and get the code merge and get it out to users which is what matters in the
end so it's a very flexible,
hierarchy system and yeah some people don't want to deal with it some people
don't want to deal with people who use ai generated patches and they can refuse
those too i mean we have lots of rules of lots of different things not a problem.
Alice do you remember the time when you entered the picture and you started
to build out all of these really great interrupt abilities between Rust and C?
How was that initial phase of building out all of these data structures in?
Things have totally changed, right? So, okay, so one way to see it is there's
this yearly conference called Linux Plumbers where all the kernel developers show up and we talk.
And I've been going there for a few years now.
And every single time I've gone there, things have totally changed from the last year, right?
I've seen this happen like four years in a row or something like that.
So, yeah, I mean, like sure, there are some subsystems left,
But there are definitely subsystems that, you know, I came to them,
we had 0% Rust code, and then we started writing some Rust code.
And then the next year, they were like, oh, I guess you have a cute toy there.
And then the next year, they have Rust code in their subsystem.
And then the next year, it's no longer an experiment.
Why would we not do this at all? I mean, it takes time. It's human behavior, right?
It's just, we're just people. we can't do mandates but
we can persuade and discuss and hopefully convince people
i was convinced i was astonished this is
not going to work at all and then i went to one of the Rust conferences and
i think i had a whole presentation aimed directly at me for like an hour and
a half to prove hey this can be done and then we went out in the hallway and
discussed for another three or four hours and then over drinks that night even
more and i was convinced because.
Of the alcohol or.
Because no no no what.
Was your main concern that people wouldn't stick around until the end.
Well my main concern again was back to this bindings is can we come up with
a way to make these things interact in a way that's going to work good for both
sides and i think Danilo and others have proven that actually it's that interaction
and that binding is going to work better than i imagine so i'll call out they've
done an amazing job there.
Yeah a quick round of applause for that well deserved.
I certainly wouldn't contribute to the C side of the kernel because I would
feel intimidated, honestly.
I would not know if I could contribute in a safe manner, but I would definitely
be interested in contributing to the rust side nowadays.
Have you ever noticed any change in maintenance or maybe contribution behavior?
Are there more rust contributors now in comparison to maybe two, three years ago?
If there are more people yeah for sure yeah absolutely yeah.
And the other side, is that another thing where you say, okay,
if you build Rust bindings for a C subset, you have to have a lot of empathy
for the C code of that subset.
But when you introduce new C code to the library, do you also keep Rust in mind now?
And to say, well, okay, maybe eventually we want to call this from C,
so maybe we build out our code very differently now in comparison to before Rust existed.
Well, yes and no. So we don't try and add code to the kernel that's not used.
So you can't just say, I want to add a binding and here's an interface.
Sometime in the future, somebody can call us from Rust. We try not to add that.
It's been a little chicken and egg problem with a lot of the bindings in the
beginning because we didn't have a real user.
Now we almost have a real user for everything. But when you try and add something, we need a real user.
If you're just adding new functionality to the c side that nobody
in the Rust side is using why would you add the binding because nobody's
going to use it it's not going to be there just add it when it's needed
and away you go just add stuff as needed so you see a lot of the initial bindings
that are coming in for there's tiny subsets of the bindings like look at some
of the stuff for the file systems that you added it was just a tiny subset of
interacting with files for what was needed for one driver and then we'll grow
from there or you're doing adding stuff for netlink,
tiny subset of, we can't really do a full network packet, but maybe we can do
a netlink message that way.
So we'll just grow over time. And again, it's just evolution.
You don't want to add giant chunks at once because it's unreviewable.
And if it's not being used, why is it even there in the first place?
Can you use a lot of the Rust standard library in the Linux kernel,
or do you really have to write everything from scratch?
I remember that the Linux kernel has its own allocator and there's a lot of
emphasis on linked lists, for example, on the C side, but I think there's also
an implementation for it on the Rust side.
Is there a huge overlap between what the standard library provides and what
the Linux kernel needs and where does that overlap end?
So the Rust standard library is split into three parts.
There's core, which is the core stuff, which we do use in the kernel.
Then there's alloc, which is the things that only require an allocator and nothing else.
We used to use that, but we wrote our own now. And then there's the std,
the actual thing most people think of as the standard library,
which provides things like files and TCP stream, But they require that there's
an operating system below you, which there isn't in Linux.
So it doesn't make sense to use std.
Originally, we were using core and alloc, and alloc was our allocator.
Eventually, we found out that the Linux kernel does a bunch of interesting stuff
for its allocators, and so we wrote our own alloc crate, but yeah.
What interesting stuff.
Think about this. Memory needs to be allocated, and you can say, don't swap.
Don't do any I.O. to give me this memory, or give it to me on that NUMA node over there,
or give it over here or give it out of this bucket or that bucket or merge these
buckets and memory is i say it's just a simple driver for memory chips right
but that's a really complex beast underneath there so interacting with that
from the rest of the kernel is not just malloc and free it's free but the malloc
part is a little more nuanced.
That means that allocator is very much specific to the linux kernel itself.
Yes
It doesn't.
Make sense to use it anywhere else? Or is that a standalone library that I could
use for any other purpose?
For now, it's specific to the kernel.
It's built on top of the C side. So I mean, it's built on top of what we have in the kernel today.
So there's just some, when you call and you ask for memory, you have to give
it some hints of what type of memory you want and what you're doing at that moment in time.
That in a normal Rust user space program makes no sense at all.
So you have to actually give it more options, more parameters.
You started to adopt Rust in the Linux kernel on the, I would say,
outskirts or maybe on the driver's side of things.
And I think that was very clever because, well, drivers are isolated,
less risk, less of a blast radius. You can experiment more there.
They don't have to be in tree. You can pull them back in tree if you like to,
but only once they are mature enough.
Do you find that Rust finds its way into the core of Linux now with maybe eventually
replacing this allocator that was written in C with rust.
So actually, drivers...
It makes it sound like drivers are the easiest thing to do for Rust.
Drivers are the hardest thing to do because drivers consume from all the kernel.
So Alice, to write a driver in Rust, had to make bindings to everything,
like to the alloc crate, to do this, to do IO, to touch this file system,
to do a memory malloc, to do all this stuff.
So drivers are just, they look like a leaf on a tree, but they consume from the whole trunk, right?
It's easier to write a core piece of the trunk in Rust.
In fact, the first code in Rust was a standalone bit of code that was added to the kernel.
It was to write, do the QR code when the kernel crashes.
Do that logic. And famously, it had a memory bug.
You can write memory unsafe code in Rust very easily in the kernel.
Memory safety means something different in the kernel. So that standalone piece
of Rust code was actually very easy and simple to do.
It was just one C in and one C function out.
But to do drivers was very much more complex. And we had to agree as a community
to accept this as going to be a possibility even more to do it as a driver.
Going forward who knows we'll see we still don't have full support for all architectures
that we support we saw what s390 got posted again today.
Yeah one.
Of those architectures so we want to get all the main main architectures to
have it before we'll be willing to put a core code to it because we just don't want to break systems.
Yeah the big question is if you're going to have a core piece of logic is i
mean if it's a new core piece then okay but if it's an existing core piece do
you now have two implementations no this is this is great right so the architecture thing is pretty,
important factor here in whether or not we use it in the call right.
Because the allocator is shared between the rust side and the c side and it's
very critical i also thought of schedulers for a moment but again we have schedulers
they do work i don't even know what the latest scheduler is that is used in
the default Linux kernel.
You can write schedulers in BPF. So people are hacking their little steam boxes
to write schedulers from user space to do better framework.
And that same scheduler actually works really well in giant servers.
So it's kind of fun. So you can write schedulers from user space in Rust,
compiled to BPF, injected into the kernel. And people have been doing that for a few years now.
What organizations use Rust for is mostly foundational to an organization.
It's in the core, the thing that cannot fail. This is where you get the most
leverage from the language. It's memory safe.
And this is how you can hook into that system, sort of.
And the thing that, in my opinion, comes closest to that in the Linux kernel would be I.O., I guess.
We have a couple of different I.O. systems in the kernel. I can't list all of
them, but they were poll-based ones. There's IO Uring.
And Alice, you will also give a talk on IO. I'm not sure if that is related
to your work on the Linux kernel, or if you can maybe share what this work is about.
So I have a talk about completion based IO, but that's about user space,
async Rust, and the Tokio side.
Here the question is, it's actually kind of difficult to use IO Uring from async Rust,
because we designed our trades to work really well with ePol and then Linux
came around in 2019 and said hey we have this thing called io_uring which does
not work in the same way and is not necessarily so easy to just swap them around,
so yeah that's what my talk later today is going to be about.
What would have to change in Tokio to support this new I.O. model in Linux better?
So in some places we already have started using it actually.
So if you have a file in Tokio, then you can go through,
then if you enable the AyoUring stuff, then it will transparently use AyoUring
instead of the normal background thread pool to do file IO, which is pretty great.
But it's going to be much more tricky to use it for networking,
for example. Because for networking, Yeah.
Well, because networking is where ePoll is used today. And ePoll is kind of
difficult to translate to our hearing without changing the API we have, right?
And that's the thing, right? In user space, if we change the API,
are people going to use it?
You need some sense of backwards compatibility for people to adopt your new stuff.
That's the mantra of the Linux kernel. Don't break user space.
The kernel is always at fault then.
Yeah, but we have this boundary of what's user space
what's the kernel that's very well defined but within that we change everything
all the time and so those are our apis are internally like you say the binding
we get it wrong the first time great we'll rewrite it and make it work that's
not seen by user space at all so yeah our goal is never to break user space
you should always be able to update to the latest kernel and nothing should
ever break if so then we did something wrong.
Rust could also make that boundary a bit safer.
Has nothing to do with rust sorry that
that's not that's not the the issue
that api of how you enter into the kernel and how you exit back out is language
agnostic let's just say it's just a way of getting data in and out and rust
no language matters there we traditionally have c wrappers but rust has rust
wrappers and go has go wrappers and all etc how that goes.
Now I find it interesting also, Ellis, that you work on both sides.
You work on, say, quote-unquote, a user space application and the kernel space.
And if you find a thing that maybe in Tokio could have been done differently
on the Linux kernel side, is there a way for you to maybe encourage a discussion around that?
To say, oh, can we change this API that would help us on the user space so much?
Or do you strictly separate that work between Tokio and the Linux kernel right now?
So obviously, if I want to make some Linux kernel changes motivated by my user
space work, it would be a lot easier now that I know how the Linux kernel community works.
I don't think we've had user space API changes per se.
We have had a case where we had, I think, a kernel bug, and then I already knew
the person on the Linux kernel side that I needed to speak to,
and then we could figure out what the bug was and get it fixed.
So in that sense, the crossover is quite useful.
Now, looking ahead a little bit into the future of the Rust for Linux project,
what would you say will land in the next 6 to 12 months?
What can people expect there? Or is it mostly going to be fixing small little
paper cuts, making the APIs a little nicer? Or are there any big plans?
I mean, everybody asks what's next in Linux. It's whatever happens to show up.
We don't know. We don't know what people are working on. We have hints of what people are working on.
I have hints of people saying, I'm going to be doing this and this and this.
Like, that's great. I believe it when I see it.
So whatever shows up, we'll deal with and we'll handle it then.
So we have ideas that people are, you see hints of what people are writing new
drivers in that are in Rust.
We have the graphics drivers being written in Rust. We have other things being written in Rust.
Let's see what happens. Let's see what shows up.
, what's on your agenda?
Well,
We have some pretty cool projects ongoing. So actually here at the conference,
we have a booth where you can try to play a racing game on a device running
a Rust GPU driver in the kernel.
So, you know, that shows that it's working. It's very much a prototype at this point.
But, you know, hopefully something will happen with that.
I have one more thing. It might be a little bit political, So just tell me if
you don't want to answer to this,
but are maintainers now inclined to also review the Rust code or is it still
okay to maybe not touch the Rust code whenever you make a patch to the C part of things,
if you don't want to look at the Rust at all?
There have always been some maintainers who are interested in the Rust code
and have, you know, been curious to see, hey, what is this and how does it work?
Remember, we also have 700 maintainers. So to classify everybody is all agreeing
to do the same thing is kind of tough.
We maybe have 150, 200 core maintainers. Even getting those people to agree is hard, right?
So we all work differently. And the goal of the kernel, any change that comes
in should never break the build.
And if it breaks the build on the Rust side, we should be able to fix it up.
I mean, so far, the things that have come in really haven't broken much.
It hasn't been an issue. it really and it's part of that's one reason we agreed
this experiment is over it looks like this is actually working because they
put the work in in the past couple years we haven't had a really big roadblocks
and things seem to be working okay and let's try it.
But if i send in a patch to the c side and it breaks things on the rust side
is is there any discussion about that or.
It depends on the depends on the maintainer if i
even if i notice it i'm like hey you broke that can you fix that so
our build bot might just smack you
down on the on the mail on the thing saying you broke the bill so you can't
do that so like if you break the bill automatically i'm like okay come on let's
figure this out if you say i don't know how to do the rust side great i'll offer
help and things like that i don't think we've ever nobody's ever turned down
help or things like that a lot of people like oh cool i've wanted to try this
rust thing let me try this out.
Ah that's a very interesting way to solve it because you take away this objectiveness
and you make it objective by making it a technical problem you failed the build
doesn't mean you made a mistake or you should have been more careful or why
didn't you look at the rust part.
You just say we all make mistakes right and we
have checklists and that's why we have bots that run on the thing to say hey
you did this wrong that's why we have like some subsystems have ai tools that
are running on patch reviews and say hey that looks good but what about this
what about this what about this it's like back to my very first patch i submitted
i'm like oh i got all those things wrong great That's the technical thing I
got wrong. I'll learn from this and move on.
My next submission go. On average, it takes at least three tries to get a patch
accepted into the kernel.
And we're still running a huge, almost 10 changes an hour for the past 20, 15 years.
Our rate of change is huge. And our rate of actually submissions and numbers
of times that have to get accepted is a lot.
That's normal. That's just how things go. I've had some patch series that's
taken like 25 times to get a patch series in mergeable state.
And that's just normal. On the C side, I don't think we've seen anything that's
really broken the Rust side yet.
What are our Clippy settings?
I don't know. They're there.
Do we use Clippy in the kernel?
Yes, yeah, and they keep breaking. But as a new version of the Clippy, it's fine.
And then the backport, the stable kernels, and I take patches that will fix those up.
Is it already set to pedantic mode?
I don't know.
No, it's not set up. So we enable some set of things, right? And we do enable the...
We don't enable the pedantic group, but we do enable the...
Group of lints. And so when Rust compiler or Clippy adds a new lint in the default
group, then suddenly we have to go update our code because we want it to be lint-free.
And of course, as Craig said, then we have to backport the fix as well because
we also want the code on the stable releases to be lint-free ideally.
But yes, so that's a yes on Clippy.
Isn't it ironic because C developers also, by osmosis, use Clippy now.
If they break the build and maybe the Rust side broke and Rust uses Clippy to
just enforce certain rules, also just the Rust compiler being stronger or maybe detecting more bugs,
does that mean the C code also profits from that ecosystem?
Does it mean the C code over time gets better? because there's more and more
edge cases which get uncovered by the Rust tooling?
Maybe, but we've had static code analysis in the kernel for decades.
And we've fixed up things based on rules and we make sure we don't break the rules.
The new version of compilers come out and they find more patterns and we fix
those again, backboard those.
So we're used to this. This is good code hygiene. We have no objection to this at all.
So if another tool comes along and says, need to fix this up in the C code, great, we'll fix it up.
I don't think clippy is so revolutionary like they have lintz in the c code too i.
Think the thing that's more interesting.
Is the rust format.
Because on.
The c side the formatting is.
Very loose we have rules yes we do but they're guidelines and are not as strict
as the as the Rust i but i like that but we i mean a code formatting is just
there to make things universal and pattern matching you just want to have it
consistent not that you agree with it or whatnot but that's why we have a code style.
And now we have a Rust code style. Great, we'll follow that.
Wonderful, not a big deal.
Yep, just run us through the formatter.
Do you use the default Rust format settings? Will you make any changes to it?
I think there are some very minor changes to the settings, but almost.
Is there even a C formatter?
Yeah. It's been around for decades.
50 years. But it's harder to introduce it than to start with one.
We've had it. We've followed the rules. I gave a talk 25 years.
One of my first talks I ever gave in a public thing was, here's our coding style,
and here's why we have a coding style, and we abide by this coding style.
Yeah, we've had that. And we've had, there's a bad little script in the kernel
that will actually reformat your code to the coding style if you really want to run it.
But it's been around for a long time.
So it is used across all subsystems.
Yeah, yeah, yeah. Any new code coming in has to follow those rules.
We have some really old code in the kernel that doesn't follow the rules,
but that's fine. That's nobody's touching those drivers anyway.
But all new stuff, it's been, again, this is 25, 30 years we've been doing this.
Yeah, that's not an argument here.
It's not like we copied the C code style over to Rust. Like we're not using tabs.
Yeah, I wish. That's fine. Our editors just work. It's okay.
These are common standards for both the kernel side and the Rust side and we
just live with them. It's all good.
I remember that there was a Linux tool which was very impressive. It found...
Code semantically it wasn't a grab it was more of a semantic code search.
Coconut yeah exactly so yes that's what got me to move to europe wait.
What that's a very.
Interesting turn of events so julia luau who's a professor in paris has a tool
called came up with a tool called coconut and it's a semantic tool that does
code transformations and you can write code and rules and it'll, it'll fix problem.
It'll transform the code based on these rules and do that long,
long time ago. So you realize bugs are follow a common pattern, right?
And if you define the pattern, like saying, oh, we forgot to check the error
value of this function, always check that and fix it up. So we do it right.
And then we have these set of rules that you can run them and make sure any
new code coming in doesn't break these rules.
So her work has fixed more security bugs in the kernel than anybody else.
Fixed them, got them out of the code, and it prevents them from coming back
in. Now, it turns out we haven't been running some of those recently,
so some of those bugs are creeping back in.
But there's also a graduate student here. I saw him at the conference.
He's working on that tool for Rust.
He's presented at the Rust conferences a few times, and he's doing some really
good work at Vrye University in Amsterdam here on that.
So that tool is also available for Rust for code transformations of Rust code
based on semantic rules.
And that's been around for a long time, and that's a good thing to be doing for Rust.
And that's all these LLMs do as well. you can say look at this pattern over
here is this pattern match over there and we've been doing that for the stable
kernels for decades again we've fallen all the bug fixes that we didn't forget
that we forgot to mark as bug fixes and now we know how to backport them just based on the semantic,
analysis of the patch and the code it works well in rust now too any.
Other tooling that people might not know that is used to yeah write rust code in the linux kernel So.
There's a pretty cool project called KLint, which is short for kernelLint,
which is a Rust compiler extension, kind of like how Clippy is a Rust compiler extension,
but it's a custom run one written for the kernel, which checks some kernel specific things.
One of the ones that I'm the most excited about, which is still a work in progress,
but there's a lint for...
So in some parts of the kernel code base you're not allowed to sleep.
And then klint has this lint that checks in those pieces of code if you call something that sleep.
And so basically it's a repository with custom clippy lints.
Are these actually clippy lints or is that a separate system well.
It's a it's a program compiler plug-in in the same way as how clippy is a compiler plug-in.
Okay yeah i always find it interesting to cool yeah to to hook into the the
rust kernel like that and extend it like this Because.
It has to do a lot of pretty complex analysis where you run the Rust compiler,
you get all the type information, and then based on that, you can then make the lint, right?
You can't just do this check based on, oh, this word is here.
It requires much more complex analysis. You need to run the compiler to get
the information you need to do the lint. So that's why it's a compiler plugin.
So we have to come to the end. And I thought about the final question for a
long time. Typically, our final question is, what's your message to the Rust community?
But I won't let you off the hook today that easily.
So I will try to reframe it. I will rephrase it. And I have two questions.
In fact, I have one for each of you.
It's the same question, but just with a twist. So who wants to start?
Okay.
Alice, what can Rust developers learn from C developers?
Empathy,
what can Rust developers learn from C developers.
I want to hear this it's.
A bit of a curveball.
That's a hard question.
And your question will be the,
if you like you can also answer first.
Whoever has to answer first.
Put you on the spot there yeah.
You really did.
So i i can answer so rust rust
free so when switching from c to rust and back to c i'm like oh no and c i gotta
okay was this pointer doing who owned this pointer who did this who where did
this go and i have to like mentally think a lot more so many years ago a friend
of mine said you gotta try this rust language. It's really fun.
It makes programming fun again.
It's like, nah, it's fine. I'm fine with C.
And I should have taken him up on it. Many years later, he was right.
Rust to me makes programming fun again. So it makes, it takes the,
I know what logic and what I want to express and how I want to get something done.
And Rust provides me with the tools to know that I'm doing so in a way that
I'm not doing all the stupid things I do in C wrong.
And I don't have to keep track of that it's like okay it compiles logic looks
right okay now i can worry about is my logic right i'm worried about all this
other meta stuff i have to keep in my head as a c programmer so my thing is
so from a c programming point of view learning from the rust developers is hey
thanks for making programming fun again really appreciate it,
you.
Can say the same thing.
You can say working.
With c has made.
Rust fun again.
Programming fun again.
I mean c is pretty fun right you got to do a,
You know, you get to set up all these complex things and keep it all in your
head. But I don't really know if I want to learn from that.
I think part of it is also maintaining a project like Linux for the better half of three decades.
Is that the way to say that? Anyhow, you know what I mean. For a long time.
Because you go through multiple different phases of, you know,
different trends, people using different paradigms and
different tooling and yeah generations of
developers essentially and how do
you make that work like the long-term vision of
such a project that's the most impressive thing when i
think about the linux kernel no matter the language really
maybe in 30 years from now it might be a different language that gets added
but what won't change is how you think about the project from an engineering
perspective and you also think about the users So I think I can speak for all
of us when I say thanks for both of your very important work,
making the world a better place. Thanks. Thank you very much.
Rust in Production is a podcast by Corrode. It is hosted by me,
Matthias Endler, and produced by Simon Brüggen. For show notes,
transcripts, and to learn more about how we can help your company make the most
of Rust, visit corrode.dev.
Thanks for listening to Rust in Production.
Alice
00:01:16
Matthias
00:01:34
Alice
00:01:38
Greg
00:01:40
Matthias
00:02:05
Greg
00:02:21
Matthias
00:03:15
Greg
00:03:26
Matthias
00:04:23
Greg
00:04:28
Matthias
00:04:34
Alice
00:04:40
Matthias
00:04:47
Alice
00:04:54
Matthias
00:05:05
Alice
00:05:13
Matthias
00:05:43
Alice
00:06:52
Matthias
00:06:53
Alice
00:06:57
Greg
00:06:59
Alice
00:07:01
Matthias
00:07:05
Alice
00:07:06
Matthias
00:07:42
Greg
00:07:48
Matthias
00:08:34
Alice
00:08:44
Matthias
00:09:34
Greg
00:09:37
Matthias
00:10:10
Greg
00:10:13
Matthias
00:10:35
Greg
00:10:38
Matthias
00:10:49
Alice
00:11:22
Matthias
00:11:45
Greg
00:11:55
Matthias
00:13:07
Greg
00:13:12
Matthias
00:13:23
Greg
00:13:25
Matthias
00:13:29
Alice
00:13:57
Greg
00:14:57
Matthias
00:15:51
Greg
00:15:56
Matthias
00:16:00
Greg
00:16:04
Matthias
00:17:39
Greg
00:18:05
Matthias
00:19:19
Alice
00:19:38
Greg
00:20:25
Matthias
00:20:58
Greg
00:20:59
Matthias
00:21:02
Greg
00:21:06
Matthias
00:21:26
Alice
00:22:02
Matthias
00:22:08
Greg
00:22:38
Matthias
00:23:44
Alice
00:24:12
Matthias
00:25:06
Greg
00:25:08
Matthias
00:25:39
Greg
00:25:43
Matthias
00:25:44
Alice
00:25:50
Greg
00:25:52
Matthias
00:26:11
Greg
00:26:45
Alice
00:28:01
Greg
00:28:03
Alice
00:28:11
Matthias
00:28:33
Greg
00:28:52
Matthias
00:29:11
Alice
00:29:55
Matthias
00:30:31
Alice
00:30:38
Matthias
00:31:29
Greg
00:31:34
Matthias
00:31:56
Greg
00:31:58
Matthias
00:32:25
Alice
00:33:03
Matthias
00:33:36
Greg
00:33:59
Matthias
00:34:30
Alice
00:34:33
Matthias
00:35:02
Alice
00:35:24
Greg
00:35:33
Matthias
00:36:15
Greg
00:36:25
Matthias
00:36:53
Greg
00:37:10
Matthias
00:38:01
Greg
00:38:04
Matthias
00:38:06
Greg
00:38:08
Matthias
00:38:17
Greg
00:38:19
Alice
00:38:21
Matthias
00:38:58
Greg
00:39:31
Alice
00:39:54
Greg
00:40:00
Alice
00:40:01
Greg
00:40:02
Alice
00:40:04
Greg
00:40:06
Alice
00:40:28
Matthias
00:40:30
Alice
00:40:35
Matthias
00:40:40
Greg
00:40:42
Matthias
00:40:45
Greg
00:40:48
Matthias
00:41:09
Greg
00:41:12
Alice
00:41:29
Greg
00:41:34
Matthias
00:41:45
Greg
00:41:58
Matthias
00:42:04
Greg
00:42:05
Matthias
00:43:35
Alice
00:43:43
Matthias
00:44:31
Alice
00:44:36
Matthias
00:44:44
Alice
00:44:55
Matthias
00:45:19
Alice
00:45:57
Greg
00:46:05
Matthias
00:46:07
Alice
00:46:10
Matthias
00:46:13
Greg
00:46:22
Matthias
00:46:24
Alice
00:46:27
Greg
00:46:27
Matthias
00:47:44
Greg
00:47:45
Matthias
00:47:46
Greg
00:47:48
Matthias
00:47:50
Alice
00:47:51
Matthias
00:48:07