Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: I'm your host, Matthias Endler from corrode, and today we talk to Julius Gustavsson
Matthias: from Volvo about putting the good kind of Rust into cars.
Matthias: Julius, thanks for being a guest. Thanks for taking the time.
Matthias: Can you introduce yourself and Volvo, the company you work for?
Julius: I'm Julius Gustafsson, and I work for Volvo Cars, and I am the main architect
Julius: and team lead for the LPA project at Volvo Cars.
Julius: And the LPA, which stands for the low-power processor, is the first ECU in the
Julius: automotive industry, at least as far as we know, that is fully written in Rust,
Julius: and it's rolling off the production line as we speak.
Matthias: The first time i heard about this project
Matthias: was even a few years
Matthias: ago when we were at a conference i
Matthias: can't even remember which one but someone mentioned that
Matthias: volvo was working with rust and by
Matthias: rust we mean rust the programming language of course not rusty
Matthias: oxidation process but was it was
Matthias: way back in the day and i didn't even
Matthias: know what parts of it were public or
Matthias: if that was a separate vendor now take us
Matthias: back to maybe 2018 or so when when this project started what was the situation
Matthias: with rust back then what was the situation especially with embedded rust and
Matthias: how did you even think about using rust at volvo yeah.
Julius: Yeah so so i joined volvo at in 2017 and that's where about the same time as
Julius: as this spa2 project started and and right,
Julius: basically that when i was joining i was also discovering or i was i'd already
Julius: discovered rust a couple years back but i was getting more and more certain
Julius: that this was something that we needed to look into.
Julius: And so we started out with a few products when we were in these very,
Julius: very initial stages of that project. We were doing all kinds of proof of concepts.
Julius: And the first thing I did there to test out Rust was to, since I came from an
Julius: Android background, so I knew the ins and outs of the Android system.
Julius: So I was able to build an Android HAL, an automotive haul.
Julius: In android was able to get that to build and
Julius: link and do all kinds of shenanigans to get that
Julius: to work because the build system didn't really support it but
Julius: i was able to work around that but and and
Julius: so so this android how basically the the
Julius: point of it was to use the android ui to control
Julius: the the ac or the fans in the
Julius: car and yeah and we had
Julius: already built this small system that was running
Julius: on a separate like a raspberry pi that would
Julius: actually send the can signals to the fan so we
Julius: needed to communicate with that to send commands that system basically and and
Julius: so i i used or i created a how that actually communicated with that over grpc
Julius: if i remember correctly using futures and and.
Julius: Back then there was no async await so this was like yeah uh
Julius: futures zero zero point oneer yeah and
Julius: and it was it was such an amazing experience
Julius: because yeah you know you had
Julius: to do all these things to get everything to work because there
Julius: was so many moving parts to get like the things to
Julius: build in a build system and and yeah getting
Julius: the whole like you know from the button to the actual
Julius: methods being called in the hall and then having that send out something over
Julius: grpc over wi-fi to to that other chip and and basically when i had it all wired
Julius: up and it started to build it took me a while the fans just turned on on the first try so it's just.
Julius: Super amazing i've i had never experienced that before and and like yeah so
Julius: that was like a first proof point that this is definitely something how.
Matthias: Much of that was because of rust
Matthias: and how much of it was because of your experience with embedded systems.
Julius: I would say of course my experience didn't hurt.
Julius: But I know for a fact that I've been doing C and C++ for mostly C though,
Julius: but a fair bit of C++ as well for the better part of 20 years.
Julius: And every single time there is always something or at least something creeps
Julius: up, you know, especially when you're least expecting it.
Julius: There is something that you didn't think about or someone else didn't think
Julius: about or something you thought of that wasn't written down and then someone
Julius: else broke that assumption.
Julius: And, you know, there's always something that makes these kind of,
Julius: especially when it's so many moving parts in this little proof of concept that I was doing there,
Julius: that, yeah, that just wouldn't have happened, or at least not for me, that's for sure.
Julius: So yeah rust definitely did the
Julius: did the heavy lifting there or at
Julius: least made sure that i had because rust tends to
Julius: make you you know think through the
Julius: whole design up front basically or at least you know a much larger part so so
Julius: yeah so so there are these fine you know loose ends that you also need to tie
Julius: up before it actually builds and runs and and i think that those are the things
Julius: that make the difference why.
Matthias: Did you decide to build that with asyncrosed given that it was still in an alpha
Matthias: version couldn't you have done something with syncrosed.
Julius: Probably but the the the nature of the since we were doing like g rpc communication
Julius: over wi-fi it was it was very asynchronous in its nature so i wanted to to try
Julius: it out and see how it how it would work,
Julius: so yeah that was that was the idea a.
Matthias: Lot of back-end developers hearing that might be a bit scared by the way async
Matthias: rust worked back in the day where even in back in the day i think you needed
Matthias: to build your own futures was that still the case back then.
Julius: Yeah yeah yeah that was definitely the case and.
Matthias: You still saw the potential there even though it was a bit let's say half baked
Matthias: or it wasn't completely fleshed out yet.
Julius: Yeah absolutely well needless to say then maybe we'll come to that later when
Julius: how we came to the LPA project but where we're not using it at least not yet,
Julius: But yeah, I had done a fair bit of, and also a lot of embedded systems are,
Julius: you know, callback driven in their nature because you are getting all kinds
Julius: of events from hardware peripherals, from,
Julius: you know, other things than your main thread constantly.
Julius: And also you're, you know, when you're achieving something, you usually need to do it in steps.
Julius: So you do something, register a callback, wait for that callback to be called, then you continue.
Julius: And and and this and even in in that stage in that stage of futures at the time
Julius: it was it was still like removing a lot of boilerplate already then so that's
Julius: that's why i thought it was intriguing given.
Matthias: Your background as a c developer and probably also as someone who has a fair
Matthias: share of experience with embedded systems how would you have done that with
Matthias: the technologies that you used before c how would that look like.
Julius: It's a good it's a
Julius: good question i i guess i would i would
Julius: most likely use some sort of async framework in in c++ i guess since the yeah
Julius: we we had gb grpc sorry and and so so that was kind of yeah and and grpc protobuf
Julius: all that stuff was already fairly well supported in C++,
Julius: so I guess I would go for that.
Julius: And I don't remember if those async frameworks at that time in C++ were, how far they'd come.
Julius: They probably had similar sort of features.
Julius: So the problem there is that you
Julius: usually don't, when the closure that you register actually gets called,
Julius: you're basically on your own because there are all sorts of assumptions that
Julius: need to be upheld for that to actually work properly.
Matthias: For instance, which assumptions?
Julius: For example, memory or variables or pointers and things that might not exist when it comes back.
Julius: So yeah, dangling pointers is a common issue.
Matthias: Even for someone like you who has quite a ton of experience in that area?
Julius: I would probably trip on those a couple of times before I get it working.
Julius: But because it's it's so easy to yeah it's it's super easy to to misstep when it comes to those things.
Matthias: So for some context we are
Matthias: in 2018 ish and you built
Matthias: your first hell your first hardware abstraction layer for
Matthias: controlling the fans from android and you
Matthias: saw it working on first try that must have made you pretty excited about rust
Matthias: and its potential to go from idea to first prototype in a reliable manner did
Matthias: you show that prototype to other colleagues did you talk about rust back in
Matthias: the day or was it more of a hobby and side project.
Julius: No i was i was quite probably quite obnoxious i'm pretty sure there were some
Julius: colleagues that were getting quite fed up because i was i was talking about
Julius: it a lot and and i did i remember i did a talk back in in 2017 where or No,
Julius: it must have been early 2018,
Julius: where I tried to do a basic introduction, but also to point out all the upsides.
Julius: And then when this SPA2 project started to crystallize more and we started to
Julius: productify everything that we had in our plants,
Julius: I started looking for places where, or see if this was actually viable as a language to use. And.
Julius: Unfortunately, none of the platforms, so if we go back a little,
Julius: the SPA2 project, it's the electrical system in the Volvo cars that are coming to market now.
Julius: And the main big feature of that system is the core computer,
Julius: which you can probably Google and read about.
Julius: And this core computer is essentially a centralized box with multiple different
Julius: processors, all running different functional domains, doing different things.
Julius: Like you have security, you have high integrity for like braking and steering
Julius: and these safety critical things.
Julius: And then you have a high-performance compute for these more big number-crunching
Julius: things and ADAS functionality, autonomous driving, or that kind of use cases.
Julius: And all these were all different platforms running different operating systems
Julius: or hardware, and none of them actually had good Rust support.
Julius: Except for the security gateway. But we, yeah, we were, or yeah,
Julius: that's a long story why we didn't start off there.
Julius: But yeah, so we had the, yeah, so we had a QNX system and there was no QNX support.
Julius: We have had Tricore based units, which Tricore is a common automotive based microcontroller,
Julius: specifically designed for safety-critical type of use cases.
Julius: And none of these had Rust support at the time.
Matthias: I'm trying to put myself into the position of a team member or a colleague of
Matthias: yours in 2018. So you're telling me about Rust.
Matthias: You're enthusiastic about it. I see the prototype. I see that it's working.
Matthias: But I don't really know. I don't really have any Rust experience.
Matthias: And I don't know how long it took to build that prototype.
Matthias: So I look at all of the tooling that we built in C and C++ and the ecosystem that it requires,
Matthias: and then I ask you about QNX support or tricore support for Infineon CPUs,
Matthias: and you say, well, it doesn't exist or it's a work in progress.
Matthias: Suffice to say, I'd be a little skeptical.
Julius: True, and Volvo is also an automotive or a car maker,
Julius: and automotive people tend to be quite conservative when it comes to you know
Julius: new things we like to use well-proven technologies so yeah definitely but I
Julius: always saw that you know this was actually something different something that,
Julius: could actually provide real value.
Matthias: Thinking about the situation in 2018
Matthias: and 19 when things were not completely fleshed out was everyone on board already
Matthias: or was there some political infighting as well about whether to go forward with
Matthias: rust or maybe be a bit more conservative here because you also mentioned that
Matthias: the car industry is conservative yeah yeah.
Julius: So i wouldn't say political infighting But the one question that always came
Julius: up, and that was, is it safety qualified?
Julius: So is it qualified and certified to use in a safety critical setting?
Julius: And that was basically a no. I mean, nothing had been done on that front at that time.
Julius: So that was kind of a non-starter because most of these functional domains that
Julius: I mentioned earlier these the cpus they are running they have a safety rating
Julius: so to say these acil levels that are that the automotive standards define why.
Matthias: Are they necessary for toolchain to fulfill.
Julius: The safety standards that the automotive industry
Julius: uses basically they they talk about how do you provide the evidence that the
Julius: product that you've developed actually does what you think it's doing and that
Julius: there are no hidden errors or dangers that you could have foreseen and mitigated.
Julius: That's basically the reason why everything needs to be,
Julius: vetted so to say both the tooling but also the code and everything that you that you use.
Matthias: Did you back in the day already know about the efforts of building a compiler
Matthias: safety critical rust compiler that that will be allowed to to be used in such
Matthias: environments like things that ferris systems built for example no.
Julius: So and and first system they actually started maybe six months later or something,
Julius: we met them at a conference in, I think it was OxidizeConf, and they soon after
Julius: announced the ferrocene, or it was called Sealed Rust at that time.
Julius: But yeah, so that started soon after.
Julius: But I always felt that that wouldn't be an issue over time, because obviously
Julius: we are doing safety-critical systems in C and C++,
Julius: and we know all the, you know and it's riddled with
Julius: things that that you can where you
Julius: can shoot yourself in the foot and you need all kinds of
Julius: tooling to basically ensure that you're
Julius: not doing the things that you're not not allowed to do while rust is preventing
Julius: most of those things up front you know just by design you can't do most of those
Julius: things so so right from the so i felt right from the get-go that,
Julius: and it was quite ironic that here all of a sudden you have a language that.
Julius: Basically, you know, does all that stuff for you, but it doesn't have the,
Julius: all the paperwork or all the necessary evidence in place.
Julius: So therefore you can't use it even if it's, or at least according to me, you know, much better.
Julius: But, but, but yeah, at the same time, I realized that it was probably a non-starter
Julius: for anything where safety certification was needed.
Julius: And then we come to basically the fifth processor on this chip,
Julius: and that is the small low-power microcontroller or processor, which is called the LPA.
Julius: And that one was bare metal Cortex-M4-based chip that no one was essentially
Julius: working on at that time because they had some basic software running on it just to power up the thing,
Julius: but otherwise it was being neglected because everyone was focusing on the other SOC.
Julius: And then it turned out that Cortex-M4 was actually the best supported embedded
Julius: target at that time for Rust.
Julius: And the 2018 edition of Rust
Julius: was actually the first edition that made building
Julius: things for embedded possible and stable so so yeah so the the stars aligned
Julius: a bit there so we me and a colleague who who was and is as enthusiastic about rust we teamed up and and.
Julius: And got help from some of our managers who had seen my talk, for example,
Julius: and was also convinced that we needed to really need to look into this for the
Julius: future because we needed, basically,
Julius: we need better tools to get our things done more effectively and more with better quality.
Julius: So we've convinced them that hey we have this chip here that no one is working on yet,
Julius: it's perfect testbed for as a rust project it doesn't have a safety.
Julius: Level on it because it's actually constructed in a way that so we power on the
Julius: we power on the board or this box when certain events happen but we can't power
Julius: them off by ourselves each each cpu has to power itself off,
Julius: and and and since a car in an off state is not considered is it where or it's
Julius: considered safe when it's when it's power off so so going from off to on is
Julius: it's not a safety critical thing but But of course,
Julius: going from on to off, you know, when you're driving, that would be completely, you know, high safety.
Julius: No, exactly. So the hardware is constructed in a way that makes that impossible.
Julius: So the shutdown is done in a safety certified way where we are not involved.
Julius: And yeah, so that was like the third or three things that added up.
Julius: It was it was pretty well supported by rust already it wasn't safety critical
Julius: so we could use it as a testbed and no one else was actually paying attention
Julius: to it so we were able to form a team and and get to work it.
Matthias: Sounds like a good strategy for testing out new things in general and maybe
Matthias: specifically testing out rust in embedded environments or maybe in larger organizations.
Julius: Yeah absolutely and yeah if you have yeah if you have these conditions then it's definitely a,
Julius: something i would recommend one thing we also did from the start was to get help external help so we,
Julius: we got a few people from from grepit in in here in sweden they're placed up in lulio north sweden,
Julius: and many of them are are deeply involved in in the arctic scheduling framework,
Julius: yeah it's not really a real-time os it's but
Julius: but but yeah so so and and
Julius: needless to say we also use arctic in in in
Julius: lpa yeah so so we
Julius: brought on these experts they were the foremost experts
Julius: in sweden at the time and they
Julius: they also they helped us get started
Julius: and and helped us with a lot of the driver development and and things like that
Julius: so so i would say that is also a big piece of the puzzle to get good people
Julius: to help you get started either either from within the company or externally
Julius: depending on what options do you have.
Matthias: Who made the decision to hire externals was it you because you maybe wanted
Matthias: to get an extra pair of hands was it the management was it a requirement from volvo itself.
Julius: It was it was a team together with management we we we also needed to we needed
Julius: to start fairly quickly and we needed to like yeah size up the team fairly fast
Julius: with some knowledgeable people and and and also,
Julius: At the time, we weren't really sure how much effort the driver situation would be.
Julius: So we felt that we needed to have someone to hold their hand there.
Matthias: Did you have a deadline on, for example, you decided that you would be done
Matthias: by that time or you would have something to show by that time?
Matthias: And if not, then the project would be canceled?
Julius: Uh well yeah so they i don't think there
Julius: was there was never a risk of it being cancelled as
Julius: such but but yeah we we wanted to
Julius: we set ourselves a deadline that yeah
Julius: so so like i said earlier we already had some c code running there that that
Julius: was basically produced by by by this or not the stock vendor but the the company
Julius: that made the actual hardware and and And that was doing most of the basic things.
Julius: And we felt that it would be fairly straightforward to revert back to that and
Julius: build that out in case. Yeah.
Julius: So we had always that as a backup. And in fact, we started off working from
Julius: that base and trying to build Rust on top of that.
Julius: But we quite quickly figured that having a 100% Rust-based solution would actually be a much...
Julius: A better choice why yeah because you
Julius: you you don't have to worry about all these ffi boundaries
Julius: between c and and and also when you're building it on on on bare metal embedded
Julius: it kind of depends on what compiler you're using what linkers you're using how
Julius: the because the c libraries they expect.
Julius: Certain initialization to be done in a certain way and so there may be incompatibilities
Julius: there that, yeah, so even though Rust and C can interoperate seamlessly, there are,
Julius: there can be some issues when you're doing it on a bare metal target like this
Julius: and since we were taking us already
Julius: working C project and then adding Rust on top and that was yeah, we,
Julius: It was bringing us more headache than just rewriting that little piece of,
Julius: or those few lines of code that...
Matthias: How long did it take you to rewrite these pieces of code?
Julius: So there was mostly Grepit that did the initial bring up, and it took them
Julius: about a couple of months. They did it over the summer, basically.
Julius: So yeah, when we came back after vacation, They already had a demo of an application
Julius: that did basically exactly the same as the one that we had.
Julius: And when we had that working, we saw that, okay, now there's no point to...
Julius: Carry on with the other one but.
Matthias: Even then going forward did you compare your maybe output with the output of
Matthias: the c-based version in order to make sure that it was compatible or was it mostly you know.
Julius: It's up now.
Matthias: We can compare the gRPC outputs essentially.
Julius: Yeah so now now we're now we're not doing gRPC anymore anymore so so it's a
Julius: different kind of communication but but yeah we We already had started working
Julius: on a CI pipeline with tests and things like that.
Julius: So yeah, that was like the first goal to ensure that the same test suite would
Julius: pass with the new Rust version.
Matthias: Interesting. So you don't use gRPC anymore. What were some of the issues that you faced?
Julius: Well, basically, so that prototype that I did back then, that was basically
Julius: the Android infotainment system,
Julius: communicating with a prototype of this core system that we are now building.
Julius: So there was that communication. But now we are inside the core system and communicating between the CPUs.
Julius: That's on a much lower level. So yeah, we're not doing gRPC on this node.
Matthias: I can understand. Is that a public protocol, an open protocol,
Matthias: or is that a Volvo-specific protocol?
Julius: Yeah, so the one we're using now is actually Volvo-specific.
Julius: The LPA is connected via UARTs to all the CPUs.
Julius: That's the only way to communicate with it and,
Julius: We searched high and low for a protocol that would give us, because in a car
Julius: you need to deal with a lot of EMC, electromagnetic interference.
Julius: And so it's quite common that you get corrupt messages and things like that.
Julius: So we need some sort of protocol that is similar to TCP, but much lightweight,
Julius: of course. It's just a point-to-point.
Julius: But something where you can detect that messages are missing or corrupt or malformed in various ways.
Matthias: Right. And my assumption would be that you had to build out a Rust library for
Matthias: reading and writing that format.
Julius: Yes. Yes.
Matthias: And was it hard for you? Did you use any existing libraries to build that parser
Matthias: or to serialize the messages over the bus?
Julius: So we actually use both, or this protocol allows us to send different types
Julius: of payloads over the, so depending on if the payload is,
Julius: we'll probably get to that later, but we also need to comply with the automotive
Julius: diagnostic standards called UDS, universal diagnostic.
Julius: And so depending
Julius: on if the message is a UDS
Julius: payload then it has certain format that is standardized by that
Julius: standard or if it's like in-band messages between the processors then we're
Julius: actually using CBOR as a messaging format and at that time there wasn't any
Julius: CBOR library for no STD. So we wrote that.
Julius: And then, of course, we used 30 as a,
Julius: which is a super awesome yeah and.
Matthias: What about the automotive diagnostic standard i guess you called it.
Julius: Yeah what.
Matthias: About that protocol was there an existing implementation or was it easy to write.
Julius: It's it's fairly straightforward it's
Julius: basically made for so up
Julius: on up until now the these easy
Julius: use in in cars these electronic control
Julius: units which are the basically the building blocks of of an electrical system
Julius: in a car and those those have existed for a long time they're always increasing
Julius: in numbers now we're up to almost 200 or something and in the current current,
Julius: generations and and and these are traditionally very small microcontrollers
Julius: with very little ram flash things like that so so so that protocol is is fairly
Julius: compact and and simple but but yeah we we needed to write a parser from scratch
Julius: for for that that's super amazing.
Matthias: Because there are a couple other vendors that use rust in production now for
Matthias: example renault which is also a prominent car manufacturer i wonder if they
Matthias: use similar protocols or if they have a different setup i do know that everyone
Matthias: sort of used or uses CAN bus.
Julius: But then the wire protocols might be different right you.
Matthias: Have different requirements.
Julius: Yeah is.
Matthias: There any collaboration going on between those car vendors on.
Julius: A message.
Matthias: Bus level on a protocol level.
Julius: They're not directly at the moment but things are actually they are I guess
Julius: they are improving now with all these different SDV or software-defined vehicle
Julius: organizations that are sprouting all over the place.
Julius: You have Eclipse SDV and you have Covisa and you have others.
Julius: Through those, I hope we can or we will be able to collaborate more.
Julius: But but yeah so no during this project we we didn't have a chance to do too
Julius: much of that we were mostly focused on getting our product out the door are.
Matthias: You planning to open source some of that work and are people ever gonna be able
Matthias: to take a look at this and maybe use it in some way.
Julius: I i hope so so we we already so all the drivers that we developed have have
Julius: been open sourced or or have been upstreamed to the Atzamd project.
Julius: And then we commissioned Grepit to write this MCAN library,
Julius: which is hardware abstraction for this MCAN peripheral that is quite common,
Julius: so it should be available to...
Julius: Or, yeah, if you have a SOC that has that peripheral, then you can use that crate out of the box.
Julius: Then we have a bunch of other things that we've developed that might make sense
Julius: to open source, but we need to look at that more going forward.
Julius: But for example, the UDS stack might be something, or this diagnostic stack.
Julius: We have this seaboard library, for example.
Julius: Now actually we have, there are a few in the community already.
Julius: But yeah, so we have some compression, error correction.
Julius: Yeah a bunch of things things like for example.
Julius: Trace so being able to tag the tests with requirements so that you can you know
Julius: produce reports where you can connect because that's a big part of providing
Julius: evidence that your code actually works.
Julius: Or that the code actually does what it's supposed to do
Julius: rather is that you have you know
Julius: these higher level requirements and then they are broken down into
Julius: you know individual component requirements and
Julius: then they are then there are tests that ensure that
Julius: those requirements are actually being met and and
Julius: then you need to show basically in a in a traceability matrix
Julius: that you know all the requirements are are actually being tested and functioning
Julius: according to you know and And so we've developed a lot of tooling around that
Julius: to show our coverage of requirements and things like that.
Julius: Some of those might be something that might be good to open source.
Julius: Others are fairly Volvo-specific, so they might not be as...
Julius: It doesn't make as much sense, but we'll see.
Matthias: It feels like to be able to
Matthias: pull this off you first had to invent or
Matthias: create the universe because there were so many small little
Matthias: bits and pieces that someone would have to write and
Matthias: you needed to build this entire tool chain so
Matthias: moving forward probably as a
Matthias: next project this will be way easier because you you
Matthias: built out a lot of the tooling but i wondered if the rust ecosystem also helped
Matthias: you so what are some of the things that you used from day one or maybe you even
Matthias: use today a couple of nice projects that you want to mention here um yeah.
Julius: So i mentioned 30 already probe rs is definitely something.
Matthias: Shout out to Noah, who is one of the maintainers of Probe.
Julius: Yeah, they've done an amazing job.
Julius: We actually use, so what we do with Probe RS is we created our own wrapper around it.
Julius: So it's basically a command line tool that is tailored for this particular chip.
Julius: And it does flashing and debugging and setting up all the peripherals in a correct
Julius: way and disabling write locks on certain things so that you can flash or erase
Julius: different things in different orders and stuff like that.
Julius: So we have our own called LPA probe, which is then basically a wrapper around ProbeRess.
Julius: But we also have a Python binding package around ProbeRess so that we can use
Julius: that in our system tests which are Python-based.
Julius: And then you can essentially use it as any other Python library within our system tests.
Matthias: I do wonder, are there any things in Probe RS that are missing right now and
Matthias: things in the embedded or automotive ecosystem that you would like to have reflected
Matthias: in Rust in the Rust ecosystem?
Julius: I don't know about Probe RS for the use cases that we have had. There hasn't been.
Julius: I can't recall anything specific that we haven't been able to fix together with them.
Julius: But otherwise sure there are
Julius: you know we want to make Rust a
Julius: fully first class alternative to
Julius: C and C++ when it comes to safety critical
Julius: and in order to get there we need you
Julius: know the toolchain needs to be certified which is now
Julius: actually available but
Julius: we also need there are still libraries that
Julius: needs certification and we need tooling around
Julius: it for example mcdc coverage and and
Julius: other tooling for for better you know traceability within the to yeah trace
Julius: the requirements in a even better way to get better coverage metrics over you
Julius: know for the code reports clippy for For example,
Julius: it would be nice if Clippy would produce reports, even if you can set it to,
Julius: warnings as errors, it would still be nice to give an overview of what warnings
Julius: you actually have or what remarks it has in the project.
Matthias: Do you have a requirement to create a software bill of materials.
Julius: Yes i was gonna get to that as
Julius: well we we need to we do that as part
Julius: of and that's part of our our build infrastructure
Julius: that we've built out cargo by
Julius: default it it just but yeah it
Julius: helps you to get the total list of everything that
Julius: you're using in in the project but we want
Julius: to know what what are we actually shipping so how much
Julius: of that is actually tooling that is that is never
Julius: leaving leaving the company and what is actually
Julius: the binary that is getting flashed in in the in the car so we had to do yeah
Julius: add some some filtering on top of that to to produce a an s-bomb that that is
Julius: yeah more accurate in terms of what is actually being deployed and.
Matthias: Where do these files end up you certainly use it for internal documentation,
Matthias: but do you also have to hand them out to some other authority which approves certain software?
Julius: I'm not sure, but... They need to be available at request.
Julius: And we have... There's open source portal at Volvo Cars that you can...
Julius: Actually, if you own a car, you're always able to request,
Julius: the license notices basically for for everything and and yeah so so those files
Julius: are used for that but also they're used for internal monitoring for example
Julius: for cargo audit or or other forms of,
Julius: auditing and monitoring to ensure that if there's a vulnerability somewhere
Julius: that we know about it and are able to react to it and things like that.
Matthias: I know from my car that there is a list of tools or a list of software that
Matthias: got used especially i would say maybe larger applications or larger,
Matthias: libraries for example curl is a very popular thing and we had done your stand
Matthias: back in the podcast as well but i wonder how far it goes down because you have
Matthias: dependencies and sub dependencies and dependencies of these dependencies is
Matthias: that all listed somewhere can i access that as a user can i,
Matthias: maybe get the entire cargo tomo of the project or would that be too much so.
Julius: The entire cargo toml would contain much more than we're actually shipping.
Julius: So we're not providing that as part of the car.
Julius: There it's only the software that is actually running on the car.
Matthias: For example, Sardis, would it show up somewhere?
Julius: It would show up, definitely. So yeah, all dependencies that are,
Julius: getting compiled and linked into the binary are would show up.
Matthias: Let's say I started Volvo and I'm in your team and I want to start working on this project in Rust.
Matthias: What would my day-to-day look like? I plug myself into an embedded device? Certainly not.
Matthias: There's probably some other device somewhere I need authentication for.
Matthias: And then we have some CICD process. We have ProBriest somewhere in the tool chain. We have an IDE.
Matthias: And then we have a build process in the end and eventually there will be a binary
Matthias: and a software bill of materials and probably some sort of integration of all
Matthias: the components later on. So walk us through all the steps.
Julius: Absolutely. But first we must talk about the hardware.
Julius: So this core computer that I described in the beginning,
Julius: where we're actually running inside the car,
Julius: that one so when we're
Julius: running there it's it's quite hard to test the lpa and on its own because it's
Julius: it's essentially you know a small island surrounded by a bunch of other processors
Julius: so and and there are not that many connections to the outside world towards
Julius: that one so so yeah if you if you want to.
Julius: Test the lpa individually you would have to
Julius: essentially deploy some sort of distributed
Julius: test program on the other cpus
Julius: and then have them have those test programs
Julius: like probing and prodding the lpa
Julius: and sending back results and stuff and that would be quite cumbersome
Julius: so so we fairly quickly
Julius: we realized that for this to be effective we need
Julius: to have our own development hardware and and
Julius: yeah we sketched out basically a
Julius: block diagram how it would look where basically we take the lpa circuitry as
Julius: it looks on the in the core computer take it over to a separate pcb and then
Julius: we hook up all kinds of components like like a debug probe a usb to can device,
Julius: USB to GBIO devices and current measurements and other things.
Julius: And so what we end up with is a small circuit board with one USB cable and it
Julius: costs a few hundred euros a piece.
Julius: And we make ourselves, or at least Grepit, who are also hardware specialists, they.
Julius: Yeah they basically did it in a just a few few number of weeks we had the first
Julius: samples so so these so we have these in in some or at least big enough quantity
Julius: that every developer can have one,
Julius: at his or her desk and then we have a bunch of those in in ci nice which is based on soul,
Julius: and soul ci where where we then so for every patch that we do it it then runs
Julius: the whole test suite on the hardware.
Matthias: What's so special about Zuul? Because not many people might know it.
Julius: Zuul CI is quite interesting. So I would say its main selling point is that
Julius: it can do speculative merging.
Julius: And what that means is that you can have, because normally especially if you're
Julius: multiple projects working in a common code base you can have,
Julius: one team or one developer doing a
Julius: change and it's working fine and
Julius: all the tests are going through at the same time someone else is doing a different
Julius: patch for something else and that is also fine and going through so both of
Julius: them get merged but it turns out that those two together don't actually work
Julius: so when both of them get merged it breaks,
Julius: Zuul actually is able to speculatively check all the patches that are in flight
Julius: and testing them together before they actually,
Julius: get merged so so you have much higher likelihood of of the master branch actually working.
Matthias: Isn't it also true that we're dealing with embedded
Matthias: devices here i know roughly how
Matthias: that would work for a backend service say but
Matthias: for embedded devices i would be scared that the device would be in a weird limbo
Matthias: state or wasn't completely flashed when the test ran so how do you ensure that
Matthias: the tests are always correct and and start from a deterministic position good.
Julius: Question yeah so so we so the
Julius: actual test framework is is robot framework which is a python based behavior
Julius: driven development type of thing similar to cucumber or or yeah systems like
Julius: that where we where the tests are basically on a system level,
Julius: like you essentially tell statements about how the system should behave.
Julius: And then you connect those statements to actual code that is being run on the target.
Julius: And of course, every test suite then has quite extensive setup loop basically
Julius: where it takes the device and...
Julius: Flashes the correct version, resets it into the correct state and sets up whatever
Julius: preconditions that need to be in place before the test starts.
Julius: And then the same when it's done, it will do some cleanup.
Julius: But that's all.
Julius: So Zool doesn't care about that. It just starts the test framework and it does
Julius: its magic and then it can report back.
Matthias: How long does a typical build take?
Julius: And so on the local machine, it takes about, depending on the machine,
Julius: maybe between five and 10 minutes.
Julius: And then the test itself, I think another 15, 20.
Julius: But then in the CI, we're doing more extensive things. So that is around an hour or so.
Matthias: And that includes setting up all the devices and running the integration test.
Julius: Yeah. Yeah. yeah so yeah around an hour if it starts to start to take more than
Julius: an hour we usually try to sit down and figure out what is if we can parallelize
Julius: it more if we need to add more,
Julius: units to to the ci so that we can run more in parallel or if we if we can optimize
Julius: the test some also that they run does that include.
Matthias: All of the components or does it include just the rust components.
Julius: So it includes all the all of the lpa components yeah and.
Matthias: How often is rust the culprit of failing a build.
Julius: Almost never i mean that that is is since it's mostly taking care of at build
Julius: time so you can't really you know if if it doesn't build you already know that
Julius: in the build step and that's usually something you catch locally does.
Matthias: It mean you don't even flash to the device that often because you have a very
Matthias: iterative way of building the software with your IDE and with the type system.
Julius: Yeah, I definitely would think so that that,
Julius: Compared to another system where less of the things were taken care of up front,
Julius: you would absolutely need to flash it more often just to see that, no, that didn't work.
Julius: So yeah, I would definitely think that that is the case.
Matthias: How big is the codebase, the Rust codebase?
Julius: So the Rust codebase is around 75,000 lines.
Julius: And then we have additional 50,000 lines of system tests and CI and other tooling.
Julius: So yeah, around 125,000, 30,000 lines.
Matthias: And that was developed by how many people? What's the team size?
Julius: So we've been ranging between five and ten developers overall during this three and a half years.
Matthias: Yeah, very nice. A very substantial code base, but also, I would say, still manageable.
Matthias: So I would assume a lot of work also went into specking out things,
Matthias: creating architecture diagrams, making sure that everything is wired up correctly,
Matthias: communication and so on.
Matthias: There's a lot of complexity introducing a completely new tool chain new language everything.
Julius: Absolutely absolutely and and and like i said earlier i didn't even go through
Julius: the whole list of things that we have had to build out as well to get this get this working so,
Julius: so yeah my my team has done an amazing amazing job there couldn't give them enough credit,
Julius: and but but i've also found that,
Julius: using rust or this setup that
Julius: we have is super empowering for
Julius: everyone so they so everyone feels quite confident when they're
Julius: when they're working in the code base that and especially that
Julius: you that you're not as afraid of breaking things
Julius: because the compiler will actually you know
Julius: let you know up front so so you
Julius: know especially when we're onboarding new new developers to the team they are
Julius: usually quite quick to get up to speed because they can hack around without
Julius: fear because as soon as the bills pass and the tests pass and everything,
Julius: then you're fairly certain that nothing weird has been done.
Matthias: How long until they are fully ramped up?
Julius: Everything from two weeks, which was the extreme case, to maybe three to six
Julius: months, something like that.
Matthias: Yeah. So it's a bit of a myth that the Rust learning curve is a huge problem
Matthias: for people that want to work with Rust professionally.
Julius: I would say so. It's absolutely something to respect,
Julius: and I would definitely, like we did,
Julius: we we seeked help and we got some some
Julius: people on board that were really already really really
Julius: good at this i i wasn't you know i wasn't
Julius: super proficient in rust when when we started i had
Julius: done this project and some other smaller hobby things
Julius: and and actually none of our regular developers
Julius: are are i had had worked
Julius: with rust in any big capacity before so we all picked it up on the job and and
Julius: but of course they are experienced you know c and c++ programmers not not everyone
Julius: even embedded so so they've had to learn that as well and.
Matthias: How did you learn how to write ideomatic rust do you have any coding guidelines
Matthias: did you talk to the people from on how to write ideomatic rust are there any
Matthias: resources that you maybe recommended to your team.
Julius: We read a lot of code from the community.
Julius: We also use Clippy a lot to avoid the bigger issues.
Julius: Rust format and Clippy, of course, are mandatory, and we use Clippy-petentic.
Julius: And no warnings allowed, so to say.
Julius: So that gives us a lot of pointers.
Julius: But then, I mean, our code has evolved over time, going back to some things
Julius: that we did in the beginning.
Julius: You can absolutely see that there are some C-isms here and there.
Matthias: When I would be tasked with building code for cars,
Matthias: is I would be extremely defensive because I know that I wouldn't be able to
Matthias: just connect to the car or maybe have a shell connection and just troubleshoot on the live machine.
Matthias: These cars, they run for years and sometimes they have to go without maintenance
Matthias: or people don't do those updates that regularly as they should.
Matthias: So how defensive is the code?
Matthias: How, let's say, painstaking, Is it to work with inputs and outputs?
Matthias: How does the error handling story look like?
Matthias: Would it be normal Rust code that people could understand or is it very specific Rust code?
Julius: There are definitely exceptions, but I would say in general,
Julius: it's very common Rust code.
Julius: The robustness of just Rust in its default form is really high.
Julius: So, yeah, you can't just unwrap all over the place.
Julius: You have to ensure that errors are actually getting handled and logged properly.
Julius: And we have a quite extensive, so we talked about this diagnostic stack,
Julius: and we also have like a framework for creating diagnostic monitors that are monitoring different,
Julius: conditions and and reporting that do.
Matthias: You have a custom panic handler.
Julius: Yes yes.
Matthias: That means failing is not an option you need to handle every condition because
Matthias: if you're the thing that boots things then.
Matthias: It boots.
Matthias: You if you fail.
Julius: Yeah and and of course yeah so
Julius: if if it panics we need to reset back to to a working condition but we also
Julius: use so the chip has a watchdog that we also need to activate so that in case
Julius: it would lock up for some reason there the watchdog would would then you know hard reset it are.
Matthias: There any rules around allocation.
Julius: Yes and no of course there are there are rules about everything when it comes
Julius: to automotive software in our case we don't use any allocation we don't have
Julius: an allocator so we don't use so everything is static that.
Matthias: Means you disabled the allocator.
Julius: Yeah or yeah so we don't use that dependency at all and and the heap is zero
Julius: size basically but that means that we use,
Julius: yeah we use the stack a lot and we use heap less for a lot of these things but
Julius: also So, yeah, we put our buffers and things. They are statically allocated.
Matthias: Because you know the sizes of the payloads that you have to handle at compile time.
Julius: Yeah, exactly.
Matthias: It's similar to how Oxide does it. They have this predictable scheduler,
Matthias: I would say, or part of the operating system, Hubris.
Matthias: And they do something similar where they only have certain message types that they can handle.
Julius: Yeah, exactly the same way. So everything is statically known beforehand or the worst case is known.
Julius: So yeah, we never had to deal with any unknown buffer sizes.
Julius: And then when it comes to input, you asked about input validation.
Julius: That is, of course, super important.
Julius: And that's why it's important also to have a well-defined, so that all messages
Julius: that we are sending are of well-defined format so that you can easily, at the parsing stage,
Julius: see if it's actually valid or not and reject it already there.
Matthias: If someone wants to buy a car that has rust components now, which cars would they need to buy?
Matthias: You said there are production cars. There's probably one Volvo and one Polestar, as far as I'm aware.
Julius: Yeah, that's true. It's the Polestar 3 and EX90, Volvo EX90.
Julius: And then just keep on the lookout for spa 2 based cars and so the more will be,
Julius: coming soon did.
Matthias: You have any issues in production.
Julius: With the with hardware or with well.
Matthias: Let's say both but i'm curious if you sold a car and then it came back because
Matthias: of a rust bug because of an issue in the rust code.
Julius: That has not happened yet is.
Matthias: That a common thing when you ship certain components and they are c-based for
Matthias: example that usually you covered most of the edge cases when things hit production
Matthias: or are there any rollbacks or changes.
Julius: Yeah i i can't really say how
Julius: i would guess it's it's fairly common in
Julius: in automotive in general i mean if you look at you
Julius: know what's called warranty warranty in
Julius: general and and things that are being replaced
Julius: due to that that is quite often software that
Julius: is at fault so so yeah i would
Julius: definitely and and of course it's it's still early time
Julius: so i'm not going to say that that is not going to happen for us but
Julius: but so far it's it has we have seen some some manufacturing issues but those
Julius: have been hardware related so far and and hopefully something that we we have
Julius: or we'll be able to catch in the early stages so to say can.
Matthias: You give us an outlook into the future maybe one two three for five years down
Matthias: the road what is rust usage at volvo gonna look like.
Julius: I'm just gonna hop into my time machine and get back now i would of course love to see it,
Julius: in more places so that is that is what i'm actively working on now to to find
Julius: or to see where it would fit because it doesn't make sense to rewrite or replace,
Julius: everything with rust because if when you're already when you already have the
Julius: thing working yeah and and it actually it's tested and it's according to specification,
Julius: it's seldom that it actually makes sense to rip it out but there are cases where
Julius: that is justified especially when it comes to security critical like cyber security
Julius: critical like user-facing code,
Julius: things that need to do validation of, you know, data that is coming from the
Julius: outside world, things like that.
Julius: So I definitely see a possibility for those kind of use cases where we're interfacing
Julius: with internet, for example, or things of that,
Julius: Now that the hurdles I mentioned earlier, that's the hurdles that we had in the beginning.
Julius: You know, we didn't have support for neither the hardware nor the OSs that are commonly used.
Julius: But all of those hurdles are falling one by one.
Matthias: Now, I find it extremely surprising that a lot of the dependencies that I use
Matthias: for writing Rust and writing Rust in production is the same that you use for
Matthias: an embedded project at Volvo.
Matthias: And this is really cool because you can reuse the code. You can share ideas.
Matthias: You can share different bits and pieces. And the entire ecosystem becomes much
Matthias: better. so you kind of cross different boundaries with rust and how does it
Matthias: look like in comparison between dev and prod.
Julius: Yeah it's funny you say that because there is actually no difference we we build
Julius: one binary and and that is the binary that we test on our test hardware and
Julius: that is the binary that goes unmodified to the core computer.
Julius: And so far, there's nothing that we can't actually test on our dev board,
Julius: or at least very few things.
Julius: There are, of course, few things that change when you're in the context of a whole car.
Julius: But when it comes to the LPA functionality, we can test essentially everything before we deploy it.
Julius: So it's almost unheard of that we get some sort of issues due to the fact that
Julius: something was working differently on our test hardware.
Julius: So, yeah, so one binary for no difference between dev and prod in that sense.
Julius: But what you said with possibility to reuse, and that is definitely one of Rust's
Julius: huge strong points that.
Julius: The fact that you can actually you know take
Julius: different components from the community and
Julius: you can re use them with with confidence
Julius: that you don't really have another i mean both hassle-free almost always it's
Julius: just builds and works and you can just use it and and also there you know all
Julius: these things about that you always have to take care of first you need to okay
Julius: can you get it to build in my build system?
Julius: And first time, probably not. You probably need to spend a lot of time to get that to work.
Julius: But then when it actually builds and you start to use it, does it use memory
Julius: in the same way as you do? Does it make the same assumptions?
Julius: Like who is allocating this buffer? Who is freeing it?
Julius: Who has the responsibility to do what?
Julius: Those are not standardized in
Julius: c and c++ meaning that yeah so
Julius: even if you can build it it's no guarantee that you can
Julius: actually use it in a you know
Julius: in any productive way so but
Julius: in rust on the other hand everything is maybe
Julius: it's not true to say everything but but i mean
Julius: you're you're pretty much guaranteed
Julius: that it will work and and and
Julius: and also like you said i i don't think of i don't think there's any language
Julius: that is as scalable like from the lowest smallest microcontrollers to like backend
Julius: systems or or whatever bigger biggest server farms that you can imagine and
Julius: everything in between that.
Matthias: Sounds really empowering and probably very encouraging for a lot of people who
Matthias: might listen would that be your message to the Rust community?
Julius: Yeah empowering is definitely the key word that,
Julius: We have been super, super happy with this project.
Julius: And it really shows that Rust has a bright future.
Julius: The hurdles that we mentioned before when we started the project and which made
Julius: it basically a non-starter for most of the ECUs that we wanted to use it,
Julius: those are all coming down or have come down already.
Julius: So now you have QNX support, for example. you have Infineon Tri-Core support.
Julius: There was recently an article on how to run Rust together with Autostar Classic.
Julius: Autostar Classic is the common automotive software framework that most automotive software runs in.
Julius: And now that is becoming available for Rust as well.
Julius: There is work being done on various different automotive platforms and components.
Julius: So for automotive, it's definitely a bright future.
Julius: And we'll see where we'll take it at Volvo.
Matthias: And where can people learn more about this project, about Rust at Volvo?
Julius: Yeah, so there are two interviews out, actually.
Julius: One from a couple of years back, where we were kind of in a starting position.
Julius: And you can find that on the Volvo Tech blog. It's called Why Volvo Thinks You
Julius: Should Have Rust in Your Car or something similar, silly pun.
Julius: And then now recently there was an article, it's Tvericholf's blog, where we talked about,
Julius: this this project and and how it has been going yeah.
Matthias: That was a really nice one i list i read this one and also this prompted me
Matthias: to reach out and where can people learn more about yourself.
Julius: You can find me on on linkedin and and
Julius: on x mastodon all these different platforms i'm also I also was fortunate enough
Julius: to join the Rust Safety Critical Consortium that was started last month at RustConf in Montreal.
Julius: And together we are aiming to close the final hurdle, which is to make Rust
Julius: a fully viable alternative to do safety critical software.
Julius: So that is what we're doing now. and you can find more about that at the Rust Foundation webpage.
Julius: And there's also a GitHub repo where you can join if you like. Admission is free.
Matthias: That means people will run into you at some point or another and we hope to
Matthias: see you at a conference or at an event speaking about Rust and Volvo or safety critical components.
Julius: Yeah, let's hope so. So we are at least super stoked about the current project
Julius: and how well it has worked out.
Julius: Definitely going to be pushing that forward within the company and hopefully
Julius: we can find some other exciting avenues for it.
Matthias: Julius, thanks for taking the time and thanks for being an ambassador of Rust
Matthias: in the car manufacturing space.
Julius: Yeah, thanks for having me. It was a great chat.