Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we talk to Sebastian Scholz
Matthias: from Gama Space about controlling solar sails and satellites with Rust.
Matthias: Sebastian, thanks very much for taking the time for the interview today.
Matthias: Can you say a few words about yourself and about Gama?
Sebastian: Yes, thank you very much for having me, Matthias. I'm very excited to be here.
Sebastian: My name is Sebastian Scholz, and I'm an electronics and embedded system engineer at Gama Space.
Sebastian: We are based in Paris, and I've been with Gama Space since January 2023.
Sebastian: Our mission, our Gama Space's mission, is to develop and provide a propulsion
Sebastian: technology called solar sails in space.
Sebastian: And to do that, we are using rust in various places. And that's what I'm here to talk about.
Matthias: Now, not everyone might be familiar what a solar sail is. Can you describe the principle behind it?
Sebastian: Yes. So the concept is actually rather simple.
Sebastian: You know, when you have a sail on a boat, you have the wind blowing into it.
Sebastian: In space, you don't have that, but you have the sun. And the photons of the
Sebastian: sun provide some amount of pressure on a sail.
Sebastian: So if you have reflective space, the reflections of the photon give you an impulse
Sebastian: and so provide momentum to your sail.
Sebastian: So basically, it's a way of moving in space, but without fuel and just relying on the sun.
Matthias: Wow.
Matthias: Basically, free propulsion.
Sebastian: Basically, free propulsion. That's exactly it. The main advantage of it,
Sebastian: of combustion propulsion, is that you don't need to bring your fuel with you.
Sebastian: So, as you might know, it's still kind of hard to get mass into space.
Sebastian: So, by saving all of this potential fuel mass, it's going to be easier and cheaper to go into space.
Matthias: And the trade-off is speed, I'm assuming, right?
Sebastian: Yes, of course, there are some downsides compared to a normal combustion engine.
Sebastian: The impulse, the specific impulse that you get is rather low.
Sebastian: So that means the acceleration is extremely low, actually, in the area of a
Sebastian: few newtons versus kilonewtons that you get from normal propulsion.
Sebastian: So that means you have to plan your trajectories accordingly.
Sebastian: But, I mean, the benefit is that this propulsion is infinite compared to the
Sebastian: fuel, because at some point fuel runs out. But the sun, it's going to take a
Sebastian: very long time for it to run out.
Matthias: Unless you're too far away from the sun.
Sebastian: Yeah, but even if you're too far away from the sun, you still get some amount
Sebastian: of propulsion. I mean, it's not like you're on an empty tank.
Sebastian: You still have some amount of movability even if the sun is very low.
Sebastian: But if you're on the outside of the solar system, you're going to have to move
Sebastian: a bit slower. Yes, that's absolutely true.
Matthias: It's not a physics podcast, but at the same time, I'm always curious about space.
Matthias: It's a thing that's fascinated me since I was a child.
Matthias: And a lot of people, they requested an episode with you and with the project
Matthias: that you do, which we're here to talk about, because you also happen to use Rust.
Sebastian: Yes, quite extensively, as a matter of fact.
Sebastian: So for the solar sails, we have launched in January 2023.
Sebastian: So right around the time when I joined the company, we've launched a satellite,
Sebastian: a demonstration mission for this technology, the solar sails.
Sebastian: And the onboard data handling system, our part of the satellite,
Sebastian: is more or less completely written in rust.
Sebastian: So we have a few sealers underneath the hood, but the main code and all of the
Sebastian: supporting test equipment was written using Rust.
Matthias: Now, isn't it extremely risky to use Rust for such a mission?
Matthias: You don't really have that many tries anyway.
Matthias: It's not like redeploying a backend or so.
Sebastian: Yes, there's one of the great dangers if you develop something for space.
Sebastian: You get one shot with your
Sebastian: satellites and if you mess it up it's not like you
Sebastian: could just go up and try to reboot it you know you you
Sebastian: really have only one shot but the mindset that
Sebastian: we have at Gama is the mindset of the new space companies so we try to move
Sebastian: fast and fail fail fast so we try a bunch of new things and if we do fail it's
Sebastian: unfortunate of course but it's not a big deal That's kind of how we do learn.
Sebastian: And so what we wanted to do with this mission, it's a demonstration mission.
Sebastian: So we wanted to demonstrate our technology, but we also wanted to try out new things.
Sebastian: And so using Rust as the main driver for our onboard data handling system was
Sebastian: one of these decisions that we wanted to try out to see if it's going to be
Sebastian: usable for the future as well.
Matthias: And before we talk about the onboard data handling system, what do you define
Matthias: as new space and what are maybe other companies in that realm?
Sebastian: Yeah, so the main one that is probably known to everyone is SpaceX in that regard.
Sebastian: And you can definitely see in the way they work that they like to fail fast as well.
Sebastian: I mean, just look at the Starship launches. The development of this kind of
Sebastian: rocket is extremely difficult, but they're doing it very fast and they're failing very fast as well.
Sebastian: I mean, many of these launches end with spectacular explosions.
Sebastian: And so the mentality is, yeah, OK, we're going to have to spend a lot of resources,
Sebastian: but we're going to learn a lot of it as well.
Sebastian: You're doing the failure so yeah that's
Sebastian: that's just one way of moving forward versus in the past
Sebastian: the kind of old space which
Sebastian: is the big entities like nasa they want to do it once and want to do it correctly
Sebastian: and i mean this mindset is also very important if you're doing like deep space
Sebastian: missions as well where it takes a really long time to get to places for example
Sebastian: i mean as well if you want to go to Mars.
Sebastian: You kind of get one shot and if you don't do it correctly, you're going to have
Sebastian: to wait a couple 10 or more years to get back.
Matthias: Okay, it's a test mission and yet it is foundational software that you're writing
Matthias: because you might as well just use some of the more established or some of the
Matthias: more common languages like C,
Matthias: C++ or maybe even ADA for the test mission for Gama Alpha but you chose rust why.
Sebastian: So this mainly boils down to our
Sebastian: software engineer at the time so this was
Sebastian: before i started working at the company i i didn't mention but the Gama Space
Sebastian: itself is only about five years old now so it was founded in the 2020s and we
Sebastian: well We kind of move relatively fast and the team only grows relatively so.
Sebastian: So right now we are, I think, 21 people in the company.
Sebastian: And at that point, when we launched Alpha, it was more like 12.
Sebastian: And most of our team is actually in mechanics because as you can imagine,
Sebastian: I mean, the folding and creation of the sale is its own topic.
Sebastian: It's one of the main things that we do. And so we only had one team
Sebastian: electronics and embedded systems engineer the one that was
Sebastian: responsible for the creation of of
Sebastian: the on-board data handling system at the time which is
Sebastian: a colleague of mine called Chris and his background was mainly in c plus plus
Sebastian: and so he decided but he he knew that c plus plus was not well suited for well
Sebastian: i mean it was difficult to use to get a reliable piece of software for the space environments.
Sebastian: And so when he looked into options and other kind of languages to use,
Sebastian: he decided that Rust would be a perfect fit for our case. He had a little bit
Sebastian: of prior experiences in it as well.
Sebastian: And so that's why we didn't choose Ada, for example.
Matthias: And how did he evaluate Rust? I know that you were not there,
Matthias: but maybe you talked to them about it. Why did they choose it?
Sebastian: Yes. So basically, he did a trade-off of Rust versus other languages.
Sebastian: And I mean, it came out clearly on top. The memory safety, just eradicating
Sebastian: a whole bunch of errors that you can have with C and C++ is really critical
Sebastian: for space applications.
Sebastian: It's these kind of avoidable errors that can be caught before you deploy your
Sebastian: code, that can just be denied by the compiler.
Sebastian: And in general, the help that the compiler gives you compared to other languages
Sebastian: in writing safe and good code, this I think was the main driver.
Sebastian: And, of course, the ability to still go into the embedded systems level,
Sebastian: still be real-time and be reliable.
Matthias: How can I imagine day-to-day at Gama?
Matthias: Do you use any special software that us normal, let's say, earthly Rust developers don't use?
Matthias: Or would you say, no, it's mostly the same components, it's mostly the same
Matthias: code structure, it's the same CI/CD platform like any other project?
Sebastian: Yeah, actually, it's very, very similar. So in a bunch of companies,
Sebastian: you would find setups with GitLab, for example, and so do we have it.
Sebastian: We use a lot of crates that are just out there for many people to use,
Sebastian: like ZeroCopy, for example, and other things, probe-rs for our debugging environment.
Sebastian: It's a little bit more specific because we work with hardware.
Sebastian: So in many software companies, you have your code, you have unit tests,
Sebastian: you have integration tests that you run on your CI, and that's that.
Sebastian: We have to interface with actual real hardware.
Sebastian: And so maybe one of the key differences is that we have a flat-sat for the Gama Alpha
Sebastian: product, so for theGama Alpha satellite. A
Sebastian: flat-sat, I should say, is basically the electronics of the satellite unwrapped,
Sebastian: laid out on a flat on a table, hence the name flat set.
Sebastian: And it's a one-to-one, pretty much a one-to-one copy of the actual hardware
Sebastian: of the satellite and we integrated it in our CI so that our tests,
Sebastian: we have mainly integration tests and they run on the actual hardware and then
Sebastian: we have probes attached to look at the actual output of the system and make
Sebastian: sure that everything is all right.
Matthias: It's kind of cool that you use probe-rs especially because,
Matthias: All of the embedded, or I call it embedded, but like all of the low-level projects
Matthias: that I know of use it to some extent.
Matthias: Was it a natural choice for you? Did you evaluate anything else?
Matthias: Is there even anything else in that space? Or is probe-rs the de facto standard now?
Sebastian: Yeah, it took us a while to get all of these systems up and running.
Sebastian: And in the process of doing so, we did evaluate multiple things.
Sebastian: But there was not a lot of choice for this sort of thing.
Sebastian: We could have also just used regular C debuggers, as in just start a different
Sebastian: process with the debugger and then interface with that.
Sebastian: That was one of the other options that we looked at.
Sebastian: I mean, that would basically have been the normal C approach,
Sebastian: you know, like just use a GDB, for example, as a debugger.
Sebastian: But the benefit of probe-rs was that, of course, it was written in Rust,
Sebastian: and it allowed for a lot more fine-grained control of what we wanted to do with our integration tests.
Sebastian: For example, we have a system to compile,
Sebastian: test Rust binaries, and then upload them via probe-rs onto the actual hardware,
Sebastian: and then look at the output and look for certain signals to make sure that the
Sebastian: test that this binary represents passes.
Sebastian: So that was very easy to do in Rust and using probe-rs.
Matthias: And how does that system look like? How does it work? What's the end-to-end development flow? So,
Matthias: So I'm guessing you develop code in your IDE. I'm not sure which IDE you use.
Sebastian: Yeah, so we use VS Code. And so if we want to develop a new feature or work on some code,
Sebastian: the typical workflow would be, so you write your code, you push it onto a new
Sebastian: branch, and as soon as you push,
Sebastian: our CI starts, and it reserves a slot on the flat side.
Sebastian: The interesting thing is that we also use the flat side for manual debugging.
Sebastian: So we have kind of a little bit of a system where you can reserve a slot on
Sebastian: the flat set. During that time, the CI can't run and has to wait.
Sebastian: But if the CI is running, you have to wait for a bit until it's finished.
Sebastian: And so we kind of have a little server, which unsurprisingly is also written in Rust.
Sebastian: Yes, that kind of manages the access to the flat set.
Matthias: And you wrote that yourself?
Sebastian: Yeah, we wrote it ourselves. I think, oh, it's been a while,
Sebastian: but I think it uses Hyper as a web server.
Sebastian: I think it's called Hyper. Yeah.
Sebastian: And basically, it manages access to it. At that time, so it has a web interface,
Sebastian: and at that time, we wanted to kind of get fancy, and I have a little bit of
Sebastian: a background in Flutter development as well.
Sebastian: So the web interface is based on Flutter, but the actual logic and the serving of the files is in Rust.
Matthias: Oh, wow. that's a
Matthias: really cool project in and of itself because because i
Matthias: really wondered how would you how would you test your hardware components you
Matthias: kind of want to abstract that away you don't want to unplug and plug it back
Matthias: in again you kind of want to have a system for that and i'm assuming that there's
Matthias: not much on the market especially for flat sets like this it's probably a,
Matthias: Still an embedded setup, just like any other embedded setups.
Matthias: But the moment you go into the details here, you will find that you probably
Matthias: need a custom solution, right?
Sebastian: Exactly. So this is one of the unfortunate truth. If you do develop your own
Sebastian: system, there's no one solution fits all kind of a thing.
Sebastian: So we did try to find generic solutions for embedded testing.
Sebastian: And I don't know how it looks right now, but a couple of years ago,
Sebastian: two and a half years ago, So there wasn't really a lot that helped you write
Sebastian: actual tests for the embedded target and run them.
Sebastian: So yes, this solution of the integration tests and the test suite itself is
Sebastian: also a custom solution that we developed in Rust as well.
Sebastian: So the whole integration test suite is also written in Rust and developed in-house.
Matthias: But the integration tests are still in a tests folder, just like you would expect
Matthias: from any normal Rust project, right?
Sebastian: Yes, pretty much. so basically it's a
Sebastian: binary that has all of the tests in it and then we just upload this binary to
Sebastian: the hardware and then let it run and we look at the output via communication
Sebastian: protocol I think for that we use just UART very common low-level communication protocol.
Matthias: Now earlier you mentioned correctness in a different context but I think it
Matthias: fits in here as well because couldn't you test a lot of the logic with unit tests?
Matthias: Or is it so that you kind of want to always test the inputs and outputs as well
Matthias: because it's kind of a monolithic service?
Sebastian: Yeah, we have some unit tests and for smaller functionalities we do use those to test.
Sebastian: The theme that we have in our code is we have one,
Sebastian: big project but it's it's built
Sebastian: up in smaller crates so the big kind of onboard data
Sebastian: handling system software uses a lot of smaller crates
Sebastian: that we also develop and the smaller crates they
Sebastian: have unit tests for their functionality but for
Sebastian: the big onboard data handling system it's harder
Sebastian: to test these things because they're very much hardware specific so for example
Sebastian: if i want to test that our persistence works so we are able to store data in
Sebastian: some sort of flash memory i can't just write a unit test for it because i need
Sebastian: the actual hardware i sure i could try to emulate it somehow,
Sebastian: but that's a bunch of extra work and i'm not guaranteed that it is going to
Sebastian: actually work on the hardware itself so in an embedded environment you you kind of can't,
Sebastian: avoid doing integration tests because you need to test on the hardware so you're
Sebastian: just going to test the whole system as a whole yeah.
Matthias: Well when you say that you store files on
Matthias: flash in a normal scenario it wouldn't be a problem if the flash storage fails
Matthias: you can't just you know fix it because it's maybe right next to you but it's
Matthias: harder to do in space do you have any failover for that sort of hardware failure?
Matthias: And if so, how do you test that?
Sebastian: Yeah. So in fact, all of our flash memory, first of all, it's a very specific
Sebastian: chip usually that has some built-in error detection and correction.
Sebastian: So on a hardware level, we are protected against it.
Sebastian: But on top of that, for specific parts of code that is stored on this flash,
Sebastian: because that's the important thing.
Sebastian: Code needs to be stored somewhere. And so if that code gets corrupted,
Sebastian: you're going to be in big trouble.
Sebastian: So one of the steps that we do in our bootloaders is that we check this code for errors.
Sebastian: For example, for bit flips. In space, you have radiation.
Sebastian: And so just random bits in your data and your code in that instance can just
Sebastian: randomly fit, which can mess up the entire code flow.
Sebastian: So we have steps. We have code written to verify that the stored code that we
Sebastian: have in our flash memory is correct and that can potentially even recover from errors.
Sebastian: So there's something called a hemming code, which means you basically reserve
Sebastian: some extra space for redundant bits and you can use those bits later to detect
Sebastian: and to recover from flipped bits.
Sebastian: And this kind of code we have integration tests that uses prop.is to manually
Sebastian: write into the flash into memory even and to flip some bits,
Sebastian: And then we'll let it boot, see that it handles these kind of corruptions that
Sebastian: we introduce correctly and fixes itself, yes.
Matthias: And that's also a custom solution?
Sebastian: Yes. I mean, this is one of the tests that we run during our integration tests.
Sebastian: So, yes, the integration test suite has a component that allows it to write into this flash memory.
Sebastian: And that is developed by us, and that's a custom solution.
Matthias: Indeed let's talk
Matthias: a little bit about your background where you
Matthias: came from what your other strong languages were and how you got onboarded on
Matthias: the project and your first experiences with the code base a lot of people might
Matthias: be curious about you know exactly this process of working for quote-unquote a space company so.
Sebastian: During my studies when i joined I joined a lot of projects that do things in space.
Sebastian: So for example, one of the cool things that we did back then was something called
Sebastian: Bexus and Rexus, which is a European project for students that allow you to do some sort of project,
Sebastian: some sort of scientific mission on a small suborbital rocket or weather balloon
Sebastian: that they launched from Sweden.
Sebastian: And so that's kind of where I got exposed to my first real mission
Sebastian: Bayes mission experience using those projects
Sebastian: and we used a bunch of different languages though
Sebastian: mainly C and Python and I
Sebastian: must admit before I joined Gama and in my university days I have never heard
Sebastian: of or used Rust so I'm kind of a late joiner in that regard but after finishing
Sebastian: my studies I joined Gama then in January 2023 and that's
Sebastian: The point when I started to learn about Rust and when I kind of fell in love
Sebastian: with it because of the security guarantees that it gives you.
Sebastian: You mentioned earlier that all of the stuff that we have talked before must
Sebastian: have been in place before. And that's actually not the case.
Sebastian: This is one of the reasons I got onboarded because at the time it was literally
Sebastian: just Chris responsible for the software.
Sebastian: And so we really needed more support in the electronics and in the software
Sebastian: department, embedded software department.
Sebastian: And so that's where I came in. Although I couldn't directly start working on
Sebastian: the onboard data handling system.
Sebastian: Well, I did some small contributions here and there. But when I started,
Sebastian: I got assigned to a kind of a side project.
Sebastian: So you have to imagine the satellite has just launched a few days ago.
Sebastian: I come into the company, I get used to everyone, everything,
Sebastian: meet everyone, and then, ah, okay, we have kind of a validation of our satellite.
Sebastian: We want to do a zero-g flight. So that is a flight where the plane.
Sebastian: Does a parable so it goes up and
Sebastian: then it flies in a way so that you you cancel
Sebastian: out gravity so you for a few seconds
Sebastian: like 30 seconds you are weightless and we
Sebastian: wanted to use that and develop a system to have a
Sebastian: small scale solar sail deployed and to
Sebastian: see how it how it behaves in COG because we
Sebastian: they had simulations at the time but there
Sebastian: was no practical experience and so we wanted to we wanted
Sebastian: to see if our simulations actually match what we see
Sebastian: in real life and so i got assigned to build this
Sebastian: specific project and the code for it and of course because we were using rust
Sebastian: this was also done in rust so they it's kind of it was kind of a relative simple
Sebastian: thing we just had three motors and we were controlling a a disk inside of a box and on the disk so we
Sebastian: were able to spin the disk and then release something that released our small-scale sail.
Sebastian: And the main part of this was just to do this control, kind of account for vibrations
Sebastian: of the plane because those kind of disturb the data and then record the data, of course.
Sebastian: There was a lot of data coming in because we wanted to record as much as possible from this experience.
Sebastian: We had cameras that captured the entire thing, but we also had various sensors,
Sebastian: accelerators, and stuff like that.
Sebastian: So that was kind of my first...
Sebastian: My my foot into into the company and that's
Sebastian: kind of also still what we do today if we
Sebastian: if somebody else joins the company they're gonna have to get used to the way
Sebastian: we work around here they're gonna have to get used to rust potentially and so
Sebastian: they they usually get a small project on the site where they can they can work
Sebastian: on it's still important to us of course and then they they kind of join the main force,
Sebastian: like that and we have a bunch of back and forth where we because you're learning
Sebastian: the way of how to write code not only for Rust but also for space there's a bunch of things that we,
Sebastian: how to's that we have in the company that we've gathered over the years on how
Sebastian: to write Rust safely as possible because unfortunately, it's still possible to screw up in Rust.
Matthias: Is it? Yes.
Sebastian: So, yeah, one of the main things in the beginning, I thought,
Sebastian: ah, it's so cool. Like, I can't get wrong with this.
Sebastian: If my code compiles, it's perfect.
Sebastian: And it's true in most of the cases, but some logic bugs can still sneak in.
Sebastian: Mostly if you don't write your code in a way that prevents them because that's
Sebastian: one of the cool things about Rust and what we use extensively here as well, if you write your code,
Sebastian: Using the strong type guarantees,
Sebastian: you can make it so that any invariants that need to hold true,
Sebastian: so any preconditions that your code has and that need to be true for it to work
Sebastian: correctly, you can express that in the type system.
Sebastian: Maybe not all of them, but most of them.
Sebastian: And so, for example, we use that extensively for configuration.
Sebastian: We have things, we have code that can only run once the hardware is initialized correctly.
Sebastian: And so we pass around serial-sized tokens, types that can only get created in
Sebastian: one place in the system, and that's after initialization of that specific hardware.
Sebastian: And so we use, for example, this sort of a token in other places to prove,
Sebastian: yes, this code will only ever run after initializing that specific hardware.
Matthias: Couldn't you also use a type state pattern for that?
Sebastian: Well if you do that type state pattern you kind of,
Sebastian: You kind of make code dependent. You encapsulate the code. Not encapsulate.
Sebastian: You make it dependent on each other a lot more, I feel like.
Sebastian: So instead of writing a huge state machine, basically, like that,
Sebastian: you can write your code more freely, just at any given structure that you want,
Sebastian: just passing around these types of keys.
Matthias: Yeah, I never heard about that types with keys pattern, the token pattern that
Matthias: you described. Is that a thing that you invented, or does it come from any other project?
Sebastian: Well, it's definitely used in the embedded world a lot.
Sebastian: Maybe not in the same kind of way, but we've got the idea from crates like Embassy
Sebastian: and actually various hardware abstraction layer crates that if you,
Sebastian: in the beginning, they give you one struct that contains all of the peripherals
Sebastian: but it's just zero-sized types for these peripherals and the idea behind that
Sebastian: is that you can then when you want to use one of these peripherals you have to consume that type.
Sebastian: And that prevents you from using the same peripheral twice,
Sebastian: for example pins if you think about a microcontroller
Sebastian: it has pins that you can toggle on or off and if
Sebastian: you decided at one point yes okay i'm gonna use this
Sebastian: pin to i don't know to talk to a motor for example
Sebastian: and then later you think okay right i
Sebastian: have a i have another use for a pin maybe i
Sebastian: don't know i want to toggle an led or something let me
Sebastian: see ah maybe pin one is still available i'm gonna try to use it ah no i've already
Sebastian: used this before and rust tells you this because you moved the type you moved
Sebastian: the variable into the function that used it and so you can't use it later and
Sebastian: so that's that's kind of the the same idea.
Sebastian: You have a token for each peripheral, for each pin that you can only use once. And you
Sebastian: This is one of the things I really love about Rust. It's the ability to detect
Sebastian: these kind of configuration errors at compile time.
Sebastian: I mean, the compiler literally prevents you from writing codes that is wrongly
Sebastian: configured that would be valid code just from a processor perspective.
Sebastian: I mean, it's okay if you use the same pin for two different things.
Sebastian: You can write to the registers and it's not going to care. But the end result
Sebastian: is that the pin is not going to do what you want because you're using it for
Sebastian: two things at the same time.
Sebastian: And by expressing this notion of only one part of your program can use this
Sebastian: specific thing via a token, the compiler can ensure that this, in fact, is the case.
Matthias: Now I understand part about loosely coupled components as well because I'm not
Matthias: sure, but I guess Embassy used the type state pattern and a lot of generics for the pins before.
Matthias: I'm not sure if they dialed it down a bit or dialed it back because,
Matthias: sometimes want to repurpose pins and if
Matthias: you set up your system once it's not
Matthias: as flexible as if you were to be able to
Matthias: you know change that dynamically sort
Matthias: of and at the same time you just use the normal ownership rules in rust to enforce
Matthias: the same behavior so you can still encode it in the type system without really
Matthias: having a lot of overhead and a lot of complexity on the on the type system level.
Sebastian: Exactly yes we as i
Sebastian: said we use this in a bunch of places also for example for
Sebastian: wrapping c libraries it's kind of
Sebastian: an interesting thing we during our
Sebastian: Gama Alpha project we used a communication
Sebastian: protocol called CSP and unfortunately
Sebastian: at the time there was no pure rust implementation of it
Sebastian: so we had no choice but to wrap
Sebastian: well either implement the whole thing ourselves but that would
Sebastian: have taken quite a bit of time which we didn't have or
Sebastian: to wrap the the c library and the c library has
Sebastian: a bunch of invariants when you use it you have one
Sebastian: of them is you have to call a function called csp_init before
Sebastian: you do anything else and so this kind of a thing once
Sebastian: again we we have a series struct in
Sebastian: rust called CSP it has an init function and it
Sebastian: gives you is the only place where you can get an instance of this CSP struct
Sebastian: and then the CSP struct allows you to call other functions on itself but you
Sebastian: need a self so you need an instance of the struct and so that proves that you
Sebastian: have initialized the CSP the underlying c library.
Matthias: Because that's sort of the only place where you can get that token.
Sebastian: From there's only one place to get this token and in that place we initialized
Sebastian: the library just before.
Sebastian: So it's kind of a way of you have one place in your code where you know about
Sebastian: these invariants and you only need to take,
Sebastian: to express these invariants at that place
Sebastian: so if i for example if i have function of this library
Sebastian: and i know it needs to be initialized before all
Sebastian: i need to do is i need to take a reference to
Sebastian: to this token and that guarantees
Sebastian: me yes the library has to be initialized and so as a user of this library which
Sebastian: is also us but sometimes we can forget maybe you forget that you have to initialize
Sebastian: the library before or you're just you know you're moving one line of code a
Sebastian: few lines up by accident before the initialization because you think, ah, this might be,
Sebastian: more efficient that way or something like that.
Sebastian: You forget about these environments much too easy.
Sebastian: But by encoding them into the type system and by encoding them into the function
Sebastian: signatures, we can still enforce it and we can let the compiler help us in not forgetting about it.
Matthias: That's pretty impressive. At the risk of going a bit meta here,
Matthias: how do you come up with these patterns?
Matthias: Do you find them online in certain other crates?
Matthias: Or do you think really hard about those problems and then start to encode those invariants?
Matthias: It's not really discoverable.
Sebastian: Yeah. So I think the way we discover them is by failing, by writing code that
Sebastian: is invalid and where we then introduce certain bugs.
Sebastian: And then later discovering, ah, yeah, okay, we need to prevent us from ever
Sebastian: writing that code again. And so...
Sebastian: For example, with the case of the library, it was literally the case that we
Sebastian: called the function before we initialized it.
Sebastian: And so that's how we got the idea of, okay, we need to prevent that.
Sebastian: How do we prevent that? Let's use this token system. Invest.
Matthias: Do you write down those correctness guidelines somewhere?
Sebastian: Well, we do have a bunch of rules like that, but the point is you only have to write them in code.
Sebastian: Right so the once once you corrected that mistake once you made sure via the
Sebastian: type system that it's impossible to use a function in an incorrect state you
Sebastian: don't have to you don't have to write anything anymore because the compiler will enforce it.
Matthias: Well i guess you review a lot of rust code nowadays a lot of rust code that should or cannot fail,
Matthias: how do you review rust code for correctness what do you look out for what are
Matthias: some common patterns that you found useful here.
Sebastian: Yeah for reviewing i
Sebastian: like to to think about
Sebastian: all possible states that a certain
Sebastian: rust code can be in and all possible inputs it's
Sebastian: kind of like like fuzzing you know where you
Sebastian: but but manually fuzzing we we
Sebastian: like to split up our our code and our
Sebastian: metric as into small chunks and so it's relatively easy usually for us to to
Sebastian: think about all the possible inputs and what can go wrong and then i like to
Sebastian: make sure that we use the type system to its full extent and to encode any preconditions
Sebastian: as i said into into the types.
Sebastian: It's kind of like if you have a precondition for using a value that can't be
Sebastian: zero, in Rust, you have the non-zero type.
Sebastian: So Rust itself uses a lot of these types for preconditions as well.
Sebastian: Same with the C string, a string that has to be zero, null byte terminated.
Sebastian: So Rust itself gives us a lot of these ideas as well.
Sebastian: And so when I examine code, I'm just trying to think of all of these different
Sebastian: invariants that need to hold true for a specific code.
Sebastian: I'm trying to find them out and make sure that they're expressed as much as possible.
Matthias: Are you a friend of debug assertions?
Sebastian: Yes. Well, I mean, we use them extensively as well, but only in places where
Sebastian: we can't guarantee something just with the type system.
Sebastian: I mean, there's always certain things that you can't guarantee at compile time,
Sebastian: especially if you're interacting with hardware, right?
Sebastian: Some of these things you just can't do with just the type system.
Matthias: For example?
Sebastian: Some of the runtime state that can't be known at compile time,
Sebastian: we, for example, have our own heap.
Sebastian: And so the idea of a heap is that you have a bunch of memory available and you
Sebastian: at runtime allocate things out of that heap as needed.
Sebastian: And you can never kind of really be sure how much you allocate just from looking
Sebastian: at your code because it depends on timing.
Sebastian: It depends on when certain pieces of code need certain amount of memories.
Sebastian: And so for those kind of,
Sebastian: So if we request a piece of memory, but we don't have any more,
Sebastian: for example, like this, there has to be still debug assertions.
Sebastian: And in fact, Rust itself brings a lot of debug assertions as well.
Sebastian: Every time you access an array, for example, or a slice, you have debug assertions
Sebastian: that make sure that you don't go over any of the limits in memory.
Sebastian: And that's one of the strong systems of Rust as well.
Sebastian: That's the memory safety all of
Sebastian: these checks at runtime as well of course you can disable them but this is for
Sebastian: us it's a tradeoff of size versus security and for the Gama Alpha project at
Sebastian: least we were able to leave debug assertions on.
Sebastian: Because that's another part of the space industry.
Sebastian: Usually you work with hardware that is not like your modern laptops and computers,
Sebastian: where you have basically unlimited memory and space.
Sebastian: No, our programs need to fit in tiny flash memories.
Sebastian: And so removing debug assertions can be one of the ways where you can have more
Sebastian: actual code, but with a smaller size.
Matthias: And by tiny, you mean how large?
Sebastian: Think like a couple hundred kilobytes oh wow
Sebastian: yeah it kind of depends i mean for for the
Sebastian: alpha satellite we had i think so the
Sebastian: processor itself had an integrated flash memory
Sebastian: of 256 kilobytes and
Sebastian: then we had an external chip with i
Sebastian: think an additional one megabyte of flash if
Sebastian: i'm not mistaken so not a lot of space
Sebastian: and some of the some of the space you need for for
Sebastian: data as well because yeah in you
Sebastian: talked about this in a previous episode you talked with i think ksat a provider
Sebastian: of ground stations so you might be familiar already but the satellite doesn't
Sebastian: have a direct communication to the ground all of the time we also need to store
Sebastian: data on the satellite itself before we can pass it down to the ground stations.
Matthias: Okay. It turns out there's not a lot of space in space.
Matthias: You work on tiny hardware. And shout out to Vegard as well. It was an amazing episode too.
Matthias: Now, Rust has fallible allocations for vectors nowadays.
Matthias: But I'm guessing that you even want to avoid allocations in the first place,
Matthias: especially the dynamic allocations.
Matthias: And Oxide, for example, has a scheduler called Hubris and Steve,
Matthias: please correct me if I'm wrong here but Steve Klapnik, that is.
Matthias: I think what they did was they used static allocation,
Matthias: static memory and every payload that they run on that scheduler has a fixed
Matthias: size known at compile time which allows you to have predictable allocations.
Matthias: Do you use that pattern as well? if you can?
Matthias: So having a fixed struct and knowing that only this struct will need to be allocated,
Matthias: quote unquote, on the stack?
Sebastian: Yes, as much as possible. So whenever we can use it, we try to avoid the allocations
Sebastian: and just use stack variables.
Sebastian: Or we have our own stack allocated statically and use elements out of that.
Sebastian: But sometimes it's just not possible to know at compile time,
Sebastian: what kind of, how much memory you need to use.
Sebastian: For example, think about the telecommand stack.
Sebastian: So for the Alpha Satellite, we were able to send telecommands to it.
Sebastian: And on the satellite itself, you don't know how many telecommands will come
Sebastian: in at a given time and how long it takes for each telecommand to be worked on. And so we had a cube.
Sebastian: Of telecommands and this queue of course
Sebastian: has an upper limit but the the memory
Sebastian: out of it was kind of heap allocated you
Sebastian: know because it was it was it had an upper limit but it
Sebastian: was taken out of a big slab but in
Sebastian: in all other cases where we can we really try
Sebastian: to avoid using the heap
Sebastian: i mean that's the no std environment for you you
Sebastian: don't get heap for free you need to provide your own
Sebastian: heap if you need it and so you're very
Sebastian: very specific about the places
Sebastian: that you that you need a heap and
Sebastian: you try to avoid it in all other cases one thing that can help with that a lot
Sebastian: is a crate called zero copy for example in our telecommand and telemetry system
Sebastian: we use it very extensively so the the way it works is you you define your structs
Sebastian: the data that you want to send,
Sebastian: want to be able to send, and we use zero copy to convert them or let them be
Sebastian: converted safely into bytes and from bytes without the use of an extra buffer,
Sebastian: without the use of an extra allocation.
Sebastian: Just the same amount of memory. It's basically a safer transmute.
Sebastian: And so, because we need to send the data as bytes, obviously,
Sebastian: so we need to do this conversion to and from bytes and zero copy is a great
Sebastian: help for that sort of work if.
Matthias: I understand correctly that means you don't initialize the struct you have a
Matthias: view into this byte block in this block of bytes and zero copy kind of makes that safe for you.
Sebastian: Yes in in a way so there's two ways of using it first of all the easy way is
Sebastian: you already have a struct and
Sebastian: you want to convert it into bytes and send those bytes over the network.
Sebastian: ZeroCopy helps you with that. But then the other way around...
Sebastian: You receive a bunch of bytes, and then you want to pass these bytes as a struct.
Sebastian: And ZeroCopy makes this passing, this conversion from just a view of the bytes
Sebastian: into a reference to the actual struct possible and save.
Sebastian: By making sure that there's no invariants that can't be uphold.
Sebastian: Or in newer versions, at the time they didn't have that, but in newer versions
Sebastian: you have a try-variant of all of their functions, which allows to test for certain invariants.
Sebastian: For example, if you want to receive a Boolean, a Boolean can either be 1 or
Sebastian: 0, but a byte can be 0 to 255, right?
Sebastian: So you have a bunch more values that normally shouldn't be allowed,
Sebastian: and these need to be verified.
Sebastian: And that's also one of the critical things.
Sebastian: In C, I saw a bunch of code where you just convert from your bytes into integers
Sebastian: and then you use those integers somewhere else and so it's very easy to mess up.
Sebastian: With just not validating your inputs enough.
Sebastian: And especially if you accept input over satellite connection,
Sebastian: there's a bunch of bits that can flip.
Sebastian: There's a bunch of stuff that can just go wrong. And so you really need to be
Sebastian: sure to pass it correctly.
Sebastian: And once again, we can use Rust's type system by making the constructors to
Sebastian: all of these payload structs private and letting it just take bytes and doing
Sebastian: all of this validation in this new function, in the constructor,
Sebastian: and returning an option of ourself or a result of ourself.
Sebastian: And so if any of the preconditions, they're just local, they're right next to
Sebastian: the struct, so it's easy to verify that all of the preconditions that we need
Sebastian: when converting from a byte string are checked for.
Sebastian: And then afterwards, you're sure if you have this struct, its data is validated.
Matthias: Well, that's really nice. It's a nice example of making illegal state impossible
Matthias: to represent and parsing instead of validating. That's kind of cool.
Matthias: You said that zero copy is a bit like a safe transmute.
Matthias: Would that mean you can ditch transmute altogether and always use zero copy?
Matthias: Or are there any cases where zero copy just doesn't provide a safe interface?
Sebastian: Actually, we have no instances of transmute.
Sebastian: And it's on the list of things that we really try to avoid.
Sebastian: And so far, we have not come across a use case that is not covered by a zero copy.
Sebastian: So, of course, sometimes we have to convert between different types in Rust, and we use as for that.
Sebastian: But those are safer conversions than just using transmute.
Sebastian: And the fact that the type system guarantees no just random conversion between
Sebastian: different types is really helpful to reason about these invariants in your code.
Sebastian: So, for example, just think about in C, I can create any struct at any point in any time.
Sebastian: In Rust, you need unsafe to do it. And it's a big kind of flag.
Sebastian: If you see unsafe in the code, it helps you during the review because when you
Sebastian: use unsafe, you always need to prove, at least with a comment,
Sebastian: that certain invariants are holding up.
Sebastian: And that makes it very easy to prove that your code can be safe,
Sebastian: even if you're using unsafe.
Sebastian: So unfortunately, we still have C libraries underneath, which are all unsafe.
Sebastian: I mean, it's C code, so there's no help from the Rust compiler there.
Sebastian: And so interfacing with those does require unsafe boundaries, unsafe blocks.
Sebastian: But when we write our wrapper functions for these functions in the C libraries,
Sebastian: we make sure that all of the unsafe free conditions that need to be met for
Sebastian: this library to be called are met.
Sebastian: And one of the examples is, as I said, the CSP library, where we make sure that
Sebastian: it's initialized before we use any of the functions.
Matthias: Okay, you try to minimize unsafe code. Do you also try to minimize unwraps?
Matthias: Are there any better patterns that you use?
Sebastian: Yeah. So unwraps, direct unwraps are also somewhat forbidden in our code.
Sebastian: The best that we can do are expects, where you provide a message and a reasoning
Sebastian: of why this certain thing can never fail.
Sebastian: And there's still a few pain points in Rust with this, where you do need to unwrap certain things.
Sebastian: For example, think about creating a non-zero value.
Sebastian: You can do this unwrapping or this expecting in const code, which is very nice.
Sebastian: So you can make it a compile time error. Yet still, it's unfortunate that certain
Sebastian: things can't be expressed as you want them and need to be unwrapped.
Sebastian: But we are constantly fighting that.
Sebastian: For example, in cases where we do need a buffer for certain things and we know
Sebastian: beforehand that we need a certain size, you can,
Sebastian: Add certain trade bounds using the unstable const generic features that prevent
Sebastian: you from needing to use Unwrap and then just giving you the actual value that
Sebastian: you want without using a result or an option.
Sebastian: And that's only possible because you let the compiler validate these preconditions at compile time.
Matthias: In my mind, you learned Rust in hardcore mode. it's
Matthias: in the space industry as a startup straight
Matthias: out of university with a python background and then also needing to ship features
Matthias: really early on you know working there what would you recommend others to do
Matthias: on how to learn rust effectively learning quickly learn it well how how would
Matthias: flatten the learning curve?
Sebastian: Flattening the learning curve, that's a big kind of issue in Rust,
Sebastian: I would say, compared to Python, for example.
Sebastian: I think just trying things out and learning by example is the easiest way to do.
Sebastian: Unfortunately, I don't have a fix, like a solution for everyone.
Sebastian: But the thing is, struggling a bit with the compiler is a bit frustrating in
Sebastian: the beginning, but it's a very valuable learning experience.
Sebastian: So if you start out using Rust and you're having trouble with lifetimes,
Sebastian: with borrowing, whatever, it is the compiler trying to help you.
Sebastian: And once you realize this, once you understand that the structures and the ways
Sebastian: you did it before are a little bit flawed in regard, once you understand this, it does help you.
Sebastian: Unfortunately, it is a hard pill to swallow sometimes, but it's very useful.
Sebastian: I, myself, after learning Rust this way, have come back to various Python projects
Sebastian: and found out that code that I thought was perfectly fine, that I wrote many years ago,
Sebastian: actually had subtle problems that could occur under some circumstances.
Sebastian: Not all of the time, but sometimes. And so these, for example,
Sebastian: if you have multiple threads and you access certain variables for multiple threads
Sebastian: in Python, there's very little help for you to avoid...
Sebastian: Accessing a variable at the same time as in fact there's none you need to do that yourself,
Sebastian: and in rust you can let the compiler help you with that and initially of course
Sebastian: it's it's harder because you need to prove that you're doing you're doing these
Sebastian: things in a safe way but it does help you to avoid certain logic bugs that would
Sebastian: be easily to would be easy to make otherwise.
Matthias: So see the compiler as an ally.
Sebastian: Exactly
Matthias: Try to
Matthias: take a step back and take your ego out look at what the compiler is trying to
Matthias: tell you and try to improve.
Sebastian: Actually that's one of the very valuable lessons Rust is amazing and one of
Sebastian: the main things is the compiler helping you so much and once you understand
Sebastian: that it's not you versus the compiler but it's you and the compiler versus bugs,
Sebastian: then you have a much easier time moving forward, even if it takes a bit of time
Sebastian: to get there, admittedly.
Matthias: 100% sure that there are people out there who would like to send in a job application
Matthias: right away or would like to work in that space.
Matthias: What are some next projects for Gama? Things that the team will be working on?
Matthias: Maybe also on the Rust side, what are the next challenges? What's coming up?
Sebastian: Yeah, so if people are interested in joining Gama, we do have some job applications
Sebastian: usually on our website, gamaspace.com so that's where people can find us.
Sebastian: So right now the Gama Alpha satellite actually has been, the work on it has
Sebastian: been finished for more than a year and we are moving forward with a different kind of sail,
Sebastian: a so-called drag sail which is a is a way of de-orbiting a satellite so bringing
Sebastian: a satellite into lower atmosphere where it can burn up so that it doesn't take
Sebastian: up any space anymore in space because even though space is rust the space around
Sebastian: earth is not and yes so there's there's a need for cleaning that up.
Matthias: Does that mean Rust does not have a garbage collector but you
Matthias: use it as a garbage
Sebastian: In a
Sebastian: way i guess although i must say so for this product that we are developing there
Sebastian: and actually that has already been sold in q1 this year we decided that we needed
Sebastian: a lot and like a real lot of reliability.
Sebastian: So this is one of the main drivers of why you need to have this kind of a system.
Sebastian: Your satellite is five years or 10 years in space. There's a lot of radiation,
Sebastian: there's a lot of events and it can fail at any moment.
Sebastian: And so our component needs to be as reliable as possible.
Sebastian: And for that, we decided to have actually no code whatsoever on this product itself.
Sebastian: But that doesn't mean that Rust isn't involved at all. In fact,
Sebastian: all of our test equipment, we have a dedicated little microcontroller and setup
Sebastian: for testing our equipment, is still 100% written in Rust.
Sebastian: Or as much as possible. But the actual drag cell that we're developing right
Sebastian: now is purely based in electronics.
Sebastian: The version that we're doing right now in the future, so the version we're doing
Sebastian: right now is for small satellites, but in the future, the next big project will
Sebastian: likely be a larger version of it using a lot more complicated hardware,
Sebastian: which will require once again software.
Sebastian: And for that, we're going to be able to reuse most of the codes that we've written
Sebastian: for the Alpha satellites, so all of the software stack, the different bootloaders
Sebastian: that we have, the onboard data handling system, most of it can be rewritten,
Sebastian: and so it can be repurposed, basically.
Sebastian: The hardware is going to change a bit, but the.
Sebastian: Because we split up everything into multiple crates, we have a crate specifically
Sebastian: for drivers of the hardware.
Sebastian: So that is the thing that mostly is going to change.
Sebastian: But other logic on top of that can stay the same.
Sebastian: In fact, that's one of the reasons why we split up the organization of our code into so many crates.
Sebastian: We have, I think, 59 crates in total in our internal crate registry.
Sebastian: Not all of them are directly related to the onboard data handling system.
Sebastian: Some of them are also for the test equipment on ground.
Sebastian: But splitting up the codes into well-defined little bits allows us to reuse most of them in space.
Sebastian: It was kind of the idea from the very beginning.
Matthias: Traditionally, the final question is, what's your message to the Rust community?
Sebastian: Well i think that rust
Sebastian: itself is an amazing tool and the rust community is doing an amazing job in
Sebastian: in providing it in in caring for it and also providing no std crates that's
Sebastian: it's very important for for our purpose and i think for for those of you in
Sebastian: the rust community that that provide these crates,
Sebastian: let's try to work on letting the compiler help us as much as possible.
Sebastian: So leaning into constgenerics, leaning into the ability of having these marker
Sebastian: types, because anything that we can handle at compile time is something that
Sebastian: doesn't fail at runtime.
Sebastian: And failing at runtime in a mission-critical environment like space is never a good idea.
Sebastian: So So if we can, let's try to work towards letting the compiler help us as much as possible.
Matthias: Sebastian, thanks a bunch for taking the time.
Sebastian: Thank you very much.
Matthias: And with that, we're closing out Season 5 of Rust in Production.
Matthias: Thanks to all our amazing guests and thanks to you for listening.
Matthias: A special shout-out to Simon Brüggen, who's been editing the show since Episode 1.
Matthias: We'll be back soon with more production stories. In the meantime,
Matthias: if you're thinking about migrating to Rust, we'd love to help.
Matthias: Head over to corrode.dev to learn more. And
Matthias: hey if you enjoyed the show the best way to support us is to leave a review
Matthias: on your favorite podcast platform or share an episode with a colleague or friend
Matthias: it really makes a difference have a nice break we'll see you in season six and
Matthias: thanks for listening to Rust in Production.