This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. I'm your host,Matthias Endler fromcorrode, and today we are talking to Jakub Valta from Fusion Engineering about flying drones with Rust. Jakub, thanks so much for being on the show. Can you introduce yourself to the audience? Hi, thanks for having me. So my name is Echo Valtar. I'm originally from the Czech Republic and I've been with Fusion for almost four years now. all my background. Originally I'm a software engineer with a focus on computer graphics. I can write game engines and for a lot of time I've been writing interactive installations and with creative coding community. That's resulting in me going to art school and getting an art and technology degree which I think is a nice way to round up a skill set. Sounds amazing, I always was interested in the intersection between art and code so I can totally relate. Yeah, it's really interesting when code is something either physical or visual for me, a piece of software which outputs few numbers isn't as exciting as a software makes something really pretty or it makes something intuitive which people can play with or it does something like making things fly. Exactly, that's a good prompt because you work at Fusion Engineering, so tell us a little bit about Fusion Engineering, the company you work for right now. Yes, we are engineering. We help flight controllers, which are brains that control the whole drone. Our company was founded in 2017, and the original plan was to make for maintenance of wine to combine blades. The wine to blades is that they get done by quite extreme weather. Salt, the leading damage diamond that reduces the efficiency of the whole thing. So normally people have them inspect the blades usually of them so they have to go up or down and manually inspect it there and know asical and apply some coating to fix all the all the materials used with that So with the drones, you could land the drone on the blade and if you either have something that drives over the blade and inspects it or the drone can fly by the blade and do some remote inspectioning, some maybe a laser scanner or something like that So if I understand correctly, it almost sounds like rocket science to me. Do you have these big turbines, wind turbines? And I never even thought about maintenance for these devices. But for sure, you need to get up there and you need to inspect those blades because, as you said, weather conditions and all of that stuff. And you can do that with drones. If you were to fly up there, you could inspect them. And it would say if you probably a lot of time, and it would probably also be way less complicated than doing it by walking up there. Yeah, so I'm not sure what exactly is the frequency of the maintenance, but you have to have people doing work in a higher altitude above ground. And that's dangerous, people fall, you need to do logistics, like getting the equipment there, people need to go to and stuff like that. you have to take breaks going down all the way up if there are stronger winds that puts people in danger but yeah being able to drum let's say comfort would be big yes it is very complicated so but it also sounds very complicated Originally, this DIA behind the company, but yeah, I realized that this is very issues for as a project and we need to work on it, let's say piece by piece. So yeah, we decided to switch the focus from the full drone to just the flight controller and flight controllers think for different use cases. In Genome, Drones are used reconnaissance, they can be used version, they can be a delivery. But yeah, these are these are you're seeing so far, but there will be probably more as people with new ideas how to use this technology. And where our, let's say, is to make the next generation of flight controllers would focus on ease of use, flight performance and safety. Okay, guess this hints at why Rust is such a nice language for flight controllers, but maybe if you can say it in your own words, given that flight controllers have existed before and probably they were written in other languages, like maybe C or C++, I don't know, what's the new big idea that Rust brings to the table now? So Rust is interesting in terms of Lightroll because it focuses on the same power goal, so that's a performance, reliability and productivity. So that, yeah, bugs which we could make in other programming are impossible in Rust because of the type system of the memory safety, of the thread safety. Yeah, because those drones, they can't really be operated remotely, they, well, they are operated remotely, but like it's kind of a problem if they crash or if the software fails because that could mean potentially that you harm people or you harm the device at least and you kind of want to avoid that. Like a sack fault in the air is probably not a good idea. Yeah, you definitely don't want things crashing in software or not in hardware. So, yeah, if something goes wrong and continues fly, that means your drone will probably hit something. There will be material there. So, if you're lucky, it might be just few hundred euros to rip some parts on the drone. If you're unlucky, it might actually harm somebody. But what happens if you have a situation where a rotor, for example, breaks? This is something that is completely outside of your control. How would you handle such a situation in the air? Yeah, so it's not that the road brake completely and more common is that the ESC that's a chip which controls the mode burns out or the motor just stops working and we have a few different approaches for that. One approach is that we are getting the feedback that the ESC is the motor controller so we know how fast the motor is spinning and if we lose this information, we know something's wrong and what we can do is we can figure out what contract to the renters and keep flying. This is possible, for example, on hexa-copters or octa-copters which have a lot of different motors. It's very challenging on quadcopters. It's very difficult to fly with only three motors. You basically need to start spinning really, really fast. You could argue makes the drone modern. This is the area of research but for drones with enough motors we can just start ignoring the motor and use the remainters to continue flying or let's say do some fail-safe manoeuvre. Yeah, so from a technical perspective or maybe from a software architecture perspective, how does it work? all of these detectors they have to run somewhere and I assume they run on the controller and it's not enough to have a single detector for all the rotors because essentially you need to kind of monitor all of them separately because they are individual rotors they could break at any point in time and separately and I do wonder do you do you run these things in parallel on the controller with different threads or do you have a different way of running all of these detections more or less in parallel and also for how long do you keep the incoming signals from the rotors, for example. Yeah. So this detection I was talking about, it's still a prototype or area of research, so nothing is really set in stone. But in general, you need to run real different things on the flight controller at the time to maintain the operation of the drone. The way being it is that we run several different services in different processes. Each service outputs certain information in the bio-different services and together the data flows between these services from the sensors and from the remote control all the way to the output stores. Yeah, those are different Linux processes. to those services running separate threats. So we are running this on Linux and generally each service is a single thread which took forever and reads the data it needs to read does its thing and then writes the data it needs to write. I'm kind of surprised that it's an entire Linux system there. I thought that there would be problems with battery life or maybe with hard real-time capabilities, but maybe that's not longer the case. Maybe modern Linux operating systems work really well in such environments. Yeah, so currently we are using a Kanthi4 Raspberry Pi to run the Linite, therefore course and for the scheduling we are using Headline Scheduler is quite robust for process for each thread. You can request a period how often do you want to run. You can request runtime that is how much time you want to use within each period. And the scheduler will make sure to wake you up at the fixed interval and make sure that it allocates to you enough runtime to do your thing. That means if you already have all of those capabilities, how does the software look like the Rust code itself? Is it structured like any other, say, backend Rust application? Do you have any limitations on the crates you can use? so in the in the core part it's we use async or tokyo the flights at 1 kilohertz which means that we had a second to do what we need to do within the loop and really does this like few hundred microseconds sleep and let the other services do their things so running at this frequency means that Can you use something like Tokyo or Async Rust in there? How does it look like? Yeah, quite careful about how the individual services are scheduled. That's why we use the operating system scheduler, which can provide this to us. But we are still exploring to moving to more hard time operating system in the future. But for now, for the, let's say, lower volume print we have right now, we are using the compute module with the toolinux, which has been working. Fine. we are exploring options to move this to a real-time operating system but more into detail into the services are not that fancy we use standard the service usually starts where the other services are write it puts and runs another reading the files, doing some processing and then outputting again to its own file. And yeah, we don't use even that many things from the standard library. It would be postcard noise TD, but sometimes it's nice to have certain features from there. Do you use any external crates or do you only focus on the standard library? We don't want to do the work, which has already been done by other people, much better than we were able to do it. So we use generally web rates, which are maintained. We are on the maintenance of those. we are using a cargo denied to check that self-contradon use several different versions of one credit and use yanked crates and security issues. So that helps a lot with managing the cellular dependencies. but in general we use creates like anyhow, this error and algebra is a big thing, we need to use a lot of math to calculate everything for interfacing the writing through the next, we use an unquote for a process mhm. We use it back out because we don't really want to allocate it in an hour or so. Few allocations here and there are fine. But if possible, we want to avoid allocate use Eric, which is on stack as a face. And we have bounds on how big the vectors can get. So yeah, we don't have to do any allocation there. that means a lot of checks can be done statically because you know at compile time how big your data structure is going to be. Yeah, so that's a big thing. We want predictability. We want to do as much checking as possible during well time. That's pretty nice. And yeah, that also means we don't use that many boxes. Most of the things are enums. If we need lesson modes to do one thing, usually we put an enum there. In aviation, there is a rule. I don't know if it's true for all the vendors, but it was kind of fascinating to me that when the machine, like the plane boots up, not even flies, but when it starts, there's a promise that there won't be any more additional allocation. So everything has to happen on the ground more or less. and maybe you don't have to go to such extremes but I feel like there's the same energy with what you described where you want to avoid allocations as much as possible. It's certainly the same kind of energy, but for us it's not as strict. We cannot locate if we want to and if it runs far. But yeah, every allocation puts it in your runtime. Why is that? Maybe to explain to someone who might not know about this. Yes, so if you have variables on stack they are known and it's now at compile time how much space stack the function needs. Why is an allocation a risk, especially a memory, a dynamic allocation on the heap? When you call a function this gets expanded to include all the variables you are using there and function ends. To put it simple the stack goes back to the previous size. with a dynamic allocation, a user box, for example, or a VAC. There has to be a cold locator, which is a P software which keeps track of words on an empty. and it finds an emphasis that you request and it gives it back to you and when you're done with the you have to go back to the allocator tell it hey I'm done with this memory please take it back make it free again so it can be used later and yeah that's another call to the allocator and There are a few things that come with it. I think the main one is that you run out of the memory. The action fails and you crash basically. That's not a big of a problem for us because we don't need that many memory to run. mm-hmm We are not, we are maybe using 10 part to have probably even less. So most of our course can run with very, very little memory actually. Yeah, this tracking of a free memory can be unheard of. Yeah, I guess that's a core point also predictable performance because on a stack allocation more or less takes constant time but on the heap that's no longer the case because as you said you need to find a free page or multiple pages even where you can put your object but like this can sometimes take very short time but sometimes it can also take a long time especially if you have a data structure like a vector that might need to be moved to a different location than to fit into memory and fit into the space and that is a bit of a problem. Yeah, so Vector has, let's say, additional danger there. If you keep pushing elements into the Vector into a VAC, at some points it will need to grow, usually doubles its size, so it doesn't relocate for every element because that costs time, that costs resources and it will, it would kill your performance. So yeah, as you push more things there that will be relocated, the alt memory has to be freed and the data has to be over. trivial thing,washing one element into a back and have a big influence on the performance. If you have a thousand big, you put extra, it pushes it over the capacity, then your memory has to be allocated, everything has to be copied and it has to be free. So, yeah. What are some other areas where you would say people that don't deal with systems programming don't know much about these things but they are very relevant to your line of work? So we talked about allocations but maybe the other part that I was curious about was error handling because yeah it's not just maybe easy enough to just add a question mark and then propagate the error to the caller because at some point there won't be any caller anymore and you want to avoid a crash so probably you need to be extremely careful or at least have a crash loops which kind of catches all of the bugs and then maybe restarts the service or something like this. yeah so in general If there is error in something essentially you really need like we need this data, you need to read this data from our function in this service and the data which is in a file is an incompatible format which can happen if you compile your services independently from different versions of your different Git versions let's say. Then, well, there is not much you can do, this drone can't fly, so you just question market out of the main and you can do really anything. Mm. but the focus is on, yeah, if we really are in a way we can do anything, crashing is the best overdone with spinitors doing something we can't control but in general, for all the error cases, you kind of need to handle them at a regular, let's say, flow of the, as a regular control flow so if something can happen, and handle it, then it's part of your, you have a code which implements that functionality and yeah, that error case is handled that can keep flying. While you explained I had two questions, first was how do you test this and second, how do you document this? Let's start with documentation because when you started with this project in 2017 and like it's been ongoing, you probably uncover a lot of edge cases and things that might not be documented by the vendors or your hardware behaves differently or maybe you also just want to have your documentation very close to your code do you use cargo dock extensively to even document that, your edge cases, even if you use those libraries internally only. In general, we do write documentation around our struct and our own functions. Actually, when there is assumptions being in the type system, you have to mention it somewhere otherwise. this knowledge gets and you will run into same issue when somebody tries to fix something they will break something else because they didn't know what works like this but the general approach is to encode as much as possible in the code in the structure of the program in the pipes and make it impossible to compile it when your assumptions are not correct. Sometimes this is difficult but really unfeasible so you end up writing docs like okay this can't be called here or this can't be called with these values or something like that but the perfect way is to type system and that really gives you a piece of mind that it can't be misused Yeah, so that's a good thing is to be able to represent only valid states of your program, but so be able to express them on a representative unique way. What you're trying essentially is to make a legal state a representable. So your problem is Not in the code, your program is a conceptual thing, right? The logic, how you design that work, you can explain it on a whiteboard in any code, right? This is a place where logic bugs occur, right? You design the thing wrong, you take it into account something, you have a logic bug, right? But then you can't run a drone of whiteboard drawings, right? You need to encode this in a source code. Yeah, generally you want to structure the code in the same way as you had structured your program conceptually so that if you need to make some change in how the things work on the logic level, you can go into the code, find the place where this piece of did and change it there. Do you use any patterns in the code? Can you give me an example for how such a thing might look like making a legal state irrepresentable that is? So about the patterns, I would say We have, let's say, we run certain problems and we have certain solutions. Some of them are used more frequently than what you would call a pattern. For example, we use a new type of pattern a lot. This relates to if you've heard of a Boolean blindness. Okay, let's say you have a function with a parameter that can be true, but in and of itself, it doesn't have any semantic mean. Can you explain it? You see true and okay, it's a value. What does it mean, right? And this leads to code like, okay, somewhere is a function call and there is a true as one of the parameters, but you have no idea what it means. What you would do is you would create an enum which represents the different cases than my names and suddenly everything becomes much clearer. It's harder to make errors. It's, let's say, self-documenting. mhm. If there is a third key and just added to the enum in the future and it's easily extensible. And the same thing, a lot of people stop with just the bullying and but this is the case with all primitive types you let's say if your function takes flow it also doesn't really mean anything it's just you can add a kilogram and apples and it doesn't make any sense but the system to do that and you can mix it up because we are all able to make mistakes so by using a new type you can make sure that you can encode in type system that okay this number represents number represents weight and kilograms and the code is much easier to know what things are when you read the code and also it gives you a place to implement the functionality on the type itself. So for example, You can, I don't know, from kilograms to grams. Okay, you have a new type and you have a place where you can implement your function to do something with that value. If you use just type, you don't have that played a lot of functions, which let's say do something with a weight, but they're all over the place and this is kind of a maintenance problem. Yes, and this happened to the very best of us. For example, in 1999 the NASA Mars Orbiter, which was used for, you know, measuring the atmosphere of Mars, I guess, and was famously lost because of a metric book. and I think at least four years or even more in development and actually running the mission, right? And yeah, that sort of thing can happen all the time. That cost 125 million. So yeah, if you can make your code not compile, if you make a mistake like that and I just looked it up, which is kind of, yeah, quite a lot of cash. It doesn't have any effect on the runtime, right? Yeah. Because all types just go away when you compile the code and exist in the output. Then it's a argue against this, right? It seems like a no-brainer. Do you use the type state pattern as well? Maybe for some explanation, the type state pattern is when you have a generic type and it takes a T as an argument and that T represents the state you're currently in. So when the drone is on the ground, it has a different state than when it is in the air, for example. I'm familiar with the pattern. I would say we don't use it very often because And that means that some methods are just not available in the air, though you can't call them. and that points out logic errors. Do you use that pattern in your code? the controller needs to be quite uniform so in a loop you have the same set of data which are coming in, some of them mobile options or you know, some stuff like and you need to output the set points for the motors very loop Let's, you have to perform the, you have the same inputs, the same outputs, you're calling the same function, but the state would resent maybe by a different variant of an enum. Mm-hmm So, we have a style, the, maybe the motors are not even connected, and we just check for signals that we should switch to a different state. Once we switch that to an armed state, and initialize the motors, make the connection start sending the signal to them and from then on every update is processing the data in a different way. There is many, the core loop is a stage machine but there is a lot of different stages to that calculation and the different stages can operate in different role based on how you want to fly the drone. right so it's essentially the same thing it's a state machine where you need to transition between the different states For example, you want to control the position of where the drone is. In a different module, the angle of the drone for much pitch and role is there. Hm? and for this, I'm not sure if it's exactly a pattern, but we represent the comicic agents with and the controller in the next stage looks at the ENA and it's just appropriate tool to handle that command and initialize date for that mode and then stays in that mode until it's switched to another mode from the upstream controller. This let's say a pattern which we are using quite often and it is a very common pattern that I see being used in other environments too and yeah it's great to hear that you can use that even for drone controllers and that's kind of cool to know. Do you use any unsafe code or is it completely safe Rust? I mean, when you control a drone, you kind of need to interface with the hardware at some point. so in general we try to isolate the iron unsafe to let's say library crates So I do wonder if you need to use Unsafe for this or if you can go through maybe other ways like connecting to the pins with files or some other way of IO that doesn't require Unsafe. so that we can write tests for the create, we can ensure it works correctly and then we don't have to deal with the unsafe on a higher level. I guess that's a pretty common approach in Rust. There might already be a create which is just what we want. So in that case, we use that. In another case, it needs something more custom. Yeah, we use unsafe, we use a lot of unsafe in certain levels because mm-hmm Yeah, those operations are inherently unsafe. And if you don't program them correctly, you might introduce undefined behavior or it's really the way you need to say, like, okay, you understand this is dangerous, but I promise I did it right. and you write a test with it, you test it and you keep an eye on that and it's solved and the rest of the code doesn't have to deal with that. Speaking of which, let's talk about testing for a moment. Can you test most of the logic with unit tests or would you say you need a lot of integration tests and functional tests? So the testing, different pieces of code are, some code is easier to test, some code is very hard to test for the library crates. Or would you say that you can test on a much lower level with a smaller scope through unit tests? but you can be right and they run on every comment on our CI so without everything passing we can't even build the code. So a lot of our code is split into bits which are tested. Some higher level creates for example, you have a create which does a PID control, that's a very popular form of a control process. Mm. And there might be a test which gives it a series of inputs and then checks that the outputs match what's expected of this algorithm to output over time. So, yeah, we run little tests faster than real time by using, let's say, virtual time. We went through a tiny simulated scenario which we already know what the inputs and outputs should be in this way. Okay. Nice. That's very nice. And once you have a case which you haven't, you know, covered yet, maybe you found an issue in the wild. What the issue is, a lot of the time is, sometimes it's purely a logic bug and then you can look at the flight log for example and you can put your detective hat on and see what went and follow the data through the code and see where a logic bug and then you can try to replicate it and see if Can you also recreate that in the lab and maybe make this a test for the future? mm-hmm the problem or if it's somewhere else. But in a lot of cases, it's a problem. It could be a problem in hardware. So, for example, some wire is loose. There is not a good contact. But a lot of the communication is using digital. So, it won't until it breaks down. mm-hmm as the connection breaks. So in those, when there is something strange happening, the general approach we are using is add extra logging, make sure we have those diagnostics, try to mitigate the issue, make sure we have the data log, and then try to mitigate the issue. we have some telemetry which can be viewed remotely however don't generally use this for logging we write a log file on the drone itself and we flush it quite regularly so that in case the thing really the whole power or something we don't lose that much data Can you lock that data remotely? Say a drone is in the air and it crashes, you want to know what happened right before? Do you have some sort of black box on the device or can you even send the logs over the air? How does it work? and yeah we log a lot of that information, we log all communication between the services so we use shared memory for communicating between different processes and we also have a logger which can scrape all of it and save it in a structured way into a log so in general files are what we use to diagnose the issues after the fact One other thing I wondered while you explained was do you use a lot of traits to mock controllers? We don't use mocking a lot. In general, we write, for example, we write a driver for something, a piece of software which does a communication, and we take it directly with the hardware, we check with the specification of the piece, make sure For example, you have a real implementation that works against a real piece of hardware and then you have a mock implementation which you can use for testing. what we have matches the expectation and not what we might accidentally make source of bugs where instead of following the specification and what are different functions promised to take in return you rely on the current behavior and the current implementation and maybe you get different hard version of the sensor and because you didn't rely on the specification it said only so I think that's the approach we are currently taking So our flight controller, we also developed the hardware ourselves. Nice PCB with sensors and STM32 on it. Speaking of which, which hardware gets used in your environment? What are the things that you have to deal with out there in the wild? And as I said before, we used the compute module at this stage, which clicks forward. and yeah, the board has a row of different interfaces. There's UART i-square-C,can at Wi-Fi antenna. Yes, the firmware which we run on SDM32. is we try to keep it quite minimal because it's a difficult day from it to have visibility into running there and in our use case it's mostly used for IO so the STM32 is the thing that commotors it has an implementation of the protocols that are used for communicating with the motors like PWM or D-Shot and do you use any hardware abstraction layers for the STM processor? and so responsible for some analog inputs. There's some york on the remote control. So yeah, the main ways in which the STM32 is used. For now, the firmware is based on STM2F4XL. and yes, the hull is quite, let's say, everything has a different type and if you want to treat a certain uniform way, sometimes there is maybe missing trait which doesn't allow you to love some other very firm way. I think this is a design decision of the crate and this code was originally written maybe four or five years ago and of course we've been maintaining it, upgrading it to new versions of the crates over the years and Would that be something you would like to change if you had the luxury of rewriting that crate? Would that be something that you would introduce or is it a limitation of hardware itself and it's just not able to be used like this? I can see that the how to program bedded systems is evolving and I think the structure of the crates available also reflects the way in which we think about this and we actually want to end the type system what we might not want to encode there because it makes life of developers much harder for example. Yeah, sometimes you run into some edge case, which nobody else ran into that like, oh, I would like to do this with these 16 pins, which are controlled by four different timers, which use a different DMA channel and stuff like that. Sometimes it's a bit of challenge to make something for it, or in the worst case, you can always use a macro to implement all of this. but it's a bit hard to work with. It breaks the code navigation and we try not to use the macros after. So your colleague, Mara Boss, she's a very famous Rust session and I guess also a Rust team member. She wrote a book, watched the Tomics and logs. So she is definitely a big proponent of open source. Yeah, so in the crates we are using, if we find some issue or some bug or if we need new functionality, we tend to work to create implement effects or implement new functionality and submit it upstream. I do wonder how you maybe contribute back as part of fusion engineering. What's the mindset there? Do you fix box upstream? Do you maybe contribute in other ways? How does it look like? These crates are usually in our GitHub repository, which contains works. How do you think about the Rust ecosystem as a whole? case is when we have some functionality which could be useful to a broader Rust ecosystem, we turn it into a public crate. so for example we have an inline Python create that allows you to run Python from Rust that's quite useful for outputs of simulations and stuff like that we have an int bits create which allows you to set your bits or access individual bits of which is for if you do any drivers for any hardware because each bit means something neat and way to access them and one more crate we are using quite often is Gitva which allows you to compile in the Git version of your code into the program itself Oh well,ok So that we can use this, for example, to ensure that our version of the STM32 firmware is the same as the version of this, that's Linuxite. And if it's not, we can reflash the firmware and make it synchronized. you could say that we tend to ship a gettable comment so that yeah we don't run into these issues where things are different versions and they don't match That means you use that information for reproducibility, you want to know what gets shipped and what the firmware contains, you want to have like a build order of materials for your drone software. Can you update the controllers once they are shipped? Yes, it is possible. We have a way to check for updates and install them using a package manager. Can you reflash the memory in the field? How does that part work? Let's say I buy 100 drones from you or drone controllers that is and then I want to keep them up today because maybe there is a fix or some critical issue that I want to have patched. We're still working on the fleet management, so if you have 100 controllers, it's probably going to be still some manual work to update all of them. Is that even possible? but we do have an hour and we do have a real pipeline for GitHub we can our CI build test builds which we can immediately download to our drones but we can also release build after testing and fixing everything which then can be download the flight controller's use binds mm-hmm For now, we are using a package manager of the Linux distribution. Yeah, the server builds the packages, updates them in its own repository and private repository, and then the flight comps can access the new versions. That's pretty amazing. And you built all of that in-house, the package manager for that and the uploader and so on. Thank you, but you're also a way to update it offline just by connecting directly to a computer. Oh wow, okay. But there are still some improvements in how all of this could work. We learn more as we, as our people are using our flight controls and we see different use cases. Way more modern than I anticipated. Kudos to that. So when the company started, we, that was before I joined the company, we were using C++, but it turned out that we have quite multiple discipline. It sounds like Rust was a big investment for the company but it also paid off well, I would say. How often do you need to drop into C and do FFI things? Is it something that you even have to do or can you use Rust for everything and you're pretty much covered now? Our team, you need software engineers, you need control engineers, which know all the math of making things fly. You don't need mechanical engineers, people who work more with the hardware. and it turns out people who a lot of external software engineers learnings can be quite difficult and it's very, let's say, create bugs or memory violations or stuff like that. So we tried to use Rust and it turned out that The people are not professional software engineers, pick it up quite fast, can be productive with it, the code quality is because Rust, usually there are idiomatic ways how to do this and there is not like 15 ways to do one thing, which over the years, that's maybe thanks to also Rust being a younger language, which doesn't have that much of a baggage So the benefit for us was that more people on the team are able to code and the quality of the code is either. Yes, that's right. wow that's a nice testament to Rust and if I understood you correctly that also means that you can bridge the gap between different disciplines or departments, you have the electrical engineers, you have the high-level software engineers, you have the people that know how to use simulation tools and they all speak the same language. I think so and the Rust already comes with the package management, it comes with testing, it comes with a formatter, a documentation generator. So all of these things make everything much easier. People can read the book or there is very good documentation for all of these pieces of the Rust ecosystem and we are basically full of this. It's pretty impressive. Do you see Rust's spreading across the industry from your line of business to others? So maybe related fields like maintenance work for industrial machines or Turbines or things that are kind of, you know, very complex pieces of machinery and you want a common language to talk about a system across disciplines and do you see that happening? I'm not really familiar with how this works in other disciplines. I can imagine that Rust is getting there, people are becoming aware of it, of the benefits. There are initiatives to make Rust easier self for mission-critical applications, for example, Ferrosin, which is a project of Ferros systems. Is Rust on the forefront here or would you say it still requires time? Yeah, I think there is a lot of work underway to make Rust more suitable for these curriculations and people do notice they investigate. I think a lot of this code can be built, especially the infrastructure, for example, you have a railway interest infrastructure and this code, me, decades old even. and any change is very and it has to be very well thought out and so extremely risk of our use cases. Am I still taking some time for Rust to enter them? However, I think people, as I said, are becoming more fair with it, seeing the benefits and maybe introducing it bit by bit into their current systems. to be honest, when I'm a fusion engineering, I didn't know anything about drones. Absolutely zero. But as I said before, I had some experience from Amazing answer, how do people even get started with such an endeavor? interactive applications and it turns out that's quite useful when you're working on drones because even though the flight controller runs at much higher frequency or typical game, 1,000hertz versus 30 or 60, the general structure is very similar and we care about it as games would, which means Because I wonder if it's a very accessible field for people who only know Rust right now. You need the predictor, but you need to get yours out in time. Same as we need to tell the motors what to do in time. And the whole structure of looped services, which run in ticks, is very similar. other than that there is of interfacing with different hardware and software. So to work with different protocols, serialization, deserialization, even network services, all of them be really useful when working on drones. That is true. Yeah, initially when you described the process of looping through inputs and then having these hard real-time constraints, I was sort of reminded of game development. I would say thank you and keep up the good work and I think together we can really improve how software is being built and the quality of the software in general. I wanted to into that anyway. And you heard it here first. So if you have a background with Rust and game development, maybe drones could be something for you and lastly it has become a bit of a tradition around here to ask you if you have a message to the Rust community anything that comes to mind this is the time to say it out loud Thank you for having me. I couldn't have said it better. Jakub, thanks so much for your time. It was a pleasure to have this interview with you. Thank you very much. Rust in Production is a podcast by Corode. It is hosted by me,Matthias Endler and produced by Simon Brüggen. For show notes, transcripts and to learn more about how we can help your company make the most of Rust, visit corode.dev. Thanks for listening to Rust in Production.