It's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias from Corrode, and today we talk to Anders Hellerup Madsen and Gorm Casper from Veo about analyzing sports matches in Rust. Anders and Gorm, thanks so much for taking the time to do the interview today. Can you say a few words about yourself? Yeah, so my name is Anders, and I work for Veo Technologies, and I started here five and a half years ago, and I'm a staff developer. Yeah, and I write software mostly for cameras and our video processing pipelines. Yeah, and my name is Gorm, and I joined a year and a half ago. And previously, I've been working mostly in the front end, and then more and more in the backend. About six years or so ago, I transitioned to the backend. And yeah, I'm very excited to be here. What does Veo do? Veo Technologies is a company that makes equipment for recording sports activities of all kinds. It started as mostly for making cameras that record football matches, but now it's a whole, it's multiple different sports types and a lot of like tooling for coaches so they can help improve their players or teams. Most of it AI-assisted, There's a lot of like social services also where people can share recordings and highlights with each other. And what's the backstory behind that company? How did it get founded? The legend is that our CEO had a friend who missed his son scoring a goal in a football match. And then as our CEO was about to start a company, he wanted to know like what sort of company should I start and what should we do? And his friend said, if you could only make something that could record football matches so I could have seen my son's goal, then that would be perfect. And then that's what Veo does today. That's a pretty fun story. When you started, what was the company's technology stack like? When I started, there barely was tech stack. There was sort of a proof of concept of a camera that was just two off-the-shelf board surveillance cameras put in a box, and then it could record the left half of a field and the right half of a field. And then there was some sort of OpenGL-based C++ application that could do some image manipulation to sort of make it look like a virtual camera. And the project I was hired to work on was to make the camera able to actually do all this image manipulation itself, like build a new camera that would be able to record and live stream a football match. And we could choose our software stack, mostly ourselves, because nothing really existed before. It was all proof of concept. And the first version was just a C++ application. Was it all in one box or was there another PC somewhere? That he did the post-processing from the cameras? Yeah, the only active component in the original Vero camera was a Raspberry Pi, and all it could do was upload the recorded videos from these two surveillance cameras. So all the processing would happen server-side, and it would use C and C++ and then OpenGL to do some graphics rendering to make it faster. Sounds like a very reasonable MVP. What were the issues with that setup? But the most direct issue was, of course, that it required that you record the whole football match and then upload the whole football match and then process the whole football match, and then you could watch it. So in those days, there was an SLA of eight hours from you finished uploading before you could watch it. And this is typically not how you want to use a camera. So the challenge for building the first real video camera that you could actually really use was to make the camera able to sort of stream the football match as it happened more or less. So that was around when, 2021, 2020-ish? Yeah, I think I started fall 2020. And did you immediately realize that you wanted to rewrite all of that in Rust? No, we started writing it in C mostly because we picked this multimedia streaming library called GStreamer. And GStreamer is a massively multi-threaded video or media processing framework that can be used to build media processing pipelines and this is what we were building so we sort of picked that and it has integrations to OpenGL and CUDA for GPU accelerations and we could use a hardware platform from NVIDIA to sort of, it's called Jetson and it's sort of similar to a Raspberry Pi, just with an active GPU from NVIDIA on it. And we could use all this to build an application that could do the same as our cloud rendering used to do. And then to sort of orchestrate it and tie it together with the rest of the camera and the iOS app you use to control it, we would build a Python application and to connect to some of the AI elements that need to go in the pipeline to sort of detect where the camera should look and where the action is on the field, we would also use Python elements of the pipeline. So it was mostly C and Python in those days. It's been a while since I used GStreamer, but I can still remember that I ran into some really nasty edge cases when building up larger pipelines. It was all sort of stringy typed, or maybe I was holding it wrong, but it felt horrific, to be honest, once it got above a certain number of lines of code. Was I the only one? No, not really. it's an odd thing but I don't think it's really GStreamer's fault. It's because it's built on like the G in GStreamer it's the same G as in GTK or GNOME or the GIMP, like the image manipulation program. And I think all of this shares this standard library called GLib and GLib comes with sort of an object model that is meant for integration with dynamically typed languages, and this means that there's a lot of like it feels like you're writing in a reflection plane all the time so you're sort of writing c code which barely has a type system but then you have to at runtime adhere to these these sort of types that are aren't really types they're just sort of c structs and and conventions and there's a lot of macros to make it fit together and it can be very difficult. But that means you don't have to use GStreamer like I used it by just, you know, writing out the pipeline on your shell. You can put that into a file. You can maybe use tooling to improve that workflow. What's it like to use GStreamer in production? It's fun because GStreamer can do, you can use it from the pipeline, from the command line just to write out GStreamer pipelines. But you can, of course, also make the elements that go in the pipeline, and those are sort of like small mini libraries. And similar to how a browser would render an HTML tree, then the HTML is just like a list of strings, really tags. But there's an object model behind all that in the browser, like the DOM. Similar to that, there's an object model behind a GStreamer pipeline. So you can use queries, you can select elements and you can run commands and send events to them and do stuff and all this is what we used to do with the Python application, Because you sort of have access to this object model, even though it's written in C, you have access to it from other languages because the whole glib thing is built for that. Why didn't you just use FFmpeg? Yeah, we get this question a lot, actually. It's because FFmpeg is very good at just running one command and doing that. But if you sort of need to hook in and do complex nodes in the processing pipeline, like run your own GPU kernels and image processing and fork off some images and give them to AI elements, get responses out of the AI elements and queue up other parts of the pipeline and merge all this together, then FFmpeg starts being easy. It stops being easy. And this is exactly what GStreamer is built for. This is where this whole concept of elements come into play. And GStreamer also has a processing advantage in that each of these elements can be in their own threads. So you can sort of have a very, very massively multi-threaded application that does a lot in the same time. And you can have each of these threads dispatch work off to GPUs and do this. And GStreamer helps you tie all of it together. And it gives you a lot of tools to sort of add probes to different places in the pipeline and monitor how the pipeline works. Walk us through your first year on the project. You open a code base, you do some refactoring, and you build some saner version, too, of the entire pipeline. How was that process like, moving from what you saw when you started to a version that maybe you released a year after starting? We started from scratch, more or less. we looked at what the existing C++ and OpenGL program did, and then we said, let's build some GStreamer elements that can do the same. And then once we had a prototype that sort of worked, then we worked for a long time on this sort of embedded, operating system trying to get the new environment to work so it could run the tooling we had. And then we ended up a really long phase of just testing, trying to get everything to be fast enough and iron out bugs, So it's like in all startups, it was mostly just hard work. There was no really difficult things to challenge. It was just a lot of hard work. And the second version, or maybe the first version that you built, that was in Python then? Was the entire pipeline converted to Python? The orchestration of the pipeline, the bit that put the pipeline together, was converted to Python. But the actual elements, like the bits and pieces that the pipeline was made of, they were written in C and CUDA for the GPU part. CUDA is NVIDIA's proprietary C++ version that can run on GPUs. And this is actually where we ran into performance problems because GStreamer is so multithreaded. And when we are orchestrating all of it in Python, Python is very strongly single-threaded. It has a global interpreter log that makes sure that one thread and only one thread runs at a time. And none of GStreamer's bits and pieces were sort of built for that. So we constantly run into deadlocks. And we tried to solve it in C and we tried to solve it in Python. And we kept on running into these things. And then eventually, I just decided that since the whole idea of glib and GStreamer is that you can use it from any programming language, let's try and use it from a language that guarantees that there is no deadlocks. And then I picked Rust because this was what I had heard about Rust at this time. And then I rewrote the Python parts of the GStreamer pipeline in Rust. And this was super successful. It fixed so many bugs and avoided so many pitfalls. And then we had Rust in production. So it was basically just like that. It's one thing to have heard about Rust, but it's another thing to bet the future of a company on top of it. Back in 2020, we already had decent G-Streamer abstractions in Rust. I think Sebastian Dröge worked on that a lot. Did you build on top of his work? Yes. Sebastian Dröge is amazing. And I love his presentations and all the work he has done on G-Streamer and the Rust bindings. And it was because I saw that these were there. and we could see how many exciting GStreamer plugins that were being published open source. And we thought, why aren't we doing this? And then about the risk of putting Rusk in production, at this point, the risk was more that we were a startup that were running out of money and we kind of needed to have a product to sell. So we needed to get it working and we couldn't get the Python stuff to work. So we just tried whatever we could and when we tried Rust it worked and then there wasn't so much to talk about it was just in production like that because the important thing was that we had something that worked. Right around the time Gorm you also used Rust as far as I remember but in a different company how was that like? Yeah, that's true, actually. It's the company. Anders and I didn't work in the... We worked in the same company before, but not at the same time. But yeah, around 2020 also. We did the same thing. We said, okay, let's try and rewrite this in Rust. I was working in a company that was doing PDF stuff. We would try to analyze uploaded PDFs. And that code base was in OCaml, which is fine. It's a very nice language. I actually really like it. But we needed more people on the team it was just me who had taken over from some some previous developers and yeah i'm one of those previous developers no no yeah and the codebase was good like it was great the problem was i couldn't hire for it i couldn't find anyone to work with me on this and we were trying and i hired some developers that was like yeah okay let's try and we would work on it, and it was just so difficult. And in the end, I said, okay, let's try and pick, let's take one month and rewrite it in Rust. Because I had taught myself Rust at the time. And I honestly thought, okay, this is going to be easier to teach Rust than teach OCaml to the one guy that I was working with. And so we ended up rewriting I think it was around 20,000 lines in three months. And, just worked so yeah i was smiling before when you said like it just came into production just like that and that was the same for us so it was like oh it works okay let's just use that then and go from there um the fact that. Rust is a descendant of ocaml might have helped here. For sure yeah yeah. Were you able to mechanically convert ocaml to rust or did you have to change the semantics did you have to change the logic? It was almost a one-to-one translation. We started with all the small helper functions doing all the affine transformations of coordinates and whatnot. So we started by actually porting the unit tests and then all the functions. And then we could move on to actually doing the business logic thing of figuring out where things are on the page and what they are. Coming back to Veo, we are at a stage now where Rust gets discovered. We have GStreamer. I was wondering, can you just install those GStreamer plugins like normal crates? How's the workflow to get those plugins into your system? GStreamer has this really advanced plugin system where it scans for specifically designed shared object files, like DLLs and Windows or SO files on Linux or Mac. And then if they follow a specific naming pattern, it tries to load them and then looks for a symbol that has the same sort of naming structure. And then if that symbol exists, Then it calls it, and this sounds very dangerous, but sometimes it goes well, and then it's just a loaded plugin, right? Then it can call register function and can set itself up that way. So what you have to do in Rust is you have to compile your Crate as a C dylib. You can set that up and you have to implement a couple of external C functions and you have to be very specific about naming and use no mangle and unsafe and everything. But once you sort of have hooked in and get loaded by the plugin system, then you can sort of stay in safe Rust for the whole thing and all the calls into core GStreamer, which is C, of course, and other things just managed by the amazing Rust bindings to GStreamer. Well, it sounds like the transition from C++ to GStreamer to Rust was a big success. Yeah, it was. So we finished the camera and it started to sell and the company started to grow and we were hiring plenty of people. And we started working on our backend systems and improving them. And my role in this was mostly to work on porting the pipeline we had now built for the camera so it could run in cloud instead of the old C, C++, OpenGL application. But the rest of the company was mostly focused on our Django application, our Python and JavaScript backends and frontends that users use to manage all the videos and see the videos and make highlights and edit videos and send things to each other. And as we grew, we wanted to hire more people. And this is when we hired Gorm initially for my team, I think. Yeah, it was for your team, actually. I had come all the way from the front end a long time ago and working at that previous company, right, in OCaml and a bunch of other languages. And so I always wanted to go deeper and deeper into the stack. So I wanted to join to see what it's like to work next to C and with C and just how low in the stack can I go. So I joined around that time. And then we decided to build more of the backend, like more of Veo in Rust. And so now I'm on a different team. Yeah, building out a new product with the whole team in Rust. But you got hired as a Rust developer. Definitely, I did, yeah. I almost want to say I decidedly did not want to work in Python. I can see that you have some horror stories to tell from the trenches. Yeah, yeah. I mean, don't we all, yeah. How many Rust developers were working at Veo back in the day when you joined? Was it just Anders? Yeah, I think so. So at the time, some of the people who write firmware for a camera, they had also started doing some of the microcontroller stuff in Rust. So there were maybe a handful of Rust developers, but spread over different teams. Yeah, no, it was also split, right? Like it was some C and some Rust. It wasn't like, oh, I only do Rust. It was just like, oh, let's do this part in Rust and this other thing is in C or whatever. Mm-hmm. Did you have any connection points with the embedded team that also worked with Rust? Did you share any learnings or share any code? Yeah, I mean, we do have a, on Slack, we have a Rust channel and people, at least in the beginning, our teams could ask for reviews and often I would jump on them just to provide some feedback. And so sometimes I would end up helping actually write stuff also. What are the types of things that you pointed out most often when you reviewed the code? Any patterns that you found that they commonly use? It depends on what language they come from, right? You tend to write code in the style of the language that you know best. But for C in particular, it's just having the whole type system and being very strict about what data is where and what type it is. Yeah, and using enums effectively. Enums, I should say. To me, it almost feels like people are afraid to use those high-level abstractions because they are afraid they would make the program slower or bigger. Maybe they need to kind of lean more into the type system. I would agree with that. I think there's also a case of when you program in c you're so close to the middle it's so low level or it's so raw that you know exactly what what bytes am i writing out like what what does it exactly look like but like what does an option of a float what does that look like you you don't really know right like it's it's yeah at least when you see program you don't know what that looks like you don't, So I think that's that part also, like the uncertainty of like letting go, like, is this actually what I think it is? Is it fast enough? How do I know these things? I think there's a big aspect of that as well. No, I mean, how do you learn those memory models? How do you learn how it really looks like in memory? Or is there any tooling or documentation? The Rustonomicon comes to mind, for example. I learned mostly by failing. So most of the time, it doesn't matter what sort of memory model something is, even when you're working in lower-level stuff, because it uses one byte more or less what really cares. But sometimes it matters. And typically when you're crossing boundaries between languages like FFI, and then something fails horribly, crashes, and then you have to figure out why did this happen and then you learn what the difference between a pointer to some object and the pointer to trait object, how they're different and what a fat pointer is and you learn all these difficult things. You sort of learn them by just having to learn them when it doesn't work. That is at least what I did. Yeah, it's interesting, actually, because I also worked with interfacing with C, and in the beginning, I didn't even realize that the memory model is so different that you have to do this, like, no mangle stuff. And so for a lot of Rust types or structs, I still don't know what the memory model looks like. But I do know in C, because C cares so much, right? Like, it will be angry if you do it wrong. Did anyone come from C++ learning Rust? Yeah, in the team I'm in, I work with the colleagues that come from C++. And what are common patterns that C++ developers tend to bring over to Rust? So one pattern is a little bit of reluctance, maybe, to pick up Rust. It's maybe a subjective thing, I don't know. But it feels to me like people who come from Python and JavaScript or functional programming languages, they tend to see more of the immediate benefits of Rust. But C++ programmers, they come from, yeah, but the lifetime and reference, that's just smart pointers. And we have that already. And it's like, they can see it's a more convenient way and it's a more modern way to do stuff. But it's also a lot of new things that they already have in C++ so so it's my impression that people from C++ is a little bit less excited over Rust than they are, than people coming from other languages. But this is maybe just my sort of, maybe that's just my perception of what C++ developers feel when they come to Rust. I don't know. No, I agree. Because I also see that in workshops at times when they say, we have smart pointers at home. Yes. We have an ad hoc makeshift version of the borrow checker. And Rust is just a C++ dialect with extra steps. And you have this almost hubris initially where people say oh yeah this is just bad programmers making these mistakes typically where you get them is when you do some refactoring and then you realize well okay now this error gets exposed and it wouldn't happen in rust and also what i do in workshops also is sometimes using them as allies to explain lower level concepts because this is still very helpful information and people coming from higher level languages don't know so this is where c++ developers feel strong or confident and maybe bringing them in like this is helpful and then showing them two three tricks that would be totally doable in c++ but just much harder. My impression of C++ is also that it's a hard language. It's very hard. So I have a lot of respect for people who are very good at C++ and, I get maybe why when you spend so much time getting very good at something so difficult then maybe you're more reluctant to pick up something, that claims to solve all the problems with what you have. But that also mirrors what I've seen. Like that, if there isn't like an immediate or a very clear benefit, then why bother, right? Why bother? Like, so it really has to... Not just be just as good or slightly better it has to be much better like much better. Yeah in our in our like video processing code we also we also that's also a little bit of an uphill battle because it is already C++ so porting it to rust is let's take something that is written and write it again in another program language just because that's a hard sell and then we are using NVIDIA GPUs, both in our cameras and in our cloud processing, like we use AWS instances. And NVIDIA has their own sort of C++ dialect for writing code that can run on GPUs. And it is convenient that you write the GPU code and the sort of host side code in the same programming language. And you're also losing that convenience when you switch to Rust. So it is sort of a double hard sell, but on the other hand then you get all these nice integrations with GStreamer which is a super nice thing so it's a little bit you have to pay something to get something else back and then of course we are then sneaking in a much better programming language on the site and then everyone will eventually be super happy. Will you keep some of the code in C++? Now we do keep the stuff that runs on the GPU and the code that immediately surrounds it, like the lowest levels of that part of the stack in C++. And then we call into that from Rust through some bind-gen-generated FFI stuff. Have you seen the latest announcement from NVIDIA? Yes, I have. They built a CUDA wrapper for Rust or a library for Rust. Yes, it's super amazing. But also, there's the whole Rust GPU thing also, which is also very, very interesting. Did you try it? I haven't tried it yet, but it was just announced yesterday at the time of recording, I think, or two days ago maybe. But it's very interesting because NVIDIA so far I don't think they ever made it possible to write GPU code in a different language than C++ so you could sort of call some CUDA things in Python but you could never write actual code to run a GPU in another language so this is very. Very exciting Also, I haven't tried it yet but what I like is that it's an official project it comes from NVIDIA itself. That means you have corporate backing for Rust now which is a major step. Maybe you can use that in the future to replace some of the C++ code or maybe at least the glue code between Rust and CUDA. Or Rust and the GPU which is CUDA. Would it translate to CUDA code or would it run other native code on the graphics card? So I think that CUDA is the part that runs on the graphics card, and this is typically compiled to an intermediate representation called PTX. And then there are some functions that sort of do the thing about uploading the code to the graphics card and, telling it what to do, like copy memory from here to there and this stuff. And all these functions are sort of just a C library or C++ library, really. So you can call into that from Rust with normal language bindings, like FFI bindings. And that has always been great to do that. The new thing is that you can now write Rust code that compiles to PTX, which I think is super interesting. And especially interesting because there has been open source projects before that tried to make compilers for other languages that compile to PTX and NVIDIA has been actively trying to close them down, even threatening lawsuits and now they are officially promoting Rust version, that Rust thing that can compile to PTX. I think that is very exciting. Okay, we went on a deep end, pretty low level And of course, you can use Rust on the other side as well to use it for quote-unquote high-level code. I don't really like that term, but what I mean is the backend side of things. How does Veo use Rust on the backend? Yes, so currently we are rewriting the whole sales pipeline in Rust. And it has previously, I haven't really looked at the old one, But as far as I understand, it's been a collection of tools, mostly in Python, and sort of built ad hoc over time. And I don't want to say barely holding it together, but you know how these tools go, right? It was time for a rewrite, and so we decided to do it in Rust. We had a few languages to pick from. Yeah. But yeah, we decided on Rust and then I transitioned into that team to lay the foundations that we could build on top of, as well as get the team up to speed on Rust so they can develop quickly enough, let's say. And how did that go? I think it went very well. We started about a year ago. And in the beginning, the team had started up just before me, like without me. So I arrived at some, I hope they don't mind me saying, horrendous Rust. Let's call it Python Rust. And they didn't really have a SDK. So the whole thing is built using Temporal. And Temporal doesn't have an SDK. or it didn't have an SDK. They do now. They just released an Alpha Version. But at the time, they didn't. So they just sort of built their own thing to work with Temporal. And it was all not very good. Not very idiomatic Rust, let's say. I mean, nothing wrong with the code. Like, nothing wrong with the... Yeah, nothing wrong with the code, I guess. But not good Rust. What makes you say that? What would you say was the code like? It was very Pythonic. Yeah, exactly. And one of the things that I realized, and same for the previous company, was that we should take people from Python to Rust. You don't have to, unlike with C and C++, you don't have to sell Rust. Because there is a, what is the company? Astral or something? That does UV and these things. So Rust is already like, proven, like it's there and it's good. And the only problem with Rust is that it's hard to learn. So that's so I don't have to sell it. I just have to teach Rust, essentially. And I think the major thing there was being super, super strict about the type system and very, very narrow, like try to narrow things down. So not to use string as anything like, you know, two different strings as an enum, right? but no, okay, fine. The string can only be one of these two. Let's make an enum. Let's tighten it down so we know what's going on. And then a bunch of... Basic things like not knowing how to use Serdy. I remember I came in and everything was deserialized into a Serdy value, which is just like a raw JSON blob almost, right? And then they would build the struct from that with a bunch of unwraps. And it was all very a lot of work for no benefit, right? If you know the tools. There's also just that part, knowing the ecosystem. I find it this kind of work hard to do because you have a working backend and you tell people no you need to start introducing those stronger types you need to start from the very foundation because otherwise you still have the pressure from the outside incoming requests kind of leaking into the api and maybe the service layer and maybe further down yeah and and it's a very fragile, system but if you do it the other way you have to build out those core types and then work your way forward towards the api that's equally hard how did you navigate that. I luckily had the seniority to say no this is how we do it so and i mean the benefits are clear like i didn't i didn't really have to sell it. Did you start from the domain layer did you build a strong domain and then have some sort of onion architecture where on the outer levels you were more lenient with the inputs and outputs? Or did you create a bounded context of sorts? We essentially have an API in front and we would be super strict about the types there and just reject anything that wasn't our thing. That couldn't deserialize into our specific types. And we then did a lot of work to like, okay, let's build a JSON schema, something like that. And do a lot of work on the error types, right? So you make a request and it fails. You want to know why. Why is that? So those things we spend a lot of time on. But then it's so obvious, right? Once you have your own type, once you have the whole thing structured and you know it's like this, you know that the enum can only be these five variants. And if it's the third one, it has, I don't know, a field on it, right? then it's so obvious that this is good and creates code that we can maintain. That means you receive data and then you try to convert it as quickly as possible to a stronger type. At the very outer layer, yeah, yeah, exactly. We did have, when we're calling other endpoints and you get some structure back, right? Like that's not our JSON structure. Like if we call another SaaS service and we get JSON blobs back. That often would be more loosely defined just so we can retrieve the thing and then we would convert it into our own. And that's in part because we would take the opportunity there to validate a lot of assumptions about the data. Yeah, maybe like, oh, you can only have one subscription or you can only have, if you have this add-on, and you cannot have this other thing. So there's a lot of like, Yeah, validation, essentially. Parsing and validation of the data that we store in other services. When you store it and you retrieve it back, do you do similar conversion from a raw type that you stored to a stronger type in your service? Or do you trust the database to return the right thing? We don't actually have a database. We work with other vendors, I guess, like other companies that takes care of all the, you know, having an account and a subscription and the payment and all these things. But it's all so loosely defined in there that we have our own assumptions about what does VAO want here. I mean, there are whole things that we don't use, right? So when we then fetch data from them, we make sure that these things are not present. Do you convert them to a different type or do you use one type from the API? No, each request and response has its own type. I like that sometimes call it the pushback pattern where you try to push back as much invalid state as you can on the boundary to sort of build a safe space that you can work with. Exactly, exactly. Do you use a lot of try-froms to do the conversion? Or is it mostly 30? What's the serialization logic looking like? It is so much certainty with a lot of our own deserializers instead of the derived one. I mean, I want to say maybe 80% is just derived serialize. But then sometimes we do want to make sure that a, let's say, a specific ID, conforms to a certain structure. So in that case, we would make our own deserializer. How much of that code is async? All the requests that we use. Is it a request? Yeah, it's a request. And that's async code. The whole thing is async. Of course, deserializers are not async, but everything else is async. And we embrace that, right? Because I think that's what everybody does with servers, essentially, like handling incoming requests. Async is amazing for that. It is what most people do one could make the argument that you could go into structured concurrency on the other side to kind of have a boundary between sync and async, so distributing work across channels but it's very much depends on the nature of the work Yeah. Yeah. And in particular, the temporal, like our code is very structured around the temporal workflows. And a workflow is just one function that is async, has to be async, because you have to be able to stop it at some point and replay it and start over and stop it when you try to call something that you don't yet know. So for us, async is definitely the correct way. And it hasn't been a problem either. In terms of learning, there hasn't been much of a problem. You talked about temporal workflows. What's that? Temporal workflow is at the core of temporal. Temporal essentially is just trying to solve the issue of having long-running workflows. And so a workflow is at the core where it's essentially like if you strip it all down it's just a deterministic function, so all you have a function that will do things like it will call out to another service, it might wait for a signal to come in like in our case we might process an order so we might do a bunch of things and then we wait for a customer so we wait for a signal that the customer actually paid the money. And then we tell the shipping company, oh, we need to ship this many cameras. And then we pause again and wait for the shipment notification to come back. And then we withdraw the money, let's say, or we send out the invoices or what have you. So that's the core. But all of these side-effective things, because it's a deterministic function, they happen by telling the context that is provided to the function that, oh, I would like to run this, what's called an activity, which in our case is just HTTP calls out. So you say, oh, I would like to run the, yeah, what's a good example? The fetch account activity because I need the account details for this. And then what Temporal does is instead of continuing the running, it will just shut down the workflow and it can do that because it's async. There's an await point there. and so the workflow function hands over control to the, Tokio, I guess. And so it just shuts everything down and then runs the activity you want, gets the result, and then it starts up the workflow again. And it gets to the same point and you say, hey, I would like to call the fetch account. And then it says, oh, that's funny. I have the response for you right here. And then it continues. And then you say, oh, now I want to wait for the signal. And then, okay, I don't have a signal. Let's shut everything down. And then it waits for a signal. And when it gets the signal, it starts it up again and yeah, runs it over again, right? Replace it until you have the point where you receive the signal. That means the temporal crate has first-party support for async Rust. Yeah, yeah, definitely, definitely. That means they wrapped all of their code into a future that you can, you know, just use with Tokio. Yes. Isn't temporal also written in Rust? Not sure. No, I think it's Go, actually, on their back end. And when we started this project, there wasn't really an SDK. They had some code that was, I think it was for internal use, I'm not sure, but it's open source because that's how they roll, which is nice. And so I took that and built an SDK for us. So it wasn't like fully featured SDK. It was an SDK of the things that we needed. So one thing I noticed is it's actually... What we're doing, like both of us, what we initially picked Rust for, right, for the temporal thing and what we use for Dixiemer, that is like the core of Python's domain because what Python is really for is for linking things made in other languages and other things together. Like Python is super good at that. You make the hardware performance things part in C or C++ or whatever, and then you just have an interface that you use from Python. And this is exactly like where you would expect to use Python. That's exactly where we've chosen to use Rust. So in T-Streamer, we basically do the setting up of the pipelines and running and all the orchestration of a pipeline that you were sort of supposed to do in a language like Python. We do that in Rust. And it seems like also in Temporal, the idea was that you should sort of use Python or maybe even JavaScript or Ruby to sort of orchestrate the flows, but you use Rust to do it. And I think it's very interesting that Rust can sort of slot in to something that is really where Python is just traditionally the right choice. And now Rust is not just a different choice. It's also a choice that feels better once you start using it. It feels like a stronger and more mature modern choice. Yeah, I found it interesting, too, because as far as I understand, you come from low-level programming, quote-unquote, and maybe you were reaching for Python because it was just quicker to write and you were able to get to hit the ground running. So you had those high-level forenses, and GORM, you kind of wanted to go down lower on the stack, but you end up meeting in the same place because, well, with Rust, you don't have to make a compromise. Now you can have the best of both worlds. Yeah. That's also the, at least for me, that's also the thing. You're exactly right. But there's also the thing of just having things work. Like, I'm so tired of things not working, like of code not working, of libraries not working. And I feel like when we write things in Rust, they work. That's what I see. I mean, not 100% of the time, but they work. Like to a much, much larger degree than when I write something in Python or JavaScript or whatever. Because like you'll write the initial version in Python maybe and everything's great and you make it. You're going to run into all the issues at some point. At some point, you run into the global interpreter log. There's just no way around that. So you might as well face them up front. It's not only just a global interpreter log. What I find frustrating working with larger Python code bases is that there's easy escape hatches for all the type safety. If you're not strict enough, you can always sidestep the interpreter and do weird things. and then everyone on the team has to suffer. And that's frustrating because you look at a piece of code, in isolation it looks fine, but you never know how it can catch you. There might be weird things, spooky action in the distance, given that Python is such a dynamic language. Yeah, and it's hard to lock down the interface, right? In particular in Python where everything is open. Like anyone at any point can always go in and call the function that you were trying to hide in some file deeply nested somewhere. But yeah, you don't know. And so suddenly you're not sure that you can change it. Yeah, it is like the nicest feature of Rust, I think, that it just works. I think I have professional experience with, I don't know, a lot of different programming languages. I've worked with the Microsoft stuff like C Sharp and I've worked with C and C++ and TypeScript, and I've even worked with Perl and of course obviously Python a lot and I think the only language I've worked with that has this feature that things just work other than Rust is OCaml and, And I think that's one of the, like when I stopped working at our previous company issue and started in Veo, that one of the things I really missed was having a language where you could write stuff that just worked and you never have to think about it again because it just works. Why was OCaml not that language that became so successful? I think the syntax is too esoteric. I think if it had the less esoteric syntax, then it would have become that language. and they tried that. There's this other thing, the other OCaml. I don't even know what it's called now. Is it ReasonML? Yeah, ReasonML. That was OCaml with a different syntax. But I think that failed just to lack of publicity and maybe also it was too confusing. I 100% agree on the... Yeah, just how... Like ML in general is hard to read if you're not used to it. And at least when I was trying to work with my other developer back then, trying to teach OCaml, just something like the currying is so elegant and it's so nice and it's so confusing when you have six parameters and it's like you think you have a value, but you just have a function that, okay, and then later then it's really difficult to find your way when you're not used to reading ML. Also something like in, so Rust have generics, which can be hard to understand, but at least most other popular programming languages has a feature that's similar to it somehow. So most developers would come to this knowing what it is. But OCaml has something like generally applicable modules that is very, very different, but sort of solves the same purpose and no one would ever, like it takes a lot of tries before you even understand what the feature is for and how it can be used and how to read it and understand it. And I think this is like... Rust is just, even if it's different in some ways, like I don't know if any other languages have a borrow checker, you can come to it and you can always find something that you think you know, like the syntax is, it looks like C or C Sharp or something. And generics, the way it trades, look like interfaces in Java. And it looks familiar and you just need to know, sort of learn what the differences are. Is there a risk that Rust will become too complicated? Of course there already is, yeah there is but honestly I felt a bit worried two or three years ago about exactly that but I also feel like when I read the, the updates like the changelog for the new versions there's nothing, crazy going on right like it's just even when async came out it just felt like or was it async traits when that came, it was such a big feature, but it was just like, oh, I don't need to use that async trait crate anymore. Great, it just works. It wasn't a big deal. But it still comes up all the time in discussions, especially about async, that there are all these things that are not immediately obvious why you can and cannot do. If you have experience with async from, say, TypeScript or C Sharp or something where it looks similar, but because they have garbage collection languages, there's a lot of things you can do in those languages with Async that you cannot in Rust. And that can be hard for people to understand. And I think this is why it comes up so often that Async is like, do you need Async or is it better to not use Async? I think that's because there's some complexity in it that is not immediately obvious. Part of the complexity of async that I see in workshops is that it's a weird split between stuff that is in the standard library and supported by the compiler itself and stuff that is outside of the standard library and is sort of wrapped into a library. That's its biggest benefit and its biggest threat because the standard library documentation is amazing. It's not always the case that all of the edge cases of async are documented well, and people run into weird compatibility issues. But that has gotten much better in the last couple of years. Back in the day when we had async STD and Tokio, that was a bigger problem. Nowadays, it's mostly about standardizing the core concepts. It also depends on what you're building, right? like how complex your async code is. One thing is to write a HTTP server and trying to get async handlers working and all of that. Another thing is to write the handlers like just an async handler. And I think, I mean, at least coming from the front end and coming from a more high level, it's fine. If you want to do an async handler, like at least for us, that hasn't been the complexity. Then I would say the borrow checker is much worse in that way. From a teaching perspective people get scary looking errors that are hard to explain when suddenly the compiler asks you to add send and sync bounds or you need to pin certain things for the compiler to be happy and that's still a bit of a leaky abstraction. Yeah, that's a good way to put it. But I agree that if you use it just to get the job done, it's amazing. You don't have to deal with any of that complexity unless you work in the trenches or unless you make a mistake and don't understand the error message, and then you can backtrack. But you have to learn the methodology behind it in order to backtrack. Yeah, I actually agree 100%. I mean, now we worked on this project for a year. And so far, I don't think any of the other guys have seen pinning, for instance, or had any of these horrendous error messages that you can't really wrap your head around. But I, on the other hand, have, because I built the SDK that we were working in. So I definitely did run into all these. And it's difficult, right? It helped that I had to build sort of the same thing in my previous company. But it also helped, honestly, to have an AI, you know, you can copy-paste your error message and spend an hour trying to, like, dig into and ask questions about what does this mean and what does that mean. And eventually you learn it. Like, it's okay. So, but yeah, I think there's a discrepancy between, I don't know if it's a discrepancy, but, like, there's a difference between, yeah, using async and building for async, essentially. And the story is much better for using it than for building it for it still. Where do you strike the balance between pragmatism and idiomatic Rust? So for us... We did a lot of work around, also because we wanted the API boundary to be really nice. The stuff that we return to the front end, because that's a different team and we want to have good communication there. Because otherwise they just call us all the time and ask. So it's better to have to document it up front and being obvious about it. So we actually started eliminating options. And my favorite example there is that for a subscription, it may or may not have an end date. It may or may not end at some point. If it doesn't end, it auto-renews. And so instead of having that, we said, okay, let's put a field called renews, which has an enum that just says, it's called renews. And then it says yes as one variant or no with an end date as the second variant. And so we try to do this as a way of documenting what is the option, like why is something an optional or a result or a, you know, so we try to spend that extra lines of code just to make it ultra clear what this is. And I think that has helped a lot, both on the boundary to different teams, but even as the code increases in size. I did a line count just before I came here. It's more than 50,000 lines now. It's, I think, fairly big. So even for ourselves, it's quite nice to have this, like an enum instead of just an option. So when we use enums instead of options, it also eliminates the problem of when you do a function call and you see none, none, none, none, or true, true, true, true, or string, string, string. And that's also a very good reason for doing these new types, which we also do a lot. Because we have tons and tons of IDs and numbers, right, order numbers, subscription number, account number, and again, subscription numbers, IDs, you know, There's just so many small bits that are really just a string, But to us have meaning and should not be confused. Because if you try to look up an account by the subscription ID, you're not going to find an account. And so we do new types a lot. A lot. And it's a thing where we like periods in programming, right? Where you're like, okay, everything must be a new type. And then we do everything as new type. And then we fall off a bit because, oh, we need to deliver. We need to deliver. It's like everything is a string suddenly. And then, wait, wait, no, let's make this a new type and this one. And then, okay, let's continue. So, yeah, we use the type system a lot like that. And because we do so many small IDs, they're rarely larger than a UUID, usually just at most 10 characters. Because of that, we also are very, very liberal with cloning. In the beginning, I wanted us to not have so many clones, but then I think one day I was just like, no, it doesn't matter. It actually doesn't matter. Let's focus on something else. Let's focus on the type safety instead, for instance. Anders, where do you draw the line between pragmatism and idiomatic code? Probably draw it lower than Gorm. Yeah. But this is also because sometimes you have the C code to look at what it was before and then it looks so nice just that it is now Rust and it has program types. So we also use a lot of cloning, but this is because almost everything in our environment is a smart pointer and they are cheap to clone. Pragmatism comes in when you have a deadline. As I said in the beginning... When we initially introduced Rust, we were sort of on a deadline. We had a product that needed to be finished, and we just needed it to work. And when it worked, then we shipped it, and then the code was in the state it was. And then when you add new code to it, you can sort of look at what the old code was, and maybe you don't raise the bar always as high as we maybe should. So there's a lot more none in our code. It's also, I think a lot of our code is also, it's library integration, right? We call a lot into the G-Streamer bindings and if they return none, then there is just a none there. And if the function takes option as an argument, then that's what it does. And we just follow that. And I guess you could sort of make an abstraction on top of that abstraction to sort of say, yeah, but this is the view wrapper around GStreamer that follows patterns that we think are nicer, but we haven't done that. We follow sort of the guidelines from the official Rust GStreamer plugins, which also means that we use a lot of unwrap because that's sort of what the official bindings are made for. They have this pattern that if something if there's some expectation that you as a programmer know is true then you should assert that, in the code and it's just an error if it happens in production and you want these stack trace, which you don't get so easily if you use like a, the nicer error handling, things like error. So we could use expect and write messages everywhere. This is why we expect. But if you have 10 things that you know will succeed and it's a complete error, the programs will just crash if they don't succeed. And instead of writing, this succeeds because it just does. And this succeeds because that's what it does. In the expecting, then you quickly get used to just writing unwrap. So we end up doing that. That's interesting because we have the same pattern and because we're dealing with money we have to be very sure that what we're doing is correct and so, we don't just do unwraps like we will actually make an assert and make a long documentation about why this is like this but then also crash like we will still crash because we should not be here right we shouldn't continue if something is violated like that, But yeah, so yeah, I was just to say, we do actually a whole chunk if we do this, where I think you're more like, yeah, let's just do an unwrap. It's fine. We get a line number and we can figure it out from there. Yeah, I think it's also... The stakes are different, right? If I'm writing an element that has a gstream element, it receives some data, and there's one pad it can receive data through. So I know that that pad is there, and if it's not there, that's a complete programming error. Someone did not write it when they described this gstream element in Rust. They did not write, this pad is there. And then it's very annoying every time you have to refer to this pad to sort of say, get this pad, and then if it's not there, then log an error, but if it is there, then do this. And there's just so many of these functions that is in the GStreamer API that is by necessity, they can do a lot of different things, but in the scope of a specific GStreamer element, they will always only return one thing. and, In those cases, we typically just assume there's one thing that we know it is in this element and unwrap or expect as early as possible. And then we know we'll get compiler errors if someone changes it. Sounds like there's a need for a higher-level GStreamer API on top of the low-level one, which enforces those invariants. There could be, but it sounds like it, especially when we talk about it like this. In practice, it's actually kind of nice to work with. And I think you can make it nicer. It's just because we are kind of cheating out a little bit. One thing I think could be nicer because most of these functions have ways to propagate errors, but we chose to not propagate the error because this has turned out to be very hard to debug. Then you just get the notification that there was a failure somewhere. There was a missing pad. and what you actually as a developer want is here's a stack trace it failed right here because it was supposed to be a pad and there was not and the, It would be convenient if there was a way to add stack traces to errors in an easy way, similar to how exceptions work in C++, for example. So you still fail early and you still push back, but you say it's okay to unwrap. I agree with that in part, because the only thing that's worse than no abstraction is to have the wrong abstraction. If you build up a library that is just not great to use or doesn't cover all your use cases or it's just leaky, then it's not helping either. Sometimes the pragmatic approach is the best. In our case, it's not so much that it's pragmatic. It's more of a... For us, it's not shorter to return an error because a workflow can return an error. It can error out. that's a very valid thing for us it's a way of documenting to the next developer and to ensuring that documenting the invariant and making sure that it holds, and so we for us at least we only do it if it's a programmer for sure like it's never like oh I couldn't find that account or I couldn't you know do something it's only if yeah if we have, something that should hold true, an assertion an assertion essentially yeah yeah exactly now if it's not that we use yeah just your normal results results type everywhere. Where can people learn more about Veo? They can learn more from our website like Veo.com and then on the website there's a career path and they can look for jobs we're hiring Rust candidates also so that would be awesome then people can come here and learn more about Rust what's. Your message to the Rust community? community. Keep making awesome things. Like this weekend, I was thinking about text adventures, which is from the early 80s and 90s. And I knew there exist like virtual machines for writing text adventures with esoteric programming languages that are really good for text. And it turns out that there exists a framework called bedquilt which ties into WASM compilers and all sorts of wackiness that allows you to write Rust code that compiles to these text adventure virtual machines something called Glulx. And it's just amazing. I was just so impressed that this exists and that we've been talking about high-level temporal workflow things and low-level GStreamer pipelines and we also briefly mentioned you could write Rust in the GPU and just keep more of that, like everyone, if you have something strange to make Rust for it. Okay, and for me, I was thinking about it this morning and I remember when I came from TypeScript and actually the first backend language I was learning was Go. And I like Go. I like it a lot. I like the community. I don't want to say anything bad about them. But there was this specific thing, this thing that, I don't know if to call it a motto or whatever, but this "don't communicate by sharing memory, share memory by communicating". And at the time, I just didn't get it. I didn't understand what that meant, in particular, because TypeScript, you don't have parallel computing. There's only concurrency. So it doesn't mean as much when you come from TypeScript. And I just nodded along, and I felt like, wow, what are we talking about? I don't understand. I am not smart enough. And then I came to Rust, and I was reading the Rust book, and it just goes into such detail about particularly that sentence, like what is communicating by sharing memory and then later what is sharing memory by communicating like what does that mean, exemplified with like channels and a mutex I think it is and so this thing of, Being kind, that's how I think of it. Being like, going to the level where the reader is and the person trying to learn this new, quite difficult language is such a strength in the community. And yeah, I just, I hope we keep that. And the more of that, yes, please. That was amazing. Anders and Gorm, thanks so much for taking the time to do the interview today. Thanks for having us. Yeah, thank you very much. Rust in Production is a podcast by Corrode It is hosted by me, Matthias Endler and produced by Simon Brüggen. For show notes, transcripts and to learn more about how we can help your company make the most of Rust visit corrode.dev. Thanks for listening to Rust in Production.