Welcome to Season 5 of Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from corrode, and today we talk to Alexandru Ene, Principal Engineer at Prime Video, about building modern user interfaces in Rust. Alex, thanks so much for taking the time. Can you introduce yourself and the company you work for? So, hi, thanks for inviting me to do this. Super fun. I'm Alex. I'm a principal engineer for Prime Video London. I've been with them for about eight years. Actually, almost nine. And before that, I used to do video games, game engines and such things at Ubisoft. And right now, as I said, I work with Prime Video on the Prime Video app that you have on your living room devices. And you do a lot of rust there. so that's why i'm yeah and. And when you say you did video games does that mean you also used rust for that or was it in some other language. No that was c++ and c sharp for unity but mainly c++ at ubisoft i don't know what they're using now i left in 2016 and i learned about rust in 2017 18 i think 2017 so i didn't even know rust existed actually for the most time i ignored it thinking it's a garbage collected language but then i saw this guy on youtube doing demo scene type of things with rust which is this super tiny executables that do art and nice visuals and music so i was like oh wait a second it doesn't have a garbage collector so how do they do all of this magic with memory that they keep saying so that's how i kind of got into it and use it then for some tiny emulators in the beginning and then since then i'm still building a game in my spare time with rust i would say that was my big rust project outside of work and how i got into it. So that's interesting because you come from a c plus plus background and i think a lot of listeners also have the same background so what kind of convinced you of rust in the end was it just the fact that it didn't have a garbage collector or did you also learn to like the language at some point. So it was i needed i wasn't opposed to something other than c++ i just hated build systems in c++ they only work on windows i could only use Visual Studio. With CMake, it was always a pain. So when I saw something with Rusty, just cargo new, that was mind-blowing to me. I mean, people from JavaScript and these other languages are like, yeah, we did that for ages. What are you talking about? But for a C++ guy, to do a cargo new or cargo run and things to just work that's just mind-blowing at the time to me and then i started liking bits of the language like enums things like that i really really like those but yeah the build system was just magical i still remember the first time i was watching this guy on youtube ferris and he was building some demo scene things and he starts a new he's building an emulator for n64 i believe and he starts a new project with cargo new and then cargo run i was like what is happening this is just so cool so that's why i got into it. Shout out to ferris he's also one of the first streamers i watched when i came to rust and, he's just amazing i guess everyone should watch watch his stuff on on youtube it's still there highly underrated i would even say because he really built an n64 emulator from scratch not sure how far he got but the idea itself was so great right and yeah i wish he he would do more rust stuff again. Yeah we make a petition to bring him back or at least on the podcast we should have him it's this good guy yeah but. In c++ you have things like conan or package managers I'm not familiar about the state of package management in C++. Can you enlighten us? So I never used any of those. I think they weren't that popular with game developers at the time. So all I was using is raw big Visual Studio and then create a project and then make all my settings. And actually, I made the game with some friends that we ported to Linux and none of us, like we only, we use directly make files, not even CMake. So just put your files in the make file and then in the right place and then it will work. That's i'm not familiar with i never use conan people keep bringing it up whenever i mention build systems but i i never tried and i don't think they supported at the time windows that well i'm not sure what the status was mainly i'm a windows developer i never like only recently i bought a mac i. Think there's something to be said about conan not being officially, part of the c++ how could i say standard or you know it's it's a separate project and i guess there's a couple good things about being separate but in reality, for something as central as a package manager in 2025 it's really beneficial to have that as part of the normal ecosystem part of the runtime that you install. Yeah and it makes for such a nice smoother starting experience I would say so it is a little bit of a shame but yeah I haven't written C++ since 2018 so I'm so happy now because all of my side projects used to be in C++ like all sorts of game mini game engines or whatever so now I have used Rust for that and now I use Rust at work. As part of your work you also probably review a lot of code how has that review process changed ever since you moved from c++ to rust. So that actually is really you probably know because you've interviewed other people but i can tell you i'm so much more relaxed on code reviews so for example we do still have c++ code in in my team for the thing we deploy on devices we have a mini layer that abstracts the system there so that's written in C++ and comes with the virtual machines and all that that the UI uses later on. So that's in C++. And then in RAS code, it's just like paying attention only to you know, code is doing what it's supposed to do, not this. What are you doing with this memory? Where is it coming from? Where is it going? Even with smart pointers or unique pointers, it's so easy to shoot yourself in the foot as a newbie. And that it's For reviewing code, it's definitely more relaxing. I think it's way easier to write correct code when it. Comes to memory access it's just trivial now the only thing i would say which helped rust, adoption a lot is that it doesn't require years and years of experience so for example if i were to propose c++ for the same thing i'm using rust now for performance reasons it would be so hard it's so hard to onboard people onto c++ so unis i did c and c++ in unis but today that's not true anymore and very few of them, the new grads even do C++, very few of them so it's really hard to onboard them, it takes so much time and with Rust it's so much easier it's just like, so basically the team that switched to Rust in Prime Video switched from JavaScript to Rust and it's fine I mean it's a new language, it has some onboarding but imagine, switching people who only know JavaScript and Java to Rust to C++ it's just gonna have It crashes everywhere, basically. It's a given. So I would say for adoption from teams that know other languages, which isn't C++, it's way easier to onboard new people, even new grads or experts. So to summarize, the difference between reviewing C++ code and Rust code is that in C++, you need to be really careful with where memory goes. Memory management is way more central in the review process, whereas in Rust, it's more about the business logic. It's more about the data flow, but not on a systems level, more on a business logic level. Yes, you're just verifying that this code is achieving what it's supposed to achieve, but not at the low level of what are you doing with this memory? Is it safe to access it where you're accessing it? Are you sharing it weirdly between threads or so on? My main problem with learning C++ in the past was that there was so much hidden knowledge. There were so many unwritten things that you needed to know about this language. It was a big, big language that you needed to learn. Whereas in Rust, a lot of things are documented really well. And yes there is a learning curve but after this you have pretty good documentation you have the compiler helping you along and the language feels smaller still. Yeah the thing with c++ is everyone picks their own flavor you know and then there's all sorts of parts of the language that oh don't use that but it's there like as a newbie you don't know what you shouldn't touch pointers are still there like row pointers they're everywhere they're just and it's so easy to just access them to get around certain things that maybe they would catch so like a unique pointer oh let me just let me just grab the pointer out of this unique pointer and do my thing or whatever and it's just maybe share it accidentally to some other place and then save it in some other place and that's how you get use after free very easy so that's the problem whereas other languages which are newer like Rust, they don't have this kind of everyone-needs-to-pick-a-subset, problem. Use the whole language, it's fine. The whole... Features are more cohesively implemented so i think that's a big deal. Well one could argue that one main reason for c++ having all of these caveats is that it's language that has historically grown over time and also is still compatible with c i'm actually not sure if that is still true or if that's an urban legend by now i would like to hear your opinion on this and second i think when you build larger systems in c++ you run into all of these really gnarly data management problems because this will kind of haunt you and bite you later it doesn't bite you at the start usually not at compile time so you're kind of in a situation where later on you have a larger C++ code base, and then suddenly these problems pop up? Yeah, I think one, there is definitely the aspect that it grew over time, and even multi-threaded programming became popular when I was in kind of uni, or just entering uni, when like Herb Sutter and those guys were like, there's no free lunch, they were writing these articles. So it was a language that just couldn't have predicted how the computing evolves and towards multi-threaded and all of those things. So I think history definitely has a role. It's not like they implemented it wrong. They just didn't know. So multi-threading wasn't a thing at this scale until way later on, the history of the language. So there were hard-learned lessons by C++ there, whereas Rust already came with this benefit of hindsight and, knowledge that was kind of occurred over time so I think. Kind of, it's already had way more knowledge. So when they designed Rust, I assume they had way more knowledge about, okay, what does computing look like today? What are the common problems with other things? Let's fix those in a breaking manner. Because it's not backwards compatible with C++ in any way, right? You only can, they don't have that constraint. So I think they did a really good job, the Rust designers and the people who contributed there. C++ keeps trying to do all of these things. There's various efforts, but honestly, I think they should probably stop adding things to the language. Just, it's fine. We keep, you know, you keep adding, they added recently something like option or result type of, and it's just so, it looks so ugly. Don't understand how people can criticize Rust for looking ugly. Like these guys who write c and then you look at c plus plus and c and c plus there's just god what's happening it it wasn't that bad when i started honestly c plus plus 11 okay that's fine it's just that's a good language just stop adding things to it. What do you think about all of these recent security efforts by people like bionis duestrup who try to make c plus plus safer is going anywhere? I haven't kept up to date with those. I know there's some efforts. Herb also, the Microsoft guy, I think Herb Sutter was there. He also had some ideas, even with some sort of a smartphone, but I'm not up to date with what's happening there. And in general, about security in the fields I worked in, the thread models were such that you didn't, there were other, performance was definitely on the first, order of business rather than security so for video game engines for example you're more concerned with that than security especially for a single player game where you don't you don't really interact with the outside basically so do. You think that performance and security are always at odds or has rust proven that you can have both. No you can have both and i think in good c++ you can have both as well today from what i can see the baseline libraries we use are in c the open ssl and those things curl everyone uses curl in everywhere basically and security is super important there so i think you can get it in both languages but with various degrees of effort they're not at odds but they for certain languages they take definitely more effort than others now Now. I'm not sure if you still write any C++, but if so, or even if you just read C++ code nowadays, has your way of dealing with the language changed the writing style or what you consider good C++? So I think one thing that helped me when I learned Rust is it kind of formalized the way you're supposed to think about memory and memory access patterns. So I don't think I was ever reviewing C++ code with the idea of, hmm, who has access in a mutable way to this data? I just didn't have this concept in my head. I mostly look at stuff like, did you free it in the right time, and it's... Maybe a vague concept of lifetimes, but not quite as formal as Rust makes it. So it definitely gave me at least some vocabulary and more structured way of thinking about catching those issues when I review code. Because, I mean, in the beginning, I was just using Rust at home for my side projects and passion stuff. So then when I got to work, I was writing C++ and it made me think a little bit more like, oh, what's the order of access here? How long does this live and it gives you a bit of vocabulary so even if you write c++ i think it makes you a good better c++ programmer yeah. And what have you done in rust at prime video. So i'm just a guy who rewrites things in rust slowly recently completing almost the the whole app rewrite so basically i started working on the so the team i work in is the client team, which is a little bit special because we work on living room devices so basically with the way prime video works we have one application that goes on all of these devices no matter how performant they are so it could be all of the gaming consoles all of the like tvs setup boxes streaming sticks they have the same application running on them which has a tiny c++ layer and it used to have a javascript and react layer on top and basically my work started in that c++ layer and then i moved on and started rewriting parts of the javascript bit in rust and web assembly so instead of vending things over the air with a javascript bundle now we send a wasm bundle and that people download, and we have a WebAssembly virtual machine on the device that runs that code, mainly for performance reasons. That's kind of a lot of the work I do. So recently, we started with the UI low-level bits, like, for example, just the scene management, animations, things like that. Those became Rust four years, five years ago. I don't remember. Maybe four. My timelines are a bit wonky. But we started with a more baseline level. And then today we even wrote our own UI SDK in Rust to replace the React part. And now our UI client engineers, front-end equivalent, are writing Rust as their day job for about a year or so. So this application has already been deployed on a lot of devices. But you can imagine it's quite quite a few given it runs on all of the gaming consoles all of the setup boxes you can think of and tvs samsung lg sony everything so yeah and i guess it also runs on. Some of the consoles like the playstation 4 and 5 and the xbox i guess. Yeah gaming consoles ps4 5 um all of the xboxes xbox one xbox one s there's a lot of them these days but yeah so a. Few people might even have more than one deployment of. Your application somewhere i wouldn't be surprised because at least you have a tv which if you bought it since 2017 or so it's probably one of these tvs that has apps on them so it will have the prime video app because. They they sometimes even come pre-installed right. Yes yes so they this was one of the problems because as i said like the hardware in these devices especially on the cheaper tvs or the setup boxes isn't super powerful even on the streaming sticks they're powered by usb right so you can't have crazy good hardware inside because it's just whatever usb connection can feed so basically ideally we would write our code into something like C or whatever Rust and, From the beginning, let's say Rust wasn't on the table, but we would do everything in C++ and have the best performance ever because we already had it for the native layer. The problem is on these devices, you don't have app stores. So sometimes your app goes with the firmware update. Sometimes it's quite a manual process. So that's why we move more towards JavaScript and React so we can keep shipping bug fixes and updates to customers. As a consequence performance wasn't fantastic as you can imagine and we had to put a lot of effort into optimizing that and even so it was very difficult so rust and web assembly helped a lot on the performance side and it's code that you can still ship and download and has. Way better performance than a javascript code would have because if you imagine javascript today you probably imagine it on a laptop or phone or whatever where you have jit enabled but on these devices you maybe don't have enough memory to even enable a jit virtual machine like a v8 or whatever so non-jitted javascript is just so slow you most people don't know the pain so yeah. Yeah that's one thing and the other thing is i'm not even sure about garbage collection on these devices how efficiently it works because it might be that eventually you run out of memory or it's just very laggy it gets slow when you use it for a longer time did you ever encounter that. Yes so we we trigger so because garbage collection is such a problem in in in javascript non-jitted, we disable garbage collection if we have an animation playing and then we call GC on the VM manually whenever we know it's safe to do a little bit of stalls like your idle or whatever so that was a huge problem as well for us because. These devices run for a very long time and sometimes people have them on for hours and maybe even days at times you can't really just you know, close it and start it back up again. That's not an option. Well, we do that if we... When we... This was actually... This was a better experience, by the way, than we're going to get there probably, but because you have exceptions, if something unexpected happens, you can just catch it and reboot the whole app. We lost that with transitioning to Rust, so I kind of miss a mega try-catch kind of, situation especially with web assembly where you don't have you don't have stuck unwinding and you can't quite do the same things. In raw rust you could unwind the stack that's true you could, technically panic or throw an error and then unwind on that right or yeah if you have a panic you can handle it if you wanted to but if you cross the rust to web assembly boundary this is where things get really hard. Yes, in WebAssembly, it's not possible today. So we lost some features, I would say. But at least, it's in general very stable. But if you do have a panic, it is a very unpleasant experience. And actually, one thing I would say, writing panic-free code in Rust is extremely difficult because you can't quite know when a panic would happen. It could be in one of your dependencies. It could be anywhere. It's not like in... You can't do anything about it. So one of the panics came... I remember because we got paged because of this edge case somewhere that had to do with the way third-party library worked. When the third-party library detected that you shouldn't call certain things in a certain way, it would panic it was like this is incorrect but then the customer has no app anymore man what did you do so then we had to kind of like fix that and patch it and return a result because. The degrees of correctness people sometimes attach to things i feel they're a bit too much at times where we could be a bit more lenient to return a result but don't take down the whole thing because in some deep deep library somewhere you found some inputs that were a bit dodgy so if. I started such a project i'd be concerned that i could never update my deployment once it's on this box it will stay there forever and sometimes people don't do any updates on their devices so it needs to be sort of perfect or at least very close to robust, was that a concern going in and probably you already had the problem before rust so it wasn't a new problem. Yeah we already had the problem i mean the rust code we download as a web assembly bundle so that didn't change this doesn't sit on the device forever this is It's just every time you start the app, we download a new app if one was released. On the C++ side, this is why a lot of low-level code that managed our UI scene, like doing the animations, creating the scene tree with the nodes, the hierarchy of nodes, and all of that was in JavaScript initially. Normally, browsers or other UI engines do this in C++ for speed. But because I had that fear that, man, whatever I write here, we write and stays here mainly forever is not quite true because if something happens you can ask for an update work with the partners but it's a very expensive time consuming process every update would be different they multiply very fast we solved that by minimizing the amount of C++ code that you put on device and has to be there but then we suffered for performance issues due to the fact that we moved it to javascript but in the meantime we got lucky and web assembly got invented and we could take that technology and deploy it and then invent rust code over the air so now. A lot of people have some rust experience maybe even some production rust experience but they've never really written a platform because to me it sounds like what you're building here is way more foundational than what people touch on a regular basis do you have any tips for people who would start with such a large project or would you even disagree and say well it's not really a platform because we already still have the old c++ layer below but we just replace javascript with web assembly. We well, It was easy for us because we had the backup of switching to the JavaScript version, right? So when we were doing this project, it always starts as an experimental, let's see what could happen. It's not exactly like that, but it starts a bit as a research project and you have a way to go back to the old app. I think one thing I would say about, because we replaced React with our own UI SDK eventually, and that is a pretty big project. I think you're right. It is a huge... It's not foundational because we have only one client, one app that uses us, right? So it's not like we're React that has to service billions of projects. But I don't think you can build these UI SDKs without fully, deeply understanding the problems in the things you're wanting to replace, right? So this was for us an eight-year journey overall since I've been there and probably longer because the app existed from before me. And the lessons that you learn are very important in the success of any future replacement. So I would be very cautious in allowing, like, for example, if some team comes to me and they say they want to rewrite something in Rust, the first thing I would test them is like, how deeply do you understand the problems with your current system? Or are you just a fan like me? And it's nice. I obviously am happy when these things happen because I like the language. But I think if you want to be successful here, I would say it's really, really important to deeply, deeply understand all the pain points with the previous thing and advantages, right? Because it's not like we rewrote some JavaScript in Rust and it's better. Some things are worse. Like I can't panic anymore, which is really difficult to never panic, okay? So access an array at the wrong point, that's it, no more app. This didn't happen in JavaScript, you know? So it's a trade-off always, I would say. It's so true because you need to understand where you're coming from and you need to understand your problem domain. If you don't have that knowledge, you're missing out on more than 50% of the advantages of porting to any new language, such as Rust. And a lot of people, they just think they move to greener pastures, whereas in reality, they just switch one problem with another. It's really true and like we didn't start like this right we started with just doing the things we knew rust would do a good job at because we already knew we did a bad job at in javascript which is managing the tree of ui elements so we moved that away from c++ into javascript for updatability but we knew the trade-off there we built it and we're like oh god this is so slow but at least we can update it so when we made the decision let's keep the react app but put the management of ui tree rust and web assembly in a different vm and these two things communicate through some message bus then it was kind of an easy choice starting from a smaller project and it's it has a backup you can always fall back to the thing that works and it gives the team reading space to kind of research here and deploy it. Without even having a billion teams collaborating. So this initial thing, when we moved the UI tree management to Rust, I think was done by seven people over a seven, eight month period, but they didn't interact. The other teams didn't even know we switched the thing underneath them. So it was quite nice. When we rewrote the UI, everyone knew because there's no other, there's no way you impact a lot of people but at least you come from a more experienced, you already deployed it in production once you know you have quite a bit of experts in the team so it's a bit of an easier sell. I guess the transition was also easier because you kept the react part and then you had this bridge to web assembly and to rust so you didn't have to replace the full thing it wasn't a complete rewrite from the get-go it was more or less a gradual adoption yes. Yeah we we did one thing at the time like that one screen page at the time basically yeah. Interesting that you went page by page sort of because the way i thought about it was you kind of started with animations or was that not the entry point? Yeah, we started with the animations and those things, but the pages were still in React. But by the time we started, moved pages to Rust. It was one page at a time because we didn't want to have on the same screen both JavaScript code changing the screen and Rust code changing the screen. So the way they're separated at the lower level in the engine, it's two different worlds. Like in the ECS world, they're completely separate entity component systems for those worlds we use entity component systems there which is what bevy and those engines use in rust so it's a bit like a 2d engine but like two different separate worlds for us so we didn't want to deal with one thing if you do that which is very problematic is focus management so focusing a thing that may have been a react ui element from a rust ui element it's just, very brittle to keep track of all of those things so it was simpler for us to move a whole page at a time it simplifies things which helps delivering it faster then that. Means that if you use the app you might have switched between javascript and rust multiple times throughout the interaction. Yeah, yeah. So we started with three pages, and then we kind of ordered them in order of use and complexity. This was one of the things that was called out by people. In Amazon, we review a lot of docs with directors, VPs, and so on when we do these big changes. And a lot of people are a little bit concerned that it will look jarring to go from the support smooth Rust pages to the old JavaScript pages. And how do we make sure it's not like shocking. But it was fine in the end. I mean, it was unavoidable. Did you start with the slowest pages or with the least used pages or what was the strategy? Well, we started from the, when you start the app, you see the user profiles. So first we replaced that profiles page because of a few reasons. It allowed us to put SoundCloud in production. It wasn't a complicated page because it's like three buttons with a bit of animations. So we didn't require that much UI engine bits to be built. One thing to remember is this UI engine didn't exist, right? So it was developed at the same time with the app pages. So we picked something that didn't have too many widgets because the UI engine team that made this kind of React in Rust equivalent, built like the concept of a button of focus management and that's it and then we started with that profiles page and once you selected it you went you go to the main app page where you have all your movies and we call it the collections page so the main app page was the second one that we replaced and then from there that was the most complex one and then from there we just replace things on the journey to playback basically it was more in order of complexity plus use i would say do. You have any metrics um in terms of usage of the app did you see the number of panics spike or the latency go down. We saw latency go significantly down especially input latency on the worst devices went from stuff like 400 something milliseconds on the worst device to 30 milliseconds. So it was a huge drop in input latency, mainly because the way React works is It's a very complicated way. And it does a lot of work to figure things out and to figure out like divs in the tree and layout is also very slow. So we simplified a lot on that critical path while allowing for certain cute features like every property could be animated and things like that. So simplifying those allowed us to have quite a huge gain in performance. That plus Rust being just a, Rust and WebAssembly just being a faster option than equivalent JavaScript code. So it was, I would say, two optimizations. One in the logic that we simplified, and then you get the base layer performance boost from Rust. And I can certainly relate to that because at least to me, when I use any of these applications, the input feels like some of the slowest parts of the entire application. It's like annoying if you type a letter, for example, and it takes ages for it to appear and annoys me to no end. So it must have felt like a free upgrade of my hardware somehow. No, people did message me that they knew I worked. So it's like, oh, you finally updated it on my TV. It's like, yeah. So you could definitely tell, especially since this rollout is usually gradual because we wanted to also monitor if they impact like business metrics, if people use the app more, watch more video. And those experiments are a bit more long running. So not everyone got it at the same time. did. People only notice it based on speed or were there also any ui differences so did you tackle updating the ui at the same time or did you try to be one to one compatible with the old version. We were well neither so both the old version so both javascript and rust so the prime video app went through a redesign and the redesign happened on both JavaScript and then the Rust bit implemented the new look only. So they were both, both of the UIs were getting kind of a redesign and then Rust came and just did the new thing. We didn't try to do much more than add a few animations I think here, but very few. So you have a bit of a, because we wanted to showcase layout animations, those are animations that, move an element in such a way that it requires you to recompute the layout of the other elements in the scene just to showcase that there it's really fast on the new ui engine in rust but that's just i think in one or two places used or it was we try to keep them as close as possible because then it kind of invalidates all the you can't quite attribute oh why did people use the up more on rust maybe it was because of the animation or was it because of the it's hard to find the cause so we try to be as one-to-one with the javascript version as possible right. The new version had better performance and i'm assuming it also used less memory did that mean it allowed you to do more with the devices eventually because now you have more headspace Now you could experiment more. On the memory side, we use more, actually, because we have still the JavaScript version. In the same like we didn't delete javascript code if we have it in Rust because we kept it there if we needed to switch to a fallback javascript full javascript experience so overall with memory that became quite a bit of um quite a bit higher i'd say because there is now it's like two apps there obviously as we port things over the js part will be deleted or minimized for things like just having logs there or whatever it's another kind of free thread that you can shove things to with a message bus and then they can do some work for example right now we send metrics and logs from javascript to our backends so from rust app we collect them we send them to javascript but JavaScript massages them in the right JSON format and then reports things from time to time. So that bit probably is not as critical to delete, but yeah. So running JavaScript and WebAssembly VMs in separate threads is interesting. And how do you handle synchronization and stake consistency between those two worlds? You mentioned messagebusses maybe one or multiple, I don't know. Is that the way you generally handle memory synchronization, or are there some other patterns? We don't quite fully share memory. Well, you can't fully share memory because the way of WebAssembly memory model is. So we just send messages between the two virtual machines, and through messages, they synchronize their state. So, for example, let's say you want to synchronize the history. Stack like if you press back where where would that go you just send a message that hey javascript updated and so on and then it's just message passing that's that's all it is and similarly javascript can tell the Rust vm it's usually one of them is the source and the other one is just synchronizing with the source so there's only one source of truth that propagates its state to the virtual machine that just follows i. Guess one of the big advantages of using a message bus here and please correct me if i'm wrong is that you avoid interrupting either world so for example if the wasm thread does any work it it does not need to react immediately to a message coming in right. No yeah and it's also way easier to implement than anything else basically so you can always poll a message whenever you want, whenever you have time to deal with them. Exactly. So you don't, they're not, obviously if it takes a long time to process a message, you've blocked the UI there. So it still has those problems, but it gives you a bit of wiggle room on how you, yeah. And while you did that transition, did you fix any bugs? Did you find any logic bugs in JavaScript? Any incorrect behavior that you were able to clean up then? Or even just technical depth as well to reduce the maintenance burden? No, but I can tell you one thing I did. When I was porting things from JavaScript to Rust, I found a place in the UI that was a bit dodgy. And I did it the right way. I think it was to how some transparency was being calculated, or I don't remember. And I broke the app. So basically, then I became very careful into telling people, please maintain whatever JavaScript was doing. That's the ground truth. We're not thinking here, we're porting it over. Let's not try to... It's really easy to fall into the trap of, oh, let me do it the right way, but then you don't realize that other things on top of you depend on it being wrong. It's that whole classic meme of whatever the API is doing is what the API should do. And yeah we got bit by that a couple of times me specifically because i was feeling so smug about it's like ah look at this silly thing i fixed it and it's like hey Alex did you notice this thing doesn't work at all no no so yeah it was with i would say because i rewrote a couple of these systems one of my lessons learned is to just move it with whatever behavior it has. Try not to be smart about that and then you can you can iterate on things later but it already these projects of rewriting things i feel they take longer than people expect especially since they're usually quite optimistic like oh what's what's so complicated about a ui tree you know it's just a tree has some properties you update it you traverse the tree you update the children and can do it in a week if you're very naive about it. But then it turns out it takes a year and then every little bug in the thing needs to be translated because there's stuff that depends on those bugs that people put workarounds for. And then they work differently if you change your code. So I'll just keep things as they are when I translate them to a new language and then iterate on them, I would say. I would advise against trying to fix anything. Yeah, I fully agree with that. Because otherwise you would tackle two problems at once. So I know a lot of companies who did the same. They started to put over the old logic, including the bugs and everything, and then started to clean up. That's a way more productive way to work in that area. Especially for people that are not know like especially for people that are not comfortable or not experienced with porting code it's a skill that you need to learn it's it's a different skill than writing code from scratch because you're taking code that already existed and is running in production yeah. And what's correct in this scenario is the behavior that is already there They're not what you think should happen. So what correct means changes if you're replacing a system, I'd say. And speaking of old behavior, you must still be able to handle old devices as well that don't support WebAssembly. Do you have two code bases or two separate code paths or some sort of fallback? We still have the javascript pages that are maintained in parallel i would say they're not getting all of the new features like all of the new animations and whatnot but because there's significant even at low percentages worldwide those are a lot of customers that are still, due to, well, historical reasons, they're getting only the JavaScript pages because we didn't put Rust on the C++ layer yet. So we still have to maintain those. And we're going through efforts of, kind of backporting this native WebAssembly VM so we can allow them to run WebAssembly. But for now, they're both maintained. Okay, so that means we are in 2025 now. How big is the code base nowadays and what's the team size? So team size is around 100 or more contributors to the whole project. We have a lot of people from backend teams that just want to surface some messaging or something like that, and they have to write a bit of UI code. So that's why the number is so high. And maybe on the code base size, below 100,000 lines of code and more than 50, I would say. I didn't check it recently. Also, it's a little bit hard to check because we rely heavily on macros because we replaced React and we wanted to give people this kind of JSX-like experience of compositing UI. So that expands quite a lot. So maybe you write five lines of a macro where you compose some UI elements, but then the code generated by that is quite significant. And so it's hard to tell, yeah. How would that feel like if I were to write a component in that system? Would it be easy for me even if I wasn't a Rust developer? Or would it be very specific? You mentioned the macro and it being a bit JSX-like. Would I be able to understand as a JavaScript developer? Or would I need to learn Rust for real first? So I think on the UI programming side, So if you're a front-end type of engineer, you don't need to learn that much Rust to be able to write code. The reason for this is that we are very relaxed with copying data in the UI side, and we wrap stuff in RC pointers and things like that. So they don't quite hit ownership issues that you are usually having to deal with when you write new Rust code. It's not super optimal, but it doesn't matter compared to JS is way faster anyway. So we used to use Leptos for this framework for effects and signals. And this is very familiar to people who know React or SolidJS and those type of things. So in that sense, it's a bit easier. I think in general, when you go to Rust from a higher level language, it's easier because, I don't know what's happening in that browser underneath me when I write React code. This code sometimes is being called once every time I click a button. This other code is being... Because it's not like you read the code and that's how it's executed. It's always they go into different places. So I think in our framework, if you read the code, this is how things are happening. We just expand it so you don't have to write a lot of things by hand. But there's no magic, except maybe in the effects and signals, which are still a little bit like magic because they're functions that are called when you set a value on another thing. How does it know to call the function? It's just a little bit, it's not quite intuitive, but it is intuitive to people who know React because they got used to like effects and this type of behavior. For me, it was just so weird to have this kind of, oh, whatever, what's happening here? And it turns out what's happening is very ugly. It's horribly ugly, but, you know, it's the way reactive UI works, right? Did you have to port that stuff over to Rust or did you find a different way to solve that? So that is already done by Leptos and there's a few reactive frameworks that we relied on. we eventually moved a bit away from it because we wanted to optimize certain things. And I think the open source projects is going a bit in a different direction. So for simplicity, we had to change some stuff there. But basically, there are libraries who did this. And I think those type of libraries pushed what's possible with Rust UI in general and making it more pleasant to use for people who are front-end engineers. Do you use a fork of Laptos now that you maintain yourself? And which libraries did you mean? So there's a few reactive ones. I don't remember the name. We used to use Leptos and then forked it and then wrote our own thing on top instead. And the libraries that you mentioned, the ones that allow you to do the signal or event handling in Rust? No, this is the signal. So Leptos is basically signals, effects, things like that. they do provide also a way to declaratively do your ui but we didn't use that we only use the signals bit. Very early on in our conversation we talked about game development and i wonder how much of that design was inspired by game development or how can you compare those two worlds i understand that this is not a game but it feels like you have things that are similar to games like reacting to events some sort of ui some sort of event loop and maybe can you compare those two worlds and how did you model that i'd. Say basically it's a lot like a game engine under the hood so right now bevy popularized this but we did use we also use entity component systems for all our low-level bits. So while we kept front-end engineers happy with leptos-like things with effects and signals and this reactive way of writing UI, under the hood, it's basically a game engine, more or less. So it has entity component systems, it has systems. Every UI element becomes an entity, and we attach components to that entity based on what behavior it needs to have. And then we have systems like the focus system that looks at every focusable entity and takes the focusable component and does stuff based on your input. So it's very, very close to game engine. One thing we maybe didn't add and are added later, which I think makes it a bit different than a game engine, is async stuff. We didn't allow async at all, basically, because you had no runtime. So you can't do anything with a future, right? So I wish we had added stuff earlier. and that's not a concept you usually find in video games, at least not the way you write it in Rust today with async whatever and tokio and all of these things. It's more like something you find on the back end because async programing is like waiting on multiple things, whereas what a game does is processing multiple things. So it's more like thread. You create some threads and you do work on those threads instead of waiting. So that. Was a concept that was added a bit later. So my understanding is that with entity component systems what you want to do is to work in parallel on the same entity just looking at it from different angles a certain facet of say elements and you only deal with these facets but you do deal with them in parallel whereas in a classical let's say io bound system you deal with things concurrently as they come in and so is that where those two worlds collide where you have events that are handled asynchronously or concurrently whereas you kind of want to update the ui in parallel or you want to update all of these properties in parallel with your entities? So the place where they kind of interact is when you have UI elements that are waiting on the network call. This is not very common in video games, but it's very common in UI applications that most of their data comes from some backend. It is a little bit on games because you get data from disk that you process and then it gets to show up on your screen. But async programming is a bit foreign to game developers as understood by this kind of idea of a promise or a future that you pull on and it's not a thing that happens often in games, I think. So because our UI engine is very driven with responses from the backend, And I think we could have added async stuff a bit earlier. I personally, so I was the tech lead on this project. And I think it was a bit my fault because I didn't quite fully understand async because it's just, I think async code is very useful, but it's also very hard to read code and have a mental model of how it gets executed. And for this reason, I decided to, okay, let's keep it a bit simple, not have any async, and then give you some callbacks. But then it turns out actually some things would be simpler with async if you don't overdo it. So we added it eventually recently. So now you can declare a UI component that shows a spinner or something, and it will automatically be replaced by something else when we download some data. And that's done with async now. We call it deferred components, I think they're called in the wild. But for those scenarios, I think that's where it differs from a game engine. But other than that all of the graphics all of the traversal of things all of the entity component systems that we have are very similar to a game engine yeah. I could also see two approaches to that problem one would be the simple way to say you have a proxy around your component and the component itself is completely sync whereas the data proxy that is around it kind of is async and then passes in the data that it receives right that is one way and the other way would be to say let's completely embrace async and do everything async even in the components and it feels like you went from one to the other at some point or. You changed your mind on this we didn't so we went with the first one that you described so basically you kind of wrap it into something and then because it's a async block you can write your code and await on network calls a bit nicer create the ui component but then the ui itself will be just it replaces something in the tree and that's just normal code once it ends executing that future this is the difficulty with async code is very kind of viral it just spreads everywhere and then you need to have this sync points with non-async code or, You either go all in and it's async everywhere, but we didn't do that. We just allow a sort of a wrapper. It goes into a task that executes it. And at the end of the day, the UI code you're going to get is going to be normal code. It goes out of that future at some point. Usually once we're done with the processing of a network call or some message pass that waits on JavaScript or something. yeah. In this case it sounds like it was easier to keep the lower level components the leaf nodes sync and have an async runtime around it. Yeah yeah because ultimately in ui a lot of patterns are like set timeout and those type of tasks can easily be async tasks as well that you then run from time to time. And that's another place where we... You just, as a UI engineer, you have this way of creating a function and shoving it in our executor. And then whatever comes out of it might be attached to the UI tree later. But it's kind of like an extra feature that you get rather than all the UI code is async everywhere. Do you port a lot of promises from JavaScript to Rust or was it not the case? There were some around, things that deal with network usually are like that with fetch and then do something and then you wait on those. We have just a way right now that you can say you write an async function you do your way however you want and then you take that future and send it to us with our api which is a task and this is the task and then we pull it whenever we have time usually per frame but yeah yeah. That means you don't usually use the vanilla futures you have async functions for the most part and then you, pull them from the outside but you don't build up your own future as you would with let's say a vanilla JavaScript promise. No, no, no. Yeah, it's an async function that returns the, yeah. And will you be able to use some of the newer features of CPUs nowadays with SIMD, for example? So we do work a lot with the WebAssembly. So we contribute to the WebAssembly VM. This is on GitHub. It's a WebAssembly micro runtime. So we were quite involved there with the community and also in the WebAssembly community working group. So we worked there on adding threads to web assembly simd is another thing that we are quite keen on that was already done so with these things it's a little we're suffering from our own eagerness of using things because you want to use the latest things and you're contributing to them but then it kind of creates a build time decision on our side because the WebAssembly bytecode is not backwards compatible or guaranteed to be backwards compatible, let's say. Well, maybe with... No, with SIMD it's not, so you need the feature enabled. With threads, you're missing some functions, so you could make it work, but it's not great. So we're a little bit careful of how many of those builds we issue from our pipeline because we don't want to overly complicate our pipeline. So it's a bit, it's a tricky thing. We are, it didn't help that much because we're not doing a lot of things that were CMD, for example, could help us with. Maybe if we re-architect some of our lower-level systems. But we are definitely looking into it and contributing. So yeah, we're interested in those. It's definitely a hot new space. And there's a lot of things going on. I also know that there is a concept around garbage collection in WebAssembly now. Or at least there's a work-in-progress sort of draft for this. That's kind of crazy as well and i'm not sure if there's any work being done around panic handling in web assembly which would also be kind of interesting right. As far as i know there was some work done on stack unwinding which is the missing bit that we have on around panics, there's kind of these two web assembly is funny because there's these two or more factions that work on it and they have completely different use cases. So the browser vendors, they care about completely different things than people in the IoT space care about. So no one who works on IoT-like devices that I know is excited about garbage collection because it's just extra stuff, right? It's useful as a concept and it's useful when you talk about maybe communicating with But it's just, it is a project that I think was useful for so many use cases that it now suffers a little bit because of it, because it's pulled into all of these different directions. So there's also compute-at-edge type of things that use WebAssembly, and those situations have completely different hardware and feature requirements than someone using it on an IoT device, for example, or even us on TVs. We're kind of in the middle because they're not IoT devices, but more powerful. So simpler is, in my view, better. And then there's the different use with the web browser. So right now WebAssembly is this hot kind of hotbed for solutions in all of these different fields. I'm quite curious how it will evolve. But yeah, it is definitely evolving quite fast, I would say, compared to other standards or things. All right. Finally, you're a principal engineer. And you have a lot of experience now with working with these worlds, JavaScript, Rust, C++, WebAssembly. My question is, how do you train a team to be productive in Rust without losing any momentum? What are your learnings? What would you do differently now if you started over? How do you get a team from zero to production with Rust? This is really difficult. One thing I would say, and I found it successful, is starting with a small project that's quite self-contained was critical for us. And that's for two things. First thing, you get used to the language. And you get used to even things that people don't quite think about when it comes to rewriting things in Rust. At Amazon, we have fairly good, let's say, ecosystem inside of the company. But if you're at a company that doesn't have that ecosystem to rely on, to deploy some RAS code, even deploying something small and keeping itself contained is going to pave the way for other people to be more productive in the future. And that's why I think starting for a small project is what I would definitely advocate. Another thing that I found successful is finding a place where latency or speed matters to whoever your customer is in there. So for us, for example, having smooth animations mattered. And that's how the first project started for us. Just do the low-level tree management, move these UI elements fast, and then people were on board with it because we thought, yes, of course, smooth animations, What do we have to lose? And that some companies maybe have different maths. They do. But for example, if you think about at our scale, a team of seven engineers for a year is not. Crazy investment maybe it's going to be one or two people for other companies for six months but it will pay off right if you give people time and if you have a problem that could be solved by rust i would say usually around stuff like processing lots of data or doing something around that and that's kind of an easy way to get your foot through the door how do you use it at scale is then a little bit more tricky. And I think here, for example, we could have done better. So once we did the port of the UI engine, even then, the team didn't start with 100 people. It started with maybe 15, and we grew and grew and grew. The problem is, after the first two pages, we were so successful that we reminded it, okay, now, everyone, you're using Rust because look at this, it's cooler, it's faster, it's smoother, is people use it more at home, use the app more, they watch more stuff, they don't abandon sessions. You know, as we saw boosts everywhere. But the problem is then you have so much more new people than people who can help them in cold reviews and then it becomes a little bit more difficult. So I think they're having a more gradual adoption curve would have been nicer. And that's the thing you asked me what I would do better. I would advocate for us to put the brakes a little bit on moving absolutely everything all at once. Just iteratively delivering, because it's a little bit like throughout... What I would say is this kind of things where you propose something, like let's move this thing to Rust or let's build this new thing in Rust and so on, or in any other language. It's a little bit like... Betting a little bit of your the trust that you have in your group or your peers so you gain a bit of trust you ship something you gain more trust you you bet your trust again that you have accumulated on a new project and this is how you can keep the momentum growing as a technologist in my view and yeah i mean there's also like maybe i'm survivorship biased here but if you make the right bets, it's fairly, it's usually fine. If you bet all of your trust on some mega project that then never ships, then it's going to be really difficult. So I just take more careful bets where I know I have a fairly good chance of delivering. On the training side, actually, this is a thing we're still figuring it out because people learn so differently. And I'm kind of oblivious to this because i'm the guy i don't know read the book guys what's the problem just it's a book we have a time to read it but not everyone learns the same way so i think having a sort of more a better way would be to we're trying to do now some courses and things like that a bit more applied to our ui and we have some documentation but it's more like it's it's knowledge that you get from your peers today it's not like oh i'm consulting the documentation and i'm learning. It's like some experienced guy told me in the code review this is better, so I'm doing it now and then I'll tell someone else. So this is kind of like, I don't know, knowledge that's trying to make a bit more, to spread it in a bit more organized way through some courses and things like that. Another thing I would say in Rust-specific case that really helped us, especially everyone's moving fast. We want to shift fast. Everyone, who has time to write documentation, right? One thing we did well was we heavily, heavily relied on the ability to write examples in Rust. So example.rs, right? So everything has an example. All the UI elements, we have hundreds of examples in our... People usually use VS Code or Z, I ported our launch JSON to Z. But basically, we have hundreds of examples you can pick from, run, and see how they behave. Put some breakpoints in a very self-contained place. So I think that helps for onboarding new people, having like a big amount of examples, especially with bigger projects. So you can kind of get an idea of what's happening easier. So that's what I would say. But yeah, starting small and then gaining a bit of momentum, doing a bit more and a bit more is definitely the way. Definitely structured training is really important. I guess that's also why we are here. So that's kind of nice. What do you look for in Rust engineers? You must have interviewed quite a lot. How do you know they can quickly pick up the language? What are some traits that you see in the real world where you can judge how people will, if people will be successful with that language or not? Oh, this is actually quite interesting because we don't test people for language skills on hiring. So my opinion as long as you don't some people just don't like the language and then you move teams or whatever but basically I don't think it's that special I think the whole, Rust is hard to learn thing is just something that is a meme about this language. I never quite got it because I kept trying to learn C++ for 10 years, man. You have no idea what hard is. If you're telling me Rust is hard, try C++. So in my view, I think that's a non-issue. And in general, so at Amazon, and actually all of, I think all of the big tech companies don't quite test on specific language skills, because I think it's assumed that you'll figure it out and learn and so on. And also, it's also assumed that you'll probably work with more than one language throughout your stay there. So it's not necessarily something that, but obviously, if you're willing to learn, that's quite important. because if you're resisting this type of change, you don't learn. It's going to be harder for you to adapt, right? Maybe some people just want to read JavaScript, right? So then in that case, they can move teams or whatever. It's a big company. But I would say overall, we don't do anything special for Rust training and for testing Rust skills. That's clever. Yeah, I fully understand. And traditionally, the final question is, what is your statement to the Rust community? Well, first of all, I'm really, really happy with everything. We have such a great array of crates out there to learn from. It's one of the communities, by the way, I really want to say this, that's been friendly as it grew. And I've never quite noticed this in another niche community that becomes more popular. It usually kind of degrades but the quality in interaction has been very consistent with the Rust community so I really like that one thing I wish we were focusing on more in the ecosystem is build speeds like iteration times just getting those down I think right now we more or less accepted that Rust will be slower to compile or whatever but I think there's examples out there in other languages where people have gotten great results, like Zig, for example, is pushing a lot on the build speed and the iteration time. And I think we all win if we can iterate in our codes super fast, like press whatever button you're pressing to run your code. And if that's like two seconds, it's just magical. Once it gets past a certain threshold, my productivity just, I get distracted. So I would just work a lot and investing on if I were like. If I had the power to invest in anything in the Rust community I would say this would be a really good place to invest in iteration times and build speed to decrease them. Who knows maybe at some point we will use cargo sick build for building our Rust projects because there is a sick compiler back end for Rust nowadays yeah. Especially for debug code where you don't need the whole optimizations from llvm and so on yeah. Yes i would like that Alex thanks a lot for taking the time i really enjoyed the calm and level-headed conversation and thanks for the interview. Thank you. This was lovely. Thanks so much for having me. I loved it. Rust in Production is a podcast by corrode. It is hosted by me, Matthias Endler, and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.