Matthias: Welcome to Season 5 of Rust in Production, a podcast about companies who use
Matthias: Rust to shape the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we talk to Alexandru Ene,
Matthias: Principal Engineer at Prime Video, about building modern user interfaces in Rust.
Matthias: Alex, thanks so much for taking the time. Can you introduce yourself and the company you work for?
Alex: So, hi, thanks for inviting me to do this. Super fun.
Alex: I'm Alex. I'm a principal engineer
Alex: for Prime Video London. I've been with them for about eight years.
Alex: Actually, almost nine. And before that, I used to do video games,
Alex: game engines and such things at Ubisoft.
Alex: And right now, as I said, I work with Prime Video on the Prime Video app that
Alex: you have on your living room devices.
Alex: And you do a lot of rust there. so that's why i'm yeah and.
Matthias: And when you say you did video games does that mean you also used rust for that
Matthias: or was it in some other language.
Alex: No that was c++ and c sharp
Alex: for unity but mainly c++ at
Alex: ubisoft i don't know what they're using now i left
Alex: in 2016 and i learned about rust
Alex: in 2017 18 i think 2017 so
Alex: i didn't even know rust existed actually for the most time
Alex: i ignored it thinking it's a garbage collected language but
Alex: then i saw this guy on youtube doing demo scene
Alex: type of things with rust which is this super tiny
Alex: executables that do art and nice visuals
Alex: and music so i was like oh wait a
Alex: second it doesn't have a garbage collector so how
Alex: do they do all of this magic with memory that
Alex: they keep saying so that's how i kind of got into it and use it then for some
Alex: tiny emulators in the beginning and then since then i'm still building a game
Alex: in my spare time with rust i would say that was my big rust project outside
Alex: of work and how i got into it.
Matthias: So that's interesting because you come from a c plus plus background and i think
Matthias: a lot of listeners also have the same background so what kind of convinced you
Matthias: of rust in the end was it just the fact that it didn't have a garbage collector
Matthias: or did you also learn to like the language at some point.
Alex: So it was i needed i wasn't opposed to something other than c++ i just hated
Alex: build systems in c++ they only work on windows i could only use Visual Studio.
Alex: With CMake, it was always a pain. So when I saw something with Rusty,
Alex: just cargo new, that was mind-blowing to me.
Alex: I mean, people from JavaScript and these other languages are like,
Alex: yeah, we did that for ages. What are you talking about? But for a C++ guy,
Alex: to do a cargo new or cargo run and things to
Alex: just work that's just mind-blowing at
Alex: the time to me and then i started liking bits of
Alex: the language like enums things like that
Alex: i really really like those but yeah
Alex: the build system was just magical i still
Alex: remember the first time i was watching this guy on youtube ferris
Alex: and he was building some demo scene things and he starts a new he's building
Alex: an emulator for n64 i believe and he starts a new project with cargo new and
Alex: then cargo run i was like what is happening this is just so cool so that's why i got into it.
Matthias: Shout out to ferris he's also one of the first streamers i watched when i came to rust and,
Matthias: he's just amazing i guess everyone should
Matthias: watch watch his stuff on on youtube it's still there highly underrated i would
Matthias: even say because he really built an n64 emulator from scratch not sure how far
Matthias: he got but the idea itself was so great right and yeah i wish he he would do more rust stuff again.
Alex: Yeah we make a petition to bring him back or at least on the podcast we should
Alex: have him it's this good guy yeah but.
Matthias: In c++ you have things like conan or package managers I'm not familiar about
Matthias: the state of package management in C++. Can you enlighten us?
Alex: So I never used any of those. I think they weren't that popular with game developers at the time.
Alex: So all I was using is raw big Visual Studio and then create a project and then make all my settings.
Alex: And actually, I made the game with some friends that we ported to Linux and
Alex: none of us, like we only, we use directly make files, not even CMake.
Alex: So just put your files in the make file and then in the right place and then it will work.
Alex: That's i'm not familiar with i never use
Alex: conan people keep bringing it up whenever i mention build systems but i i never
Alex: tried and i don't think they supported at the time windows that well i'm not
Alex: sure what the status was mainly i'm a windows developer i never like only recently i bought a mac i.
Matthias: Think there's something to be said about conan not being officially,
Matthias: part of the c++ how could i say standard or you know it's it's a separate project
Matthias: and i guess there's a couple good things about being separate but in reality,
Matthias: for something as central as a package manager in 2025 it's really beneficial
Matthias: to have that as part of the normal ecosystem part of the runtime that you install.
Alex: Yeah and it makes for such a nice smoother
Alex: starting experience I would say so it
Alex: is a little bit of a shame but yeah I haven't written C++ since 2018 so I'm
Alex: so happy now because all of my side projects used to be in C++ like all sorts
Alex: of game mini game engines or whatever so now I have used Rust for that and now I use Rust at work.
Matthias: As part of your work you also probably review a lot of code how has that review
Matthias: process changed ever since you moved from c++ to rust.
Alex: So that actually is really you
Alex: probably know because you've interviewed other people but
Alex: i can tell you i'm so much more relaxed on
Alex: code reviews so for example we do
Alex: still have c++ code in in my team for the thing we deploy on devices we have
Alex: a mini layer that abstracts the system there so that's written in C++ and comes
Alex: with the virtual machines and all that that the UI uses later on.
Alex: So that's in C++. And then in RAS code, it's just like paying attention only
Alex: to you know, code is doing what it's supposed to do, not this.
Alex: What are you doing with this memory?
Alex: Where is it coming from? Where is it going? Even with smart pointers or unique
Alex: pointers, it's so easy to shoot yourself in the foot as a newbie.
Alex: And that it's For reviewing code, it's definitely more relaxing.
Alex: I think it's way easier to write correct code when it.
Alex: Comes to memory access it's just trivial now the only thing i would say which helped rust,
Alex: adoption a lot is that it doesn't require years and
Alex: years of experience so for example if i were to propose c++ for the same thing
Alex: i'm using rust now for performance reasons it would be so hard it's so hard
Alex: to onboard people onto c++ so unis i did c and c++ in unis but today that's
Alex: not true anymore and very few of them,
Alex: the new grads even do C++,
Alex: very few of them so it's really hard to onboard them, it takes so much time
Alex: and with Rust it's so much easier it's just like,
Alex: so basically the team that switched to Rust in Prime Video switched from JavaScript
Alex: to Rust and it's fine I mean it's a new language, it has some onboarding but imagine,
Alex: switching people who only know JavaScript and Java to Rust to C++ it's just
Alex: gonna have It crashes everywhere, basically.
Alex: It's a given. So I would say for adoption from teams that know other languages,
Alex: which isn't C++, it's way easier to onboard new people, even new grads or experts.
Matthias: So to summarize, the difference between reviewing C++ code and Rust code is
Matthias: that in C++, you need to be really careful with where memory goes.
Matthias: Memory management is way more central in the review process,
Matthias: whereas in Rust, it's more about the business logic.
Matthias: It's more about the data flow, but not on a systems level, more on a business logic level.
Alex: Yes, you're just verifying that this code is achieving what it's supposed to
Alex: achieve, but not at the low level of what are you doing with this memory?
Alex: Is it safe to access it where you're accessing it?
Alex: Are you sharing it weirdly between threads or so on?
Matthias: My main problem with learning C++ in the past was that there was so much hidden knowledge.
Matthias: There were so many unwritten things that you needed to know about this language.
Matthias: It was a big, big language that you needed to learn.
Matthias: Whereas in Rust, a lot of things are documented really well.
Matthias: And yes there is a learning curve but after this you have pretty good documentation
Matthias: you have the compiler helping you along and the language feels smaller still.
Alex: Yeah the thing with c++ is everyone
Alex: picks their own flavor you know and then
Alex: there's all sorts of parts of the language that oh don't
Alex: use that but it's there like as a newbie you
Alex: don't know what you shouldn't touch pointers are still
Alex: there like row pointers they're everywhere they're just and it's
Alex: so easy to just access them to get around
Alex: certain things that maybe they would
Alex: catch so like a unique pointer oh let me just let me just grab the pointer out
Alex: of this unique pointer and do my thing or whatever and it's just maybe share
Alex: it accidentally to some other place and then save it in some other place and
Alex: that's how you get use after free very easy so that's the problem whereas other
Alex: languages which are newer like Rust,
Alex: they don't have this kind of everyone-needs-to-pick-a-subset,
Alex: problem.
Alex: Use the whole language, it's fine. The whole...
Alex: Features are more cohesively implemented so i think that's a big deal.
Matthias: Well one could argue that one
Matthias: main reason for c++ having all of these caveats
Matthias: is that it's language that has historically
Matthias: grown over time and also is still compatible
Matthias: with c i'm actually not sure if that is still
Matthias: true or if that's an urban legend
Matthias: by now i would like to hear your opinion on
Matthias: this and second i think when you
Matthias: build larger systems in c++ you run into
Matthias: all of these really gnarly
Matthias: data management problems because this will kind of haunt you and bite you later
Matthias: it doesn't bite you at the start usually not at compile time so you're kind
Matthias: of in a situation where later on you have a larger C++ code base,
Matthias: and then suddenly these problems pop up?
Alex: Yeah, I think one, there is definitely the aspect that it grew over time,
Alex: and even multi-threaded programming
Alex: became popular when I was in kind of uni, or just entering uni,
Alex: when like Herb Sutter and those guys were like, there's no free lunch,
Alex: they were writing these articles.
Alex: So it was a language that just couldn't have predicted how the computing evolves
Alex: and towards multi-threaded and all of those things.
Alex: So I think history definitely has a role. It's not like they implemented it
Alex: wrong. They just didn't know.
Alex: So multi-threading wasn't a thing at this scale until way later on,
Alex: the history of the language.
Alex: So there were hard-learned lessons by C++ there, whereas Rust already came with
Alex: this benefit of hindsight and,
Alex: knowledge that was kind of occurred over time so I think.
Alex: Kind of, it's already had way more knowledge.
Alex: So when they designed Rust, I assume they had way more knowledge about,
Alex: okay, what does computing look like today?
Alex: What are the common problems with other things? Let's fix those in a breaking manner.
Alex: Because it's not backwards compatible with C++ in any way, right?
Alex: You only can, they don't have that constraint.
Alex: So I think they did a really good job, the Rust designers and the people who contributed there.
Alex: C++ keeps trying to do all of these things. There's various efforts,
Alex: but honestly, I think they should probably stop adding things to the language. Just, it's fine.
Alex: We keep, you know, you keep adding, they added recently something like option
Alex: or result type of, and it's just so, it looks so ugly.
Alex: Don't understand how people can criticize Rust for looking ugly.
Alex: Like these guys who write c and then you look at c plus plus and c and c plus
Alex: there's just god what's happening it it wasn't that bad when i started honestly
Alex: c plus plus 11 okay that's fine it's just that's a good language just stop adding things to it.
Matthias: What do you think about all of these recent security efforts by people like
Matthias: bionis duestrup who try to make c plus plus safer is going anywhere?
Alex: I haven't kept up to date with those. I know there's some efforts.
Alex: Herb also, the Microsoft guy, I think Herb Sutter was there.
Alex: He also had some ideas, even with some sort of a smartphone,
Alex: but I'm not up to date with what's happening there.
Alex: And in general, about security in the fields I worked in, the thread models
Alex: were such that you didn't, there were other, performance was definitely on the first,
Alex: order of business rather than security so for video game engines for example
Alex: you're more concerned with that than security especially for a single player
Alex: game where you don't you don't really interact with the outside basically so do.
Matthias: You think that performance and security are always at odds or has rust proven that you can have both.
Alex: No you can have both and i think in good
Alex: c++ you can have both as well today from what
Alex: i can see the baseline libraries we
Alex: use are in c the open ssl and those things curl
Alex: everyone uses curl in everywhere basically and security is super important there
Alex: so i think you can get it in both languages but with various degrees of effort
Alex: they're not at odds but they for certain languages they take definitely more
Alex: effort than others now Now.
Matthias: I'm not sure if you still write any C++, but if so, or even if you just read C++ code nowadays,
Matthias: has your way of dealing with the language changed the writing style or what you consider good C++?
Alex: So I think one thing that helped me when I learned Rust is it kind of formalized
Alex: the way you're supposed to think about memory and memory access patterns.
Alex: So I don't think I was ever reviewing C++ code with the idea of,
Alex: hmm, who has access in a mutable way to this data?
Alex: I just didn't have this concept in my head. I mostly look at stuff like,
Alex: did you free it in the right time, and it's...
Alex: Maybe a vague concept of lifetimes, but not quite as formal as Rust makes it.
Alex: So it definitely gave me at least some vocabulary and more structured way of
Alex: thinking about catching those issues when I review code.
Alex: Because, I mean, in the beginning, I was just using Rust at home for my side
Alex: projects and passion stuff.
Alex: So then when I got to work, I was writing C++ and it made me think a little
Alex: bit more like, oh, what's the order of access here?
Alex: How long does this live and it gives you a bit of vocabulary so even if you
Alex: write c++ i think it makes you a good better c++ programmer yeah.
Matthias: And what have you done in rust at prime video.
Alex: So i'm just a guy who rewrites things in rust slowly recently completing almost
Alex: the the whole app rewrite so basically i started working on the so the team
Alex: i work in is the client team,
Alex: which is a little bit special because we work on
Alex: living room devices so basically with the
Alex: way prime video works we have one application that goes on
Alex: all of these devices no matter how performant they
Alex: are so it could be all of the gaming consoles all
Alex: of the like tvs setup boxes streaming sticks
Alex: they have the same application running on them
Alex: which has a tiny c++ layer
Alex: and it used to have a javascript and react layer on top
Alex: and basically my work started
Alex: in that c++ layer and then i moved on and started rewriting parts of the javascript
Alex: bit in rust and web assembly so instead of vending things over the air with
Alex: a javascript bundle now we send a wasm bundle and that people download,
Alex: and we have a WebAssembly virtual machine on the device that runs that code,
Alex: mainly for performance reasons.
Alex: That's kind of a lot of the work I do. So recently, we started with the UI low-level
Alex: bits, like, for example, just the scene management, animations, things like that.
Alex: Those became Rust four years, five years ago. I don't remember. Maybe four.
Alex: My timelines are a bit wonky.
Alex: But we started with a more baseline level. And then today we even wrote our
Alex: own UI SDK in Rust to replace the React part.
Alex: And now our UI client engineers, front-end equivalent, are writing Rust as their
Alex: day job for about a year or so.
Alex: So this application has already been deployed on a lot of devices.
Alex: But you can imagine it's quite quite a few given it runs on all of the gaming
Alex: consoles all of the setup boxes you can think of and tvs samsung lg sony everything
Alex: so yeah and i guess it also runs on.
Matthias: Some of the consoles like the playstation 4 and 5 and the xbox i guess.
Alex: Yeah gaming consoles ps4 5 um all of the xboxes xbox one xbox one s there's
Alex: a lot of them these days but yeah so a.
Matthias: Few people might even have more than one deployment of.
Alex: Your application somewhere i wouldn't be surprised because at least you have
Alex: a tv which if you bought it since 2017 or so it's probably one of these tvs
Alex: that has apps on them so it will have the prime video app because.
Matthias: They they sometimes even come pre-installed right.
Alex: Yes yes so they this was
Alex: one of the problems because as i said like the hardware
Alex: in these devices especially on the cheaper
Alex: tvs or the setup boxes
Alex: isn't super powerful even on the streaming sticks they're powered by usb right
Alex: so you can't have crazy good hardware inside because it's just whatever usb
Alex: connection can feed so basically ideally we would write our code into something
Alex: like C or whatever Rust and,
Alex: From the beginning, let's say Rust wasn't on the table, but we would do everything
Alex: in C++ and have the best performance ever because we already had it for the native layer.
Alex: The problem is on these devices, you don't have app stores.
Alex: So sometimes your app goes with the firmware update. Sometimes it's quite a manual process.
Alex: So that's why we move more towards JavaScript and React so we can keep shipping
Alex: bug fixes and updates to customers.
Alex: As a consequence performance wasn't fantastic
Alex: as you can imagine and we had to put a lot of effort into optimizing that and
Alex: even so it was very difficult so rust and web assembly helped a lot on the performance
Alex: side and it's code that you can still ship and download and has.
Alex: Way better performance than a javascript code would
Alex: have because if you imagine javascript today
Alex: you probably imagine it on a laptop or phone or
Alex: whatever where you have jit enabled but on these devices you maybe don't have
Alex: enough memory to even enable a jit virtual machine like a v8 or whatever so
Alex: non-jitted javascript is just so slow you most people don't know the pain so yeah.
Matthias: Yeah that's one thing and the other thing is i'm not even sure about garbage
Matthias: collection on these devices how efficiently it works because it might be that
Matthias: eventually you run out of memory or it's just very laggy it gets slow when you
Matthias: use it for a longer time did you ever encounter that.
Alex: Yes so we we trigger so because garbage collection is such a problem in in in javascript non-jitted,
Alex: we disable garbage collection if we have an animation playing and then we call
Alex: GC on the VM manually whenever we know it's safe to do a little bit of stalls
Alex: like your idle or whatever so that was a huge problem as well for us because.
Matthias: These devices run for a very long time and sometimes people have them on for
Matthias: hours and maybe even days at times you can't really just you know,
Matthias: close it and start it back up again. That's not an option.
Alex: Well, we do that if we...
Alex: When we... This was actually... This was a better experience,
Alex: by the way, than we're going to get there probably, but because you have exceptions,
Alex: if something unexpected happens, you can just catch it and reboot the whole app.
Alex: We lost that with transitioning to Rust, so I kind of miss a mega try-catch kind of,
Alex: situation especially with web assembly where you don't have you don't have stuck
Alex: unwinding and you can't quite do the same things.
Matthias: In raw rust you could unwind the stack that's true you could,
Matthias: technically panic or throw an error and then unwind on that right or yeah if
Matthias: you have a panic you can handle it if you wanted to but if you cross the rust
Matthias: to web assembly boundary this is where things get really hard.
Alex: Yes, in WebAssembly, it's not possible today.
Alex: So we lost some features, I would say.
Alex: But at least, it's in general very stable.
Alex: But if you do have a panic, it is a very unpleasant experience.
Alex: And actually, one thing I would say, writing panic-free code in Rust is extremely
Alex: difficult because you can't quite know when a panic would happen.
Alex: It could be in one of your dependencies. It could be anywhere. It's not like in...
Alex: You can't do anything about it. So one of the panics came...
Alex: I remember because we got paged because of this edge case somewhere that had
Alex: to do with the way third-party library worked.
Alex: When the third-party library detected that you shouldn't call certain things in a certain way,
Alex: it would panic it was like this is incorrect but then the customer has no app
Alex: anymore man what did you do so then we had to kind of like fix that and patch
Alex: it and return a result because.
Alex: The degrees of correctness people sometimes
Alex: attach to things i feel they're
Alex: a bit too much at times where we could be a bit more lenient to return a result
Alex: but don't take down the whole thing because in some deep deep library somewhere
Alex: you found some inputs that were a bit dodgy so if.
Matthias: I started such a project i'd be concerned that i could never update my deployment
Matthias: once it's on this box it will stay there forever and sometimes people don't
Matthias: do any updates on their devices so it needs to be sort of perfect or at least very close to robust,
Matthias: was that a concern going in and probably you already had the problem before
Matthias: rust so it wasn't a new problem.
Alex: Yeah we already had the problem i mean the rust code we download as a web assembly
Alex: bundle so that didn't change this doesn't sit on the device forever this is
Alex: It's just every time you start the app,
Alex: we download a new app if one was released.
Alex: On the C++ side, this is why a lot of low-level code that managed our UI scene,
Alex: like doing the animations,
Alex: creating the scene tree with the nodes, the hierarchy of nodes,
Alex: and all of that was in JavaScript initially.
Alex: Normally, browsers or other UI engines do this in C++ for speed.
Alex: But because I had that fear that, man, whatever I write here,
Alex: we write and stays here mainly forever
Alex: is not quite true because if something happens you
Alex: can ask for an update work with the partners but
Alex: it's a very expensive time consuming process
Alex: every update would
Alex: be different they multiply very fast we
Alex: solved that by minimizing the amount
Alex: of C++ code that you put on device and has to
Alex: be there but then we suffered for performance issues due to the fact that we
Alex: moved it to javascript but in the meantime we got lucky and web assembly got
Alex: invented and we could take that
Alex: technology and deploy it and then invent rust code over the air so now.
Matthias: A lot of people have some rust experience maybe
Matthias: even some production rust experience but they've never
Matthias: really written a platform because to me
Matthias: it sounds like what you're building here
Matthias: is way more foundational than what people touch on a regular basis do you have
Matthias: any tips for people who would start with such a large project or would you even
Matthias: disagree and say well it's not really a platform because we already still have
Matthias: the old c++ layer below but we just replace javascript with web assembly.
Alex: We well,
Alex: It was easy for us because we had the backup of switching to the JavaScript version, right?
Alex: So when we were doing this project, it always starts as an experimental,
Alex: let's see what could happen.
Alex: It's not exactly like that, but it starts a bit as a research project and you
Alex: have a way to go back to the old app.
Alex: I think one thing I would say about, because we replaced React with our own
Alex: UI SDK eventually, and that is a pretty big project.
Alex: I think you're right.
Alex: It is a huge... It's not foundational because we have only one client,
Alex: one app that uses us, right?
Alex: So it's not like we're React that has to service billions of projects.
Alex: But I don't think you can build these UI SDKs without fully,
Alex: deeply understanding the problems in the things you're wanting to replace, right?
Alex: So this was for us an eight-year journey overall since I've been there and probably
Alex: longer because the app existed from before me.
Alex: And the lessons that you learn are very important in the success of any future replacement.
Alex: So I would be very cautious in allowing, like, for example, if some team comes
Alex: to me and they say they want to rewrite something in Rust, the first thing I
Alex: would test them is like, how deeply do you understand the problems with your current system?
Alex: Or are you just a fan like me? And it's nice. I obviously am happy when these
Alex: things happen because I like the language.
Alex: But I think if you want to be successful here, I would say it's really,
Alex: really important to deeply, deeply understand all the pain points with the previous
Alex: thing and advantages, right?
Alex: Because it's not like we rewrote some JavaScript in Rust and it's better.
Alex: Some things are worse. Like I can't panic anymore, which is really difficult to never panic, okay?
Alex: So access an array at the wrong point, that's it, no more app.
Alex: This didn't happen in JavaScript, you know? So it's a trade-off always, I would say.
Matthias: It's so true because you need to understand where you're coming from and you
Matthias: need to understand your problem domain.
Matthias: If you don't have that knowledge, you're missing out on more than 50% of the
Matthias: advantages of porting to any new language, such as Rust.
Matthias: And a lot of people, they just think they move to greener pastures,
Matthias: whereas in reality, they just switch one problem with another.
Alex: It's really true and like we
Alex: didn't start like this right we started with
Alex: just doing the things we knew rust would do
Alex: a good job at because we already knew we did
Alex: a bad job at in javascript which is managing the tree
Alex: of ui elements so we moved that away from c++
Alex: into javascript for updatability but we
Alex: knew the trade-off there we built it and we're like
Alex: oh god this is so slow but at least we can update
Alex: it so when we made the decision let's keep
Alex: the react app but put the
Alex: management of ui tree rust and web assembly
Alex: in a different vm and these two things communicate through some
Alex: message bus then it was kind of an easy choice starting from a smaller project
Alex: and it's it has a backup you can always fall back to the thing that works and
Alex: it gives the team reading space to kind of research here and deploy it.
Alex: Without even having a billion teams collaborating.
Alex: So this initial thing, when we moved the UI tree management to Rust,
Alex: I think was done by seven people over a seven, eight month period,
Alex: but they didn't interact.
Alex: The other teams didn't even know we switched the thing underneath them.
Alex: So it was quite nice.
Alex: When we rewrote the UI, everyone knew because there's no other,
Alex: there's no way you impact a lot of people but at least you come from a more experienced,
Alex: you already deployed it in production once you know you have quite a bit of
Alex: experts in the team so it's a bit of an easier sell.
Matthias: I guess the transition was also easier because you kept the react part and then
Matthias: you had this bridge to web assembly and to rust so you didn't have to replace
Matthias: the full thing it wasn't a complete
Matthias: rewrite from the get-go it was more or less a gradual adoption yes.
Alex: Yeah we we did one thing at the time like that one screen page at the time basically yeah.
Matthias: Interesting that you went page by page sort of because the way i thought about
Matthias: it was you kind of started with animations or was that not the entry point?
Alex: Yeah, we started with the animations and those things, but the pages were still
Alex: in React. But by the time we started,
Alex: moved pages to Rust. It was one page at a time because we didn't want to have
Alex: on the same screen both JavaScript code changing the screen and Rust code changing the screen.
Alex: So the way they're separated at the lower level in the engine,
Alex: it's two different worlds.
Alex: Like in the ECS world, they're completely
Alex: separate entity component systems for those
Alex: worlds we use entity component systems there
Alex: which is what bevy and those engines use
Alex: in rust so it's a bit like a 2d engine but like two different separate worlds
Alex: for us so we didn't want to deal with one thing if you do that which is very
Alex: problematic is focus management so focusing a thing that may have been a react
Alex: ui element from a rust ui element it's just,
Alex: very brittle to keep track of all of those things so it was simpler for us to
Alex: move a whole page at a time it simplifies things which helps delivering it faster then that.
Matthias: Means that if you use the app you might have switched between javascript and
Matthias: rust multiple times throughout the interaction.
Alex: Yeah, yeah. So we started with three pages, and then we kind of ordered them
Alex: in order of use and complexity.
Alex: This was one of the things that was called out by people.
Alex: In Amazon, we review a lot of docs with directors, VPs, and so on when we do these big changes.
Alex: And a lot of people are a little bit concerned that it will look jarring to
Alex: go from the support smooth Rust pages to the old JavaScript pages.
Alex: And how do we make sure it's not like shocking.
Alex: But it was fine in the end. I mean, it was unavoidable.
Matthias: Did you start with the slowest pages or with the least used pages or what was the strategy?
Alex: Well, we started from the, when you start the app, you see the user profiles.
Alex: So first we replaced that profiles page because of a few reasons.
Alex: It allowed us to put SoundCloud in production.
Alex: It wasn't a complicated page because it's like three buttons with a bit of animations.
Alex: So we didn't require that much UI engine bits to be built.
Alex: One thing to remember is this UI engine didn't exist, right?
Alex: So it was developed at the same time with the app pages. So we picked something
Alex: that didn't have too many widgets because the UI engine team that made this
Alex: kind of React in Rust equivalent,
Alex: built like the concept of a button of focus
Alex: management and that's it and then
Alex: we started with that profiles page and once you selected it
Alex: you went you go to the main app page where
Alex: you have all your movies and we call it the collections page
Alex: so the main app page was the second one that we replaced and then from there
Alex: that was the most complex one and then from there we just replace things on
Alex: the journey to playback basically it was more in order of complexity plus use i would say do.
Matthias: You have any metrics um in terms of usage of the app did you see the number
Matthias: of panics spike or the latency go down.
Alex: We saw latency go significantly down especially input latency on the worst devices
Alex: went from stuff like 400 something milliseconds on the worst device to 30 milliseconds.
Alex: So it was a huge drop in input latency, mainly because the way React works is
Alex: It's a very complicated way.
Alex: And it does a lot of work to figure things out and to figure out like divs in
Alex: the tree and layout is also very slow.
Alex: So we simplified a lot on that critical path while allowing for certain cute
Alex: features like every property could be animated and things like that.
Alex: So simplifying those allowed us to have quite a huge gain in performance.
Alex: That plus Rust being just a, Rust and WebAssembly just being a faster option
Alex: than equivalent JavaScript code.
Alex: So it was, I would say, two optimizations.
Alex: One in the logic that we simplified, and then you get the base layer performance boost from Rust.
Matthias: And I can certainly relate to that because at least to me, when I use any of
Matthias: these applications, the input feels like some of the slowest parts of the entire application.
Matthias: It's like annoying if you type a letter, for example, and it takes ages for
Matthias: it to appear and annoys me to no end.
Matthias: So it must have felt like a free upgrade of my hardware somehow.
Alex: No, people did message me that they knew I worked. So it's like,
Alex: oh, you finally updated it on my TV. It's like, yeah.
Alex: So you could definitely tell, especially since this rollout is usually gradual
Alex: because we wanted to also monitor if they impact like business metrics,
Alex: if people use the app more, watch more video.
Alex: And those experiments are a bit more long running.
Alex: So not everyone got it at the same time. did.
Matthias: People only notice it based on speed or were there also any ui differences so
Matthias: did you tackle updating the ui at the same time or did you try to be one to
Matthias: one compatible with the old version.
Alex: We were well neither so both the old version so both javascript and rust so
Alex: the prime video app went through a redesign and the redesign happened on both
Alex: JavaScript and then the Rust bit implemented the new look only.
Alex: So they were both, both of the UIs were getting kind of a redesign and then
Alex: Rust came and just did the new thing.
Alex: We didn't try to do much more than add a few animations I think here, but very few.
Alex: So you have a bit of a, because we wanted to showcase layout animations,
Alex: those are animations that,
Alex: move an element in such a way that it requires you
Alex: to recompute the layout of the other elements in
Alex: the scene just to showcase that there it's really
Alex: fast on the new ui engine in
Alex: rust but that's just i think in one or
Alex: two places used or it was we try
Alex: to keep them as close as possible because then it kind of invalidates all the
Alex: you can't quite attribute oh why did people use the up more on rust maybe it
Alex: was because of the animation or was it because of the it's hard to find the
Alex: cause so we try to be as one-to-one with the javascript version as possible right.
Matthias: The new version had better performance and i'm assuming it also used less memory
Matthias: did that mean it allowed you to do more with the devices eventually because
Matthias: now you have more headspace Now you could experiment more.
Alex: On the memory side, we use more, actually, because we have still the JavaScript version.
Alex: In the same like we didn't
Alex: delete javascript code if we have it in Rust because
Alex: we kept it there if we needed to switch to a
Alex: fallback javascript full javascript
Alex: experience so overall with memory
Alex: that became quite a bit of um quite a
Alex: bit higher i'd say because there is now it's like two
Alex: apps there obviously as we port things
Alex: over the js part will be deleted
Alex: or minimized for things like
Alex: just having logs there or whatever it's another
Alex: kind of free thread that you can shove things
Alex: to with a message bus and then they can do some work for example right now we
Alex: send metrics and logs from javascript to our backends so from rust app we collect
Alex: them we send them to javascript but JavaScript massages them in the right JSON
Alex: format and then reports things from time to time.
Alex: So that bit probably is not as critical to delete, but yeah.
Matthias: So running JavaScript and WebAssembly VMs in separate threads is interesting.
Matthias: And how do you handle synchronization and stake consistency between those two worlds?
Matthias: You mentioned messagebusses maybe one or multiple, I don't know.
Matthias: Is that the way you generally handle memory synchronization,
Matthias: or are there some other patterns?
Alex: We don't quite fully share memory.
Alex: Well, you can't fully share memory because the way of WebAssembly memory model is.
Alex: So we just send messages between the two virtual machines, and through messages,
Alex: they synchronize their state.
Alex: So, for example, let's say you want to synchronize the history.
Alex: Stack like if you press back where
Alex: where would that go you just send a message that hey
Alex: javascript updated and so on and then it's just
Alex: message passing that's that's all it is
Alex: and similarly javascript can tell the Rust vm it's usually one of them is the
Alex: source and the other one is just synchronizing with the source so there's only
Alex: one source of truth that propagates its state to the virtual machine that just follows i.
Matthias: Guess one of the big advantages of using a message bus here and please correct
Matthias: me if i'm wrong is that you avoid interrupting either world so for example if
Matthias: the wasm thread does any work it it does not need to react immediately to a message coming in right.
Alex: No yeah and it's also way easier to implement than anything else basically so
Alex: you can always poll a message whenever you want, whenever you have time to deal with them.
Alex: Exactly. So you don't, they're not, obviously if it takes a long time to process
Alex: a message, you've blocked the UI there.
Alex: So it still has those problems, but it gives you a bit of wiggle room on how you, yeah.
Matthias: And while you did that transition, did you fix any bugs?
Matthias: Did you find any logic bugs in JavaScript?
Matthias: Any incorrect behavior that you were able to clean up then? Or even just technical
Matthias: depth as well to reduce the maintenance burden?
Alex: No, but I can tell you one thing I did. When I was porting things from JavaScript
Alex: to Rust, I found a place in the UI that was a bit dodgy.
Alex: And I did it the right way. I think it was to how some transparency was being
Alex: calculated, or I don't remember.
Alex: And I broke the app. So basically, then I became very careful into telling people,
Alex: please maintain whatever JavaScript was doing. That's the ground truth.
Alex: We're not thinking here, we're porting it over.
Alex: Let's not try to... It's really easy to fall into the trap of,
Alex: oh, let me do it the right way, but then you don't realize that other things
Alex: on top of you depend on it being wrong.
Alex: It's that whole classic meme of whatever the API is doing is what the API should do.
Alex: And yeah we got bit by
Alex: that a couple of times me specifically because
Alex: i was feeling so smug about it's like ah look at this
Alex: silly thing i fixed it and it's like hey Alex
Alex: did you notice this thing doesn't work at all no no so yeah it was with i would
Alex: say because i rewrote a couple of these systems one of my lessons learned is
Alex: to just move it with whatever behavior it has.
Alex: Try not to be smart about that and then
Alex: you can you can iterate on things
Alex: later but it already these projects
Alex: of rewriting things i feel they take longer
Alex: than people expect especially since they're usually quite optimistic like oh
Alex: what's what's so complicated about a ui tree you know it's just a tree has some
Alex: properties you update it you traverse the tree you update the children and can
Alex: do it in a week if you're very naive about it.
Alex: But then it turns out it takes a year and then every little bug in the thing
Alex: needs to be translated because there's stuff that depends on those bugs that
Alex: people put workarounds for.
Alex: And then they work differently if you change your code.
Alex: So I'll just keep things as they are when I translate them to a new language
Alex: and then iterate on them, I would say.
Alex: I would advise against trying to fix anything.
Matthias: Yeah, I fully agree with that. Because otherwise you would tackle two problems at once.
Matthias: So I know a lot of companies who did the same.
Matthias: They started to put over the old logic, including the bugs and everything,
Matthias: and then started to clean up.
Matthias: That's a way more productive way to work in that area.
Matthias: Especially for people that are not know like especially for people that are
Matthias: not comfortable or not experienced with porting code it's a skill that you need
Matthias: to learn it's it's a different skill than writing code from scratch because
Matthias: you're taking code that already existed and is running in production yeah.
Alex: And what's correct in this scenario is the behavior that is already there They're
Alex: not what you think should happen.
Alex: So what correct means changes if you're replacing a system, I'd say.
Matthias: And speaking of old behavior, you must still be able to handle old devices as
Matthias: well that don't support WebAssembly.
Matthias: Do you have two code bases or two separate code paths or some sort of fallback?
Alex: We still have the javascript pages
Alex: that are maintained in parallel i
Alex: would say they're not getting all of the new features like all of the new animations
Alex: and whatnot but because there's significant even at low percentages worldwide
Alex: those are a lot of customers that are still,
Alex: due to, well, historical reasons, they're getting only the JavaScript pages
Alex: because we didn't put Rust on the C++ layer yet.
Alex: So we still have to maintain those. And we're going through efforts of,
Alex: kind of backporting this native WebAssembly VM so we can allow them to run WebAssembly.
Alex: But for now, they're both maintained.
Matthias: Okay, so that means we are in 2025 now.
Matthias: How big is the code base nowadays and what's the team size?
Alex: So team size is around 100 or more contributors to the whole project.
Alex: We have a lot of people from backend teams that just want to surface some messaging
Alex: or something like that, and they have to write a bit of UI code.
Alex: So that's why the number is so high. And maybe on the code base size,
Alex: below 100,000 lines of code and more than 50, I would say.
Alex: I didn't check it recently. Also, it's a little bit hard to check because we
Alex: rely heavily on macros because we replaced React and we wanted to give people
Alex: this kind of JSX-like experience of compositing UI.
Alex: So that expands quite a lot.
Alex: So maybe you write five lines of a macro where you compose some UI elements,
Alex: but then the code generated by that is quite significant. And so it's hard to tell, yeah.
Matthias: How would that feel like if I were to write a component in that system?
Matthias: Would it be easy for me even if I wasn't a Rust developer? Or would it be very specific?
Matthias: You mentioned the macro and it being a bit JSX-like.
Matthias: Would I be able to understand as a JavaScript developer? Or would I need to
Matthias: learn Rust for real first?
Alex: So I think on the UI programming side, So if you're a front-end type of engineer,
Alex: you don't need to learn that much Rust to be able to write code.
Alex: The reason for this is that we are very relaxed with copying data in the UI
Alex: side, and we wrap stuff in RC pointers and things like that.
Alex: So they don't quite hit ownership issues that you are usually having to deal
Alex: with when you write new Rust code. It's not super optimal, but it doesn't matter
Alex: compared to JS is way faster anyway.
Alex: So we used to use Leptos for this framework for effects and signals.
Alex: And this is very familiar to people who know React or SolidJS and those type of things.
Alex: So in that sense, it's a bit easier.
Alex: I think in general, when you go to Rust from a higher level language, it's easier because,
Alex: I don't know what's happening in that browser underneath me when I write React code.
Alex: This code sometimes is being called once every time I click a button.
Alex: This other code is being... Because it's not like you read the code and that's how it's executed.
Alex: It's always they go into different places. So I think in our framework,
Alex: if you read the code, this is how things are happening.
Alex: We just expand it so you don't have to write a lot of things by hand.
Alex: But there's no magic, except maybe in the effects and signals,
Alex: which are still a little bit like magic because they're functions that are called
Alex: when you set a value on another thing.
Alex: How does it know to call the function? It's just a little bit,
Alex: it's not quite intuitive, but it is intuitive to people who know React because
Alex: they got used to like effects and this type of behavior.
Alex: For me, it was just so weird to have this kind of, oh, whatever,
Alex: what's happening here? And it turns out what's happening is very ugly.
Alex: It's horribly ugly, but, you know, it's the way reactive UI works, right?
Matthias: Did you have to port that stuff over to Rust or did you find a different way to solve that?
Alex: So that is already done by Leptos and there's a few reactive frameworks that
Alex: we relied on. we eventually moved a bit away from it because we wanted to optimize certain things.
Alex: And I think the open source projects is going a bit in a different direction.
Alex: So for simplicity, we had to change some stuff there.
Alex: But basically, there are libraries who did this.
Alex: And I think those type of libraries pushed what's possible with Rust UI in general
Alex: and making it more pleasant to use for people who are front-end engineers.
Matthias: Do you use a fork of Laptos now that you maintain yourself? And which libraries did you mean?
Alex: So there's a few reactive ones. I don't remember the name.
Alex: We used to use Leptos and then forked it and then wrote our own thing on top instead.
Matthias: And the libraries that you mentioned, the ones that allow you to do the signal
Matthias: or event handling in Rust?
Alex: No, this is the signal. So Leptos is basically signals, effects,
Alex: things like that. they do provide also a way to declaratively do your ui but
Alex: we didn't use that we only use the signals bit.
Matthias: Very early on in our conversation we
Matthias: talked about game development and i
Matthias: wonder how much of that design was inspired
Matthias: by game development or how can
Matthias: you compare those two worlds i understand that
Matthias: this is not a game but it feels like you have things that are similar to games
Matthias: like reacting to events some sort of ui some sort of event loop and maybe can
Matthias: you compare those two worlds and how did you model that i'd.
Alex: Say basically it's a lot like a game engine under the hood so right now bevy
Alex: popularized this but we did use we also use entity component systems for all our low-level bits.
Alex: So while we kept front-end engineers happy with leptos-like things with effects
Alex: and signals and this reactive way of writing UI,
Alex: under the hood, it's basically a game engine, more or less.
Alex: So it has entity component systems, it has systems.
Alex: Every UI element becomes an entity, and we attach components to that entity
Alex: based on what behavior it needs to have.
Alex: And then we have systems like the focus system that looks at every focusable
Alex: entity and takes the focusable component and does stuff based on your input.
Alex: So it's very, very close to game engine.
Alex: One thing we maybe didn't add and are added later, which I think makes it a
Alex: bit different than a game engine, is async stuff.
Alex: We didn't allow async at all, basically, because you had no runtime.
Alex: So you can't do anything with a future, right?
Alex: So I wish we had added stuff earlier. and that's not a concept you usually find
Alex: in video games, at least not the way you write it in Rust today with async whatever
Alex: and tokio and all of these things.
Alex: It's more like something you find on the back end because async programing is
Alex: like waiting on multiple things, whereas what a game does is processing multiple
Alex: things. So it's more like thread.
Alex: You create some threads and you do work on those threads instead of waiting.
Matthias: So that.
Alex: Was a concept that was added a bit later.
Matthias: So my understanding is that with
Matthias: entity component systems what you want to do is
Matthias: to work in parallel on the same entity just looking
Matthias: at it from different angles a certain
Matthias: facet of say elements and
Matthias: you only deal with these facets but you do deal
Matthias: with them in parallel whereas in a
Matthias: classical let's say io bound
Matthias: system you deal with things concurrently
Matthias: as they come in and so is that where those two worlds collide where you have
Matthias: events that are handled asynchronously or concurrently whereas you kind of want
Matthias: to update the ui in parallel or you want to update all of these properties in
Matthias: parallel with your entities?
Alex: So the place where they kind of interact is when you have UI elements that are
Alex: waiting on the network call.
Alex: This is not very common in video games, but it's very common in UI applications
Alex: that most of their data comes from some backend.
Alex: It is a little bit on games because you get data from disk that you process
Alex: and then it gets to show up on your screen.
Alex: But async programming is a bit foreign to game developers as understood by this
Alex: kind of idea of a promise or a future that you pull on and it's not a thing
Alex: that happens often in games, I think.
Alex: So because our UI engine is very driven with responses from the backend,
Alex: And I think we could have added async stuff a bit earlier.
Alex: I personally, so I was the tech lead on this project. And I think it was a bit
Alex: my fault because I didn't quite fully understand async because it's just,
Alex: I think async code is very useful, but it's also very hard to read code and
Alex: have a mental model of how it gets executed.
Alex: And for this reason, I decided to, okay, let's keep it a bit simple,
Alex: not have any async, and then give you some callbacks.
Alex: But then it turns out actually some things would be simpler with async if you don't overdo it.
Alex: So we added it eventually recently. So now you can declare a UI component that
Alex: shows a spinner or something,
Alex: and it will automatically be replaced by something else when we download some
Alex: data. And that's done with async now.
Alex: We call it deferred components, I think they're called in the wild.
Alex: But for those scenarios, I think that's where it differs from a game engine.
Alex: But other than that all of the graphics all of the traversal of things all of
Alex: the entity component systems that we have are very similar to a game engine yeah.
Matthias: I could also see two approaches
Matthias: to that problem one would be the simple way
Matthias: to say you have a proxy around your component
Matthias: and the component itself is completely sync whereas
Matthias: the data proxy that is around it kind of is async and then passes in the data
Matthias: that it receives right that is one way and the other way would be to say let's
Matthias: completely embrace async and do everything async even in the components and
Matthias: it feels like you went from one to the other at some point or.
Alex: You changed your mind on this we didn't so
Alex: we went with the first one that you described so basically
Alex: you kind of wrap it into something and then
Alex: because it's a async block
Alex: you can write your code and await on
Alex: network calls a bit nicer create the
Alex: ui component but then the ui itself will be just
Alex: it replaces something in the tree and that's just normal code once it ends executing
Alex: that future this is the difficulty with async code is very kind of viral it
Alex: just spreads everywhere and then you need to have this sync points with non-async code or,
Alex: You either go all in and it's async everywhere, but we didn't do that.
Alex: We just allow a sort of a wrapper. It goes into a task that executes it.
Alex: And at the end of the day, the UI code you're going to get is going to be normal
Alex: code. It goes out of that future at some point.
Alex: Usually once we're done with the processing of a network call or some message
Alex: pass that waits on JavaScript or something. yeah.
Matthias: In this case it sounds like it was easier to keep the lower level components
Matthias: the leaf nodes sync and have an async runtime around it.
Alex: Yeah yeah because ultimately in ui a lot of patterns are like set timeout and
Alex: those type of tasks can easily be async tasks as well that you then run from time to time.
Alex: And that's another place where we... You just, as a UI engineer,
Alex: you have this way of creating a function and shoving it in our executor.
Alex: And then whatever comes out of it might be attached to the UI tree later.
Alex: But it's kind of like an extra feature that you get rather than all the UI code is async everywhere.
Matthias: Do you port a lot of promises from JavaScript to Rust or was it not the case?
Alex: There were some around,
Alex: things that deal with network usually are like that with fetch and then do something
Alex: and then you wait on those.
Alex: We have just a way right
Alex: now that you can say you write an async function you do your way however you
Alex: want and then you take that future and send it to us with our api which is a
Alex: task and this is the task and then we pull it whenever we have time usually per frame but yeah yeah.
Matthias: That means you don't usually use the vanilla futures you have async functions
Matthias: for the most part and then you,
Matthias: pull them from the outside but you don't build up your own future as you would
Matthias: with let's say a vanilla JavaScript promise.
Alex: No, no, no. Yeah, it's an async function that returns the, yeah.
Matthias: And will you be able to use some of the newer features of CPUs nowadays with SIMD, for example?
Alex: So we do work a lot with the WebAssembly. So we contribute to the WebAssembly VM. This is on GitHub.
Alex: It's a WebAssembly micro runtime.
Alex: So we were quite involved there with the community and also in the WebAssembly
Alex: community working group.
Alex: So we worked there on
Alex: adding threads to web assembly simd is
Alex: another thing that we are quite keen on that was
Alex: already done so with these
Alex: things it's a little we're suffering from our
Alex: own eagerness of using things because you want
Alex: to use the latest things and you're contributing to them but then it kind of
Alex: creates a build time decision on our side because the WebAssembly bytecode is
Alex: not backwards compatible or guaranteed to be backwards compatible, let's say.
Alex: Well, maybe with... No, with SIMD it's not, so you need the feature enabled.
Alex: With threads, you're missing some functions, so you could make it work, but it's not great.
Alex: So we're a little bit careful of how many of those builds we issue from our
Alex: pipeline because we don't want to overly complicate our pipeline.
Alex: So it's a bit, it's a tricky thing.
Alex: We are, it didn't help that much because we're not doing a lot of things that
Alex: were CMD, for example, could help us with.
Alex: Maybe if we re-architect some of our lower-level systems.
Alex: But we are definitely looking into it and contributing. So yeah,
Alex: we're interested in those.
Matthias: It's definitely a hot new space.
Matthias: And there's a lot of things going on. I also know that there is a concept around
Matthias: garbage collection in WebAssembly now.
Matthias: Or at least there's a work-in-progress sort of draft for this.
Matthias: That's kind of crazy as well and i'm not sure if there's any work being done
Matthias: around panic handling in web assembly which would also be kind of interesting right.
Alex: As far as i know there was some work done on stack unwinding which is the missing
Alex: bit that we have on around panics,
Alex: there's kind of these two web assembly is funny because there's these two or
Alex: more factions that work on it and they have completely different use cases.
Alex: So the browser vendors, they care about completely different things than people
Alex: in the IoT space care about.
Alex: So no one who works on IoT-like devices that I know is excited about garbage
Alex: collection because it's just extra stuff, right?
Alex: It's useful as a concept and it's useful when you talk about maybe communicating with But it's just,
Alex: it is a project that I think was useful for so many use cases that it now suffers
Alex: a little bit because of it, because it's pulled into all of these different directions.
Alex: So there's also compute-at-edge type of things that use WebAssembly,
Alex: and those situations have completely different hardware and feature requirements
Alex: than someone using it on an IoT device, for example, or even us on TVs.
Alex: We're kind of in the middle because they're not IoT devices, but more powerful.
Alex: So simpler is, in my view, better. And then there's the different use with the web browser.
Alex: So right now WebAssembly is this hot kind of hotbed for solutions in all of
Alex: these different fields. I'm quite curious how it will evolve.
Alex: But yeah, it is definitely evolving quite fast, I would say,
Alex: compared to other standards or things.
Matthias: All right. Finally, you're a principal engineer. And you have a lot of experience
Matthias: now with working with these worlds, JavaScript, Rust, C++, WebAssembly.
Matthias: My question is, how do you train a team to be productive in Rust without losing any momentum?
Matthias: What are your learnings? What would you do differently now if you started over?
Matthias: How do you get a team from zero to production with Rust?
Alex: This is really difficult. One thing I would say, and I found it successful,
Alex: is starting with a small project that's quite self-contained was critical for
Alex: us. And that's for two things.
Alex: First thing, you get used to the language. And you get used to even things that
Alex: people don't quite think about when it comes to rewriting things in Rust.
Alex: At Amazon, we have fairly good, let's say, ecosystem inside of the company.
Alex: But if you're at a company that doesn't have that ecosystem to rely on,
Alex: to deploy some RAS code, even deploying something small and keeping itself contained
Alex: is going to pave the way for other people to be more productive in the future.
Alex: And that's why I think starting for a small project is what I would definitely advocate.
Alex: Another thing that I found successful is finding a place where latency or speed
Alex: matters to whoever your customer is in there.
Alex: So for us, for example, having smooth animations mattered.
Alex: And that's how the first project started for us. Just do the low-level tree
Alex: management, move these UI elements fast, and then people were on board with
Alex: it because we thought, yes, of course, smooth animations, What do we have to lose?
Alex: And that some companies maybe have different maths.
Alex: They do. But for example, if
Alex: you think about at our scale, a team of seven engineers for a year is not.
Alex: Crazy investment maybe it's
Alex: going to be one or two people for other companies for six
Alex: months but it will pay off right if you
Alex: give people time and if you have a problem that could
Alex: be solved by rust i would say usually around stuff like processing lots of data
Alex: or doing something around that and that's kind of an easy way to get your foot
Alex: through the door how do you use it at scale is then a little bit more tricky.
Alex: And I think here, for example, we could have done better.
Alex: So once we did the port of the UI engine, even then, the team didn't start with 100 people.
Alex: It started with maybe 15, and we grew and grew and grew.
Alex: The problem is, after the first two pages, we were so successful that we reminded it, okay, now,
Alex: everyone, you're using Rust because look at this, it's cooler,
Alex: it's faster, it's smoother, is people use it more at home, use the app more,
Alex: they watch more stuff, they don't abandon sessions.
Alex: You know, as we saw boosts everywhere.
Alex: But the problem is then you have so much more new people than people who can
Alex: help them in cold reviews and then it becomes a little bit more difficult.
Alex: So I think they're having a more gradual adoption curve would have been nicer.
Alex: And that's the thing you asked me what I would do better. I would advocate for
Alex: us to put the brakes a little bit on moving absolutely everything all at once.
Alex: Just iteratively delivering, because it's a little bit like throughout...
Alex: What I would say is this kind of things where you propose something,
Alex: like let's move this thing to Rust or let's build this new thing in Rust and
Alex: so on, or in any other language. It's a little bit like...
Alex: Betting a little bit of your the trust that
Alex: you have in your group or your peers so
Alex: you gain a bit of trust you ship something you gain
Alex: more trust you you bet your trust again that you have accumulated on a new project
Alex: and this is how you can keep the momentum growing as a technologist in my view
Alex: and yeah i mean there's also like maybe i'm survivorship biased here but if
Alex: you make the right bets, it's fairly, it's usually fine.
Alex: If you bet all of your trust on some mega project that then never ships,
Alex: then it's going to be really difficult.
Alex: So I just take more careful bets where I know I have a fairly good chance of delivering.
Alex: On the training side, actually, this is a thing we're still figuring it out
Alex: because people learn so differently.
Alex: And I'm kind of oblivious to this because i'm the
Alex: guy i don't know read the book guys what's the problem just
Alex: it's a book we have a time to read it but not
Alex: everyone learns the same way so i think having a sort
Alex: of more a better way would be to we're trying to do now some courses and things
Alex: like that a bit more applied to our ui and we have some documentation but it's
Alex: more like it's it's knowledge that you get from your peers today it's not like
Alex: oh i'm consulting the documentation and i'm learning.
Alex: It's like some experienced guy told me in the code review this is better,
Alex: so I'm doing it now and then I'll tell someone else.
Alex: So this is kind of like, I don't know, knowledge that's trying to make a bit more,
Alex: to spread it in a bit more organized way through some courses and things like that.
Alex: Another thing I would say in Rust-specific case that really helped us,
Alex: especially everyone's moving fast. We want to shift fast.
Alex: Everyone, who has time to write documentation, right?
Alex: One thing we did well was we heavily, heavily relied on the ability to write examples in Rust.
Alex: So example.rs, right?
Alex: So everything has an example. All the UI elements, we have hundreds of examples in our...
Alex: People usually use VS Code or Z, I ported our launch JSON to Z.
Alex: But basically, we have hundreds of examples you can pick from,
Alex: run, and see how they behave.
Alex: Put some breakpoints in a very self-contained place. So I think that helps for
Alex: onboarding new people, having like a big amount of examples,
Alex: especially with bigger projects.
Alex: So you can kind of get an idea of what's happening easier.
Alex: So that's what I would say. But yeah, starting small and then gaining a bit
Alex: of momentum, doing a bit more and a bit more is definitely the way.
Matthias: Definitely structured training is really important. I guess that's also why
Matthias: we are here. So that's kind of nice.
Matthias: What do you look for in Rust engineers? You must have interviewed quite a lot.
Matthias: How do you know they can quickly pick up the language?
Matthias: What are some traits that you see in the real world where you can judge how
Matthias: people will, if people will be successful with that language or not?
Alex: Oh, this is actually quite interesting because we don't test people for language skills on hiring.
Alex: So my opinion as long as you don't some people just don't like the language
Alex: and then you move teams or whatever but basically I don't think it's that special I think the whole,
Alex: Rust is hard to learn thing is just something that is a meme about this language.
Alex: I never quite got it because I kept trying to learn C++ for 10 years, man.
Alex: You have no idea what hard is. If you're telling me Rust is hard, try C++.
Alex: So in my view, I think that's a non-issue.
Alex: And in general, so at Amazon, and actually all of, I think all of the big tech
Alex: companies don't quite test on specific language skills, because I think it's
Alex: assumed that you'll figure it out and learn and so on.
Alex: And also, it's also assumed that you'll probably work with more than one language
Alex: throughout your stay there.
Alex: So it's not necessarily something that, but obviously, if you're willing to
Alex: learn, that's quite important. because if you're resisting this type of change, you don't learn.
Alex: It's going to be harder for you to adapt, right?
Alex: Maybe some people just want to read JavaScript, right? So then in that case,
Alex: they can move teams or whatever.
Alex: It's a big company. But I would say overall, we don't do anything special for
Alex: Rust training and for testing Rust skills.
Matthias: That's clever. Yeah, I fully understand. And traditionally, the final question
Matthias: is, what is your statement to the Rust community?
Alex: Well, first of all, I'm really, really happy with everything.
Alex: We have such a great array of crates out there to learn from.
Alex: It's one of the communities, by the way, I really want to say this,
Alex: that's been friendly as it grew.
Alex: And I've never quite noticed this in another niche community that becomes more popular.
Alex: It usually kind of degrades but
Alex: the quality in interaction has been
Alex: very consistent with the Rust community so I really
Alex: like that one thing I wish we were focusing on
Alex: more in the ecosystem is build speeds
Alex: like iteration times just getting those down I think right now we more or less
Alex: accepted that Rust will be slower to compile or whatever but I think there's
Alex: examples out there in other languages where people have gotten great results,
Alex: like Zig, for example, is pushing a lot on the build speed and the iteration time.
Alex: And I think we all win if we can iterate in our codes super fast,
Alex: like press whatever button you're pressing to run your code.
Alex: And if that's like two seconds, it's just magical. Once it gets past a certain
Alex: threshold, my productivity just, I get distracted.
Alex: So I would just work a lot and investing on if I were like.
Alex: If I had the power to invest in anything in the Rust community I would say this
Alex: would be a really good place to invest in iteration times and build speed to decrease them.
Matthias: Who knows maybe at some point we will use cargo sick build for building our
Matthias: Rust projects because there is a sick compiler back end for Rust nowadays yeah.
Alex: Especially for debug code where you don't need the whole optimizations from llvm and so on yeah.
Matthias: Yes i would like that Alex thanks a lot for taking the time i really enjoyed
Matthias: the calm and level-headed conversation and thanks for the interview.
Alex: Thank you. This was lovely. Thanks so much for having me. I loved it.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.