This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from corrode, and today we're talking to Jess Frazelle from Zoo about creating modern hardware CAD tools with Rust. Jess, so glad you could make it. Can you introduce yourself to the audience? Yeah, I'm super excited to be here. Jessie Frazelle i a programmer but now i am ceo of a startup called Zoo and we build mechanical CAD tools and hardware automation stuff. Right but what did you do before because a lot of people know you from the community they probably know you for a very long time you don't have to sell yourself short here i know that you have a very impressive background. Yeah so i guess i started out my career mostly working on infrastructure tooling even before that though like i worked at kind of like agencies where they'd build like your website for you and then we were building all these websites and we were serving them over like an ftp server and i got really interested in like docker i worked at a startup that was doing well something that would be insanely popular now but it wasn't then it was like web hosting for data scientists so we take like your r in python and run a docker container this is like 2013 so like sadly it was like not the right time It was way too early. But now there's like 17 startups doing this. But anyways, so we used LXC and then we moved over to Docker. Then I got really into Docker, just like running everything in containers. So I got a job there on the core team. And that was really fun. Working on an open source project was like eye opening because it's crazy. And then I worked on Kubernetes. And then I started a company with two other co-founders called Oxide Computer Company. And that was like. We built rack scale servers and that was my first hardware project and so we were using solidworks it took like over an hour to load our cad file it was just super big and massive and then sometimes it would just air for random reasons like random files were missing or something and then and then like there wasn't a way to just automate things such that like if our industrial designers were changing the side of the rack to make it more aesthetic it would take like weeks to run a simulation to know that like airflow was messed up so it's not like you're like pushing to ci like a like a standard software development workflow you would push the ci, ci would fail and then you'd go fix your design that's how it should work that is like so far from how it works today and so i started like asking other people what they were doing and one of our investors in oxide at the time introduced me to jordan and jordan who is now one of my co-founders he was the co-founder though of relativity space at the time and so i was like they were 3d printing rockets and i was like surely they're doing something better for cad and he was like no we're not we're using the same shape tools you are and then after ranting about it for so long we decided to start this company to fix the problem is. This how Zoo.dev got started. Yes yes totally and then and when we first started we were mostly like trying to create a cad engine from the ground up, which we're still doing, obviously, but no one's built a CAD engine in like 30 years. So if you're using like SolidWorks, you're using their 30-year-old CAD engine. If you're using Onshape, you're using Parasolid. Almost everything else that is like a new CAD tool uses Parasolid, which is a 30-year-old CAD engine. So none of them use a GPU. None of them are built modernly. I think it was either Parasolid or the SolidWorks engine that we talked to one of the engineers and like this was this predated c like c and so it's like fortran but they called it c plus and like just the whole thing is it's just an archaic code base and it seems like it's very hard to change things to make things better so yeah and now like we have a cad app built on top of our engine and that's pretty cool it actually uses like code cad and in a modern way such that like you can actually click and point to create drawings like you would in a traditional workflow, but it generates code on the side, or you could just raw code it if you want. And does that have something to do with the name zoo.dev? Because it's a very peculiar name. What's the story behind it? So when we originally started the company, we called it KittyCAD because it's a cool pun. And my co-founder Jenna's grandma came up with it, which is pretty funny. But then we came out with Text2CAD, which is our machine learning model for like taking a prompt and then generating a CAD model from it. And we were like, we should have more animals. so we came up with MLephant another pun and so then we were like oh it's a whole zoo and then we rebranded the zoo and surprisingly zoo.dev was not taken also it was super cheap so that was cool. It's a brilliant name too yeah so that's the big idea behind it you have something new you have something shiny but also it's a different way of talking about cad because before as you said a lot of people do point and click sort of cad and i can still remember from my time at university we certainly had to do that and was a very numbing experience i guess my numbing but at the same time what you do now is you bring a lot of things that we take for granted in software development like version control or being able to talk about models as code to, the present day. Yeah it's it's honestly really cool and i i think we're the only like graphical user interface where one side updates the other and it like you click in point and it generates code and then the code updates the model and like i don't think like you couldn't make a ui like that for react like there's lots of aspects about this language that are literally built for like code mods on the fly and so i think that aspect of is really cool it's also really hard and i i feel like we've gone through a few different architectures for the app just because like there were so many unknown unknowns when we started and i think now finally we've like realized okay so this is how we're going to do the code mods this is how like things are going to update and stuff like that and the ast and yeah i also like it it's it's also cool from the aspect that that like software engineers just get it and then also like executives that were previously software engineers really get it so they're like oh finally we can like rev our model in Git with our firmware, which is a little crazy to think about. There are some weird things where it's like, why would you want that? Obviously, you can update the firmware along with the model, but it's nice to have it. It's something that if you were doing something like this, you were probably paying hundreds of thousands of dollars a year for PLM software to do that. When it's just like, if you just use Git, it just works out of the box it's crazy and then there's stuff like OpenSCAD that already existed but but open scad was never really made for enterprises it's like we actually our main front-end engineer is the person that made the fillet plugin for OpenSCAD which just goes to show like everything has fillets but there is no like first class primitive to make fillets in OpenSCAD and for anyone who's listening that doesn't know what a fillet is if you like look at the edge of your desk like the reason why you don't like cut your finger on it is because it has this little rounded corner and like, it's just a fillet. And like, now that you know what they are, you'll see them everywhere. Yeah, the fact that OpenSCAD doesn't make that easy means like obviously no real mechanical engineer is going to use it. It's also entirely like mesh versus B-rep, which just means when you go to then machine it, like the surfaces that are there, they're just triangles. It's not actually a defined surface. So it's a lot harder to machine and make super precise and watertight. So, yeah. Anyways, there's so many wins here. And then we've made our own CAD kernel, which people think we're crazy for. But it's really cool to actually be able to use a GPU for some of the computations. And yeah, there are some like computations where CPU is actually better. And so like, we're not like insane. We don't like, if it's faster on a CPU, we'll use it. But like a lot of things are faster on a GPU. So yeah, sorry. That was a long rant. I can tell that you obviously put a lot of research into that product, into that idea before you even started working on it. Can you tell me a little bit about the research process that you went through when you tried to survey all the existing CAD programs and maybe their pros and cons. Yeah so I mean I didn't really like like we wanted this to exist we didn't necessarily want to have to be the ones to build it so I first like looked at a lot of the open source CAD tools and honestly just none more ready for the enterprise for a lot of the same reasons I listed open SCAD for and I, And then I was trying to find like a startup that was doing something similar where it was just like really pluggable. You could automate anything. And just everyone was more focused on most of the startups in the space are like, OK, we'll be the GitHub, but for CAD. And so they'll take your like SolidWorks file and then you can comment on it and stuff like that. But it's just like that's not like we have Git. Git already exists. Let's use Git. So, like, we wanted to fix it from first principles and, like, really make it, like, a clean new start because, like, in our minds, there's no way in 10 years from now we're still using these 30-year-old kernels and, like, the bottleneck should not be, like, opening your file at the start of the day. So, yeah, I mean, we really wanted to just fix it from first principles, but we researched like every single CAD program that there was and tried to find any startup that was doing something similar and there just sadly wasn't. But along the way, we met some really cool people. So that was cool. But yeah, I mean, I would recommend anyone else starting a company to like just really scope out the space first. And you even wrote a paper on it, right? Yeah, yeah. For ACM Queue, yeah. Can you talk a little bit about that paper what was the title what was it about. Oh yeah so it was called a new era for mechanical cad to be honest like a few things have changed in it because one of our engineers pointed this out the other day and that like i was like never use electron it should be all native apps and we just switched to electron which was the whole thing like actually we were on Tauri which is the rust gooey app and i swear we were using like every single weird tauri feature there was whether it was like you know the ability to like double click a file to open it or like from your finder and then just like random features that like i don't think anybody else was using and then we actually had to upgrade to the beta way too early because they wrap webkit and then webkit didn't have the the CebRTC stuff that we needed so we like paid this company who was amazing by the way if you ever need like WebRTC stuff or stuff like that igalia is great uh they have like the coolest nerds on the planet but they ended up fixing it upstream in WebKit. Then we had to be on the beta to use it. And it was kind of just a shit show. Sorry, I could go on all day about Tauri. I feel like we were just using too many weird things and had too weird of an app. So Electron is way more production ready for us. But it's just funny because I was like, in that paper, I was like, never use Electron, I'll use native apps. And here we are on Electron. And actually, you can make Electron fast, and we have, without really trying. But we're going to try to make it even faster. But yeah, sorry, that was a side note. But yeah, most of what I went over in the paper was just like all the apps that we tried and like the dream of the future, which is like, you know, this GUI that you can... Write, write, like code in, it could be pushed to Git and you'd have a model. And at the same time, as a developer, I already think of code reusability where I could maybe model out a part and then parameterize it and then maybe call a function that would generate, various kinds of things that I modeled once and give me variants from that. The other thing that I was wondering about was, so you mentioned you moved away from Tauri. Are you set on this? Or could you imagine ever moving back to Tauri if things change or if it's stable enough for your use case? It kind of depends. So the one architecture decision that I really take issue with, which we've had calls with the Tauri team, they're great. I don't want to diss them. But they ended up, I kind of assumed that the file permission stuff in Tauri was native to each OS. You would think that they would have abstracted away and then say, like, you don't have permission for a file, it should prompt a native dialogue to get permission. Like, that's how you would assume it works. They did not do it that way. They actually rolled their own permissions. So you can never prompt that native dialogue. And that was just a non-starter for us. That was like issue number one. Issue number two, which I think is maybe more fixable, is Electron, you get playwright testing out of the box. It's in beta, but it works crazy well. And we had no way to test, aside from Windows, the app on each OS, which is also a non-starter for us. Because if we don't have tests, then I don't think things are working. And as most of our development team works on Linux or macOS, we really needed to be able to test on those platforms. Mostly macOS was the one that had a lot of issues but linux was where we hit issues with the web rtc stuff and so we were never actually able to run the app on linux until we fixed upstream webkit and so that was the whole thing too but yeah it was mostly that i. See one of the things that i personally like about rust is being explicit about certain things do you prefer explicitness over implicitness. Yeah. No, I love... This is why I was so sad to give up Rust and have to write things on JavaScript. I hate JavaScript. And I actually was going to use... I think it's called Nappy, like the Rust, it converts from, it embeds Rust in JavaScript. And I hooked it all up for our app just to use it for one function because I refused to rewrite it in JavaScript. And I was like, oh, this is hell for the whole like front end team that will have to like maintain this from here on out. So I got rid of it. But yeah, no, I was super sad to get rid of Rust because I love the explicitness is like literally the win. Like you want things to like air hard and like do the right like have like no like magic hands like i feel like things in javascript are just like oh yeah magic's happening behind the scenes but like rust is like very like it does the thing or it fails yeah do. You know leptos and dioxus those are two other different rust ui frameworks. Oh no i don't there was another one that we looked into that i forgot the name of but i don't think it was any of those but yeah at that point we were just like electron is used by so many people and i know there are shitty electron apps the thing is like vs code is very fast and we read all the performance docs from vs code so our app right now is fast but we need to make sure it doesn't like get slower so we have ideas for testing to make sure it doesn't get slower and stuff like that but yeah there's ways to do it good basically so yeah okay. So that's the front end side that's the ui the user facing side but the core of any cad program is the cad kernel why did you decide to build your own rather than using an existing one. Yeah so like i was saying before like a lot of the the cad kernels are they're so old that like it predates like c++ it predates any normal language and they're all slow and a lot of the pain points at Oxide was just how slow the apps were. For any complex project, it's slow. And we hear this all the time from when we're on sales calls, showing. MEs like our project like the the mechanical engineers that have the most opinions on like oh i'll never write code blah blah blah like they like hate on all those aspects they're like oh absolutely write your own kernel like that is the one thing no one can fight against because they all know how slow it is so even though like everyone knows it's hard and everything no one at least on the mechanical engineer side is against it and these people most of them are like oh you can pry SolidWorks from my cold dead hands. They absolutely love the kernel aspect of what we're doing. So that was like, it was a huge undertaking for us though and we're still like, we're almost done. There's just a few things missing but we're hoping to launch this year. But yeah, it's been a lot of work. Our CAD kernel is also sadly not written in Rust, it's written in C++ and that was because Vulkan and a lot of the CUDA libraries like all of the good graphics libraries and like NURBS libraries are in C++ and you could generate bindings for them, but it will never be the same. I've used the CUDA Rust bindings and like it leaves a lot to be desired. I actually think hopefully there will be better ones soon. But yeah, it's just like they're just not there yet. Maybe in the future we'll decide to use them, but a lot of our graphics team like knows C++ and that's their native language and so they're a lot more comfortable with it but every other part of our infrastructure is in Rust and we just use bindings to talk to the C++ layer. Could you gradually migrate that C++ code to Rust if you wanted to? Or does it even make any sense to do that? Or would you rather say, let's switch out the C++ code base completely with an equivalent once it's ready? Yeah, it just depends at that time when we decide. There also has to be a lot of wins with it. And today there just wouldn't be that many because we'd probably be fleshing out a lot of these libraries ourselves. But once there are some wins, then we'll have to weigh the pros and cons a bit, like a traditional engineering decision. I just feel like it's a very daunting task to even start such an endeavor where you don't even know what you're getting yourself into, I guess. I don't know. To me, it feels like, for example, when you start with thinking about a CAD kernel or even wrapping it somehow, where would I even start? How does that process go? can you walk me through that thought process. Yeah so i mean one i one i think like the naiveness helps plays a part because if the people who knew how hard this was when we were talking to them originally were like absolutely don't do that like that's way too hard like just don't do it but like that made me want to do it more just because like it was like okay well how hard could it be it turns out it is very hard but it's not like as hard as they made it out to be because they were like, oh, that will take you 10 years or whatever. So yeah, where we started was on file conversions because that's the input output of any CAD kernel. So it just seems like a natural starting point. And then we got a really, really good graphics engineer who he built like the original, basically Blender, but for iOS, which isn't like a CAD app, but like it is still like a modeling app and he's worked like it and topology which is like they do implicit modeling and then he also knew this person who he introduced us to who became a consultant who's been in the industry for so long like he's seen every single cad application that there is and he was one of the only kind of people who came from the old cad world that like agreed with what we were doing he was like absolutely use a gpu i have ideas for how you could do this and like he he like saw how it could be done and like one of his things we've gotten patented just for defensibility but yeah like having people like that on your side helps a lot because like i personally would not know like where even to start when it comes with a framework like vulcan it's just not something that i've been like introduced to but like when it comes to kubernetes like i know how to use that so yeah. Knowing you, you were probably very hands-on during that early phase of the development. Which parts did you start? Which parts did you sketch out? Where did you bring in your experience? Yeah, so I helped a lot at the whole C++ to Rust bindings layer, which now I kind of hate what I made there. But we use it in anger sometimes. But most of the time, we don't have to change much with it. But it does work. Yeah, I kind of tried to help out where I could on our early infrastructure, which has now been like like our early infrastructure used nomad hashi corp stuff now we use kubernetes, so like eventually all this stuff that like you built gets like you know redone in a different way but yeah i i help out like a lot on the like now uh we our programming language for mechanical engineers is called kcl kitty cad language and i help out a lot like they're pulling features through to the engine like if they say that they have loft available i'll pull it through and add it and yeah like generating docs automatically from react macros just random stuff whatever i'm capable of i help with. That's pretty amazing what's not to like about the c++ to rust bindings. I would say like just it doesn't come natural to me but i've like learned a little bit and i can like swing a racket if i have to but it's not great but but i don't have any i guess like on the c++ side it's totally fine on the on the rust side it's totally fine it's just we use the thing called CXX Bridge, which there is absolutely nothing wrong with it. It's great for what you need, but it's actually overkill for us. What we should have done is like just raw bindings, And so we could go back and do that. We're kind of waiting until we get like past v1 so that there's a very, very clear API. And then we know like what we have to export, what we don't. But from there, it should be like super easy to just get rid of this. Like it's just kind of heavier than what we needed. There's a lot of like this interop between types that we don't need. And yeah, it's... So if I understand you correctly, if you had to start again with that part, you would write it by hand? Yeah, I mean, it's also like we're moving so fast that maybe it kind of helps how it is today. And then once we do call it v1, we'll do it that way. I don't know. I mean, I go back and forth on it. It works today. It just feels a little bloated. Oh, that's fair. But it was pragmatic too. Tell me a little bit about the file formats that KittyCAD supports. Do you have to write some of the file formats yourself? So we support STEP, which is like a kind of standard of manufacturing. If you were to send your manufacturer something, you'd probably send them a STEP file. And that's all B-rep based. And then we support a lot of the meshes like OBJ, STL, something that you throw at a 3D printer, PLY, GLTF. And we actually extended the GLTF format, which is traditionally just a mesh format to include B-rep data. And the reason we did that is like we really liked GLTF, it's JSON based, you can parse it really quickly. And all the older file formats like step, it's pretty hard to write a parser for. Whereas JSON so much easier to write a parser for once you know the like, kind of structure of it. And so we just kind of feel like that's a more modern format. You don't necessarily have to adopt like our GLTF version. We're just kind of giving that back to the like community, but we use it as our interop layer. So like if you get something out of our engine, we personally pull out this like GLTF layer with the B-rep data, and then we can convert that to any other format. So it's just the interop layer that we use. But we also think like in the future, if you could just send your manufacturer JSON, I feel like that would work a lot better. And there's so much that you can add there. Like step files lose a lot of data when it comes to like, oh, I pulled this part off of McMaster Car. It's this part number. Here's the SKU. And then like say you wanted to just order that part. You can't really like it doesn't have any of that data. It doesn't have the part number. It doesn't have the SKU. But you could put that into GLTF. Then you could create your bill of materials it could double as a bill of materials basically which again you're most likely paying some vendor hundreds of thousands of dollars for you to do that for you which is stupid like like there's just a bunch of like stupid crap around this ecosystem because things have been so hard like it's been so hard to communicate with the underlying geometry yeah. That means step doesn't have any way to add metadata. It does have metadata and like annotations but nobody uses it correctly oh. Wow and did you have to write a parser for step yourself or could you use an existing one from the rust ecosystem. There is a rust step parser it uses it uses all drives so it like takes a really really long time to compile like whenever we add to a project it just takes a long time to compile so we might write our own sorry to the maintainer of rust step i think that's what's called but like what they did was really cool i mean it's crazy that they took this like kind of weird text-based format that like isn't xml isn't like anything normal and they made a bunch of derives for it like that is honestly like pretty admirable it's just the whole derive thing takes way too long to come back because it's just generating so much code. Yeah so if i understand you correctly maybe at some point because of the compile times you have to step away from the implementation if you. Allow me that pun yeah i couldn't resist. But what about compile times in general do you have any huge problems with that with the code base right now does it feel like it's in the right spot could it be faster or is it slow. I think it's pretty good, although our C++ engineers would be like, I can't believe you just said that. Because they compile their C++ code with CMake, and it takes way less time. I think there's something with the cache in Rust where it's like popping the cache somewhere it shouldn't. And we've thought about looking into it, but I think it's below a lot of our lines. But the C++ engineers make fun of us all the time for what we put up with. But I think like our project is mostly like any other project. We do have an engineer that like every so often will be like, why is this taking so long to compile? And then he'll realize like some dependency really sucks and then he'll fix it. And so that's really great. I think without that, we would be like having huge, very long compile times, but they're pretty okay right now. Do you use a lot of generics or macros or any special features that maybe make the compile times longer than they could be? Yeah we use okay so we write a lot of our own macros i love macros like absurdly love macros i think what people hate about macros is that you don't get good compile time errors for them but like you can solve this and this is how i write all my macros which there was an engineer at oxide shout out to adam he came up with this way of writing macros such that it's easier. To you don't like get the compile time errors necessarily but what you can get is like if you write tests and then you like look at the output of the macro you can very easily see what changed and then you can see what the code is actually doing and it's just you get so much more visibility that like i think all the pain with macros goes away so like we use macros a lot of places we use macros to generate docs we use macros to just generate a shit ton of code because i hate like manually writing code in boilerplate. So we use that a lot. And then we also use on every single type that is... User facing in the API, we have JSON schema on it. And so that's this Rust package called schemars, which is great. And so then we can use that to generate open API specs. And so schemars had a bug at one point that I think got fixed that was making it really slow to compile because it was just getting exponentially longer with every single type that you had it on. And if you imagine that we had a bajillion types, yeah. So that got fixed, which made it better. And yeah we love macros i absolutely love macros i think probably some of the engineers on our team were like can you just stop using macros but i love them a. Tool which helped me personally dealing with some of the more gnarlier issues with macros was cargo expand where you could see code that was rendered somehow as an intermediate step i don't know if you have heard about it. Yeah i think it's called cargo expand or cargo something like that yeah yeah. Yeah exactly that's by David Tolney. Do you use dropshot by any chance that's the tool from oxide for their open api. Yeah yeah we use that it's great we use it for all of our servers internally like my one beef with it is that they don't have middleware but we solved it with a macro a macro that wraps every single api endpoint and so it was fine um yeah sorry. I didn't catch that they don't have what. Middleware so you can't just like have off middleware that runs on everything or just come up with you know your 87 different middlewares and that's kind of the reason why they didn't do it is because every other server framework that they'd used in the past had middleware but it was never like defined where things were being modified what order were they going in and so i do understand the reasoning and us having the macro makes it very clear what order things are getting modified or headers are getting injected or whatever. And so I get it. It just is not like the most, like, like if you come from a different web environment, you're like, oh, where's my middleware? But you can just use a macro. Can you explain to someone who maybe is listening to us two talking now and doesn't have a lot of experience with open API, why this is so necessary in today's world and why you want to have maybe an API that is based on a spec? Why is it so important? Yeah, so I feel like not enough companies do this. Like one, the win is if you have a bunch of different microservices, you can generate API clients for communicating between them with just a spec change. And so when we say like, so our engine server is separate from like the API server that does authentication and stuff like that and kind of routes things, but it will route to the engine. So when engine changes their spec, a PR from main is automatically opened on the API server. And then that will, in turn, automatically generate a new engine client. And then we can just redeploy the things like it's like everything's automatically done except for pushing of the merge button. And so that that's a huge win. But then when it comes to external users, you can take that spec and you can generate a Python client from it. You can generate a Go client, a Rust client, whatever from it. I will say the off the shelf client generators like just suck. They aren't idiomatic. They're just terrible. So like we wrote our own Go client generator, our own Rust one. And then I wrote a Python one, but our Python engineers hate it. And I need to make it more idiomatic. But like Python is one language I don't really like know the idiomatic like things for. So it is not idiomatic. Like, I think they hate it with, like, a passion, but it is getting better. But I think I spent, like, a day last week or the week before just making it better. But I will say that's our worst one. But and then you can also generate docs from the API spec. And then we generate parts of our CLI from the API spec. So you can just, like, not do a lot of work. Like, if the API changes, we add a new endpoint. Like, the whole world automates around us. We merge PRs that are automatically generated on all these things. And then the docs are automatically updated. And it's just like the world's greatest experience. Like, whereas most other companies either don't have SDKs, they don't have an API to begin with, or they have like a ton of engineers doing this manually every time, which I think is a waste as well. But yeah, it's pretty cool. Do you have any special tooling for creating those pull requests? Just GitHub Actions, yeah. And what do you run inside those GitHub Actions? Because there must be something which... So on the API server repo, if it sees that the spec changed and then that PR is merged in domain, it iterates over an array of repos like kittycad.ts, kittycad.go, kittycad.py, the website repo, and it just... It clones that repo and then it updates the spec in that repo and then opens a pull request. And then in each of those repos, there are scripts for generating a new SDK or generating new website docs off of the spec. And so those automatically get triggered in their own GitHub action on that repo. And then sometimes like way back in the day, we'd get like failures when a new type that was added that like maybe the generator didn't account for. But it's like super rare these days when that happens. I think the most recent one we got was on the Python library where something was named global and that's actually a keyword. It's just like weird stuff like that where you don't think of it the first time you write it. So if the generator fails, you just go fix the generator and then most of the time it just succeeds. And if it's green, it will like push code back up to that repo and then you merge it and then you can cut a release. And then for a lot of these SDKs, they generate a this is like where we get like really meadow with the open api spec where they generate an open api spec patch such that it adds examples for the go sdk or it adds examples for the python sdk so each sdk is generating all their own examples for each function and then all of those get merged into the website repo which then like is like okay take all of these merge them into one open API spec and now generate the docs for them. It's crazy. I really love that approach. And let's contrast that with an article that you wrote titled Boring Infrastructure, because I think it also highlights one other angle of your approach towards programming and software engineering, I guess, because in there you explain how you migrated parts of the code base from C++ to Rust. Can you take us back to that time and walk us through the decision process for that. Yeah, so originally, when we wrote the engine API, we wrote it in all C++, even the server itself was in C++. And I found like a lot of, I think I found two different C++, sorry, this was a while ago, I found two different C++, like server frameworks that kind of did something like Dropshot, where they would generate an open API spec. And that was really cool. But like being able like i don't have the same familiarity with c plus plus as i do with like rust like running it in production like i know how it can fail like i i've seen the ways that it can fail like i've done it before but like c plus plus is just like this like blob of like i don't know like we were hitting just like weird issues where like just blobs were too big being sent to the server and then it would just like crash and i was like what the fuck like i would have known how to fix this in rust and like we would have done it from the start and just like stupid stuff like that. So I was like, the API layer, there's nothing to that that needs to be in C++. Like we can do that in Rust, we can use the same frameworks that we're using everywhere else, we can then still do our open API stuff. And then like, engineers working on one microservice don't have to like, learn a whole new framework to work on another. And so it just made a lot of sense to move it back over to Rust and use Dropshot and all the tools that we already knew. But yeah, that's when we got into the bridge stuff, which is both good and bad. That means you didn't do it purely because of performance reasons or all of the other buzzwords that people throw around. You did it because of pragmatic reasons, because you wanted a sane infrastructure that, as you said in your own words, is boring and maintainable, I guess. Totally, yeah. I guess it's nice to hear from someone that works with Rust a lot that people choose it not only because of the hype, but because it provides real value in such a context. And another area where you transitioned from another language to Rust is KittyCAD, where you started with Go and then you transitioned to Rust. Is that for a similar reason or what was that decision process like? Yeah so i think that was like i wrote the cli originally in go mostly we had like we had rust everywhere and then for some reason i wrote the cli and go and i can't remember like what i was thinking that day but like i even before oxide i wrote go like i wasn't like a rust programmer and so i think i was still in that phase where like sometimes it was just easier for me to pick up go or something i don't know but eventually i was just like this is stupid like like if you think about like long term at a company you can only have so many languages before it's just weird and so i just feel like no one on the team unless they had that like experience using go before would be able to contribute to the cli if it was in just like this other language that we randomly use for this one product like i know that there's like the go sdk but like that's just like a one off whereas like having just random things written in go sprawled across like code bases is a little weird. So basically, I feel like as a company, we've just converged on like. Everything's in Rust, unless it's like really low level graphics, then it's in C++. And then any front end stuff is in TypeScript. But I mean, there's there are days where like me and a couple engineers on the team are like, Oh, let's write a UI framework in Rust. But like, we wouldn't do that to the rest of our team. But like, we're very close to like, thinking about it, especially back when Tauri was being like, annoying. Yeah, we we consider it like React really gets to me some days. But it would also, I do feel like our front-end engineers know Rust. It's just everyone can move a lot faster on their end in React, although I hate React, but yeah. I know that the set people wrote their own UI framework, so they went all the way. Maybe you want to talk to them. We had them on the podcast as well. So if someone wants to listen to that episode, it's just a couple of episodes before that one. Well, I'll have to listen to that one. That's dope. And what about now is it easier for you to write something new in go or rust. I i don't even think i could write go anymore i i mean i i fix bugs in the sdk when they arise but like yeah it's so it's so out of my mind like i i feel like it's just like one of those things where those memory bits have now been replaced by like probably some sort of weird debugging of something else that that i i i can write it but it's just it's not a natural thing anymore like i'm so much more like i writing rust is so much more natural to me that like even even like typescript is hard for me to write because i'm just like looking for like a match statement or oh like just just things where like i want to abort a promise like in tokio you can abort a like async operation you can't abort a promise in Rust or in like TypeScript, sorry. That like drives me nuts. Yeah. Looking back, what do you think about Go? Did they make the right decisions in regards to the standard library? Or would you say that maybe they should have done it differently? Or is it fine for what it is and you just moved on? Yeah, I think Go is great. I think there are like, unless you're writing firmware, which we were at Oxide, you probably don't need Rust or you're doing something where like if we were to write like a modern gaming engine, we would do it in Rust because memory management and stuff like that. But like if you're writing HTTP servers, like I get that we don't do this in Go. But like if your app is just like some HTTP server, you should write in Go. I get that we don't do that. But... Yeah, I think Go really made the right decision when it came to the standard library stuff. Like one of the huge pain points I had with even learning Rust was like, oh, there's no like HTTP standard library. Like instead you have these like four disparate like frameworks, one of which like someone gave up and then one's like half done. And then one's like about to go to 1.0 and it's going to like fuck the ecosystem. There are so many different things going on where I think, what was it? Tokio went to 1.0 when we were at Oxide. And I swear, because at runtime, it would fail if you had competing versions of Tokio in your project. That was the worst. I get that they had to do it. But holy shit, that dependency hell sucked. Because some projects were behind, some were not, some were out of our control. And I think, who's doing that right now? Is it HTTP or Hyper? Like, HTTP called it a 1.0, and then they changed, like, a few things, and then, like, that also kind of, like, effed the ecosystem. I think some things are upgraded, some things aren't. But at least at runtime, they aren't failing you, which, like, is great. Tokio had to do that for some reason. It was very weird. But, yeah, I think the standard library in Go is, like, unmatched. There are so many things that Go did well that we also try to do with, like, KCL, for example, our language, where before Go was 1.0, if they did a breaking change, they would always have some way for you to automatically code mod your code so that you didn't have to update it by hand. And that's how they kept people using the language when it was super early days. And we try to do that with all of our breaking changes for KCL. Yeah, I think there's a lot of good things about Go. And unless you really need memory management or stuff like that, like although i would say go doesn't have enums like proper enums that i really like that in rust if you really like love like type structure and stuff like that like type safety like go is type safe but you can't be as like pedantic about your types that you can with rust where you can have like these like crazy nested enums of objects and shit like that it's just really cool i don't know there's pros and cons. But your backend is in Rust. And I wonder what's the rational behind it. I mean, honestly, and I think I just admitted this in that boring infrastructure article, like at the time of starting the company, like Rust was just the thing that I was like most fluent in. And it's just stayed that way. It doesn't mean that once you get over this initial hurdle of like the learning curve, it gets better. and then maybe you get used to it and then it feels more natural to write things in Rust rather than go? Yeah, for sure. I mean, it took me like a year to get really like natural at writing Rust, which I don't think there's another language that I can say it took me that long to feel like. I was writing like idiomatic Rust, not even like bad Rust. So Steve Klabnik works at Oxide and like in the early days of Oxide, I remember it was like my first Rust project and I asked him like to review it because I like really wanted feedback and he was like really nice about it. Like one he noticed that like i always run clippy so there were like no clippy ears but then he was like you know you don't have to like panic everywhere like all these unwraps you could like return errors and then i was like oh right i could do that and so then i changed everything to use like anyhow and errors and like make it really nice but it's just like it's not even stuff like that like that was pretty easy to like learn and and do versus just like unwrapping everywhere like that was stupid but it's like the the idiomaticness of like if let and stuff like that like i really like that like you can get really into just like the aesthetics of like how you write something i guess which go was also like you were actually supposed to write like if statements and then make sure that the last one was just like a plain return and like return early in the if statement stuff like that like i love the go linter yeah i don't know it just there is a learning curve especially with generics and traits but i like i love traits i love traits like whenever there is something where i'm like oh my gosh this should be a trait i like freak out because it's so it's just very nice you can have like traits and traits and i just it's it's once you get over that curve like it's it's fun to write i guess oh. Yeah i can totally agree with that it's definitely fun fun to write careful with the traits though they can get out of hand talking from experience but yeah overall i fully agree are there any other parts from go that you miss the most in rust that includes the go ecosystem. I actually don't really miss much go stuff i would say the things that i wish for in go if i was still like a go programmer like all the time i would want like real enums like in rust and then macros i love macros in rust and like I use them a lot and, They have this generate thing, which I do use a lot in Go, like the ability to run things before other things. That's how our SDK is generated. It's just different than the macros. Macros are really cool. I never really needed generics in Go. I never really understood that whole thing. But yeah, there were always ways around it if you needed them. Do you think that generics were a mistake in Go? I haven't used them, so I can't really comment to it if it's a bad interface or something. And if I actually had a use case for generics, I would like try it just to see what it's like. But I don't know if it's a mistake. Like the community literally complained about it like nonstop. So, I mean, at some point you do have to like please your users. But yeah, I don't know how they went about it, to be honest. Right. In another podcast, the Embedded FM podcast, you mentioned that not everybody was convinced about the transition from Go to Rust in the beginning. Has that changed over time? Oh, I don't know about that. because I think it was just the CLI. I wonder what I was referring to. Yeah, I can't remember, to be honest. I'm sorry. Yeah, I guess you mentioned the CLI in that podcast and that initially people were not really convinced about the rewrite. But I wonder if nowadays everyone, so more or less is fine with the tech stack, is fine with the decision. I know that you don't only have Rust, but specifically about the Rust part, are there any people that are more vocal about maybe Rust being the wrong choice or is everyone pretty okay with it? I would say everyone loves Rust. We have specifically hired Rust engineers since day one. And to be fair, the C++ folks also write Rust. Sometimes they'll like parts of it, and then sometimes they'll hate other parts. It depends. I definitely know they hate the compile times. There's a lot of stuff with like reference counts stuff that i don't think like parts of our engine would like convert over well but yeah i mean they're definitely like reading the book and learning about it but yeah i would say they probably have the best take i just can't think off the top of my head of their like hates other than the like kind of like issues with pointers and then yeah like the compile time stuff Pretty sure they love it. Yeah They are not here to tell us otherwise so we just assume it's fine there's one third rewrite or maybe one third transition from another language to rust and that was for KittyCAD the language KCL so we got the cli that It was written initially in Go and then transitioned to Rust. And now we also have the language KittyCAD. And initially it was written in TypeScript. Can you talk a little bit about that? What were the learnings from that process? Yeah, so I ported, I think one of our engineers ported the parser originally over to Rust. And then I ported the executor. And to be honest, like ChatGPT did a lot of it because it was just like raw, basically convert this thing. And then I went through and I was like, wait, what are these things? And I fixed a lot of them. But then I think a few weeks later, someone found a lot of other stuff that was like, why do we need this? Yeah, I feel like that was definitely the right decision because... Well, we embed Wasm into the like desktop app and, you know, web app. They both work the same because it's Electron. And then we can actually use the CLI to run the same code as in the app. And being able to share that Rust code multiple places is absolutely amazing. Also, Rust like is just good for writing a language in. So one of our engineers, his name is Adam Chalmers. He just gave a talk at RustConf on like KCL. So if anything, like you should probably watch his talk because it's probably a lot better than what I'm saying. But it allowed like everyone on the team to contribute to the language. It makes hiring for like the front end role harder because we need folks that like either are open to learning Rust or have experience in Rust. And so finding someone that like is a front end engineer and also knows Rust is actually like it's a very small like Venn diagram of people in the world. And then they also need to know like languages because we're writing our own language and like how to you know traverse an ast and and then it gets into like how do you generate code mobs for these things so like our app is like very very complex and then and then you have to do math with geometry so like on top of all these things the venn diagram it's smaller and smaller and smaller on these subsets of people that would fill it and but actually like being able to have like the same infrastructure engineer, write stuff in the language is really cool. People who work on various systems and microservices can also contribute to the language. I can contribute to the language easier. It just feels more type-safe. It feels better. Everything about it, I think, is better. It's probably faster. We didn't really test it, but VA, it's really fast. So it's hard to compete with JavaScript, honestly. Kitty is by all means and purposes a real programming language right so that means that you can do calculations inside of a CAD program if you want to but doesn't it also require a language and a parser and a lexer all of these very hard computer sciencey things. Yeah so that's why hiring for this app is really hard because like we'd want a front-end engineer that knew all of those things as well and so yeah there is elixir there's parser uh we use winnow which is really cool it's like nom but better i think it's like one of the maintainers of nom like split off and did his own thing or someone who's really fed up with nom split off and did their own thing i forget the story there but it's basically just like nom but better and and yeah like tools like that just exist in the rust ecosystem which also makes it a lot easier oh. That's that's nice to here i want to switch gears for a second because i know that your company built an async Rust debugging framework and i wonder why was that necessary what's the current standpoint on async Rust and and why did you end up building a debugging framework for async rust. Okay so yeah in our engine server because the engine communicates over web rtc and web sockets we spawn a lot of async jobs I mean not a lot but like it's like okay so we have a job for the web socket like read write, type of transmission and then we have a job for web rtc and like weird things were happening because also then like libraries that we use are spawning their own things and we needed a way to know like what was either like getting deadlocked or what was what was not killing off like acing jobs because a ton of people like okay so when go first started everyone was using channels wrong like literally everyone is using channels wrong they were just like creating shit they weren't like keeping track of it whatever this is the exact same problem i think people are having with async like just spawning stuff so and the web rtc library specifically in rust that we use was literally just like a raw port of the go one and so there's just a lot of things that are very weird about it and and and basically this framework allows you to watch the processes and figure out which one is spawning something that is then the one that leads to issues. And yeah, that's basically what led to that is just trying to find like either what isn't being cleaned up or what is just hanging around. Yeah. How can I avoid these issues myself if I wanted to avoid issues with spawning, async tasks and maybe also cleaning up and doing it properly? Are there any tips, any best practices that you could share? Yeah, like... Instead of just like raw spawning something, like assign it a variable and then keep that variable around to either abort it if you have to or whatever. I mean, if you're just doing a CLI and then it'll abort when you kill the process. But if you have a long running service, like a server, you really need to keep track of them and abort them when they're done. Yeah. So it's mostly about hygiene when you have long running servers. I get that. Yeah. Because otherwise you run into all sorts of weird issues like you can run out of file handles or you that's a classic, i i definitely can remember those times from go and spawning work groups and so on and yeah like, go does not prevent you from doing this and i think rust is kind of the same if you have a long-running async task then yeah totally keeping this this file handle around we're coming towards Towards the end, I do have a couple of questions regarding your background at various companies. You contributed to Golang, Kubernetes, Docker. You worked at Google. How does that experience influence your work at su.dev? I feel like in a lot of different ways, just like I was explaining, like with Go, we kind of stole their idea to like use code mods pre v1 to help people like migrate to breaking changes. Then there's like, let me think, like Docker, a lot of experience there when it comes to like keeping a large amount of people happy, like making sure, especially with an open source project, like you want to build a community around it. You want to like be engaged you want to reply to people but also it's like if you add a feature you can never remove it and so being very picky about the features that you add, to projects like that's kind of one of the main learnings that i have used the rest of my life there and then when it comes to like working at big companies like microsoft or google i think like a lot of the folks that we hired and stuff like that like no one really likes middle managers yeah everyone contributes at the company and like you never feel like one of your co-workers just isn't doing anything or like that your manager like doesn't really add value or that you know things are just coming from above with no reasoning behind it and so yeah there's like a lot more trust on our team i would say than necessarily at a big company and i like that about it yeah i would say those are probably the main things. And at least to an outsider at the core of zoo there's a technical team so it's a tech company in the best sense of the word, you have this one anecdote on your blog where you described it you went on vacation i guess and you asked around for books to read and i guess it was Bryan Cantrill who suggested the soul of a new machine and you said that it inspired you in some way can you talk a little bit about that book what is it about why did you like it so much. Yeah so i think that book like like a lot of it went into the founding of oxide and that like but it also like the founding of any company they were building a new computer it was a lot of god i'm trying to remember the book a lot of like debugging a lot of making it work and it's just like one of those stories from like the early days of a company where like things are crazy and everyone's trying to like get things done and it's that excitement and like, it's it's like written really well so you can feel the excitement of it although i will say like at a startup like sometimes when you're in it it doesn't feel exciting but it does like after the fact and yeah like those those hard days are hard but and i think the the book also like goes into that well but yeah the the dream of like building something new and actually executing on it is pretty cool. And also this rewarding process of building something from the ground up from first principles. Yes now. Knowing you you are in a very unique position to answer the following question do you have any advice to decision makers who aren't sure about choosing rust because you've seen it all you come from very diverse background and i think now you're also heavily invested in rust with this company are there any tips anything that you would maybe share with a fellow cto or decision maker someone that is maybe on the cusp of giving rust a chance what would you tell them. Yeah i mean honestly our experience has been great i i don't know what their fears would be but maybe it would be like there aren't enough rust engineers in the wild but like that's not true we have had like new we haven't like had a hard time finding rust engineers if anything it actually helps you out to be that odd like not odd company but like that want a company that's doing stuff in rust like it helps you like find engineers and really good engineers because i would say the the folks using rust today are like the the folks using any language early even though it's not early days anymore are like a cut above the rest i would say because you have to have that like willpower to go use something that no one else is and so yeah and then just when it comes to type safety like resiliency like i have time to do like greenfield stuff. Like when I have time to add new commands to KCL, I have time to do a bunch of stuff that is newer because a lot of our stuff just kind of self-maintains itself. And I know a lot of that is a mix of just like architecting things well and writing tests and making sure there aren't a lot of bugs. When there are bugs, we obviously fix them. But I do think. The language lends itself really well for maintainability and really exactness. There is no time that I'm like, oh, is this going to fail with some random input? It's like, no, you directly define what the input is going to be. And then as long as it's that, you're fine. And it will fail well. I mean, it will air well in the other cases. And so there isn't just this ambiguous, vague definition of things. It's very, very exact. And in the beginning, when you learn the language, that can be off-putting, but I do feel like it's the hugest pro of the language is the exactness of it. Because with any bug, with any language, when you're doing stuff with the computers, it's deterministic. If there's a bug, something in the stack is wrong, and you have to go debug it. I will say this is the thing that drives me absolutely insane about artificial intelligence and that is not deterministic but like if there's a bug in your code or the computer is misbehaving that means like you coded something wrong or something in your dependency is coded wrong like something is telling the computer the wrong thing to do and i feel like because of how exact rust is there is fewer times where it's ambiguous what is going wrong, And like where or where the problem is coming from, just because nothing's really vague. Yeah, sorry, that was long and rambly. No, it was very nice summary. And I fully agree with that. The reliability part is sometimes more important than the raw performance or the fearless concurrency. At the end of the day, you're running a business here and outages have a cost, a real cost to it. And also you lose trust with your customers. So I think these are things that are rarely talked about. These are important. And I guess Rust can help there. Yeah. And as someone who writes React a lot of the time for our front end, I mean, it is like night and day. Because in JavaScript land, React land, it is ambiguous. There's so much magic happening behind the scenes. When things go wrong, you're like, I have no idea what the hell is happening right here. Or it is hard to account for every single scenario that could go wrong because it's just so vague. and yeah it just makes me love rust in contrast how. Much of it is because of the type system strong versus weak typing and dynamic versus static typing. Yeah i mean types of it is a lie at the end of the day like at runtime it's not gonna do shit yeah i really think i really think it's like the the whole package like having actual types and very very very strong types whereas like go has types but like they lack enums like what i was saying and you can't do like really pedantic things with types that you can do in other languages you can but it's just weird um and people don't do it as much for us i feel like at oxide people were like coming up with the world's weirdest types but they were like very very readable and they made the whole thing readable but every time it would break the sdks which is the stuff i was writing and i was like oh my gosh i didn't account for this one weird off type but yeah but i appreciate the exactness of it. Okay so it's a bit of a double-edged sword but it. Pays off in. The end now traditionally the final question is do you have a message to the Rust community. Yeah, I guess like, thank you for all that you do. Yeah, it's, it's cool, like, seeing, like, whenever I go to like docs.rs, and I look for a library that like I need, and if it's actually there, and it's actually good, that makes me really excited. I do know there are like things missing and stuff. So to new members of the Rust community, there's always things that you can build that haven't been done before, which is cool. But yeah it's pretty cool to be a part of something like rust because it's a good language and yeah i enjoy like whenever i like google something weird there's like a blog post or something so yeah. Jessie thanks so much for your time i really appreciate it was a pleasure to talk to you. Yeah thank you so much for having me this has been fun. Rust in Production is a podcast by corrode it is hosted by me Matthias Endler and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust visit corrode.dev thanks for listening to rust in production.