It's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. I'm Matthias Endler from corrode, and today's guest is Tom Hacohen from Svix. We talk about reliable webhooks at scale with Rust. Tom, thanks a lot for being a part of this podcast. Can you introduce yourself and Svix, the company you founded? It yeah for sure so i'm tom the founder and ceo of Svix what we do is webhook sending as a service so we help companies send webhooks you know people use our simple apis and sdks and really with a few lines of code they get the state-of-the-art webhook sending system and actually i mean now that i think about it we also finally publicly launched the receiving aspect as well so we also help you with receiving webhooks so you know if you use a vendor that has not so great webhooks i mean preferably you just tell them to use svix and you know they can improve the the you know improve everything for everyone but if not they can just you can use the receiving svix and kind of get a great experience in front of that. What's so special about webhooks isn't it just a fancy link i mean i know how to serve an api and i can probably take a call from someone but why is it so special why should i use a service for that. Yeah so i think first of all yeah it's really just a simple HTTP post request. You know, that's all you need to do. But I think like everything in product and engineering, the devil is into detail. And it kind of like when you start moving to production, all of a sudden, you know, it needs to scale. And when you have multiple customers, you need to make sure that one customer having issues is not affecting the other customer. So you have like noisy neighbor problems and you have like security because webhooks, you know, you kind of mentioned API. It's well understood. You have, you know, like a request comes in you do some off token checks and then you respond but with webhooks you actually make calls from your own server to an address controlled by your customer slash attacker maybe and they can point it to an internal system and you know trick you this is called with server-side request forgery so essentially without going too much into the details there's just a lot many more things that you need to worry about than you then people that people are not used to but additionally you know, when it comes to product infrastructure, which is where I place us. The best you can do, let's assume you spend a lot of time on it, and we spend a lot of time on it, right? We're a big team just working on webhooks. If you spend even a fraction of the time we spend on it, and you're doing a great job and you're wasting a lot of your team's time instead of advancing the product roadmap, and everything just works, then your customers aren't happy with you because you haven't built the features that they wanted. And if you do anything if you do your the job less anything less than stellar then that means uptime a downtime sorry and so like you're gonna have issues so really i think when it comes to infrastructure like really the best you can do even if you do everything perfect is just have a neutral effect on your customers your customers won't notice that you spend all of this time on it so and this is part of what we help you to do is really just like give your customers a state-of-the-art experience they don't think about they don't worry about it just works and for you you don't have to maintain it and worry about any of that. What do i get when i use fix other than a endpoint is it also providing some logs monitoring metrics and uptime statistics what do i get from that. Yeah so that's a great question because you know when it comes to webhooks you know again going back to the api request if your customer of yours tries to make an api request and that api request fails right they would know it fails it failed they got an error code back with web hooks they don't know that you were just sending a web hook just now and it failed for whatever reason they have zero visibility because it's generated by the sender so logs and and observability are essential there and yes we offer that out of the box and we actually have a pre-built ui that our customers can embed in their customers fully white labeled sorry embed in their dashboards for their customers and then their customers can just go and use that to manage the webhooks, see the observability. We have automatic retries, but they can also manually retry once they fix an issue, really have full visibility and full control into the whole webhooks sending system. Before you built Svix, there must have been a moment when you thought, wow, this is really challenging to pull off and there is no existing service that solves that problem. Take us back to that time. Yeah so it's exactly how it started and it was really just by luck so at my previous company people were asking us for webhooks i mean we kind of like looked into it started designing the system you know i did everything to kind of like to plan for the build and then realized it's actually so much work and we just don't have the capacity to build it at the moment so we said no and then people kept on asking us and we kind of like went back to the drawing board like okay we can find the time to you know to do the initial build but then realized there's going to be a lot of maintenance you know ongoing maintenance and like you know they're going to be like random requests by customers that we're going to have to build otherwise they're going to be unhappy so we said no again and then people kept on asking us still and we kind of like okay you know we're going to find the time to build it we're going to allocate the resource to maintain it but we have this beautiful api great product we're not going to have a terrible webhook experience right so we multiply the first two estimates by like 3x and 4x and again said no and. I i wasn't smart enough to realize at that point that there was a business there but it was like a few months later that a friend of mine asked me a question about webhooks in like in a slack community and i kind of like started explaining to her you should do x and you should do y and i realized she couldn't care less about anything i was saying she only wanted to send webhooks you don't have to think about it and that's kind of like when the you know the apple dropped on my head i mean and i realized that you know what i had the same problem and and i asked her like hey if i build a service that does this will you pay me and she's like hell yeah and that's kind of like how it all started and by the way spoiler alert she never paid but other people from that slack that did become our first customers so that was a win and i'm eternally grateful for for her asking that question. I really like that story because it has a few ingredients which i look for in companies they solve a very narrow focus problem and no one's gonna take away your lunch because, other people don't want to care about that problem it's essentially quote-unquote boring for them someone needs to solve it but then if you look behind it there's a ton of complexity and you need to do it right in order to provide a good service but can you talk about those customers a little bit who were the people that asked for webhooks all along. Yeah so those initial few were small you know friends of mine like it's like a startup slack like small startups one was like a jira competitor i guess you would say and and you know they needed webhooks because jira has webhooks and customers rely on it to to build those integrations you know when you close a ticket they want to run some workflow but since then we know we started we've been working with all the way from like very small startups so indie developers all the way up to the fortune 50, and the use cases are very varied to the point that I actually, I kind of like look at us and I draw the parallels to like Twilio for SMS or SendGrid for email in the sense that, you know people just send emails i you know like sandwich customers just send emails i sandwich doesn't care what's in the email and similar to us we just see so many different use cases that people just use us for for really whatever kind. Of makes me wonder if you were not really scared of being a bottleneck or a single point of failure for all of these customers. Yeah man this is this was at least in the beginning was like a dreading thought and you know one that literally kept us up at night with page duty and you know and everything but but you know like a lot of startups they would focus you know it's kind of move fast and break things right like you would focus on speed as the main feature and similar to other startups fee speed is extremely important for us but for us stability has been like the feature that we offer like we we have a stellar track record and it's because we spend a lot of time effort and money on redundant infrastructure to make sure that we do, you know, we are that potential point of failure and we make sure that we don't ever go down, knock on wood, obviously, but we spend a lot of time and effort there. And to the point that, we are more stable than homegrown webhook systems. Like we are, you know, we work with some of the best companies out there, whether it's like Brex and Benchling and Lob. And again, as I said, I have Fortune 50, so we can't mention their names, unfortunately. And we are, you know, they trust us and use us because we are stable. We are reliable. And again, and we focus all our time on it. So it makes sense. Okay, you said speed matters, especially for startups. And I fully agree with that. But then immediately a lot of people might think is rust the best application it's it's rust the best language to build such a service then if you want to iterate quickly can't you build a quicker prototype in another language. So first of all we did build initial version of Svix was actually written in python so i guess there you know there are legs to that statement. And i think you know rust is problematic for fast iteration at the at the beginning of the company so like the way i think about it is if you're building i mean okay if you're very proficient in rust and you know you you move very quickly in rust just do it and you just use rust you don't have to worry about it just use you know use the tool that you know but at the beginning the really the beginning of the company when you don't even know what you're building and you're trying to figure it out and really what matters is just getting something clunky out there working and i think rust with the emphasis on correctness can hold you back but i think the moment you have even like a hint of traction like you know like even like a few months in and you realize that what you're building is generally you know again generally not it doesn't have to be precise but generally what you're going to build i think then rust is extremely powerful because rust you know when we were using python we we used to write a lot of tests we still write a lot of tests but much less and also kind of like have issues in staging and sometimes very rarely have issues in production as well and with rust it's kind of like you spend all of that time in the beginning but the likelihood of errors is significantly diminished later on so it's kind of like this it's almost there's like a preservation of energy so preservation of development time But with Python. You don't spend a lot of energy in the beginning, but you do spend it once it hits production, once you need to debug, once you need to refactor. And with Rust, you spend more in the beginning, but actually then later on, you don't have to spend as much energy. And I think now we move fast. Wasn't that transition painful. Oh extremely yeah i think you know like as i said like being reliable for our customers is the most important thing for us so we couldn't do what some companies do even even some of our competitors do which is you know oh yeah we're going to be down for a few hours on saturday we have a maintenance period we can't do anything like that so it was all like live transitions and you know making sure we don't break it to you know don't break anything for anyone like a lot of testing and what's i guess challenging about what it is that we do is or one of the things is you know we send http requests to a variety of servers right i mean like webhooks so that that means that we interact with you know i well well what's the name of that microsoft one ias iss why ias yeah so like ias apache nginx you know all of the wide variety of servers and some of them have very specific requirements for how a client should interact with them at the moment you know we switch to another client and like, like for example in rust we we were facing issues and i think actually a small tangent you know like one of the best thing and one of the worst thing about the rust ecosystem is the love for perfection and unfortunately the world out there is like ugly and dirty and when you interact with that world. Sometimes you can't be perfect like yeah you can adhere to the spec perfectly but if in our case but if the servers that we send the webhooks to expect something that is not required by the spec but i'm sorry that's what i expect you know we need to follow that we don't care that they're wrong we have to, We have to deliver that message. And so that, yeah, there were just like a variety of challenges. It just kind of triggered a bit of PTSD, I guess, from that time. If we go back to the first Python prototype that you built, it somehow also worked, right? It was running in production, but what was the general sentiment like at this point in time? Did you trust in the application? what was the deployment process the icd testing how many issues did you find in production versus development. And so on yes so actually a bit of again a tangent before like i was when i started coding i was i guess pearl in the beginning or whatever but i was like my first professional job was like a c embedded c engineer and again i have the scars to show for like you know like the living as a c engineer where like everything can break everything is scary and and and a very primitive type system as well so one of the things that i was doing i remember that as like a very junior engineer is adding a lot of typing as much as i can to see with like you know there are like a few tricks you do like opaque structs and like opaque type depths with like opaque structs whatever like a few a few ways a few tricks to do it um but that experience just made me obsessed with, telling the compiler what I really mean and having the compiler complain at build time about all the issues so I don't have to think about runtime. I know that if it compiles, it works. And by the way, I think that's partially why I'm such a big fan of Rust. So the reason why I told you all of this is that when we were writing Python code, it was already very good with typing. We had hackery around SQL Alchemy to make it more type safe. And we had even like scaffolding for fast API to make that even better and Pydantic. We had like a few places where we're doing, you know, kind of like we're doing what we could to make sure that Python has a very rich type system. So I think we were already doing like a fairly good job in terms of like catching a lot of issues at compile time. But the problem with Python is that those type, all the hacks that I mentioned were not real, right? It's not really a type. It's kind of like, you know, an annotation that we added. So we still had issues where we got the annotation wrong. And to account for that, like what we did is that we had a lot of unit tests, but we also had a lot of end-to-end tests. That was really the big thing that we did. Like we had a lot of end-to-end tests. And the end-to-end test suite like runs on staging, on real production, like against real load balance, everything, just to make sure that whatever it is that we do just goes, you know, safely goes into production. So we were pretty good. Like, I don't think we had a lot of production issues... I mean, definitely no serious ones, but it could be that we added a new API and that API maybe had an error when it was just introduced in some scenarios. So it was fairly good in that regard. But the problem was when you want to refactor, when you change something. That is the scariest experience you can have as a Python developer because you don't have that assurance of if it compiles, it just works. And man, finding those and going... Like one of the most annoying things is that like, and this used to be a common occurrence, doesn't happen for us anymore, is that like staging would be broken because someone merged something and that for whatever reason broke something. And then you kind of like, oh, I have another fix. And then like someone else brings a fix and someone else brings another change. And all of a sudden like unwinding those in staging is like a big pain. And that just doesn't happen anymore. So I don't think customers fully noticed the experience from a stability standpoint, but we definitely did in terms of just like how healthy our staging environment was and how big of a pain it was. The one thing that you triggered in me was also this PTSD that I had when refactoring larger Python codebases, where you end up with a... An exception somewhere really down the call stack in production and you have no idea what's happening you have no idea of the state that the application was in and you trigger that maybe once in a million requests or so because it's a very weird combination of conditions so yeah i can totally relate to that although i was wondering shouldn't the type system or the types that you added to the python code have you know saved you from that wasn't that the idea. I think the problem is that the python type system is just not expressive enough to really i mean maybe you know maybe nowadays it's like slightly better but like enums for example i don't think i mean maybe now they exist i don't think so and it's still not to the same level but it's like enums and being able to catch all of those you know all of the all of the variants yeah thank you and another thing that you said that triggered me is like exceptions oh my god i think that is the billion dollar programming mistake like i think rust really got it right in how to do error propagation which is you don't you can't just like willingly throw random exceptions all the way up to the stack like you have to catch them at every step and define the you know like properly define the um the signature and kind of like the contract that you have with the caller um i think is extremely important so yes python helped a lot but again as i said we missed some places like we didn't get the typing exactly correct in some areas we didn't the python was expensive enough to really get everything that we wanted in others it was scary it wasn't uh just walking the park what we trust and we can cover that later like it really we really utilized the type system to just make many runtime errors just like impossible. I don't want to put words into your mouth but this other thing that i really don't like about python deployments is that they are large you have large containers. Yeah and. In rust you have small binaries in comparison did you. Run into that problem okay so we we were using aws lambda for some things back in the days and you would hit i'm pretty sure we hit lambda size limits, maybe i'm misremembering no i'm pretty sure we hit lambda size limits it's like 150 megabyte but even if it wasn't a hard limit i think it was you know those were hard limits but even if they weren't i think it was just you'd have performance issues maybe or cost i don't remember it was annoying though it was really annoying yeah i mean nothing more to add there it was really annoying and like and also because there's a runtime, part to it like you would have to wait for for lambda for aws to support python 3.10 before you can upgrade so we still like we were like a few versions behind it was really like a big pain by having like the runtime being shipped by someone that's not us okay. So we agree python might not be the best choice for what you wanted to build as fix a production grade service for webhooks, but Rust might not have been the only choice that you might have considered for example what about other languages like Elixir or Golang wouldn't that be an option too. Yeah so I can only speak to how it was when we made the decision maybe things I mean I know some things have changed in Elixir land and maybe a bit in Go so Elixir was even more esoteric than Rust back then and like choosing an esoteric language like Rust was already a bit of a gamble. So I had to, you know, I had to like hedge my bet to an extent. Like I couldn't go, and it's not that I wanted to choose Elixir and that's why I didn't choose it, but it was just like off the table just because of that. And also I had experience with Rust before and I didn't have any experience with Elixir. So, you know, the amount of unknowns was much higher, lower. Like I built production system in Rust before that. Not web, by the way, web was not ready before Sphix. like we were really at the cusp of like bus being web ready but yeah i've built like production stuff before in rust and then go it just and again maybe things have improved but the type system is so bare and as i said like i'm all in it's kind of felt like going almost backwards from like python like and then really i i mean yeah compilation speed amazing like i wish we had it in rust like i can we can probably talk about it later like i will complain you know for hours at no end, about a rough compilation time. But. The Go type system was just not expressive enough to really capture everything that I thought should be captured. And even without that, we actually wrote the first version of the rewrite of Sphix in both Go and Rust. And I say like first version, like we spent a few days on the Rust one, a few days on the Go one. And like someone, I wrote the Rust one and like someone else from the team that knew Go very well wrote the Go one. And then as a team, we just looked at both and we just decided like Rust just makes much more sense. In terms of like again all the areas that we cared about like type type safety but also just in terms of like how it is to use the language and there are like a few things that are done and go that i know go go for his love but i i just can't understand like the implicit implicit imports drives me mad like i like not implicit just like non-descriptive imports like the capitalization for visibility also is a bit annoying reminds me of like Hungarian notation, but i think also like the way to do json parsing if i remember correctly that was like i think it's done in comments or something i don't remember i remember it was just or not comments like a go specific comment thing annotation or something tags labels. Or whatever i guess tags yeah. Yeah again i don't remember it's been a while but i it just essentially we kind of like we look at the resulting code and we just okay there is no comparison here like we can build something on top of Rust, then I don't think we can build the same thing on top of Go. Did that person that wrote the Go prototype agree, or did they challenge the decision? You know, so I will admit that I'm the CEO of the company, so there's obviously like a bit of a power imbalance there, but from what I remember, there was like buying from everyone. The only concern was like finding Rust talent, which I shared. And... In this prototype was there already some sort of concurrency did you use you know go functions or so or go routines actually to maybe make some things concurrent or or was it linear. No no no so we i mean we built a tiny part of the product but we built it like we think it would look like in the end so even you know including like some weird scaffolding that we would you know normally you wouldn't want in like a first version but it's kind of like okay how how would it look like if we actually did this scaffolding. When you built this initial prototype you probably compared go concurrency with rust concurrency what what was the verdict there. Yeah i don't think there was you know like a a strong comparison there in the sense of you know it's like benchmarking or stuff like that we really we cared a lot about you know the typing and developer experience really what swayed us but we did use both right i mean go concurrency is kind of like nice in the sense you just like it just happens but i think i personally but this is a subjective thing i like i like the fact that there's like an await syntax i like the fact that we color functions by saying you know what this one is you know this one can run you know in an async context like this one has io this one has whatever i think that's a nice thing but this is really just a syntax thing i don't think we had a deep you know kind of any anything deep there and i i will say though that we and this is like a bit of a tangent we there was like a rust library that was doing magical. Io in the background so it was like you wouldn't the function wouldn't await but in the background it would trigger something that would make changes in the server and it's kind of just like again in a background thread or something like again like magical even though it wasn't like you didn't call a weight on the function and i remember that experience just remind me how much i love the fact that like, any io functions you know that io functions are you know have to be awaited because you know, you know you can easily look at the signature and know that something affects the server or if something doesn't affect the server is this like a local configuration or is this like a a remote configuration i know this is like an esoteric example but it really just we spent so much time debugging this because we were so confused by the fact that this non-async call was actually you know touching the server no. I can relate to that because a lot of people say function coloring is an issue but it feels like what you allude to is more of more the opposite where you say you want to know if something is async you want to be explicit about it and if you structure your code differently maybe it's not going to be a huge problem because you know that this part of the application has side effects and does ao in an asynchronous way and the other part does not it's. Interesting it's kind of like you use the word you know like function coloring right and i think that's a great example like you know we color we have syntax highlighting right in our code in our code we want to highlight things that behave differently than things than other things and especially things as important as you know i mean potentially if we don't even have like asking away like just blocking operations which is like i don't know how anyone can advocate for like, not having a way to know what our function is blocking like that yeah just beyond me again for blocking operations yeah. Okay so programming other programming languages were ruled out we talked about golang a little bit and elixir was just too cutting edge back in the day so it does make sense to take a closer look at rust but the other option that i had in mind at least maybe you discussed it internally was maybe we just rewrite parts of the application in rust maybe by using language bindings with py03 and then you rewrite parts of the critical stuff and you keep the rest did you consider that option. Yeah i mean i think for us the the kind of like two parts that cause latency or you know memory usage all of that like the performance either lies in the database queue or you know other like background systems that we use, or in the json serialization aspect and all of that so there isn't really a lot of code for us that we could out so there's not like an like an ai operation that you can just like okay i'm gonna or you know like a numpy thing like you know some like heavy scientific operation you can just do rewrite in c or in rust and we'll just be faster and everything else can just stay in python for us, the the the wrapping is the code like the you know like how quickly we can respond you know how you know the latency when we respond to http calls that is something that matters to us like how much memory use you know we use when we respond to an http call that's what matters so the shell was as important as the code itself so we we didn't have anything obvious or anywhere obvious to make that yeah and that change especially since by the way like all the serialization costs are just as significant so, It just didn't make sense. And, you know, actually another thing that comes to mind, we talked about, you know, the typing as well. That's another place to lose type. Like really there was no, we didn't even check that option. Like we just decided to commit. Yeah, because really what you build is a platform. It's sort of runtime for web things like webhooks, for example. And yeah, I see that you also need to handle a lot of JSON. and then if you transform a lot of JSON objects from Python to Rust and back, that also incurs memory overhead. Okay, that means we finally arrived at the final decision to rewrite the application in Rust, but it probably wasn't an easy path. It wasn't an easy migration. Can you talk a little bit about the learnings from migrating from Python to Rust? Yeah, I think, you know, like having making sure that the database types are the same wasn't that hard but still wasn't like super easy and and then just you know making sure that the ready serialization is identical because this one did it slightly different than the other one or maybe we you know we messed up something in python it was actually like named you know slightly in a weird manner that's inconsistent whether it rust we just kind of like did you know camel case everything and then in python we had to you know any whatever we had to like create weird aliases for like some of the cache stuff but i think other than that and other than the difficulties of just like a rewrite with zero downtime i think it went fairly smoothly and we're super okay first of all i'm a big fan of rust i'll i'll you know stand on the roof and shout it to anyone for anyone to hear but we did have some areas that were unexpected that where rust wasn't perfect actually and there were the areas that we expected like compilation i mean compilation app actually was worse than what we expected and there were other areas like finding engineers that was a bit of a challenge back then nowadays you know there are like a lot of rust engineers but there were actual, areas where rust or python i think i don't know if better is the right word but like there were just like less foot guns i guess i'm happy to elaborate on those if you want. Yeah please in your blog post you mentioned a few things like heap fragmentation uh, and memory efficiency maybe these would be a few things that. Yeah i think kind of like you know like if you summarize all of those in like in one you know kind of sentence like with great power comes great responsibility so kind of like you have more control now and so that means you have to be more careful about things the one example that you mentioned was like the heap fragmentation, so rust gives you more control there and i guess for whatever it just uses the default system allocator by default which just didn't work very well when it comes to fragmentation so like a long running process so heap fragmentation for those who don't know is essentially when you allocate big chunks or smaller chunks and then you can like every allocation just adds more to the memory instead of like reusing those chunks because they don't fit you know let's assume you you allocated a chunk of like 90 bytes then you freed it and then you want to allocate another chunk of like 95 bytes you can't reuse that one so you're going to get another chunk of 95 bytes then maybe you're going to reuse it for half of that smaller chunk but then you're not going to be able to you know another 70. Byte chunk would again would have to go on top and essentially what happens is that your memory just grows and grows and grows even though in practice you're actually not using that much memory using the same amount of memory but just because you allocate and deallocate in different sizes all the time it just causes fragmentation this was actually fixed by just switching to jmalloc like just changing the system allocated to one that you know is very good and really did wonders for us but the fact that we had to think about memory layouts and investigate like memory usage you know it's kind of like it just gives an example of like some of the stuff that you now have to worry about when you start using rust it's. Funny because the rust compiler itself or the Rust ecosystem moved away from Jamalock to the system allocator, but apparently for some use cases that isn't the best choice, the system allocator it is. Yeah, for us it definitely wasn't, yeah. But, where would you notice heap fragmentation was it a very occurring a reoccurring pattern that you could have solved with for example a bump allocator or an arena allocator was it expected, load of small objects that you allocated at once and could de-allocate at once. Yeah so i think for us i don't you know it's it was also like a year or something ago but like i think for us it was a lot of parsing json and the json would come from customers and we would send their json for them so that means you know because like it's a webhook so that means that that json can have you know for you know can be of varying sizes and can be and would normally be very large as well, i don't know we probably could have you know made so do you use a different allocate or something like that but it's just like changing the the default allocator would just you know solve everything i think that was probably the main thing another thing is you know we use some aws libraries and they i i believe they allocate we make http calls and that i believe allocates when it kind of creates generates the body that we send and so there were just like a few areas where we were allocating large arbitrary sized structures all the time yeah did. You notice a difference in memory usage when you switched the allocator. Oh yeah so before that it was i guess you can't see me it's only audio only but like you would see you know you'd see like a bump i guess because of some you know like high loads that would cause a lot of fragmentation and then that bump would just like stay flat and then there would be another bump and then that would stay flat as well, with jmadoc you immediately see just like, First of all, it becomes more jagged instead of flat. You'd see the graph of memory usage becomes way more jagged. And the low watermark was much lower. The baseline was significantly lower. We can post the link to that blog post as well in the show notes. You'll see there's a few graphs there. You can really see it. It's very obvious. Was memory usage even a concern for you? even if you used more memory would you run out of that. We did that's how we found out okay and again not immediately but just like every now and then like you get an oom and you're like what the hell is going on you know we see our usage we like we have 95 buffer like what's going on but we then we look at some of those and we see that there's like a memory leak but nothing is leaking right we did like a run valgrind we ran everything like nothing was leaking and then that's kind of how we discovered a few actually more things about like memory usage that were surprising again like you'd think russ is much more memory efficient than python right i mean like if you ask anyone on reddit or twitter or whatever they would they would tell you exactly that, but one thing was with 30 json like when you were passing generic json values so just you know like essentially again a dict but like a json dict so like not a not a string string but more just like generic, again, generic JSON, that would, for whatever reason, in serde, that causes like an explosion in memory. And we didn't, We had no idea. And I think, you know, like in a language like Python, you would think that just the generic case, or, you know, you think, I think the generic case just kind of like works, like because they optimize for that, they think about that, like, well, in Rust, you know, like using an untyped structure, that is such an, you know, such an odd case. And we were just like surprised by how bad that was. And maybe this is, by the way, was the reason for the. For the, well one of the reasons for the you know fragmentation that we're talking about i'm not sure and there are solutions so you had to know that you don't need you're not supposed to use json value you need to use json raw value that just like treats the string it doesn't power set but like again in a in a world you know in python stuff like that would just happen magically in the background you know kind of like copy and why and parse on access and all of those kind of things and i think with rust everything is much more rigid like what you ask for is what you get and it just like was like a funny gotcha that we that we got does. Python deserialize lazily. Probably not but there are things that i'm sure it does like to be efficient but maybe it does by the way i really have no idea well. Especially if you share those large json objects then i can see where the efficiency comes from because. In rust. You would generally clone unless you put it behind an arc but in python everything is ref counted so you get cheap copies for free. That's another like example of where you know memory use can just just go out of the window because as you said in like in python you don't really care about you know kind of like what was it like fearless concurrency is that the yeah you don't really care about that like yeah of course space conditions that's just a fact of life in python so well i guess maybe people just don't use threading as much and so they just as you said pass like ref counts all over like in rust like you would clone so much data if you're not careful like you would accidentally clone that json structure like that heavy when you pass it to another function and you can accidentally clone something else and it's very easy to accidentally use much more memory than you would use in python just because of the you know like the implicit ref counting and just like passing of references instead of cloning yeah. I always wanted a lazy json library in rust and probably it exists but i couldn't bother looking it up um but it would be so nice to have a system that does not create 30 json values all the time but it more or less is a view into the data and then it only deserializes what you actually want to look at. Yeah i think so kind of the 30 row value kind of semi lets you do that by the fact that it doesn't serialize that part and then you can maybe explicitly deserialize that part. I mean, again, you have to build the scaffolding yourself, but it is semi-possible. Or at least that's what we're doing now. True. And for you, the serde interface is probably, I'm assuming, one of the most important parts of what you use in the Rust ecosystem. What else do you use very heavily? What do you depend on heavily? Which creates? So serde would be one. Axum, we use SeaORM as our ORM. I mean, mainly for the query building, less for ORM stuff. We use redis rust we use aid for open telemetry generation i think those would be the main ones and actually i mean you know like this is a good segue for like ecosystem maturity because we you know we have the maintainer of one of the maintainers of axiom on the team we have one of the maintainers of redis rust on the team we have one of the maintainers of aid on the team and it's not you know i wish we didn't right i mean i'm happy to have those people they're great and like they're very you know you know they're smart and capable but like we we didn't actually have to we have to spend time and effort on these you know like core libraries that we depend on like we have to spend time and effort on the ecosystem to make sure that it's where we need it to be, which is a bit i mean i think it's less less so nowadays and i guess if you're starting a swix competitor you have us to maintain this for you so like you don't have to incur the same cost but but it was a challenge and it definitely is still is but. On the other side you need to have some skin in the game. Yeah again no no not complaining really not complaining but you know if we had the alternative which would be like having the perfect redis library just existing and we didn't have to you know spend so much time on it that would have been better but again no complaints we're happy that it's well maintained by people that are well meaning and are capable so. What were some of the missing features or issues that you had with the existing redis libraries. So when it's james for my team when he took over the maintainership it was essentially, abandoned i mean i don't know if like abandoned is the right word but it wasn't like being released or maintained or i guess for all intents and purposes abandoned it had no support for redis cluster just like didn't support that async the async version of the lib was fairly broken if i remember correctly oh sorry it had support for for blocking redis cluster and blocking redis and that was fairly okay in the lib not perfect but the async variants for either which is the async variant for the for the non-cluster version was limited and the async cluster version was like an external library by someone else that's also unmaintained. And James merged them together and really streamlined the library and made everything better. Well, that's nice to hear because it benefits a wider network, array of people of potential users um any similar story for any other dependency you mentioned that you have your own open api client i guess but there's drop shot from oxide computer i wonder if if you looked at that or what was the reason for building your own. Yeah so i don't remember what the drop shot is i'll check it out again after this but aid essentially is a tool to automatically generate an open api spec from your axiom code so you define you know like what you define the the api routes what you accept what you send and we kind of like we automatically generate it automatically generates the open api spec from that it was great we looked at multiple options back when we adopted it it was by far the best one and unfortunately it went unmaintained a few months ago or even longer than that and and we kind of like had to take over because you know we had fixes for upstream that just weren't being merged and even you know i don't remember how much how long we waited but we waited a long time for one of the fixes to be merged but then no release was being made so that also was fairly useless in that regard so yeah now we kind of like we were you know we've been merging prs and like making releases and yeah happy about it yeah. Then again these dependencies make sense to be owned by a company like yours because they are so central to what you do they are central to the web ecosystem and you know all of what you described sounds reasonable at least from an outsider's perspective. Yeah and also thanks for. Maintaining by the way. I don't know with pleasure and you know we actually we even had to create a new library so like we created a library for kcuids that just didn't exist we built one actually we also wrote one in python that's fairly popular but we did that we also built or we created the omni queue so essentially you know like our open source project supports a variety of queues in the background so like redis sqs you know just because it needs a queue and i remember like i remember celery from python very fondly it's like a library that just lets you you know it just like abstracts away the queue back end and then people that use your library could just choose their own queue of choice so we kind of like we built the equivalent of that for rust yeah it was also also great now. Let's talk about production rust okay you have that thing in production i have two main questions how does it work on the load and how do you avoid breaking changes when when you might when you maintain a larger application in Rust. We came from Python, and Python, let's say, is not notorious for being fast. And I think it's much better nowadays. But back, I think it was like Python 3.7 or 3.8 when we switched. I think it still has the global interpreter lock, or maybe just the just release a version without. I don't know. But the point is, with Python, you run the way to scale up. Multi-threading, yeah. But multi-process as well, because of the global interpreter lock. So you would run multiple threads and multiple processes. in order to use processes are. Probably fine because those are separate scopes of memory but if you have. Multiple threads. In the same python interpreter then you need the global interpreter log or you have the new interpreter. What's it called yeah yeah yeah yeah but the problem in multi-process yeah it doesn't have the jail limitations that's why they recommend it but it you can share memory you like connection pools are separate like for like to the database like So actually it comes with its own challenges and just not having to worry about any of that, first of all, was fun. It was great. But also we've just seen, and it could be when you rewrite, you probably optimize things a bit and all of that. You know the system better. But we've seen all of Magnitude's improvement in memory usage. Latency was, I mean, just like shot down significantly. And again, we changed some infrastructure as we were doing, so it could be semi-related to that as well. But just every metric improved drastically. I think it was, I don't remember the exact numbers, but I think it was probably 40x or something, the difference between the amount of Python runners that we needed to run compared to the Rust ones. And the thing about that is that it actually compounds. So if you go from, let's even go for the basic case of just like two Rust runners and like 80 Python ones, all of a sudden, like the two Rust runners, they hit in-memory cache like all the time because they're just two of them, right? So let's say 50% of the time, but probably more than 50% because like the moment they flip, it gets to the other one, that one now has it cached. So like in-memory caching is actually extremely effective all of a sudden. We have more than two, but still not way more than two i mean more way more than two but not way more to the point as like not effective anymore and with python we just we couldn't do effective in-memory caching like you know connection database connection pools were just not being reused as much like so, switching to us actually significantly reduced our database load our redis load like all the loads and all of the upstream system you would never think um like you'd really never think that this would be i mean i guess you would think but it really the downstream effects of just being able to run less processes was just you know we're just we're just insane for us so we're very happy about that no. One really thinks about cache locality when you think about reducing the number of nodes you have but it's so vital at a certain point. Yeah i think again like i don't have the exact numbers but like the calls to redis which is like, went essentially to nothing like fairly quickly i mean not fairly immediately when we switch but like essentially for not to nothing so right. And now let's come to the second question which is about avoiding breaking api changes or breaking changes in the api. Yeah so i think you know i have we haven't released it yet we have a we i wrote a blog post exactly about this and what we do there and i'm happy to give a bit of a glimpse so first of all let's start with the interface and then we can talk about the internals the blog post is more about the interface anyway so when you when i mentioned we use this tool called aid and aid generates an open api spec open api by the way it's kind of like a formal definition language for describing http apis. So what aid would do it would take our call let's let's say the api calls create application, it will automatically you know make a json schema out of the ink you know the body that comes in the body that goes out all the path parameters query parameters the path itself authentication requirements all of that will just aid will automatically create that open api spec and we you know we did a lot of extra enrichment on our end to you know to add oh this this field also only accepts a regex expects a regex that like matches this and this integer can only be from 1 to 50 and this is an array that can't be empty like we added like a lot of additional kind of restrictions and the nice thing as well by the way again a small tangent is that the code that enforces it for us is the same as the code that generates the aid description so like the the you know the the size limitation is literally what we enforce if it's in the open api spec it's what we enforce if it's not it's not what we enforce and barring a few exceptions that we purposely broke that but like. Everything is just like, all of this is automatic. And what this means is that we have a JSON file with the exact description of our exact API that we can just generate whenever we want. So what we do, we generate that. We commit it whenever you make a change. We commit it to Git. And then in CI, we generate it again, and then we compare it. And if it's different, CI fails. So essentially what it means is that if you make a change that affects our api you consciously have to update that file like regenerate that files first of all you as the first the developer as the first line of defense you know that you just did something that's potentially breaking and again maybe it's on purpose maybe you added a new api it's fine but you know that and also we have github code owners on that specific spec file that forces people you know like that have extra people that you know that know what we expect from our api like what do we want to release what we don't want to release they will have to review it as well and just say you know what this api changes is reasonable so essentially we kind of like what we did is we leverage the you know the rust type system with some like macro magic and and a few other things to just to just make it so we can't accidentally change our api signatures we have to like we are have to be aware of it like we can't accidentally release an api we don't want to release. We can't just do those kind of things. So that's been great in that regard. How would you even learn that? If you did not know that you could build an OpenAPI spec from your XM handlers, how would you find out that you could use the Rust type system to enforce that and to make testing easier and to make breaking changes explicit? Are there any tips that you could share in order to really hook into the Rust type system and encode the invariance that you depend on. Yeah, so I think this is like a hack that we did. I don't know if, I don't think it's a common practice. Maybe it is, and we're just unaware of it. And that's why, again, we wrote that blog post. I mean, I think it is, I think more people should be doing exactly this. Although, you know what, now that I think about it, I'm sure like large companies have their own tricks as well. But really the key is, I'm a big fan. So some people, what they do, they write the open API spec by hand, and then they generate user generated to generate the backend code i don't think that's good because there's going to be a limit to how well that backend code could be generated so you're going to edit it manually in the end but i think when you generate it from the code and the code is the source of truth which you know let's face it is it is a source which that's what's run in production, i think generating from that like really makes sense so just i think just become obsessed with types and annotations and everything that you want to describe as part of your contract, i think if you're obsessed you know if you're just obsessed about that like everything else will just fall into place yeah i mean i think really that's that's the main thing and i guess related to that is you have to be careful about having too you know too wide of a contract, so if you by default let's say there's like a an age i mean this is like the most uh you know. CS101 example, but like you have like a person's name and age and you let the age be negative by default because, you know, you didn't limit it, you just put an int, then you can like, by default, you exposed a wider contract than what you want to expose. So I think actually being very explicit about annotating correctly, like everything that you actually believe this type should have and should hold, I think is extremely important. And I think that could go a long way. And how far can you take this? I think in your blog post, you also mentioned that your Redis get and set commands are strongly typed. Yeah. And so actually, this is a different blog post, that one. But this is the blog post you're referring to. That's where we use it for internal stability. And it's exactly that. Internally, we use the type system to protect us from a lot of things, like essentially define all the internal contracts as well. So you just mentioned Redis. you know redis lets you store random strings like it doesn't have any typing associated with it but what we did is we created a new type and we kind of we added a new redis interface and and and the way we do it is you have to define a type for the key and that type for the key has to be associated with a specific schema for the value so if you use that key to get a value you know that it's always going to be the correct schema because we parse at the moment we get it and when you write that value same thing you always know it's going to be the same schema because you the only way to write into redis would be using this interface does that make sense yeah. It's a bit like a binary protocol that you talk to with redis so it's it's a spec in between that is the contract between whatever redis expects and whatever your code sends and receives. Exactly yeah we kind of like add this like nice contract between the you know like kind of what i said earlier the messy world out there like untyped redis and and our clean code you. Recently moved to redis streams does that principle still apply do you also use a typed stream of some sort. Yeah so the red redis streams they just accept you know payloads whatever you know you just put whatever you put there we just wrap around that as well with the same schema schema type it. Feels like we both are really big fans of rust's type safety but there are other traits about rust that people often mention for reasons for moving to rust and the things that people usually mention is number one safety slash security number two fearless concurrency number three performance and number four stability or robustness and if you could order these four things by priority what would be number one two and so on for you. Let me first give you a cop-out which is i think the rust typing system is what enables all of those so i just choose the rust typing system and that awesome yeah it just gives us all of that um. Oh you like that. Yeah i think you know so i'm man i don't know i i think fearless concurrency i don't care about as much i mean i i do and we do use it and whatever so like don't yeah we definitely care about it but like i mean compared to like safety and correctness but again it they kind of come together right the fearless concurrency comes from the fact that you can be safe and correct, so i guess like all of my answers will just go back to the type system but let's say, safety correctness would be first performance would be second the fearless concurrency would be last what what's the third one okay so stability will be second performance will be third yeah i think that i think that's and i will add developer experience somewhere in the middle as well like, ergonomics yeah ergonomics exactly um it just yeah the the fact that you just write the code that you want to write and you kind of trust the compiler to do a good job but like the zero zero cost optimizations it's just it's fun yeah yeah it's. Kind of interesting that you rank stability slash robustness higher than raw performance. I think you know like at the end of the day and i know a lot of people are going to be angry at me for saying this you can buy larger servers at the end i mean again i gave the python example you can't buy 100x larger server there is a limit but if you know it's about like 30 improvement you can buy a 30 larger server but like the the safety and like stability is just you can't buy a larger server for that yeah. Okay we've witnessed your journey from the very beginning from the idea to now if you started over with that project, and with everything you know now what advice would you give to yourself back in the day when you started. So i think we did a lot of things well but i think we did a lot well i guess a lot of things as well unwell not as well you know one thing that we did in the beginning and i think like i would bet that every startup founder does the same mistake is we assumed that today even though today we have zero customers by end of next week we're probably going to have facebook load or like google load on our systems and that's you know like we we just we will we will drown and die if we don't fix this load issue now immediately before anything else and we start designing our system to support that load and like yeah this can go from like zero to a billion customers, already and like this is the best system and all of that and and i think we made the same mistake of thinking that we should care about that early on and and we our initial system you know was built to support like a thousand x of the load that was you know where we had even in the you know close horizon so not just like one customer to 1 000 customers more kind of like you know realistically we're not going to get more than like 10 000 customers in the next year so like let's build to like 10 million whatever it just didn't make any sense but we still did it. And by doing that we actually built a much worse system because we were not serving those future customers we were serving the current customers and the current customers cared about like speed of iteration and and you know latency and and you know stability and all of that and you can't explain like it's not a good answer to tell the customers like i know this call is a bit slower than what you expect but it's because we need to support one billion people like you but it's like but you don't like yeah it's not a good answer so nowadays like what we follow is that we want to be able to scale 10x our current loads in a week like or whatever it is a very short period of time we want to see the path to being able to scale you know like one order of magnitude immediately but we don't care about like scaling 100x or 1000x that we assumed we're gonna like have at least like a month or two to figure out and then and that's what we do kind of like that i mean again by the way we don't it doesn't mean that we neuter our existing system we don't build it slow on purpose like we try to build it to the best of our ability but the the what we aim for is no longer that like a thousand x it's just that like 10x with a clear path to like 100x maybe, Does that make sense? Yeah. So your advice to your past self would be focus on the now, focus on the today rather than focus on what will be further down the road. You cross the bridge when you get there and focus on building something that is robust and can scale reasonably well instead of going for hypergrowth, by the way. But what about the rust side? the advice that you would give when you started on this rust journey. I guess one thing is that it's okay to clone you know like one thing that i think we did in the beginning way too much was just to figure out how to have completely cloneless cloneless setup like all the way down the stack and the problem is is that at some point you would have a dependency that requires ownership, and you're either going to just normally, without all of this cloneless setup, you're just going to, would have just passed it the reference, the ownership, like all the way down. And now you kind of like have to clone. So essentially, you're just, it's a premature optimization trying to figure out like the best way to just not clone. And like the previous example, you don't have to be degenerate about it. Like you don't have to just like, oh, I'm just going to clone anywhere. Like no more passing as references. No, definitely pass as reference where it makes sense. But if you have to spend half an hour trying to figure out the signature for this function just to make it so you don't have to clone this fairly small structure, don't worry about it. I think that's... My mileage may vary depending on the exact situation but i think that that's one thing that we would change would. I be wrong in summarizing it with keep it simple even if you write rust. 100 and i think you know one of the we kind of talked about like one of the vices of the rust ecosystem is the obsession with correctness which again is great in most cases but you have to remember that sometimes the world outside is dirty and like you have you can't it's it's useless if you're correct if everything else everyone else is incorrect and you can't communicate with them i think the other one is like extreme obsession with performance i think it's great and healthy and we should continue that that's you know what makes rust so great or part of what makes rust great but we also need to be pragmatic so. We have two tips already we have focus on the 10x not the 1000x and we have focus on simplicity um cloning is fine for example anything else that you would want to add. Yeah so the the other two i mean the first one would be it's very similar to the 10x instead of a thousand x one that i gave earlier which is you know don't aim for 100 uptime, you know it's kind of like in the beginning where we were just obsessed as you said in the beginning as well like we can never go down so we were obsessed with just making sure always up the the thing is though the moment we stopped obsessing with 100 uptime and just gave ourselves an actually attainable goal, which was like five nines of uptime, we actually reached the 100%. Like we got to where we wanted to go before. And the reason for that is, again, similar to the previous point, is when you aim for something that's unattainable, you start doing things that are just crazy, that don't add any value, and actually make things more complex. So an example, I think it's a fair assumption. That we don't need to be up if an asteroid destroys the Earth, right? I think that's a fair assumption. But if we didn't make that assumption, we would maybe send a server to Mars or do something like insane like that, and then we'd have to deal with insane latency and all of that. All of a sudden, we added a lot of complexity for our system for really no good reason because if Earth is destroyed, I don't think anyone cares if we have a bit of downtime. So i think that that is one of those areas that i mean again we didn't have a server on mars just to be clear but like we were making crazy we just like crazy complex systems just to support that 0.00001 percent of i mean just like it didn't make any sense it was actually making us less stable because the complexity made things worse and i guess the other the other like advice i would give my old self is really be diligent about avoiding unnecessary technical debt so i think well i guess so maybe even yeah because i think technical debt is fine technically it's great you take a loan on your own on future self in order to move faster now in order to build something now i don't think that's necessarily a bad thing a technical debt but there are kinds of technical debt that are just unnecessary like naming things poorly in the database for example. If you have a poorly named table and you know that and you're just lazy about changing that in staging or in local development essentially what you did is that everyone from now on till the future of the company you know till the end of the company until 20 years in the future will have to deal with that poor naming because like naming a date renaming a database table is a pain renaming a database column is a pain no one is going to bother with doing that it's not the same is like just like having poor names in the code which again unnecessary but it's fine we can fix it later here it's actually like just an unnecessary you know avoidable piece of debt that i would really encourage people to be extremely diligent about don't let people hand away if it's like hey it's not a big deal just a name no it does matter does. It help to be explicit about the things that you store in a table or how would you find a good name i know it's a bit of a segue but i'm curious now. I think it's really depends on the case but just if if someone or more than one person on the team find the name confusing i think that's warrant to change because people finding something confusing means they're going to go down a rabbit hole in a few months from now when they've debug you know when they're trying to figure out yeah i mean actually okay one quick example that we have we have a few things that are named, like restricted to mean that they actually have more access because they kind of like they have restricted access well in and it's so confusing and i get it wrong because we also have limited and restricted and another name and sometimes we take the the position of the elevated access sometimes we take the position of the data and it's it's really confusing that's that costs us like many many developer hours over the last of the years and again it's it's not a big deal but it is something that's like too big of a pain to change and i wish we were just a bit more thoughtful about it in the beginning would. Privileged access have been a better choice. Yeah so privilege would have been better or just deciding are we talking about the data or are we talking about the access just being consistent about that and always doing that i would have gone a long way and. Finally it's become a bit of a tradition around here to ask that one last question do you have a message to the rust community anything that comes to mind. So first of all message of love i'm a big fan keep up the good work but in terms of a message that drives action i think we got to fix the compilation time i don't know what we can do it's really i think it's it is like a bottleneck for a lot of people it is it is a pain and it's it even affects you know like id like you know language servers and all of that like when it's when things are slow and so we got to fix that thing yeah and i know people are working on it so yeah. That's that's true let's all wish for faster compile times this year and maybe the next couple years it has improved but i i shared a sentiment with you so that wraps it up thanks so much for all the insights and i can tell that If I had production-level webhooks that I wanted to serve anywhere, then I would take a very, very close look at Svix. Good job there, and good luck with the future, and thanks for being a guest. Thank you. Thank you for having me. Rust in Production is a podcast by corrode. It is hosted by me, Matthias Endler, and produced by Simon Brüggen. For show notes, transcripts, and to learn more about how we can help your company make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.