Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: I'm Matthias Endler from corrode, and today's guest is Tom Hacohen from Svix.
Matthias: We talk about reliable webhooks at scale with Rust.
Matthias: Tom, thanks a lot for being a part of this podcast.
Matthias: Can you introduce yourself and Svix, the company you founded?
Tom: It yeah for sure so i'm tom
Tom: the founder and ceo of Svix what we
Tom: do is webhook sending as a service so we
Tom: help companies send webhooks you know people use
Tom: our simple apis and sdks and really with a
Tom: few lines of code they get the state-of-the-art webhook sending system and
Tom: actually i mean now that i think about it we also finally publicly launched
Tom: the receiving aspect as well so we also help you with receiving webhooks so
Tom: you know if you use a vendor that has not so great webhooks i mean preferably
Tom: you just tell them to use svix and you know they can improve the the you know
Tom: improve everything for everyone but if not they can just you can use the receiving
Tom: svix and kind of get a great experience in front of that.
Matthias: What's so special about webhooks isn't it just a fancy link i mean i know how
Matthias: to serve an api and i can probably take a call from someone but why is it so
Matthias: special why should i use a service for that.
Tom: Yeah so i think first of all yeah it's really just a simple HTTP post request.
Tom: You know, that's all you need to do. But I think like everything in product
Tom: and engineering, the devil is into detail.
Tom: And it kind of like when you start moving to production, all of a sudden,
Tom: you know, it needs to scale.
Tom: And when you have multiple customers, you need to make sure that one customer
Tom: having issues is not affecting the other customer.
Tom: So you have like noisy neighbor problems and you have like security because
Tom: webhooks, you know, you kind of mentioned API.
Tom: It's well understood. You have, you know, like a request
Tom: comes in you do some off token checks and
Tom: then you respond but with webhooks you actually make
Tom: calls from your own server to an address controlled by
Tom: your customer slash attacker maybe and they can point it to an internal system
Tom: and you know trick you this is called with server-side request forgery so essentially
Tom: without going too much into the details there's just a lot many more things
Tom: that you need to worry about than you then people that people are not used to
Tom: but additionally you know,
Tom: when it comes to product infrastructure, which is where I place us.
Tom: The best you can do, let's assume you spend a lot of time on it,
Tom: and we spend a lot of time on it, right?
Tom: We're a big team just working on webhooks. If you spend even a fraction of the
Tom: time we spend on it, and you're doing a great job and you're wasting a lot of
Tom: your team's time instead of advancing the product roadmap, and everything just works,
Tom: then your customers aren't happy with you because you haven't built the features that they wanted.
Tom: And if you do anything if you do your the job less
Tom: anything less than stellar then that means uptime a
Tom: downtime sorry and so like you're gonna have issues so really
Tom: i think when it comes to infrastructure like really the best you
Tom: can do even if you do everything perfect is just have a neutral effect on your
Tom: customers your customers won't notice that you spend all of this time on it
Tom: so and this is part of what we help you to do is really just like give your
Tom: customers a state-of-the-art experience they don't think about they don't worry
Tom: about it just works and for you you don't have to maintain it and worry about any of that.
Matthias: What do i get when i use fix other than a endpoint is it also providing some
Matthias: logs monitoring metrics and uptime statistics what do i get from that.
Tom: Yeah so that's a great question because you
Tom: know when it comes to webhooks you know again going back
Tom: to the api request if your customer of yours tries to
Tom: make an api request and that api request
Tom: fails right they would know it fails it failed
Tom: they got an error code back with web hooks they
Tom: don't know that you were just sending a web hook just now
Tom: and it failed for whatever reason they have zero visibility because it's
Tom: generated by the sender so logs and
Tom: and observability are essential there and yes we offer that out of the box and
Tom: we actually have a pre-built ui that our customers can embed in their customers
Tom: fully white labeled sorry embed in their dashboards for their customers and
Tom: then their customers can just go and use that to manage the webhooks, see the observability.
Tom: We have automatic retries, but they can also manually retry once they fix an
Tom: issue, really have full visibility and full control into the whole webhooks sending system.
Matthias: Before you built Svix, there must have been a moment when you thought,
Matthias: wow, this is really challenging to pull off and there is no existing service
Matthias: that solves that problem. Take us back to that time.
Tom: Yeah so it's exactly how it started
Tom: and it was really just by luck so at my previous company
Tom: people were asking us for webhooks i mean we kind of like
Tom: looked into it started designing the system you know
Tom: i did everything to kind of like to plan for
Tom: the build and then realized it's actually so much work and we
Tom: just don't have the capacity to build it at the moment so we
Tom: said no and then people kept on asking us and we kind of like went
Tom: back to the drawing board like okay we can find the time to you know to
Tom: do the initial build but then realized there's going
Tom: to be a lot of maintenance you know ongoing maintenance and like
Tom: you know they're going to be like random requests by customers that we're going
Tom: to have to build otherwise they're going to be unhappy so we said no again and
Tom: then people kept on asking us still and we kind of like okay you
Tom: know we're going to find the time to build it we're going to allocate the resource to maintain it
Tom: but we have this beautiful api great product we're not
Tom: going to have a terrible webhook experience right so we multiply the
Tom: first two estimates by like 3x and 4x and again said no and.
Tom: I i wasn't smart enough to realize at that point that there was a business
Tom: there but it was like a few months later that a
Tom: friend of mine asked me a question about webhooks in like in a
Tom: slack community and i kind of like started explaining to her you should do x and you
Tom: should do y and i realized she couldn't care less about anything i
Tom: was saying she only wanted to send webhooks you don't have
Tom: to think about it and that's kind of like when the you know the
Tom: apple dropped on my head i mean and i realized that
Tom: you know what i had the same problem and and i
Tom: asked her like hey if i build a service that does this will you pay me and she's
Tom: like hell yeah and that's kind of like how it all started and by the way spoiler
Tom: alert she never paid but other people from that slack that did become our first
Tom: customers so that was a win and i'm eternally grateful for for her asking that question.
Matthias: I really like that story because it has a few ingredients which i look for in
Matthias: companies they solve a very narrow focus problem and no one's gonna take away your lunch because,
Matthias: other people don't want to care about that problem it's
Matthias: essentially quote-unquote boring for them someone needs to solve it but then
Matthias: if you look behind it there's a ton of complexity and you need to do it right
Matthias: in order to provide a good service but can you talk about those customers a
Matthias: little bit who were the people that asked for webhooks all along.
Tom: Yeah so those initial few
Tom: were small you know friends of
Tom: mine like it's like a startup slack like small startups one
Tom: was like a jira competitor i guess you would say and and you know they needed
Tom: webhooks because jira has webhooks and customers rely on it to to build those
Tom: integrations you know when you close a ticket they want to run some workflow
Tom: but since then we know we started we've been working with all the way from like
Tom: very small startups so indie developers all the way up to the fortune 50,
Tom: and the use cases are very varied to the point that I actually,
Tom: I kind of like look at us and I draw the parallels to like Twilio for SMS or
Tom: SendGrid for email in the sense that,
Tom: you know people just send emails i you know like sandwich customers just send
Tom: emails i sandwich doesn't care what's in the email and similar to us we just
Tom: see so many different use cases that people just use us for for really whatever kind.
Matthias: Of makes me wonder if you were not really scared of being a bottleneck or a
Matthias: single point of failure for all of these customers.
Tom: Yeah man this is this was at least in the beginning was
Tom: like a dreading thought and you know
Tom: one that literally kept us up at night with page duty and you know
Tom: and everything but but you know
Tom: like a lot of startups they would focus you know it's kind of move fast and break things
Tom: right like you would focus on speed as
Tom: the main feature and similar to other startups fee speed is extremely important
Tom: for us but for us stability has been like the feature that we offer like we
Tom: we have a stellar track record and it's because we spend a lot of time effort
Tom: and money on redundant infrastructure to make sure that we do,
Tom: you know, we are that potential point of failure and we make sure that we don't
Tom: ever go down, knock on wood, obviously, but we spend a lot of time and effort
Tom: there. And to the point that,
Tom: we are more stable than homegrown webhook systems.
Tom: Like we are, you know, we work with some of the best companies out there,
Tom: whether it's like Brex and Benchling and Lob.
Tom: And again, as I said, I have Fortune 50, so we can't mention their names, unfortunately.
Tom: And we are, you know, they trust us and use us because we are stable. We are reliable.
Tom: And again, and we focus all our time on it. So it makes sense.
Matthias: Okay, you said speed matters, especially for startups. And I fully agree with that.
Matthias: But then immediately a lot of people might think is rust the best application
Matthias: it's it's rust the best language to build such a service then if you want to
Matthias: iterate quickly can't you build a quicker prototype in another language.
Tom: So first of all we did build initial version of Svix was actually written in
Tom: python so i guess there you know there are legs to that statement.
Tom: And i think you know rust is problematic for
Tom: fast iteration at the at the beginning of
Tom: the company so like the way i think about it is if
Tom: you're building i mean okay if you're very proficient in rust
Tom: and you know you you move very quickly in rust just do
Tom: it and you just use rust you don't have to worry about it just use
Tom: you know use the tool that you know but at the beginning the
Tom: really the beginning of the company when you don't even know what you're building and you're
Tom: trying to figure it out and really what matters is
Tom: just getting something clunky out there working and i
Tom: think rust with the emphasis on correctness can
Tom: hold you back but i think the moment you have even like
Tom: a hint of traction like you know like even like a few months in and
Tom: you realize that what you're building is generally you
Tom: know again generally not it doesn't have to be precise but
Tom: generally what you're going to build i think then rust is
Tom: extremely powerful because rust you
Tom: know when we were using python we we used
Tom: to write a lot of tests we still write a lot of tests but much
Tom: less and also kind of like have issues
Tom: in staging and sometimes very rarely
Tom: have issues in production as well and with rust it's kind of like you spend
Tom: all of that time in the beginning but the likelihood of errors is significantly
Tom: diminished later on so it's kind of like this it's almost there's like a preservation
Tom: of energy so preservation of development time But with Python.
Tom: You don't spend a lot of energy in the beginning, but you do spend it once it
Tom: hits production, once you need to debug, once you need to refactor.
Tom: And with Rust, you spend more in the beginning, but actually then later on,
Tom: you don't have to spend as much energy. And I think now we move fast.
Matthias: Wasn't that transition painful.
Tom: Oh extremely yeah i
Tom: think you know like as i said like being reliable for
Tom: our customers is the most important thing for us
Tom: so we couldn't do what some
Tom: companies do even even some of our competitors do which is
Tom: you know oh yeah we're going to be down for a few hours on saturday
Tom: we have a maintenance period we can't do anything like
Tom: that so it was all like live transitions and you
Tom: know making sure we don't break it to you know don't break anything for
Tom: anyone like a lot of testing and what's i
Tom: guess challenging about what it is that we do is or
Tom: one of the things is you know we send http requests
Tom: to a variety of servers right i mean like webhooks so that
Tom: that means that we interact with you
Tom: know i well well what's the name of that microsoft one ias iss why ias yeah
Tom: so like ias apache nginx you know all of the wide variety of servers and some
Tom: of them have very specific requirements for how a client should interact with
Tom: them at the moment you know we switch to another client and like,
Tom: like for example in rust we we were facing issues and i think actually a small
Tom: tangent you know like one of the best thing and one of the worst thing about
Tom: the rust ecosystem is the love for perfection and unfortunately the world out
Tom: there is like ugly and dirty and when you interact with that world.
Tom: Sometimes you can't be perfect like yeah you can adhere to the spec perfectly
Tom: but if in our case but if the servers that we send the webhooks to expect something
Tom: that is not required by the spec but i'm sorry that's what i expect you know
Tom: we need to follow that we don't care that they're wrong we have to,
Tom: We have to deliver that message. And so that, yeah, there were just like a variety of challenges.
Tom: It just kind of triggered a bit of PTSD, I guess, from that time.
Matthias: If we go back to the first Python prototype that you built, it somehow also worked, right?
Matthias: It was running in production, but what was the general sentiment like at this point in time?
Matthias: Did you trust in the application? what was the deployment process the icd testing
Matthias: how many issues did you find in production versus development.
Tom: And so on yes so actually
Tom: a bit of again a tangent before like i was when i
Tom: started coding i was i guess pearl in the beginning
Tom: or whatever but i was like my first professional job was
Tom: like a c embedded c engineer and again
Tom: i have the scars to show for like you know like the living as
Tom: a c engineer where like everything can break everything is scary and
Tom: and and a very primitive type system as
Tom: well so one of the things that i was doing
Tom: i remember that as like a very junior engineer is adding a lot of typing as
Tom: much as i can to see with like you know there are like a few tricks you do like
Tom: opaque structs and like opaque type depths with like opaque structs whatever
Tom: like a few a few ways a few tricks to do it um but that experience just made me obsessed with,
Tom: telling the compiler what I really mean and having the compiler complain at
Tom: build time about all the issues so I don't have to think about runtime.
Tom: I know that if it compiles, it works. And by the way, I think that's partially
Tom: why I'm such a big fan of Rust.
Tom: So the reason why I told you all of this is that when we were writing Python
Tom: code, it was already very good with typing.
Tom: We had hackery around SQL Alchemy to make it more type safe.
Tom: And we had even like scaffolding for fast API to make that even better and Pydantic.
Tom: We had like a few places where we're doing, you know, kind of like we're doing
Tom: what we could to make sure that Python has a very rich type system.
Tom: So I think we were already doing like a fairly good job in terms of like catching
Tom: a lot of issues at compile time.
Tom: But the problem with Python is that those type, all the hacks that I mentioned
Tom: were not real, right? It's not really a type.
Tom: It's kind of like, you know, an annotation that we added.
Tom: So we still had issues where we got the annotation wrong.
Tom: And to account for that, like what we did is that we had a lot of unit tests,
Tom: but we also had a lot of end-to-end tests.
Tom: That was really the big thing that we did. Like we had a lot of end-to-end tests.
Tom: And the end-to-end test suite like runs on staging, on real production,
Tom: like against real load balance, everything, just to make sure that whatever
Tom: it is that we do just goes, you know, safely goes into production.
Tom: So we were pretty good. Like, I don't think we had a lot of production issues...
Tom: I mean, definitely no serious ones, but it could be that we added a new API
Tom: and that API maybe had an error when it was just introduced in some scenarios.
Tom: So it was fairly good in that regard. But the problem was when you want to refactor,
Tom: when you change something.
Tom: That is the scariest experience you can have as a Python developer because you
Tom: don't have that assurance of if it compiles, it just works.
Tom: And man, finding those and going... Like one of the most annoying things is
Tom: that like, and this used to be a common occurrence, doesn't happen for us anymore,
Tom: is that like staging would be broken because someone merged something and that
Tom: for whatever reason broke something.
Tom: And then you kind of like, oh, I have another fix. And then like someone else
Tom: brings a fix and someone else brings another change. And all of a sudden like
Tom: unwinding those in staging is like a big pain.
Tom: And that just doesn't happen anymore. So I don't think customers fully noticed
Tom: the experience from a stability standpoint, but we definitely did in terms of
Tom: just like how healthy our staging environment was and how big of a pain it was.
Matthias: The one thing that you triggered in me was also this PTSD that I had when refactoring
Matthias: larger Python codebases, where you end up with a...
Matthias: An exception somewhere really down the
Matthias: call stack in production and you have no idea
Matthias: what's happening you have no idea of the state that the application was
Matthias: in and you trigger that maybe once in a
Matthias: million requests or so because it's a very weird combination
Matthias: of conditions so yeah i can totally relate to that although i was wondering
Matthias: shouldn't the type system or the types that you added to the python code have
Matthias: you know saved you from that wasn't that the idea.
Tom: I think the problem is that the python type system is
Tom: just not expressive enough to really i mean maybe you
Tom: know maybe nowadays it's like slightly better but like enums for
Tom: example i don't think i mean maybe now they exist i don't think so
Tom: and it's still not to the same level but it's like enums and being able to
Tom: catch all of those you know all of the all
Tom: of the variants yeah thank you and another thing
Tom: that you said that triggered me is like exceptions oh my
Tom: god i think that is the billion dollar programming mistake like
Tom: i think rust really got it right in how to
Tom: do error propagation which is
Tom: you don't you can't just like willingly throw random exceptions all
Tom: the way up to the stack like you have to catch them at every
Tom: step and define the you know like properly define
Tom: the um the signature and kind of like the
Tom: contract that you have with the caller um i think is extremely important so
Tom: yes python helped a lot but again
Tom: as i said we missed some places like we didn't get the typing exactly correct
Tom: in some areas we didn't the python was expensive enough to really get everything
Tom: that we wanted in others it was scary it wasn't uh just walking the park what
Tom: we trust and we can cover that later like it really we really utilized the type system to
Tom: just make many runtime errors just like impossible.
Matthias: I don't want to put words into your mouth but this other thing that i really
Matthias: don't like about python deployments is that they are large you have large containers.
Tom: Yeah and.
Matthias: In rust you have small binaries in comparison did you.
Tom: Run into that problem okay so we we were using aws lambda for some things back
Tom: in the days and you would hit i'm pretty sure we hit lambda size limits,
Tom: maybe i'm misremembering no i'm pretty sure we hit lambda size
Tom: limits it's like 150 megabyte but even if it wasn't a hard limit i think it
Tom: was you know those were hard limits but even if they weren't i think it was
Tom: just you'd have performance issues maybe or cost i don't remember it was annoying
Tom: though it was really annoying yeah i mean nothing more to add there it was really
Tom: annoying and like and also because there's a runtime,
Tom: part to it like you would have to wait for for lambda for aws to support python
Tom: 3.10 before you can upgrade so we still like we were like a few versions behind
Tom: it was really like a big pain by having like the runtime being shipped by someone that's not us okay.
Matthias: So we agree python might not be the best choice for what you wanted to build
Matthias: as fix a production grade service for webhooks,
Matthias: but Rust might not have been the only choice that you might have considered
Matthias: for example what about other languages like Elixir or Golang wouldn't that be an option too.
Tom: Yeah so I can only speak to how it was when we made the decision maybe things
Tom: I mean I know some things have changed in Elixir land and maybe a bit in Go
Tom: so Elixir was even more esoteric than Rust back then and
Tom: like choosing an esoteric language like Rust was already a bit of a gamble.
Tom: So I had to, you know, I had to like hedge my bet to an extent.
Tom: Like I couldn't go, and it's not that I wanted to choose Elixir and that's why
Tom: I didn't choose it, but it was just like off the table just because of that.
Tom: And also I had experience with Rust before and I didn't have any experience with Elixir.
Tom: So, you know, the amount of unknowns was much higher, lower.
Tom: Like I built production system in Rust before that.
Tom: Not web, by the way, web was not ready before Sphix. like
Tom: we were really at the cusp of like bus being web
Tom: ready but yeah i've built like production stuff before
Tom: in rust and then go it just
Tom: and again maybe things have improved but
Tom: the type system is so bare and as i said like i'm all in it's kind of felt like
Tom: going almost backwards from like python like and then really i i mean yeah compilation
Tom: speed amazing like i wish we had it in rust like i can we can probably talk
Tom: about it later like i will complain you know for hours at no end,
Tom: about a rough compilation time. But.
Tom: The Go type system was just not expressive enough to really capture everything
Tom: that I thought should be captured.
Tom: And even without that, we actually wrote the first version of the rewrite of
Tom: Sphix in both Go and Rust.
Tom: And I say like first version, like we spent a few days on the Rust one, a few days on the Go one.
Tom: And like someone, I wrote the Rust one and like someone else from the team that
Tom: knew Go very well wrote the Go one.
Tom: And then as a team, we just looked at both and we just decided like Rust just makes much more sense.
Tom: In terms of like again all the areas that
Tom: we cared about like type type safety but also
Tom: just in terms of like how it is to use the
Tom: language and there are like a few things that are done and go that i know go
Tom: go for his love but i i just can't understand like the implicit implicit imports
Tom: drives me mad like i like not implicit just like non-descriptive imports like
Tom: the capitalization for visibility also is a bit annoying reminds me of like Hungarian notation,
Tom: but i think also like the way to do json parsing if i remember correctly that
Tom: was like i think it's done in comments or something i don't remember i remember
Tom: it was just or not comments like a go specific comment thing annotation or something tags labels.
Matthias: Or whatever i guess tags yeah.
Tom: Yeah again i don't remember it's been a while but i it just essentially we kind
Tom: of like we look at the resulting code and we just okay there is no comparison
Tom: here like we can build something on top of Rust, then I don't think we can build
Tom: the same thing on top of Go.
Matthias: Did that person that wrote the Go prototype agree, or did they challenge the decision?
Tom: You know, so I will admit that I'm the CEO of the company, so there's obviously
Tom: like a bit of a power imbalance there, but from what I remember,
Tom: there was like buying from everyone.
Tom: The only concern was like finding Rust talent, which I shared.
Matthias: And...
Matthias: In this prototype was there already some sort of concurrency did you use you
Matthias: know go functions or so or go
Matthias: routines actually to maybe make some things concurrent or or was it linear.
Tom: No no no so we i mean we built a tiny part of the product but we built it like
Tom: we think it would look like in the end so even you know including like some
Tom: weird scaffolding that we would you know normally you wouldn't want in like
Tom: a first version but it's kind of like okay how how would it look like if we
Tom: actually did this scaffolding.
Matthias: When you built this initial prototype you probably compared go concurrency with
Matthias: rust concurrency what what was the verdict there.
Tom: Yeah i don't think there was
Tom: you know like a a strong comparison there
Tom: in the sense of you know it's like benchmarking or
Tom: stuff like that we really we cared a lot about you know the typing
Tom: and developer experience really what swayed us but we did
Tom: use both right i mean go concurrency is kind of like nice in
Tom: the sense you just like it just happens but i
Tom: think i personally but this is a subjective thing i like
Tom: i like the fact that there's like an await syntax i like
Tom: the fact that we color functions by saying you know what this
Tom: one is you know this one
Tom: can run you know in an async context like
Tom: this one has io this one has whatever i think that's a nice thing but this is
Tom: really just a syntax thing i don't think we had a deep you know kind of any
Tom: anything deep there and i i will say though that we and this is like a bit of
Tom: a tangent we there was like a rust library that was doing magical.
Tom: Io in the background so it was like you
Tom: wouldn't the function wouldn't await but in
Tom: the background it would trigger something that would make changes in the server
Tom: and it's kind of just like again in a background thread or something like again
Tom: like magical even though it wasn't like you didn't call a weight on the function
Tom: and i remember that experience just remind me how much i love the fact that like,
Tom: any io functions you know that io functions are you know have to be awaited because you know,
Tom: you know you can easily look at the signature and know that something affects
Tom: the server or if something doesn't
Tom: affect the server is this like a local configuration or is this like a
Tom: a remote configuration i know this is like an esoteric example but it really
Tom: just we spent so much time debugging this because we were so confused by the
Tom: fact that this non-async call was actually you know touching the server no.
Matthias: I can relate to that because a lot of people
Matthias: say function coloring is an issue but it
Matthias: feels like what you allude to
Matthias: is more of more the opposite where you say
Matthias: you want to know if something is async you want to be explicit about it and
Matthias: if you structure your code differently maybe it's not going to be a huge problem
Matthias: because you know that this part of the application has side effects and does
Matthias: ao in an asynchronous way and the other part does not it's.
Tom: Interesting it's kind of like you use the word you know like function coloring
Tom: right and i think that's a great example like you
Tom: know we color we have syntax highlighting right in our code in our code we want
Tom: to highlight things that behave differently than things than other things and
Tom: especially things as important as you know i mean potentially if we don't even
Tom: have like asking away like just blocking operations which is like i don't know
Tom: how anyone can advocate for like,
Tom: not having a way to know what our function is blocking like that yeah just beyond
Tom: me again for blocking operations yeah.
Matthias: Okay so programming other
Matthias: programming languages were ruled out we talked about
Matthias: golang a little bit and elixir was just
Matthias: too cutting edge back in the day so it does
Matthias: make sense to take a closer look at rust but
Matthias: the other option that i had in mind at least maybe you discussed it internally
Matthias: was maybe we just rewrite parts of the application in rust maybe by using language
Matthias: bindings with py03 and then you rewrite parts of the critical stuff and you
Matthias: keep the rest did you consider that option.
Tom: Yeah i mean i think for us the the kind of like two parts that cause latency
Tom: or you know memory usage all of that like the performance either lies in the
Tom: database queue or you know other like background systems that we use,
Tom: or in the json serialization aspect and
Tom: all of that so there isn't really a lot of code for us that we could out so
Tom: there's not like an like an ai operation that you can just like okay i'm gonna
Tom: or you know like a numpy thing like you know some like heavy scientific operation
Tom: you can just do rewrite in c or in rust and we'll just be faster and everything
Tom: else can just stay in python for us,
Tom: the the the wrapping is the code like the
Tom: you know like how quickly we can respond you know how you know
Tom: the latency when we respond to http calls that is something that
Tom: matters to us like how much memory use you know we use when we respond to an
Tom: http call that's what matters so the shell was as important as the code itself
Tom: so we we didn't have anything obvious or anywhere obvious to make that yeah
Tom: and that change especially since by the way like all the serialization costs
Tom: are just as significant so,
Tom: It just didn't make sense. And, you know, actually another thing that comes
Tom: to mind, we talked about, you know, the typing as well.
Tom: That's another place to lose type. Like really there was no,
Tom: we didn't even check that option. Like we just decided to commit.
Matthias: Yeah, because really what you build is a platform. It's sort of runtime for
Matthias: web things like webhooks, for example.
Matthias: And yeah, I see that you also need to handle a lot of JSON. and then if you
Matthias: transform a lot of JSON objects from Python to Rust and back,
Matthias: that also incurs memory overhead.
Matthias: Okay, that means we finally arrived at the final decision to rewrite the application
Matthias: in Rust, but it probably wasn't an easy path.
Matthias: It wasn't an easy migration. Can you talk a little bit about the learnings from
Matthias: migrating from Python to Rust?
Tom: Yeah, I think, you know, like having making
Tom: sure that the database types are the
Tom: same wasn't that hard but still wasn't like
Tom: super easy and and then just you know making sure that the
Tom: ready serialization is identical because this one did it
Tom: slightly different than the other one or maybe we you know
Tom: we messed up something in python it was actually like named you
Tom: know slightly in a weird manner that's inconsistent whether it rust we just
Tom: kind of like did you know camel case everything and then in python we
Tom: had to you know any whatever we had to like create weird aliases
Tom: for like some of the cache stuff but i
Tom: think other than that and other than the difficulties of just like a rewrite
Tom: with zero downtime i think it went fairly smoothly and we're
Tom: super okay first of all i'm a big fan of rust i'll i'll
Tom: you know stand on the roof and shout it to anyone for anyone to hear but we
Tom: did have some areas that were
Tom: unexpected that where rust wasn't perfect actually and there were the areas
Tom: that we expected like compilation i mean compilation app actually was worse
Tom: than what we expected and there were other areas like finding engineers that
Tom: was a bit of a challenge back then nowadays you know there are like a lot of
Tom: rust engineers but there were actual,
Tom: areas where rust or python i think i don't know if better is the right word
Tom: but like there were just like less foot guns i guess i'm happy to elaborate on those if you want.
Matthias: Yeah please in your blog post you mentioned a few things like heap fragmentation uh,
Matthias: and memory efficiency maybe these would be a few things that.
Tom: Yeah i think kind of like you know like if you summarize all of those in like
Tom: in one you know kind of sentence like with great power comes great responsibility
Tom: so kind of like you have more control now and so that means you have to be more
Tom: careful about things the one example that you mentioned was like the heap fragmentation,
Tom: so rust gives you more control there and
Tom: i guess for whatever it just uses the default system allocator
Tom: by default which just didn't work
Tom: very well when it comes to fragmentation so like a long running process
Tom: so heap fragmentation for those who don't know is essentially
Tom: when you allocate big chunks or smaller chunks and then
Tom: you can like every allocation just adds more to the memory
Tom: instead of like reusing those chunks because they don't fit you know let's assume
Tom: you you allocated a chunk of like 90 bytes then you freed it and then you want
Tom: to allocate another chunk of like 95 bytes you can't reuse that one so you're
Tom: going to get another chunk of 95 bytes then maybe you're going to reuse it for
Tom: half of that smaller chunk but then you're not going to be able to you know another 70.
Tom: Byte chunk would again would have to go on top and essentially what happens is
Tom: that your memory just grows and grows and grows even though
Tom: in practice you're actually not using that much
Tom: memory using the same amount of memory but just
Tom: because you allocate and deallocate in different sizes all the time it
Tom: just causes fragmentation this was
Tom: actually fixed by just switching to jmalloc like
Tom: just changing the system allocated to one that you know is very good and really
Tom: did wonders for us but the fact that we had to think about memory layouts and
Tom: investigate like memory usage you know it's kind of like it just gives an example
Tom: of like some of the stuff that you now have to worry about when you start using rust it's.
Matthias: Funny because the rust compiler itself or the Rust ecosystem moved away from
Matthias: Jamalock to the system allocator, but apparently for some use cases that isn't
Matthias: the best choice, the system allocator it is.
Tom: Yeah, for us it definitely wasn't, yeah.
Matthias: But,
Matthias: where would you notice heap fragmentation was it a very occurring a reoccurring
Matthias: pattern that you could have solved with for example a bump allocator or an arena
Matthias: allocator was it expected,
Matthias: load of small objects that you allocated at once and could de-allocate at once.
Tom: Yeah so i think for us i don't you know it's it was also like a year or something
Tom: ago but like i think for us it was a lot of parsing json and the json would
Tom: come from customers and we would send their json for them so that means you
Tom: know because like it's a webhook so that means that that json can have
Tom: you know for you know can be of varying sizes and can be and would normally be very large as well,
Tom: i don't know we probably could have you
Tom: know made so do you use a different allocate or something like that
Tom: but it's just like changing the the default allocator would just
Tom: you know solve everything i think that was probably the main
Tom: thing another thing is you know we use some aws libraries and they
Tom: i i believe they allocate we make http calls and that i believe allocates when
Tom: it kind of creates generates the body that we send and so there were just like
Tom: a few areas where we were allocating large arbitrary sized structures all the time yeah did.
Matthias: You notice a difference in memory usage when you switched the allocator.
Tom: Oh yeah so before that it was i guess you can't see me it's only audio only
Tom: but like you would see you know you'd see like a bump i guess because of some
Tom: you know like high loads that would cause a lot of fragmentation and then that
Tom: bump would just like stay flat and then there would be another bump and then
Tom: that would stay flat as well,
Tom: with jmadoc you immediately see just like,
Tom: First of all, it becomes more jagged instead of flat. You'd see the graph of
Tom: memory usage becomes way more jagged.
Tom: And the low watermark was much lower.
Tom: The baseline was significantly lower.
Tom: We can post the link to that blog post as well in the show notes.
Tom: You'll see there's a few graphs there. You can really see it. It's very obvious.
Matthias: Was memory usage even a concern for you? even if you used more memory would you run out of that.
Tom: We did that's how we found out okay and
Tom: again not immediately but just like every now and then like you
Tom: get an oom and you're like what the hell is going on you know
Tom: we see our usage we like we have 95 buffer
Tom: like what's going on but we then we look at some of those and we
Tom: see that there's like a memory leak but nothing is leaking right
Tom: we did like a run valgrind we ran everything like nothing was leaking and then
Tom: that's kind of how we discovered a few actually more things about like memory
Tom: usage that were surprising again like you'd think russ is much more memory efficient
Tom: than python right i mean like if you ask anyone on reddit or twitter or whatever
Tom: they would they would tell you exactly that,
Tom: but one thing was with 30 json like when you were passing generic json values
Tom: so just you know like essentially again a dict but like a json dict so like
Tom: not a not a string string but more just like generic, again, generic JSON,
Tom: that would, for whatever reason, in serde, that causes like an explosion in memory. And we didn't,
Tom: We had no idea. And I think, you know, like in a language like Python,
Tom: you would think that just the generic case, or, you know, you think,
Tom: I think the generic case just kind of like works, like because they optimize
Tom: for that, they think about that, like, well, in Rust,
Tom: you know, like using an untyped structure, that is such an, you know, such an odd case.
Tom: And we were just like surprised by how bad that was. And maybe this is,
Tom: by the way, was the reason for the.
Tom: For the, well one of the reasons for the you know
Tom: fragmentation that we're talking about i'm not sure and there
Tom: are solutions so you had to know that you don't need you're not supposed to
Tom: use json value you need to use json raw value that
Tom: just like treats the string it doesn't power set but like again in
Tom: a in a world you know in python stuff like that would just happen magically
Tom: in the background you know kind of like copy and why and parse on access and
Tom: all of those kind of things and i think with rust everything is much more rigid
Tom: like what you ask for is what you get and it just like was like a funny gotcha that we that we got does.
Matthias: Python deserialize lazily.
Tom: Probably not but there are things that i'm sure it does like to be efficient
Tom: but maybe it does by the way i really have no idea well.
Matthias: Especially if you share those large json objects then i can see where the efficiency comes from because.
Tom: In rust.
Matthias: You would generally clone unless you put it behind an arc but in python everything
Matthias: is ref counted so you get cheap copies for free.
Tom: That's another like example of where you know
Tom: memory use can just just go out of the window because
Tom: as you said in like in python you don't really care about you know kind of
Tom: like what was it like fearless concurrency is that the yeah
Tom: you don't really care about that like yeah of course space conditions that's
Tom: just a fact of life in python so well i guess maybe
Tom: people just don't use threading as much and so
Tom: they just as you said pass like ref counts all over
Tom: like in rust like you would clone so much data
Tom: if you're not careful like you would accidentally clone that json structure
Tom: like that heavy when you pass it to another function and you can accidentally
Tom: clone something else and it's very easy to accidentally use much more memory
Tom: than you would use in python just because of the you know like the implicit
Tom: ref counting and just like passing of references instead of cloning yeah.
Matthias: I always wanted a lazy json library in rust and probably it exists but i couldn't
Matthias: bother looking it up um but it would be so nice to have a system that does not
Matthias: create 30 json values all the time but it more or less is a view into the data
Matthias: and then it only deserializes what you actually want to look at.
Tom: Yeah i think so kind of the 30 row value kind of semi lets you do that by the
Tom: fact that it doesn't serialize that part and then you can maybe explicitly deserialize that part.
Tom: I mean, again, you have to build the scaffolding yourself, but it is semi-possible.
Tom: Or at least that's what we're doing now.
Matthias: True. And for you, the serde interface is probably, I'm assuming,
Matthias: one of the most important parts of what you use in the Rust ecosystem.
Matthias: What else do you use very heavily? What do you depend on heavily? Which creates?
Tom: So serde would be one. Axum, we use SeaORM as our ORM.
Tom: I mean, mainly for the query building, less for ORM stuff.
Tom: We use redis rust we use
Tom: aid for open telemetry generation i think
Tom: those would be the main ones and actually i mean you
Tom: know like this is a good segue for like ecosystem maturity
Tom: because we you know
Tom: we have the maintainer of one of the maintainers of axiom on the team we have
Tom: one of the maintainers of redis rust on the team we have one of the maintainers
Tom: of aid on the team and it's not you know i wish we didn't right i mean i'm happy
Tom: to have those people they're great and like they're very you know you know they're
Tom: smart and capable but like we we didn't actually have to
Tom: we have to spend time and effort on these you know like core libraries that
Tom: we depend on like we have to spend time and effort on the ecosystem to make
Tom: sure that it's where we need it to be,
Tom: which is a bit i mean i think it's less less so nowadays and i guess if you're
Tom: starting a swix competitor you have us to maintain this for you so like you
Tom: don't have to incur the same cost but but it was a challenge and it definitely is still is but.
Matthias: On the other side you need to have some skin in the game.
Tom: Yeah again no no not complaining really not complaining but you know if we had
Tom: the alternative which would be like having the perfect redis library just existing
Tom: and we didn't have to you know spend so much time on it that would have been
Tom: better but again no complaints we're happy that it's well maintained by people
Tom: that are well meaning and are capable so.
Matthias: What were some of the missing features or issues that you had with the existing redis libraries.
Tom: So when it's james for my team when he took over the maintainership it was essentially,
Tom: abandoned i mean i don't know if like abandoned is the right word but it wasn't like
Tom: being released or maintained or i guess for
Tom: all intents and purposes abandoned it had no
Tom: support for redis cluster just like didn't support that
Tom: async the async version of
Tom: the lib was fairly broken if
Tom: i remember correctly oh sorry it had support for for blocking
Tom: redis cluster and blocking redis and that was fairly okay in the lib not perfect
Tom: but the async variants for either which is the async variant for the for the
Tom: non-cluster version was limited and the async cluster version was like an external
Tom: library by someone else that's also unmaintained.
Tom: And James merged them together and really streamlined the library and made everything better.
Matthias: Well, that's nice to hear because it benefits a wider network,
Matthias: array of people of potential users um any similar story for any other dependency
Matthias: you mentioned that you have your own open api client i guess but there's drop
Matthias: shot from oxide computer i wonder
Matthias: if if you looked at that or what was the reason for building your own.
Tom: Yeah so i don't remember what the drop shot is
Tom: i'll check it out again after this but aid essentially
Tom: is a tool to automatically generate an open api spec
Tom: from your axiom code so you
Tom: define you know like what you define the the
Tom: api routes what you accept what you send and we kind
Tom: of like we automatically generate it automatically generates the open api spec
Tom: from that it was great we looked at multiple
Tom: options back when we adopted it it was
Tom: by far the best one and unfortunately it went unmaintained a
Tom: few months ago or even longer than that and and we kind
Tom: of like had to take over because you know we had fixes for upstream that
Tom: just weren't being merged and even you know i
Tom: don't remember how much how long we waited but we waited a long time for one
Tom: of the fixes to be merged but then no release was being made so that also was
Tom: fairly useless in that regard so yeah now we kind of like we were you know we've
Tom: been merging prs and like making releases and yeah happy about it yeah.
Matthias: Then again these dependencies make sense to be owned by a company like yours
Matthias: because they are so central to what you do they are central to the web ecosystem
Matthias: and you know all of what you described sounds reasonable at least from an outsider's perspective.
Tom: Yeah and also thanks for.
Matthias: Maintaining by the way.
Tom: I don't know with pleasure and you know we actually we even
Tom: had to create a new library so like we created a library
Tom: for kcuids that just didn't
Tom: exist we built one actually we also wrote one
Tom: in python that's fairly popular but we did
Tom: that we also built or we created the omni queue
Tom: so essentially you know like our open source project supports a
Tom: variety of queues in the background so like redis sqs you
Tom: know just because it needs a queue and i remember like
Tom: i remember celery from python very fondly it's like a library that just lets
Tom: you you know it just like abstracts away the queue back end and then people
Tom: that use your library could just choose their own queue of choice so we kind
Tom: of like we built the equivalent of that for rust yeah it was also also great now.
Matthias: Let's talk about production rust okay
Matthias: you have that thing in production i have two main questions how does it work
Matthias: on the load and how do you avoid breaking changes when when you might when you
Matthias: maintain a larger application in Rust.
Tom: We came from Python, and Python, let's say, is not notorious for being fast.
Tom: And I think it's much better nowadays.
Tom: But back, I think it was like Python 3.7 or 3.8 when we switched.
Tom: I think it still has the global interpreter lock, or maybe just the just release
Tom: a version without. I don't know.
Tom: But the point is, with Python, you run the way to scale up.
Matthias: Multi-threading, yeah.
Tom: But multi-process as well, because of the global interpreter lock.
Tom: So you would run multiple threads and multiple processes. in order to use processes are.
Matthias: Probably fine because those are separate scopes of memory but if you have.
Tom: Multiple threads.
Matthias: In the same python interpreter then you need the global interpreter log or you
Matthias: have the new interpreter.
Tom: What's it called yeah yeah yeah yeah but the problem in multi-process yeah it
Tom: doesn't have the jail limitations that's why they recommend it but it you can
Tom: share memory you like connection pools are separate like for like to the database
Tom: like So actually it comes with its own challenges and just not having to worry
Tom: about any of that, first of all, was fun. It was great.
Tom: But also we've just seen, and it could be when you rewrite, you probably optimize
Tom: things a bit and all of that.
Tom: You know the system better. But we've seen all of Magnitude's improvement in memory usage.
Tom: Latency was, I mean, just like shot down significantly.
Tom: And again, we changed some infrastructure as we were doing, so it could be semi-related
Tom: to that as well. But just every metric improved drastically.
Tom: I think it was, I don't remember the exact numbers, but I think it was probably
Tom: 40x or something, the difference between the amount of Python runners that we
Tom: needed to run compared to the Rust ones.
Tom: And the thing about that is that it actually compounds.
Tom: So if you go from, let's even go for the basic case of just like two Rust runners
Tom: and like 80 Python ones, all of a sudden, like the two Rust runners,
Tom: they hit in-memory cache like all the time because they're just two of them, right?
Tom: So let's say 50% of the time, but probably more than 50% because like the moment
Tom: they flip, it gets to the other one, that one now has it cached.
Tom: So like in-memory caching is actually extremely effective all of a sudden.
Tom: We have more than two, but still not way more than two i mean more way more
Tom: than two but not way more to the point as like not effective anymore and with
Tom: python we just we couldn't do effective in-memory caching like you know connection
Tom: database connection pools were just not being reused as much like so,
Tom: switching to us actually significantly reduced our database load
Tom: our redis load like all the
Tom: loads and all of the upstream system you would never think um like you'd really
Tom: never think that this would be i mean i guess you would think but it really
Tom: the downstream effects of just being able to run less processes was just you
Tom: know we're just we're just insane for us so we're very happy about that no.
Matthias: One really thinks about cache locality when you think about reducing the number
Matthias: of nodes you have but it's so vital at a certain point.
Tom: Yeah i think again like i don't have the exact numbers but like the calls to redis which is like,
Tom: went essentially to nothing like fairly quickly i mean not fairly immediately
Tom: when we switch but like essentially for not to nothing so right.
Matthias: And now let's come to the second question which is about avoiding breaking api
Matthias: changes or breaking changes in the api.
Tom: Yeah so i think you know
Tom: i have we haven't released it yet we have a we i wrote a
Tom: blog post exactly about this and what we do there and i'm
Tom: happy to give a bit of a glimpse so first
Tom: of all let's start with the interface and then we can talk about the internals
Tom: the blog post is more about the interface anyway so when you when i mentioned
Tom: we use this tool called aid and aid generates an open api spec open api by the
Tom: way it's kind of like a formal definition language for describing http apis.
Tom: So what aid would do it would take our call let's let's say the api calls create application,
Tom: it will automatically you know make
Tom: a json schema out of the ink you know the body that comes in
Tom: the body that goes out all the path parameters
Tom: query parameters the path itself authentication requirements all
Tom: of that will just aid will automatically create that open
Tom: api spec and we you know we did a lot
Tom: of extra enrichment on our end to you know
Tom: to add oh this this field
Tom: also only accepts a regex expects a
Tom: regex that like matches this and this integer can only be
Tom: from 1 to 50 and this is an array that
Tom: can't be empty like we added like a lot of additional kind of
Tom: restrictions and the nice thing as well by the way again a small tangent is
Tom: that the code that enforces it for us is the same as the code that generates
Tom: the aid description so like the the you know the the size limitation is literally
Tom: what we enforce if it's in the open api spec it's what we enforce if it's not
Tom: it's not what we enforce and barring a few exceptions that we purposely broke that but like.
Tom: Everything is just like, all of this is automatic. And what this means is that
Tom: we have a JSON file with the exact description of our exact API that we can
Tom: just generate whenever we want.
Tom: So what we do, we generate that.
Tom: We commit it whenever you make a change. We commit it to Git.
Tom: And then in CI, we generate it again, and then we compare it.
Tom: And if it's different, CI fails.
Tom: So essentially what it means is that if you make a change that
Tom: affects our api you consciously have to update that
Tom: file like regenerate that files first of all you as the
Tom: first the developer as the first line of defense you know that you
Tom: just did something that's potentially breaking and again maybe it's on
Tom: purpose maybe you added a new api it's fine but you
Tom: know that and also we have github code
Tom: owners on that specific spec file
Tom: that forces people you know like that have extra
Tom: people that you know that know what we expect from
Tom: our api like what do we want to release what we don't want to release they
Tom: will have to review it as well and just say you know
Tom: what this api changes is reasonable so essentially
Tom: we kind of like what we did is we leverage the you know the rust type system
Tom: with some like macro magic and and a few other things to just to just make it
Tom: so we can't accidentally change our api signatures we have to like we are have
Tom: to be aware of it like we can't accidentally release an api we don't want to release.
Tom: We can't just do those kind of things. So that's been great in that regard.
Matthias: How would you even learn that? If you did not know that you could build an OpenAPI
Matthias: spec from your XM handlers,
Matthias: how would you find out that you could use the Rust type system to enforce that
Matthias: and to make testing easier and to make breaking changes explicit?
Matthias: Are there any tips that you could share in order to really hook into the Rust
Matthias: type system and encode the invariance that you depend on.
Tom: Yeah, so I think this is like a hack that we did. I don't know if,
Tom: I don't think it's a common practice. Maybe it is, and we're just unaware of it.
Tom: And that's why, again, we wrote that blog post. I mean, I think it is,
Tom: I think more people should be doing exactly this.
Tom: Although, you know what, now that I think about it, I'm sure like large companies
Tom: have their own tricks as well.
Tom: But really the key is, I'm a big fan. So some people, what they do,
Tom: they write the open API spec by hand, and then
Tom: they generate user generated to generate the backend code i don't think that's
Tom: good because there's going to be a limit to how well that backend code could
Tom: be generated so you're going to edit it manually in the end but i think when
Tom: you generate it from the code and the code is the source of truth which you
Tom: know let's face it is it is a source which that's what's run in production,
Tom: i think generating from that like really makes sense so just i think just become
Tom: obsessed with types and annotations and everything that you want to describe as part of your contract,
Tom: i think if you're obsessed you know if you're just obsessed about that like
Tom: everything else will just fall into place yeah i mean i think really that's
Tom: that's the main thing and i guess related to that is you have to be careful
Tom: about having too you know too wide of a contract,
Tom: so if you by default let's say there's like a an age i mean this is like the most uh you know.
Tom: CS101 example, but like you have like a person's name and age and you let the
Tom: age be negative by default because, you know, you didn't limit it, you just put an int,
Tom: then you can like, by default, you exposed a wider contract than what you want to expose.
Tom: So I think actually being very explicit about annotating correctly,
Tom: like everything that you actually believe this type should have and should hold,
Tom: I think is extremely important. And I think that could go a long way.
Matthias: And how far can you take this? I think in your blog post, you also mentioned
Matthias: that your Redis get and set commands are strongly typed.
Tom: Yeah. And so actually, this is a different blog post, that one.
Tom: But this is the blog post you're referring to. That's where we use it for internal
Tom: stability. And it's exactly that.
Tom: Internally, we use the type system to protect us from a lot of things,
Tom: like essentially define all the internal contracts as well.
Tom: So you just mentioned Redis. you know redis lets you
Tom: store random strings like it doesn't have any typing associated with
Tom: it but what we did is we
Tom: created a new type and we kind of we added a new redis
Tom: interface and and and the
Tom: way we do it is you have to define a type for the key and that
Tom: type for the key has to be associated with a specific schema
Tom: for the value so if you use that key to get a value you know that it's always
Tom: going to be the correct schema because we parse at the moment we get it and
Tom: when you write that value same thing you always know it's going to be the same
Tom: schema because you the only way to write into redis would be using this interface
Tom: does that make sense yeah.
Matthias: It's a bit like a binary protocol that you talk to with redis so it's it's a
Matthias: spec in between that is the contract between whatever redis expects and whatever
Matthias: your code sends and receives.
Tom: Exactly yeah we kind of like add this like nice contract between the you know
Tom: like kind of what i said earlier the messy world out there like untyped redis
Tom: and and our clean code you.
Matthias: Recently moved to redis streams does that principle still apply do you also
Matthias: use a typed stream of some sort.
Tom: Yeah so the red redis streams they just accept you know payloads whatever you
Tom: know you just put whatever you put there we just wrap around that as well with
Tom: the same schema schema type it.
Matthias: Feels like we both are really big fans of rust's
Matthias: type safety but there are other traits
Matthias: about rust that people often mention for
Matthias: reasons for moving to rust and the
Matthias: things that people usually mention is number one
Matthias: safety slash security number two fearless concurrency number three performance
Matthias: and number four stability or robustness and if you could order these four things
Matthias: by priority what would be number one two and so on for you.
Tom: Let me first give you a cop-out which is i think the rust typing system is what
Tom: enables all of those so i just choose the rust typing system and that awesome
Tom: yeah it just gives us all of that um.
Matthias: Oh you like that.
Tom: Yeah i think you know
Tom: so i'm man i don't know i i think
Tom: fearless concurrency i don't care about as much i mean i i do and we do use
Tom: it and whatever so like don't yeah we definitely care about it but like i mean
Tom: compared to like safety and correctness but again it they kind of come together
Tom: right the fearless concurrency comes from the fact that you can be safe and correct,
Tom: so i guess like all of my answers will just go back to the type system but let's say,
Tom: safety correctness would be first performance would be second the fearless concurrency
Tom: would be last what what's the third one okay so stability will be second performance
Tom: will be third yeah i think that i think that's and i will add developer experience
Tom: somewhere in the middle as well like,
Tom: ergonomics yeah ergonomics exactly um it just yeah the the fact that you just
Tom: write the code that you want to write and you kind of trust the compiler to
Tom: do a good job but like the zero zero cost optimizations it's just it's fun yeah yeah it's.
Matthias: Kind of interesting that you rank stability slash robustness higher than raw performance.
Tom: I think you know like at the end of the day and
Tom: i know a lot of people are going to be angry at me for saying this you can buy larger
Tom: servers at the end i mean again i gave the python example you can't buy 100x
Tom: larger server there is a limit but if you know it's about like 30 improvement
Tom: you can buy a 30 larger server but like the the safety and like stability is
Tom: just you can't buy a larger server for that yeah.
Matthias: Okay we've witnessed your journey from the very beginning from the idea to now
Matthias: if you started over with that project,
Matthias: and with everything you know now what advice would you give to yourself back
Matthias: in the day when you started.
Tom: So i think
Tom: we did a lot of things well but i think we did a lot well i
Tom: guess a lot of things as well unwell not as well you know
Tom: one thing that we did in the beginning and i
Tom: think like i would bet that every startup founder does
Tom: the same mistake is we assumed that today
Tom: even though today we have zero customers by end
Tom: of next week we're probably going to have facebook load or
Tom: like google load on our systems and that's you know like we we just we will
Tom: we will drown and die if we don't fix this load issue now immediately before
Tom: anything else and we start designing our system to support that load and like
Tom: yeah this can go from like zero to a billion customers,
Tom: already and like this is the best system and all of that and and
Tom: i think we made the same mistake of thinking that we
Tom: should care about that early on and and
Tom: we our initial system you know was built to support like a thousand x of the
Tom: load that was you know where we had even in the you know close horizon so not
Tom: just like one customer to 1 000 customers more kind of like you know realistically
Tom: we're not going to get more than like 10 000 customers in the next year so like
Tom: let's build to like 10 million whatever it just didn't make any sense but we still did it.
Tom: And by doing that we actually built a much
Tom: worse system because we were not serving
Tom: those future customers we were serving the
Tom: current customers and the current customers cared about like speed
Tom: of iteration and and you know latency and and
Tom: you know stability and all of that and you can't explain like it's not
Tom: a good answer to tell the customers like i know this call is a bit slower
Tom: than what you expect but it's because we need to support one
Tom: billion people like you but it's like but you don't like yeah
Tom: it's not a good answer so nowadays like
Tom: what we follow is that we want to be able
Tom: to scale 10x our current loads in
Tom: a week like or whatever it is a very short period of time we want
Tom: to see the path to being able to scale you know
Tom: like one order of magnitude immediately but we
Tom: don't care about like scaling 100x or 1000x that we
Tom: assumed we're gonna like have at least like a month
Tom: or two to figure out and then and that's
Tom: what we do kind of like that i mean again by the way we don't it doesn't mean
Tom: that we neuter our existing system we don't build it slow on purpose like we
Tom: try to build it to the best of our ability but the the what we aim for is no
Tom: longer that like a thousand x it's just that like 10x with a clear path to like 100x maybe,
Tom: Does that make sense?
Matthias: Yeah. So your advice to your past self would be focus on the now,
Matthias: focus on the today rather than focus on what will be further down the road.
Matthias: You cross the bridge when you get there and focus on building something that
Matthias: is robust and can scale reasonably well instead of going for hypergrowth, by the way.
Matthias: But what about the rust side? the advice that you would give when you started on this rust journey.
Tom: I guess one thing is that it's
Tom: okay to clone you know like one thing that i think we did in the beginning way
Tom: too much was just to figure out how to have completely cloneless cloneless setup
Tom: like all the way down the stack and the problem is is that at some point you
Tom: would have a dependency that requires ownership,
Tom: and you're either going to just normally, without all of this cloneless setup,
Tom: you're just going to, would have just passed it the reference,
Tom: the ownership, like all the way down.
Tom: And now you kind of like have to clone. So essentially, you're just,
Tom: it's a premature optimization trying to figure out like the best way to just not clone.
Tom: And like the previous example, you don't have to be degenerate about it.
Tom: Like you don't have to just like, oh, I'm just going to clone anywhere.
Tom: Like no more passing as references.
Tom: No, definitely pass as reference where it makes sense.
Tom: But if you have to spend half an hour trying to figure out the signature for
Tom: this function just to make it so you don't have to clone this fairly small structure,
Tom: don't worry about it. I think that's...
Tom: My mileage may vary depending on the exact situation but i think that that's
Tom: one thing that we would change would.
Matthias: I be wrong in summarizing it with keep it simple even if you write rust.
Tom: 100 and i think you know
Tom: one of the we kind of talked about like one
Tom: of the vices of the rust ecosystem is the obsession with
Tom: correctness which again is great in most cases but
Tom: you have to remember that sometimes the world outside is
Tom: dirty and like you have you can't it's it's useless
Tom: if you're correct if everything else everyone else is incorrect and you can't
Tom: communicate with them i think the other one is like extreme obsession with performance
Tom: i think it's great and healthy and we should continue that that's you know what
Tom: makes rust so great or part of what makes rust great but we also need to be pragmatic so.
Matthias: We have two tips already we have focus on the 10x not the 1000x and we have
Matthias: focus on simplicity um cloning is fine for example anything else that you would want to add.
Tom: Yeah so the the other two i mean the first one would be it's very similar to
Tom: the 10x instead of a thousand x one that i gave earlier which is you know don't aim for 100 uptime,
Tom: you know it's kind of like in the beginning where we were just obsessed as you
Tom: said in the beginning as well like we can never go down so we were obsessed
Tom: with just making sure always up the the thing is though the moment we stopped
Tom: obsessing with 100 uptime and just gave ourselves an actually attainable goal,
Tom: which was like five nines of uptime, we actually reached the 100%.
Tom: Like we got to where we wanted to go before.
Tom: And the reason for that is, again, similar to the previous point,
Tom: is when you aim for something that's unattainable, you start doing things that
Tom: are just crazy, that don't add any value, and actually make things more complex.
Tom: So an example, I think it's a fair assumption.
Tom: That we don't need to be up if an asteroid destroys the Earth,
Tom: right? I think that's a fair assumption.
Tom: But if we didn't make that assumption, we would maybe send a server to Mars
Tom: or do something like insane like that, and then we'd have to deal with insane latency and all of that.
Tom: All of a sudden, we added a lot of complexity for our system for really no good
Tom: reason because if Earth is destroyed, I don't think anyone cares if we have a bit of downtime.
Tom: So i think that that is
Tom: one of those areas that i mean again we didn't
Tom: have a server on mars just to be clear but like we were making
Tom: crazy we just like crazy
Tom: complex systems just to support that 0.00001
Tom: percent of i mean just like it didn't make any sense it was
Tom: actually making us less stable because the complexity made things
Tom: worse and i guess the other the other
Tom: like advice i would give my old self is
Tom: really be diligent about avoiding unnecessary
Tom: technical debt so i think well i guess so maybe even yeah
Tom: because i think technical debt is fine technically it's great you take a loan
Tom: on your own on future self in order to move faster now in order to build something
Tom: now i don't think that's necessarily a bad thing a technical debt but there
Tom: are kinds of technical debt that are just unnecessary like naming things poorly
Tom: in the database for example.
Tom: If you have a poorly named table and
Tom: you know that and you're just lazy about changing that in staging or
Tom: in local development essentially what you did
Tom: is that everyone from now on till the future
Tom: of the company you know till the end of the company until 20 years
Tom: in the future will have to deal with
Tom: that poor naming because like naming a date renaming a database table
Tom: is a pain renaming a database column is a pain no one
Tom: is going to bother with doing that it's not the same is
Tom: like just like having poor names in the code which again unnecessary but it's
Tom: fine we can fix it later here it's actually like just an unnecessary you know
Tom: avoidable piece of debt that i would really encourage people to be extremely
Tom: diligent about don't let people hand away if it's like hey it's not a big deal
Tom: just a name no it does matter does.
Matthias: It help to be explicit about the things that you store in a table or how would
Matthias: you find a good name i know it's a bit of a segue but i'm curious now.
Tom: I think it's really depends on the case but
Tom: just if if someone or more
Tom: than one person on the team find the name confusing i think that's warrant to
Tom: change because people finding something confusing means they're going to go
Tom: down a rabbit hole in a few months from now when they've debug you know when
Tom: they're trying to figure out yeah i mean actually okay one quick example that
Tom: we have we have a few things that are named,
Tom: like restricted to mean that they
Tom: actually have more access because they kind
Tom: of like they have restricted access well in and
Tom: it's so confusing and i get it wrong because we also have limited and
Tom: restricted and another name and sometimes we
Tom: take the the position of the elevated access sometimes we take the position
Tom: of the data and it's it's really confusing that's that costs us like many many
Tom: developer hours over the last of the years and again it's it's not a big deal
Tom: but it is something that's like too big of a pain to change and i wish we were
Tom: just a bit more thoughtful about it in the beginning would.
Matthias: Privileged access have been a better choice.
Tom: Yeah so privilege would have been better or just deciding are we talking about
Tom: the data or are we talking about the access just being consistent about that
Tom: and always doing that i would have gone a long way and.
Matthias: Finally it's become a bit of a tradition around here to ask that one last question
Matthias: do you have a message to the rust community anything that comes to mind.
Tom: So first of all message of love i'm a big fan
Tom: keep up the good work but in terms of a
Tom: message that drives action i think
Tom: we got to fix the compilation time i don't know what we can do
Tom: it's really i think
Tom: it's it is like a bottleneck for a lot of people it is it is a pain and it's
Tom: it even affects you know like id like you know language servers and all of that
Tom: like when it's when things are slow and so we got to fix that thing yeah and
Tom: i know people are working on it so yeah.
Matthias: That's that's true let's all wish for faster compile
Matthias: times this year and maybe the next couple years it has improved but i i shared
Matthias: a sentiment with you so that wraps it up thanks so much for all the insights
Matthias: and i can tell that If I had production-level webhooks that I wanted to serve anywhere,
Matthias: then I would take a very, very close look at Svix.
Matthias: Good job there, and good luck with the future, and thanks for being a guest.
Tom: Thank you. Thank you for having me.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.