WEBVTT 00:00:01.450 --> 00:00:06.090 It's Rust in Production, a podcast about companies who use Rust to shape the 00:00:06.090 --> 00:00:07.030 future of infrastructure. 00:00:07.310 --> 00:00:11.910 I'm Matthias Endler from corrode, and today's guest is Tom Hacohen from Svix. 00:00:12.110 --> 00:00:15.410 We talk about reliable webhooks at scale with Rust. 00:00:19.090 --> 00:00:23.270 Tom, thanks a lot for being a part of this podcast. 00:00:23.730 --> 00:00:27.950 Can you introduce yourself and Svix, the company you founded? 00:00:27.950 --> 00:00:30.790 It yeah for sure so i'm tom 00:00:30.790 --> 00:00:33.510 the founder and ceo of Svix what we 00:00:33.510 --> 00:00:36.250 do is webhook sending as a service so we 00:00:36.250 --> 00:00:39.130 help companies send webhooks you know people use 00:00:39.130 --> 00:00:42.030 our simple apis and sdks and really with a 00:00:42.030 --> 00:00:45.050 few lines of code they get the state-of-the-art webhook sending system and 00:00:45.050 --> 00:00:47.950 actually i mean now that i think about it we also finally publicly launched 00:00:47.950 --> 00:00:52.410 the receiving aspect as well so we also help you with receiving webhooks so 00:00:52.410 --> 00:00:57.990 you know if you use a vendor that has not so great webhooks i mean preferably 00:00:57.990 --> 00:01:01.170 you just tell them to use svix and you know they can improve the the you know 00:01:01.170 --> 00:01:05.390 improve everything for everyone but if not they can just you can use the receiving 00:01:05.390 --> 00:01:07.850 svix and kind of get a great experience in front of that. 00:01:07.850 --> 00:01:12.810 What's so special about webhooks isn't it just a fancy link i mean i know how 00:01:12.810 --> 00:01:19.130 to serve an api and i can probably take a call from someone but why is it so 00:01:19.130 --> 00:01:20.810 special why should i use a service for that. 00:01:20.810 --> 00:01:26.230 Yeah so i think first of all yeah it's really just a simple HTTP post request. 00:01:26.490 --> 00:01:30.610 You know, that's all you need to do. But I think like everything in product 00:01:30.610 --> 00:01:33.330 and engineering, the devil is into detail. 00:01:33.490 --> 00:01:36.290 And it kind of like when you start moving to production, all of a sudden, 00:01:36.530 --> 00:01:37.670 you know, it needs to scale. 00:01:37.930 --> 00:01:41.670 And when you have multiple customers, you need to make sure that one customer 00:01:41.670 --> 00:01:43.790 having issues is not affecting the other customer. 00:01:43.970 --> 00:01:46.950 So you have like noisy neighbor problems and you have like security because 00:01:46.950 --> 00:01:48.830 webhooks, you know, you kind of mentioned API. 00:01:49.730 --> 00:01:52.750 It's well understood. You have, you know, like a request 00:01:52.750 --> 00:01:55.450 comes in you do some off token checks and 00:01:55.450 --> 00:01:58.290 then you respond but with webhooks you actually make 00:01:58.290 --> 00:02:01.930 calls from your own server to an address controlled by 00:02:01.930 --> 00:02:06.570 your customer slash attacker maybe and they can point it to an internal system 00:02:06.570 --> 00:02:10.050 and you know trick you this is called with server-side request forgery so essentially 00:02:10.050 --> 00:02:13.690 without going too much into the details there's just a lot many more things 00:02:13.690 --> 00:02:17.250 that you need to worry about than you then people that people are not used to 00:02:17.250 --> 00:02:18.510 but additionally you know, 00:02:19.170 --> 00:02:23.250 when it comes to product infrastructure, which is where I place us. 00:02:24.760 --> 00:02:28.300 The best you can do, let's assume you spend a lot of time on it, 00:02:28.560 --> 00:02:30.020 and we spend a lot of time on it, right? 00:02:30.120 --> 00:02:34.660 We're a big team just working on webhooks. If you spend even a fraction of the 00:02:34.660 --> 00:02:37.840 time we spend on it, and you're doing a great job and you're wasting a lot of 00:02:37.840 --> 00:02:42.660 your team's time instead of advancing the product roadmap, and everything just works, 00:02:43.100 --> 00:02:48.260 then your customers aren't happy with you because you haven't built the features that they wanted. 00:02:48.700 --> 00:02:51.640 And if you do anything if you do your the job less 00:02:51.640 --> 00:02:54.820 anything less than stellar then that means uptime a 00:02:54.820 --> 00:02:57.940 downtime sorry and so like you're gonna have issues so really 00:02:57.940 --> 00:03:00.780 i think when it comes to infrastructure like really the best you 00:03:00.780 --> 00:03:04.780 can do even if you do everything perfect is just have a neutral effect on your 00:03:04.780 --> 00:03:07.580 customers your customers won't notice that you spend all of this time on it 00:03:07.580 --> 00:03:11.720 so and this is part of what we help you to do is really just like give your 00:03:11.720 --> 00:03:16.080 customers a state-of-the-art experience they don't think about they don't worry 00:03:16.080 --> 00:03:20.100 about it just works and for you you don't have to maintain it and worry about any of that. 00:03:20.100 --> 00:03:28.160 What do i get when i use fix other than a endpoint is it also providing some 00:03:28.160 --> 00:03:33.380 logs monitoring metrics and uptime statistics what do i get from that. 00:03:33.380 --> 00:03:36.680 Yeah so that's a great question because you 00:03:36.680 --> 00:03:39.440 know when it comes to webhooks you know again going back 00:03:39.440 --> 00:03:42.460 to the api request if your customer of yours tries to 00:03:42.460 --> 00:03:45.340 make an api request and that api request 00:03:45.340 --> 00:03:48.540 fails right they would know it fails it failed 00:03:48.540 --> 00:03:51.840 they got an error code back with web hooks they 00:03:51.840 --> 00:03:54.720 don't know that you were just sending a web hook just now 00:03:54.720 --> 00:03:57.660 and it failed for whatever reason they have zero visibility because it's 00:03:57.660 --> 00:04:00.940 generated by the sender so logs and 00:04:00.940 --> 00:04:05.500 and observability are essential there and yes we offer that out of the box and 00:04:05.500 --> 00:04:09.120 we actually have a pre-built ui that our customers can embed in their customers 00:04:09.120 --> 00:04:14.580 fully white labeled sorry embed in their dashboards for their customers and 00:04:14.580 --> 00:04:18.780 then their customers can just go and use that to manage the webhooks, see the observability. 00:04:19.500 --> 00:04:23.220 We have automatic retries, but they can also manually retry once they fix an 00:04:23.220 --> 00:04:28.340 issue, really have full visibility and full control into the whole webhooks sending system. 00:04:29.460 --> 00:04:32.200 Before you built Svix, there must have been a moment when you thought, 00:04:32.380 --> 00:04:36.740 wow, this is really challenging to pull off and there is no existing service 00:04:36.740 --> 00:04:39.800 that solves that problem. Take us back to that time. 00:04:41.080 --> 00:04:43.840 Yeah so it's exactly how it started 00:04:43.840 --> 00:04:46.740 and it was really just by luck so at my previous company 00:04:46.740 --> 00:04:49.460 people were asking us for webhooks i mean we kind of like 00:04:49.460 --> 00:04:52.320 looked into it started designing the system you know 00:04:52.320 --> 00:04:55.180 i did everything to kind of like to plan for 00:04:55.180 --> 00:04:57.960 the build and then realized it's actually so much work and we 00:04:57.960 --> 00:05:00.660 just don't have the capacity to build it at the moment so we 00:05:00.660 --> 00:05:03.380 said no and then people kept on asking us and we kind of like went 00:05:03.380 --> 00:05:06.060 back to the drawing board like okay we can find the time to you know to 00:05:06.060 --> 00:05:08.660 do the initial build but then realized there's going 00:05:08.660 --> 00:05:11.460 to be a lot of maintenance you know ongoing maintenance and like 00:05:11.460 --> 00:05:14.080 you know they're going to be like random requests by customers that we're going 00:05:14.080 --> 00:05:17.420 to have to build otherwise they're going to be unhappy so we said no again and 00:05:17.420 --> 00:05:20.280 then people kept on asking us still and we kind of like okay you 00:05:20.280 --> 00:05:23.320 know we're going to find the time to build it we're going to allocate the resource to maintain it 00:05:23.320 --> 00:05:26.080 but we have this beautiful api great product we're not 00:05:26.080 --> 00:05:29.140 going to have a terrible webhook experience right so we multiply the 00:05:29.140 --> 00:05:33.060 first two estimates by like 3x and 4x and again said no and. 00:05:33.060 --> 00:05:35.820 I i wasn't smart enough to realize at that point that there was a business 00:05:35.820 --> 00:05:38.560 there but it was like a few months later that a 00:05:38.560 --> 00:05:41.200 friend of mine asked me a question about webhooks in like in a 00:05:41.200 --> 00:05:43.920 slack community and i kind of like started explaining to her you should do x and you 00:05:43.920 --> 00:05:46.940 should do y and i realized she couldn't care less about anything i 00:05:46.940 --> 00:05:49.600 was saying she only wanted to send webhooks you don't have 00:05:49.600 --> 00:05:52.280 to think about it and that's kind of like when the you know the 00:05:52.280 --> 00:05:55.020 apple dropped on my head i mean and i realized that 00:05:55.020 --> 00:05:58.260 you know what i had the same problem and and i 00:05:58.260 --> 00:06:02.820 asked her like hey if i build a service that does this will you pay me and she's 00:06:02.820 --> 00:06:06.600 like hell yeah and that's kind of like how it all started and by the way spoiler 00:06:06.600 --> 00:06:11.480 alert she never paid but other people from that slack that did become our first 00:06:11.480 --> 00:06:15.740 customers so that was a win and i'm eternally grateful for for her asking that question. 00:06:16.380 --> 00:06:21.580 I really like that story because it has a few ingredients which i look for in 00:06:21.580 --> 00:06:29.600 companies they solve a very narrow focus problem and no one's gonna take away your lunch because, 00:06:30.240 --> 00:06:33.280 other people don't want to care about that problem it's 00:06:33.280 --> 00:06:38.180 essentially quote-unquote boring for them someone needs to solve it but then 00:06:38.180 --> 00:06:43.160 if you look behind it there's a ton of complexity and you need to do it right 00:06:43.160 --> 00:06:48.340 in order to provide a good service but can you talk about those customers a 00:06:48.340 --> 00:06:51.880 little bit who were the people that asked for webhooks all along. 00:06:51.880 --> 00:06:54.980 Yeah so those initial few 00:06:54.980 --> 00:06:57.700 were small you know friends of 00:06:57.700 --> 00:07:01.160 mine like it's like a startup slack like small startups one 00:07:01.160 --> 00:07:05.700 was like a jira competitor i guess you would say and and you know they needed 00:07:05.700 --> 00:07:09.220 webhooks because jira has webhooks and customers rely on it to to build those 00:07:09.220 --> 00:07:12.660 integrations you know when you close a ticket they want to run some workflow 00:07:12.660 --> 00:07:16.660 but since then we know we started we've been working with all the way from like 00:07:16.660 --> 00:07:19.900 very small startups so indie developers all the way up to the fortune 50, 00:07:21.160 --> 00:07:25.120 and the use cases are very varied to the point that I actually, 00:07:25.140 --> 00:07:29.640 I kind of like look at us and I draw the parallels to like Twilio for SMS or 00:07:29.640 --> 00:07:32.360 SendGrid for email in the sense that, 00:07:33.070 --> 00:07:36.290 you know people just send emails i you know like sandwich customers just send 00:07:36.290 --> 00:07:39.850 emails i sandwich doesn't care what's in the email and similar to us we just 00:07:39.850 --> 00:07:46.550 see so many different use cases that people just use us for for really whatever kind. 00:07:46.550 --> 00:07:52.830 Of makes me wonder if you were not really scared of being a bottleneck or a 00:07:52.830 --> 00:07:55.390 single point of failure for all of these customers. 00:07:56.170 --> 00:07:59.110 Yeah man this is this was at least in the beginning was 00:07:59.110 --> 00:08:01.830 like a dreading thought and you know 00:08:01.830 --> 00:08:04.550 one that literally kept us up at night with page duty and you know 00:08:04.550 --> 00:08:07.210 and everything but but you know 00:08:07.210 --> 00:08:10.150 like a lot of startups they would focus you know it's kind of move fast and break things 00:08:10.150 --> 00:08:13.570 right like you would focus on speed as 00:08:13.570 --> 00:08:17.490 the main feature and similar to other startups fee speed is extremely important 00:08:17.490 --> 00:08:23.330 for us but for us stability has been like the feature that we offer like we 00:08:23.330 --> 00:08:28.450 we have a stellar track record and it's because we spend a lot of time effort 00:08:28.450 --> 00:08:31.810 and money on redundant infrastructure to make sure that we do, 00:08:32.170 --> 00:08:36.810 you know, we are that potential point of failure and we make sure that we don't 00:08:36.810 --> 00:08:40.750 ever go down, knock on wood, obviously, but we spend a lot of time and effort 00:08:40.750 --> 00:08:42.450 there. And to the point that, 00:08:43.250 --> 00:08:46.470 we are more stable than homegrown webhook systems. 00:08:46.770 --> 00:08:49.630 Like we are, you know, we work with some of the best companies out there, 00:08:49.750 --> 00:08:51.550 whether it's like Brex and Benchling and Lob. 00:08:52.610 --> 00:08:55.930 And again, as I said, I have Fortune 50, so we can't mention their names, unfortunately. 00:08:56.570 --> 00:09:01.290 And we are, you know, they trust us and use us because we are stable. We are reliable. 00:09:01.730 --> 00:09:04.610 And again, and we focus all our time on it. So it makes sense. 00:09:05.210 --> 00:09:10.090 Okay, you said speed matters, especially for startups. And I fully agree with that. 00:09:10.630 --> 00:09:17.230 But then immediately a lot of people might think is rust the best application 00:09:17.230 --> 00:09:21.210 it's it's rust the best language to build such a service then if you want to 00:09:21.210 --> 00:09:24.610 iterate quickly can't you build a quicker prototype in another language. 00:09:24.610 --> 00:09:29.370 So first of all we did build initial version of Svix was actually written in 00:09:29.370 --> 00:09:32.850 python so i guess there you know there are legs to that statement. 00:09:33.550 --> 00:09:37.030 And i think you know rust is problematic for 00:09:37.030 --> 00:09:39.770 fast iteration at the at the beginning of 00:09:39.770 --> 00:09:43.190 the company so like the way i think about it is if 00:09:43.190 --> 00:09:46.130 you're building i mean okay if you're very proficient in rust 00:09:46.130 --> 00:09:49.090 and you know you you move very quickly in rust just do 00:09:49.090 --> 00:09:51.890 it and you just use rust you don't have to worry about it just use 00:09:51.890 --> 00:09:55.250 you know use the tool that you know but at the beginning the 00:09:55.250 --> 00:09:57.930 really the beginning of the company when you don't even know what you're building and you're 00:09:57.930 --> 00:10:00.750 trying to figure it out and really what matters is 00:10:00.750 --> 00:10:04.150 just getting something clunky out there working and i 00:10:04.150 --> 00:10:07.590 think rust with the emphasis on correctness can 00:10:07.590 --> 00:10:10.530 hold you back but i think the moment you have even like 00:10:10.530 --> 00:10:13.790 a hint of traction like you know like even like a few months in and 00:10:13.790 --> 00:10:17.130 you realize that what you're building is generally you 00:10:17.130 --> 00:10:19.950 know again generally not it doesn't have to be precise but 00:10:19.950 --> 00:10:22.930 generally what you're going to build i think then rust is 00:10:22.930 --> 00:10:26.070 extremely powerful because rust you 00:10:26.070 --> 00:10:28.790 know when we were using python we we used 00:10:28.790 --> 00:10:31.750 to write a lot of tests we still write a lot of tests but much 00:10:31.750 --> 00:10:35.190 less and also kind of like have issues 00:10:35.190 --> 00:10:38.270 in staging and sometimes very rarely 00:10:38.270 --> 00:10:42.730 have issues in production as well and with rust it's kind of like you spend 00:10:42.730 --> 00:10:48.070 all of that time in the beginning but the likelihood of errors is significantly 00:10:48.070 --> 00:10:52.290 diminished later on so it's kind of like this it's almost there's like a preservation 00:10:52.290 --> 00:10:55.970 of energy so preservation of development time But with Python. 00:10:56.150 --> 00:11:00.770 You don't spend a lot of energy in the beginning, but you do spend it once it 00:11:00.770 --> 00:11:03.110 hits production, once you need to debug, once you need to refactor. 00:11:03.570 --> 00:11:07.810 And with Rust, you spend more in the beginning, but actually then later on, 00:11:07.810 --> 00:11:11.490 you don't have to spend as much energy. And I think now we move fast. 00:11:12.540 --> 00:11:14.620 Wasn't that transition painful. 00:11:14.620 --> 00:11:18.620 Oh extremely yeah i 00:11:18.620 --> 00:11:21.980 think you know like as i said like being reliable for 00:11:21.980 --> 00:11:24.820 our customers is the most important thing for us 00:11:24.820 --> 00:11:28.040 so we couldn't do what some 00:11:28.040 --> 00:11:31.020 companies do even even some of our competitors do which is 00:11:31.020 --> 00:11:34.020 you know oh yeah we're going to be down for a few hours on saturday 00:11:34.020 --> 00:11:36.800 we have a maintenance period we can't do anything like 00:11:36.800 --> 00:11:40.200 that so it was all like live transitions and you 00:11:40.200 --> 00:11:42.960 know making sure we don't break it to you know don't break anything for 00:11:42.960 --> 00:11:47.580 anyone like a lot of testing and what's i 00:11:47.580 --> 00:11:51.560 guess challenging about what it is that we do is or 00:11:51.560 --> 00:11:55.140 one of the things is you know we send http requests 00:11:55.140 --> 00:11:58.680 to a variety of servers right i mean like webhooks so that 00:11:58.680 --> 00:12:02.020 that means that we interact with you 00:12:02.020 --> 00:12:08.480 know i well well what's the name of that microsoft one ias iss why ias yeah 00:12:08.480 --> 00:12:13.340 so like ias apache nginx you know all of the wide variety of servers and some 00:12:13.340 --> 00:12:17.820 of them have very specific requirements for how a client should interact with 00:12:17.820 --> 00:12:21.000 them at the moment you know we switch to another client and like, 00:12:21.560 --> 00:12:27.500 like for example in rust we we were facing issues and i think actually a small 00:12:27.500 --> 00:12:30.260 tangent you know like one of the best thing and one of the worst thing about 00:12:30.260 --> 00:12:34.860 the rust ecosystem is the love for perfection and unfortunately the world out 00:12:34.860 --> 00:12:37.760 there is like ugly and dirty and when you interact with that world. 00:12:39.140 --> 00:12:42.700 Sometimes you can't be perfect like yeah you can adhere to the spec perfectly 00:12:42.700 --> 00:12:47.880 but if in our case but if the servers that we send the webhooks to expect something 00:12:47.880 --> 00:12:52.480 that is not required by the spec but i'm sorry that's what i expect you know 00:12:52.480 --> 00:12:55.360 we need to follow that we don't care that they're wrong we have to, 00:12:56.350 --> 00:13:00.970 We have to deliver that message. And so that, yeah, there were just like a variety of challenges. 00:13:01.150 --> 00:13:04.510 It just kind of triggered a bit of PTSD, I guess, from that time. 00:13:04.730 --> 00:13:10.890 If we go back to the first Python prototype that you built, it somehow also worked, right? 00:13:11.070 --> 00:13:19.910 It was running in production, but what was the general sentiment like at this point in time? 00:13:20.150 --> 00:13:26.730 Did you trust in the application? what was the deployment process the icd testing 00:13:26.730 --> 00:13:30.130 how many issues did you find in production versus development. 00:13:30.130 --> 00:13:33.410 And so on yes so actually 00:13:33.410 --> 00:13:36.130 a bit of again a tangent before like i was when i 00:13:36.130 --> 00:13:38.830 started coding i was i guess pearl in the beginning 00:13:38.830 --> 00:13:42.210 or whatever but i was like my first professional job was 00:13:42.210 --> 00:13:46.010 like a c embedded c engineer and again 00:13:46.010 --> 00:13:49.350 i have the scars to show for like you know like the living as 00:13:49.350 --> 00:13:53.070 a c engineer where like everything can break everything is scary and 00:13:53.070 --> 00:13:56.370 and and a very primitive type system as 00:13:56.370 --> 00:13:59.030 well so one of the things that i was doing 00:13:59.030 --> 00:14:03.670 i remember that as like a very junior engineer is adding a lot of typing as 00:14:03.670 --> 00:14:06.790 much as i can to see with like you know there are like a few tricks you do like 00:14:06.790 --> 00:14:10.870 opaque structs and like opaque type depths with like opaque structs whatever 00:14:10.870 --> 00:14:17.450 like a few a few ways a few tricks to do it um but that experience just made me obsessed with, 00:14:17.970 --> 00:14:22.930 telling the compiler what I really mean and having the compiler complain at 00:14:22.930 --> 00:14:26.910 build time about all the issues so I don't have to think about runtime. 00:14:27.070 --> 00:14:30.710 I know that if it compiles, it works. And by the way, I think that's partially 00:14:30.710 --> 00:14:32.090 why I'm such a big fan of Rust. 00:14:32.750 --> 00:14:37.210 So the reason why I told you all of this is that when we were writing Python 00:14:37.210 --> 00:14:39.830 code, it was already very good with typing. 00:14:40.050 --> 00:14:47.150 We had hackery around SQL Alchemy to make it more type safe. 00:14:47.450 --> 00:14:54.050 And we had even like scaffolding for fast API to make that even better and Pydantic. 00:14:54.470 --> 00:14:58.990 We had like a few places where we're doing, you know, kind of like we're doing 00:14:58.990 --> 00:15:04.250 what we could to make sure that Python has a very rich type system. 00:15:04.730 --> 00:15:07.830 So I think we were already doing like a fairly good job in terms of like catching 00:15:07.830 --> 00:15:10.730 a lot of issues at compile time. 00:15:10.830 --> 00:15:15.570 But the problem with Python is that those type, all the hacks that I mentioned 00:15:15.570 --> 00:15:17.350 were not real, right? It's not really a type. 00:15:17.450 --> 00:15:20.190 It's kind of like, you know, an annotation that we added. 00:15:20.350 --> 00:15:22.570 So we still had issues where we got the annotation wrong. 00:15:23.590 --> 00:15:27.850 And to account for that, like what we did is that we had a lot of unit tests, 00:15:28.070 --> 00:15:29.850 but we also had a lot of end-to-end tests. 00:15:30.070 --> 00:15:33.350 That was really the big thing that we did. Like we had a lot of end-to-end tests. 00:15:33.550 --> 00:15:37.670 And the end-to-end test suite like runs on staging, on real production, 00:15:37.810 --> 00:15:41.690 like against real load balance, everything, just to make sure that whatever 00:15:41.690 --> 00:15:45.990 it is that we do just goes, you know, safely goes into production. 00:15:46.890 --> 00:15:50.030 So we were pretty good. Like, I don't think we had a lot of production issues... 00:15:51.220 --> 00:15:55.180 I mean, definitely no serious ones, but it could be that we added a new API 00:15:55.180 --> 00:16:00.080 and that API maybe had an error when it was just introduced in some scenarios. 00:16:01.140 --> 00:16:07.060 So it was fairly good in that regard. But the problem was when you want to refactor, 00:16:07.200 --> 00:16:08.120 when you change something. 00:16:08.700 --> 00:16:13.500 That is the scariest experience you can have as a Python developer because you 00:16:13.500 --> 00:16:16.620 don't have that assurance of if it compiles, it just works. 00:16:17.320 --> 00:16:21.840 And man, finding those and going... Like one of the most annoying things is 00:16:21.840 --> 00:16:24.800 that like, and this used to be a common occurrence, doesn't happen for us anymore, 00:16:25.040 --> 00:16:29.220 is that like staging would be broken because someone merged something and that 00:16:29.220 --> 00:16:30.520 for whatever reason broke something. 00:16:30.820 --> 00:16:34.040 And then you kind of like, oh, I have another fix. And then like someone else 00:16:34.040 --> 00:16:36.460 brings a fix and someone else brings another change. And all of a sudden like 00:16:36.460 --> 00:16:38.600 unwinding those in staging is like a big pain. 00:16:40.280 --> 00:16:46.080 And that just doesn't happen anymore. So I don't think customers fully noticed 00:16:46.080 --> 00:16:50.900 the experience from a stability standpoint, but we definitely did in terms of 00:16:50.900 --> 00:16:54.520 just like how healthy our staging environment was and how big of a pain it was. 00:16:54.720 --> 00:16:59.360 The one thing that you triggered in me was also this PTSD that I had when refactoring 00:16:59.360 --> 00:17:04.000 larger Python codebases, where you end up with a... 00:17:04.740 --> 00:17:08.120 An exception somewhere really down the 00:17:08.120 --> 00:17:10.920 call stack in production and you have no idea 00:17:10.920 --> 00:17:14.060 what's happening you have no idea of the state that the application was 00:17:14.060 --> 00:17:17.380 in and you trigger that maybe once in a 00:17:17.380 --> 00:17:20.660 million requests or so because it's a very weird combination 00:17:20.660 --> 00:17:28.360 of conditions so yeah i can totally relate to that although i was wondering 00:17:28.360 --> 00:17:34.540 shouldn't the type system or the types that you added to the python code have 00:17:34.540 --> 00:17:37.520 you know saved you from that wasn't that the idea. 00:17:37.520 --> 00:17:40.420 I think the problem is that the python type system is 00:17:40.420 --> 00:17:43.320 just not expressive enough to really i mean maybe you 00:17:43.320 --> 00:17:46.080 know maybe nowadays it's like slightly better but like enums for 00:17:46.080 --> 00:17:48.860 example i don't think i mean maybe now they exist i don't think so 00:17:48.860 --> 00:17:51.660 and it's still not to the same level but it's like enums and being able to 00:17:51.660 --> 00:17:55.320 catch all of those you know all of the all 00:17:55.320 --> 00:17:58.560 of the variants yeah thank you and another thing 00:17:58.560 --> 00:18:01.720 that you said that triggered me is like exceptions oh my 00:18:01.720 --> 00:18:04.760 god i think that is the billion dollar programming mistake like 00:18:04.760 --> 00:18:07.880 i think rust really got it right in how to 00:18:07.880 --> 00:18:10.640 do error propagation which is 00:18:10.640 --> 00:18:13.720 you don't you can't just like willingly throw random exceptions all 00:18:13.720 --> 00:18:16.540 the way up to the stack like you have to catch them at every 00:18:16.540 --> 00:18:20.140 step and define the you know like properly define 00:18:20.140 --> 00:18:22.840 the um the signature and kind of like the 00:18:22.840 --> 00:18:26.660 contract that you have with the caller um i think is extremely important so 00:18:26.660 --> 00:18:30.740 yes python helped a lot but again 00:18:30.740 --> 00:18:34.480 as i said we missed some places like we didn't get the typing exactly correct 00:18:34.480 --> 00:18:38.480 in some areas we didn't the python was expensive enough to really get everything 00:18:38.480 --> 00:18:43.920 that we wanted in others it was scary it wasn't uh just walking the park what 00:18:43.920 --> 00:18:50.480 we trust and we can cover that later like it really we really utilized the type system to 00:18:50.920 --> 00:18:54.640 just make many runtime errors just like impossible. 00:18:56.940 --> 00:19:01.560 I don't want to put words into your mouth but this other thing that i really 00:19:01.560 --> 00:19:06.420 don't like about python deployments is that they are large you have large containers. 00:19:06.420 --> 00:19:07.980 Yeah and. 00:19:07.980 --> 00:19:11.560 In rust you have small binaries in comparison did you. 00:19:11.560 --> 00:19:18.400 Run into that problem okay so we we were using aws lambda for some things back 00:19:18.400 --> 00:19:24.140 in the days and you would hit i'm pretty sure we hit lambda size limits, 00:19:24.640 --> 00:19:27.520 maybe i'm misremembering no i'm pretty sure we hit lambda size 00:19:27.520 --> 00:19:31.860 limits it's like 150 megabyte but even if it wasn't a hard limit i think it 00:19:31.860 --> 00:19:35.360 was you know those were hard limits but even if they weren't i think it was 00:19:35.360 --> 00:19:39.880 just you'd have performance issues maybe or cost i don't remember it was annoying 00:19:39.880 --> 00:19:44.760 though it was really annoying yeah i mean nothing more to add there it was really 00:19:44.760 --> 00:19:46.920 annoying and like and also because there's a runtime, 00:19:48.070 --> 00:19:53.090 part to it like you would have to wait for for lambda for aws to support python 00:19:53.090 --> 00:19:57.970 3.10 before you can upgrade so we still like we were like a few versions behind 00:19:57.970 --> 00:20:03.850 it was really like a big pain by having like the runtime being shipped by someone that's not us okay. 00:20:03.850 --> 00:20:08.730 So we agree python might not be the best choice for what you wanted to build 00:20:08.730 --> 00:20:13.170 as fix a production grade service for webhooks, 00:20:14.310 --> 00:20:18.590 but Rust might not have been the only choice that you might have considered 00:20:18.590 --> 00:20:23.430 for example what about other languages like Elixir or Golang wouldn't that be an option too. 00:20:23.430 --> 00:20:29.030 Yeah so I can only speak to how it was when we made the decision maybe things 00:20:29.030 --> 00:20:33.350 I mean I know some things have changed in Elixir land and maybe a bit in Go 00:20:33.350 --> 00:20:37.710 so Elixir was even more esoteric than Rust back then and 00:20:37.750 --> 00:20:41.670 like choosing an esoteric language like Rust was already a bit of a gamble. 00:20:41.910 --> 00:20:45.430 So I had to, you know, I had to like hedge my bet to an extent. 00:20:45.550 --> 00:20:48.930 Like I couldn't go, and it's not that I wanted to choose Elixir and that's why 00:20:48.930 --> 00:20:52.170 I didn't choose it, but it was just like off the table just because of that. 00:20:52.730 --> 00:20:57.290 And also I had experience with Rust before and I didn't have any experience with Elixir. 00:20:57.450 --> 00:21:00.570 So, you know, the amount of unknowns was much higher, lower. 00:21:00.790 --> 00:21:02.690 Like I built production system in Rust before that. 00:21:03.390 --> 00:21:06.430 Not web, by the way, web was not ready before Sphix. like 00:21:06.430 --> 00:21:09.450 we were really at the cusp of like bus being web 00:21:09.450 --> 00:21:12.310 ready but yeah i've built like production stuff before 00:21:12.310 --> 00:21:15.850 in rust and then go it just 00:21:15.850 --> 00:21:18.850 and again maybe things have improved but 00:21:18.850 --> 00:21:24.810 the type system is so bare and as i said like i'm all in it's kind of felt like 00:21:24.810 --> 00:21:30.050 going almost backwards from like python like and then really i i mean yeah compilation 00:21:30.050 --> 00:21:33.430 speed amazing like i wish we had it in rust like i can we can probably talk 00:21:33.430 --> 00:21:36.330 about it later like i will complain you know for hours at no end, 00:21:36.470 --> 00:21:39.010 about a rough compilation time. But. 00:21:40.190 --> 00:21:43.490 The Go type system was just not expressive enough to really capture everything 00:21:43.490 --> 00:21:44.770 that I thought should be captured. 00:21:45.250 --> 00:21:50.530 And even without that, we actually wrote the first version of the rewrite of 00:21:50.530 --> 00:21:52.130 Sphix in both Go and Rust. 00:21:52.230 --> 00:21:56.790 And I say like first version, like we spent a few days on the Rust one, a few days on the Go one. 00:21:57.030 --> 00:22:00.010 And like someone, I wrote the Rust one and like someone else from the team that 00:22:00.010 --> 00:22:01.750 knew Go very well wrote the Go one. 00:22:01.930 --> 00:22:07.870 And then as a team, we just looked at both and we just decided like Rust just makes much more sense. 00:22:07.870 --> 00:22:10.690 In terms of like again all the areas that 00:22:10.690 --> 00:22:13.690 we cared about like type type safety but also 00:22:13.690 --> 00:22:16.350 just in terms of like how it is to use the 00:22:16.350 --> 00:22:19.530 language and there are like a few things that are done and go that i know go 00:22:19.530 --> 00:22:24.010 go for his love but i i just can't understand like the implicit implicit imports 00:22:24.010 --> 00:22:29.630 drives me mad like i like not implicit just like non-descriptive imports like 00:22:29.630 --> 00:22:35.210 the capitalization for visibility also is a bit annoying reminds me of like Hungarian notation, 00:22:36.750 --> 00:22:41.930 but i think also like the way to do json parsing if i remember correctly that 00:22:41.930 --> 00:22:45.510 was like i think it's done in comments or something i don't remember i remember 00:22:45.510 --> 00:22:51.750 it was just or not comments like a go specific comment thing annotation or something tags labels. 00:22:51.750 --> 00:22:53.450 Or whatever i guess tags yeah. 00:22:53.450 --> 00:22:57.130 Yeah again i don't remember it's been a while but i it just essentially we kind 00:22:57.130 --> 00:23:00.930 of like we look at the resulting code and we just okay there is no comparison 00:23:00.930 --> 00:23:05.390 here like we can build something on top of Rust, then I don't think we can build 00:23:05.390 --> 00:23:06.430 the same thing on top of Go. 00:23:06.690 --> 00:23:11.630 Did that person that wrote the Go prototype agree, or did they challenge the decision? 00:23:12.030 --> 00:23:16.070 You know, so I will admit that I'm the CEO of the company, so there's obviously 00:23:16.070 --> 00:23:19.470 like a bit of a power imbalance there, but from what I remember, 00:23:19.610 --> 00:23:22.510 there was like buying from everyone. 00:23:22.710 --> 00:23:26.710 The only concern was like finding Rust talent, which I shared. 00:23:28.210 --> 00:23:28.690 And... 00:23:29.700 --> 00:23:35.540 In this prototype was there already some sort of concurrency did you use you 00:23:35.540 --> 00:23:38.500 know go functions or so or go 00:23:38.500 --> 00:23:43.440 routines actually to maybe make some things concurrent or or was it linear. 00:23:43.440 --> 00:23:49.020 No no no so we i mean we built a tiny part of the product but we built it like 00:23:49.020 --> 00:23:53.100 we think it would look like in the end so even you know including like some 00:23:53.100 --> 00:23:56.660 weird scaffolding that we would you know normally you wouldn't want in like 00:23:56.660 --> 00:23:59.500 a first version but it's kind of like okay how how would it look like if we 00:23:59.500 --> 00:24:00.520 actually did this scaffolding. 00:24:00.520 --> 00:24:05.640 When you built this initial prototype you probably compared go concurrency with 00:24:05.640 --> 00:24:08.440 rust concurrency what what was the verdict there. 00:24:08.440 --> 00:24:11.440 Yeah i don't think there was 00:24:11.440 --> 00:24:14.840 you know like a a strong comparison there 00:24:14.840 --> 00:24:17.480 in the sense of you know it's like benchmarking or 00:24:17.480 --> 00:24:20.460 stuff like that we really we cared a lot about you know the typing 00:24:20.460 --> 00:24:23.320 and developer experience really what swayed us but we did 00:24:23.320 --> 00:24:26.320 use both right i mean go concurrency is kind of like nice in 00:24:26.320 --> 00:24:29.040 the sense you just like it just happens but i 00:24:29.040 --> 00:24:32.100 think i personally but this is a subjective thing i like 00:24:32.100 --> 00:24:34.880 i like the fact that there's like an await syntax i like 00:24:34.880 --> 00:24:38.280 the fact that we color functions by saying you know what this 00:24:38.280 --> 00:24:41.600 one is you know this one 00:24:41.600 --> 00:24:44.780 can run you know in an async context like 00:24:44.780 --> 00:24:49.480 this one has io this one has whatever i think that's a nice thing but this is 00:24:49.480 --> 00:24:54.440 really just a syntax thing i don't think we had a deep you know kind of any 00:24:54.440 --> 00:24:58.840 anything deep there and i i will say though that we and this is like a bit of 00:24:58.840 --> 00:25:03.900 a tangent we there was like a rust library that was doing magical. 00:25:04.700 --> 00:25:07.420 Io in the background so it was like you 00:25:07.420 --> 00:25:10.460 wouldn't the function wouldn't await but in 00:25:10.460 --> 00:25:13.940 the background it would trigger something that would make changes in the server 00:25:13.940 --> 00:25:18.180 and it's kind of just like again in a background thread or something like again 00:25:18.180 --> 00:25:22.520 like magical even though it wasn't like you didn't call a weight on the function 00:25:22.520 --> 00:25:27.820 and i remember that experience just remind me how much i love the fact that like, 00:25:28.490 --> 00:25:33.470 any io functions you know that io functions are you know have to be awaited because you know, 00:25:34.130 --> 00:25:37.910 you know you can easily look at the signature and know that something affects 00:25:37.910 --> 00:25:39.050 the server or if something doesn't 00:25:39.050 --> 00:25:41.670 affect the server is this like a local configuration or is this like a 00:25:42.110 --> 00:25:45.610 a remote configuration i know this is like an esoteric example but it really 00:25:45.610 --> 00:25:49.190 just we spent so much time debugging this because we were so confused by the 00:25:49.190 --> 00:25:54.730 fact that this non-async call was actually you know touching the server no. 00:25:54.730 --> 00:25:57.450 I can relate to that because a lot of people 00:25:57.450 --> 00:26:01.190 say function coloring is an issue but it 00:26:01.190 --> 00:26:03.850 feels like what you allude to 00:26:03.850 --> 00:26:06.530 is more of more the opposite where you say 00:26:06.530 --> 00:26:10.790 you want to know if something is async you want to be explicit about it and 00:26:10.790 --> 00:26:16.350 if you structure your code differently maybe it's not going to be a huge problem 00:26:16.350 --> 00:26:20.030 because you know that this part of the application has side effects and does 00:26:20.030 --> 00:26:24.250 ao in an asynchronous way and the other part does not it's. 00:26:24.250 --> 00:26:27.370 Interesting it's kind of like you use the word you know like function coloring 00:26:27.370 --> 00:26:30.090 right and i think that's a great example like you 00:26:30.090 --> 00:26:34.610 know we color we have syntax highlighting right in our code in our code we want 00:26:34.610 --> 00:26:39.370 to highlight things that behave differently than things than other things and 00:26:39.370 --> 00:26:44.590 especially things as important as you know i mean potentially if we don't even 00:26:44.590 --> 00:26:47.690 have like asking away like just blocking operations which is like i don't know 00:26:47.690 --> 00:26:49.250 how anyone can advocate for like, 00:26:49.930 --> 00:26:55.450 not having a way to know what our function is blocking like that yeah just beyond 00:26:55.450 --> 00:26:58.330 me again for blocking operations yeah. 00:26:58.330 --> 00:27:02.310 Okay so programming other 00:27:02.310 --> 00:27:05.170 programming languages were ruled out we talked about 00:27:05.170 --> 00:27:08.170 golang a little bit and elixir was just 00:27:08.170 --> 00:27:10.930 too cutting edge back in the day so it does 00:27:10.930 --> 00:27:14.010 make sense to take a closer look at rust but 00:27:14.010 --> 00:27:17.490 the other option that i had in mind at least maybe you discussed it internally 00:27:17.490 --> 00:27:25.250 was maybe we just rewrite parts of the application in rust maybe by using language 00:27:25.250 --> 00:27:31.170 bindings with py03 and then you rewrite parts of the critical stuff and you 00:27:31.170 --> 00:27:33.810 keep the rest did you consider that option. 00:27:33.810 --> 00:27:40.470 Yeah i mean i think for us the the kind of like two parts that cause latency 00:27:40.470 --> 00:27:44.210 or you know memory usage all of that like the performance either lies in the 00:27:44.210 --> 00:27:48.130 database queue or you know other like background systems that we use, 00:27:48.810 --> 00:27:52.430 or in the json serialization aspect and 00:27:52.430 --> 00:27:57.770 all of that so there isn't really a lot of code for us that we could out so 00:27:57.770 --> 00:28:01.950 there's not like an like an ai operation that you can just like okay i'm gonna 00:28:01.950 --> 00:28:06.510 or you know like a numpy thing like you know some like heavy scientific operation 00:28:06.510 --> 00:28:10.130 you can just do rewrite in c or in rust and we'll just be faster and everything 00:28:10.130 --> 00:28:11.730 else can just stay in python for us, 00:28:12.430 --> 00:28:15.330 the the the wrapping is the code like the 00:28:15.330 --> 00:28:18.490 you know like how quickly we can respond you know how you know 00:28:18.490 --> 00:28:21.390 the latency when we respond to http calls that is something that 00:28:21.390 --> 00:28:25.370 matters to us like how much memory use you know we use when we respond to an 00:28:25.370 --> 00:28:29.790 http call that's what matters so the shell was as important as the code itself 00:28:29.790 --> 00:28:33.950 so we we didn't have anything obvious or anywhere obvious to make that yeah 00:28:33.950 --> 00:28:37.410 and that change especially since by the way like all the serialization costs 00:28:37.410 --> 00:28:38.790 are just as significant so, 00:28:39.420 --> 00:28:42.900 It just didn't make sense. And, you know, actually another thing that comes 00:28:42.900 --> 00:28:45.220 to mind, we talked about, you know, the typing as well. 00:28:45.780 --> 00:28:48.340 That's another place to lose type. Like really there was no, 00:28:48.560 --> 00:28:51.560 we didn't even check that option. Like we just decided to commit. 00:28:52.020 --> 00:28:57.080 Yeah, because really what you build is a platform. It's sort of runtime for 00:28:57.080 --> 00:29:00.020 web things like webhooks, for example. 00:29:00.840 --> 00:29:05.480 And yeah, I see that you also need to handle a lot of JSON. and then if you 00:29:05.480 --> 00:29:09.920 transform a lot of JSON objects from Python to Rust and back, 00:29:10.060 --> 00:29:12.160 that also incurs memory overhead. 00:29:13.300 --> 00:29:18.920 Okay, that means we finally arrived at the final decision to rewrite the application 00:29:18.920 --> 00:29:23.180 in Rust, but it probably wasn't an easy path. 00:29:23.460 --> 00:29:27.740 It wasn't an easy migration. Can you talk a little bit about the learnings from 00:29:27.740 --> 00:29:30.080 migrating from Python to Rust? 00:29:30.500 --> 00:29:33.260 Yeah, I think, you know, like having making 00:29:33.260 --> 00:29:36.440 sure that the database types are the 00:29:36.440 --> 00:29:39.340 same wasn't that hard but still wasn't like 00:29:39.340 --> 00:29:41.960 super easy and and then just you know making sure that the 00:29:41.960 --> 00:29:45.140 ready serialization is identical because this one did it 00:29:45.140 --> 00:29:48.020 slightly different than the other one or maybe we you know 00:29:48.020 --> 00:29:51.620 we messed up something in python it was actually like named you 00:29:51.620 --> 00:29:54.360 know slightly in a weird manner that's inconsistent whether it rust we just 00:29:54.360 --> 00:29:57.100 kind of like did you know camel case everything and then in python we 00:29:57.100 --> 00:30:00.420 had to you know any whatever we had to like create weird aliases 00:30:00.420 --> 00:30:03.140 for like some of the cache stuff but i 00:30:03.140 --> 00:30:06.200 think other than that and other than the difficulties of just like a rewrite 00:30:06.200 --> 00:30:10.240 with zero downtime i think it went fairly smoothly and we're 00:30:10.240 --> 00:30:13.320 super okay first of all i'm a big fan of rust i'll i'll 00:30:13.320 --> 00:30:16.840 you know stand on the roof and shout it to anyone for anyone to hear but we 00:30:16.840 --> 00:30:19.620 did have some areas that were 00:30:19.620 --> 00:30:23.940 unexpected that where rust wasn't perfect actually and there were the areas 00:30:23.940 --> 00:30:26.760 that we expected like compilation i mean compilation app actually was worse 00:30:26.760 --> 00:30:30.500 than what we expected and there were other areas like finding engineers that 00:30:30.500 --> 00:30:34.680 was a bit of a challenge back then nowadays you know there are like a lot of 00:30:34.680 --> 00:30:36.980 rust engineers but there were actual, 00:30:37.760 --> 00:30:42.380 areas where rust or python i think i don't know if better is the right word 00:30:42.380 --> 00:30:46.980 but like there were just like less foot guns i guess i'm happy to elaborate on those if you want. 00:30:48.320 --> 00:30:54.060 Yeah please in your blog post you mentioned a few things like heap fragmentation uh, 00:30:54.810 --> 00:30:59.870 and memory efficiency maybe these would be a few things that. 00:30:59.870 --> 00:31:05.210 Yeah i think kind of like you know like if you summarize all of those in like 00:31:05.210 --> 00:31:08.430 in one you know kind of sentence like with great power comes great responsibility 00:31:08.430 --> 00:31:12.050 so kind of like you have more control now and so that means you have to be more 00:31:12.050 --> 00:31:15.590 careful about things the one example that you mentioned was like the heap fragmentation, 00:31:16.070 --> 00:31:19.210 so rust gives you more control there and 00:31:19.210 --> 00:31:23.030 i guess for whatever it just uses the default system allocator 00:31:23.030 --> 00:31:26.290 by default which just didn't work 00:31:26.290 --> 00:31:29.410 very well when it comes to fragmentation so like a long running process 00:31:29.410 --> 00:31:32.790 so heap fragmentation for those who don't know is essentially 00:31:32.790 --> 00:31:35.610 when you allocate big chunks or smaller chunks and then 00:31:35.610 --> 00:31:38.530 you can like every allocation just adds more to the memory 00:31:38.530 --> 00:31:41.730 instead of like reusing those chunks because they don't fit you know let's assume 00:31:41.730 --> 00:31:46.150 you you allocated a chunk of like 90 bytes then you freed it and then you want 00:31:46.150 --> 00:31:50.150 to allocate another chunk of like 95 bytes you can't reuse that one so you're 00:31:50.150 --> 00:31:53.650 going to get another chunk of 95 bytes then maybe you're going to reuse it for 00:31:53.650 --> 00:31:57.910 half of that smaller chunk but then you're not going to be able to you know another 70. 00:31:58.410 --> 00:32:01.230 Byte chunk would again would have to go on top and essentially what happens is 00:32:01.230 --> 00:32:04.470 that your memory just grows and grows and grows even though 00:32:04.470 --> 00:32:07.210 in practice you're actually not using that much 00:32:07.210 --> 00:32:09.910 memory using the same amount of memory but just 00:32:09.910 --> 00:32:12.670 because you allocate and deallocate in different sizes all the time it 00:32:12.670 --> 00:32:15.690 just causes fragmentation this was 00:32:15.690 --> 00:32:18.910 actually fixed by just switching to jmalloc like 00:32:18.910 --> 00:32:24.290 just changing the system allocated to one that you know is very good and really 00:32:24.290 --> 00:32:28.710 did wonders for us but the fact that we had to think about memory layouts and 00:32:28.710 --> 00:32:32.530 investigate like memory usage you know it's kind of like it just gives an example 00:32:32.530 --> 00:32:38.270 of like some of the stuff that you now have to worry about when you start using rust it's. 00:32:38.270 --> 00:32:44.550 Funny because the rust compiler itself or the Rust ecosystem moved away from 00:32:44.550 --> 00:32:50.610 Jamalock to the system allocator, but apparently for some use cases that isn't 00:32:50.610 --> 00:32:52.710 the best choice, the system allocator it is. 00:32:53.810 --> 00:32:55.970 Yeah, for us it definitely wasn't, yeah. 00:32:56.850 --> 00:32:57.150 But, 00:32:58.110 --> 00:33:04.770 where would you notice heap fragmentation was it a very occurring a reoccurring 00:33:04.770 --> 00:33:09.070 pattern that you could have solved with for example a bump allocator or an arena 00:33:09.070 --> 00:33:11.790 allocator was it expected, 00:33:13.010 --> 00:33:18.150 load of small objects that you allocated at once and could de-allocate at once. 00:33:18.150 --> 00:33:22.770 Yeah so i think for us i don't you know it's it was also like a year or something 00:33:22.770 --> 00:33:27.670 ago but like i think for us it was a lot of parsing json and the json would 00:33:27.670 --> 00:33:32.230 come from customers and we would send their json for them so that means you 00:33:32.230 --> 00:33:35.190 know because like it's a webhook so that means that that json can have 00:33:35.610 --> 00:33:40.610 you know for you know can be of varying sizes and can be and would normally be very large as well, 00:33:41.230 --> 00:33:44.170 i don't know we probably could have you 00:33:44.170 --> 00:33:46.930 know made so do you use a different allocate or something like that 00:33:46.930 --> 00:33:49.670 but it's just like changing the the default allocator would just 00:33:49.670 --> 00:33:52.310 you know solve everything i think that was probably the main 00:33:52.310 --> 00:33:55.710 thing another thing is you know we use some aws libraries and they 00:33:55.710 --> 00:34:01.270 i i believe they allocate we make http calls and that i believe allocates when 00:34:01.270 --> 00:34:05.050 it kind of creates generates the body that we send and so there were just like 00:34:05.050 --> 00:34:12.330 a few areas where we were allocating large arbitrary sized structures all the time yeah did. 00:34:12.330 --> 00:34:15.770 You notice a difference in memory usage when you switched the allocator. 00:34:15.770 --> 00:34:20.050 Oh yeah so before that it was i guess you can't see me it's only audio only 00:34:20.050 --> 00:34:24.010 but like you would see you know you'd see like a bump i guess because of some 00:34:24.010 --> 00:34:28.190 you know like high loads that would cause a lot of fragmentation and then that 00:34:28.190 --> 00:34:31.570 bump would just like stay flat and then there would be another bump and then 00:34:31.570 --> 00:34:32.750 that would stay flat as well, 00:34:33.550 --> 00:34:36.030 with jmadoc you immediately see just like, 00:34:36.730 --> 00:34:40.450 First of all, it becomes more jagged instead of flat. You'd see the graph of 00:34:40.450 --> 00:34:41.850 memory usage becomes way more jagged. 00:34:42.350 --> 00:34:48.570 And the low watermark was much lower. 00:34:49.750 --> 00:34:52.310 The baseline was significantly lower. 00:34:53.670 --> 00:34:57.230 We can post the link to that blog post as well in the show notes. 00:34:57.390 --> 00:35:00.450 You'll see there's a few graphs there. You can really see it. It's very obvious. 00:35:01.810 --> 00:35:08.090 Was memory usage even a concern for you? even if you used more memory would you run out of that. 00:35:08.090 --> 00:35:11.990 We did that's how we found out okay and 00:35:11.990 --> 00:35:14.770 again not immediately but just like every now and then like you 00:35:14.770 --> 00:35:17.330 get an oom and you're like what the hell is going on you know 00:35:17.330 --> 00:35:21.150 we see our usage we like we have 95 buffer 00:35:21.150 --> 00:35:24.410 like what's going on but we then we look at some of those and we 00:35:24.410 --> 00:35:27.350 see that there's like a memory leak but nothing is leaking right 00:35:27.350 --> 00:35:30.530 we did like a run valgrind we ran everything like nothing was leaking and then 00:35:30.530 --> 00:35:35.310 that's kind of how we discovered a few actually more things about like memory 00:35:35.310 --> 00:35:39.810 usage that were surprising again like you'd think russ is much more memory efficient 00:35:39.810 --> 00:35:43.350 than python right i mean like if you ask anyone on reddit or twitter or whatever 00:35:43.350 --> 00:35:44.650 they would they would tell you exactly that, 00:35:45.450 --> 00:35:51.230 but one thing was with 30 json like when you were passing generic json values 00:35:51.230 --> 00:35:56.390 so just you know like essentially again a dict but like a json dict so like 00:35:56.390 --> 00:36:00.250 not a not a string string but more just like generic, again, generic JSON, 00:36:00.530 --> 00:36:05.770 that would, for whatever reason, in serde, that causes like an explosion in memory. And we didn't, 00:36:06.370 --> 00:36:11.590 We had no idea. And I think, you know, like in a language like Python, 00:36:12.390 --> 00:36:16.070 you would think that just the generic case, or, you know, you think, 00:36:16.110 --> 00:36:19.190 I think the generic case just kind of like works, like because they optimize 00:36:19.190 --> 00:36:21.430 for that, they think about that, like, well, in Rust, 00:36:22.110 --> 00:36:27.410 you know, like using an untyped structure, that is such an, you know, such an odd case. 00:36:27.570 --> 00:36:30.930 And we were just like surprised by how bad that was. And maybe this is, 00:36:30.970 --> 00:36:32.410 by the way, was the reason for the. 00:36:33.350 --> 00:36:36.390 For the, well one of the reasons for the you know 00:36:36.390 --> 00:36:39.630 fragmentation that we're talking about i'm not sure and there 00:36:39.630 --> 00:36:42.310 are solutions so you had to know that you don't need you're not supposed to 00:36:42.310 --> 00:36:45.110 use json value you need to use json raw value that 00:36:45.110 --> 00:36:47.970 just like treats the string it doesn't power set but like again in 00:36:47.970 --> 00:36:52.070 a in a world you know in python stuff like that would just happen magically 00:36:52.070 --> 00:36:55.890 in the background you know kind of like copy and why and parse on access and 00:36:55.890 --> 00:36:59.550 all of those kind of things and i think with rust everything is much more rigid 00:36:59.550 --> 00:37:06.870 like what you ask for is what you get and it just like was like a funny gotcha that we that we got does. 00:37:06.870 --> 00:37:09.070 Python deserialize lazily. 00:37:10.410 --> 00:37:15.570 Probably not but there are things that i'm sure it does like to be efficient 00:37:15.570 --> 00:37:18.690 but maybe it does by the way i really have no idea well. 00:37:18.690 --> 00:37:25.790 Especially if you share those large json objects then i can see where the efficiency comes from because. 00:37:25.790 --> 00:37:26.990 In rust. 00:37:26.990 --> 00:37:33.070 You would generally clone unless you put it behind an arc but in python everything 00:37:33.070 --> 00:37:36.930 is ref counted so you get cheap copies for free. 00:37:37.750 --> 00:37:40.830 That's another like example of where you know 00:37:40.830 --> 00:37:43.610 memory use can just just go out of the window because 00:37:43.610 --> 00:37:46.270 as you said in like in python you don't really care about you know kind of 00:37:46.270 --> 00:37:50.150 like what was it like fearless concurrency is that the yeah 00:37:50.150 --> 00:37:53.010 you don't really care about that like yeah of course space conditions that's 00:37:53.010 --> 00:37:55.830 just a fact of life in python so well i guess maybe 00:37:55.830 --> 00:37:59.830 people just don't use threading as much and so 00:37:59.830 --> 00:38:02.930 they just as you said pass like ref counts all over 00:38:02.930 --> 00:38:05.750 like in rust like you would clone so much data 00:38:05.750 --> 00:38:08.970 if you're not careful like you would accidentally clone that json structure 00:38:08.970 --> 00:38:12.850 like that heavy when you pass it to another function and you can accidentally 00:38:12.850 --> 00:38:18.750 clone something else and it's very easy to accidentally use much more memory 00:38:18.750 --> 00:38:22.870 than you would use in python just because of the you know like the implicit 00:38:22.870 --> 00:38:27.030 ref counting and just like passing of references instead of cloning yeah. 00:38:27.030 --> 00:38:32.970 I always wanted a lazy json library in rust and probably it exists but i couldn't 00:38:32.970 --> 00:38:38.210 bother looking it up um but it would be so nice to have a system that does not 00:38:38.210 --> 00:38:43.070 create 30 json values all the time but it more or less is a view into the data 00:38:43.070 --> 00:38:46.310 and then it only deserializes what you actually want to look at. 00:38:46.310 --> 00:38:52.070 Yeah i think so kind of the 30 row value kind of semi lets you do that by the 00:38:52.070 --> 00:38:56.230 fact that it doesn't serialize that part and then you can maybe explicitly deserialize that part. 00:38:56.870 --> 00:39:01.530 I mean, again, you have to build the scaffolding yourself, but it is semi-possible. 00:39:02.090 --> 00:39:03.530 Or at least that's what we're doing now. 00:39:04.620 --> 00:39:09.160 True. And for you, the serde interface is probably, I'm assuming, 00:39:09.420 --> 00:39:14.640 one of the most important parts of what you use in the Rust ecosystem. 00:39:15.000 --> 00:39:20.340 What else do you use very heavily? What do you depend on heavily? Which creates? 00:39:20.880 --> 00:39:26.480 So serde would be one. Axum, we use SeaORM as our ORM. 00:39:26.600 --> 00:39:29.140 I mean, mainly for the query building, less for ORM stuff. 00:39:29.960 --> 00:39:33.340 We use redis rust we use 00:39:33.340 --> 00:39:36.980 aid for open telemetry generation i think 00:39:36.980 --> 00:39:40.340 those would be the main ones and actually i mean you 00:39:40.340 --> 00:39:43.520 know like this is a good segue for like ecosystem maturity 00:39:43.520 --> 00:39:46.680 because we you know 00:39:46.680 --> 00:39:49.900 we have the maintainer of one of the maintainers of axiom on the team we have 00:39:49.900 --> 00:39:53.060 one of the maintainers of redis rust on the team we have one of the maintainers 00:39:53.060 --> 00:39:59.260 of aid on the team and it's not you know i wish we didn't right i mean i'm happy 00:39:59.260 --> 00:40:02.280 to have those people they're great and like they're very you know you know they're 00:40:02.280 --> 00:40:05.400 smart and capable but like we we didn't actually have to 00:40:05.820 --> 00:40:11.060 we have to spend time and effort on these you know like core libraries that 00:40:11.060 --> 00:40:14.180 we depend on like we have to spend time and effort on the ecosystem to make 00:40:14.180 --> 00:40:15.920 sure that it's where we need it to be, 00:40:16.400 --> 00:40:20.460 which is a bit i mean i think it's less less so nowadays and i guess if you're 00:40:20.460 --> 00:40:24.340 starting a swix competitor you have us to maintain this for you so like you 00:40:24.340 --> 00:40:31.160 don't have to incur the same cost but but it was a challenge and it definitely is still is but. 00:40:31.160 --> 00:40:33.480 On the other side you need to have some skin in the game. 00:40:33.480 --> 00:40:38.540 Yeah again no no not complaining really not complaining but you know if we had 00:40:38.540 --> 00:40:43.780 the alternative which would be like having the perfect redis library just existing 00:40:43.780 --> 00:40:47.140 and we didn't have to you know spend so much time on it that would have been 00:40:47.140 --> 00:40:50.820 better but again no complaints we're happy that it's well maintained by people 00:40:50.820 --> 00:40:52.700 that are well meaning and are capable so. 00:40:53.400 --> 00:40:58.420 What were some of the missing features or issues that you had with the existing redis libraries. 00:40:59.800 --> 00:41:04.700 So when it's james for my team when he took over the maintainership it was essentially, 00:41:05.920 --> 00:41:08.680 abandoned i mean i don't know if like abandoned is the right word but it wasn't like 00:41:08.680 --> 00:41:11.620 being released or maintained or i guess for 00:41:11.620 --> 00:41:14.860 all intents and purposes abandoned it had no 00:41:14.860 --> 00:41:17.900 support for redis cluster just like didn't support that 00:41:17.900 --> 00:41:20.620 async the async version of 00:41:20.620 --> 00:41:23.480 the lib was fairly broken if 00:41:23.480 --> 00:41:27.320 i remember correctly oh sorry it had support for for blocking 00:41:27.320 --> 00:41:30.900 redis cluster and blocking redis and that was fairly okay in the lib not perfect 00:41:30.900 --> 00:41:36.200 but the async variants for either which is the async variant for the for the 00:41:36.200 --> 00:41:42.340 non-cluster version was limited and the async cluster version was like an external 00:41:42.340 --> 00:41:44.260 library by someone else that's also unmaintained. 00:41:45.020 --> 00:41:50.220 And James merged them together and really streamlined the library and made everything better. 00:41:50.680 --> 00:41:55.240 Well, that's nice to hear because it benefits a wider network, 00:41:55.720 --> 00:42:02.560 array of people of potential users um any similar story for any other dependency 00:42:02.560 --> 00:42:07.800 you mentioned that you have your own open api client i guess but there's drop 00:42:07.800 --> 00:42:11.360 shot from oxide computer i wonder 00:42:11.360 --> 00:42:15.280 if if you looked at that or what was the reason for building your own. 00:42:15.280 --> 00:42:18.320 Yeah so i don't remember what the drop shot is 00:42:18.320 --> 00:42:22.360 i'll check it out again after this but aid essentially 00:42:22.360 --> 00:42:25.400 is a tool to automatically generate an open api spec 00:42:25.400 --> 00:42:28.260 from your axiom code so you 00:42:28.260 --> 00:42:31.000 define you know like what you define the the 00:42:31.000 --> 00:42:33.740 api routes what you accept what you send and we kind 00:42:33.740 --> 00:42:37.180 of like we automatically generate it automatically generates the open api spec 00:42:37.180 --> 00:42:40.140 from that it was great we looked at multiple 00:42:40.140 --> 00:42:43.240 options back when we adopted it it was 00:42:43.240 --> 00:42:46.220 by far the best one and unfortunately it went unmaintained a 00:42:46.220 --> 00:42:48.980 few months ago or even longer than that and and we kind 00:42:48.980 --> 00:42:51.860 of like had to take over because you know we had fixes for upstream that 00:42:51.860 --> 00:42:54.620 just weren't being merged and even you know i 00:42:54.620 --> 00:42:57.480 don't remember how much how long we waited but we waited a long time for one 00:42:57.480 --> 00:43:02.040 of the fixes to be merged but then no release was being made so that also was 00:43:02.040 --> 00:43:07.660 fairly useless in that regard so yeah now we kind of like we were you know we've 00:43:07.660 --> 00:43:12.840 been merging prs and like making releases and yeah happy about it yeah. 00:43:12.840 --> 00:43:19.080 Then again these dependencies make sense to be owned by a company like yours 00:43:19.080 --> 00:43:24.080 because they are so central to what you do they are central to the web ecosystem 00:43:24.080 --> 00:43:30.740 and you know all of what you described sounds reasonable at least from an outsider's perspective. 00:43:30.740 --> 00:43:32.700 Yeah and also thanks for. 00:43:32.700 --> 00:43:33.660 Maintaining by the way. 00:43:33.660 --> 00:43:36.520 I don't know with pleasure and you know we actually we even 00:43:36.520 --> 00:43:39.920 had to create a new library so like we created a library 00:43:39.920 --> 00:43:42.760 for kcuids that just didn't 00:43:42.760 --> 00:43:45.600 exist we built one actually we also wrote one 00:43:45.600 --> 00:43:48.580 in python that's fairly popular but we did 00:43:48.580 --> 00:43:51.620 that we also built or we created the omni queue 00:43:51.620 --> 00:43:54.820 so essentially you know like our open source project supports a 00:43:54.820 --> 00:43:58.280 variety of queues in the background so like redis sqs you 00:43:58.280 --> 00:44:01.180 know just because it needs a queue and i remember like 00:44:01.180 --> 00:44:05.120 i remember celery from python very fondly it's like a library that just lets 00:44:05.120 --> 00:44:09.120 you you know it just like abstracts away the queue back end and then people 00:44:09.120 --> 00:44:12.900 that use your library could just choose their own queue of choice so we kind 00:44:12.900 --> 00:44:18.660 of like we built the equivalent of that for rust yeah it was also also great now. 00:44:18.660 --> 00:44:22.420 Let's talk about production rust okay 00:44:22.420 --> 00:44:29.240 you have that thing in production i have two main questions how does it work 00:44:29.240 --> 00:44:36.680 on the load and how do you avoid breaking changes when when you might when you 00:44:36.680 --> 00:44:39.940 maintain a larger application in Rust. 00:44:40.740 --> 00:44:45.540 We came from Python, and Python, let's say, is not notorious for being fast. 00:44:45.780 --> 00:44:47.240 And I think it's much better nowadays. 00:44:47.640 --> 00:44:52.820 But back, I think it was like Python 3.7 or 3.8 when we switched. 00:44:53.220 --> 00:44:56.740 I think it still has the global interpreter lock, or maybe just the just release 00:44:56.740 --> 00:44:57.960 a version without. I don't know. 00:44:58.400 --> 00:45:04.040 But the point is, with Python, you run the way to scale up. 00:45:04.300 --> 00:45:04.680 Multi-threading, yeah. 00:45:05.200 --> 00:45:07.860 But multi-process as well, because of the global interpreter lock. 00:45:07.960 --> 00:45:12.240 So you would run multiple threads and multiple processes. in order to use processes are. 00:45:12.240 --> 00:45:17.380 Probably fine because those are separate scopes of memory but if you have. 00:45:17.380 --> 00:45:18.000 Multiple threads. 00:45:18.000 --> 00:45:21.980 In the same python interpreter then you need the global interpreter log or you 00:45:21.980 --> 00:45:23.260 have the new interpreter. 00:45:23.260 --> 00:45:29.160 What's it called yeah yeah yeah yeah but the problem in multi-process yeah it 00:45:29.160 --> 00:45:32.580 doesn't have the jail limitations that's why they recommend it but it you can 00:45:32.580 --> 00:45:36.680 share memory you like connection pools are separate like for like to the database 00:45:36.680 --> 00:45:40.740 like So actually it comes with its own challenges and just not having to worry 00:45:40.740 --> 00:45:42.820 about any of that, first of all, was fun. It was great. 00:45:43.100 --> 00:45:48.620 But also we've just seen, and it could be when you rewrite, you probably optimize 00:45:48.620 --> 00:45:49.820 things a bit and all of that. 00:45:49.900 --> 00:45:53.620 You know the system better. But we've seen all of Magnitude's improvement in memory usage. 00:45:54.680 --> 00:45:58.400 Latency was, I mean, just like shot down significantly. 00:45:58.600 --> 00:46:02.280 And again, we changed some infrastructure as we were doing, so it could be semi-related 00:46:02.280 --> 00:46:05.860 to that as well. But just every metric improved drastically. 00:46:06.900 --> 00:46:12.460 I think it was, I don't remember the exact numbers, but I think it was probably 00:46:12.460 --> 00:46:16.480 40x or something, the difference between the amount of Python runners that we 00:46:16.480 --> 00:46:18.280 needed to run compared to the Rust ones. 00:46:19.560 --> 00:46:22.580 And the thing about that is that it actually compounds. 00:46:22.920 --> 00:46:27.800 So if you go from, let's even go for the basic case of just like two Rust runners 00:46:27.800 --> 00:46:32.100 and like 80 Python ones, all of a sudden, like the two Rust runners, 00:46:32.320 --> 00:46:37.480 they hit in-memory cache like all the time because they're just two of them, right? 00:46:37.520 --> 00:46:41.220 So let's say 50% of the time, but probably more than 50% because like the moment 00:46:41.220 --> 00:46:43.980 they flip, it gets to the other one, that one now has it cached. 00:46:44.300 --> 00:46:47.760 So like in-memory caching is actually extremely effective all of a sudden. 00:46:47.880 --> 00:46:52.600 We have more than two, but still not way more than two i mean more way more 00:46:52.600 --> 00:46:57.340 than two but not way more to the point as like not effective anymore and with 00:46:57.340 --> 00:47:01.260 python we just we couldn't do effective in-memory caching like you know connection 00:47:01.260 --> 00:47:04.900 database connection pools were just not being reused as much like so, 00:47:05.780 --> 00:47:08.800 switching to us actually significantly reduced our database load 00:47:08.800 --> 00:47:11.740 our redis load like all the 00:47:11.740 --> 00:47:15.940 loads and all of the upstream system you would never think um like you'd really 00:47:15.940 --> 00:47:20.300 never think that this would be i mean i guess you would think but it really 00:47:20.300 --> 00:47:26.100 the downstream effects of just being able to run less processes was just you 00:47:26.100 --> 00:47:29.760 know we're just we're just insane for us so we're very happy about that no. 00:47:29.760 --> 00:47:35.120 One really thinks about cache locality when you think about reducing the number 00:47:35.120 --> 00:47:39.380 of nodes you have but it's so vital at a certain point. 00:47:39.380 --> 00:47:45.780 Yeah i think again like i don't have the exact numbers but like the calls to redis which is like, 00:47:46.520 --> 00:47:50.740 went essentially to nothing like fairly quickly i mean not fairly immediately 00:47:50.740 --> 00:47:54.440 when we switch but like essentially for not to nothing so right. 00:47:54.440 --> 00:48:01.480 And now let's come to the second question which is about avoiding breaking api 00:48:01.480 --> 00:48:04.040 changes or breaking changes in the api. 00:48:04.040 --> 00:48:07.420 Yeah so i think you know 00:48:07.420 --> 00:48:10.140 i have we haven't released it yet we have a we i wrote a 00:48:10.140 --> 00:48:12.940 blog post exactly about this and what we do there and i'm 00:48:12.940 --> 00:48:16.520 happy to give a bit of a glimpse so first 00:48:16.520 --> 00:48:21.320 of all let's start with the interface and then we can talk about the internals 00:48:21.320 --> 00:48:25.340 the blog post is more about the interface anyway so when you when i mentioned 00:48:25.340 --> 00:48:30.560 we use this tool called aid and aid generates an open api spec open api by the 00:48:30.560 --> 00:48:34.760 way it's kind of like a formal definition language for describing http apis. 00:48:35.680 --> 00:48:43.720 So what aid would do it would take our call let's let's say the api calls create application, 00:48:44.400 --> 00:48:47.240 it will automatically you know make 00:48:47.240 --> 00:48:50.200 a json schema out of the ink you know the body that comes in 00:48:50.200 --> 00:48:53.140 the body that goes out all the path parameters 00:48:53.140 --> 00:48:56.280 query parameters the path itself authentication requirements all 00:48:56.280 --> 00:48:59.120 of that will just aid will automatically create that open 00:48:59.120 --> 00:49:01.920 api spec and we you know we did a lot 00:49:01.920 --> 00:49:04.800 of extra enrichment on our end to you know 00:49:04.800 --> 00:49:07.800 to add oh this this field 00:49:07.800 --> 00:49:11.000 also only accepts a regex expects a 00:49:11.000 --> 00:49:13.760 regex that like matches this and this integer can only be 00:49:13.760 --> 00:49:16.540 from 1 to 50 and this is an array that 00:49:16.540 --> 00:49:19.800 can't be empty like we added like a lot of additional kind of 00:49:19.800 --> 00:49:22.820 restrictions and the nice thing as well by the way again a small tangent is 00:49:22.820 --> 00:49:28.020 that the code that enforces it for us is the same as the code that generates 00:49:28.020 --> 00:49:34.080 the aid description so like the the you know the the size limitation is literally 00:49:34.080 --> 00:49:37.660 what we enforce if it's in the open api spec it's what we enforce if it's not 00:49:37.660 --> 00:49:42.520 it's not what we enforce and barring a few exceptions that we purposely broke that but like. 00:49:43.280 --> 00:49:48.080 Everything is just like, all of this is automatic. And what this means is that 00:49:48.080 --> 00:49:53.980 we have a JSON file with the exact description of our exact API that we can 00:49:53.980 --> 00:49:55.380 just generate whenever we want. 00:49:55.540 --> 00:49:58.160 So what we do, we generate that. 00:49:58.600 --> 00:50:01.380 We commit it whenever you make a change. We commit it to Git. 00:50:01.560 --> 00:50:04.080 And then in CI, we generate it again, and then we compare it. 00:50:04.640 --> 00:50:09.900 And if it's different, CI fails. 00:50:10.300 --> 00:50:13.180 So essentially what it means is that if you make a change that 00:50:13.180 --> 00:50:15.920 affects our api you consciously have to update that 00:50:15.920 --> 00:50:18.700 file like regenerate that files first of all you as the 00:50:18.700 --> 00:50:22.440 first the developer as the first line of defense you know that you 00:50:22.440 --> 00:50:25.060 just did something that's potentially breaking and again maybe it's on 00:50:25.060 --> 00:50:28.040 purpose maybe you added a new api it's fine but you 00:50:28.040 --> 00:50:31.060 know that and also we have github code 00:50:31.060 --> 00:50:34.180 owners on that specific spec file 00:50:34.180 --> 00:50:37.120 that forces people you know like that have extra 00:50:37.120 --> 00:50:39.880 people that you know that know what we expect from 00:50:39.880 --> 00:50:43.140 our api like what do we want to release what we don't want to release they 00:50:43.140 --> 00:50:45.800 will have to review it as well and just say you know 00:50:45.800 --> 00:50:48.920 what this api changes is reasonable so essentially 00:50:48.920 --> 00:50:53.500 we kind of like what we did is we leverage the you know the rust type system 00:50:53.500 --> 00:50:59.740 with some like macro magic and and a few other things to just to just make it 00:50:59.740 --> 00:51:04.300 so we can't accidentally change our api signatures we have to like we are have 00:51:04.300 --> 00:51:07.240 to be aware of it like we can't accidentally release an api we don't want to release. 00:51:07.360 --> 00:51:12.100 We can't just do those kind of things. So that's been great in that regard. 00:51:13.150 --> 00:51:22.030 How would you even learn that? If you did not know that you could build an OpenAPI 00:51:22.030 --> 00:51:24.050 spec from your XM handlers, 00:51:24.750 --> 00:51:30.550 how would you find out that you could use the Rust type system to enforce that 00:51:30.550 --> 00:51:35.170 and to make testing easier and to make breaking changes explicit? 00:51:35.330 --> 00:51:40.670 Are there any tips that you could share in order to really hook into the Rust 00:51:40.670 --> 00:51:44.510 type system and encode the invariance that you depend on. 00:51:44.990 --> 00:51:47.970 Yeah, so I think this is like a hack that we did. I don't know if, 00:51:48.210 --> 00:51:51.490 I don't think it's a common practice. Maybe it is, and we're just unaware of it. 00:51:51.810 --> 00:51:54.930 And that's why, again, we wrote that blog post. I mean, I think it is, 00:51:55.130 --> 00:51:57.210 I think more people should be doing exactly this. 00:51:58.210 --> 00:52:00.530 Although, you know what, now that I think about it, I'm sure like large companies 00:52:00.530 --> 00:52:01.610 have their own tricks as well. 00:52:02.450 --> 00:52:05.990 But really the key is, I'm a big fan. So some people, what they do, 00:52:06.130 --> 00:52:09.110 they write the open API spec by hand, and then 00:52:09.110 --> 00:52:13.210 they generate user generated to generate the backend code i don't think that's 00:52:13.210 --> 00:52:16.350 good because there's going to be a limit to how well that backend code could 00:52:16.350 --> 00:52:19.730 be generated so you're going to edit it manually in the end but i think when 00:52:19.730 --> 00:52:23.930 you generate it from the code and the code is the source of truth which you 00:52:23.930 --> 00:52:26.430 know let's face it is it is a source which that's what's run in production, 00:52:27.090 --> 00:52:31.170 i think generating from that like really makes sense so just i think just become 00:52:31.170 --> 00:52:37.490 obsessed with types and annotations and everything that you want to describe as part of your contract, 00:52:37.950 --> 00:52:40.970 i think if you're obsessed you know if you're just obsessed about that like 00:52:40.970 --> 00:52:45.850 everything else will just fall into place yeah i mean i think really that's 00:52:45.850 --> 00:52:50.050 that's the main thing and i guess related to that is you have to be careful 00:52:50.050 --> 00:52:54.050 about having too you know too wide of a contract, 00:52:54.850 --> 00:53:02.370 so if you by default let's say there's like a an age i mean this is like the most uh you know. 00:53:03.470 --> 00:53:08.310 CS101 example, but like you have like a person's name and age and you let the 00:53:08.310 --> 00:53:12.470 age be negative by default because, you know, you didn't limit it, you just put an int, 00:53:13.310 --> 00:53:18.470 then you can like, by default, you exposed a wider contract than what you want to expose. 00:53:18.750 --> 00:53:22.410 So I think actually being very explicit about annotating correctly, 00:53:22.650 --> 00:53:26.710 like everything that you actually believe this type should have and should hold, 00:53:26.870 --> 00:53:29.750 I think is extremely important. And I think that could go a long way. 00:53:30.910 --> 00:53:34.110 And how far can you take this? I think in your blog post, you also mentioned 00:53:34.110 --> 00:53:37.770 that your Redis get and set commands are strongly typed. 00:53:38.850 --> 00:53:41.990 Yeah. And so actually, this is a different blog post, that one. 00:53:42.130 --> 00:53:46.250 But this is the blog post you're referring to. That's where we use it for internal 00:53:46.250 --> 00:53:48.730 stability. And it's exactly that. 00:53:49.890 --> 00:53:54.410 Internally, we use the type system to protect us from a lot of things, 00:53:54.630 --> 00:53:57.870 like essentially define all the internal contracts as well. 00:53:58.490 --> 00:54:01.410 So you just mentioned Redis. you know redis lets you 00:54:01.410 --> 00:54:04.570 store random strings like it doesn't have any typing associated with 00:54:04.570 --> 00:54:07.290 it but what we did is we 00:54:07.290 --> 00:54:10.130 created a new type and we kind of we added a new redis 00:54:10.130 --> 00:54:12.930 interface and and and the 00:54:12.930 --> 00:54:16.170 way we do it is you have to define a type for the key and that 00:54:16.170 --> 00:54:19.190 type for the key has to be associated with a specific schema 00:54:19.190 --> 00:54:23.970 for the value so if you use that key to get a value you know that it's always 00:54:23.970 --> 00:54:27.830 going to be the correct schema because we parse at the moment we get it and 00:54:27.830 --> 00:54:31.370 when you write that value same thing you always know it's going to be the same 00:54:31.370 --> 00:54:36.490 schema because you the only way to write into redis would be using this interface 00:54:36.490 --> 00:54:38.930 does that make sense yeah. 00:54:38.930 --> 00:54:46.170 It's a bit like a binary protocol that you talk to with redis so it's it's a 00:54:46.170 --> 00:54:50.590 spec in between that is the contract between whatever redis expects and whatever 00:54:50.590 --> 00:54:52.070 your code sends and receives. 00:54:52.070 --> 00:54:55.850 Exactly yeah we kind of like add this like nice contract between the you know 00:54:55.850 --> 00:54:59.810 like kind of what i said earlier the messy world out there like untyped redis 00:54:59.810 --> 00:55:02.330 and and our clean code you. 00:55:02.330 --> 00:55:07.250 Recently moved to redis streams does that principle still apply do you also 00:55:07.250 --> 00:55:09.970 use a typed stream of some sort. 00:55:09.970 --> 00:55:16.350 Yeah so the red redis streams they just accept you know payloads whatever you 00:55:16.350 --> 00:55:19.410 know you just put whatever you put there we just wrap around that as well with 00:55:19.410 --> 00:55:22.370 the same schema schema type it. 00:55:22.370 --> 00:55:25.890 Feels like we both are really big fans of rust's 00:55:25.890 --> 00:55:28.990 type safety but there are other traits 00:55:28.990 --> 00:55:32.530 about rust that people often mention for 00:55:32.530 --> 00:55:35.730 reasons for moving to rust and the 00:55:35.730 --> 00:55:38.730 things that people usually mention is number one 00:55:38.730 --> 00:55:44.990 safety slash security number two fearless concurrency number three performance 00:55:44.990 --> 00:55:52.790 and number four stability or robustness and if you could order these four things 00:55:52.790 --> 00:55:57.670 by priority what would be number one two and so on for you. 00:55:58.480 --> 00:56:02.100 Let me first give you a cop-out which is i think the rust typing system is what 00:56:02.100 --> 00:56:05.620 enables all of those so i just choose the rust typing system and that awesome 00:56:05.620 --> 00:56:08.120 yeah it just gives us all of that um. 00:56:08.660 --> 00:56:09.540 Oh you like that. 00:56:09.540 --> 00:56:12.920 Yeah i think you know 00:56:12.920 --> 00:56:15.640 so i'm man i don't know i i think 00:56:15.640 --> 00:56:21.860 fearless concurrency i don't care about as much i mean i i do and we do use 00:56:21.860 --> 00:56:25.460 it and whatever so like don't yeah we definitely care about it but like i mean 00:56:25.460 --> 00:56:29.680 compared to like safety and correctness but again it they kind of come together 00:56:29.680 --> 00:56:33.780 right the fearless concurrency comes from the fact that you can be safe and correct, 00:56:34.640 --> 00:56:37.980 so i guess like all of my answers will just go back to the type system but let's say, 00:56:38.640 --> 00:56:44.400 safety correctness would be first performance would be second the fearless concurrency 00:56:44.400 --> 00:56:49.800 would be last what what's the third one okay so stability will be second performance 00:56:49.800 --> 00:56:55.320 will be third yeah i think that i think that's and i will add developer experience 00:56:55.320 --> 00:56:56.820 somewhere in the middle as well like, 00:56:58.000 --> 00:57:03.420 ergonomics yeah ergonomics exactly um it just yeah the the fact that you just 00:57:03.420 --> 00:57:06.180 write the code that you want to write and you kind of trust the compiler to 00:57:06.180 --> 00:57:13.200 do a good job but like the zero zero cost optimizations it's just it's fun yeah yeah it's. 00:57:13.200 --> 00:57:19.380 Kind of interesting that you rank stability slash robustness higher than raw performance. 00:57:19.380 --> 00:57:22.140 I think you know like at the end of the day and 00:57:22.140 --> 00:57:25.360 i know a lot of people are going to be angry at me for saying this you can buy larger 00:57:25.360 --> 00:57:29.960 servers at the end i mean again i gave the python example you can't buy 100x 00:57:29.960 --> 00:57:34.160 larger server there is a limit but if you know it's about like 30 improvement 00:57:34.160 --> 00:57:39.920 you can buy a 30 larger server but like the the safety and like stability is 00:57:39.920 --> 00:57:43.040 just you can't buy a larger server for that yeah. 00:57:43.040 --> 00:57:50.940 Okay we've witnessed your journey from the very beginning from the idea to now 00:57:50.940 --> 00:57:54.400 if you started over with that project, 00:57:54.920 --> 00:58:01.120 and with everything you know now what advice would you give to yourself back 00:58:01.120 --> 00:58:02.220 in the day when you started. 00:58:02.220 --> 00:58:05.960 So i think 00:58:05.960 --> 00:58:09.540 we did a lot of things well but i think we did a lot well i 00:58:09.540 --> 00:58:12.900 guess a lot of things as well unwell not as well you know 00:58:12.900 --> 00:58:15.900 one thing that we did in the beginning and i 00:58:15.900 --> 00:58:18.880 think like i would bet that every startup founder does 00:58:18.880 --> 00:58:22.660 the same mistake is we assumed that today 00:58:22.660 --> 00:58:25.940 even though today we have zero customers by end 00:58:25.940 --> 00:58:29.640 of next week we're probably going to have facebook load or 00:58:29.640 --> 00:58:35.500 like google load on our systems and that's you know like we we just we will 00:58:35.500 --> 00:58:41.100 we will drown and die if we don't fix this load issue now immediately before 00:58:41.100 --> 00:58:45.100 anything else and we start designing our system to support that load and like 00:58:45.100 --> 00:58:47.880 yeah this can go from like zero to a billion customers, 00:58:48.860 --> 00:58:51.560 already and like this is the best system and all of that and and 00:58:51.560 --> 00:58:54.820 i think we made the same mistake of thinking that we 00:58:54.820 --> 00:58:58.700 should care about that early on and and 00:58:58.700 --> 00:59:03.540 we our initial system you know was built to support like a thousand x of the 00:59:03.540 --> 00:59:08.260 load that was you know where we had even in the you know close horizon so not 00:59:08.260 --> 00:59:11.760 just like one customer to 1 000 customers more kind of like you know realistically 00:59:11.760 --> 00:59:14.800 we're not going to get more than like 10 000 customers in the next year so like 00:59:14.800 --> 00:59:18.520 let's build to like 10 million whatever it just didn't make any sense but we still did it. 00:59:19.660 --> 00:59:22.960 And by doing that we actually built a much 00:59:22.960 --> 00:59:26.020 worse system because we were not serving 00:59:26.020 --> 00:59:28.840 those future customers we were serving the 00:59:28.840 --> 00:59:31.700 current customers and the current customers cared about like speed 00:59:31.700 --> 00:59:35.820 of iteration and and you know latency and and 00:59:35.820 --> 00:59:38.660 you know stability and all of that and you can't explain like it's not 00:59:38.660 --> 00:59:41.460 a good answer to tell the customers like i know this call is a bit slower 00:59:41.460 --> 00:59:44.400 than what you expect but it's because we need to support one 00:59:44.400 --> 00:59:47.460 billion people like you but it's like but you don't like yeah 00:59:47.460 --> 00:59:50.680 it's not a good answer so nowadays like 00:59:50.680 --> 00:59:53.620 what we follow is that we want to be able 00:59:53.620 --> 00:59:57.080 to scale 10x our current loads in 00:59:57.080 --> 00:59:59.980 a week like or whatever it is a very short period of time we want 00:59:59.980 --> 01:00:03.100 to see the path to being able to scale you know 01:00:03.100 --> 01:00:05.740 like one order of magnitude immediately but we 01:00:05.740 --> 01:00:09.160 don't care about like scaling 100x or 1000x that we 01:00:09.160 --> 01:00:12.000 assumed we're gonna like have at least like a month 01:00:12.000 --> 01:00:15.580 or two to figure out and then and that's 01:00:15.580 --> 01:00:18.000 what we do kind of like that i mean again by the way we don't it doesn't mean 01:00:18.000 --> 01:00:21.780 that we neuter our existing system we don't build it slow on purpose like we 01:00:21.780 --> 01:00:27.320 try to build it to the best of our ability but the the what we aim for is no 01:00:27.320 --> 01:00:33.160 longer that like a thousand x it's just that like 10x with a clear path to like 100x maybe, 01:00:34.200 --> 01:00:34.740 Does that make sense? 01:00:35.580 --> 01:00:44.160 Yeah. So your advice to your past self would be focus on the now, 01:00:44.480 --> 01:00:48.900 focus on the today rather than focus on what will be further down the road. 01:00:49.040 --> 01:00:52.540 You cross the bridge when you get there and focus on building something that 01:00:52.540 --> 01:00:58.360 is robust and can scale reasonably well instead of going for hypergrowth, by the way. 01:00:58.700 --> 01:01:05.240 But what about the rust side? the advice that you would give when you started on this rust journey. 01:01:06.400 --> 01:01:10.000 I guess one thing is that it's 01:01:10.000 --> 01:01:15.100 okay to clone you know like one thing that i think we did in the beginning way 01:01:15.100 --> 01:01:20.840 too much was just to figure out how to have completely cloneless cloneless setup 01:01:20.840 --> 01:01:25.640 like all the way down the stack and the problem is is that at some point you 01:01:25.640 --> 01:01:29.100 would have a dependency that requires ownership, 01:01:29.980 --> 01:01:33.980 and you're either going to just normally, without all of this cloneless setup, 01:01:34.140 --> 01:01:35.920 you're just going to, would have just passed it the reference, 01:01:36.440 --> 01:01:37.560 the ownership, like all the way down. 01:01:37.700 --> 01:01:40.540 And now you kind of like have to clone. So essentially, you're just, 01:01:40.640 --> 01:01:44.760 it's a premature optimization trying to figure out like the best way to just not clone. 01:01:45.080 --> 01:01:48.240 And like the previous example, you don't have to be degenerate about it. 01:01:48.320 --> 01:01:51.000 Like you don't have to just like, oh, I'm just going to clone anywhere. 01:01:51.120 --> 01:01:52.800 Like no more passing as references. 01:01:53.000 --> 01:01:54.880 No, definitely pass as reference where it makes sense. 01:01:55.340 --> 01:02:00.120 But if you have to spend half an hour trying to figure out the signature for 01:02:00.120 --> 01:02:05.260 this function just to make it so you don't have to clone this fairly small structure, 01:02:05.280 --> 01:02:07.260 don't worry about it. I think that's... 01:02:08.560 --> 01:02:13.060 My mileage may vary depending on the exact situation but i think that that's 01:02:13.060 --> 01:02:15.440 one thing that we would change would. 01:02:15.440 --> 01:02:20.080 I be wrong in summarizing it with keep it simple even if you write rust. 01:02:20.080 --> 01:02:23.380 100 and i think you know 01:02:23.380 --> 01:02:26.200 one of the we kind of talked about like one 01:02:26.200 --> 01:02:29.320 of the vices of the rust ecosystem is the obsession with 01:02:29.320 --> 01:02:32.180 correctness which again is great in most cases but 01:02:32.180 --> 01:02:35.020 you have to remember that sometimes the world outside is 01:02:35.020 --> 01:02:38.040 dirty and like you have you can't it's it's useless 01:02:38.040 --> 01:02:40.900 if you're correct if everything else everyone else is incorrect and you can't 01:02:40.900 --> 01:02:45.620 communicate with them i think the other one is like extreme obsession with performance 01:02:45.620 --> 01:02:48.840 i think it's great and healthy and we should continue that that's you know what 01:02:48.840 --> 01:02:55.220 makes rust so great or part of what makes rust great but we also need to be pragmatic so. 01:02:55.220 --> 01:03:02.240 We have two tips already we have focus on the 10x not the 1000x and we have 01:03:02.240 --> 01:03:08.200 focus on simplicity um cloning is fine for example anything else that you would want to add. 01:03:08.200 --> 01:03:12.520 Yeah so the the other two i mean the first one would be it's very similar to 01:03:12.520 --> 01:03:18.460 the 10x instead of a thousand x one that i gave earlier which is you know don't aim for 100 uptime, 01:03:19.200 --> 01:03:22.180 you know it's kind of like in the beginning where we were just obsessed as you 01:03:22.180 --> 01:03:25.240 said in the beginning as well like we can never go down so we were obsessed 01:03:25.240 --> 01:03:29.660 with just making sure always up the the thing is though the moment we stopped 01:03:29.660 --> 01:03:32.880 obsessing with 100 uptime and just gave ourselves an actually attainable goal, 01:03:33.000 --> 01:03:37.320 which was like five nines of uptime, we actually reached the 100%. 01:03:37.320 --> 01:03:39.620 Like we got to where we wanted to go before. 01:03:40.220 --> 01:03:42.900 And the reason for that is, again, similar to the previous point, 01:03:43.140 --> 01:03:49.120 is when you aim for something that's unattainable, you start doing things that 01:03:49.120 --> 01:03:53.280 are just crazy, that don't add any value, and actually make things more complex. 01:03:53.580 --> 01:03:56.440 So an example, I think it's a fair assumption. 01:03:57.510 --> 01:04:03.650 That we don't need to be up if an asteroid destroys the Earth, 01:04:03.730 --> 01:04:05.210 right? I think that's a fair assumption. 01:04:05.410 --> 01:04:12.010 But if we didn't make that assumption, we would maybe send a server to Mars 01:04:12.010 --> 01:04:16.590 or do something like insane like that, and then we'd have to deal with insane latency and all of that. 01:04:16.790 --> 01:04:20.490 All of a sudden, we added a lot of complexity for our system for really no good 01:04:20.490 --> 01:04:25.190 reason because if Earth is destroyed, I don't think anyone cares if we have a bit of downtime. 01:04:25.190 --> 01:04:28.070 So i think that that is 01:04:28.070 --> 01:04:30.890 one of those areas that i mean again we didn't 01:04:30.890 --> 01:04:33.930 have a server on mars just to be clear but like we were making 01:04:33.930 --> 01:04:36.970 crazy we just like crazy 01:04:36.970 --> 01:04:41.690 complex systems just to support that 0.00001 01:04:41.690 --> 01:04:44.430 percent of i mean just like it didn't make any sense it was 01:04:44.430 --> 01:04:47.250 actually making us less stable because the complexity made things 01:04:47.250 --> 01:04:50.530 worse and i guess the other the other 01:04:50.530 --> 01:04:54.790 like advice i would give my old self is 01:04:54.790 --> 01:04:58.370 really be diligent about avoiding unnecessary 01:04:58.370 --> 01:05:01.870 technical debt so i think well i guess so maybe even yeah 01:05:01.870 --> 01:05:05.050 because i think technical debt is fine technically it's great you take a loan 01:05:05.050 --> 01:05:09.110 on your own on future self in order to move faster now in order to build something 01:05:09.110 --> 01:05:13.990 now i don't think that's necessarily a bad thing a technical debt but there 01:05:13.990 --> 01:05:18.650 are kinds of technical debt that are just unnecessary like naming things poorly 01:05:18.650 --> 01:05:19.830 in the database for example. 01:05:20.590 --> 01:05:24.010 If you have a poorly named table and 01:05:24.010 --> 01:05:26.950 you know that and you're just lazy about changing that in staging or 01:05:26.950 --> 01:05:29.630 in local development essentially what you did 01:05:29.630 --> 01:05:32.550 is that everyone from now on till the future 01:05:32.550 --> 01:05:35.910 of the company you know till the end of the company until 20 years 01:05:35.910 --> 01:05:39.030 in the future will have to deal with 01:05:39.030 --> 01:05:42.090 that poor naming because like naming a date renaming a database table 01:05:42.090 --> 01:05:44.750 is a pain renaming a database column is a pain no one 01:05:44.750 --> 01:05:47.550 is going to bother with doing that it's not the same is 01:05:47.550 --> 01:05:51.910 like just like having poor names in the code which again unnecessary but it's 01:05:51.910 --> 01:05:56.490 fine we can fix it later here it's actually like just an unnecessary you know 01:05:56.490 --> 01:06:00.330 avoidable piece of debt that i would really encourage people to be extremely 01:06:00.330 --> 01:06:03.110 diligent about don't let people hand away if it's like hey it's not a big deal 01:06:03.110 --> 01:06:05.350 just a name no it does matter does. 01:06:05.350 --> 01:06:09.490 It help to be explicit about the things that you store in a table or how would 01:06:09.490 --> 01:06:13.090 you find a good name i know it's a bit of a segue but i'm curious now. 01:06:13.090 --> 01:06:15.890 I think it's really depends on the case but 01:06:15.890 --> 01:06:18.870 just if if someone or more 01:06:18.870 --> 01:06:22.710 than one person on the team find the name confusing i think that's warrant to 01:06:22.710 --> 01:06:26.150 change because people finding something confusing means they're going to go 01:06:26.150 --> 01:06:29.290 down a rabbit hole in a few months from now when they've debug you know when 01:06:29.290 --> 01:06:33.550 they're trying to figure out yeah i mean actually okay one quick example that 01:06:33.550 --> 01:06:36.730 we have we have a few things that are named, 01:06:37.530 --> 01:06:41.350 like restricted to mean that they 01:06:41.350 --> 01:06:44.250 actually have more access because they kind 01:06:44.250 --> 01:06:47.330 of like they have restricted access well in and 01:06:47.330 --> 01:06:50.910 it's so confusing and i get it wrong because we also have limited and 01:06:50.910 --> 01:06:53.790 restricted and another name and sometimes we 01:06:53.790 --> 01:06:57.670 take the the position of the elevated access sometimes we take the position 01:06:57.670 --> 01:07:03.250 of the data and it's it's really confusing that's that costs us like many many 01:07:03.250 --> 01:07:07.010 developer hours over the last of the years and again it's it's not a big deal 01:07:07.010 --> 01:07:11.110 but it is something that's like too big of a pain to change and i wish we were 01:07:11.110 --> 01:07:13.550 just a bit more thoughtful about it in the beginning would. 01:07:13.550 --> 01:07:15.810 Privileged access have been a better choice. 01:07:15.810 --> 01:07:20.170 Yeah so privilege would have been better or just deciding are we talking about 01:07:20.170 --> 01:07:24.270 the data or are we talking about the access just being consistent about that 01:07:24.270 --> 01:07:27.550 and always doing that i would have gone a long way and. 01:07:27.550 --> 01:07:32.910 Finally it's become a bit of a tradition around here to ask that one last question 01:07:32.910 --> 01:07:37.830 do you have a message to the rust community anything that comes to mind. 01:07:37.830 --> 01:07:40.670 So first of all message of love i'm a big fan 01:07:40.670 --> 01:07:43.370 keep up the good work but in terms of a 01:07:43.370 --> 01:07:46.750 message that drives action i think 01:07:46.750 --> 01:07:49.470 we got to fix the compilation time i don't know what we can do 01:07:49.470 --> 01:07:52.290 it's really i think 01:07:52.290 --> 01:07:57.750 it's it is like a bottleneck for a lot of people it is it is a pain and it's 01:07:57.750 --> 01:08:02.250 it even affects you know like id like you know language servers and all of that 01:08:02.250 --> 01:08:06.990 like when it's when things are slow and so we got to fix that thing yeah and 01:08:06.990 --> 01:08:09.710 i know people are working on it so yeah. 01:08:09.710 --> 01:08:13.130 That's that's true let's all wish for faster compile 01:08:13.130 --> 01:08:18.230 times this year and maybe the next couple years it has improved but i i shared 01:08:18.230 --> 01:08:23.950 a sentiment with you so that wraps it up thanks so much for all the insights 01:08:23.950 --> 01:08:29.850 and i can tell that If I had production-level webhooks that I wanted to serve anywhere, 01:08:30.090 --> 01:08:32.510 then I would take a very, very close look at Svix. 01:08:32.890 --> 01:08:37.350 Good job there, and good luck with the future, and thanks for being a guest. 01:08:37.930 --> 01:08:39.050 Thank you. Thank you for having me. 01:08:39.470 --> 01:08:43.210 Rust in Production is a podcast by corrode. It is hosted by me, 01:08:43.510 --> 01:08:46.270 Matthias Endler, and produced by Simon Brüggen. 01:08:46.430 --> 01:08:50.730 For show notes, transcripts, and to learn more about how we can help your company 01:08:50.730 --> 01:08:56.170 make the most of Rust, visit corrode.dev. Thanks for listening to Rust in Production.