WEBVTT 00:00:01.490 --> 00:00:05.830 It's Rust in Production, a podcast about companies who use Rust to shape the 00:00:05.830 --> 00:00:06.690 future of infrastructure. 00:00:07.130 --> 00:00:12.430 My name is Matthias Endler from corrode and today I talk to Jon Gjengset from 00:00:12.430 --> 00:00:16.910 Helsing about keeping critical infrastructure secure and resilient with Rust. 00:00:19.410 --> 00:00:23.050 Unless you've been living under a rock, you know the guest of the show, 00:00:23.230 --> 00:00:25.870 but I will let him introduce himself. 00:00:26.870 --> 00:00:28.110 Jon, happy to have you. 00:00:29.230 --> 00:00:34.270 Thanks, Matthias. So I'm Jon Gjengset. You may know me as @jonhoo online, 00:00:34.470 --> 00:00:38.830 otherwise on the various channels, although YouTube I think is the primary one. 00:00:39.750 --> 00:00:41.630 And I guess, who am I? 00:00:42.850 --> 00:00:46.570 So professionally, I work at a company called Helsing, which I guess we'll talk 00:00:46.570 --> 00:00:50.450 more about today. I work as a principal engineer, which means I end up jumping 00:00:50.450 --> 00:00:55.390 all over the stack, owning wherever the biggest areas of attention are and where 00:00:55.390 --> 00:00:56.690 I can be the highest leverage. 00:00:56.890 --> 00:01:01.910 So I'm not necessarily in one particular team persistently over time, 00:01:02.010 --> 00:01:03.670 but rather seeking out where I'm needed. 00:01:03.990 --> 00:01:07.510 I've worked there for the past three years. Before that, I worked at Amazon. 00:01:07.510 --> 00:01:10.590 I maintained and built their Rust build infrastructure. 00:01:11.310 --> 00:01:17.050 And then before that, I did a PhD at MIT where I built a sort of novel distributed 00:01:17.050 --> 00:01:21.970 systems database project that has since turned into a startup called ReadySet. 00:01:22.730 --> 00:01:26.350 That's sort of the professional side of my career. And then the thing I'm more 00:01:26.350 --> 00:01:30.410 widely known for in the Rust community is the Rust educational materials that 00:01:30.410 --> 00:01:32.970 I make, primarily in the forms of videos on YouTube, 00:01:33.190 --> 00:01:38.570 where I do long form content on building things from scratch in Rust and showing 00:01:38.570 --> 00:01:42.370 people the actual code as we develop it and trying to give you know a realistic 00:01:42.370 --> 00:01:46.930 view of the the the actual development workflow like what does it look like 00:01:46.930 --> 00:01:49.950 when you go from zero to one on a code base in rust. 00:01:51.070 --> 00:01:54.030 Yes and that's exactly the reason why i 00:01:54.030 --> 00:01:56.830 was super excited to have you on the show to do 00:01:56.830 --> 00:01:59.830 this interview and i guess i can speak for everyone 00:01:59.830 --> 00:02:05.170 listening right now when i say thanks so much for all the content for educating 00:02:05.170 --> 00:02:09.730 all of us you're probably one of the most famous rustaceans out there and 00:02:09.730 --> 00:02:15.750 overall from what i can tell an amazing human being always happy to share 00:02:15.750 --> 00:02:16.030 I 00:02:16.030 --> 00:02:17.330 appreciate that thank you! 00:02:19.290 --> 00:02:26.330 And a lot of people know you from this educational context, but maybe not everyone 00:02:26.330 --> 00:02:29.110 knows that you're a principal engineer at Helsing. 00:02:29.430 --> 00:02:32.450 So talk a little bit about your role. 00:02:32.710 --> 00:02:36.770 You mentioned it already, but maybe you can get into the details. 00:02:36.770 --> 00:02:40.150 What do you do there and what is Helsing's responsibility right now? 00:02:40.470 --> 00:02:44.030 Sure. So Helsing is a defense company based in Europe. 00:02:44.350 --> 00:02:50.250 They started fairly recently. So it's only, I want to say, four or five years old. 00:02:50.330 --> 00:02:54.370 I forget the exact start date, but around there. I joined in... 00:02:55.510 --> 00:03:01.430 Sort of towards the end of, or middle to end of 2023, after I moved back to Europe. 00:03:01.930 --> 00:03:07.470 And Helsing operates in sort of across the entire defense spectrum in Europe, 00:03:07.610 --> 00:03:10.910 but they're specifically focused on software-enabled defense. 00:03:11.130 --> 00:03:16.210 So how can we use software as the primary driver for, you know, 00:03:16.650 --> 00:03:20.610 keeping up the deterrence capabilities of especially democracies and especially 00:03:20.610 --> 00:03:23.610 focused on on Europe, given that's where the company is based as well. 00:03:24.630 --> 00:03:28.510 It's already become quite a large company. So in the sense that, 00:03:28.650 --> 00:03:33.210 you know, we're now somewhere in the vicinity of a thousand employees we operate 00:03:33.210 --> 00:03:37.190 in, or we have offices in, I think, six different countries now. 00:03:37.310 --> 00:03:42.690 So we have offices in Estonia and Poland, in the UK, Germany, France. 00:03:43.670 --> 00:03:48.830 Am I missing any? I think that those are the sort of primary office locations we have. 00:03:49.330 --> 00:03:52.390 And then obviously, you know, I work remotely from Norway for the company and 00:03:52.390 --> 00:03:56.430 we have other people working elsewhere as well. But those are the sort of primary locations. 00:03:56.670 --> 00:04:02.350 But for a company that's only four to five years old, that's quite the growth. 00:04:02.670 --> 00:04:06.670 And the, you know, the reality is that that is partially because of the world 00:04:06.670 --> 00:04:10.110 situation and the way things are developing in Europe, where a pretty, 00:04:10.150 --> 00:04:15.690 you know, a pretty severe investment into European defense has been needed. 00:04:16.170 --> 00:04:20.610 And Helsing has sort of seen fit to try to fill at least some of that space. 00:04:20.790 --> 00:04:23.910 And so we operate across land, air, maritime. 00:04:24.710 --> 00:04:31.030 We recently announced some work in space. And so the observation is that by 00:04:31.030 --> 00:04:32.310 focusing on the software, 00:04:32.450 --> 00:04:35.250 there's a lot we can bring to the table and we can bring it to the table pretty 00:04:35.250 --> 00:04:37.050 rapidly because software in 00:04:37.050 --> 00:04:41.770 general has more rapid development cycles than traditional hardware does. 00:04:42.360 --> 00:04:48.360 And so, you know, I, in my work at Helsing, I've been primarily working in the air domain. 00:04:48.600 --> 00:04:53.460 So working with, for example, the capability upgrades for the Eurofighter, 00:04:53.660 --> 00:04:55.520 which is a European jet fighter program. 00:04:55.800 --> 00:05:00.980 And currently I'm working more on the CA-1, which is our recently introduced 00:05:00.980 --> 00:05:03.840 product that is essentially an autonomous UAV. 00:05:04.080 --> 00:05:11.060 And so I'm working on building a lot of the software that's going to underpin that entire stack. 00:05:11.800 --> 00:05:20.660 If you compare Helsing's usage of Rust with AWS, can you see any differences? 00:05:21.480 --> 00:05:24.660 Yeah, I mean, I think there are a number of differences, actually. 00:05:25.000 --> 00:05:26.900 One of them is that Helsing was... 00:05:28.000 --> 00:05:31.040 A sort of rust first company so the it 00:05:31.040 --> 00:05:34.060 was very early on decided that the the entire stack 00:05:34.060 --> 00:05:36.800 there should be rust based and and 00:05:36.800 --> 00:05:42.600 should be you know wherever possible rust would be the language of choice and 00:05:42.600 --> 00:05:46.000 we get we can get into exactly why that was but but that has sort of shaped 00:05:46.000 --> 00:05:48.940 a lot of the software engineering it shaped a lot of the software that we've 00:05:48.940 --> 00:05:54.160 built and the way that we built software whereas at amazon you know amazon was 00:05:54.160 --> 00:05:56.720 originally a sort of java company and, 00:05:57.320 --> 00:05:59.540 some healthy amounts of Perl in there. 00:05:59.700 --> 00:06:04.840 And then over time, it's sort of grown into this polyglot company where there 00:06:04.840 --> 00:06:08.260 are lots of different languages in use in different parts of the company. 00:06:08.500 --> 00:06:12.520 And Rust is obviously one that's, I think it's beyond up-and-coming now. 00:06:12.620 --> 00:06:17.680 It's actually being adopted in pretty serious use cases across AWS in particular. 00:06:19.360 --> 00:06:23.560 But Rust was the sort of up-and-coming competitor. 00:06:23.620 --> 00:06:26.740 It was not the incumbent. and that changes 00:06:26.740 --> 00:06:30.080 how you adopt it right because it means that at amazon there's 00:06:30.080 --> 00:06:33.940 a lot of infrastructure there's a lot of tooling that's not built for rust that 00:06:33.940 --> 00:06:38.360 was built with other languages in mind and where rust now has to make inroads 00:06:38.360 --> 00:06:41.560 into that ecosystem and integrate with all the things that people are used to 00:06:41.560 --> 00:06:45.100 working with whereas the tell thing we can build everything specifically for 00:06:45.100 --> 00:06:49.360 using rust because that is the the primary target language yeah. 00:06:49.360 --> 00:06:57.540 What i get from this is aws had infrastructure before and it was sort of a brownfield 00:06:57.540 --> 00:07:02.160 adoption of rust whereas at Helsing it more or less might have been a greenfield 00:07:02.160 --> 00:07:04.520 adoption i'm not sure if that is true. 00:07:04.520 --> 00:07:07.320 Yeah i think that's right i mean it was a 00:07:07.320 --> 00:07:11.360 is a pretty principled choice from the beginning of the company to say you know 00:07:11.360 --> 00:07:16.140 we're going to be building technology where you know a lot of the technology 00:07:16.140 --> 00:07:22.920 is critical right it's like it it it ends up having implications for life and 00:07:22.920 --> 00:07:25.820 death decisions it ends up having implications for you know, 00:07:27.110 --> 00:07:30.050 the correctness in in the military 00:07:30.050 --> 00:07:32.730 domain like messing things up here is 00:07:32.730 --> 00:07:35.630 very very costly and i don't mean in terms of monetary 00:07:35.630 --> 00:07:40.390 cost right it's just costly in like a human cost and so as a result you need 00:07:40.390 --> 00:07:44.170 to make sure that you build systems that are highly resilient highly reliable 00:07:44.170 --> 00:07:50.070 highly predictable and and robust and rust is one of the mechanisms that we 00:07:50.070 --> 00:07:53.730 wanted to use from very early on to make that be the case. 00:07:54.550 --> 00:07:58.550 And, you know, because the company is relatively young, 00:07:58.810 --> 00:08:04.870 we could also then say we're going to take the attitude here of everything is 00:08:04.870 --> 00:08:08.650 going to be Rust from the get-go rather than say, you know, let's just try out 00:08:08.650 --> 00:08:11.290 different languages and over time figure out what to choose. 00:08:11.410 --> 00:08:13.930 We're just going to say that is what the stack is going to be built in. 00:08:14.070 --> 00:08:16.690 So it's very much a greenfield approach in that way. 00:08:17.360 --> 00:08:23.320 And yet, even if you base everything on Rust, you still need to interface with 00:08:23.320 --> 00:08:26.440 existing libraries that might not be written in Rust. 00:08:26.580 --> 00:08:32.160 You might drive controllers or other hardware that maybe has firmware that isn't 00:08:32.160 --> 00:08:34.220 written in Rust. How is that story like? 00:08:34.220 --> 00:08:41.400 Yeah, and I actually think that's one of the reasons why Rust was able to gain 00:08:41.400 --> 00:08:48.480 such adoption in such a wide set of areas is because its interoperability story is decently good. 00:08:48.480 --> 00:08:52.720 There are things I would still like to see Rust grow on here for sure, 00:08:52.900 --> 00:08:57.380 but it gives you low enough control to be able to write firmware, 00:08:57.840 --> 00:09:00.240 operating systems, embedded devices in it. 00:09:00.400 --> 00:09:05.740 But it also makes it relatively easy to plug Rust into existing code bases, 00:09:05.940 --> 00:09:10.180 whether that is underneath something like Python. 00:09:10.820 --> 00:09:14.520 Python cryptography is a good example here where they are now using Rust under 00:09:14.520 --> 00:09:16.660 the hood for some of the native components, 00:09:17.020 --> 00:09:21.120 but also being able to put Rust above other things where you have an existing 00:09:21.120 --> 00:09:26.860 C code base or even C++ code base, and you want to be able to interface with it from Rust. 00:09:27.080 --> 00:09:31.580 Well, you can also do that. So you can sandwich Rust into wherever in the stack 00:09:31.580 --> 00:09:35.060 you feel like it's appropriate, and then it can grow out from there. 00:09:35.380 --> 00:09:39.580 And that's much harder with some other languages. Like if you have a runtime, 00:09:39.820 --> 00:09:43.460 for example, and the being able to do foreign function interfaces either above 00:09:43.460 --> 00:09:45.400 or below you tends to be more painful. 00:09:46.580 --> 00:09:51.160 Did you see cases where vendors provide Rust SDKs now that it's becoming more popular? 00:09:51.920 --> 00:09:57.960 It's a mix. There are some domains where we're seeing more interest in supporting 00:09:57.960 --> 00:09:59.580 Rust from industry vendors. 00:09:59.800 --> 00:10:06.680 And then there's others where it's still, you know, all like all C++ or you'll 00:10:06.680 --> 00:10:10.940 see places like the, if you look at the Terraform Kubernetes ecosystem, 00:10:10.940 --> 00:10:14.440 a lot of that is Go and they're not really planning or they're not. 00:10:14.440 --> 00:10:18.820 They don't seem particularly interested in saying we're going to also provide Rust bindings now. 00:10:19.060 --> 00:10:22.920 And then you have, you know, more embedded ecosystems where maybe they already 00:10:22.920 --> 00:10:26.680 have an existing sort of stack for development that's C-based maybe, 00:10:26.700 --> 00:10:29.280 and they don't necessarily want to change that. 00:10:29.460 --> 00:10:34.780 But I do think we're seeing vendors now think more about maybe a Rust SDK is 00:10:34.780 --> 00:10:36.040 also something we want to provide. 00:10:36.400 --> 00:10:39.640 I mean, Amazon is a good example where they, you know, now have a Rust SDK. 00:10:39.640 --> 00:10:43.420 And it was because there was sufficient demand for, you know, 00:10:43.480 --> 00:10:47.780 we want to build things that use AWS from Rust. So please let us do that. 00:10:48.040 --> 00:10:53.740 Maybe for some additional context, could you list the things where Rust is front 00:10:53.740 --> 00:10:56.560 and center at Helsing? What makes the stack? 00:10:57.780 --> 00:11:03.040 So Rust is used throughout our entire stack, essentially. 00:11:03.360 --> 00:11:08.460 So we use it for obviously anything that's backend services and the like, 00:11:08.540 --> 00:11:12.180 but we also use it for anything that's close to edge devices. 00:11:12.180 --> 00:11:16.380 So if you are writing code that's going to run on a UAV, on a drone, 00:11:16.580 --> 00:11:20.500 on an underwater autonomous submersible, whatever it might be, 00:11:21.200 --> 00:11:26.040 realistically, you have to use a language there where you have fairly tight 00:11:26.040 --> 00:11:31.140 control over things like power usage, performance, predictable performance over 00:11:31.140 --> 00:11:35.000 time, low overhead compute, and hardware control. 00:11:35.000 --> 00:11:38.680 And so we use it a lot there in sort of embedded or bare metal systems, 00:11:38.840 --> 00:11:42.320 but also things that are almost embedded or bare metal, right? 00:11:42.460 --> 00:11:47.100 So things like, you know, essentially single application, but still runs on 00:11:47.100 --> 00:11:51.620 Linux, but on a very particular compute board would still be something where 00:11:51.620 --> 00:11:53.000 we write those applications in Rust. 00:11:53.000 --> 00:11:58.620 We use it for a lot of our tooling is built in Rust a lot of our networking 00:11:58.620 --> 00:12:03.040 technologies are built in Rust in fact it might be easier to sort of list the 00:12:03.040 --> 00:12:06.540 things that are not done in Rust which I'd say, 00:12:07.240 --> 00:12:10.680 there's primarily three categories so there's. 00:12:12.000 --> 00:12:16.840 Web frontends, we tend to build in TypeScript instead, because you can do some 00:12:16.840 --> 00:12:20.180 of it in WebAssembly, but realistically, TypeScript is where you're going to 00:12:20.180 --> 00:12:21.500 get the most mileage here. 00:12:22.000 --> 00:12:27.920 And then for AI research, like for the people actually working on the underlying 00:12:27.920 --> 00:12:32.100 machine learning algorithms and training and the like, there we try to enable 00:12:32.100 --> 00:12:35.120 people to use Python because that is where they're the most productive. 00:12:35.380 --> 00:12:38.060 That's where a lot of the state-of-the-art research happens. 00:12:38.280 --> 00:12:41.840 And that's where a lot of the existing tooling and libraries and such exist. 00:12:42.000 --> 00:12:46.280 And so we don't really want to move all of that to Rust, even though we think 00:12:46.280 --> 00:12:47.420 it's probably feasible. 00:12:47.840 --> 00:12:53.480 It's not clear that it gives the bang for the buck in that area. 00:12:53.480 --> 00:12:57.800 And then the last is we have some things around infrastructure that's in Go 00:12:57.800 --> 00:13:03.360 I mean I mentioned Terraform and Kubernetes and stuff so there's like some stuff like that where, 00:13:03.960 --> 00:13:09.600 Go has the best support ecosystem for writing things in that environment but 00:13:09.600 --> 00:13:14.160 I'd say that's a that's a pretty small minority I'd say sort of in in order 00:13:14.160 --> 00:13:19.800 of in order of adoption percentage I think it's Rust by far and then Python 00:13:19.800 --> 00:13:23.200 and then TypeScript and then there's like a tiny bit of Go there and then there's like 00:13:23.480 --> 00:13:28.600 There's always Bash because CI and everything, but I'd say those are the primary languages. 00:13:29.220 --> 00:13:34.800 According to our internal statistics, a large percentage of our listeners do 00:13:34.800 --> 00:13:37.220 use Rust in production in some way. 00:13:37.560 --> 00:13:42.060 But what does it feel like to work in a codebase where Rust is truly the answer 00:13:42.060 --> 00:13:43.980 everywhere, not just in one layer? 00:13:45.180 --> 00:13:50.720 It's very convenient, right? Because it means that whenever I go to any codebase 00:13:50.720 --> 00:13:53.840 across the company, chances are I know how to read that codebase. 00:13:53.960 --> 00:13:55.140 Chances are it's written in Rust. 00:13:55.760 --> 00:13:59.240 It's not just about being able to read the code. It's also the structure is 00:13:59.240 --> 00:14:01.960 kind of predictable because they're all cargo projects. 00:14:02.180 --> 00:14:04.600 They all have, you know, the layout you would expect from that. 00:14:04.760 --> 00:14:10.140 It also means we can write tooling that is specifically designed for Rust projects 00:14:10.140 --> 00:14:13.360 and then, you know, add support for Python, for TypeScript. 00:14:13.560 --> 00:14:18.820 But we can build things specifically for, as an example, we have an internal 00:14:18.820 --> 00:14:24.680 linter that we use to catch things that are more company preferences rather. 00:14:24.680 --> 00:14:28.680 So we run Clippy, and then we also run this other thing that tries to lint for 00:14:28.680 --> 00:14:32.520 things that are preferences we have for software engineering in Rust. 00:14:32.980 --> 00:14:38.200 And so this can be things like looking for preferred libraries we have for things 00:14:38.200 --> 00:14:39.480 like logging, for example. 00:14:39.760 --> 00:14:46.760 Or it can be things like how we believe you should take internal dependencies, 00:14:47.000 --> 00:14:49.000 like how they should be expressed in your Cargo.toml. 00:14:49.000 --> 00:14:54.360 Or it can be things like, you know, how we think you should be using expect, 00:14:54.560 --> 00:14:58.680 like the sentence structure we believe you should have inside of expect statements. 00:14:58.840 --> 00:15:02.460 Like there's a bunch of that sort of stuff that you can do linting that I don't 00:15:02.460 --> 00:15:06.900 think necessarily makes sense for the, like, it wouldn't make sense for upstreaming 00:15:06.900 --> 00:15:09.520 into Clippy, but it does make sense for enforcing things internally. 00:15:09.520 --> 00:15:13.220 And the fewer languages you have, the more you can build tooling specifically 00:15:13.220 --> 00:15:17.040 for those languages to encourage the sort of software engineering excellence 00:15:17.040 --> 00:15:19.460 practices you want to have for that language. 00:15:20.080 --> 00:15:25.140 Yeah, I'm sure a lot of people will be interested in taking a peek at that. 00:15:25.540 --> 00:15:30.220 So please open source it if you can, even if you don't want to upstream it. 00:15:30.220 --> 00:15:36.720 I think it's mostly uninteresting in the sense that Clippy already has a relatively 00:15:36.720 --> 00:15:41.600 straightforward mechanism where you can basically implement your own lints. 00:15:41.800 --> 00:15:45.020 And so our internal tool is just that. 00:15:45.160 --> 00:15:50.400 It's just a Clippy, but where all the lints are replaced with things we decided we wanted to look for. 00:15:50.400 --> 00:15:55.860 And where there are things that we think are actually useful to Rust users more 00:15:55.860 --> 00:16:01.640 broadly, then we would obviously open source those or upstream those into Clippy itself. 00:16:01.920 --> 00:16:06.480 But for a lot of the others that are just like encoding of our software quality 00:16:06.480 --> 00:16:10.820 standards, it's like, I don't actually think they're all that interesting outside 00:16:10.820 --> 00:16:15.860 of the company. And there's certainly nothing about the linting engine itself 00:16:15.860 --> 00:16:16.820 that's interesting, right? 00:16:16.900 --> 00:16:22.840 Because it is just the clippy tooling that exists that we've added our own rules to, if you will. 00:16:23.540 --> 00:16:27.800 Which sort of begs the question, do you use that tool across the stack? 00:16:27.820 --> 00:16:32.180 Or do you make differences between embedded developers and backend developers? 00:16:32.400 --> 00:16:33.420 Do they use Rust differently? 00:16:34.240 --> 00:16:38.040 No, that tool is used across the stack. It's not currently mandated. 00:16:38.240 --> 00:16:42.480 So it's a thing that you can choose to add to your CI. And then we are strongly 00:16:42.480 --> 00:16:44.500 encouraging everyone to use it in their CI. 00:16:44.940 --> 00:16:50.020 But most of the rules here are around things that are good practice, 00:16:50.020 --> 00:16:51.800 no matter where in the stack you operate. 00:16:52.180 --> 00:16:57.320 This can be things like if you have an error type that is, let's say, 00:16:57.480 --> 00:17:02.220 error result, and you are propagating that error type using the question mark 00:17:02.220 --> 00:17:05.640 operator, you should have a call to dot context before it. 00:17:06.420 --> 00:17:11.920 Right right and that makes sense for for any code base you're operating in the 00:17:11.920 --> 00:17:17.520 same thing like we can include encode rules like always prefer dot context over 00:17:17.520 --> 00:17:21.880 dot wrap error or vice versa right but we can at least encode that we believe 00:17:21.880 --> 00:17:23.540 it should always be one or always be the other, 00:17:24.220 --> 00:17:27.780 and so it's like those kinds of things more so 00:17:27.780 --> 00:17:30.960 than you know it's harder to write a lint for 00:17:30.960 --> 00:17:35.500 i don't know how you should handle back pressure in an application which is 00:17:35.500 --> 00:17:38.300 the kind of thing that would differ between embedded development and cloud development 00:17:38.300 --> 00:17:42.880 you can't really write a lint for that at the clippy level it's more it's almost 00:17:42.880 --> 00:17:47.920 more architectural and so those things are not things we lint for do. 00:17:47.920 --> 00:17:50.680 You have an example for when dot context is helpful. 00:17:50.680 --> 00:17:53.720 Oh i mean i love dot context i use it everywhere 00:17:53.720 --> 00:17:56.440 the the thing that context gives you is the 00:17:56.440 --> 00:17:59.460 ability to you know when you propagate an error up 00:17:59.460 --> 00:18:02.300 through the stack that error might just be 00:18:02.300 --> 00:18:05.120 something like permission denied or not found or something 00:18:05.120 --> 00:18:08.400 and if you just propagate it with question mark then at 00:18:08.400 --> 00:18:11.320 the point where you emit the error you know in your main in 00:18:11.320 --> 00:18:14.200 your binary or something you would just get an 00:18:14.200 --> 00:18:18.720 error printed that says file not found or permission denied and that's completely 00:18:18.720 --> 00:18:23.400 useless and if by adding context you can include not just information about 00:18:23.400 --> 00:18:27.640 like which file was being accessed but why was that file being accessed so imagine 00:18:27.640 --> 00:18:31.740 things like you have a imagine you have config files that have include directives. 00:18:31.920 --> 00:18:34.760 And so you might be like three levels down in an include hierarchy, 00:18:34.760 --> 00:18:37.920 and then you reach a file you're not allowed to read. 00:18:38.080 --> 00:18:41.940 Well, then the question is, which file included the one that said that you should read this file? 00:18:42.200 --> 00:18:45.820 So even if you had the file name without the context, the file name might be 00:18:45.820 --> 00:18:48.960 like, well, I don't know why foo.bar is being included. 00:18:49.160 --> 00:18:51.820 That's not any of the files that are listed in my top level config. 00:18:52.260 --> 00:18:54.060 And so the context allows you to give. 00:18:54.750 --> 00:18:58.910 Both things like data context, but also programmatic context, 00:18:59.090 --> 00:19:02.210 like why are we loading a config file here in the first place? 00:19:02.490 --> 00:19:06.650 Or in which part of the application did we load this configuration? 00:19:06.930 --> 00:19:10.850 Like was it, you know, I think that's loaded at startup, or maybe it's like 00:19:10.850 --> 00:19:13.870 the code that handles configuration changes at runtime. 00:19:14.110 --> 00:19:17.590 They both end up reading the config files, but which of them failed with this error? 00:19:17.990 --> 00:19:22.510 And so in general, adding context like this is very, very helpful for making 00:19:22.510 --> 00:19:27.450 your errors actually be actionable where they are where they're emitted yeah. 00:19:27.450 --> 00:19:33.410 And it also works across different crates even in the same namespace or outside. 00:19:33.410 --> 00:19:36.190 Um well i mean context is just 00:19:36.190 --> 00:19:40.010 the the way this is set up is if you use anyhow or you use eyre or you use miette 00:19:40.230 --> 00:19:46.470 and you have this sort of opaque error type dot context allows you to a take 00:19:46.470 --> 00:19:50.630 such an opaque error type and turn it into another opaque error that has the 00:19:50.630 --> 00:19:54.790 original one plus this additional context as a part of the chain. 00:19:55.150 --> 00:20:00.370 But it also lets you take an error that is not one of these opaque errors, 00:20:00.510 --> 00:20:04.470 like one that comes out of some external library, and then chain this context 00:20:04.470 --> 00:20:07.250 on top of that error and turn it into one of the opaque errors. 00:20:07.830 --> 00:20:10.990 It obviously doesn't work so well if you have enumerated errors. 00:20:11.110 --> 00:20:15.010 So if you have ones that are like, I have an error enum, and these are the variants, 00:20:15.290 --> 00:20:19.910 then you can't easily call dot context on that unless you're willing to erase 00:20:19.910 --> 00:20:25.290 the the concrete type of that error and turn it into an opaque error like an anyhow error. 00:20:25.290 --> 00:20:26.050 How 00:20:26.050 --> 00:20:28.290 high is the code reuse at Helsing 00:20:29.780 --> 00:20:32.620 I think it depends on where in the stack you look, right? 00:20:32.780 --> 00:20:43.380 So we have projects that span from relatively new sort of experimental, 00:20:43.380 --> 00:20:44.840 we're trying something out, 00:20:45.240 --> 00:20:49.160 to this is in a product and has been in development for several years. 00:20:49.420 --> 00:20:53.620 And the amount of code reuse you do in one versus the other is pretty significant. 00:20:54.100 --> 00:20:59.680 It's also the amount of code reuse you do within the domain versus across domains varies. 00:20:59.780 --> 00:21:08.420 So there are more commonalities between something like a UAV and a strike drone. 00:21:08.620 --> 00:21:16.700 Those have a lot more similar components than, let's say, a radar analysis system and a submersible. 00:21:16.940 --> 00:21:22.580 Those don't share as much in common. So it's hard to give you one number for the amount of reuse. 00:21:22.580 --> 00:21:27.620 But we do try to lean pretty heavily into if you've built something that is 00:21:27.620 --> 00:21:31.220 useful elsewhere at the company or indeed outside of the company, 00:21:31.420 --> 00:21:33.280 then build it as a reusable library. 00:21:33.760 --> 00:21:38.960 And so, you know, you've already seen some of this come out on the public side of Helsing. 00:21:39.020 --> 00:21:42.780 So we have some open source repositories like we have a tool called buffrs, 00:21:42.840 --> 00:21:47.700 which is basically a package manager for protobuf files. 00:21:47.700 --> 00:21:50.680 So it lets you take a collection of protobuf files, 00:21:50.820 --> 00:21:56.460 create basically a proto.toml, which is equivalent to a cargo.toml that says, this is the version, 00:21:56.600 --> 00:21:59.620 this is the package name, and you can take dependencies on other protos and 00:21:59.620 --> 00:22:06.080 it resolves those and runs proto and gives you the flattened set of all the 00:22:06.080 --> 00:22:08.180 files and resolves the dependencies between them. 00:22:08.800 --> 00:22:15.800 We have a tool called Sguaba, which is a library for doing spatial math, 00:22:16.000 --> 00:22:20.140 like rigid body dynamics or rigid body transformations in a type-safe manner. 00:22:20.320 --> 00:22:23.860 And this is something that we use across a lot of our different code bases where 00:22:23.860 --> 00:22:28.020 it's a real pain to get them right once, and you don't want every team to have 00:22:28.020 --> 00:22:29.940 to get them right again and again and again. 00:22:29.960 --> 00:22:34.300 So we build it once, and then we make that be a central library, 00:22:34.480 --> 00:22:38.280 a central utility that every team can make use of, and then we also open source it. 00:22:38.800 --> 00:22:42.280 And then we have other versions of this internally, like we have some tooling 00:22:42.280 --> 00:22:47.820 for the Avro ecosystem where anyone who uses that internally now has access to our tooling. 00:22:48.000 --> 00:22:51.000 And some of that we might open source, some of it we've already open sourced. 00:22:51.380 --> 00:23:00.360 And so I'd say there's a pretty broad swath of reuse that's pretty intentional. 00:23:00.880 --> 00:23:08.520 It's something where we see that there's a lot of cost to telling every team to reinvent the wheel. 00:23:08.960 --> 00:23:13.160 And it's cost not just in terms of engineering time, but also in terms of correctness, right? 00:23:13.460 --> 00:23:17.200 It means that you only manifest the bug once, you only fix it once, 00:23:17.340 --> 00:23:21.600 rather than every team having to wrangle with the same complexities and the same bugs over time. 00:23:22.300 --> 00:23:27.080 Yeah, that's also what I like about Rust in general, because undefined behavior 00:23:27.080 --> 00:23:33.220 is a thing that you can centralize and then fix once, and then the entire ecosystem profits from the fix. 00:23:33.960 --> 00:23:36.860 Yeah and i mean this is partially a property of just having a 00:23:36.860 --> 00:23:42.020 good packaging system right like having a package manager like cargo means it's 00:23:42.020 --> 00:23:45.240 pretty easy to turn something into a library it's pretty easy to take a dependency 00:23:45.240 --> 00:23:50.540 on it and so that incentivizes doing this kind of of sharing which would be 00:23:50.540 --> 00:23:55.360 more annoying and at least some other language ecosystems yeah. 00:23:56.440 --> 00:24:03.220 Also, thanks a lot. I always like it when companies open source their work. It's super amazing. 00:24:03.620 --> 00:24:08.320 And Squabble is certainly an amazing library too. You gave a talk about it. 00:24:08.500 --> 00:24:09.940 We will link to it in the show notes. 00:24:10.730 --> 00:24:15.150 And can you allude a little bit more to buffers? 00:24:15.450 --> 00:24:18.930 Why is it important to have a package manager for protobuf definitions? 00:24:19.050 --> 00:24:22.670 You must really like protobuf definitions and probably use them everywhere. 00:24:23.110 --> 00:24:30.290 Well, so it comes as a pretty natural outcome of not having a monorepo, right? 00:24:30.430 --> 00:24:36.610 Because you end up with different teams having protobuf files for their configuration 00:24:36.610 --> 00:24:38.950 or their data exchange or whatever it might be. 00:24:38.950 --> 00:24:44.690 And then you have some other team that also wants to make use of that team's protobuf files, 00:24:45.430 --> 00:24:49.730 well then either you need to like be in the same repository as them or you need 00:24:49.730 --> 00:24:54.150 to copy paste the files or you need to have a mechanism for publishing your 00:24:54.150 --> 00:24:58.070 protofiles and then getting them into another codebase, 00:24:58.090 --> 00:25:02.690 at which point you need versioning because you're not tagging a particular commit 00:25:02.690 --> 00:25:04.250 of it, you need the version of it. 00:25:04.490 --> 00:25:09.110 And thus you start getting transitive dependencies where maybe there's a shared 00:25:09.110 --> 00:25:13.610 definition for maybe just data types even, so not full structure of gRPCs, 00:25:13.670 --> 00:25:15.690 just some of the core data types. 00:25:15.970 --> 00:25:20.870 And you want those to be shared across two different repositories inside of one team. 00:25:20.990 --> 00:25:23.930 And then they both take your dependencies on that. And then anyone who takes 00:25:23.930 --> 00:25:26.750 a dependency on either now also needs the transitive dependency. 00:25:27.090 --> 00:25:31.230 And so you very quickly run into the situation of we basically need a package 00:25:31.230 --> 00:25:33.510 manager here and we need versioning, we need packages. 00:25:33.890 --> 00:25:35.910 And that's what Buffers was built to solve. 00:25:36.550 --> 00:25:38.290 What do you use protobuf for internally? 00:25:40.630 --> 00:25:47.530 So we tend to like having sort of textual code representations of protocols 00:25:47.530 --> 00:25:53.370 because it makes it a lot easier to decouple the two sides of any given communication pattern. 00:25:54.370 --> 00:25:57.990 And, you know, protobuf is one way to do that. Avro is another, 00:25:58.250 --> 00:26:00.470 and there exist many others as well. 00:26:00.690 --> 00:26:03.310 It also means that it's easier to, 00:26:04.290 --> 00:26:08.810 Once you create that protocol divide, and if you encode the protocol separately 00:26:08.810 --> 00:26:10.550 from any given code base, 00:26:10.790 --> 00:26:14.970 it now makes it easier as well to change the technology choices on either side, 00:26:14.970 --> 00:26:18.510 or just completely reinvent one side by rebuilding it from scratch, 00:26:18.670 --> 00:26:20.630 but being able to reuse the definitions. 00:26:21.530 --> 00:26:25.710 Why Protobuffer specifically? Specifically, it's a pretty mature toolchain and 00:26:25.710 --> 00:26:31.450 ecosystem, and it has a proven track record of working well, 00:26:31.570 --> 00:26:34.110 being efficient, having support for many languages. 00:26:34.930 --> 00:26:39.910 And for that reason, I think it's a pretty obvious default choice. 00:26:40.390 --> 00:26:43.470 But as you'll see, or as you might have seen from my streams, 00:26:43.590 --> 00:26:47.050 for example, there are things where we use Avro instead of protobuf. 00:26:47.050 --> 00:26:51.350 And the rationale for that is there are some features that Avro has that Protobuf 00:26:51.350 --> 00:26:56.430 does not, where if for your particular use case, those features are worthwhile, 00:26:56.610 --> 00:26:58.910 well, then you should pick the technology that has those. 00:26:59.150 --> 00:27:02.870 So as an example, Avro support for this thing called logical types, 00:27:03.090 --> 00:27:09.210 where you can annotate particular fields of a type with, you know, this is a F64, 00:27:09.710 --> 00:27:14.790 but I'm going to annotate it with the logical type velocity in meters per second. 00:27:14.790 --> 00:27:17.730 And protobuf doesn't really have that 00:27:17.730 --> 00:27:21.150 mechanism for adding sort of richness to the the types 00:27:21.150 --> 00:27:23.930 of fields there's like you you 00:27:23.930 --> 00:27:27.990 can kind of hack your way there in protobuf but in abro that's just directly 00:27:27.990 --> 00:27:33.710 supported by the tooling it also has better support for streams of blobs in 00:27:33.710 --> 00:27:38.170 protobuf you don't really have that in protobuf you have it in gRPC but we don't 00:27:38.170 --> 00:27:42.270 necessarily use these protocol definitions for RPC mechanisms. 00:27:42.270 --> 00:27:46.910 We often use them to represent just the data format that's being exchanged in 00:27:46.910 --> 00:27:48.950 various different formats and protocols. 00:27:49.250 --> 00:27:52.890 And so just because we're using protobuf does not mean we're using gRPC everywhere. 00:27:53.130 --> 00:27:57.730 Same thing for Avro. We might be using the Avro IDL for expressing data types 00:27:57.730 --> 00:28:03.110 for protocols. We're not necessarily using the RPC mechanisms that are built on top of Avro. 00:28:03.690 --> 00:28:06.270 Is it time to write an Avro package manager now? 00:28:07.560 --> 00:28:11.940 Maybe. It's not impossible. I mean, running a package manager, 00:28:11.940 --> 00:28:17.940 if it is as simple as, you know, assign a version, create a bundle, 00:28:18.140 --> 00:28:20.800 upload it somewhere, is not that bad. 00:28:21.120 --> 00:28:26.040 And I mean, if you look at buffers, you know, there is some amount of complexity 00:28:26.040 --> 00:28:30.220 there, but it's not, you know, monumental. And we could probably pretty easily 00:28:30.220 --> 00:28:33.260 take buffers and create an Avro version of buffers. 00:28:34.140 --> 00:28:37.820 The thing where it starts to get really gnarly is when you want sophisticated 00:28:37.820 --> 00:28:40.040 semantic versioning resolution, for example. 00:28:40.940 --> 00:28:44.200 There's been some really cool work on... 00:28:44.720 --> 00:28:51.180 There's this effort called PubGrub, which is essentially trying to write a version 00:28:51.180 --> 00:28:54.560 resolver that can be reused across packaging ecosystems, 00:28:54.560 --> 00:29:01.620 across different types of both specifier semantics in your dependency thing, thing, 00:29:02.040 --> 00:29:04.920 but also write it in such a way that it's reusable. 00:29:05.140 --> 00:29:08.600 So you could use it in NPM, you could use it in cargo, you could use it in whatever 00:29:08.600 --> 00:29:14.260 package manager you dream of, and just get really rich resolution of package 00:29:14.260 --> 00:29:17.080 dependencies, because it turns out that's a fairly complicated problem. 00:29:17.360 --> 00:29:21.580 And then PubGrab also tries to give you good error messages from resolver failures, 00:29:21.740 --> 00:29:24.780 which tends to be something that many package managers struggle with, 00:29:24.780 --> 00:29:29.000 because they build version resolution in this kind of ad hoc way. 00:29:29.860 --> 00:29:31.380 And with PubGrub, 00:29:32.090 --> 00:29:37.710 Cargo hasn't adopted PubGrub yet, but it's sort of on the long-term roadmap. 00:29:37.710 --> 00:29:39.870 And it has been for a while, so don't hold your breath. 00:29:40.230 --> 00:29:45.450 But it does mean that, for example, buffers as of today only has exact version lookups. 00:29:45.670 --> 00:29:52.590 So if you take a dependency on 1.2.3 and 1.2.4 is released, then buffers will not pick it up. 00:29:52.730 --> 00:29:58.130 But if we integrated PubGrub into buffers, we would get this version resolution 00:29:58.130 --> 00:30:02.490 behavior that we would say, the default should be semantic version like caret 00:30:02.490 --> 00:30:03.910 matching like what Cargo does. 00:30:04.690 --> 00:30:11.530 And at that point, moving to say Buffer's version that supports Avro feels like 00:30:11.530 --> 00:30:18.010 it shouldn't be that bad because there's not too much that is protobuf specific inside of Buffer's. 00:30:18.130 --> 00:30:23.510 It is more like a collection of files with some metadata that annotates version 00:30:23.510 --> 00:30:25.410 and dependencies and then being 00:30:25.410 --> 00:30:28.830 able to bundle those up into tarballs that you can publish somewhere. 00:30:29.410 --> 00:30:33.610 And if you squint at it, that's like most of what many package managers are. 00:30:34.490 --> 00:30:39.950 Okay, so we understand that management of those protobuf definitions is really 00:30:39.950 --> 00:30:42.790 important for housing. But what do you use it in the first place? 00:30:43.990 --> 00:30:50.690 So we do use gRPC for some parts of the tech stack, although gRPC tends to be 00:30:50.690 --> 00:30:55.270 best suited for environments where communication is fairly reliable and predictable. 00:30:55.270 --> 00:30:58.790 And you have like TCP, you have stable IP networks and the like. 00:30:58.790 --> 00:31:02.330 So there we use gRPC, and then protobuf is a good way to make use of it. 00:31:02.670 --> 00:31:07.390 But protobuf is also useful for just describing data formats. 00:31:08.450 --> 00:31:10.470 Data formats is the wrong word, but like... 00:31:12.530 --> 00:31:17.970 The structure of messages that are going to go over a network or in some cases 00:31:17.970 --> 00:31:22.490 that will go to disk, although protobuf tends to be less well-suited for that, 00:31:22.690 --> 00:31:23.890 but certainly for anything that 00:31:23.890 --> 00:31:27.230 has to go over a network, but it doesn't necessarily need to go over gRPC. 00:31:27.790 --> 00:31:33.870 So for example, there are especially closer to the edge networks that we have 00:31:33.870 --> 00:31:39.390 where you have drones flying around where there's radios with very limited bandwidth, 00:31:39.390 --> 00:31:42.370 connections that come and go you get jammed so 00:31:42.370 --> 00:31:45.490 you lose connectivity or your bandwidth gets severely reduced because 00:31:45.490 --> 00:31:48.430 you're flying through a zone where there's a lot of interference whatever 00:31:48.430 --> 00:31:53.010 it might be in those environments gRPC is not going to work like realistically 00:31:53.010 --> 00:31:59.210 you can't use tcp it will like the the network is simply too poor and too dynamic 00:31:59.210 --> 00:32:03.890 for that to work plus you want to make use of things like you know if you have 00:32:03.890 --> 00:32:07.890 a radio network it's inherently broadcast and so you send one packet, 00:32:08.190 --> 00:32:11.730 you want to make it useful to as many other peers in your network as possible. 00:32:12.070 --> 00:32:15.090 And so that's an example where we're not using gRPC, we're not using TCP. 00:32:15.410 --> 00:32:19.730 And in fact, we've built our own stack that is reliant on CRDTs, 00:32:19.730 --> 00:32:21.870 on conflict-free replicated data types, 00:32:22.010 --> 00:32:28.050 that tries to build a distributed network where you can still get reliable exchange 00:32:28.050 --> 00:32:31.830 of information, even in the presence of severe packet loss, 00:32:32.070 --> 00:32:34.050 network reordering, packet reordering, 00:32:34.470 --> 00:32:38.390 node sending, frequent updates over time, and you want to make sure you only 00:32:38.390 --> 00:32:42.850 accumulate the updates that are newer, that old data gets erased when new data 00:32:42.850 --> 00:32:45.010 replaces it, even if you get the packets out of order. 00:32:45.810 --> 00:32:47.770 Like this sort of very... 00:32:49.500 --> 00:32:54.120 Traditional distributed systems when they're not in a cloud environment setting. 00:32:54.860 --> 00:33:00.040 But there, for the CRDT stuff we've built, we're still using protobuf for the 00:33:00.040 --> 00:33:01.120 actual data definitions. 00:33:01.460 --> 00:33:05.180 So the definitions of the schema, effectively, of what an application, 00:33:05.580 --> 00:33:10.100 what data an application might write, might read, what gets sort of stored and 00:33:10.100 --> 00:33:13.560 sent, all of that is still in protobuf files because, again, 00:33:13.720 --> 00:33:15.100 it's something that people know. 00:33:15.300 --> 00:33:19.580 The tooling is good around it. You know, it has editor support and all of that stuff. 00:33:19.720 --> 00:33:23.260 We have buffers for being able to give you versioning over those schemas. 00:33:23.320 --> 00:33:28.260 And so it makes a lot of sense to reuse that ability to describe your protocols, 00:33:28.400 --> 00:33:34.080 your data definitions, even though the underlying exchange technology is very different. 00:33:35.980 --> 00:33:40.400 And what are reasons for using CRDTs specifically? 00:33:41.240 --> 00:33:44.540 You mentioned a few things already, but what's the bird's eye view? 00:33:44.880 --> 00:33:49.020 When would you use it? when wouldn't you use it and maybe what is it in the first place. 00:33:49.020 --> 00:33:52.280 Yeah so so crdts 00:33:52.280 --> 00:33:55.220 are at the the very basic level 00:33:55.220 --> 00:33:59.140 algorithms or it's 00:33:59.140 --> 00:34:06.940 a it's a an exchange data type so it's a it's a data type that comes with some 00:34:06.940 --> 00:34:13.180 algorithms that define what to do when these messages are exchanged over a network 00:34:13.180 --> 00:34:17.520 or between peers in whatever way they are, such that if. 00:34:18.440 --> 00:34:20.860 Imagine you have set up a distributed system where you have, 00:34:20.920 --> 00:34:26.800 let's say, three nodes in that system, nodes A, B, and C, and node A and B concurrently 00:34:26.800 --> 00:34:31.500 make edits to some underlying document or whatever. 00:34:31.720 --> 00:34:35.220 Let's use document as a good example. The shared document between A, 00:34:35.320 --> 00:34:39.260 B, and C, A is making edits to the document, B is making edits to the document, 00:34:39.280 --> 00:34:43.180 and they can't talk to each other, but they're able to send packets to C. 00:34:43.720 --> 00:34:49.440 C is now going to observe the changes from both A and B. how does it reconcile the changes from A and B? 00:34:49.960 --> 00:34:57.020 And CRDTs are the data type plus algorithms that make C able to reconcile those 00:34:57.020 --> 00:35:04.160 changes in such a way that if A and B's edits were not conflicting with each other, 00:35:04.300 --> 00:35:08.640 then there is no conflict observed by C. It just observed both edits. 00:35:08.820 --> 00:35:14.140 And if they do conflict with each other, then C has a conflict resolution strategy 00:35:14.140 --> 00:35:18.940 in the sense that it can detect that there was a conflict and it can decide 00:35:18.940 --> 00:35:23.700 what to do about that conflict in such a way that afterwards there is no conflict. 00:35:25.840 --> 00:35:28.760 And so imagine for example that your document is like a key value 00:35:28.760 --> 00:35:31.600 store if a writes to key foo and b writes to 00:35:31.600 --> 00:35:35.540 key bar then all the edits to foo and all the edits to bar should just come 00:35:35.540 --> 00:35:41.840 into c and there should be no problem if a deletes foo and b updates foo so 00:35:41.840 --> 00:35:46.480 the same key and then they send to c then c is now going to have to decide what 00:35:46.480 --> 00:35:49.220 to do when it observes a delete and an update at the same time. 00:35:49.520 --> 00:35:54.380 And the CRDT would be something like, you know, it both informs you about the 00:35:54.380 --> 00:35:58.440 metadata you need to add to the packets to detect that these are the same key and they happened 00:35:58.880 --> 00:36:02.120 concurrently with each other rather than, let's say, the update happened first 00:36:02.120 --> 00:36:04.800 and the delete happened after, in which case you should always take the delete. 00:36:05.000 --> 00:36:08.700 But if they actually happen concurrently, which the CRDT metadata will tell you, 00:36:08.940 --> 00:36:16.460 then the CRDT algorithm will tell C whether it should prefer keeping the updated 00:36:16.460 --> 00:36:21.800 value or prefer the delete and the sort of baked into the data type itself, 00:36:22.080 --> 00:36:23.600 how to resolve that conflict. 00:36:24.480 --> 00:36:28.320 And is that resolution always the same, or does it depend on the use case? 00:36:28.460 --> 00:36:30.220 Can you configure the resolution? 00:36:30.720 --> 00:36:34.540 Yes, you can choose different CRDTs depending on the outcomes that you want. 00:36:34.720 --> 00:36:40.840 So for maps, for example, the most common way to construct a map is something 00:36:40.840 --> 00:36:48.480 called an observe removed map, where you can only remove items or updates that you have observed. 00:36:48.740 --> 00:36:52.620 So in the case before of A, updates foo, and B, deletes foo, 00:36:52.760 --> 00:36:58.380 I think I said it in reverse, but it doesn't matter, then because B did not 00:36:58.380 --> 00:37:02.720 observe the update to A, it is not allowed to remove the update to A. 00:37:02.940 --> 00:37:06.980 And therefore, the update to A will win out. And the result will be that foo 00:37:06.980 --> 00:37:10.480 will not be deleted in the resulting resolution. 00:37:11.100 --> 00:37:15.760 And the protocol and the algorithms and the metadata ensure that this is always the case. 00:37:15.980 --> 00:37:19.380 But you can choose, there's a different CRDT that is not the observed removed 00:37:19.380 --> 00:37:24.420 map that allows you, I forget the name of this one, but there's a CRDT for maps 00:37:24.420 --> 00:37:27.720 that is specifically a, like removes wins, 00:37:28.100 --> 00:37:32.840 where you are guaranteed that if you have a remove and an update, 00:37:33.140 --> 00:37:34.380 the update will be removed. 00:37:34.920 --> 00:37:38.780 And so these are different semantics you can choose by choosing the appropriate CRDTs. 00:37:38.980 --> 00:37:42.700 And the CRDTs tend to also be composable. So you can say that, 00:37:42.840 --> 00:37:47.820 you know, you have a key value map where the values themselves are also CRDTs of a particular type. 00:37:47.860 --> 00:37:50.540 And you can structure them in this way where you choose the 00:37:50.540 --> 00:37:53.900 semantics at every layer by choosing the appropriate crdt mechanism 00:37:53.900 --> 00:37:56.860 and the the rule of crdts is 00:37:56.860 --> 00:38:02.840 that as long as you observe all the same operations as another node in the system 00:38:02.840 --> 00:38:08.920 you agree on the final state so they're guaranteed to be sort of commutative 00:38:08.920 --> 00:38:13.240 and associative to the point where eventually everyone has the same state as 00:38:13.240 --> 00:38:14.760 long as they get to exchange all messages. 00:38:15.480 --> 00:38:20.700 I sort of have immediate follow-up questions, two of them. The first one would be... 00:38:22.170 --> 00:38:25.890 How do you decide on such data structures? Is there a team meeting and then 00:38:25.890 --> 00:38:31.690 someone kind of knows algorithms really well, algorithms and data structures and proposes that? 00:38:32.030 --> 00:38:36.210 And maybe there might be competing algorithms that maybe you considered. 00:38:36.490 --> 00:38:40.070 And have you been there when the decision was made? 00:38:40.870 --> 00:38:44.330 Yeah, so our CRDT distributed system I built. 00:38:44.630 --> 00:38:49.130 And part of that was out of a conviction that. 00:38:50.620 --> 00:38:53.420 GRPC is not something you can run 00:38:53.420 --> 00:38:56.240 on the edge network so we need something else and the 00:38:56.240 --> 00:38:59.320 question then becomes what is the something else and you 00:38:59.320 --> 00:39:02.600 know i happen to have a decent amount of background in building distributed 00:39:02.600 --> 00:39:07.220 systems and so i've had a decent idea of what the different options were and 00:39:07.220 --> 00:39:11.800 crdts felt like they fit this particular set of use cases or at least the use 00:39:11.800 --> 00:39:16.240 cases we could predict we would we're going to have quite well there are other 00:39:16.240 --> 00:39:19.280 designs you could come up with here, but they come with a different set of trade-offs. 00:39:19.540 --> 00:39:25.200 I think at the time, I went into it with a conviction that this is the right way. 00:39:25.640 --> 00:39:32.480 As you build more and more complex and compounding stacks, you start having 00:39:32.480 --> 00:39:36.760 to document these decisions as well so that the motivation and the rationale 00:39:36.760 --> 00:39:38.720 for making the choice is not lost to time. 00:39:38.900 --> 00:39:42.920 And that's where you end up writing architecture decision records like ADRs, 00:39:43.060 --> 00:39:47.700 where you write down, here's the problem statement, Here's the decision we made 00:39:47.700 --> 00:39:48.960 for what algorithm to use. 00:39:49.200 --> 00:39:52.660 Here are the options we considered and why we decided to discard them. 00:39:52.860 --> 00:39:56.400 So that for the future, people have insight into the decisions you made. 00:39:56.560 --> 00:40:01.220 And also, you know, the process of writing this document becomes the way you 00:40:01.220 --> 00:40:07.760 take the decision is you convince the group as part of writing this document 00:40:07.760 --> 00:40:11.140 that all the options should be discarded except for the one that you choose. 00:40:11.140 --> 00:40:15.700 And I'm assuming there's still ongoing research on CRDTs. 00:40:16.420 --> 00:40:21.320 Would you amend any of the prior decisions now if you had the chance? 00:40:21.680 --> 00:40:25.560 I don't think so. I think the CRDT design has actually worked out very, very well. 00:40:25.980 --> 00:40:31.000 And, you know, when we implemented the CRDTs in the first place, 00:40:31.180 --> 00:40:34.300 it was built off of a very recently released paper. 00:40:34.420 --> 00:40:37.180 So we started doing that implementation in... 00:40:38.040 --> 00:40:41.000 Shortly after I joined, actually, so this is sort of end of 2023, 00:40:41.000 --> 00:40:47.380 and we were implementing it based on a paper called DSON, which was released, I want to say 2022. 00:40:48.000 --> 00:40:52.480 So it was very much sort of bleeding edge technology already when we started implementing it. 00:40:52.700 --> 00:40:56.820 And then as part of implementing that set of data structures and algorithms 00:40:56.820 --> 00:41:02.380 and protocols, we also effectively extended the research into things that were 00:41:02.380 --> 00:41:06.340 needed for the operational use cases and production use cases we had in mind. 00:41:07.260 --> 00:41:10.200 Sounds like a decent paper and. 00:41:10.200 --> 00:41:13.000 Maybe i mean we did actually go a 00:41:13.000 --> 00:41:16.460 lot of back and forth with the authors of the paper we've also open sourced 00:41:16.460 --> 00:41:20.860 the the core of that implementation so there's a on crates.io there's a crate 00:41:20.860 --> 00:41:26.980 called decent dson that is the core of that crdt and that compound set of crdts 00:41:26.980 --> 00:41:30.560 precisely because we think this might be useful to other people and because 00:41:30.560 --> 00:41:31.820 we think we made, you know, 00:41:32.360 --> 00:41:36.520 innovations in the space that, you know, were not in the decent paper. 00:41:36.720 --> 00:41:40.620 And some of this was purely because in order to put this into production, 00:41:40.620 --> 00:41:45.360 we had to do a lot of both optimization, but also debugging to make sure it's 00:41:45.360 --> 00:41:46.980 extremely reliable and fast. 00:41:47.160 --> 00:41:52.420 And as part of that found corner cases that like in the paper were either handled in, you know, 00:41:52.720 --> 00:41:56.540 suboptimal ways, or in fact, we found some bugs, at least in their prototypical 00:41:56.540 --> 00:42:00.560 implementation, not necessarily in the algorithms that we wanted to correct. 00:42:00.560 --> 00:42:02.200 And therefore also wanted to publish. 00:42:03.310 --> 00:42:06.250 And when you did the implementation and you found those bugs, 00:42:06.510 --> 00:42:10.070 did the Rust type system surface them easily? 00:42:11.730 --> 00:42:16.590 Yes and no. There were some where the Rust type system was helpful. 00:42:17.530 --> 00:42:24.770 So the original DSON paper was accompanied by a research prototype written in JavaScript. 00:42:26.270 --> 00:42:29.090 And when writing the sort of rust 00:42:29.090 --> 00:42:31.870 encoding of that same algorithm there were 00:42:31.870 --> 00:42:34.930 definitely places where it pointed out that they had relied 00:42:34.930 --> 00:42:37.810 on the javascript type system 00:42:37.810 --> 00:42:42.950 being fairly forgiving where they were just sort of interchangeably using two 00:42:42.950 --> 00:42:46.470 different types that just like really should not be mixed because you can very 00:42:46.470 --> 00:42:50.830 easily get into bugs that way and i think we found like one or two bugs in the 00:42:50.830 --> 00:42:54.830 in the research prototype again not necessarily in the algorithm but in the 00:42:54.830 --> 00:42:56.430 research prototype that were because of this. 00:42:57.810 --> 00:43:03.650 But I think decent amount of the sort of nuances of the algorithms, 00:43:03.830 --> 00:43:05.570 especially when it comes to performance, for example, 00:43:05.970 --> 00:43:09.970 are not things that would be caused by the Rust type system as much as they're 00:43:09.970 --> 00:43:14.670 caught by a lot of testing, 00:43:15.090 --> 00:43:17.350 like property-based testing, fuss testing, 00:43:17.870 --> 00:43:21.530 and doing a lot of performance benchmarks and tracking down the root causes. 00:43:21.530 --> 00:43:25.150 They're like things that are hard to catch in the type system. 00:43:26.530 --> 00:43:30.870 Now when you look at all of these things that you've implemented so far and 00:43:30.870 --> 00:43:35.310 there's certainly a lot of amazing things in there I also want to check out 00:43:35.310 --> 00:43:37.090 your CRDT implementation now, 00:43:37.690 --> 00:43:42.870 what would you say was the biggest learning since you started modeling logic 00:43:42.870 --> 00:43:48.890 in Rust how do you model your types nowadays things that people can learn and apply in their own work. 00:43:50.150 --> 00:43:55.070 I think there's a trade-off that you learn over time of where is it worthwhile 00:43:55.070 --> 00:43:57.570 introducing more types and where is it not? 00:43:57.870 --> 00:44:00.950 Where is it worthwhile making things generic versus where is it not? 00:44:01.430 --> 00:44:05.010 I don't think there's a hard and fast rule that you can just always follow. 00:44:05.210 --> 00:44:11.090 But over time, you develop a sort of intuition for this doesn't feel like a 00:44:11.090 --> 00:44:15.310 good use of a type or this feels like it would be dangerous if I don't add a type. 00:44:15.470 --> 00:44:19.250 Like you can start to sort of predict the bugs that people will make if you 00:44:19.250 --> 00:44:20.410 don't introduce a new type. 00:44:20.910 --> 00:44:25.530 And you also, whenever you introduce a new type, you like feel a tingling in 00:44:25.530 --> 00:44:29.250 your hands about the amount of pain you just introduced to people using it because 00:44:29.250 --> 00:44:31.650 now there's an extra type or previously that it didn't need to be. 00:44:32.450 --> 00:44:36.130 And so I think the thing I've learned, and I think this is not a, 00:44:36.170 --> 00:44:41.890 it's not a moment in time learning. It's sort of a lesson over time is to better tune that trade-off. 00:44:42.070 --> 00:44:45.990 And I think one of the observations I've come to is. 00:44:47.360 --> 00:44:51.400 Representing things in the type system is extremely valuable, 00:44:51.400 --> 00:44:54.360 and people don't do it enough. 00:44:54.860 --> 00:45:02.520 But you have to balance it against the pain of using the library that you've 00:45:02.520 --> 00:45:04.340 developed that has all of these types. 00:45:04.340 --> 00:45:07.500 And you need to really keep in mind what am I actually gaining 00:45:07.500 --> 00:45:10.440 in terms of the safety I add to the system when I introduce all 00:45:10.440 --> 00:45:13.300 these types and what is the cost of the people using it 00:45:13.300 --> 00:45:16.260 and the way you do that is by making sure that when you 00:45:16.260 --> 00:45:20.880 write for example a library like Sguaba for example for the rigid body transformations 00:45:20.880 --> 00:45:26.540 also write code that uses that library while you are writing this type states 00:45:26.540 --> 00:45:31.120 library because it will immediately show you just how painful the consuming 00:45:31.120 --> 00:45:35.280 code ends up and that will help guide your way into, 00:45:35.500 --> 00:45:37.960 okay, maybe I don't need a dedicated type for this. 00:45:38.400 --> 00:45:45.940 So an example here maybe is in Sguaba, we have a type for WGS84, 00:45:46.400 --> 00:45:47.900 which is basically GPS coordinates. 00:45:48.380 --> 00:45:55.360 So latitude, longitude, and altitude. Now it turns out that WGS84 is actually a moving standard. 00:45:55.580 --> 00:46:00.400 It's a moving standard because the parameters. 00:46:01.460 --> 00:46:04.260 Earth changes over the course of 00:46:04.260 --> 00:46:07.540 time like you get you have you know plate drift 00:46:07.540 --> 00:46:10.760 and drift of the magnetic north pole and like they also update 00:46:10.760 --> 00:46:13.760 the reference ellipsoid sometimes when they get better estimates of like the 00:46:13.760 --> 00:46:18.840 roundness of the earth and so as a result there's not actually one wgs84 there's 00:46:18.840 --> 00:46:28.820 like multiple over time and if you capture a coordinate right now and then in within 10 years. 00:46:29.140 --> 00:46:34.480 You try to take the same coordinate and plot it on a map, then it wouldn't be 00:46:34.480 --> 00:46:37.020 in the same place as where the original one was. 00:46:37.180 --> 00:46:43.420 So if you read out the GPS coordinates of the Eiffel Tower, the actual GPS coordinates 00:46:43.420 --> 00:46:45.460 of the Eiffel Tower will change over time. 00:46:45.600 --> 00:46:47.640 But they'll change very little, usually. 00:46:47.940 --> 00:46:51.160 But it doesn't mean that technically, like 00:46:51.160 --> 00:46:54.380 you kind of want the type system to represent the 00:46:54.380 --> 00:46:57.460 point in time at which this measurement was 00:46:57.460 --> 00:47:00.580 taken the more extreme case here is so 00:47:00.580 --> 00:47:04.180 you have this thing called local tangent planes which is basically imagine 00:47:04.180 --> 00:47:07.140 a plane is flying and it has a some gps coordinate 00:47:07.140 --> 00:47:10.960 and then you want to know you know the the relative 00:47:10.960 --> 00:47:13.980 location to the plane so something like 00:47:13.980 --> 00:47:17.240 either in front right down coordinates or north east 00:47:17.240 --> 00:47:20.620 down coordinates so it's like this thing is one 00:47:20.620 --> 00:47:23.980 kilometer north three kilometers east and 500 00:47:23.980 --> 00:47:29.920 meters up from me then that is a it's a local tangent plane to the current location 00:47:29.920 --> 00:47:36.600 of the plane so i record that coordinate but now the plane moves then you kind 00:47:36.600 --> 00:47:40.060 of want to represent the fact that this was a coordinate relative to that plane's 00:47:40.060 --> 00:47:41.440 position at this point in time. 00:47:42.470 --> 00:47:46.390 The reality is if we actually tried to encode that information in the type system 00:47:46.390 --> 00:47:51.670 of Sguaba, it would be impossible to use because every type would be distinct. 00:47:52.170 --> 00:47:56.270 You just like, there would be no easy way to move between different coordinate systems. 00:47:56.450 --> 00:48:00.970 You would end up with these like, WGS84 would have like three different generic 00:48:00.970 --> 00:48:05.030 type parameters that are like involve time. And then how does that translate 00:48:05.030 --> 00:48:07.630 into other coordinate systems that involve time? 00:48:08.030 --> 00:48:12.590 And it would be more accurate, it, but it also would be much more painful to 00:48:12.590 --> 00:48:17.130 use, and it's not entirely clear that it eliminates very many classes of bugs. 00:48:17.490 --> 00:48:20.930 Quite to the contrary, it might introduce more, because now if you get the wrong 00:48:20.930 --> 00:48:25.450 time bases, then nothing compiles, and then you pull shortcuts to try to get 00:48:25.450 --> 00:48:27.850 out of the mess of errors you end up with. 00:48:28.270 --> 00:48:32.730 And so for Sguaba, I made the explicit choice to say, this library will not 00:48:32.730 --> 00:48:35.190 represent time in the type system. 00:48:35.190 --> 00:48:40.750 And that does reduce the safety you get from the type system, 00:48:40.750 --> 00:48:46.230 but it also makes the library much more pleasant to use, which increases the 00:48:46.230 --> 00:48:47.590 number of people who will use it, 00:48:47.710 --> 00:48:51.310 and therefore overall increases safety compared to if I had put it in. 00:48:52.110 --> 00:48:56.790 Is that an example of the conflict between ergonomics and correctness? 00:48:57.250 --> 00:48:58.150 I think that's right. 00:48:59.370 --> 00:49:07.790 And, you know, it's like, I don't actually think it's sacrificing correctness, 00:49:07.990 --> 00:49:10.350 it's sacrificing precision. 00:49:10.870 --> 00:49:15.310 And precision can be useful for correctness, but you can also have correctness 00:49:15.310 --> 00:49:20.250 without that precision, right? So you can write correct programs that don't 00:49:20.250 --> 00:49:22.710 have time represented in the type system. 00:49:22.910 --> 00:49:27.190 It just means that there's some classes of bugs that you don't get to eliminate 00:49:27.190 --> 00:49:32.050 through the type system. but that doesn't mean you end up not having correctness yeah. 00:49:32.050 --> 00:49:36.890 So from your lens it's still correct because it encodes everything that you 00:49:36.890 --> 00:49:42.210 want the type system to encode but you just explicitly leave out a thing that 00:49:42.210 --> 00:49:44.890 you don't want to be so precise on yeah. 00:49:44.890 --> 00:49:48.750 And where where i think you know i'm going to leave it to the users of this 00:49:48.750 --> 00:49:50.130 library to get that part correct. 00:49:51.340 --> 00:49:54.000 Do you make a difference between library code and binary code? 00:49:54.200 --> 00:49:59.640 Do you write different code if you had to write application-level Rust versus library-level Rust? 00:50:00.220 --> 00:50:05.020 Yeah, I think I do in the sense that when I write library-level Rust, 00:50:05.440 --> 00:50:09.300 I think a lot more about the programmatic API that I present. 00:50:09.820 --> 00:50:12.980 So that includes not just documentation, right? 00:50:13.020 --> 00:50:16.880 Obviously, you need to write good documentation for a library to be useful, 00:50:16.880 --> 00:50:20.600 but also the structure of that API. 00:50:21.080 --> 00:50:26.420 What are the backwards compatibility hazards? Where do I think I might put myself 00:50:26.420 --> 00:50:29.360 into a trap when it comes to breaking changes down the line? 00:50:29.660 --> 00:50:34.660 So I want to be conservative about what things I expose in the public API because 00:50:34.660 --> 00:50:38.220 those are things I can't change later unless I do a breaking release. 00:50:38.620 --> 00:50:42.400 The way that you propagate errors might be different because you might want 00:50:42.400 --> 00:50:50.800 library consumers to have a better ability to deconstruct the error and figure out the origin. 00:50:50.980 --> 00:50:56.020 Whereas in a binary, usually what you want is to present a chain of errors to 00:50:56.020 --> 00:50:59.880 the user that results in something actionable on their end to fix the problem. 00:51:00.280 --> 00:51:03.300 And so I do think the design ends up somewhat different. 00:51:03.640 --> 00:51:07.980 I don't think it changes the internal writing of the code very much, 00:51:08.080 --> 00:51:10.900 but it changes how much focus you put on the external API. 00:51:11.420 --> 00:51:14.660 But for a binary, of course, you have to think about what does the command line 00:51:14.660 --> 00:51:18.360 interface look like for that binary and so that also requires thought but it's 00:51:18.360 --> 00:51:22.720 it's a different kind of design process earlier. 00:51:22.720 --> 00:51:27.140 You mentioned that you also want to write the application level code that goes 00:51:27.140 --> 00:51:32.420 along with the library code or actually vice versa does that mean you start 00:51:32.420 --> 00:51:39.000 with a main.rs model out your types and then gradually move them into library crate. 00:51:39.000 --> 00:51:46.420 It can i do do that sometimes as well but more commonly it means i already have 00:51:46.420 --> 00:51:48.400 at least two code bases that, 00:51:49.240 --> 00:51:55.100 have a need for this library. And so I'm going to create the library and see 00:51:55.100 --> 00:51:56.940 how it affects those two codebases. 00:51:57.420 --> 00:52:03.140 Because then I have a real set of use cases that I can test out how the library feels. 00:52:03.940 --> 00:52:07.720 And I mean, this was the case for Sguaba, for instance, was we had codebases 00:52:07.720 --> 00:52:10.440 internally that had to do this kind of spatial math. 00:52:10.500 --> 00:52:14.120 And they already had code for it, like they were working code bases but 00:52:14.120 --> 00:52:16.860 the that code was like hard to 00:52:16.860 --> 00:52:19.740 review brittle had a bunch of like magic constants in 00:52:19.740 --> 00:52:22.740 there and so it didn't it felt like 00:52:22.740 --> 00:52:25.700 you know we've had to solve this problem at 00:52:25.700 --> 00:52:30.860 least twice so we should turn it into a reusable library that is you know well 00:52:30.860 --> 00:52:35.140 designed well tested and because it would be recommended going forward and so 00:52:35.140 --> 00:52:40.580 that is what informs the the the design of the library is the the the evident 00:52:40.580 --> 00:52:44.220 need from the things you've you've already built when. 00:52:44.220 --> 00:52:47.940 You build squab up what was your testing strategy was it based on unit tests 00:52:47.940 --> 00:52:51.020 integration tests or or property-based testing or fuzzing. 00:52:51.020 --> 00:52:55.800 It's it's all of the above so the there's both a bunch of unit tests in there 00:52:55.800 --> 00:53:00.960 there's also a lot of equivalence tests to other crates that implement some 00:53:00.960 --> 00:53:02.120 subset of the functionality. 00:53:02.500 --> 00:53:07.480 So for example, there's a crate called NavTypes that implements, 00:53:07.580 --> 00:53:13.660 for example, conversion between WGS84 and ECEF, which is another Earth-based coordinate system. 00:53:14.020 --> 00:53:20.380 And so there's a property-based test inside of Sguaba that basically generates 00:53:20.380 --> 00:53:23.760 random points on Earth and then converts them back and forth using Sguaba, 00:53:24.160 --> 00:53:28.560 converts them back and forth using NavTypes, and then checks that the results are near each other. 00:53:29.340 --> 00:53:32.300 And then there's also just general property-based testing that does things like, 00:53:32.440 --> 00:53:35.560 you know, if you pick a random coordinate on Earth, 00:53:35.980 --> 00:53:41.000 run it through like back and forth through WGS84 and ECEF, which is a lossy 00:53:41.000 --> 00:53:44.540 conversion, run it through that 10 times and see how much degradation you get. 00:53:44.740 --> 00:53:48.800 And you get guarantees about, you know, you get probabilistic guarantees about 00:53:48.800 --> 00:53:51.240 how much deterioration will you see over time. 00:53:52.050 --> 00:53:55.650 And then we also have a bunch of tests around the enforcement of the type system, right? 00:53:55.790 --> 00:54:01.290 So both tests to make sure that you can express the correct computations, 00:54:01.290 --> 00:54:07.030 but also compile fail tests that say you cannot try to use a coordinate from 00:54:07.030 --> 00:54:11.170 one coordinate system as a coordinate in a different one without an explicit conversion. 00:54:11.590 --> 00:54:18.450 You put in so much work into Sguaba, similar libraries, and then you decide 00:54:18.450 --> 00:54:25.490 to open source that work. that's a big gift why do you do that why open source those libraries i. 00:54:25.490 --> 00:54:29.510 Think it's a it's a combination of factors one 00:54:29.510 --> 00:54:35.570 of them is the the the traditional you know by having more people being able 00:54:35.570 --> 00:54:38.810 to look at a thing you're more confident that it's correct and i think that 00:54:38.810 --> 00:54:43.970 applies to to open sourcing software in this context too and and i think there's 00:54:43.970 --> 00:54:45.530 a related point to that which is, 00:54:46.390 --> 00:54:48.010 you know, we operate in the defense sector. 00:54:48.290 --> 00:54:55.050 And so the systems that we build, we want to have as much confidence as we can is correct. 00:54:55.250 --> 00:55:00.770 But we also want to sort of, to the extent that we can, give people the ability 00:55:00.770 --> 00:55:07.010 to look at how we build software for whether they think we are building software in a responsible way. 00:55:07.190 --> 00:55:11.850 And obviously we can't open source like the actual products we're developing, 00:55:12.070 --> 00:55:16.590 but at least one stepping stone is to open source some of the techniques, 00:55:16.830 --> 00:55:20.890 some of the tools that we use in order to produce this software to the level 00:55:20.890 --> 00:55:24.310 of sort of reliability that we want to give. 00:55:24.830 --> 00:55:29.190 And so open sourcing these kinds of libraries, I think, 00:55:29.550 --> 00:55:33.510 gives hopefully both some feeling of transparency on that part, 00:55:33.610 --> 00:55:37.550 but also inspires some amount of confidence that we are building this software, 00:55:37.550 --> 00:55:43.630 at least at a technical level with care and so I think it matters to demonstrate 00:55:43.630 --> 00:55:46.590 that and then I think there's a there's a. 00:55:48.040 --> 00:55:52.200 Sort of a wanting to give back kind of feeling, right? 00:55:52.400 --> 00:55:55.780 Of we get a lot from the Rust community. 00:55:55.940 --> 00:55:59.600 And I mean, we are sponsors of the Rust Foundation partially for this reason to give back. 00:55:59.740 --> 00:56:02.900 But the other way to give back is to make sure that when we build things that 00:56:02.900 --> 00:56:07.800 we think are useful to other people than us, that we make them useful to other people than us. 00:56:08.480 --> 00:56:11.000 And then I think that there's obviously a cynical angle too, 00:56:11.180 --> 00:56:14.900 right, which is you put things out there so that other people get to look at 00:56:14.900 --> 00:56:20.340 interesting things you've built and then go, I also want to work on those things, right? 00:56:20.360 --> 00:56:24.100 Do you get to actually show some of your code, show some of your development 00:56:24.100 --> 00:56:27.940 styles, show some of the problems you're working on and hopefully get other 00:56:27.940 --> 00:56:29.420 people interested as a result? 00:56:30.280 --> 00:56:35.600 Yeah. In preparation for this interview, I read through some of the blog posts 00:56:35.600 --> 00:56:37.460 on the Helsing Tech blog. 00:56:37.680 --> 00:56:44.280 And I have to say, it's astonishing and certainly enticing to know what sort 00:56:44.280 --> 00:56:45.500 of problems you're working on. 00:56:45.500 --> 00:56:53.180 And i think it attracts a certain group of people who are interested in solving 00:56:53.180 --> 00:56:58.280 hard problems and working with rust because they know that rust is the right choice. 00:56:58.280 --> 00:57:00.900 I i think that's true and if you 00:57:00.900 --> 00:57:03.760 think about the flip side right imagine we didn't open source anything we 00:57:03.760 --> 00:57:07.480 didn't write any technical blog posts then i think the the 00:57:07.480 --> 00:57:10.420 first question would be well why not what do you have to hide right but 00:57:10.420 --> 00:57:13.760 the the other observation is how do you hire like 00:57:13.760 --> 00:57:16.680 especially you know talented engineers who are curious 00:57:16.680 --> 00:57:19.900 about technical depth if the only 00:57:19.900 --> 00:57:22.600 thing they see is sort of the the product side of 00:57:22.600 --> 00:57:25.660 things externally like they don't have direct access 00:57:25.660 --> 00:57:30.080 to the engineers we have internally so you kind of you would have to apply go 00:57:30.080 --> 00:57:35.120 through interviews and then get to talk to the engineers which is a lot to ask 00:57:35.120 --> 00:57:38.120 of someone who's still like deciding whether they might want to join the company 00:57:38.120 --> 00:57:41.840 and so by opening the doors a little bit and showing some of the work we work 00:57:41.840 --> 00:57:45.180 on and the way that we actually do engineering, 00:57:46.450 --> 00:57:51.490 You give people more insight and therefore hopefully more to go on when deciding 00:57:51.490 --> 00:57:53.130 whether this is a place where they would want to work. 00:57:54.410 --> 00:57:58.890 Do you get any contributions, pull requests, people creating issues? 00:57:59.010 --> 00:58:01.870 We do. It varies between the different projects, right? 00:58:01.930 --> 00:58:05.750 So the different things we've open sourced are varying degrees of useful to 00:58:05.750 --> 00:58:07.450 other, like the Dson Crate, for example. 00:58:07.890 --> 00:58:11.590 I'm not expecting lots of people to make use of because it's a very, 00:58:11.990 --> 00:58:16.670 you know, you need to have a very particular use case for those to be the most useful to you. 00:58:16.930 --> 00:58:21.570 And then other things like the Avro tooling, for example, I actually expect 00:58:21.570 --> 00:58:27.010 could become quite popular because a lot of people use Avro and this thing gives 00:58:27.010 --> 00:58:28.730 you faster and better error. 00:58:28.970 --> 00:58:32.030 Like it's faster than the upstream version and it gives you better error messages. 00:58:32.250 --> 00:58:35.270 So we might get a bunch of people using it, but it's fairly new. 00:58:35.850 --> 00:58:38.730 Squab, I expect, would probably be quite popular. 00:58:38.790 --> 00:58:42.030 And that's also the one we've seen the most interest in, actually, 00:58:42.030 --> 00:58:46.630 of people wanting to find issues, reuse it, contribute to it, 00:58:46.670 --> 00:58:48.510 and we take those contributions seriously. 00:58:49.670 --> 00:58:53.270 Buffers has been a bit of a mix where people have had interest, 00:58:53.490 --> 00:58:58.090 but I think the companies where this becomes the most relevant, 00:58:58.350 --> 00:59:02.450 many of them have monorepos and therefore don't need this particular tech. 00:59:02.890 --> 00:59:09.370 As you need to both have a need to do versioning and packaging of extended dependency 00:59:09.370 --> 00:59:13.630 chains of buffers, of protobuf files, and also not have a monorepo. 00:59:13.890 --> 00:59:17.850 And that combination, I think it's somewhat rare, although not unique. 00:59:17.990 --> 00:59:22.150 But in general, we do see interest on the projects we put out there. 00:59:23.320 --> 00:59:27.100 And I believe one other angle is that a lot of people might just be interested 00:59:27.100 --> 00:59:30.540 in knowing how a Rust expert structures a library. 00:59:31.260 --> 00:59:38.120 And there's very little material out there outside of maybe a handful of popular 00:59:38.120 --> 00:59:42.620 crates and maybe a bunch of blog posts on how to write advanced Rust code. 00:59:42.920 --> 00:59:48.100 And a lot of people want to learn by osmosis, by reading what other people have 00:59:48.100 --> 00:59:50.720 written that they deem to be Rust experts. 00:59:51.160 --> 00:59:52.760 Yes, I think that's also true. 00:59:53.320 --> 00:59:59.740 After all those years, do you still see yourself as an educator and do you do 00:59:59.740 --> 01:00:04.440 Rust education outside of Helsing or also within Helsing? 01:00:04.760 --> 01:00:09.180 Yeah, I very much see my job as an educator. And it's something that I have, 01:00:09.240 --> 01:00:12.120 you know, I think a pretty deep passion for. 01:00:12.500 --> 01:00:18.420 I really enjoy, you know, that moment where you can experience someone else 01:00:18.420 --> 01:00:20.720 understanding something. That makes me very happy. 01:00:21.180 --> 01:00:24.460 And so, you know, I continue to do education outside of Helsing. 01:00:24.540 --> 01:00:27.660 And it's the same thing I did at Amazon as well, where, you know, 01:00:27.720 --> 01:00:30.720 a lot of my live streams and stuff, that all continued while I worked there. 01:00:30.780 --> 01:00:31.820 And it's the same thing as Helsing. 01:00:32.300 --> 01:00:35.800 I do some amount of education internally at Helsing as well. 01:00:36.000 --> 01:00:41.380 Although internally, it has more of a sort of reactive nature, right? 01:00:41.460 --> 01:00:44.880 Where people will poke me and be like, hey, Jon, why doesn't this work? 01:00:45.000 --> 01:00:49.220 Or how should we do this? Or because like we do some amount of office hours 01:00:49.220 --> 01:00:52.800 internally, but also more of that, you know, we have like a Rust help channel 01:00:52.800 --> 01:00:55.300 where people ask and then occasionally I'll get, you know, 01:00:55.680 --> 01:00:58.840 poked explicitly and be like, I think Jon wrote a blog post about this, 01:00:58.920 --> 01:01:00.760 or I think Jon implemented something along those lines. 01:01:01.840 --> 01:01:08.660 But I actually think the most amount of education I do that has value internally 01:01:08.660 --> 01:01:10.840 is actually the external education that I do. 01:01:11.140 --> 01:01:17.460 So I know that a lot of the engineers we have at Helsing have learned or partially 01:01:17.460 --> 01:01:21.440 learned Rust through my public educational resources, right? 01:01:21.560 --> 01:01:25.100 And that is also how some of them continue to learn new concepts in Rust, 01:01:25.300 --> 01:01:28.880 is to observe the same teaching resources that I put out publicly. 01:01:28.880 --> 01:01:33.520 And this is also why I think Helsing is quite supportive of me continuing to 01:01:33.520 --> 01:01:38.260 do my public education, because not only is it sort of a, cynically speaking, 01:01:38.320 --> 01:01:39.860 like a sales thing, right? 01:01:39.940 --> 01:01:44.300 Like it's valuable to the company to have someone who's seen as a Rust expert, 01:01:44.520 --> 01:01:48.220 both publicly be operating as a Rust expert and be employed by them. 01:01:48.220 --> 01:01:53.460 But I think more meaningfully, it also means that more people are able to learn 01:01:53.460 --> 01:01:57.780 Rust through the abilities or through the teaching that I do, 01:01:57.900 --> 01:02:00.780 which means that there's a bigger hiring pool for Helsing to draw from. 01:02:01.100 --> 01:02:06.320 But also, the people that we hire, hopefully, have then also learned more things 01:02:06.320 --> 01:02:09.900 about Rust because I've produced those intermediate teaching resources. 01:02:10.200 --> 01:02:15.600 And the people at the company can continue to improve their skills by me continuing 01:02:15.600 --> 01:02:20.260 to do teaching. So it ends up being a virtual cycle in a way where it's good 01:02:20.260 --> 01:02:23.560 for the company, which is good for me, which is good for the company, which is good for me. 01:02:24.460 --> 01:02:31.580 And I do think there's also some amount of recognition internally at the company that... 01:02:32.920 --> 01:02:36.480 You know, if I started just building internal teaching resources, 01:02:36.480 --> 01:02:40.680 it would be seen as a bit of a shame, right? 01:02:40.740 --> 01:02:46.880 It would be like, why are we not making this material public when it could be? 01:02:46.960 --> 01:02:50.600 There's nothing secret about it. It's just how do you build good Rust code? 01:02:50.740 --> 01:02:54.600 How do you engineer high quality Rust products? 01:02:54.960 --> 01:02:58.860 Then that feels like something we should be sharing back to the community because 01:02:58.860 --> 01:03:03.200 those aren't, they're not industry secrets, right? They are just things that 01:03:03.200 --> 01:03:06.840 are beneficial to everyone using Rust. And I think it's also the case that, 01:03:07.600 --> 01:03:11.760 If we make the Rust community better, we benefit as a result, 01:03:11.880 --> 01:03:15.620 not just through hiring, but also because the quality of the crates that exist 01:03:15.620 --> 01:03:17.480 in the open source ecosystem will be better. 01:03:17.760 --> 01:03:22.620 The tooling will be better. Everything gets better if the community improves. 01:03:22.860 --> 01:03:27.580 And so there's just a lot of positive externalities here and positive feedback 01:03:27.580 --> 01:03:32.200 loops that mean that me continuing to do the public part of education is valuable. 01:03:32.960 --> 01:03:40.680 And with 200 plus hours of video material out there of you teaching rust i sort 01:03:40.680 --> 01:03:46.720 of think it's inevitable that people share videos of you internally without you knowing. 01:03:46.720 --> 01:03:49.440 Oh that that definitely happens i 01:03:49.440 --> 01:03:52.360 mean this happened at amazon too where the moment 01:03:52.360 --> 01:03:55.360 i got on the inside i kept finding places where 01:03:55.360 --> 01:03:58.160 people had referred to either things that like videos 01:03:58.160 --> 01:04:01.280 i'd made or blog posts i'd written or crates i've published 01:04:01.280 --> 01:04:05.940 being like Jon go look at this thing or like you should read Jon's thing about 01:04:05.940 --> 01:04:10.620 this and that is always fun it's the same thing when when we do hiring a decent 01:04:10.620 --> 01:04:14.600 number of the people that go through the interview process say that either they 01:04:14.600 --> 01:04:18.440 learned rust through me or they heard about the company through me and that 01:04:18.440 --> 01:04:21.120 is a weird feeling for sure do. 01:04:21.120 --> 01:04:27.320 You see people taking it to the extreme sometimes where maybe they they learn 01:04:27.320 --> 01:04:32.520 about an advanced concept and they want to apply it at work and during code review you find well, 01:04:33.780 --> 01:04:40.500 it's expressive it's certainly concise but maybe not maintainable by a larger 01:04:40.500 --> 01:04:42.480 team and where do you draw the line. 01:04:43.480 --> 01:04:49.820 Yeah, I actually think this is not just a like intermediate Rust programmer thing. 01:04:49.980 --> 01:04:54.500 I think this is pretty common across Rust, even from the early days, 01:04:54.500 --> 01:04:58.900 is that people see all of these tools and techniques that are possible in Rust, 01:04:58.980 --> 01:05:01.660 and then they immediately want to make use of them. 01:05:01.860 --> 01:05:05.320 And I think the new type pattern is a big one, right? Like I can define custom 01:05:05.320 --> 01:05:07.680 types for everything, and then everything is type safe. 01:05:07.840 --> 01:05:11.760 And people start using that pattern to the extreme. And to the discussion we 01:05:11.760 --> 01:05:15.700 had earlier around the trade-off space here, they just navigate the trade-off 01:05:15.700 --> 01:05:18.180 space by always picking the most type-safe thing. 01:05:18.480 --> 01:05:23.580 And we see the same when you look at the hesitance to use locks, 01:05:23.960 --> 01:05:28.600 the hesitance to use RC and ARC. 01:05:28.600 --> 01:05:34.560 So people try to use lock-free algorithms and use references with lifetimes 01:05:34.560 --> 01:05:36.740 everywhere. And you end up with multiple lifetime annotations. 01:05:36.920 --> 01:05:40.820 And no one wants to clone anything. and everything has to be monomorphized so 01:05:40.820 --> 01:05:41.920 there's no dynamic dispatch. 01:05:42.240 --> 01:05:46.620 And people really lean into every possible feature that Rust gives you. 01:05:46.760 --> 01:05:50.740 And it makes it really painful to program in the language. It makes it painful to review the code. 01:05:50.960 --> 01:05:56.960 It means that you end up constructing suboptimal software architecture because 01:05:56.960 --> 01:06:01.520 you can't express the architecture you want with the borrow checker, 01:06:01.600 --> 01:06:04.480 with the type checker, or even just with your current knowledge of the language. 01:06:04.480 --> 01:06:07.680 And so I do tend to see. 01:06:08.080 --> 01:06:13.420 Especially in people who haven't built production code in Rust very much, 01:06:13.680 --> 01:06:18.520 they tend to lean overly much into some of these patterns. And then you kind 01:06:18.520 --> 01:06:19.520 of have to pull them back and be like... 01:06:21.120 --> 01:06:25.160 It's okay to clone here. It's okay to put this thing behind a mutex. 01:06:25.300 --> 01:06:29.960 It's okay to not have a new type for this particular string representation of an email, right? 01:06:30.200 --> 01:06:32.680 And over time, people learn that distinction. 01:06:32.960 --> 01:06:37.520 But that is part of the education you kind of have to do on the job is see the 01:06:37.520 --> 01:06:42.460 code that people write and then course correct as you do for where they're maybe 01:06:42.460 --> 01:06:45.020 overzealous about the use of some of Rust's features. 01:06:45.740 --> 01:06:48.360 Yeah, it's certainly a bit of a rite of passage. 01:06:49.320 --> 01:06:50.400 Yes, I think so. 01:06:50.400 --> 01:06:59.300 Do you think the key to idiomatic Rust is keeping it simple and then maybe making 01:06:59.300 --> 01:07:01.220 it right where it matters? 01:07:01.720 --> 01:07:08.220 So finding that balance between on one side simplicity and maybe ease of maintenance 01:07:08.220 --> 01:07:12.360 versus correctness for things that really are important? 01:07:13.390 --> 01:07:15.730 Or what's your working definition of idiomatic, Rust? 01:07:16.070 --> 01:07:19.710 I think it's very hard to give a sort of general definition. 01:07:19.990 --> 01:07:24.510 I tend to start from, I don't think you should start with the simplest possible 01:07:24.510 --> 01:07:28.450 thing, but I also don't think you should start with the most complicated thing. 01:07:29.170 --> 01:07:35.250 And this is where, and I don't like using this, but I think experience matters here, right? 01:07:35.310 --> 01:07:39.010 Where like over time, you just get a feel for where the balance should lie. 01:07:39.010 --> 01:07:42.090 You start writing the code and you go, it's okay to clone here, 01:07:42.130 --> 01:07:43.870 and it's not okay to clone here. 01:07:44.070 --> 01:07:48.730 And it's hard for me to distill what the principles are for making those choices. 01:07:49.490 --> 01:07:54.350 I would say in general, it is very useful to have a running system. 01:07:55.050 --> 01:07:59.590 Once you have a running system, you can then, like, refactoring with Rust tends 01:07:59.590 --> 01:08:02.930 to be a lot easier than in other languages because you have the type system 01:08:02.930 --> 01:08:06.710 and the solid compiler and type checker and borrow checker to rely on. 01:08:06.710 --> 01:08:13.850 And so I would tend to err on the side of where the type safe thing is easy, 01:08:13.870 --> 01:08:14.990 then do the type safe thing. 01:08:15.570 --> 01:08:22.270 And where the type safe thing gets in the way of you building the actual application 01:08:22.270 --> 01:08:26.530 to the end, build the application to the end first, and then mark it with a to do to come back to. 01:08:27.010 --> 01:08:30.170 And some of those to do would be very painful to fix later. 01:08:30.410 --> 01:08:34.430 But the reality is, if you don't finish the whole thing, you're never going 01:08:34.430 --> 01:08:36.610 to come back to the to do's because you didn't work in the first. 01:08:36.710 --> 01:08:43.510 Place so so it's useful to like use that that as a forcing function for making 01:08:43.510 --> 01:08:47.930 you make a suboptimal choice is like well i need to at least get to a thing 01:08:47.930 --> 01:08:50.250 that runs otherwise this whole thing is irrelevant. 01:08:51.150 --> 01:08:56.490 Rust is a huge language, and while I'm pretty sure that you know more than most 01:08:56.490 --> 01:09:02.030 people about Rust, what is one thing that you would personally want to spend more time on? 01:09:02.490 --> 01:09:08.350 If you had three months to focus on one subject that was Rust-related, what would it be? 01:09:08.750 --> 01:09:11.810 I think there are two categories for me. 01:09:12.090 --> 01:09:17.830 One of them is around WebAssembly. So I've done very little WebAssembly in Rust, 01:09:17.970 --> 01:09:23.310 and I think it's both a cool technology, And it's something that I think there's 01:09:23.310 --> 01:09:27.030 a bunch of use cases for it that I think we haven't fully explored. 01:09:27.030 --> 01:09:31.730 And I would love to fiddle around with it to see what I can make of it. 01:09:31.850 --> 01:09:37.150 But also, I think it's a very useful skill or set of knowledge to have in your toolbox. 01:09:37.970 --> 01:09:42.130 And same thing for, you know, I'm thinking of writing another sort of version 01:09:42.130 --> 01:09:45.490 or not version, but a second iteration of Rust for Rustaceans. 01:09:45.490 --> 01:09:48.830 And, you know, Rust Rustaceans doesn't have a chapter on WebAssembly, 01:09:49.090 --> 01:09:52.650 in part because I hadn't done very much WebAssembly at the time when I wrote 01:09:52.650 --> 01:09:55.770 the book and I didn't feel like I could be an authority on that subject. 01:09:55.970 --> 01:09:59.970 And I still don't think I could be. And so that is something I would want to do more of. 01:10:00.650 --> 01:10:04.130 I think the other category would be sort of deep embedded development. 01:10:04.390 --> 01:10:07.590 I've done some embedded development in Rust and I've certainly written, 01:10:07.590 --> 01:10:09.210 you know, crates for no-std and everything. 01:10:09.470 --> 01:10:14.870 But to really write something low level on a microcontroller where, like, you need to, like, 01:10:15.770 --> 01:10:19.110 You need to initialize the CPUs in the right way, and you need to handle the 01:10:19.110 --> 01:10:21.130 interrupts, and you need to write some inline assembly. 01:10:22.330 --> 01:10:26.870 Code like that is really fun to write, and I haven't written lots of it in Rust, 01:10:27.110 --> 01:10:31.690 but I would like to, because I want to see, you know, what does it feel like 01:10:31.690 --> 01:10:34.190 when I push the language in that direction a little bit? 01:10:34.410 --> 01:10:38.750 And like, where are the sharp edges, and what can I do to make those sharp edges 01:10:38.750 --> 01:10:40.670 be more ergonomic, right? 01:10:40.730 --> 01:10:44.890 Like, this could be building tooling, building libraries to make that experience better. 01:10:44.890 --> 01:10:51.070 I wanted to briefly touch on supply chain security because I believe it's kind 01:10:51.070 --> 01:10:52.290 of important for housing. 01:10:52.690 --> 01:10:56.390 You write a lot of code, you maintain a lot of code yourself, 01:10:56.750 --> 01:11:03.150 but you still need to depend on a lot of crates that are out there that we sort of take for granted. 01:11:03.150 --> 01:11:10.050 And there's been some recent challenges around some packages in Rust. 01:11:10.210 --> 01:11:12.470 I don't want to mention any names. 01:11:12.910 --> 01:11:22.190 And Cargo itself also had some sort of exploit because of the tar crate just a couple days ago. 01:11:23.090 --> 01:11:26.190 I wonder what's Helsing's stance on that? 01:11:26.430 --> 01:11:31.590 And what's the state of the Rust ecosystem in regard to supply chain security? 01:11:32.210 --> 01:11:36.850 I think Rust is not in a worse place than other ecosystems here. 01:11:37.010 --> 01:11:42.230 I think it's a bit similar to other ecosystems where when you take a lot of 01:11:42.230 --> 01:11:47.010 third-party dependencies, there's some amount of inherent risk there. 01:11:47.150 --> 01:11:51.090 And I don't think Rust's tooling is worse or Rust's risk is higher. 01:11:52.450 --> 01:11:58.570 And I think the question, as with any language like this or any project that 01:11:58.570 --> 01:12:03.210 has to take third-party dependencies, the question becomes, what do you do about those risks? 01:12:03.730 --> 01:12:08.530 And you think, you know, the reality is you have to build in defense in depth 01:12:08.530 --> 01:12:09.990 against these things, right? 01:12:10.070 --> 01:12:15.310 There's not going to be one silver bullet that just solves all your supply chain security problems. 01:12:15.310 --> 01:12:22.390 Instead, you have to have sort of a collection of processes and tools that make 01:12:22.390 --> 01:12:25.850 sort of as many parts of it as secure as you can. 01:12:26.010 --> 01:12:29.770 And then you layer them on top of each other to get coverage across your whole pipeline. 01:12:29.990 --> 01:12:35.130 So this includes everything from, you know, being judicious about your selection 01:12:35.130 --> 01:12:36.630 of dependencies in the first place. 01:12:36.910 --> 01:12:41.010 Like, don't take a dependency on some tiny, barely maintained project if you 01:12:41.010 --> 01:12:43.030 can easily just replicate the functionality yourself. 01:12:43.030 --> 01:12:46.070 Like the reason why it's worthwhile 01:12:46.070 --> 01:12:49.470 to take dependencies is if the maintenance cost 01:12:49.470 --> 01:12:52.190 of the code in that dependency is large but if 01:12:52.190 --> 01:12:55.030 the maintenance cost is actually pretty small it might not be worth taking the 01:12:55.030 --> 01:12:58.650 dependency and introducing that risk the same thing with if you have the choice 01:12:58.650 --> 01:13:02.890 between multiple dependencies then look at the different dependencies not just 01:13:02.890 --> 01:13:07.350 in terms of the the quality of them in terms of like the you know the api the 01:13:07.350 --> 01:13:10.230 documentation the current code maturity, 01:13:10.470 --> 01:13:14.290 but also look at the maintenance of that package. 01:13:14.490 --> 01:13:17.890 Who maintains it? How many people? Do they have CI? 01:13:18.230 --> 01:13:22.710 What kind of testing strategy do they have? Do they have a security disclosure policy? 01:13:23.010 --> 01:13:26.810 There's a bunch of things you can look for here that indicate something about the. 01:13:27.700 --> 01:13:31.080 The sustainability of that package and of taking a dependency on it. 01:13:31.840 --> 01:13:35.060 And then there's also the sort of ongoing monitoring part, right? 01:13:35.140 --> 01:13:38.880 So obviously you want to monitor all of the security vulnerability databases 01:13:38.880 --> 01:13:44.740 to make sure that if you run into a problem, or rather if a problem is discovered 01:13:44.740 --> 01:13:46.160 with some version of some dependency, 01:13:46.520 --> 01:13:50.940 you A, are notified, and B, that you internally have the infrastructure to find 01:13:50.940 --> 01:13:55.860 all the places where the impacted dependency are used. 01:13:55.860 --> 01:14:02.280 And so this includes being able to track provenance information for all the builds that you do, 01:14:02.700 --> 01:14:06.080 provenance for all your deployments, and this is where you get into things like 01:14:06.080 --> 01:14:10.660 generating SBOMs, like software bill of materials that list all of the dependencies 01:14:10.660 --> 01:14:11.840 that went into a given artifact, 01:14:12.540 --> 01:14:16.520 tracking which software releases are released to what customers, 01:14:16.520 --> 01:14:21.300 at what time, in what products, in what physical devices, and keeping track 01:14:21.300 --> 01:14:23.680 of that whole graph structure. 01:14:23.680 --> 01:14:29.440 And being able to do analysis over that graph over time as you learn about new vulnerabilities. 01:14:30.200 --> 01:14:33.820 And then, of course, there's also work here on security scanning. 01:14:34.100 --> 01:14:40.040 So this would mean both doing scanning of our code for insecure patterns and 01:14:40.040 --> 01:14:44.980 the like, but also running proactive scanning on dependencies that we take. 01:14:45.120 --> 01:14:48.380 The first time they're brought into the company, anytime there's a new version 01:14:48.380 --> 01:14:51.040 and so on, to actually scan them for. 01:14:52.790 --> 01:14:56.530 Is this a dependency that we want to take? And some of that could be human review. 01:14:56.770 --> 01:15:01.330 Some of that can be AI-assisted review. And there's a combination of these that also could work. 01:15:01.510 --> 01:15:09.290 We wrote a blog post recently about using AI for assisted vetting of software packages. 01:15:09.790 --> 01:15:11.990 I'll send you the link and you can put it in the show notes. 01:15:12.170 --> 01:15:17.050 And so that contains some more thoughts about how you can not necessarily replace 01:15:17.050 --> 01:15:20.530 the human review here, but at least make it more efficient for humans to review 01:15:20.530 --> 01:15:21.750 those dependencies that you take. 01:15:22.310 --> 01:15:26.370 And so there's like a whole host of techniques where you kind of need to do 01:15:26.370 --> 01:15:31.170 all of them because they end up giving you, so each one gives you sort of partial 01:15:31.170 --> 01:15:32.990 coverage of the stack that you have. 01:15:33.250 --> 01:15:37.250 And only when you combine all of them do you get the defenses that you need. 01:15:37.630 --> 01:15:42.190 But even then, you know, taking dependency ultimately is a risk. 01:15:42.390 --> 01:15:46.250 And so you have to take the calculator risk of is the upside of taking this 01:15:46.250 --> 01:15:49.430 dependency worth the potential risk that you're introducing. 01:15:49.430 --> 01:15:54.730 But I do think that there's a genuine security case for taking dependencies 01:15:54.730 --> 01:16:00.930 because the alternative, if you build everything in-house, is that you will not have the people, 01:16:01.530 --> 01:16:07.570 especially sort of subject matter experts, to maintain those internal implementations over time. 01:16:07.790 --> 01:16:10.190 So if we internally implement it, I don't know. 01:16:10.900 --> 01:16:16.780 I mean, crypto is the obvious example of like the old adage of you should not roll your own crypto. 01:16:17.120 --> 01:16:21.260 If we personally like implemented all of our own crypto libraries, 01:16:21.520 --> 01:16:25.740 I'd be deeply uncomfortable with that because we don't have enough, you know, 01:16:26.220 --> 01:16:33.100 cryptography specialists and analysts and engineers to A, build it in the first 01:16:33.100 --> 01:16:35.780 place and then B, maintain it over time. 01:16:35.780 --> 01:16:41.180 So I would much rather that be a publicly vetted, widely used. 01:16:41.960 --> 01:16:45.940 Continuously handled by a large number of security experts, and then we take 01:16:45.940 --> 01:16:51.180 a dependency on it, is a much better and more secure decision for your dependency 01:16:51.180 --> 01:16:52.900 chain than in-housing everything. 01:16:52.900 --> 01:16:58.520 And the question really becomes that risk-reward trade-off of at what point 01:16:58.520 --> 01:17:03.760 does it become better to just in-house that dependency so that you don't take 01:17:03.760 --> 01:17:08.380 an external dependency on it because the upkeep is not that bad or the upkeep 01:17:08.380 --> 01:17:12.420 is not likely to have security implications compared to the third-party dependency. 01:17:13.120 --> 01:17:20.620 There's a common trope that people use, which is Rust's package ecosystem is similar to NPMs. 01:17:20.780 --> 01:17:26.420 We have a lot of smaller packages, and that exposes us to bigger risk. 01:17:26.760 --> 01:17:27.780 What's your take on that? 01:17:28.880 --> 01:17:37.240 Yes and no. It is true that Rust tends to have more dependencies than Java or C++, for example. 01:17:37.540 --> 01:17:40.180 It tends to have more but smaller dependencies. 01:17:41.410 --> 01:17:45.650 I think the jury's still out on whether that's a good thing or a bad thing, 01:17:45.670 --> 01:17:50.350 because the downside with taking a large dependency is that A, 01:17:50.570 --> 01:17:56.550 the large dependency means the maintainer of that project is maintaining way more code. 01:17:57.030 --> 01:18:00.630 And chances are they're not an expert in the entirety of that code. 01:18:00.770 --> 01:18:06.930 And so the likelihood that any part of it is under maintained or under vetted 01:18:06.930 --> 01:18:08.750 or underdeveloped is much higher. 01:18:09.950 --> 01:18:15.530 And then the other is the breaking changes sort of update cadence tends to be 01:18:15.530 --> 01:18:18.830 worse because if you have one giant dependency that you have to, 01:18:19.050 --> 01:18:22.890 like now they make a breaking change, that might be a lot harder for you to 01:18:22.890 --> 01:18:26.110 adopt because you have to adopt all or nothing of the entire dependency. 01:18:26.390 --> 01:18:30.070 Whereas if you have many smaller dependencies, fewer of them make breaking changes 01:18:30.070 --> 01:18:31.290 at any given point in time. 01:18:31.430 --> 01:18:35.390 So more of them will be fully up to date because they don't need to align on 01:18:35.390 --> 01:18:37.010 like a single breaking change schedule. 01:18:38.230 --> 01:18:42.590 But ultimately, you know, there are some things where taking a large dependency 01:18:42.590 --> 01:18:46.430 is probably worthwhile. And we do see this in the Rust ecosystem too. 01:18:46.590 --> 01:18:51.730 If you look at things like Bevy, for example, right? Bevy is effectively one 01:18:51.730 --> 01:18:54.170 big dependency. Tauri is another one. 01:18:54.450 --> 01:18:57.290 And so Rust doesn't preclude you from doing this. 01:18:57.370 --> 01:19:01.690 It's more that I think it often makes sense to have smaller dependencies of 01:19:01.690 --> 01:19:08.230 this kind. And I don't think that's inherently a, it's not obvious to me that 01:19:08.230 --> 01:19:10.470 it's a guaranteed security risk compared to the alternative. 01:19:10.910 --> 01:19:14.030 You also have the downside with large dependencies that you could end up with 01:19:14.030 --> 01:19:16.830 because it's so large, you need to have many maintainers. 01:19:16.930 --> 01:19:19.850 And so you have a large number of maintainers that, 01:19:21.090 --> 01:19:25.550 Would it be better if that same number of maintainers each maintained a smaller 01:19:25.550 --> 01:19:28.050 library that was a subset of the overall thing? 01:19:28.310 --> 01:19:30.510 I think that might be better. I'm not sure. 01:19:31.030 --> 01:19:37.410 But yeah, it's not clear to me that Rust is in more of a danger because it has 01:19:37.410 --> 01:19:41.650 this coarse or this finer granularity of packages. 01:19:42.110 --> 01:19:49.910 I really like your take on Rust's great ecosystem and also contrasting it with 01:19:49.910 --> 01:19:52.430 whatever Node and NPM provide. 01:19:52.830 --> 01:19:59.430 What about unsafe code, though? Because this is very unique to Rust and to how 01:19:59.430 --> 01:20:01.290 we think about safety and code. 01:20:02.690 --> 01:20:07.310 If you look at this from a perspective of supply chain security, 01:20:07.690 --> 01:20:13.190 aren't we exposing ourselves to a lot of risk by taking on a lot of unsafe code? 01:20:13.190 --> 01:20:16.890 And also how would you vet for that. 01:20:16.890 --> 01:20:26.070 Well it's complicated because unsafe code is not inherently less safe even though 01:20:26.070 --> 01:20:30.710 the name kind of implies that right because if you look at java if you look 01:20:30.710 --> 01:20:36.250 at go if you look at node.js and and certainly if you look at c and c plus plus. 01:20:37.540 --> 01:20:41.520 Those languages have no guardrail for what is safe and what is unsafe. 01:20:42.200 --> 01:20:45.780 You know, if you look at Java, you have unsafe operations in Java as well, 01:20:45.840 --> 01:20:49.620 where you do direct pointer manipulation and you can do really bad things there 01:20:49.620 --> 01:20:52.920 and you can violate memory safety and you can do all these things. 01:20:53.780 --> 01:20:58.280 In C++, like all the guardrails are just off. Even if you turn on a lot of the 01:20:58.280 --> 01:21:00.840 compiler validation, like you can just do these things. 01:21:01.160 --> 01:21:05.620 It's just that in Rust, it's more obvious when you do these things than it is in those languages. 01:21:05.620 --> 01:21:09.760 And so I think in Rust, the reality 01:21:09.760 --> 01:21:15.480 is that safe and unsafe is more of a communication mechanism to say, 01:21:15.480 --> 01:21:20.600 this part of the code, you should look at more carefully because it needs to 01:21:20.600 --> 01:21:24.520 sort of uphold things that are not checked by the compiler. 01:21:24.740 --> 01:21:29.780 And so it's places where you get fewer of the benefits of safety from Rust, 01:21:30.000 --> 01:21:35.520 but they're not places that are inherently less safe than general third-party software. 01:21:36.020 --> 01:21:40.220 I do think actually that, you know, you should think a little bit more about 01:21:40.220 --> 01:21:42.680 taking a dependency that includes unsafe than one that does not. 01:21:42.840 --> 01:21:46.520 But I don't think it should be a thing that, you know, excludes you from taking 01:21:46.520 --> 01:21:49.080 that dependency or see it as significantly more risky. 01:21:49.560 --> 01:21:53.720 Where I worry more is when you have crates that have no business having unsafe 01:21:53.720 --> 01:21:54.940 code, but they do anyway. 01:21:55.220 --> 01:21:59.560 Usually this is for like performance optimizations reasons or just they want 01:21:59.560 --> 01:22:00.840 to work around the borrow checker. 01:22:01.280 --> 01:22:05.960 That kind of use I'm more skeptical of, but it's not really a binary of is it unsafe or is it not? 01:22:06.740 --> 01:22:09.880 Do you think unsafe, the term, was a misnomer? 01:22:10.580 --> 01:22:15.380 Yeah, I think in a way it is, because there are two uses of unsafe in Rust. 01:22:15.720 --> 01:22:20.700 One of them is on function definitions, and the other is on blocks inside of code. 01:22:21.980 --> 01:22:25.620 On blocks inside of code, it really should be described as safe. 01:22:25.860 --> 01:22:30.960 The goal of that annotation is to claim that the code between this curly bracket 01:22:30.960 --> 01:22:36.140 and this curly bracket is not subject to the standard compiler checks or it's 01:22:36.140 --> 01:22:40.460 allowed to break some of the rules or it's not checked that it doesn't break the rules. 01:22:40.620 --> 01:22:44.040 But trust me, I have checked that it is safe. 01:22:44.320 --> 01:22:46.760 That's what you're asserting by putting unsafe around a block. 01:22:47.100 --> 01:22:51.640 And so it's not to say this code is unsafe. It's actually to say this code is 01:22:51.640 --> 01:22:54.500 safe. It's just that it's checked by me, not by the compiler. 01:22:54.940 --> 01:23:00.240 And when it's put on a function definition, unsafe is more of an appropriate 01:23:00.240 --> 01:23:01.520 term because it's saying, 01:23:01.520 --> 01:23:08.080 this function is unsafe to call unless the following is true yeah but so i i 01:23:08.080 --> 01:23:12.820 do i do wish that there was a different name for for the the unsafe in a block. 01:23:12.820 --> 01:23:19.200 I like that you said assert in that context because yeah it feels like an assertion 01:23:19.200 --> 01:23:21.060 maybe it could be called assert safe. 01:23:21.880 --> 01:23:26.240 Yeah that is the the challenge right is that asserts we think of as something 01:23:26.240 --> 01:23:31.620 that can fail and the assertion here can't fail like the the program can't fail 01:23:31.620 --> 01:23:36.300 to run as a result of that assert it doesn't actually check anything it's like 01:23:36.300 --> 01:23:39.640 a signature that i just trust me i've checked. 01:23:42.500 --> 01:23:44.560 What's next for Helsing and for you. 01:23:46.120 --> 01:23:51.980 For Helsing, I think it's continued development of the main product lines that 01:23:51.980 --> 01:23:56.380 we currently have, all of which are fairly ambitious and have some really cool tech in them. 01:23:56.660 --> 01:24:01.520 And so this includes the CA-1 is the project I work on, which is basically an 01:24:01.520 --> 01:24:06.020 effort to try to build a self-flying jet fighter in two years. 01:24:06.180 --> 01:24:09.060 So it's a very ambitious timeline. We're building the whole thing, 01:24:09.160 --> 01:24:10.360 hardware and software, from scratch. 01:24:11.220 --> 01:24:14.320 It's no joke, but it is really interesting work. 01:24:14.940 --> 01:24:21.700 And then we have the SG-1, which is an underwater, also autonomous vehicle that 01:24:21.700 --> 01:24:27.460 aims to do business to look for things like submarine traffic in large underwater areas. 01:24:27.680 --> 01:24:31.120 Think like the Baltic Sea, for example, where you need these things that can 01:24:31.120 --> 01:24:34.620 stay underwater for very long periods of time, have very minimal compute, 01:24:34.940 --> 01:24:39.100 very limited ability to, you know, very constrained budgeting for power. 01:24:39.640 --> 01:24:42.920 They can't have any moving parts in them because then they're too easy to detect 01:24:42.920 --> 01:24:47.140 and then you still need to write software on there that does sonar analysis 01:24:47.140 --> 01:24:51.400 for example so they need to run ML models how do you do that when you have extremely 01:24:51.400 --> 01:24:52.580 constrained battery power. 01:24:53.320 --> 01:24:56.140 And so that's some interesting stuff and then obviously we do 01:24:56.140 --> 01:25:03.000 a lot of work with the HX-2 for use in Ukraine for instance where that is a lot 01:25:03.000 --> 01:25:10.580 of work on being close to the sharp end of defense but also systems where safety 01:25:10.580 --> 01:25:13.940 criticality is enormously important. 01:25:14.140 --> 01:25:19.240 And it might sound self-contradictory to talk about safety critical in a system 01:25:19.240 --> 01:25:23.520 that explodes, but the reality is that it is extremely important that it explodes 01:25:23.520 --> 01:25:26.200 in the right place for the right reasons at the right time. 01:25:26.340 --> 01:25:29.940 And that's where the safety criticality comes in. And the cost of getting that 01:25:29.940 --> 01:25:34.280 wrong can be catastrophic. as a working on these systems, I think is, 01:25:34.480 --> 01:25:40.760 you know, it's, I don't like to describe it as fun because it feels incorrect, but it is challenging. 01:25:40.940 --> 01:25:43.160 It's interesting. It is... 01:25:44.020 --> 01:25:48.180 I think it's meaningful. I think it's important. And I think, 01:25:48.240 --> 01:25:52.040 you know, in terms of the future of Helsing, a lot of that becomes continuing 01:25:52.040 --> 01:25:57.780 to work on these products and other ones to see how far we can go with, 01:25:57.940 --> 01:26:03.340 you know, using software to build really good deterrence capabilities for Europe. 01:26:03.760 --> 01:26:07.420 And for you, which streams can we expect in the near future? 01:26:08.000 --> 01:26:11.820 I actually tend to not plan my streams very far ahead. 01:26:11.820 --> 01:26:17.520 Instead, what I do is I look for things that I want to exist, 01:26:17.700 --> 01:26:21.800 either at work or in my personal life, and then I build those and then I turn 01:26:21.800 --> 01:26:23.080 on the camera while building them. 01:26:23.460 --> 01:26:27.580 Because if I try to do streams that are where the content is... 01:26:28.750 --> 01:26:34.650 Hand-picked for streaming. It sometimes works if I pick a topic that I know 01:26:34.650 --> 01:26:39.830 is particularly deep and gnarly in there, but it's much more compelling when 01:26:39.830 --> 01:26:43.030 I can say what my use case is and what I'm building towards. 01:26:43.030 --> 01:26:46.630 It means that I can build a solution that actually solves a problem and that 01:26:46.630 --> 01:26:49.710 therefore comes with additional constraints on the implementation, 01:26:50.330 --> 01:26:56.750 a less generic goal, because I think that's where a lot of really good software 01:26:56.750 --> 01:27:01.430 engineering practices come in are when you're not just writing code, 01:27:01.510 --> 01:27:03.250 but you're writing code for a purpose. 01:27:03.550 --> 01:27:07.650 And so that's why my streams tend to be when I have a thing to build, 01:27:07.910 --> 01:27:11.190 then I do a stream on that topic, which was the case recently, 01:27:11.330 --> 01:27:15.990 for example, for the Avro IDL converter was I needed one of those, so I built it. 01:27:16.170 --> 01:27:20.530 And I think increasingly we'll see other examples of this of I need a thing 01:27:20.530 --> 01:27:22.910 either for work or for my personal life, and I will build them. 01:27:23.370 --> 01:27:26.330 That's how I've done my streams until now and how they will continue. 01:27:27.490 --> 01:27:30.710 And so far, your intuition hasn't failed you around that. 01:27:31.490 --> 01:27:37.030 That's true. I have some ideas for other forms of streams or other forms of 01:27:37.030 --> 01:27:40.850 educational material that I have sort of half-baked in the back of my head. 01:27:40.870 --> 01:27:44.450 I think some of them could be really good, that the challenge is always, 01:27:45.070 --> 01:27:46.310 when will I find the time? 01:27:46.570 --> 01:27:50.970 And so this could be things like, I've had an idea for a sort of Rust intermediate 01:27:50.970 --> 01:27:56.410 course that would actually be a structured course rather than just a sort of ad hoc streams. 01:27:56.730 --> 01:27:58.670 I've had ideas for more of a, 01:27:59.530 --> 01:28:04.950 casual chat beginner's introduction to Rust that comes in shorter snippets rather 01:28:04.950 --> 01:28:05.950 than like 10-hour streams. 01:28:06.210 --> 01:28:10.150 I think that could be really cool, but finding the time is always the hard part. 01:28:10.470 --> 01:28:14.510 There's a ton more that we could discuss, but we have to get to the end. 01:28:14.650 --> 01:28:19.070 And traditionally, our final question is, what's your message to the Rust community? 01:28:19.530 --> 01:28:23.710 I think my message to the Rust community is twofold. I think the first half 01:28:23.710 --> 01:28:31.410 is Rust is now a language that is actively in use in important systems across the world. 01:28:31.930 --> 01:28:36.810 And that's a good thing, right? This is what a programming language aims for, 01:28:37.010 --> 01:28:41.730 is to be successful to the point where it's adopted for real use cases that 01:28:41.730 --> 01:28:42.610 makes a real difference. 01:28:43.490 --> 01:28:49.050 But the result of this is that companies care about what happens to the language 01:28:49.050 --> 01:28:51.850 and the direction the language goes in. And I think this is sort of, 01:28:52.980 --> 01:28:57.060 a little complicated for the Rust community, which traditionally has had very 01:28:57.060 --> 01:29:03.720 strong opinions on not just how the language is used, but also sort of there's 01:29:03.720 --> 01:29:05.560 been an association, I think, 01:29:05.740 --> 01:29:10.640 with like a value judgment on what the language is used for. 01:29:14.020 --> 01:29:20.480 And increasingly, I think the Rust community will have to come to terms with 01:29:20.480 --> 01:29:23.880 the fact that it's being used in production in ways that we don't control. 01:29:24.320 --> 01:29:28.600 And I think we need to decide how to interface with that. 01:29:28.820 --> 01:29:32.680 A good example of this is like sponsorship of Rust conferences, right? 01:29:32.760 --> 01:29:35.460 Or in general, like sponsorship of the Rust Foundation, for example, 01:29:35.600 --> 01:29:39.900 like putting money into the Rust ecosystem, where this to me is like a good 01:29:39.900 --> 01:29:42.520 thing for the Rust community. I want that to happen. 01:29:42.660 --> 01:29:47.680 I want the community to get the language of the funding that it needs to continue 01:29:47.680 --> 01:29:48.900 to grow and continue to develop. 01:29:49.240 --> 01:29:55.240 But that comes at the cost that the language needs to prove that that influx of money is worthwhile. 01:29:55.640 --> 01:30:00.260 Because in general, if you want the funds to continue to come, 01:30:00.500 --> 01:30:03.460 there needs to be something that you get back for the money you put in. 01:30:03.680 --> 01:30:07.360 And that's not to claim that I have the answer for this, but I think there's 01:30:07.360 --> 01:30:13.640 been a sort of allergic reaction in the Rust community to industry involvement in the language. 01:30:14.620 --> 01:30:21.200 And I understand why in many ways, but I do think that it is an allergic reaction. 01:30:21.200 --> 01:30:25.320 We have to figure out how to mitigate because otherwise the funding for the 01:30:25.320 --> 01:30:30.940 language dries up and we end up with a language that doesn't have the funding to keep growing. 01:30:31.920 --> 01:30:34.960 And then I think that the second half of my takeaway from the Rust community 01:30:34.960 --> 01:30:37.480 is we've built something that's really good. 01:30:37.760 --> 01:30:40.840 Like the Rust language is really good. The tooling is really good. 01:30:40.940 --> 01:30:42.060 The community is really good. 01:30:42.460 --> 01:30:45.500 And I think that the ecosystem as well is really good. 01:30:46.060 --> 01:30:51.200 But I also think I've observed a sort of stagnation in the, 01:30:52.280 --> 01:30:55.620 I don't know how to describe it, like the creativity of the use of the language 01:30:55.620 --> 01:30:57.640 compared to some of the early days. 01:30:58.420 --> 01:31:01.220 And that might be because we've solved many of the problems, right? 01:31:01.400 --> 01:31:05.720 But it does feel like there are still cool things you could do with the language 01:31:05.720 --> 01:31:07.880 that can change how the ecosystem works. 01:31:08.020 --> 01:31:12.600 serde was a good example of this from back in the day where it was a different way of doing things. 01:31:12.600 --> 01:31:18.320 And then it made us have things like derived procedural macros and the ability 01:31:18.320 --> 01:31:23.420 to do serialization in a really efficient and cross-language ecosystem way. 01:31:23.740 --> 01:31:28.680 And I want to see more of those kinds of ambitious, let's build something that 01:31:28.680 --> 01:31:35.320 is different, is better, is cool, new, innovative in the language, in the ecosystem. 01:31:35.440 --> 01:31:39.200 I'm seeing less of a hunger for that than I think I saw in the early days. 01:31:39.200 --> 01:31:40.820 And that makes me a little sad. 01:31:42.260 --> 01:31:47.840 Well here's hoping that people like you who are ambassadors of the language 01:31:47.840 --> 01:31:52.140 will bring some of that passion back and i certainly hope to see you on the 01:31:52.140 --> 01:31:56.860 stream i will follow along Jon thanks so much for taking the time and for being 01:31:56.860 --> 01:31:59.200 part of this community no. 01:31:59.200 --> 01:32:00.760 Thanks for having me it was a fun conversation. 01:32:00.760 --> 01:32:06.140 Rust in production is a podcast by corrode it is hosted by me Matthias Endler 01:32:06.140 --> 01:32:08.180 and produced by Simon Brüggen. 01:32:08.320 --> 01:32:12.660 For show notes, transcripts, and to learn more about how we can help your company 01:32:12.660 --> 01:32:15.540 make the most of Rust, visit corrode.dev. 01:32:15.760 --> 01:32:18.120 Thanks for listening to Rust in Production.