WEBVTT 00:00:01.570 --> 00:00:06.210 This is Rust in Production, a podcast about companies who use Rust to shape 00:00:06.210 --> 00:00:07.330 the future of infrastructure. 00:00:07.690 --> 00:00:12.150 My name is Matthias endler from corrode, and today we talk to Vegard Sandein 00:00:12.150 --> 00:00:15.530 from KSAT about talking to satellites with Rust. 00:00:19.310 --> 00:00:23.830 Vegard, can you introduce yourself and KSAT, the company you work for? 00:00:24.650 --> 00:00:29.010 Thanks for having me. My name is Vegard Sandengen. I have a master's in computer 00:00:29.010 --> 00:00:33.610 science, I have worked most of my professional career in the space domain, 00:00:33.970 --> 00:00:39.590 even though it's usually on the ground, and I've been working at KSAT now for the last four years. 00:00:40.310 --> 00:00:46.290 And recently, you became a father, so there's one more rustacean in this world. Congratulations. 00:00:47.049 --> 00:00:47.390 Thank you. 00:00:47.750 --> 00:00:51.810 So, can you say a few words about KSAT? I know that the slogan is, 00:00:52.130 --> 00:00:55.710 connect space and Earth, and I really like that, but what is it about? 00:00:56.490 --> 00:01:01.310 KSAT is the abbreviation of the company, which is... Kongsberg satellite sources. 00:01:01.750 --> 00:01:06.610 So we're getting data from space to Earth, and then we're using that data. 00:01:06.890 --> 00:01:10.030 So ground network operations and Earth observation networks. 00:01:10.270 --> 00:01:12.310 I work in the ground network, which 00:01:12.310 --> 00:01:17.970 is our distributed network of antennas situated all around the world. 00:01:18.110 --> 00:01:24.250 And we enable satellite owners to talk with their satellites and get their data. 00:01:24.250 --> 00:01:32.170 A lot of people only know about satellite technology from television or from popular science. 00:01:32.650 --> 00:01:38.210 And the knowledge they have is probably rooted in the 60s and 70s. 00:01:38.430 --> 00:01:41.310 But a lot has happened since then. 00:01:41.670 --> 00:01:44.310 What has happened since the 60s? 00:01:45.110 --> 00:01:50.070 Yeah, satellite industry was traditionally operated by satellite companies, 00:01:50.070 --> 00:01:53.150 and they're using their software to just deliver 00:01:53.150 --> 00:01:56.530 on whatever their satellite had and a satellite 00:01:56.530 --> 00:01:59.890 itself started like in the 60s with the russians launching 00:01:59.890 --> 00:02:02.950 Sputnik and it was very expensive i mean launching a 00:02:02.950 --> 00:02:06.290 satellite took a government agency so all 00:02:06.290 --> 00:02:11.889 the way until basically Space X and a lot of the other newcomers in the satellite 00:02:11.889 --> 00:02:16.410 business or in the launch business came along it was extremely expensive to 00:02:16.410 --> 00:02:21.550 launch satellites so it was mostly just agencies and government entities that 00:02:21.550 --> 00:02:23.970 it could afford to put satellites into orbit. 00:02:24.230 --> 00:02:30.270 And some of those satellites are geostationary satellites in delivering your 00:02:30.270 --> 00:02:34.850 satellite communications for TV or for your sat phone, if you had that. 00:02:35.430 --> 00:02:41.250 And from the old days, it was mostly communication-based, but NASA and ESA also 00:02:41.250 --> 00:02:48.830 launched scientific instruments to monitor the earth or to monitor the sun or 00:02:48.830 --> 00:02:53.910 to send probes into outer space to do some other readings. 00:02:55.230 --> 00:02:59.610 And the way satellites communicate is almost, 00:03:00.380 --> 00:03:06.040 exclusively through radio frequency communication and different wavelengths 00:03:06.040 --> 00:03:07.720 on the radio frequency spectrum. 00:03:08.199 --> 00:03:15.200 And the type of wavelength you use determines what's the quality of your transmission. 00:03:15.840 --> 00:03:20.740 And Earth observation satellites, very close to the Earth, they orbit the Earth 00:03:20.740 --> 00:03:27.100 maybe 14, and 15 times a day, they can produce a lot of data. 00:03:27.600 --> 00:03:33.580 And as the instruments have gotten better, the resolution of whatever measurements 00:03:33.580 --> 00:03:35.180 they're doing is getting higher. 00:03:35.560 --> 00:03:40.480 The amount of data is getting higher. And the majority of the way to actually 00:03:40.480 --> 00:03:43.180 get data done is to have contact with the ground station. 00:03:43.380 --> 00:03:47.460 And you have a limited visibility over a ground station. 00:03:47.600 --> 00:03:53.380 So you only get like 10 to 15 minutes of visibility maximum that's peak and 00:03:53.380 --> 00:03:58.220 you have to push down gigabytes of data so the amount of data we're talking 00:03:58.220 --> 00:04:00.160 about is ever increasing. 00:04:00.160 --> 00:04:04.840 Yeah thanks for the overview but one thing i always wondered as some somewhat 00:04:04.840 --> 00:04:12.000 of a bystander is what is the standardization of the communication protocols. Do 00:04:12.000 --> 00:04:17.480 we keep using the same protocols since the 60s or does every satellite have 00:04:17.480 --> 00:04:19.640 its own protocol or is it something in between? 00:04:20.339 --> 00:04:26.320 It's everything and nothing, unfortunately. So there is a standardization body 00:04:26.320 --> 00:04:33.140 called CCSDS that a lot of the government agencies contribute into from the 00:04:33.140 --> 00:04:37.320 early days of the 80s-ish, if I remember correctly. 00:04:37.620 --> 00:04:45.320 So a lot of the hardware-related radio frequency protocols and how to handle 00:04:45.320 --> 00:04:50.600 data on the physical link has a lot of different standards. 00:04:50.980 --> 00:04:56.760 And in order to push data over the air, you also need some error correction 00:04:56.760 --> 00:05:02.380 and you need to be able to sequence your data just like TCP/IP, 00:05:02.820 --> 00:05:07.440 there's an equivalent standard in the space industry. 00:05:09.330 --> 00:05:12.130 Coming into the new space era there's a lot of 00:05:12.130 --> 00:05:15.510 new contenders on the market that are software 00:05:15.510 --> 00:05:18.450 companies that are using spacecrafts and 00:05:18.450 --> 00:05:22.270 not spacecraft companies using software they're 00:05:22.270 --> 00:05:25.610 also not really following some 00:05:25.610 --> 00:05:31.550 of these standards from the agency era so you get a lot of compatibility issues 00:05:31.550 --> 00:05:35.910 where you're basically having to custom fit okay how do we talk to this spacecraft 00:05:35.910 --> 00:05:42.330 because This is a new software company that has just looked at the standard and said, 00:05:42.430 --> 00:05:45.190 ah, we don't really need this. We'll do it our way. And it works for them. 00:05:45.510 --> 00:05:51.890 But at some level, you have a minimum viable product that you can share on a radio frequency level. 00:05:51.910 --> 00:05:56.170 And most people are compatible with that. But after that, all bets are off. 00:05:56.970 --> 00:06:02.710 Sounds like that approach would generate a ton of legacy code in a very short time. 00:06:03.910 --> 00:06:04.390 Yeah. 00:06:05.070 --> 00:06:08.930 Now, let's talk about the size of operations at KSAT. 00:06:09.230 --> 00:06:15.089 KSAT started off 25 years ago as a company, and we started off with one antenna 00:06:15.089 --> 00:06:18.870 and one customer, and that's about it. 00:06:19.029 --> 00:06:23.610 And as KSAT grew its company and this market shift into new space with all these 00:06:23.610 --> 00:06:28.830 new software actors really exploded the number of satellites launched into space, 00:06:28.830 --> 00:06:34.490 And KSAT followed suit and built up both their antenna park on how many antennas 00:06:34.490 --> 00:06:39.010 you have and how many employees we have to deal with this and how many engineers. 00:06:39.290 --> 00:06:47.090 And at this point in time, we're roughly at ballpark between 100 to 300 active antennas. 00:06:47.350 --> 00:06:50.990 It is one of the biggest providers of commercial 00:06:51.790 --> 00:06:52.710 ground station services. 00:06:53.150 --> 00:07:00.370 I think the official website mentions 23 sites worldwide, which sounds crazy to me. 00:07:00.750 --> 00:07:05.130 What is a site specifically and what goes in there to maintain that? 00:07:05.270 --> 00:07:09.650 A antenna site for us is mostly, it's a place where we need a lot of power and 00:07:09.650 --> 00:07:12.430 we need fiber optic cable, hopefully. 00:07:12.710 --> 00:07:17.130 We don't have that at every site, but what qualifies as a good site is that 00:07:17.130 --> 00:07:22.990 it's far enough apart from any other site we have, and that it covers a lot 00:07:22.990 --> 00:07:26.430 of ground we don't actually get from other sites in the vicinity. 00:07:26.830 --> 00:07:31.470 And the placement of the sites are usually depending a bit on what orbit the satellites go in. 00:07:31.750 --> 00:07:35.370 So the satellites usually have two orbits that are relevant. 00:07:35.590 --> 00:07:39.110 It's like polar orbit, where they go from pole to pole, and then it's the other 00:07:39.110 --> 00:07:40.710 one where they just follow the equator. 00:07:41.330 --> 00:07:45.670 And if you only have ground station at the equator and you have a polar orbiting 00:07:45.670 --> 00:07:48.690 satellite, you only get the visibility twice a day. 00:07:49.150 --> 00:07:53.990 But if you have a ground station near the poles, you get 10, 00:07:54.210 --> 00:07:58.770 12, 14 contacts a day. So it really depends. 00:07:59.270 --> 00:08:05.250 Each contact has a duration of anything between 5 to 15 minutes, really. 00:08:05.490 --> 00:08:10.270 And that can generate anything from a few gigabytes to 100 gigabytes per contact. 00:08:10.790 --> 00:08:15.610 Data processing can come later, but the data exchange happens during that time frame. 00:08:16.110 --> 00:08:18.890 The data exchange between the satellite and the ground station, yes. 00:08:19.370 --> 00:08:24.350 And because of the volume of data increasing so much, our main concern going 00:08:24.350 --> 00:08:28.010 forward is not really building enough antennas, it's actually just building 00:08:28.010 --> 00:08:31.210 enough infrastructure to handle all this data. 00:08:31.210 --> 00:08:35.590 Because there's so much data and you need to push it around and you need to 00:08:35.590 --> 00:08:39.690 provide it to the customer in a reliable fashion. 00:08:40.090 --> 00:08:48.110 And it can be quite unreliable networking between a remote site in Canada or 00:08:48.110 --> 00:08:53.710 in New Zealand and you have the customer on the west coast of the US. 00:08:54.410 --> 00:08:57.970 That's a challenge really going forward. 00:08:57.970 --> 00:09:01.270 Okay, so to summarize, the setup is a bit like this. 00:09:01.330 --> 00:09:06.730 You have a ton of satellites circling the Earth on a regular basis. 00:09:07.210 --> 00:09:13.770 They go around the Earth 10 to 15 times a day or so, roughly like that. 00:09:13.929 --> 00:09:18.170 And then on the ground, you have antennas on ground stations. 00:09:18.510 --> 00:09:23.150 And then these antennas, they connect with the satellites, do the data exchange, 00:09:23.150 --> 00:09:28.750 and then you need to send the data over, say, Fiber to a central place. 00:09:30.130 --> 00:09:36.190 Usually delivered straight to the customer but due to our volume of data we 00:09:36.190 --> 00:09:44.450 also have to temporarily store it on the site itself but do not lose data in the process so yeah. 00:09:45.250 --> 00:09:48.670 Two things come to mind first it needs 00:09:48.670 --> 00:09:55.210 to be extremely reliable because if you lose the data that is big outage and 00:09:55.210 --> 00:10:00.549 probably a loss to the customer as well and the second part is how often can 00:10:00.549 --> 00:10:06.090 you make changes to that code how often can you modify the code that also needs to be reliable, 00:10:07.030 --> 00:10:11.450 i'm guessing you probably even have limitations as to how often you can access 00:10:11.450 --> 00:10:13.429 these ground stations and make changes. 00:10:13.429 --> 00:10:16.410 Yes that is correct that that 00:10:16.410 --> 00:10:19.070 has to be really reliable but you're actually right on 00:10:19.070 --> 00:10:22.210 point with how do we update the code because it's not 00:10:22.210 --> 00:10:28.070 that we're using the antenna 100 of the time but the ecosystem around the antenna 00:10:28.070 --> 00:10:33.510 with our software running in different hardware close to the antenna it is not 00:10:33.510 --> 00:10:37.650 easy to access i don't have access to it for instance but so i just have to 00:10:37.650 --> 00:10:41.309 push code and hope that someone else deploys it worst case it can take, 00:10:42.250 --> 00:10:46.429 weeks before something is deployed worldwide that 00:10:46.429 --> 00:10:49.290 is a process we're obviously trying to optimize and get better at 00:10:49.290 --> 00:10:52.370 but it's it is a pain point because it's also in 00:10:52.370 --> 00:10:56.790 inaccessible sites and the most inaccessible sites we have is probably in our 00:10:56.790 --> 00:11:02.010 antarctica and troll station that also doesn't have fiber optic cable so anything 00:11:02.010 --> 00:11:06.070 that you put down there we also have to beam up to a geostationary sites satellites 00:11:06.070 --> 00:11:09.750 so we can beam it down to earth again a place where we have fiber. 00:11:10.530 --> 00:11:14.630 I guess the huge advantage here is that for code that is written in Rust, 00:11:14.690 --> 00:11:20.710 you could just deploy a static binary and people would just be able to run it on the deploy target. 00:11:21.450 --> 00:11:27.570 It's generally that easy. I mean, everything we do nowadays is dockerized. 00:11:27.750 --> 00:11:31.230 On all our ground stations, we're running some variant of Kubernetes and just 00:11:31.230 --> 00:11:34.470 running it on OpenStack or Kubernetes directly. 00:11:34.770 --> 00:11:40.929 One could think that since you operate in the ground station and you probably 00:11:40.929 --> 00:11:45.530 have access to a rack or so, you're not resource-constrained. 00:11:45.750 --> 00:11:50.450 But one thing that people might forget is that you don't do constant updates 00:11:50.450 --> 00:11:51.730 to the hardware over there. 00:11:52.049 --> 00:11:55.549 We're definitely resource-constrained on a lot of our sites. 00:11:55.770 --> 00:12:00.090 Not all of them, but a lot of them. It can take us eight months to get a new 00:12:00.090 --> 00:12:05.610 computer just ordered from our vendor, and then we have to ship it to anywhere 00:12:05.610 --> 00:12:09.030 in the world, and you have to get people there on-site to install it. 00:12:09.030 --> 00:12:13.110 So we are resource constrained in the fact that we don't want to over-provision 00:12:13.110 --> 00:12:19.190 every data center around the world near to all our antennas on our ground station sites. 00:12:19.390 --> 00:12:23.590 Because, first of all, we don't necessarily have the resources to do that, 00:12:23.610 --> 00:12:26.670 and we don't have the ability to do it at some point. 00:12:26.830 --> 00:12:30.429 So it's nice to use something that doesn't hog all the resources. 00:12:31.110 --> 00:12:32.790 Wouldn't it then be super easy 00:12:32.790 --> 00:12:38.110 to fall into a trap of being extremely conservative about tech decisions? 00:12:38.110 --> 00:12:44.390 People might associate space technology with a lot of very old conservative 00:12:44.390 --> 00:12:49.530 technology, and maybe for a good reason, because it's tried and tested. 00:12:50.490 --> 00:12:54.250 I think the satellite industry or space industry is definitely very conservative. 00:12:54.470 --> 00:13:00.050 It takes a lot of effort to qualify something to run in space. I know, 00:13:01.160 --> 00:13:05.100 RustConf, last year there was one of the sponsors was K2 Space. 00:13:05.420 --> 00:13:11.000 They're actually a space company that with a lot of recruits from former AWS 00:13:11.000 --> 00:13:16.800 and SpaceX, they wanted to do everything in Rust. They wanted to build the satellite. 00:13:17.040 --> 00:13:20.100 They wanted to build the firmware. They wanted to build all the ground resources. 00:13:20.660 --> 00:13:25.020 100% of Rust. They had a lightning talk at RustConf. It's probably out on YouTube. 00:13:25.380 --> 00:13:29.560 So there are definitely contenders out there that want to not be so conservative. 00:13:29.560 --> 00:13:34.060 But from the old space they are very conservative but i wouldn't say that's 00:13:34.060 --> 00:13:39.120 necessarily true on the ground the ground is a bit more like we can touch this 00:13:39.120 --> 00:13:42.780 we can fix it it's not the same in space earlier. 00:13:42.780 --> 00:13:45.740 You said there was a shift in the industry so 00:13:45.740 --> 00:13:48.640 we moved from space companies using software to 00:13:48.640 --> 00:13:52.660 software companies doing space things mostly two 00:13:52.660 --> 00:13:57.080 companies come to mind right away one would be spacex and the other one would 00:13:57.080 --> 00:14:03.840 be blue origin but i'm assuming that's just a tiny little slice of the picture 00:14:03.840 --> 00:14:08.300 and maybe there are other software companies that i might have heard of that 00:14:08.300 --> 00:14:12.260 pushed into the into the space. 00:14:12.260 --> 00:14:15.460 Into the space yes you also have a few other providers that 00:14:15.460 --> 00:14:18.560 are up there trying to and successfully doing 00:14:18.560 --> 00:14:21.820 so like rocket lab but from but these 00:14:21.820 --> 00:14:25.220 are launch providers they're facilitating the software 00:14:25.220 --> 00:14:28.180 companies to launch something into space but otherwise 00:14:28.180 --> 00:14:31.180 aws is actually going right 00:14:31.180 --> 00:14:35.260 at it and they're going after the data primarily they want the data because 00:14:35.260 --> 00:14:41.040 that's aws's business model is data and there's a lot of data in space a couple 00:14:41.040 --> 00:14:46.060 of years back or three or four years back they launched a ground station service 00:14:46.060 --> 00:14:51.140 which is i I wouldn't say a direct competitor to us, but they are definitely a competitor. 00:14:51.380 --> 00:14:55.920 And we have made a strategic partnership with AWS to be, 00:14:56.840 --> 00:15:02.020 a ground network of network providers. So people can come to us and they can 00:15:02.020 --> 00:15:08.820 use the resources in AWS, their antennas, their setup, but they can do it through us. 00:15:08.900 --> 00:15:13.160 But the business model is a bit different because AWS, as I said, they're a data company. 00:15:13.260 --> 00:15:16.480 They really just care about getting their data into the AWS data center. 00:15:16.680 --> 00:15:18.720 So you can do whatever you want with it there. 00:15:19.300 --> 00:15:22.880 So the space part is just a means to an end, really. 00:15:22.880 --> 00:15:28.420 So we move from space exploration to data exploration. 00:15:28.860 --> 00:15:33.400 What has changed on the language side? How did the story go at KSAT? 00:15:34.620 --> 00:15:40.920 Initially, everything was engineers writing Perl scripts and just making it work. 00:15:41.080 --> 00:15:44.460 And that has scaled very well, but it's still written in Perl, 00:15:44.540 --> 00:15:45.860 and it's not the newest version. 00:15:46.060 --> 00:15:50.660 And at some point, we needed to have a bit more control of whatever is running 00:15:50.660 --> 00:15:56.760 on our antennas. And that was developed in Java in the mid-2000s with an Oracle 00:15:56.760 --> 00:15:58.820 database. And that has scaled well. 00:15:59.320 --> 00:16:03.680 We're very thankful for the legacy that was provided to us so that we can even 00:16:03.680 --> 00:16:06.120 be here today to do something else at a bigger scale. 00:16:06.340 --> 00:16:10.100 Because that would not be possible without the humble beginnings. 00:16:12.160 --> 00:16:18.680 The 2000s were definitely the time of Java. It has some really nice traits, 00:16:18.680 --> 00:16:23.760 and I think it resonated well with the challenges of its time. 00:16:24.100 --> 00:16:28.440 But then what happened in the 2010s at KSAT? 00:16:28.900 --> 00:16:34.300 Yeah, so at some point, we're kind of scaled up with a bit more developers and 00:16:34.300 --> 00:16:38.320 with a bit more modern scripting and kind of Python took over. 00:16:38.520 --> 00:16:42.300 We have multiple Python applications still in production today from that era. 00:16:42.300 --> 00:16:47.940 But yeah, we started to see that due to how that Java application, 00:16:48.240 --> 00:16:53.740 and not necessarily Java in itself, but just the database and all the Perl integrations 00:16:53.740 --> 00:16:57.560 that unfortunately had direct database access, 00:16:57.920 --> 00:17:02.400 meant that we had a distributed network all over the world with scripts being 00:17:02.400 --> 00:17:06.000 able to access the raw contents of our database. 00:17:06.000 --> 00:17:09.720 And that was not very scalable. We launched 00:17:09.720 --> 00:17:12.820 an initiative to move away from this world into 00:17:12.820 --> 00:17:15.800 a more modern world where we can have more 00:17:15.800 --> 00:17:18.740 control over the life cycle of the data that 00:17:18.740 --> 00:17:25.380 we put in the database. 20-25 years ago everything was on ftp xml drop boxes 00:17:25.380 --> 00:17:29.340 you can call that an api as well but we've decided that we can try to offload 00:17:29.340 --> 00:17:35.099 responsibility into like segregated new Postgres database where access to the 00:17:35.099 --> 00:17:38.240 data is tightly controlled through an HTTP API. 00:17:38.740 --> 00:17:42.560 Yeah, so we're employing a sort of a strangle pattern on that and just trying 00:17:42.560 --> 00:17:49.300 to just grope in any responsibilities and kind of rewrite and repurpose it and 00:17:49.300 --> 00:17:51.700 have successfully launched a. 00:17:52.430 --> 00:17:58.490 Competing solution in-house now where half of the antennas are on the old system 00:17:58.490 --> 00:18:07.270 the old api was written in pearl and it was strangled on the hftp layer into a coffin application, 00:18:07.869 --> 00:18:13.070 and then we nipped at it and moved different responsibilities and endpoints around and now, 00:18:13.710 --> 00:18:19.450 i would say from a responsibility point of view where like it's 40 60 in rust 00:18:19.450 --> 00:18:25.330 right now but a lot of the boring parts are in the kotlin application but we're 00:18:25.330 --> 00:18:32.190 actively working to to migrate the remaining kotlin portions as well over trust earlier. 00:18:32.190 --> 00:18:35.130 You mentioned the strangler pattern how does it work. 00:18:35.130 --> 00:18:38.090 So strangler pattern is very 00:18:38.090 --> 00:18:41.470 convenient when you have a code base 00:18:41.470 --> 00:18:44.670 or a interface layer where you 00:18:44.670 --> 00:18:47.950 can very well design know what's 00:18:47.950 --> 00:18:50.710 going in and what's going out and you know that everything below 00:18:50.710 --> 00:18:53.470 this is just complete mess and you don't 00:18:53.470 --> 00:18:56.970 understand anything but you understand the interfaces or 00:18:56.970 --> 00:19:00.550 the boundaries and you can replace the implementation 00:19:00.550 --> 00:19:03.930 under each boundary with very 00:19:03.930 --> 00:19:09.470 great control and see the differences in implementation and behavior and you 00:19:09.470 --> 00:19:14.470 make it entirely seamless to all consumers that you have actually done anything 00:19:14.470 --> 00:19:19.010 which is very nice but it It requires that you have some sort of abstractions 00:19:19.010 --> 00:19:20.790 that actually make this feasible. 00:19:21.250 --> 00:19:28.250 And from an HTTP API layer, very easy, because the contract is in how the API 00:19:28.250 --> 00:19:30.310 responds or what parameter it takes. 00:19:30.690 --> 00:19:33.150 And you can replace that in any language. 00:19:33.570 --> 00:19:38.109 It's not really that hard. It's just a lot of verification that you've actually 00:19:38.109 --> 00:19:40.330 replicated all the behavior. 00:19:41.619 --> 00:19:44.740 Now, let's focus on the API for a second. 00:19:45.440 --> 00:19:49.560 You mentioned that it's a split between Kotlin and Rust at the moment. 00:19:50.060 --> 00:19:51.780 Where do you draw the line? 00:19:52.099 --> 00:19:57.660 I don't think there's a natural split now, other than whatever developer or 00:19:57.660 --> 00:20:00.880 team took that responsibility and what they were comfortable with. 00:20:01.260 --> 00:20:06.460 So we've had a very open policy at KSAT on what languages we would use to solve 00:20:06.460 --> 00:20:12.340 whatever problem. And it has definitely been a pushback to introduce Rust in 00:20:12.340 --> 00:20:16.160 some capacity by some team members in different teams. 00:20:16.359 --> 00:20:21.359 And I'm not necessarily sure all of their concerns are, I would call, 00:20:21.460 --> 00:20:23.080 valid, but there are definitely concerns. 00:20:23.640 --> 00:20:29.140 And some of the pushback I've heard is usually it's not mature enough or the ecosystem is not there. 00:20:29.140 --> 00:20:36.180 And I feel that is a sentiment that is often held about Rust that I'm not necessarily 00:20:36.180 --> 00:20:40.400 sure is true anymore, because I feel the ecosystem is very much present. 00:20:40.619 --> 00:20:43.160 I can do everything I want in the ecosystem in Rust today. 00:20:43.520 --> 00:20:48.580 And the other part is maybe just a lack of knowledge of how do you use such 00:20:48.580 --> 00:20:51.720 complex terms, because it comes from a system background. 00:20:52.040 --> 00:20:56.840 And a lot of regarding boroughs and lifetimes and stuff like that, 00:20:56.880 --> 00:20:58.760 it can seem a bit intimidating. 00:20:59.140 --> 00:21:05.340 To someone that's usually just very happy in their Java or .NET environment 00:21:05.340 --> 00:21:11.240 where that is not necessarily a concern for 99% of what they're doing. 00:21:11.400 --> 00:21:13.940 There are also positive receptions of Rust. 00:21:14.280 --> 00:21:20.240 And I have personally been able to, I don't know, convert a couple of teams to use Rust. 00:21:20.540 --> 00:21:24.460 So yeah, we're approximately three or four teams now using Rust in production 00:21:24.460 --> 00:21:30.320 at KSAT with maybe four-ish people in each team that's actively writing Rust. 00:21:30.700 --> 00:21:35.960 How does that usually go for you when you approach a team and they are curious 00:21:35.960 --> 00:21:38.380 about Rust, but they are not entirely convinced yet? 00:21:39.930 --> 00:21:44.670 The conversation often goes in the direction of this is what is very good about 00:21:44.670 --> 00:21:47.330 Rust, and that's what I start with. 00:21:47.510 --> 00:21:52.450 And you have to make some concessions. And the concessions are obviously just, is it a good team fit? 00:21:52.810 --> 00:21:58.650 Because I don't think Rust is hard to use once you've gotten over that initial, 00:21:58.930 --> 00:22:01.270 whoa, what happened here? It's a shock. 00:22:01.790 --> 00:22:06.869 But a lot of teams have their experiences in their toolboxes in other languages 00:22:06.869 --> 00:22:07.950 and know how to solve them. 00:22:07.950 --> 00:22:10.950 And if you don't really have a champion on that team itself, 00:22:11.290 --> 00:22:16.330 I don't think it's possible to really introduce Rust into a team because the 00:22:16.330 --> 00:22:17.710 team has to embrace it themselves. 00:22:18.010 --> 00:22:22.390 That it's a no-go if the team is not championed from within, really. 00:22:22.390 --> 00:22:30.130 So my job is more just like I try to do some good mentoring and try to have 00:22:30.130 --> 00:22:35.070 some common guidelines and try to curate some crates and make some internal 00:22:35.070 --> 00:22:39.790 crates that help the process along internally with the tooling and the way we do things. 00:22:40.010 --> 00:22:45.070 But ultimately, you require that team champion as well to be on your team. 00:22:46.050 --> 00:22:50.230 What's your success rate here? Have you lost some of these battles? 00:22:51.410 --> 00:22:55.550 Not on a team level, but maybe on an individual level, yes. 00:22:56.690 --> 00:23:02.710 But the general vibe is that it's going more and more into us for a lot of our distributed systems. 00:23:02.950 --> 00:23:07.170 And just because it's so nice to use once you actually get to know it. 00:23:07.730 --> 00:23:11.730 So it's just that hurdle of inviting people in that haven't used it before. 00:23:12.960 --> 00:23:18.140 I'm almost too afraid to ask it, but has Go ever come up in that conversation? 00:23:19.000 --> 00:23:23.099 Go has come up multiple times, and we have production code in Go as well. 00:23:23.340 --> 00:23:28.520 I'm a bit annoyed at that sentiment as well, because Go is maybe annoyed is 00:23:28.520 --> 00:23:33.859 not the right word, and I'm a bit intrigued by the why don't we just do it in Go? 00:23:34.040 --> 00:23:38.220 Because Go was released in March 2012. 00:23:38.840 --> 00:23:43.520 It's three years older than Rust. at this point is 10 and 13. 00:23:43.820 --> 00:23:45.200 It's not that big and much of a difference. 00:23:45.460 --> 00:23:51.760 In terms of age, but in terms of functionality and in terms of developer ergonomics, maybe? 00:23:52.740 --> 00:23:58.440 Yeah, but Go had a very simple language to begin with. So it was very easy to get going with Go. 00:23:58.820 --> 00:24:01.920 But I also think that there is an ecosystem in Go, 00:24:02.119 --> 00:24:07.160 but the ecosystem is harder to engage with than it is the Rust ecosystem because 00:24:07.160 --> 00:24:14.460 the tooling and with cargo on the kin as just miles above any tooling you have in Go. 00:24:15.020 --> 00:24:19.060 So that makes it, for me, also a no-brainer just because, like, 00:24:19.660 --> 00:24:23.500 disregarding just the language itself and the features and ergonomics of the 00:24:23.500 --> 00:24:29.440 language, just the tooling and the ecosystem with using the language is what 00:24:29.440 --> 00:24:33.520 makes Rust the number one contender on the market. 00:24:34.460 --> 00:24:42.180 Go is very much a day-one language. and starting a project and getting to your 00:24:42.180 --> 00:24:47.880 first production version is usually very ergonomic, very quick, very elegant. 00:24:48.720 --> 00:24:52.840 The problems start to arise on day two. Not exactly day two, 00:24:53.000 --> 00:25:00.780 but when you have a larger code base, you feel the limitations of the language of the ecosystem. 00:25:00.780 --> 00:25:06.940 It's trying to constrain you somehow. almost feels like it's strangling you. 00:25:08.130 --> 00:25:09.390 And you're not strangling it. 00:25:10.290 --> 00:25:14.710 I probably would have made the same decision in your position, of course. 00:25:15.530 --> 00:25:20.270 Obviously, I'm biased, but you have to maintain this software for a very long time. 00:25:21.190 --> 00:25:26.110 Yeah, so definitely, from my experience point of view, just being able to model 00:25:26.110 --> 00:25:29.810 your code in a way that just, it just feels, 00:25:30.270 --> 00:25:34.950 you just know where the boundaries of what you've made in the main And it's 00:25:34.950 --> 00:25:39.310 very easy to move that along and refactor it. 00:25:39.750 --> 00:25:47.990 So back in eons ago, I was a C and C++ developer, and I did a bit of that and a bit of that. 00:25:48.490 --> 00:25:53.510 And just trying to refactor a C++ code base and having confidence that you've 00:25:53.510 --> 00:25:56.350 actually done it correctly, I have never had that. 00:25:56.570 --> 00:26:01.490 But Rust, if it compiles it works, it basically is that. 00:26:01.490 --> 00:26:04.350 And that sentiment is overused i think 00:26:04.350 --> 00:26:07.990 but it still feels very true at some point because the 00:26:07.990 --> 00:26:12.850 compiler is so powerful but whenever it compiles i'm confident and i also have 00:26:12.850 --> 00:26:18.970 a few tests here and there and where the tests run as well which they do 99 00:26:18.970 --> 00:26:25.230 percent of the time after i've done a major refactor i'm confident i will push no problem funny. 00:26:25.230 --> 00:26:29.370 That you say you have a few tests here and there does that mean you lean into 00:26:29.370 --> 00:26:32.490 Rust's strong type system a lot as well. 00:26:32.610 --> 00:26:36.370 And maybe you don't have to write that many tests that you would have to write 00:26:36.370 --> 00:26:39.230 in other languages, more dynamic languages like Python. 00:26:39.650 --> 00:26:44.770 Oh, definitely. Our tests is, I think there's a concept called like a diamond-shaped 00:26:44.770 --> 00:26:48.710 testing or something where you basically, you have very few unit tests, 00:26:48.830 --> 00:26:51.590 you have very few system tests, but you have a lot of integration tests. 00:26:51.790 --> 00:26:57.810 And those integration tests are placed on the boundaries of the network layer, so HTTP. 00:26:57.810 --> 00:27:00.750 And I have all my 00:27:00.750 --> 00:27:03.950 tests are basically just HTTP related API tests 00:27:03.950 --> 00:27:08.410 because I don't really care how the structs 00:27:08.410 --> 00:27:13.510 or functionality within the Rust code base behaves because what's important 00:27:13.510 --> 00:27:20.430 is just what is the contract or the HTTP boundaries so we have a few tests down 00:27:20.430 --> 00:27:27.550 to the database over the HTTP layer but from unit test point of view almost nothing Thanks. 00:27:28.210 --> 00:27:34.870 But in order for that to work, you would have to lean very heavily into the 00:27:34.870 --> 00:27:40.910 Rust mechanics, into the type system, and you would have to rely on it. 00:27:41.410 --> 00:27:47.470 Are there patterns that you commonly use to fully embrace that part of Rust? 00:27:48.030 --> 00:27:51.430 Yeah, so I use quite a lot of new types. 00:27:51.730 --> 00:27:57.650 For instance, a UID, I will new type it into a variant that represents this resource. 00:27:58.210 --> 00:28:02.470 Meaning that the API layer is very communicative of what it's actually expecting, 00:28:03.150 --> 00:28:06.030 or not the API layer, but the code base itself that serves. 00:28:06.510 --> 00:28:12.330 So it's very easy to modularize different components that work in some form 00:28:12.330 --> 00:28:17.090 of hierarchy because the types are so strong that you can convey so much with 00:28:17.090 --> 00:28:21.010 both the primitive types themselves, but also some types in form of enums. 00:28:21.210 --> 00:28:25.830 The one thing I miss every time I go to any other language is just the enum. 00:28:25.830 --> 00:28:32.110 I think this I could model very well in an enum, and I don't have this capability. And it saddens me. 00:28:33.150 --> 00:28:38.470 Do you have an example for an enum that, for example, comes to mind where modeling 00:28:38.470 --> 00:28:41.970 some certain business logic was very ergonomic? 00:28:43.090 --> 00:28:46.310 So I'm a big fan of the one-off pattern. 00:28:46.590 --> 00:28:49.810 It is represented in, for instance, OpenAPI definitions. 00:28:50.050 --> 00:28:52.950 There is like a one-off you can represent there. doing code 00:28:52.950 --> 00:28:55.670 gen for one-offs in open api to any other 00:28:55.670 --> 00:28:59.070 language is horrible but code 00:28:59.070 --> 00:29:02.250 gen 2 was very easy to use and being 00:29:02.250 --> 00:29:06.710 able to represent the fact that this resource has different 00:29:06.710 --> 00:29:13.230 properties depending on which kind it is is very powerful because even though 00:29:13.230 --> 00:29:17.730 at some level you're talking about this resource it has one resource id but 00:29:17.730 --> 00:29:23.450 it can manifest itself as different forms of different versions or represent 00:29:23.450 --> 00:29:26.090 different physical attributes on the network. 00:29:26.750 --> 00:29:31.930 And on some abstractions, you don't really care about those properties, but on others you do. 00:29:32.050 --> 00:29:36.710 It's very nice to be able to represent just the exact properties that are present 00:29:36.710 --> 00:29:42.990 and not load of optionals that are present only is this is true and this is true. 00:29:43.190 --> 00:29:47.970 And you have to carry that logic throughout the code, that makes it harder to refactor as well. 00:29:48.170 --> 00:29:52.810 If you know that this can only be set if this other value is set, 00:29:53.050 --> 00:29:57.190 and that's invariance in your code that you kind of encode with the enums instead. 00:29:57.710 --> 00:30:02.910 I'm not too familiar with it, but I know that in a schema you can say this is 00:30:02.910 --> 00:30:08.190 one of these variants, one of these kinds, and I guess it maps really well to enums. 00:30:08.610 --> 00:30:14.070 If you go further one step, you're probably also using the serde ecosystem of things 00:30:14.070 --> 00:30:18.910 and say this is my input type and so I convert it from the schema. 00:30:20.450 --> 00:30:24.730 So we're leaning heavily into serde. It's an excellent library. 00:30:25.490 --> 00:30:29.090 Any other crates that you personally like for that sort of work? 00:30:29.750 --> 00:30:34.930 You usually have to do some customizations on top of serde with serde with or 00:30:34.930 --> 00:30:38.950 stuff like that to actually do the proper transformations. 00:30:39.750 --> 00:30:44.950 I've been also experimenting now with Utopia to generate OpenAPI specifications. 00:30:45.110 --> 00:30:49.970 It's called UTO IPA. It's a very common misspelling, unfortunately. 00:30:50.450 --> 00:30:53.470 I made it a dozen times until someone pointed it out. 00:30:53.690 --> 00:30:56.150 Yeah, I will probably continue to misspell it. 00:30:56.330 --> 00:31:02.830 The reason why it's called UTOIPA, by the way, is IPA is API backwards. 00:31:03.130 --> 00:31:06.250 And it's also a good beer. That's from the READMEs. 00:31:06.410 --> 00:31:10.110 Of course. Sorry. Yeah. 00:31:10.730 --> 00:31:14.950 One slight issue I have with serde is that it's very versatile, 00:31:14.950 --> 00:31:20.590