WEBVTT 00:00:01.470 --> 00:00:06.310 Welcome back to a fresh season of Rust in Production, a podcast about companies 00:00:06.310 --> 00:00:08.830 who use Rust to shape the future of infrastructure. 00:00:09.230 --> 00:00:13.990 My name is Matthias Endler from corrode, and today we talk to Victor Ciura about 00:00:13.990 --> 00:00:16.350 large-scale Rust adoption at Microsoft. 00:00:18.950 --> 00:00:23.850 Victor, thanks so much for joining us today. Can you say a few words about yourself 00:00:23.850 --> 00:00:26.230 and about Microsoft, the company you work for? 00:00:27.330 --> 00:00:33.290 Hey, thanks for inviting me. I'm a principal engineer on the developer division at Microsoft. 00:00:34.770 --> 00:00:42.590 Before coming to Rust, I spent over 20 years doing C++ exclusively systems programming 00:00:42.590 --> 00:00:47.110 in general and developer tools, worked on various projects, including being 00:00:47.110 --> 00:00:49.610 part of the Visual C++ team at Microsoft. 00:00:50.190 --> 00:00:55.610 And in recent years, I transitioned to work on Rust tooling in developer division. 00:00:56.230 --> 00:01:03.550 We both met at Eurorost in 2024, and I really had a pleasant experience listening to your talk. 00:01:03.870 --> 00:01:07.230 It was very honest, brutally honest, let me say. 00:01:07.590 --> 00:01:13.790 And I really appreciated that. And at the same time, one could see that you're 00:01:13.790 --> 00:01:15.350 a very experienced engineer. 00:01:15.630 --> 00:01:20.490 So I'm really looking forward to that conversation today, because I think that 00:01:20.490 --> 00:01:25.010 we will not only talk about the highlights of using Rust in production, 00:01:25.010 --> 00:01:27.350 but also maybe some of the downsides. 00:01:28.150 --> 00:01:33.450 But before we start, there's probably a lot to talk about, but before we start, 00:01:33.650 --> 00:01:40.110 can you give us an overview and maybe an update of what happened since Eurorust 2024? 00:01:40.730 --> 00:01:46.370 Yeah, definitely. So things in Microsoft, across all organizations in Microsoft, 00:01:46.410 --> 00:01:53.090 Microsoft is a very big place, but things in terms of Rust have moved a lot in the past year. 00:01:53.710 --> 00:02:00.850 And there's so many things brewing. Many of the projects are publicly facing, 00:02:01.110 --> 00:02:05.270 so there's public knowledge, but many of the things we're working on are not 00:02:05.270 --> 00:02:07.490 yet ready to be shared. So stay tuned. 00:02:07.970 --> 00:02:12.630 More stuff is going to be made public this calendar year. But definitely a lot 00:02:12.630 --> 00:02:14.770 of stuff has been going on. 00:02:15.010 --> 00:02:20.530 So back then, we talked about a few of these early efforts with Rust across 00:02:20.530 --> 00:02:27.050 the company, but I have a few more sort of that I wish I could spotlight here. There's, 00:02:27.630 --> 00:02:33.390 All sorts of interesting projects, in my opinion, in my position in the developer division, 00:02:33.390 --> 00:02:39.210 because we are tasked to improve the lives of Rust developers in the company 00:02:39.210 --> 00:02:44.550 and improve developer tooling and sort of unblock them and make sure they're 00:02:44.550 --> 00:02:48.610 productive and doing their best work across all the products and services we have. 00:02:48.610 --> 00:02:54.410 But you can imagine, I hear from all teams across the company and they're either 00:02:54.410 --> 00:02:57.510 telling us how they're using Rust and what they love about it, 00:02:57.610 --> 00:03:01.010 but also complaining about stuff that doesn't work for them. 00:03:01.150 --> 00:03:02.850 And, oh, they're so loud. 00:03:03.930 --> 00:03:10.230 So my team specifically is tasked to make them productive and improve their lives. 00:03:10.410 --> 00:03:16.090 So in this role and in this team, I hear about a lot of interesting projects. 00:03:16.090 --> 00:03:21.910 So they all sort of fascinate me. So, and they're areas that I've never worked 00:03:21.910 --> 00:03:26.130 in and they're so diverse in nature from firmware, 00:03:26.330 --> 00:03:31.930 hardware stuff related to services and SDKs and all sorts of interesting things 00:03:31.930 --> 00:03:34.070 that I've never worked with. 00:03:34.230 --> 00:03:38.730 And now I get to sort of meet these projects and see how they're using Rust 00:03:38.730 --> 00:03:42.890 and the kinds of challenges they're facing and their successes along the way. 00:03:43.050 --> 00:03:47.230 So I wanted to maybe share some of these with you. yeah sure. 00:03:47.230 --> 00:03:49.750 Let's kick it off by highlighting some of the projects. 00:03:49.750 --> 00:03:52.530 So there's there's work we've been 00:03:52.530 --> 00:03:55.610 doing for example with TockOS 00:03:55.610 --> 00:03:58.430 i don't know if people are familiar with probably your audience is 00:03:58.430 --> 00:04:04.130 familiar with TockOS kernel which is written in in rust we've used this in 00:04:04.130 --> 00:04:09.570 a project for what we call the microsoft pluton security processor we've done 00:04:09.570 --> 00:04:14.670 some collaboration in open source here because we're using this and we've bet 00:04:14.670 --> 00:04:16.250 on this one-source project. 00:04:16.510 --> 00:04:23.810 And it's used in Surface PCs, and we're using this in collaboration with our 00:04:23.810 --> 00:04:27.670 OEM partners as well as how we're building laptops and PCs. 00:04:28.770 --> 00:04:31.870 Just for the people who might not be familiar with TockOS, 00:04:32.150 --> 00:04:35.210 as far as I'm aware, and I'm not an expert on the topic, 00:04:35.370 --> 00:04:40.550 but I think it is a real-time operating system that is very low level, 00:04:40.550 --> 00:04:47.610 very close to the hardware and it's perfect for when you might need a very, 00:04:48.350 --> 00:04:51.770 lean runtime and you want to run certain, 00:04:52.690 --> 00:04:57.490 processes or certain tasks on top of that operating system, which more or less 00:04:57.490 --> 00:04:59.110 have to react to real-time data. 00:04:59.730 --> 00:05:03.310 Exactly. And it's, for example, in the case of these Copilot PCs 00:05:04.190 --> 00:05:07.790 where we need it, we're using the Intel Partner Security Engine there, 00:05:08.030 --> 00:05:11.610 which is a specialized hardware for security on these devices. 00:05:11.830 --> 00:05:16.770 And we bet on TockOS as a good fit for this. 00:05:17.010 --> 00:05:21.770 And sort of that's how we ended up collaborating on this existing project, 00:05:21.870 --> 00:05:27.570 which was an amazing project and was deemed as a good fit for this. 00:05:27.790 --> 00:05:31.950 And by collaborating, you mean contributing actual code to the project? 00:05:32.810 --> 00:05:38.890 I can give you, it's a funny story. People think that 32-bit is by all means dead. 00:05:39.110 --> 00:05:44.790 So many, many people don't even think in terms of, oh, do I need 32-bit or something? 00:05:44.790 --> 00:05:50.550 So for this particular Intel chip that we use for this security thing, 00:05:50.810 --> 00:05:53.650 it's actually, it needs 32-bit, right? 00:05:53.790 --> 00:05:58.710 So we did work to port TockOS to 32-bit architecture. That's just one example. 00:05:59.190 --> 00:06:03.130 Is that already merged into mainline or what's the status on that? 00:06:04.110 --> 00:06:08.410 I don't know. The PR is there, but I don't know if it's merged yet or not. 00:06:08.570 --> 00:06:10.630 But it's definitely public. 00:06:11.590 --> 00:06:12.670 Wow. Okay. 00:06:12.850 --> 00:06:14.190 I think it should be merged by now. 00:06:14.970 --> 00:06:20.970 Yeah, I don't know how many 32-bit devices are out there, but I would say there 00:06:20.970 --> 00:06:24.490 must be millions, probably billions of these devices. 00:06:24.850 --> 00:06:29.530 Yeah, when you think about these specialized cards or specialized hardware that's 00:06:29.530 --> 00:06:32.810 sort of connected to various other more complex infrastructure, 00:06:33.210 --> 00:06:39.130 there's definitely still a need for 32-bit in some places, which people don't actually expect. 00:06:40.050 --> 00:06:43.930 They like to think just in terms of ARM64 or ARM64. 00:06:44.710 --> 00:06:46.430 Intel 64-bit architecture so. 00:06:46.430 --> 00:06:49.810 It's funny because if if 00:06:49.810 --> 00:06:53.150 i think of big tech like very large organizations 00:06:53.150 --> 00:06:59.710 sometimes i hear that it's easier for them to build their own little you know 00:06:59.710 --> 00:07:05.030 solution or proper solution for example google used that as an excuse to build 00:07:05.030 --> 00:07:08.910 fuchsia which is also written in rust they could also have contributed to an 00:07:08.910 --> 00:07:11.930 existing project but in your case for TockOS, 00:07:12.290 --> 00:07:15.610 you very much contributed to an existing open source project. 00:07:15.790 --> 00:07:19.470 Can you elaborate on that? Like what was the decision making process behind that? 00:07:19.710 --> 00:07:26.610 So I think for many decades, Microsoft has suffered the not invented here thing syndrome. 00:07:26.950 --> 00:07:31.550 And we tended to think we're the smartest and do everything from scratch because 00:07:31.550 --> 00:07:34.970 we can do it better than anyone else. I think we're cured of that. 00:07:35.110 --> 00:07:38.870 So right now we're very much in the process of looking 00:07:38.870 --> 00:07:41.990 for what's best out there and if 00:07:41.990 --> 00:07:47.670 that best in class in open source fits our needs and aligns in terms of roadmap 00:07:47.670 --> 00:07:53.950 and what we want from it with our project needs and our roadmaps and if it if 00:07:53.950 --> 00:07:58.470 we have this alignment then we're gonna bet on on open source very much the 00:07:58.470 --> 00:08:00.970 the microsoft of today is about, 00:08:02.050 --> 00:08:07.710 doing work in the open and leveraging work that is in the open source so both 00:08:07.710 --> 00:08:12.330 consuming good quality libraries and projects and contributing back. 00:08:12.610 --> 00:08:17.510 So if we don't find such a thing, we will start it, of course. 00:08:18.070 --> 00:08:22.270 But in many cases, we find good projects out there that, 00:08:23.120 --> 00:08:28.160 almost align perfectly with what we need, and then what's the point in reinventing the wheel, right? 00:08:28.600 --> 00:08:29.280 Yeah, true. 00:08:29.280 --> 00:08:30.840 When we can improve it for everyone. 00:08:31.120 --> 00:08:35.740 Yeah, absolutely. You said TockOS was used in Surface devices, 00:08:35.960 --> 00:08:38.120 but isn't Surface a 64-bit device? 00:08:38.620 --> 00:08:43.000 No, this is a specialized chip that's for the security processor. 00:08:43.360 --> 00:08:43.780 Ah, okay. 00:08:43.820 --> 00:08:46.960 It's not the main integrated CPU. 00:08:47.340 --> 00:08:52.360 Okay, perfect. Yeah, so we already covered one very... 00:08:53.400 --> 00:08:56.580 Low-level project, one that is close to the hardware. 00:08:56.740 --> 00:09:00.660 Another one was, and I think we did this a while back, is Project Moo, 00:09:01.000 --> 00:09:08.920 where we did a Wifi implementation that's actually used in Surface laptops and even in Azure boxes. 00:09:09.260 --> 00:09:14.400 So there's a custom Wifi implementation in Rust. And this is, 00:09:14.500 --> 00:09:15.740 again, an open-source project. 00:09:15.960 --> 00:09:19.660 People can actually Google for the GitHub repo for it. 00:09:20.940 --> 00:09:22.080 We can probably. 00:09:22.080 --> 00:09:23.480 Link to it in the show notes i will. 00:09:23.480 --> 00:09:26.380 Yeah i can provide you with a bunch of links to these 00:09:26.380 --> 00:09:31.380 things then again if we're thinking about these kinds of things with we have 00:09:31.380 --> 00:09:36.240 the calyptra project which is about harder root of trust foundation for building 00:09:36.240 --> 00:09:42.460 more advanced security capabilities so again this is a fully transparent industry 00:09:42.460 --> 00:09:45.720 collaboration effort where We're sort of contributing to this space. 00:09:46.800 --> 00:09:53.520 And again, Azure integrated HSMs, this is Microsoft's new in-house security chip. 00:09:53.680 --> 00:10:00.900 This is meant for isolating cryptographic keys in dedicated vaults. It's FIPS compliant. 00:10:01.560 --> 00:10:05.780 It's about securing key exchange between Azure VMs and all these things. 00:10:05.920 --> 00:10:14.140 So again, we're using Rust in all these security sensitive areas of our cloud 00:10:14.140 --> 00:10:15.240 infrastructure, right? 00:10:15.340 --> 00:10:18.660 If we're talking about hardware that supports the cloud and devices, 00:10:19.060 --> 00:10:24.480 if we're talking about the hypervisors and the services themselves and sort 00:10:24.480 --> 00:10:28.020 of how we're securing this trusted compute platform in the cloud. 00:10:28.320 --> 00:10:34.600 So all across the board, from firmware and hardware and dedicated cards up the 00:10:34.600 --> 00:10:36.740 software stack at all levels, right? 00:10:37.140 --> 00:10:42.220 If we're talking about Azure Post Agents, where we offload on hardware cards 00:10:42.220 --> 00:10:48.600 for managing Azure VM workloads and manage virtualization for storage and networking in Azure. 00:10:49.560 --> 00:10:57.260 Even Hyper-V, right? Which Hyper-V underpins the whole Azure infrastructure for all the VMs, right? 00:10:57.940 --> 00:11:01.860 If you think about classic Hyper-V has been written, I don't know. 00:11:02.720 --> 00:11:07.340 25 years now. I don't even remember. It was written in C++, right? 00:11:07.520 --> 00:11:09.040 And still the bulk of it is C++. 00:11:09.360 --> 00:11:12.340 But we're rewriting parts of it in Rust right now. 00:11:12.680 --> 00:11:17.340 Either parts that we want to secure, so we want to sort of harden various parts 00:11:17.340 --> 00:11:21.300 of the hypervisor, or we're rewriting new parts of it. 00:11:21.380 --> 00:11:26.720 For example, the instruction emulation for ARM64 in Hyper-V is written completely in Rust. 00:11:26.720 --> 00:11:30.240 So the whole hypervisor virtual stack and 00:11:30.240 --> 00:11:33.260 even more and more components in hype or vr rewritten from 00:11:33.260 --> 00:11:36.080 c++ to rust because we want to sort 00:11:36.080 --> 00:11:39.020 of harden those components and and i think that's a 00:11:39.020 --> 00:11:42.700 sort of a theme across microsoft 00:11:42.700 --> 00:11:45.700 not just in azure but maybe azure 00:11:45.700 --> 00:11:49.000 would be the most aggressive such organization in 00:11:49.000 --> 00:11:52.280 microsoft was doing these rust rewrites but 00:11:52.280 --> 00:11:55.140 in general we're we're on 00:11:55.140 --> 00:11:58.180 the one hand trying to harden c++ because we 00:11:58.180 --> 00:12:01.120 have a lot of c++ at microsoft 00:12:01.120 --> 00:12:04.220 as you can like might imagine right the billions 00:12:04.220 --> 00:12:08.280 of lines of code of c++ and and we need to harden c++ because it's not going 00:12:08.280 --> 00:12:14.560 to go away anytime soon so we're investing a lot on on securing c++ both in 00:12:14.560 --> 00:12:20.860 terms of tooling and how we're methodologies around writing c++ but especially 00:12:20.860 --> 00:12:23.080 in terms of tooling and at the same time. 00:12:23.340 --> 00:12:28.540 We're doing these tactical Rust migrations or Rust rewrites, 00:12:28.700 --> 00:12:34.400 where we're identifying pieces of code or components that have been traditionally 00:12:34.400 --> 00:12:36.800 targeted for vulnerabilities over the years, 00:12:36.920 --> 00:12:41.680 sort of the most sensitive parts or things that have most CVEs or exploits that 00:12:41.680 --> 00:12:45.320 were sort of memory caused by memory exploitations. 00:12:45.760 --> 00:12:48.460 And we're just rewriting them in Rust to 00:12:48.460 --> 00:12:54.240 secure them and to be able to prove their security as in not just oh oh it's 00:12:54.240 --> 00:13:00.080 rust it must be secure right rewriting them in rust gives us ways to reason 00:13:00.080 --> 00:13:06.920 about proving their their safety and and do all sorts of better analysis on it can you. 00:13:06.920 --> 00:13:12.820 Share any internal metrics about how security improved after rewriting parts of it in rust. 00:13:12.820 --> 00:13:15.980 I don't have numbers on hand 00:13:15.980 --> 00:13:19.280 but i've i've seen plenty of presentations coming out 00:13:19.280 --> 00:13:25.580 of we have dedicated security teams in the companies that sort of analyze these 00:13:25.580 --> 00:13:29.620 kinds of patterns across multiple organizations and projects and they regularly 00:13:29.620 --> 00:13:36.140 post statistics so I've seen such presentations internally but I don't have numbers of them but it's. 00:13:37.560 --> 00:13:40.940 People, in general, people tend to think, oh, this is anecdotal. 00:13:40.960 --> 00:13:44.580 As you know, you're rewriting it in Rust and it will just magically be better. 00:13:44.740 --> 00:13:50.140 And they sort of deride this thing. But we have hard data that shows that components 00:13:50.140 --> 00:13:53.360 that have been rewritten are much more solid. 00:13:53.500 --> 00:13:58.180 And we can actually even formally prove some of them to be memory safe, 00:13:58.620 --> 00:14:01.780 both in terms of spatial and temporal safety. 00:14:01.780 --> 00:14:07.000 And I've seen similar studies out of Google on their security blog as well about 00:14:07.000 --> 00:14:12.940 providing statistical information to prove that these rights are improving the 00:14:12.940 --> 00:14:14.140 overall security of systems. 00:14:14.140 --> 00:14:19.500 And more interesting, what I've seen from the Google security study that I read 00:14:19.500 --> 00:14:23.860 last year is that they noticed the incidents, 00:14:24.200 --> 00:14:31.280 the security memory vulnerabilities decreasing just out of writing new components 00:14:31.280 --> 00:14:34.640 in Rust without even needing to touch existing components. 00:14:34.640 --> 00:14:39.380 Because the vast majority of exploits and bugs tend to be in recent code, 00:14:39.740 --> 00:14:44.800 not in code that has been tested and tried and patched for decades, right? 00:14:45.180 --> 00:14:50.440 Most vulnerable code tends to be the most recent code because it has been less 00:14:50.440 --> 00:14:53.700 tested or less battle-hardened, let's say. 00:14:54.500 --> 00:14:56.360 Do you see the same pattern at Microsoft? 00:14:56.680 --> 00:15:02.140 Yeah, we're seeing similar things. We're not sort of madly going across the 00:15:02.140 --> 00:15:06.100 board and just like for the fun of it, Let's rewrite everything in Rust. 00:15:06.880 --> 00:15:11.000 We're doing these sort of tactical choices where we say, okay, 00:15:11.100 --> 00:15:15.520 this new component, it's better if we write it in Rust. 00:15:16.420 --> 00:15:22.300 Or this component, we feel that by rewriting it in Rust, it will give us a chance 00:15:22.300 --> 00:15:25.080 to rethink it, re-architect it, prove... 00:15:28.380 --> 00:15:31.360 Different sort of pre and post conditions about 00:15:31.360 --> 00:15:36.760 it sort of be be more deliberate and more precise about what we can prove about 00:15:36.760 --> 00:15:41.000 this particular piece of code that we're rewriting so it's it's not sort of 00:15:41.000 --> 00:15:45.720 a blanket statement that oh if we're rewriting everything everything will be 00:15:45.720 --> 00:15:49.660 magically better right we're we're doing sort of very tactical choices here. 00:15:49.660 --> 00:15:57.900 Google came forward and I said 70% of their issues with code in general were memory safety issues. 00:15:58.320 --> 00:16:02.920 And I would assume that it's similar across different organizations. 00:16:03.160 --> 00:16:06.480 I might even remember that there was a similar study from Microsoft. 00:16:06.820 --> 00:16:13.340 Yeah, I think I presented the slide on that at one or two of the conferences last year. 00:16:13.340 --> 00:16:16.160 So it definitely we're seeing the 00:16:16.160 --> 00:16:19.980 same thing as in memory safety dominates 00:16:19.980 --> 00:16:22.820 all cvs across the microsoft so it's 00:16:22.820 --> 00:16:27.480 definitely over 70 percent in microsoft the majority 00:16:27.480 --> 00:16:31.260 so the majority are memory safety issues and all across the board from heap 00:16:31.260 --> 00:16:36.260 corruption heap out of bounds step corruptions type confusions uninitialized 00:16:36.260 --> 00:16:42.120 variables use after freeze all sorts of things so definitely we're seeing the 00:16:42.120 --> 00:16:43.180 same kinds of things, right? 00:16:43.480 --> 00:16:49.780 And in terms of mitigations, yes, like I told you, we're constantly developing 00:16:49.780 --> 00:16:54.020 newer and newer technologies for C++ to address this. 00:16:54.260 --> 00:16:58.600 But with Rust, you get most of these things out of the box, right? 00:16:59.120 --> 00:17:01.940 Again, there's challenges even there, 00:17:02.080 --> 00:17:08.240 but definitely in terms of memory vulnerabilities, we can do a lot better. 00:17:08.240 --> 00:17:13.020 Was Azure the first Rust project at Microsoft? 00:17:14.470 --> 00:17:16.210 Was Azure the first to adopt Microsoft? 00:17:16.490 --> 00:17:23.910 I don't know for sure at this point. So very much when Rust started at Microsoft several years ago, 00:17:24.190 --> 00:17:30.610 it was very much a sort of bottom-up thing where various teams or various enthusiastic 00:17:30.610 --> 00:17:34.790 engineers tried out things, tried to rewrite various components, 00:17:35.390 --> 00:17:40.190 for various reasons and claimed some results on them, as in, did it work? 00:17:41.510 --> 00:17:46.270 What were the challenges? what didn't work, and so on. So it was very much sort 00:17:46.270 --> 00:17:48.030 of an experimentation phase. 00:17:48.310 --> 00:17:53.230 And then the following years, based on early successes, there's been a bunch 00:17:53.230 --> 00:17:57.330 of incubation projects across the company in all organizations, not just Azure. 00:17:57.670 --> 00:18:03.650 In Windows, in Office, in Azure, in M365, all across the company, really. 00:18:04.030 --> 00:18:10.470 And I think we did this for a couple of years and sort of had various degrees 00:18:10.470 --> 00:18:13.030 of successes and teams were, 00:18:13.210 --> 00:18:20.510 I think the most impressive thing is that teams were super enthusiastic and 00:18:20.510 --> 00:18:22.910 they felt the experience was, 00:18:23.290 --> 00:18:25.070 in general, a win, right? 00:18:25.150 --> 00:18:29.810 Even if maybe a particular project didn't get the green light to go forward, right? 00:18:29.930 --> 00:18:34.310 If it was just maybe proving something or prototyping something or, 00:18:34.670 --> 00:18:40.970 you know, not all projects get the green light to go ahead, right? For various reasons. 00:18:41.250 --> 00:18:47.050 But even in cases where we just learned from a particular such instance or such 00:18:47.050 --> 00:18:51.790 experience, teams were super enthusiastic and they felt like this could really 00:18:51.790 --> 00:18:54.770 change something at Microsoft, right, in how we do software. 00:18:55.190 --> 00:19:03.170 So after that incubation phase, we're sort of more and more projects started doing Rust seriously. 00:19:03.410 --> 00:19:06.750 We have internal hackathons, right, a couple of times a year. 00:19:06.910 --> 00:19:11.310 And we've long passed the phase where people enthusiastically, 00:19:11.510 --> 00:19:15.790 let's try something or let's incubate this and see how it goes, right? 00:19:16.030 --> 00:19:21.390 Right now, we're in a phase where Rust at Microsoft is taking really seriously. 00:19:21.850 --> 00:19:27.690 And that comes with... So it's both good and scary at the same time, right? 00:19:28.290 --> 00:19:33.390 It's good because that means more teams and developers are empowered to use Rust. 00:19:33.390 --> 00:19:38.810 So it's no longer like sort of covert pirate operation to do something in Rust. 00:19:38.810 --> 00:19:44.970 It's sort of applauded and encouraged when it's appropriate but it also presents 00:19:44.970 --> 00:19:51.610 sort of a very high bar in terms of in terms of expectations right because very 00:19:51.610 --> 00:19:55.450 very much across the company people are feeling that rust is, 00:19:56.590 --> 00:20:01.050 has graduated to be a sort of a premier programming language, right? 00:20:01.250 --> 00:20:06.070 And that means they have the same kinds of expectations from Rust that they 00:20:06.070 --> 00:20:09.870 have from C++ and C Sharp and TypeScript internally, 00:20:10.050 --> 00:20:16.490 as in those used to be traditionally the tier one, the premier languages in the company, right? 00:20:16.650 --> 00:20:23.150 And now as Rust is becoming to be seen as a tier one language at the company, 00:20:23.730 --> 00:20:28.090 all these developers who are getting more and more ambitious in building cool 00:20:28.090 --> 00:20:32.190 stuff with Rust, they sort of have these high expectations as in because they 00:20:32.190 --> 00:20:36.630 come from these worlds, they come from C++, from C Sharp, 00:20:37.450 --> 00:20:42.690 and they expect developer tools to be the same kind of maturity, right? 00:20:43.230 --> 00:20:48.170 So they expect their developer experience, their debugging experience, 00:20:48.390 --> 00:20:52.630 their tooling, their infrastructure to be ready. So they have sort of all these 00:20:52.630 --> 00:20:56.430 expectations in terms of where Rust should be, at least internally. 00:20:57.350 --> 00:21:01.650 I would assume that also very much depends on the project they are working on. 00:21:01.710 --> 00:21:05.590 For example, if I work on the Windows kernel, I have different expectations 00:21:05.590 --> 00:21:10.970 about my tooling than, let's say, if I work with cloud providers or cloud services. 00:21:11.310 --> 00:21:12.130 Definitely, yes. 00:21:12.430 --> 00:21:16.890 Or do you see similar patterns even across low level and high level? 00:21:16.890 --> 00:21:19.430 Does it even matter where people use Rust? 00:21:19.770 --> 00:21:22.890 It has the same deficiencies across the board. 00:21:23.830 --> 00:21:27.150 There are definitely things that are universal. 00:21:27.630 --> 00:21:33.310 I mentioned this. Microsoft is very much a big place and various organizations 00:21:33.310 --> 00:21:40.390 have sort of various internal protocols and specific compliance rules and specific 00:21:40.390 --> 00:21:42.170 infrastructure needs and so on. 00:21:42.170 --> 00:21:51.830 So while there are many commonalities, there's also very specific needs based on project, right? 00:21:51.830 --> 00:21:59.730 And I would say if the Rust likes or sort of, 00:21:59.830 --> 00:22:07.010 let's say, enthusiasm for Rust is maybe the universal thing across Microsoft organizations, 00:22:07.250 --> 00:22:13.170 like everyone loves fast dev compile time iterations, as in if it compiles, 00:22:13.410 --> 00:22:15.770 it most likely works to that kind of feel. 00:22:15.770 --> 00:22:21.110 Or if it's easier to write tests, people will write more tests, right? 00:22:21.350 --> 00:22:23.790 Or be more encouraged to write a lot of tests. 00:22:24.590 --> 00:22:31.110 And the richness of the ecosystems and libraries out there, memory safety guarantees 00:22:31.110 --> 00:22:35.510 that we talked about earlier, and even data race-related concurrency bugs that 00:22:35.510 --> 00:22:36.470 you can reduce with this. 00:22:36.810 --> 00:22:40.990 So sort of the things they love seem to be sort of universal. Yeah. 00:22:42.050 --> 00:22:47.050 And performance, right? Let's not forget that many of the people who are coming 00:22:47.050 --> 00:22:52.810 across in the company are coming from garbage-collected programming languages like C-sharp, right? 00:22:53.170 --> 00:22:59.490 So they like the native systems programming language feel and deterministic 00:22:59.490 --> 00:23:02.830 distraction and snappiness of Rust CodeGen. 00:23:03.210 --> 00:23:09.930 Although the likes are almost universal across the company, the dislikes are 00:23:09.930 --> 00:23:15.910 very much specific on whichever team, what problem they face most, right? 00:23:16.030 --> 00:23:17.350 Or what they struggle with. For 00:23:17.350 --> 00:23:24.010 example, in terms of infrastructure and compliance and security promises. 00:23:25.630 --> 00:23:31.810 There's sort of a common bar across the company, but some organizations have 00:23:31.810 --> 00:23:36.570 more stringent rules in terms of what they need to prove about the compliance 00:23:36.570 --> 00:23:38.430 and security of software issue, right? 00:23:38.430 --> 00:23:43.610 So this is where engineering systems need to work to fill in the gaps, 00:23:43.630 --> 00:23:51.010 as in we've worked for decades to implement all sorts of workflows and engineering 00:23:51.010 --> 00:23:57.650 systems to deal with C++ codebases and C Sharp codebases or TypeScript codebases. 00:23:57.650 --> 00:24:02.510 And we built all sorts of tools from static analysis to checkers to binary tools 00:24:02.510 --> 00:24:08.010 that check various promises and guarantees about the produced binary artifacts 00:24:08.010 --> 00:24:11.110 or binary hardening tools or all sorts of, 00:24:11.230 --> 00:24:14.410 we have a plethora of tools internal that we use online. 00:24:14.690 --> 00:24:19.650 In gen engineering systems. And as you can imagine, a lot of these are sort 00:24:19.650 --> 00:24:20.970 of not there for Rust, right? 00:24:21.110 --> 00:24:27.950 They either don't exist altogether or they just fall flat on their face with 00:24:27.950 --> 00:24:29.570 Rust-generated code, right? 00:24:30.770 --> 00:24:35.870 So we sort of need to fill in these gaps or improve existing tools if there are any, right? 00:24:36.070 --> 00:24:38.970 For example, let's see, static analysis, right? 00:24:39.210 --> 00:24:43.750 But there's also lots of tools at binary level. 00:24:43.750 --> 00:24:48.290 So do the different teams even share that feedback with one another? 00:24:48.650 --> 00:24:51.790 Say, for example, someone at Azure finds a great tool. 00:24:52.310 --> 00:24:56.670 How do you share that internally? You probably have some sort of Microsoft Teams 00:24:56.670 --> 00:25:00.390 channel or multiple even about tooling for Rust. 00:25:00.570 --> 00:25:03.490 So there's two things that run back. 00:25:03.650 --> 00:25:07.210 So yeah, definitely we have a thriving Rust-Aceans community inside. 00:25:07.210 --> 00:25:13.150 So we actually have internal forums and Teams channels for Rustasians across 00:25:13.150 --> 00:25:17.230 the company where we share sort of learnings from each other. We ask questions. 00:25:17.490 --> 00:25:23.610 We sort of discover things we share. So that's sort of a community sort of things. 00:25:23.770 --> 00:25:28.690 We have many, many, many Rust developers internally who are discussing all sorts 00:25:28.690 --> 00:25:32.230 of interesting stuff there from sharing pieces of code or asking for help or 00:25:32.230 --> 00:25:34.290 sharing a tool or asking about something. 00:25:34.450 --> 00:25:36.570 Just roughly how many people are in these channels? 00:25:37.230 --> 00:25:40.990 It's it's it's in the thousands i think i haven't 00:25:40.990 --> 00:25:43.750 looked in a while but it's a lot of people it's like 00:25:43.750 --> 00:25:47.210 always generates a lot of noise 00:25:47.210 --> 00:25:51.830 in those channels a lot of discussions every day but going back to your original 00:25:51.830 --> 00:25:57.570 question so that's sort of for for sharing and guidance and asking and discovery 00:25:57.570 --> 00:26:01.110 so there's sort of these internal communities are amazing right and i've learned 00:26:01.110 --> 00:26:04.270 so much from just being in those threads. 00:26:05.410 --> 00:26:09.010 But going back to your original question in terms of tooling and engineering 00:26:09.010 --> 00:26:11.190 systems, that's a totally different story. 00:26:11.350 --> 00:26:13.570 That's not where something happens randomly. 00:26:13.830 --> 00:26:18.790 So we have dedicated engineering systems teams inside the company, 00:26:18.790 --> 00:26:25.670 and we have policies across the board where a team cannot just build something 00:26:25.670 --> 00:26:27.710 the way they like to build it, right? 00:26:27.710 --> 00:26:30.790 There's there's policies and 00:26:30.790 --> 00:26:33.510 and specific rules about how you 00:26:33.510 --> 00:26:36.730 set up infrastructure how you set up pipelines how 00:26:36.730 --> 00:26:39.750 what kind of tools and tasks you run and 00:26:39.750 --> 00:26:44.550 how you do all sorts of processing and audits on on on the build systems so 00:26:44.550 --> 00:26:51.370 there's there's a big branch of engineering systems in the company who's responsible 00:26:51.370 --> 00:26:55.210 in supporting all the programming languages that are used in all projects across 00:26:55.210 --> 00:26:57.850 the company regardless of the organization right so. 00:26:57.850 --> 00:27:04.330 Can you share a few of the best practices that might be helpful for people outside 00:27:04.330 --> 00:27:08.690 of microsoft maybe general things that you found out about how to set up a project in rust. 00:27:09.710 --> 00:27:15.130 Oh, there's a, that's a, I think we need more than an hour just for that. 00:27:15.410 --> 00:27:20.730 So there's, hopefully, what we are doing internally is that we're setting what 00:27:20.730 --> 00:27:24.790 we call paved paths for various programming languages, right? 00:27:24.930 --> 00:27:28.790 There's paved path for C++ projects, paved path for C Sharp, 00:27:28.950 --> 00:27:31.110 paved path for Rust. We're building one right now, right? 00:27:31.550 --> 00:27:36.270 Where we're setting up workflows and templates so that people don't have to 00:27:36.270 --> 00:27:39.890 do these kinds of setups manually for each project, right? 00:27:40.250 --> 00:27:43.830 Because you don't want to end up with snowflakes across the company that are 00:27:43.830 --> 00:27:48.410 slightly configured slightly differently, or they're using slightly different things, right? 00:27:48.650 --> 00:27:51.730 So we want very much to standardize best practices. 00:27:52.270 --> 00:27:54.990 So we have pipeline templates, we have 00:27:54.990 --> 00:27:59.030 project templates, we have sort of infrastructure tasks that are shared. 00:27:59.170 --> 00:28:02.130 So it's not a manual bring up process. 00:28:02.130 --> 00:28:08.430 And that's why sort of it's such a high bar to make Rust a sort of a tier one 00:28:08.430 --> 00:28:12.330 supported language in the company because there's so many of these things that 00:28:12.330 --> 00:28:15.910 we need to make sure they're right for prime time and they satisfy the needs 00:28:15.910 --> 00:28:17.050 for all the projects, right? 00:28:17.190 --> 00:28:20.290 Because this is very much a unified rollout, right? 00:28:20.710 --> 00:28:25.550 Regardless if we're talking about secret scanning or static analysis or. 00:28:26.430 --> 00:28:31.150 Any kind of SDL requirement in terms of security analysis or any kind of binary 00:28:31.150 --> 00:28:32.930 checks that we're doing on software and so on. 00:28:33.450 --> 00:28:40.670 So there's lots of such tools and policies, and they all need to be uniformly applied. 00:28:40.670 --> 00:28:44.730 As in, when a new Rust project becomes online in the company, 00:28:45.130 --> 00:28:50.310 it needs to have this proper setup, these pipelines, these templates in place 00:28:50.310 --> 00:28:52.410 in terms of engineering systems and infrastructure. 00:28:53.110 --> 00:28:58.390 It's one thing to start something sort of as a hackathon project and sort of 00:28:58.390 --> 00:29:02.890 build something and share it with a few other colleagues in a team, right? 00:29:03.090 --> 00:29:06.970 As proof of concept, when you build it, that's sort of a different story. 00:29:07.270 --> 00:29:12.670 But when you sort of promote it to this is now a project that is proper project 00:29:12.670 --> 00:29:18.050 in the company, then it needs to be sort of integrated in this engineering system. 00:29:19.050 --> 00:29:24.330 I can see how this can work for a microservice environment where, 00:29:24.790 --> 00:29:27.510 for example, you build a new Azure component or so. 00:29:27.790 --> 00:29:33.750 You have a bootstrapping system for that. But I wonder how it works on the lower level layers. 00:29:34.190 --> 00:29:38.870 For example, in one of your talks, you mentioned that you chose direct write 00:29:38.870 --> 00:29:41.490 as one of your first Rust experiments. 00:29:41.490 --> 00:29:46.710 Experiments and direct write is like a text layout and rendering component that's 00:29:46.710 --> 00:29:50.870 that's used across windows and office and you mentioned that it has a com-like 00:29:50.870 --> 00:29:55.810 interface so it has this component object model interface and i wonder if you 00:29:55.810 --> 00:30:00.690 can even have some sort of bootstrapping pipeline for this when when you start such a project yeah. 00:30:00.690 --> 00:30:05.010 That that's actually a very good segue, I wanted to mention this, 00:30:05.290 --> 00:30:13.570 not all of these things happen as greenfield projects or some perfectly isolated 00:30:13.570 --> 00:30:17.990 thing like, oh, I'm going to extract this new functionality or this new component. 00:30:18.310 --> 00:30:23.310 And it's so nice and pristine and I can, it's self-contained and it has everything 00:30:23.310 --> 00:30:24.690 you need and I can test it. 00:30:26.460 --> 00:30:31.320 It's rarely the case that something is so nicely cut away from everything else, right? 00:30:31.520 --> 00:30:38.240