WEBVTT 00:00:01.530 --> 00:00:05.810 It's time for another episode of Rust in Production, a podcast about companies 00:00:05.810 --> 00:00:08.350 who use Rust to shape the future of infrastructure. 00:00:08.790 --> 00:00:13.370 I'm Matthias Endler from corrode and today's guest is Charlie Marsh from Astral. 00:00:13.590 --> 00:00:16.970 We talk about improving the Python ecosystem with Rust. 00:00:18.970 --> 00:00:23.670 Charlie, thanks for being a guest. Can you say a few words about yourself and 00:00:23.670 --> 00:00:25.850 about Astral, the company you work for? 00:00:26.570 --> 00:00:28.690 Yeah, of course. Thanks so much for having me. first of all. 00:00:28.810 --> 00:00:29.690 My name is Charlie Marsh. 00:00:30.130 --> 00:00:36.790 I run a company called Astral. We build high-performance developer tools for the Python ecosystem. 00:00:36.790 --> 00:00:44.050 So we're best known for two tools, RUF, which is a sort of combined linter formatter 00:00:44.050 --> 00:00:45.490 code transformation tool. 00:00:45.630 --> 00:00:48.670 You could think of it like kind of Rust format and Clippy together, 00:00:49.030 --> 00:00:51.370 which is again for Python code, but written in Rust. 00:00:51.610 --> 00:00:56.590 And then UV, which is our project manager, Python package manager, 00:00:56.770 --> 00:00:59.690 Python tool chain manager. It's, it's, it's, you could think of it a little 00:00:59.690 --> 00:01:01.010 bit like cargo and Rust up. 00:01:01.130 --> 00:01:05.790 So it tries to bootstrap or will bootstrap Python for you and then help you 00:01:05.790 --> 00:01:10.650 manage your dependencies, install things, lock them into lock files and reproducible 00:01:10.650 --> 00:01:12.390 versions and all that kind of stuff. 00:01:12.810 --> 00:01:18.130 So yeah, everything we've built so far is open source and written in Rust. 00:01:18.350 --> 00:01:26.530 We're a team of about 15 people spread from the Pacific, like Pacific time in the U.S. 00:01:26.530 --> 00:01:29.690 We have people in pacific time central time eastern time 00:01:29.690 --> 00:01:35.110 and we have one person in the uk we have four people in cet or like germany 00:01:35.110 --> 00:01:39.150 switzerland the netherlands and then one person in india so not only are we 00:01:39.150 --> 00:01:43.390 like remote we're like very distributed kind of like a lot of open source and 00:01:43.390 --> 00:01:47.150 we spend all our time basically writing rust to try and make python better i. 00:01:47.150 --> 00:01:56.630 Heard about astral the first time when you published ruff but it took off with uv i would say like. 00:01:56.630 --> 00:01:57.190 There was. 00:01:57.190 --> 00:02:04.030 Definitely a huge sympathy for what you do in the packaging space as well did 00:02:04.030 --> 00:02:08.330 you see that as well from a company perspective that there was a huge growth. 00:02:08.920 --> 00:02:12.400 I think so. Yeah. I mean, I think, you know, we started with Ruff. 00:02:12.640 --> 00:02:15.400 I started working on Ruff before the company existed. 00:02:16.040 --> 00:02:18.940 And it was just something I was building. 00:02:19.620 --> 00:02:24.040 Well, for a lot of reasons, but largely because I saw these were problems that 00:02:24.040 --> 00:02:25.700 I had experienced in my own projects. 00:02:25.860 --> 00:02:30.080 And I was like, what would it be like if I wrote, you know, Python tooling and Rust instead? 00:02:30.280 --> 00:02:33.200 And that was sort of the genesis of Ruff. and ruff 00:02:33.200 --> 00:02:36.460 grew extremely quickly like we had you know 00:02:36.460 --> 00:02:39.940 decades old projects like scipy and 00:02:39.940 --> 00:02:44.180 stuff like adopting it while it was still like what i would consider to be very 00:02:44.180 --> 00:02:49.280 unstable so it was clear that people wanted you know something in this in this 00:02:49.280 --> 00:02:55.840 arena of faster python tooling so ruff grew very fast and you know we were 00:02:55.840 --> 00:02:58.480 we were seeing what was happening with ruff and we were like well. 00:02:59.080 --> 00:03:01.900 As a company and in terms of like the problems we're trying to 00:03:01.900 --> 00:03:04.680 solve like we're not just we're not trying to 00:03:04.680 --> 00:03:07.980 build just that like we want to build a python 00:03:07.980 --> 00:03:11.020 tool chain effectively like we want to solve the hard 00:03:11.020 --> 00:03:14.120 problems in python and for me like packaging was 00:03:14.120 --> 00:03:17.280 kind of like if you want to be like the python tooling 00:03:17.280 --> 00:03:20.000 company which aspirational i guess we'd like 00:03:20.000 --> 00:03:22.840 to be then like you have to work on packaging i think 00:03:22.840 --> 00:03:25.740 because it's it's the thing that everyone has 00:03:25.740 --> 00:03:28.840 trouble with and the thing that has been so many people have tried to 00:03:28.840 --> 00:03:31.560 solve and and done good work on but but i 00:03:31.560 --> 00:03:34.440 don't think anyone would have would consider it solved and so 00:03:34.440 --> 00:03:38.920 you know i i kind of knew from early on that we wanted to do something in packaging 00:03:38.920 --> 00:03:44.920 and before we released it i definitely felt nervous in that ruff had grown 00:03:44.920 --> 00:03:48.580 well and people really liked it and it was it was kind of felt like it was going 00:03:48.580 --> 00:03:52.400 to be a tough act to follow like we don't want to be a one-hit wonder you know 00:03:52.400 --> 00:03:53.600 with the tools that we built. 00:03:54.060 --> 00:03:58.080 Like I wanted to build something that hopefully would be as exciting and grow 00:03:58.080 --> 00:04:00.960 as well and have as much of an impact as Ruff. 00:04:01.080 --> 00:04:03.920 And I think UV has actually really surpassed that in a lot of ways. 00:04:04.060 --> 00:04:09.300 It's like, I think the impact it's had on Python has probably been more significant. 00:04:09.580 --> 00:04:13.000 And I think like the, not that I don't love Ruff. 00:04:13.120 --> 00:04:18.540 I mean, it was my first big project and we still like half the people on the 00:04:18.540 --> 00:04:21.440 team are still working, are working on Ruff and the static analysis tooling. 00:04:21.440 --> 00:04:25.280 It's like a big focus area for us but i think i've been really amazed by how 00:04:25.280 --> 00:04:30.920 quickly uv has had the impact that it's had like we released it in february 00:04:30.920 --> 00:04:36.820 of last year so it's just it just is one turned one year old and in that time it's grown to like. 00:04:37.940 --> 00:04:43.380 I think last I checked, it's like a little over 12.5% of all requests to the 00:04:43.380 --> 00:04:50.500 Python index come from UV, which is like over 200 million requests a day, which is wild, right? 00:04:50.640 --> 00:04:55.780 And I spend a lot of my time talking to companies. There are tons of like 1 00:04:55.780 --> 00:04:59.260 billion, 10 billion, even $100 billion companies that are using this thing in 00:04:59.260 --> 00:05:01.060 production and have been for a long time. 00:05:01.220 --> 00:05:04.380 So just like the way it grew i think really surpassed 00:05:04.380 --> 00:05:07.240 my my expectations and i feel 00:05:07.240 --> 00:05:12.160 like we now have a a platform to like keep hopefully making python better but 00:05:12.160 --> 00:05:15.680 but i think you're right that like uv i think we shifted into a little bit of 00:05:15.680 --> 00:05:19.880 another gear we spread beyond just being like the company was just ruff before 00:05:19.880 --> 00:05:27.120 and now we are like python tooling and we try to solve like a much wider surface area of problems now. 00:05:27.120 --> 00:05:31.640 You said that ruff was your first major rust project 00:05:31.760 --> 00:05:34.800 maybe the first actual project that you tried after 00:05:34.800 --> 00:05:37.960 like learning rust i'm not sure about this but it certainly 00:05:37.960 --> 00:05:41.520 was big milestone for you in your rust journey but then 00:05:41.520 --> 00:05:49.560 you compared it with uv and did you already learn a few things that you wanted 00:05:49.560 --> 00:05:55.540 to avoid with uv that you did with ruff and also did you set up the uv code 00:05:55.540 --> 00:05:58.060 base early on for such growth. 00:05:58.060 --> 00:06:04.580 Yeah so So I would say I really learned Rust in the process of writing Rust. 00:06:05.420 --> 00:06:10.660 I had done some Rust at my previous job, but someone else on the team introduced 00:06:10.660 --> 00:06:12.240 Rust, really great engineer. 00:06:13.480 --> 00:06:16.800 And when I was contributing to that code base, I was mostly trying to get in 00:06:16.800 --> 00:06:19.820 and out as quickly as possible because I didn't really know Rust. 00:06:19.840 --> 00:06:22.480 I was like, I need to fix a bug. How do I get this to compile? 00:06:22.680 --> 00:06:26.640 I wasn't really taking the time to learn it. I was just using it when I had 00:06:26.640 --> 00:06:28.480 to. So I had some Rust exposure. 00:06:29.540 --> 00:06:33.060 But part of why I worked on ruff in the first place was I felt like I wanted 00:06:33.060 --> 00:06:35.820 to really learn rust. And I thought that in order to do that, 00:06:35.940 --> 00:06:38.800 I had to build something from scratch. And I had to like, it's just kind of 00:06:38.800 --> 00:06:41.280 the way that I like to learn. I was like, I need to build something. 00:06:41.480 --> 00:06:46.380 And like, even if that requires like, wasting a ton of time trying to understand 00:06:46.380 --> 00:06:50.060 like lifetimes, like maybe I'll burn like two days trying to get this to compile. 00:06:50.220 --> 00:06:53.240 But like, that's kind of how I learn is just like, I have to build things and 00:06:53.240 --> 00:06:56.120 like fight through the problems. And I wanted to learn the ecosystem and just 00:06:56.120 --> 00:06:57.100 the tooling and everything. 00:06:57.800 --> 00:07:02.200 So I did really learn Rust in the process of writing ruff. 00:07:02.420 --> 00:07:06.920 I think that showed, like there was probably still shows a little bit today, 00:07:07.180 --> 00:07:09.680 like, you know, especially early on, there were a lot of things I was doing 00:07:09.680 --> 00:07:14.420 that were just that I would now, I mean, this is, this is part of personal growth, right? 00:07:14.500 --> 00:07:17.160 There's lots of things I was doing that I would now look at and laugh about 00:07:17.160 --> 00:07:19.300 and say, that's obviously not the right way to do it. 00:07:19.800 --> 00:07:23.960 Whether it was like silly performance things or the way I was structuring the 00:07:23.960 --> 00:07:26.780 code or or I don't know, anything. 00:07:27.600 --> 00:07:32.600 By the time that we started working on UV, I think I felt a lot more comfortable in Rust. 00:07:33.380 --> 00:07:37.100 And I'd also learned more from looking at other projects too, 00:07:37.380 --> 00:07:40.220 I think. And we had kind of evolved Ruff a little bit over time. 00:07:40.460 --> 00:07:46.420 So for example, like we have a really, a fairly wide like crate structure. 00:07:46.720 --> 00:07:48.580 Like we just create a lot of crates. 00:07:48.900 --> 00:07:52.640 Like if you go and open up Ruff or UV, the crate subdirectory, 00:07:52.800 --> 00:07:54.360 there's like in UV, there's probably. 00:07:55.460 --> 00:08:00.020 There's at least 20 crates, maybe like 30. So we just create like lots of crates. 00:08:00.460 --> 00:08:04.900 And that was something that I started to do in ruff because our, 00:08:05.040 --> 00:08:07.140 our build and compile times got really bad. 00:08:07.160 --> 00:08:11.360 And it became like, if I just put everything in one crate, it was like our build 00:08:11.360 --> 00:08:14.720 and compile times got worse and worse. And the development loop was painful. 00:08:15.480 --> 00:08:18.300 And, you know, at the time I was reading a 00:08:18.300 --> 00:08:21.000 lot about how sort of like crates are like the atomic unit for like 00:08:21.000 --> 00:08:23.700 Rust-C and like you can parallelize across crates and you need 00:08:23.700 --> 00:08:27.180 to be thinking about like how your crate graph looks so 00:08:27.180 --> 00:08:30.080 that there's a lot of like fan out and stuff and i was like okay we're going 00:08:30.080 --> 00:08:33.480 to start creating like lots more smaller crates and we're going to sort of change 00:08:33.480 --> 00:08:38.220 the atomic unit of like what the code base is from like the module to the crate 00:08:38.220 --> 00:08:42.440 and so we started carving out like lots of little crates and we did basically 00:08:42.440 --> 00:08:47.060 the same thing and for example in ruff like you know. 00:08:48.000 --> 00:08:50.940 The core linter crate like doesn't 00:08:50.940 --> 00:08:53.700 depend on clap for example like it doesn't have it doesn't depend on 00:08:53.700 --> 00:08:56.500 any of the cli stuff the cli stuff is in a separate crate 00:08:56.500 --> 00:08:59.260 that depends on the linter crate and you just try and get this structure in this 00:08:59.260 --> 00:09:02.260 organization or like the parser and the ast those are 00:09:02.260 --> 00:09:05.000 all their own crates so we can if you just need to test the 00:09:05.000 --> 00:09:07.700 parser right it's really fast to compile and build or if people 00:09:07.700 --> 00:09:10.760 if other people even just need to pull in the parser as a library which 00:09:10.760 --> 00:09:13.420 some people do it's much easier so by the time we got 00:09:13.420 --> 00:09:16.240 to uv i think we had ironed out a 00:09:16.240 --> 00:09:19.060 lot of things that made it so we put we just 00:09:19.060 --> 00:09:22.320 put a lot of stuff in place from the start like we just had a better separation 00:09:22.320 --> 00:09:25.520 in terms of what went into what crates i think 00:09:25.520 --> 00:09:31.100 we'd ironed out a lot of our workflows for example like what's our clippy configuration 00:09:31.100 --> 00:09:36.500 how do we do how do we pin the rust toolchain version like how do we install 00:09:36.500 --> 00:09:39.780 rust in ci like there were just all these little things that we were able to 00:09:39.780 --> 00:09:43.180 just copy over from ruff and i think that made things a lot easier. 00:09:44.050 --> 00:09:48.430 At the same time, there were still a lot of design decisions in UV architecturally 00:09:48.430 --> 00:09:52.230 that were pretty different because the design of a package manager and the design 00:09:52.230 --> 00:09:56.570 of the linter are just the things they need to do are very different. 00:09:57.230 --> 00:10:01.430 So, for example, UV has a whole networking stack, right? 00:10:01.550 --> 00:10:06.410 It needs to make lots of HTTP requests and the linter doesn't need to do any of that. 00:10:06.550 --> 00:10:13.350 So suddenly we had to think about requests or basically the whole HTTP stack. 00:10:13.350 --> 00:10:18.150 We had to think about like open ssl we had to think about all these system dependencies 00:10:18.150 --> 00:10:21.930 like git that was another two like we support git dependencies so suddenly you 00:10:21.930 --> 00:10:25.070 have to think about how you depend on git i think there was just a lot of complexity 00:10:25.070 --> 00:10:27.950 that came with building a package manager that didn't we didn't necessarily 00:10:27.950 --> 00:10:29.890 have to encounter when we were working on the linter, 00:10:30.790 --> 00:10:34.350 which is good because i kind of like encountered those problems over time when. 00:10:34.350 --> 00:10:41.370 I understand you correctly to summarize you didn't have a rough start with uv because. 00:10:41.370 --> 00:10:42.750 You could copy over. 00:10:42.750 --> 00:10:48.870 A lot of the let's say templates or a lot of the best practices from the previous project. 00:10:48.870 --> 00:10:52.430 At least what we thought were best practices yeah and i guess to some degree 00:10:52.430 --> 00:10:55.390 we still think our best practices but when. 00:10:55.390 --> 00:11:03.010 You used uv when you started with uv did you already compartmentalize some of 00:11:03.010 --> 00:11:08.790 the functionality into smaller crates somewhat subconsciously did you say oh 00:11:08.790 --> 00:11:15.070 yeah this definitely goes into a separate crate now or did you still start with a single large crate. 00:11:15.070 --> 00:11:19.270 No i think we were like compartmentalizing stuff basically from the start and 00:11:19.270 --> 00:11:24.530 for example like you know we had one crate for like parsing python versions 00:11:24.530 --> 00:11:27.210 there's a whole spec around versioning that requires, 00:11:27.750 --> 00:11:32.730 a lot of there's a lot of intricacies to it so that's its own crate we had one crate for like, 00:11:33.330 --> 00:11:36.190 parsing requirements txt files that's another thing that's common in 00:11:36.190 --> 00:11:39.590 python that's there's actually no spec for that that's like implementation defined 00:11:39.590 --> 00:11:42.270 and so we have like one crate for that you know we have a crate for 00:11:42.270 --> 00:11:45.330 creating virtual environments we had a crate for the cli we 00:11:45.330 --> 00:11:48.930 had a crate for the resolver so yeah we broke things down i think pretty early 00:11:48.930 --> 00:11:55.010 on and that i mean that's generally served us very well so i'm a big fan of 00:11:55.010 --> 00:11:58.710 that structure i think it's like much the trade-off is really different when 00:11:58.710 --> 00:12:03.330 you're publishing a library like we don't publish any of our we don't, 00:12:03.870 --> 00:12:06.890 we don't publish any of anything in rougher uv as 00:12:06.890 --> 00:12:09.750 a library like we don't publish to crates io any of 00:12:09.750 --> 00:12:13.290 the stuff that we build right now like the public api for our stuff tends to 00:12:13.290 --> 00:12:17.930 be the command line so the cli the command line interface so like if we had 00:12:17.930 --> 00:12:21.710 to publish things i think we would have to think a lot more carefully well there's 00:12:21.710 --> 00:12:25.010 all sorts of considerations that come with publishing that you don't have to 00:12:25.010 --> 00:12:27.650 think about if you aren't publishing. 00:12:27.990 --> 00:12:32.350 But one is, if we had a really granular create structure, I think it becomes 00:12:32.350 --> 00:12:35.770 a bit harder for people to use your... There's a lot more of a maintenance burden, 00:12:35.810 --> 00:12:38.510 and it's also harder for people to use and compose your things. 00:12:39.910 --> 00:12:41.330 But you don't go and... 00:12:42.890 --> 00:12:48.330 Introduce a crate for types specifically because that's one thing that i see some, 00:12:49.550 --> 00:12:52.410 some people do some companies do that they have 00:12:52.410 --> 00:12:55.630 a types crate where they put everything that is related to 00:12:55.630 --> 00:12:58.510 you know their basic types into one thing 00:12:58.510 --> 00:13:02.150 and then at the end they don't 00:13:02.150 --> 00:13:07.070 really get a lot of benefits from using a workspace because you need that types 00:13:07.070 --> 00:13:13.390 crate anyway in all of your other crates and so it kind of goes against to what 00:13:13.390 --> 00:13:18.430 a workspace is about in my opinion but i want to hear it from you like what 00:13:18.430 --> 00:13:21.490 are some anti-patterns for building workspaces. 00:13:21.490 --> 00:13:24.350 Yeah so like i guess 00:13:24.350 --> 00:13:28.210 yeah we try not to do things like like 00:13:28.210 --> 00:13:30.970 that i don't know what we would call that pattern but like 00:13:30.970 --> 00:13:34.210 create a crate that's just kind of like used everywhere there's 00:13:34.210 --> 00:13:37.250 maybe some of that but we try to generally avoid it 00:13:37.250 --> 00:13:40.970 i think the other anti-patterns are like you can 00:13:40.970 --> 00:13:44.770 make your crate structure like too granular like if 00:13:44.770 --> 00:13:47.830 you find yourself creating a crate for that's like 00:13:47.830 --> 00:13:51.550 one function and sometimes 00:13:51.550 --> 00:13:54.750 you'll find yourself doing that because you have like circular dependencies 00:13:54.750 --> 00:13:57.510 or something in the dependency chain that's requiring you to do this 00:13:57.510 --> 00:14:01.290 like i have two crates that need this functionality and they can't depend on 00:14:01.290 --> 00:14:04.250 one can't depend on the other or something and then you find yourself putting 00:14:04.250 --> 00:14:08.030 a function into like like it's just a crate that's like a function at that point 00:14:08.030 --> 00:14:11.610 you've probably gone too granular and you kind of need to rethink like the organization 00:14:11.610 --> 00:14:14.810 because there is some overhead to having all these crates and like, 00:14:15.330 --> 00:14:18.770 you know it's not always like super fun to maintain i think 00:14:18.770 --> 00:14:21.670 a couple things that we do though that are kind of nice for this 00:14:21.670 --> 00:14:24.870 is like we prefix all the crates 00:14:24.870 --> 00:14:31.310 in the workspace with uv or with ruff so like it's easy to differentiate what's 00:14:31.310 --> 00:14:35.110 us create in the workspace versus what's a third party crate like all of our 00:14:35.110 --> 00:14:40.650 crates are like uv virtual env or like uv resolver or like whatever else that's 00:14:40.650 --> 00:14:42.910 pretty nice and then the other thing is we. 00:14:44.080 --> 00:14:51.760 We declare them all in the workspace root so that every crate that depends on 00:14:51.760 --> 00:14:54.640 other crates in the workspace can just use workspace equals true. 00:14:54.880 --> 00:14:59.460 This is a little bit nuanced and maybe hard to visualize, but it basically means 00:14:59.460 --> 00:15:03.380 that in the cargo.toml for all the crates that are in the workspace, 00:15:03.400 --> 00:15:05.780 it's very obvious what else is in the workspace. 00:15:07.040 --> 00:15:10.480 Another thing that we do is we actually try 00:15:10.480 --> 00:15:13.980 to use workspace equals true for all basically 00:15:13.980 --> 00:15:17.900 all dependencies in the workspace so we 00:15:17.900 --> 00:15:21.240 put everything that we depend on in the root cargo toml 00:15:21.240 --> 00:15:24.080 like the workspace root and then we use workspace equals true 00:15:24.080 --> 00:15:27.020 everywhere and that tends to simplify 00:15:27.020 --> 00:15:30.440 things a lot it basically means we have like one dependency specifier for 00:15:30.440 --> 00:15:34.160 request for example and a 00:15:34.160 --> 00:15:37.220 common set of like default extras or no default extras or 00:15:37.220 --> 00:15:40.220 sorry not extras features extras is the python version is 00:15:40.220 --> 00:15:43.300 the python analog to to features but like no default features 00:15:43.300 --> 00:15:46.260 or you know or whatever else we have like one dependency declaration for 00:15:46.260 --> 00:15:49.800 everything that we need and then all the workspace members their cargo terminals 00:15:49.800 --> 00:15:52.820 are just very straightforward it's like the things they require with workspace 00:15:52.820 --> 00:15:56.880 equals true so we don't have to think about oh do we have request dependency 00:15:56.880 --> 00:16:00.460 specifiers across like 10 or 15 different crates we just have one definition 00:16:00.460 --> 00:16:04.480 for it so like that's also i think been a really handy thing right. 00:16:04.480 --> 00:16:09.980 Workspaces seem to be one of these things that i keep missing in other ecosystems 00:16:09.980 --> 00:16:16.160 and i can't remember if there's even another language that has such a feature from the top. 00:16:16.160 --> 00:16:19.000 Of my head yeah think about one i mean we've thought about 00:16:19.000 --> 00:16:21.860 it a fair amount in python so like uv 00:16:21.860 --> 00:16:24.760 has a has workspace functionality that's like very 00:16:24.760 --> 00:16:27.940 much modeled after cargo um so 00:16:27.940 --> 00:16:30.920 you have a root it defines the members 00:16:30.920 --> 00:16:34.500 we use very similar actually like members and excludes syntax 00:16:34.500 --> 00:16:37.620 and there's nice things like that you know 00:16:37.620 --> 00:16:40.360 we you can do like uv run dash p just like you can do 00:16:40.360 --> 00:16:43.120 cargo run right so like we've we've just like 00:16:43.120 --> 00:16:46.100 copied a lot of things from cargo because it's great 00:16:46.100 --> 00:16:48.980 and we're like we want to have those things in python 00:16:48.980 --> 00:16:51.780 and for workspaces i think that's 00:16:51.780 --> 00:16:55.140 worked really well it's been very interesting to introduce them because for 00:16:55.140 --> 00:16:57.980 a lot of people it's fairly new concept like when they come to python right 00:16:57.980 --> 00:17:00.680 like like most people who use uv i've never used Rust right they've never 00:17:00.680 --> 00:17:03.500 heard of a cargo workspace before and like why should they 00:17:03.500 --> 00:17:06.920 you know if they're just working in python so it's been interesting to try and 00:17:06.920 --> 00:17:12.420 like i communicate that and like help people understand like what it's for and 00:17:12.420 --> 00:17:16.140 like why you might use it and like what what an example workspace might look 00:17:16.140 --> 00:17:19.880 like in practice there are some things that we miss that are hard to get a little 00:17:19.880 --> 00:17:22.160 bit hard to get like without standards like for example. 00:17:23.190 --> 00:17:26.750 We don't support, I just spent all this time talking about this incredibly boring 00:17:26.750 --> 00:17:29.690 workspace equals true thing that all your listeners are probably like, 00:17:29.990 --> 00:17:31.310 why is he so much time talking about that? 00:17:31.550 --> 00:17:35.610 But like, we can't really do that in UV because like there isn't really a way 00:17:35.610 --> 00:17:37.550 to express that in the standards. 00:17:38.050 --> 00:17:41.130 And so, you know, there are some things that we have to kind of either can't 00:17:41.130 --> 00:17:43.110 support or have to get creative about how we support. 00:17:43.350 --> 00:17:46.370 But I think the workspace concept is excellent. And like, I'm really glad that 00:17:46.370 --> 00:17:48.270 we made it such a first class thing in UV. 00:17:49.190 --> 00:17:50.750 Now, I'm pretty sure that we could 00:17:50.750 --> 00:17:54.710 talk about workspaces for hours because there's a lot of nuance to it. 00:17:54.870 --> 00:17:57.510 And I think a lot of people that haven't tried it themselves, 00:17:57.510 --> 00:18:00.170 they just don't know what the fuss is all about. 00:18:00.290 --> 00:18:03.590 But I do believe that there's more to it than just the term. 00:18:05.050 --> 00:18:09.070 Now, one other thing that you mentioned, though, which is very close to my heart 00:18:09.070 --> 00:18:15.530 is parsing stuff, especially the requirements txt, which is unspecified, I heard. 00:18:16.470 --> 00:18:18.270 Now, isn't it super simple? 00:18:18.270 --> 00:18:20.390 Yeah isn't. 00:18:20.390 --> 00:18:23.930 It super simple like someone might hear this and say oh yeah i just opened my 00:18:23.930 --> 00:18:28.170 editor i string split every line on the equal sign and that's my requirements 00:18:28.170 --> 00:18:32.170 txt parser why is that not the case. 00:18:32.170 --> 00:18:35.550 Yeah so for requirements txt it's like 00:18:35.550 --> 00:18:38.530 this is basically this is a file format that 00:18:38.530 --> 00:18:42.130 exists for pip really pip is 00:18:42.130 --> 00:18:45.370 like the i guess sort of how should 00:18:45.370 --> 00:18:48.070 i describe it like the reference implementation for a lot of things 00:18:48.070 --> 00:18:50.770 it's really been like the python package installer for a long 00:18:50.770 --> 00:18:53.850 time yeah um and requirements txt 00:18:53.850 --> 00:18:56.770 is a file format that exists for pip and you know 00:18:56.770 --> 00:18:59.530 kind of the way that you can think about it which 00:18:59.530 --> 00:19:03.130 maybe people don't think about it this way is like it's 00:19:03.130 --> 00:19:06.370 basically like each line is a command line argument because you 00:19:06.370 --> 00:19:10.410 can not only put requirements in there you can also put command line arguments 00:19:10.410 --> 00:19:15.210 and settings yeah which is interesting so like on pip and pip you can pass an 00:19:15.210 --> 00:19:18.910 index url on the command line which is like what registry should i use for fetching 00:19:18.910 --> 00:19:23.410 packages you can actually put that in requirements txt too you can just do dash dash index url. 00:19:24.490 --> 00:19:29.530 You can also like embed like you can also nest them so like within a requirements 00:19:29.530 --> 00:19:35.070 txt you can do dash r and point to a different requirements txt and then it's 00:19:35.070 --> 00:19:39.830 sort of like it gets inlined roughly so there's a lot of complexity to the stuff 00:19:39.830 --> 00:19:42.950 that's in requirements txt and that people don't really think about. 00:19:43.570 --> 00:19:51.230 And there are also lots of very subtle behaviors where, especially for us, we often have to decide, 00:19:52.270 --> 00:19:55.810 well, what do we want to do? Do we want to be like bug for bug compatible with 00:19:55.810 --> 00:19:58.190 pip or do we want to do things like slightly different? 00:19:58.430 --> 00:20:02.330 And a lot of these are edge cases, but there's just a lot of nuance to it. 00:20:02.470 --> 00:20:10.570 Like in a requirements.txt file, you can have, I guess what I would call like a named dependency. 00:20:10.850 --> 00:20:15.110 Like you could say like flask and then the version that you want, 00:20:15.710 --> 00:20:19.930 or you could do flask at, and then the URL that you should fetch it from or 00:20:19.930 --> 00:20:22.010 like a Git repository or something like that. 00:20:22.270 --> 00:20:26.010 But you can also just do the URL or just do the Git repository. 00:20:26.730 --> 00:20:31.910 And it turns out that in PIP, there's slightly different behaviors around how 00:20:31.910 --> 00:20:37.110 those things are parsed, like the URL, if it's just the URL versus if it's after a name dependency. 00:20:38.550 --> 00:20:43.810 And how white space is handled, like error recovery, all this stuff is just a little bit different. 00:20:44.110 --> 00:20:48.310 And so over time, we've had to add, basically, the only way to know if we're 00:20:48.310 --> 00:20:52.230 doing the right thing is we just see what PIP does. and then we try to mimic that to some degree. 00:20:52.470 --> 00:20:55.750 Or if we think we can do things that are unambiguously clearer, then we'll do that. 00:20:56.710 --> 00:21:01.290 But yeah, that one's especially hard. It's a little bit easier for other things 00:21:01.290 --> 00:21:06.570 like Python version specifiers where there's a clear standard on how these things should be parsed. 00:21:07.470 --> 00:21:12.230 And if we see something that's not clear, we can actually ask about it through 00:21:12.230 --> 00:21:15.530 the standards process and be like, how should this be handled? 00:21:15.730 --> 00:21:16.590 Like blah, blah, blah, blah, blah. 00:21:16.830 --> 00:21:19.990 In essence, you make the standards stronger for everyone 00:21:19.990 --> 00:21:23.170 which is great what kind of piqued 00:21:23.170 --> 00:21:26.070 my interest there was did you have to 00:21:26.070 --> 00:21:30.510 go and read the pip source code to be able to understand what's going on because 00:21:30.510 --> 00:21:36.870 this is a standard that sort of developed over 30 years i think python is 30 00:21:36.870 --> 00:21:42.330 years by now it's from 1994 as far as i remember so and did you really have 00:21:42.330 --> 00:21:44.030 to read the source code yes. 00:21:44.030 --> 00:21:45.850 Sometimes we definitely have to go read source code, 00:21:46.740 --> 00:21:49.460 and which is fine like i don't i don't 00:21:49.460 --> 00:21:53.080 really mind that like as long as we can figure out like why a certain behavior exists 00:21:53.080 --> 00:21:57.240 in a certain way and like what it's motivated by but yes we often have to go 00:21:57.240 --> 00:22:01.560 read the source code to understand like how does this tool handle this case 00:22:01.560 --> 00:22:05.940 and like and and also why right like a lot of the time you're trying to understand 00:22:05.940 --> 00:22:09.480 why people made certain decisions not just how it works that. 00:22:09.480 --> 00:22:15.260 Means you don't even you don't only have to read the source code you also have to read the history the. 00:22:15.260 --> 00:22:19.400 Yes version history yes but do. 00:22:19.400 --> 00:22:24.260 They have proper tests at least where you can go through and maybe implement 00:22:24.260 --> 00:22:26.580 these tests in in your implementation. 00:22:26.580 --> 00:22:29.700 Yeah they do have tests yeah it's not always like super 00:22:29.700 --> 00:22:33.180 straightforward to take their tests and move them 00:22:33.180 --> 00:22:36.500 over to uv like a lot of the times i don't 00:22:36.500 --> 00:22:39.320 know just the way that you write tests in like python and rust often tends to 00:22:39.320 --> 00:22:42.460 be quite different like or at least the way that we write tests 00:22:42.460 --> 00:22:45.400 tends to be quite different like you know like in python like 00:22:45.400 --> 00:22:48.340 mocking is very popular for example and like we 00:22:48.340 --> 00:22:51.480 don't really have we have like almost no walking in uv like 00:22:51.480 --> 00:22:54.360 almost all of our tests i think 00:22:54.360 --> 00:22:57.480 are what you would probably consider like integration tests 00:22:57.480 --> 00:23:00.900 like the whole way that we test uv is we 00:23:00.900 --> 00:23:03.960 actually run real scenarios against 00:23:03.960 --> 00:23:10.200 a real package registry and it's almost all based on snapshot testing so like 00:23:10.200 --> 00:23:14.940 we snapshot the cli output a lot of the time that's like how we detect if things 00:23:14.940 --> 00:23:18.420 are working correctly or not is we snapshot the cli output or we snapshot the 00:23:18.420 --> 00:23:22.080 lock file or we snapshot things like that but like the vast majority of the 00:23:22.080 --> 00:23:23.740 way that you get tested is. 00:23:24.560 --> 00:23:26.180 Integration tests of running real 00:23:26.180 --> 00:23:32.160 commands on the cli and snapshotting the output and that's like we have, 00:23:32.970 --> 00:23:36.390 a lot of tests and they all run through that. 00:23:36.650 --> 00:23:39.350 So it's a little different sometimes because if I go and look, 00:23:39.410 --> 00:23:43.850 we do have obviously more traditional unit tests for requirements.txt parser. 00:23:44.130 --> 00:23:47.610 That's something that's very testable, right? And we do have tests for that. 00:23:47.730 --> 00:23:53.690 But often if it's more complex scenarios around how PIP behaves versus how we 00:23:53.690 --> 00:23:57.510 behave, it's a little bit harder to shoo it into what we have. 00:23:58.290 --> 00:24:00.350 Do you always run the full test harness? 00:24:01.770 --> 00:24:02.450 In CI? 00:24:02.970 --> 00:24:08.370 Yeah, or maybe locally as well, but I wonder that's probably too much. 00:24:08.950 --> 00:24:13.730 Yeah, we do always run, we almost always run the full test harness. 00:24:14.070 --> 00:24:20.070 So like we just, we skip it for changes that are purely documentation. 00:24:20.750 --> 00:24:24.890 Which we just detect with some file filters in GitHub Actions, 00:24:25.050 --> 00:24:26.470 but otherwise we run the full test suite. 00:24:26.770 --> 00:24:30.250 Every change we run it on Linux, macOS, and Windows. 00:24:30.830 --> 00:24:34.690 And that's because we do like 00:24:34.690 --> 00:24:37.590 that that that's not as critical for ruff but 00:24:37.590 --> 00:24:41.050 it's very critical for uv because with the 00:24:41.050 --> 00:24:44.110 package manager just like more things tend to differ across 00:24:44.110 --> 00:24:47.410 platform and more behaviors differ across platform and 00:24:47.410 --> 00:24:50.470 so we always test everything on linux mac os and windows 00:24:50.470 --> 00:24:55.130 i guess one interesting piece is like we we 00:24:55.130 --> 00:24:58.450 build uv for like a lot of different targets so you 00:24:58.450 --> 00:25:01.510 know when we publish to py pi or publish anywhere 00:25:01.510 --> 00:25:04.430 or publish to github releases like we're 00:25:04.430 --> 00:25:07.250 building like 15 probably somewhere between like 00:25:07.250 --> 00:25:12.990 15 and 20 different builds or maybe is it 15 somewhere between 10 and 20 let's 00:25:12.990 --> 00:25:19.550 just say different build variants and so that's like linux x86 you know dynamically 00:25:19.550 --> 00:25:25.990 linked against glibc linux x86 sorry linux x86 musl linux arch GWC, Linux ArchMuzzle. 00:25:26.110 --> 00:25:29.910 We build for some more obscure platforms that are supported in Python, 00:25:30.210 --> 00:25:33.790 like PowerPC and S390X and stuff like that. 00:25:34.070 --> 00:25:41.890 We build for ARM and x86 Windows, ARM and x86 Mac OS. 00:25:42.190 --> 00:25:45.710 So we just have a lot of different builds. And... 00:25:46.150 --> 00:25:48.930