This is Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from corode, and today we talk to Daniel Stenberg about integrating Rust into Curl. Daniel, thanks for being here. Can you quickly introduce yourself? I don't think you need a lot of introduction, but maybe say a few words about yourself and Curl. Sure. Hi, I'm Daniel. I am a Swedish developer. I started with open source in the 90s. I'm mostly known for being the founder and lead developer of Curl and the Curl project, LibCurl. And that is what I do mostly, internet transfers and protocols and Curl. That's my daily thing. And you've been doing this for quite a while and i saw many of your talks in person you are kind of a i would say popular or well-known developer and i guess one of the reasons why people admire your work is that you've been on this for many many years decades even by now you started with curl in i guess 1996 if you might count in the pre-precurses of curl is that correct. In November 1996 i looked for i wanted to download currents rates off the web so i looked for a tool that could do that i didn't want anything big and fancy just a small tool to download HTTP and i found one called httpget so that i didn't write but i found it and you know No, that was even before Google. I imagine I used AltaVista. I don't actually remember. But I found that tool. It didn't really work the way I wanted to. So I communicated with the author, a Brazilian guy called Rafael Sagula. And so we bounced some patches back and forth. And I became the maintainer of httpget in late 1996. That's how it all started, really. Okay. Fast forward to today, how many devices does Curl run on? Do you have a rough idea? Oh, I've decided to more count installations because I don't really know. I count somewhere around 20 billion installations of Curl. And it gets that many also because it typically runs in multiple installations in each mobile phone, for example. So any typical mobile phone runs like three or five copies of curl because lots of apps ship it independently so that there's a lot of curl everywhere yeah. And the moment one of the big vendors like instagram for example adds curl to one of their apps you have another whatever like 600 million or 1 billion. Users exactly and and the other way around too right if one of them stops using it it's going to be yeah sure i can lose a billion users or a billion installations overnight like that and it happens it goes up and down and also since it's open source and usually these these companies and product people they they just you know they pick curl and ship it in the products and i don't know about it i just realize it one day once you know when i just stumble upon it for some reason someone emails me and ask me something or i just happen to find it So it's very hard to actually have any accurate numbers. That's why I say I'm guessing we're really trying to Google, find the curl license and see in what kind of products, how many products are using this and blah, blah, blah. So it's a rough estimate. I guess you can just roughly estimate the numbers from, say, licensing terms in handbooks or something. I don't know if, for example, my car vendor, which is VW, maybe has your name in their handbooks. Do you know? They do. I'm not sure it's in every VW car, because I have screenshots from several VW cars with my name in it. So I know it's there. It's actually in most cars. But I don't know exactly in how many. I mean, several hundred million, surely. Sounds scary. You know, you end up in a situation like when several hundred million installations, it doesn't really change the scale very much when you're at 20 billion. So sure, another hundred million users or so. Doesn't that keep you up at night from time to time? Doesn't it scare you? Usually not, actually. It's kind of weird in that way. But I think it's because it's been such a gradual growth and scale and sort of development that, I mean, it has taken a long time to get here. And we have come to this point because we were at the previous point, right? So it's sort of building on top of previous, maybe not experiences, but proof that we can do this. And it's fairly reliable, right? And over time, we had more and more tests, more and more systems to verify this stuff. So I'm actually, these days, I'm pretty certain that, I mean, the internet won't burn tomorrow just when I do a new release, maybe, hopefully, you know, crossing my fingers. So I usually don't get too sort of scared about most devices in the world running curl by now. Now, it feels pretty good, actually, because it's also, you know, obviously, since we have come to this point, it seems to run pretty well in all of these places. So obviously, a lot of things have been done right. And we just keep on adding, you know, tightening the bolts, doing things better, testing more, verifying more. So I think we're actually, I mean, I think we're going in the right direction rather than having any sort of. Special dangers that something will i'm not saying that it can't happen i'm sure it will happen at some point more i mean bad things happen every now and then but, with a little luck we should be able to avoid at least the worst kind of outcomes. One thing that probably a lot of people value about curl is its stability and we will get to that in a second you touched on that briefly but i wonder in one of your talks at FOSDEM2024 you mentioned that curl runs on two planets now earth and mars does it also run on the moon. A most likely i would say but but again especially the space people they don't tell me what they do right so and and like in the mars case nasa has only said that they use curl in relation to that project They won't tell me exactly what they use it for, because that seems to be some kind of secret. It's used to further their mission or whatever they said. So I don't know. And I think it has been used on Moon, but I haven't got any proof about that. So I don't say it because I don't know it for sure, but possibly or likely, I don't know. That's so funny, though. what is the most safety critical environment that curl runs in that you know. Of i guess it's safety critical i guess that depends on your point of view but i mean it's used in pretty much all kinds of vehicles for example and it's used in a lot of medical devices i guess those are maybe the most safety critical ones i don't know what about. Power grids nuclear. Reactors? I don't know any of that. But I mean, it's also used very commonly in like financial institutes, like banks, insurance companies. And so it's also... Kind of critical in that kind of aspect that it's an infrastructure tool for a lot of our kind of modern society. Does the C in curl stand for C by the way? No, it stands for client or possibly for the verb "see", to see a URL. Because when I picked a name, I wanted it to be short and easy to type on a keyboard like Unix style. So it would be easy in short but i also wanted a url in the name because it would work with urls and then it couldn't be just the url but that would be stupid but client for urls would be fine or just see, urls so we kind of it worked either way so i think that would be enough and it was pronounceable in english so curl an english word so you know naming is really hard how do you name something i think it went pretty good it worked out pretty well because if. You search for it you will find. It so it's awesome seo the. Core of curl is written in c of course and with all of your experience looking back over two decades of working on this what are some of the things that you like about c. There are there are many things to like about c i think i mean first of course we have to recognize that I started this in the 90s and there were not many alternatives in the 90s for picking a language to do something like this with. I mean, there aren't that many options today even. So, of course, I went with C. Back in the 90s, C++ wasn't even an option because of the inability to do stable APIs and stuff. But anyway, so I think one of the primary good stuff, of good reasons to go with C is that it is available everywhere, literally everywhere. I mean, we talk about 20 billion installations everywhere. One of the reasons why it can run on all of these weird platforms everywhere, it is because it's in basic C89-flavored code sets. Basically, you just need a C compiler, which exists for any platform anywhere. So you can run it on any crazy operating system, and you could run it on any crazy operating system already from the, well, early 2000s at least. Does that mean that there would still not be any alternatives today if you started a project knowing that it would be used on so many platforms there is very little other choice is there. No there really aren't but but of course you could then question is it is it sensible to aim for that many platforms if you start a new one today you could of course discuss that and sure you can debate that back and forth But I think there are a few options today, and we're talking about Rust here today, right? So, sure, Rust is, I would say, it's becoming, it's only becoming an option. But for me, sticking to C has been a key, and I think it's been a key to its development and success over the years. And what other alternatives would there be? I can imagine C++. Yeah, people have brought up C++ over the years, right? But I mean, I have a strong feeling against C++ because I think it's a messy language. And I think because typically people mess up code written in C++. So it's both ways. So I'm actually, I've never been very happy with C++. I worked for several years at Mozilla working with C++ all day. But it just solidified my opinion that C++ is not my favorite language. So I don't want C++ in curl. And I don't think it had helped curl. I think it just would have made it more complicated. So I don't think C++ would have been a better solution in any way. Was it ever raised as a topic of interest to integrate C++ into curl? Not really. More in the way that everyone sort of, you know, it's easy to just bring your opinion and say, hey, you should do this. Someone from the sidelines just throwing, you should blah, blah, blah, blah. Use your copious free time to do what I ask you to do. But I've never considered it seriously, no. The same might be true for Rust. People might say, you should use Rust or you should rewrite it in Rust. It is. There are certainly similarities. Yes. Doesn't that put you off then if people constantly evangelize rust in that way? Not really. I think it's more of a pattern in how humans work. So the people it's so easy to just bring out, you should do this because I have seen the light. I will tell you how you should do things to make things better. It's just that people are like that and it will always go on like that. I just usually, you know, don't really care so sure you can say things and and bring ideas but you know ideas are really really easy it's easy to say something it's easy to come with a good idea the hard part is not the idea the hard part is actually making it reality right implement it do it make it happen so and so sure rewrite it in whatever and sure if you come along and make it happen then we can talk then we can probably maybe make it happen and make it a reality. But just throwing it out there probably won't help that much. And now, you said that you have sort of mixed feelings about C++, and a lot of people compare Rust with C++. How do you see the differences between those languages? Where would you say Rust made the right decisions, where C++ made some wrong decisions, maybe? I think, first of all, I'm really not a Rust developer. So I'm just a rookie Rust person. and I couldn't even, you know, I couldn't write anything in Rust. So my take is more sort of understanding how the system works and my experiences with C++. I think, for example, one of the downsides with C++ is that it is so much C. So that it sort of, it brings all the C and it tries to do things C++, which allows everyone to shoot themselves in the foot and at the same time, try to do everything object-oriented in the way you never figure out who's doing what because everything is inheriting everything else and you just never understand who's doing what. So I think by not having this based on C, I think Rust has a much better position. And of course, I think Rust has a much better starting point because just of the ownership and trying to do it memory safe and stuff like that. And then half of the C++ people would say, well, you can do it with C++ as well. But no, you can't. So, yeah, I think Rust has a much more going for it as a safe language. But then, I mean, I can't really tell how it works in reality in like a big code base. Maybe they will end up with similar problems or similar challenges. I can't really tell. The one thing that a lot of people from C backgrounds mention when they talk about C++, including Linux Torvalds, for example, is that exception handling kind of goes in the way of how you would want to model a lot of low-level tools like kernels or maybe also HTTP clients that get used in multiple billions of devices. That's certainly true. True. I think in general, when you do things like low-level code kernels or libraries in general, you don't want exceptions. You just want everything to be managed basically by success or not, right? Return codes. And yes, and then we're touching on one of these, at least. I'm not sure exactly what the situation is right now in Rust, but running out of memory. What happens when you run out of memory? I think for a library, like used by millions of applications, it should just return a failure, right? The operation failed. And then the application can decide, what do you do when the transfer failed? You can try it again. You can exit. You can, yeah, well, it's not up to me to decide. I write a library that makes the transfer. The transfer can fail due to certain conditions. Running out of memory is one of those failure conditions. It should just return an error. Yeah. So that's it. One of those things that I ran into with Rust originally, years ago at least. Yeah, I remember there was a EuroRust talk from 2022, and there was a point about OOMs in Rust, like out-of-memory exceptions or like out-of-memory situations in Rust. And the problem is that, Rust panics can be handled with a panic handler but if you have an OOM in Rust that calls abort and that cannot be handled right now so there is an RFC to fix that but the problem is not panics necessarily the problem is that OOMs don't panic so they cannot be handled ah. Right okay I understand. But there is another thing in Rust, which is to try and preserve memory. For example, you can try to allocate memory for a vector. And if this operation fails, you get an error, which is just a variant of the result. These might be things that maybe would help you make this a little bit more defensive. Or if you write a library, maybe you can handle these cases. But I don't know if such patterns are very widely used. Right. And for me, you know, I'm taking more of a stance that I think these are more identifying issues that are going to likely be sort of problematic to some users. But it doesn't really stop me. It doesn't really prevent me from adding anything because I can just say that, well, this is a fundamental flaw in this setup, but I can still support Rust written libraries, right? It's just, well, this is just one effect of the total thing when you build with this. And therefore, I like, my approach is more rather to, sure, I can support more stuff and certainly libraries or subsystems written in Rust. But at the moment, I also provide others, right? And I rather let my users decide or sort of let the world pick the combination they want if the world decides that we only want the rust written libraries then sure then we will see that happen and then there will be more work and more emphasis and more attention to the to those libraries and then they will get more improvements and more you know and over time they will be better and maybe that they will become the default options going forward or whatever. So I rather sort of let the, let users and the world of who are using curl. Pick what they want and need and go with that. I can imagine that one of your main roles in curl or just in general as a lead developer is risk management so being aware of all the unknown unknowns and every new technology like a new programming language and maybe trying to gauge whether this is a big risk for the project or not further down the line is that correct Well. A little bit yes and a little bit no, right? So, of course, I want to understand sort of the ramifications and understand what it is. And is it worth spending time on at all, right? If someone throws up something really weird, I don't want to spend time on it and efforts and bring it in to support. If it's just something crazy that will just make users unhappy and just turns out to be a waste of time. So sure, I need to sort of have some faith in it before I put in effort and do something with it. And in this case, I believe in Rust as a tool, method, language to do things. So I believe in it as a... But then I don't need to actually think that rustls will replace OpenSSL over the time. It's not up to me to decide. I can support them both. So sure, if you want to build curl with rustls instead of OpenSSL, go ahead and do that and tell me what doesn't work and we can fix that and go forward. And I can also leave that risk assessment to the users who are building devices, right? If you want to build your device and you You want to do a robot dog. Do you want, which kind of TLS library do you want? You decide that. Curl supports 12 different ones. You can pick the one that works for your device. I don't have to do that. I don't have to make the decision. I do to some regard. So if I figure out that one of those 12 libraries is really crappy, you know, development stops, it seems to be a really bad choice for users, then I can drop that and say, hey, users, don't use that. Here are the other 11 you should go with. So I can raise the standards and make sure that at least I don't recommend anything that is really truly bad. I try to do that so that I don't lure users into something they shouldn't use. But other than that, I also want users to have the choice and therefore let the market decide what is right and what is wrong going forward. It helps me because then I don't have to make that decision. I don't have to look in the crystal ball and say, what is the future like, which is the right solution, which library is going to win, which is going to be the better one in five years. I don't know. I'd support several of them and let them fight out between themselves. I think that's very smart. And I want to get into all of these details in a second, but before we go and talk about these things, can you just quickly give us a timeline of Rust adoption in Curl? When was the first time you heard about rust in general when was it proposed for curl when was the first little piece of library code that was pushed to the curl code base and so on. So what was the process i i was employed by mozilla from 2014 to late 2018 and at mozilla i of course learned about rust and i think the first public release was in that time period i wasn't involved in it in any way. I've just sort of heard of it. And at the end of my time at Mozilla, we also started to merge the first components into Firefox that were written in Rust. So I certainly was aware of Rust and sort of the concept and the idea. As I already said, I don't author Rust myself, so I'm certainly not sort of good at Rust, but I understand the concept, I knew about it. And I always sort of appreciated the idea that you could do it like, we did in Firefox, and sort of replace components piecemeal, one by one, with one written in Rust instead of the C++ one. So that's certainly one of the fundamental strengths of Rust, so that you can go one by one and improve things over time. So I knew about Rust for a long time ago, but then fast forward into the Curl project, I was actually, when I started to implement, I think, yeah, I think I started to implement HTTP3 in 2019, I think we started that. And pretty quickly, I was one of the, I think the absolutely first backends for HTTP3 that we support in curl is the quichelibrary written by Cloudflare, and that is implemented in Rust. So I think the first HTTP3 support we had, the first experiment, you know, it didn't really work, but you could, you know, do some first basic HTTP3 stuff. That was using the Rust library. So using a Rust library was never any weird thing. It was just sort of just, sure, that was just the nature of it. We can use Rust libraries as well. And then a few years later, I did a work with the ISRG, the Let's Encrypt People. And then I introduced support for Hyper, the Rust HTTP library. So you can actually switch out a chunk of LibCars native HTTP code and use Hyper instead as a build time option. And so, and I think in the center, I think I started at 2021, was it early 2022, something like that a few years ago. And in the same time, we also got support, the first support for rustls merged into curl as in the TLS option and all of these, so that they fit the pattern that I already mentioned in cURL that we have a lot of backends. So you can optionally pick and choose when you build cURL which kind of different, third-party solutions you want when you want different protocols so you sort of pick different libraries to go with so you can go with keys for HTTP3,hyper for HTTP1 and 2 and rustls for tls and then you get quite a bit of rust in the execution binary that becomes current but. All of these These libraries are still, I wouldn't say experimental, but they are opt-in, right? Yes. I think we still consider them all experimental because they are all almost there, but not quite. Quite annoying, actually, because they're all maybe 90, I don't know how many percent. I don't have any exact numbers, but they're all like 95 to 98% done. They're almost there. We have a few test cases that fail with them enabled, and we have a few quirks remaining. So there's just a little friction. We just need a little bit more polish in all of those three areas, really, to make them non-experimental. I just checked, and for Hyper, there were still a few disabled tests. For Rust TLS, there were also some disabled tests, and there was one other thing about IP support in certificates, which might or might not be supported by now. I don't know. I think that's fixed. Okay. But we have another issue right now with rustls. So they fixed their issues, and we still have to catch up with their fixes and adopt them and adjust things. So I think, yeah, we just need a little bit more time, a little bit more effort to make sure that everything is glued in correctly, at least in the hyper cases I think in several of the cases it's just the integration between so it's not really curl's fault it's not really hyper's fault it's really the integration between them that needs to be, polished a little bit more. And about this integration in general, you mentioned one of the advantages of Rust would be easy interop with other languages through their FFI interface, their foreign function interface. Were there any things that you encountered when integrating a backend like Hyper into Curl that fundamentally were different between C and Rust and would prove to be some challenge for a project? Or was it more or less easygoing? Yes and no, but it's actually quite straightforward. So there's really nothing fundamental that stands in the way of doing it. I think, I mean, one of the things that I'm struggling with, actually, in all of these three projects that are all Rust libraries, they provide excellent Rust documentation. But if you're a C programmer, you want to know how it works as a C function, right? What does it actually return? What does it actually do? And none of them actually provide C documentation. And sure, they convert the Rust code into a C API. But from a C user standpoint, it's complicated to read documentation for Rust and trying to understand how that actually maps to C. I mean, that's not actually a language thing, right? It's more of an issue with me as a C user trying to interface a Rust library. But there are things like that. Otherwise, I think we had some early issues with Rust. There were some global symbols, for example, that made it impossible to load two Rust libraries statically. But I think those were just, you know, some child diseases in the beginning. So I think, otherwise, I think it's pretty straightforward. I mentioned that if you wanted to render c documentation from rust there might be a need for a certain library or a certain tool that does this i am not aware of such a tool but it would be nice if if we had one or if if i was aware of one right well what do c people use for this is there a documentation tool that renders that that i can view online then like the html output of it or is it mostly integrated into the IDEyeah. I don't think there's no there's no standard way to there are ways i mean there are tools that are converting you know header files to documentation and stuff like that in my case i have all my documentations written separately standalone because i think that's the better way to do it but that's more of a religious taste thing rather than facts. As someone who used that documentation in the past i really appreciate that part it was really extremely well done and it saved me so much so much time the reason was back then that i needed to interface through the php extension of curl and there was an issue with it and i could very clearly see how the C function worked. Even though I'm not a C developer or I'm not really very experienced in C, I could still understand what was going on roughly. Right. So kudos to that. You once mentioned, and correct me if I'm wrong here, that users in general don't test experimental stuff. They don't enable optional features and they build their own core client just to experiment with it. Is that still the case? Is that true? I think that is still true to a rather large extent, yes. And does it also mean that the Rust... Code base or the rust integration doesn't get that much exposure and maybe it's not you know it's not handling all of the error or edge cases. Yes and. And that could be a problem for. Rust integration exactly so yes so there's a little bit of a chicken and egg thing there right so so you want people to test it out so that we can find the problems and fix them so that we can make them sort of not experimental so that people will enable them but but yes so it's certainly, hashed that challenge. I actually think, in general, I think the Rust backends have that challenge in general, that it is in the land between C programmers and Rust programmers. So it's a little bit of a hesitation from Rust programmers, and there are hesitations from C programmers rather than the opposite, I think. So it is a challenge. I also think that's why We came to 95% support in all of these three backends, but now we have struggled to get that final little push over the goal line because now it's a bit difficult, challenging to get that first 90%. That was easy. The last 10%, that's where 90% of the work is, right? So, yeah, it's becoming harder and harder and slower and slower, And it takes a particular kind of mindset and person to actually manage to take this further. Do you see a future where one of these Rust backends could be enabled by default? Sure. I think so. I think it's mostly, I think all of them just, they just lack someone to, you know, step up, roll up your sleeves and make it happen. You know, don't give up so easy and just spend whatever time it's required. I can't say how much time it's required. It depends on person. But I mean, it's not that many tests. It's not rocket science. It's just code, like I used to say. It's subtly fixable. I can imagine that if you would just spend devoted time and energy on it, it wouldn't take that long. Which one of the backends would you say is the most mature one from the three? We have Hyper, rustls, and Quiche. They all serve different purposes, but one of them might be more stable than the other. I'm not sure I have the oversight to say which one. I think Quiche is the least one. Elite sort of mature mature one exactly but and also because they sort of also consider themselves beta much more than i think hyper and rustls do i i imagine i could imagine that hyper is the one that has reached the most maturity but i wouldn't really i couldn't say that for sure i think rustls is also pretty mature they complicated our lives a little bit by changing their api recently so that's one of the bumps that we have to overcome you know upgrade to the new api so. Speaking of which isn't that an annoying part in rusted a lot of those libraries are not stable yet they have breaking apis isn't that a cause of churn for curl. Yes certainly that that sort of makes it more complicated right because then so like in this case it's hard to keep up with the latest changes because oh suddenly they changed the api so we cannot just you know upgrade to the latest, we have to understand the new concept because they removed the old concept and introduced a new one with new names and a new concept. So sure, it's certainly a, limiting factor because i mean you know we all we all like every open source project there's a limited amount of people and a finite amount of resources so what do you spend your time on every day and every hour so yeah you have to decide what to spend sort of where to go and maybe, improving the rustls back end or the hyper back end or the quiche back end is not always, top the top prior. How many people work on the rust part of curl is it a single person two is it more people. I think we're closer to zero right now so it's i mean we're not a project that we we don't have that distinction really we don't have people working in areas we have people working in the project or not and we're we're not a fixed amount of people either because we're an open source and we i'm the only person who works on it full time so there's me and then there's people who only did one commit and everything in there between so and who's working on the rust stuff i i don't think we have many people working on it regularly right now so i think we're closer to zero than to one right now maybe. That's an opportunity for some of the. Oh it's certainly an opportunity. And tell us a little bit about the curl test suite do you run all of the backends through the same tests and so even if the rust part wasn't enabled by default you would still get some exposure to real-world edge cases? Exactly, that's how we do it. So we pretty much have a test suite that has a bunch of different tests. So we can run tests with the curl command line tool, which is actually the most convenient way. And we build special dedicated tests for running lib curl tests with library in ways for cases that the curl command line doesn't do. And then we have also dedicated unit tests that just runs various kinds of functions directly. correctly. And then in the test suite, we build curl and lib curl basically with all supported TLS backends in different, and even in a few different combinations. And then we run the same tests with all the different TLS backends. So ideally then whatever you build with all the tests should run the same. So if you run, build with rustls and run the test suite, and then and it should just end up saying, OK. And then we know that all what we're testing is working OK, which should cover, quite a lot of TLS cases. How long does a usual full-blown CI run, a continuous integration run, take? That's a little bit annoying because some of them are slow because of CI's infrastructure reasons. I think it takes somewhere between, in the best case, maybe 30 minutes to a few hours, maybe sometimes more than a few hours when we sort of have many CI running at the same time. And do you have your own continuous integration infrastructure or do you run it somewhere else? We run it on a lot of free CI services. I think we use six different ones right now. So we're on basically anything you can find that offers free CI services. We're not all on free tiers either. We have a few bumped up power tiers as well. And I think we run like 140 CI jobs on every pull request. So there's quite a lot of builds going on. In total, I think we're at around 190,000 tests per pull request or something. With Rust compile times taking so long, wouldn't it make sense to have dedicated infrastructure for it or at least have one of the bigger cloud providers step up and provide bigger cloud infrastructure for CI? Sure. Wishful thinking. Exactly, sure. They would be very welcome to do that. I actually think right now our biggest bottleneck is actually just, I think, some of the Windows CI infrastructures, because they don't offer enough power to us and we don't pay enough, I guess. We could probably pay more to get more. So I think that's why they are queued up sometimes. That's why it takes longer time. not because they're busy, just because they're waiting for some parallelism to go down so that we can run them. Even if you don't write ROS code yourself, do you still have to build it on your local machine from time to time? Oh yeah. I did the entire hyper adaption in curl, right? So of course I built Hyper hundreds of times locally. So yes, update Hyper, build it, run it, test it and figure out what's wrong, build it again to test it. So sure. And, Yeah, so I'm kind of used to at least updating building hyper things or Rust things in general. And I built all of those Rust things locally and I test them locally. And in particular, when we run into problems, it's convenient to be able to run it and debug it locally to figure out exactly what's going on and how to fix it. I mentioned the long build times, but I wonder if that is true in your case. Did you even have problems with long build times? No, they're really fast. So there's no issues. In general, with curl, there are never long build times, actually. It's, in comparison, a tiny project. And all of the other sort of related libraries, they're all kind of small. I mean, in the grand scheme of things. So build times is not the problem. I think running tests, they take usually longer than building things. Now the rust ecosystem is still pretty young by, Looking at it at a scale from, for example, many, many decades, like C is around for many decades now. And in comparison to Rust, it's still pretty young. What do you believe will be in Rust's future? Looking forward, maybe not just two or three years, but a couple decades from now, do you still see it to stick around? I mean, for curl, it's not a huge problem because you can just disable the backends if you want and maybe deprecate them. But just what is your personal opinion on rust is it will it stay is it something that will be as popular as c and c plus plus in the future or do you see other competitors also take that space. Well it's really i mean talking about that long future it's really impossible to say right and i've never been good at telling the future so i rarely try to i just follow along but i i mean I mean, I think Rust has everything that it needs in able to become that sort of future C alternative that could sort of replace C in virtually every aspect over time. But if it does, I'm not sure. I don't think I've seen any other viable alternatives, rather, to replace Rust. At least for... I mean, in my world, I work with libraries and low-level code, and then many of the other new popular languages, they're not really there to replace C in that kind of environment. So I think Rust is basically the only current available alternative to C in a lot of these cases. And so sure, I think it certainly has a chance to do that. Whether it does that or not, I can't really tell. And if it failed, what would be your most likely reason? If it failed, no, I think the only reason it can fail is if something else can do it better, smoother, easier, fancier with something else. I mean, but it has to be something with a huge backing. So I don't see anything of that short term. But, you know, if you're talking long term, something can happen. I don't know. Do you see that some existing C tools get rewritten in Rust now? That seems to be a trend in other languages, but I'm not sure if it's also true for C. Well, I think it is. I think it is a trend. I mean, people are rewriting a lot of the popular Linux tools, like the bin utils and stuff in Rust alternatives. So I think it's over time, it will happen more and more. And so Yes, I think so. Even if my personal opinion is also sometimes that it's a little bit of waste of time. At least, I don't really like this popular sort of thing that every language should rewrite everything in their own language again, because it seems like that's a pattern we're doing over and over. Every language should always introduce a lot of things. And now we should rewrite it again in our new language. But I also understand why. So sure. And everyone is free to do whatever they want. So why not? If that's what you want to do, go ahead and do it. Well, one reason might be to use multiple cores and use modern hardware and SIMD and all of these. Exactly. And make it more safe and easier to use more threads and then thereby make it more efficient, faster, better. So sure, there are certainly, there can be many good reasons to do it, yes. Doesn't curl also have a way to run multiple requests more or less in parallel? I'm not sure if you run them concurrently or in parallel, but there is a way to spawn multiple requests, right? And how does that work? And also, did you take a very defensive approach when you integrated, or were you going all in because you kind of knew what you were doing? They don't do threads. They are all just state machines. So it's all in the same thread. So we do any amount of transfers in the same thread concurrently. So it's just a lot of state machines. Yes. It's very similar to async programming in Rust then, right? Yeah, it is virtually the same thing, just done manually. Yeah. And under the hood, it uses an event system like epoll, or how does it work? It's actually more complicated than that, because curl itself doesn't do the event system. It allows the application to provide the event system. So an application can go with any event system it wants. You can go with any epollor LibEvent or LibEV or LibUV or whatever you want and then use curl to do that. And then because that's typically the way how you do really high performance Lib curl transfer. If you want to go like several thousand concurrent transfers in a single thread, that's really how the way to do it. That's a very smart way to do it. does the curl command line tool also do it or is there no multi. Now the current command line tool has the dash capital z option z and then it does them concurrently too so you can do parallel transfers like that and then it can do any any amount of parallel transfers but again they're in the same thread and. How does the command line tool run these things it needs some sort of it It brings their own event handling system. Yeah, but then it doesn't do it with an event-based one. It just does it with a select-based one or poll. So it's a little bit less performant, but it's fine. As long as you're sort of below a few hundred transfers, then it doesn't really matter which one you go with. You can also use curl from Rust. And that is a very nice thing to do because you get all of the advantages from cURL, all of the knowledge, all of the hard work that was put into it. And you can just use Rust from your application perspective and call into cURL. Did you ever interact with some of the maintainers of the libcurl crate? I think I've been communicating with them, but not a lot. I think they've sort of managed most of that themselves. ourselves. But I think that's also thanks to us having a pretty solid product and a pretty good documentation situation. So most people can actually figure that out themselves. They don't need to ask me. They don't have to. As long as they can just go with what we have and they understand how it works, they don't need to tell me about it. But sure, and even cargo and stuff is using libcurl. So there's a lot of curl in Rust. Oh, yeah, I didn't know that. Yeah, so every cargo command line is using curl underneath. I think I should introduce a curl counter on my terminal somehow because I think I used it thousands of times a day. It's incredible. And it also runs on all of the platforms that I use. It runs on Linux, but it also runs on FreeBSD. It runs on Android. It runs on wherever. And did you run into any specific issues regarding building for these platforms also probably people try to build rust on some of the more exotic architectures as well did you ever encounter any any issues with that. Sure i mean every basically every new platform has the opportunity to introduce some new challenges right so going you have to be pretty conservative in how to do things to enable to just open up the opportunity to build on a huge amount of platforms. So it has to be close to POSIX, but also open for POSIX violations. And that's basically how we did it from the beginning. And then we've always been very careful to do if-defs for features that are available in the platform. But sure, Sure. Whenever, if you want to go with curl and run it on a new operating system, new platform, sure, there's going to be some tweaks that might be necessary to do, not only defining just, you know, the rich set of different features that the platform might or might not have, but there are also different, you know, one of my recent things is that I've ported curl to a lot of real-time operating systems, and they tend to be more crazy than other operating systems because they typically, for example, change some of the POSIX functions for sockets or something like that, because our function that makes a connect, it should take another argument that's used for this, which is sort of crazy. So then, yeah, then you have to do a special crazy thing for just that particular operating system and things like that. So it certainly takes a lot of, sometimes just a little, sometimes a lot of work to make it run on a new platform, a new operating system. But in my case, when I have this list, I have 101 operating systems that Curl has run on now, and most of them I never ran Curl on, and most of them I never even heard of. Someone else made it run there and were happy, and they just told me about it. So apparently it is also easy enough for others to make it run on a fair amount of different systems. Does it mean that other people also maintain the Rust compilation for some of these platforms? I don't know. Usually they don't tell me exactly what kind of setup they do. So I would imagine that Rust is... For the more weird platforms, they don't go with Rust solutions at all. They go with some traditional C libraries. So I guess they don't use the Rust ones. Okay. So for most platforms, you kind of know that the C code base works and you compile a minimal version of it. And maybe for some platforms, you might also provide Rust support, but it might be something that they have to opt in and maybe add that. And like always, we just ship curl as a source code, right? And the ones who are going to build it, they decide how to build it and with what sort of components and what backend. Ends so they are it's up to them so whatever wherever you build curl you can select how sort of which back ends to use which components so you can go with the rust components or the c components that's up to you so i guess and especially since the rust ones are all experimental and beta versions still most people would go with the c versions because some some of them are not experimental are. The backends compiled in statically or dynamically or can you choose. You can choose since it's all c code so you typically go it depends on your platform usually or your religion or something but but if you go with a modern system like linux you typically go with everything shared yeah. I looked up some of the edge cases that you ran into when you integrated the Hyper backend into Curl. And there were a few things that you raised on the Hyper issue tracker, which just helped improve this grade in general. I really liked that part. Yes, I think I ran into a whole bunch of issues. And I also actually think it worked the other way around, that I actually also fixed Curl stuff because I realized that Hyper had done solutions in a clever way that I could just as well do that. Because when incorporating Hyper into Curl, I ended up in this fun solution that, again, I mentioned the test suite. I run tests and they should work the same way, right? Even if I build with Hyper or build with a native HTTP, they should run, they should produce the exact same byte sequence over the wire pretty much. And of course they didn't at first. So I had to sort of struggle. Who's to blame here? Is it Curl? Is it Hyper? Is it both? Is it neither? or how do I actually, move forward so it was a bunch of fixes into hyper bunch of fixes in curl and bunch of minor adjustments in the test suite as well and. You also phrased it very nicely because one of the issues that you created in the hyper github repository was titled what's the maximum header size limit and knowing that it comes from you you know exactly that what you're aiming for is kind of an explanation as to why it is like this or why maybe it is also undefined behavior for all we know but i really like that you were very open for suggestions and also didn't really impose anything. Yeah and it's kind of and that boils down to that i i run into this sort of, awkward situation for me that you know i try to make sure that curl and lib curl has a situation where I know exactly how it works, I can document the limits, and when it runs into this, this is returned because blah, blah, blah. But when I then use Hyper, for example, that has its own mind and its own limits and its own decisions on what is right and what is wrong, then suddenly things are different. So when I build with a native HTTP, I have one limit, and when I build with Hyper, there's another limit. And in that case, I had this situation where I actually... Long story short, if you send an endless amount of headers to a client, right? You have a malicious server and you just send headers. Header one, header two, header three, header four, you know, and it never ends. That's a lot of headers. At some point, you have to just stop and say, I refuse to accept any more headers. It's stupid. Something is wrong. You're trying to take advantage of me and return an error code. At what point should you do that? And then I made a limit in libcurl because I have to stop because of just memory restraints and, well, and Hyper makes a different choice and they stop at a different point. So that's why I wanted to understand if there had been any clever thoughts or reasoning behind their limit. And I know there's been a discussion in that issue afterwards, because others have run into their limit, which seems to be much lower than my limit. A reasonable way to solve it might be to introduce a configuration or a setting for it. Maybe, but I've come over time to be more restrictive about options because people won't know what to set it to anyway. Anyway, and then you have an option that no one knows exactly what to set it to until you run into that. And then you can bump it a little bit until the next time you run into it. It's really difficult for anyone to have a setting for it. So yes, I agree. Settings are good, but it's also, and also it makes things complicated to add settings for a lot of things because you drown in settings and then you never find the setting you actually want because there's going to be a hundred. I already had that problem in curl that we have so many options that you won't find the option you want because there are so many other options you don't care about. In my case, I ended up looking at the browsers. What kind of limits do the browsers set? Because I imagine that very few users can survive. I mean, very few users actually have servers sending more headers than, popular browsers accept so if the popular browsers don't accept that many headers i could put the limit somewhere in there in that sort of ballpark as well that's what i went with. Yeah. Because you also catch two birds with one stone on one side you aim to be pragmatic here and maybe have sane defaults which would work for most people and on the other side what people expect sometimes is what the browser does. Exactly and they want to make they want to use our tools to do the same thing they did with the browser. So having more strict rules than the browsers, that's usually a failure because then people are going to complain. I could do this with Chrome, but curl says blah, blah. That's never good. So I want to make sure that we can do it. If you could do it with Chrome, you should be able to do it with curl as well. If it sends you 200K of headers and Chrome accepts that, then curl should accept it as well. And is the reverse also true? Did you find any issues with curl when integrating hyper support port and rust in general. I found a few things or i think i've been keeping to find a few things i think if in particular i think i've made the curl HTTP parser a bit stricter because i realized when i ran a few test cases or i don't remember how many but test cases and then i realized that the hyper parser is much more strict when it parses some of the HTTP details i think status code i I don't remember exactly. There were some things that at least I ran test cases and curl said, oh, this is fine and hyper returned error. And then I realized that, wait a minute, why is curl saying this is fine? Because it's really not HTTP compliant. It's actually weird status. So I decided then that, meh, I should make the curl parser more strict as well. And then made it so that curl would return error in the same way that hyper already did. So in that way, I think it was a benefit for me to make sure that, oh, I sort of just opened my eyes because curl had been doing that for 20 years already. And then just made me think that why is it accepting a two-digit status code when the spec says it should be three digits? And I made it fail if it's not three digits, like Hyper did. So things like that. I don't remember the details, but I remember there were a few other things as well that just made me realize when I could try with hyper and native, hyper native, and then sort of realize the differences, and then figure out that some of the things that Curl did natively really wasn't making much sense. It should just be a little bit more. Different than i had it before so it has been a sort of an improvement to the native parser as well yes. And it's also not a breaking change since it doesn't change this curl api it just changes the parsing outcome. Yeah the only the only risk is that there were actually two users you know one in three one in a million that actually gets a two-digit status code you know that's always a danger and that's always that's one of the sort of that's the reverse issue that i've had with with hyper that they are they've been too strict about things because you know the spec says something yeah yeah the spec says it should be like this yes it does but you know out in the real world there are two users who deployed something in 20 years ago, and that's not compliant, but they're still in use, right? That's sort of the collision, the real world versus the spec. And that's a complicated thing because then you need to make up your mind. Who's right here? Should we adopt according to the world or according to the spec? And then that's really difficult. In my case, I often have a way that we already work, right? So I want to make sure that we keep working like that so that we don't break. Like if it worked with that weird case already and people are using that weird case, I don't want to break it. Right. But if nobody is using that weird case, then we shouldn't accept it. So it's not that easy to figure out. Rust has this notion of soft breaking changes where sometimes it would be technically a breaking change. But since they run it on all the crates with a tool called Crater, they can ensure that it's not used... Very rarely used in the real world. Right, right. Yeah, it's difficult. It would be nice to have such a tool for Curl, but I mean, no one will tell you when they use it. Yeah, and really you can't know because in Curl we have this constant dilemma that people have really, really long upgrade cycles. So sometimes it's actually quite common that someone shows up in the mailing list and say, hey, I have a problem with this curl version that, you know, 14 years old and does this, you know, it's a wonder that it still works, right? And then say, well, I only have this issue. I don't want to upgrade because I only have this tiny issue. Or the reverse. Sometimes they say, well, I did the upgrade from, you know, the version 14 years ago to the current version. And then they discovered a tiny little change in something like that. Maybe I no longer accept two-digit status codes. They have to be three. And it doesn't work for our use case because blah, blah, blah. And they've been living under a rock for 14 years and worked with that weird case, told nobody. And it worked with that version that we shipped ages ago. And now suddenly it doesn't work because at some point during those 14 years, which did a minor change, nobody noticed. Nobody cared about it because it was spec-compliant. The API and API still work the same. But a user eventually figured out that it broke something somewhere. So it is a challenge. And then, of course, again, who's right and who's wrong? Perhaps one more question about hyper integration. Recently HTTP2 support for hyper was disabled and there is a pull request, but apparently it seems like there is a redesign that needs to happen for re-enabling it. Can you still remember some of the details of what is missing there? Sure. So Hyper supports HTTP 1 and HTTP 2. And I think they have experimental HTTP 3 support even, which I think is going to add even more challenges for me to support that. But I've sort of not gone into the HTTP 3 one with Hyper yet. We have four backends for HTTP 3 already in curl. So the HTTP 1 and HTTP 2 one turned out to be a problem that I had misunderstood how basically where the storage and how to manage HTTP 2 requests is. HTTP 2 can do many streams right over the same physical connection and and curl of course supports vp2s and you can also do streams over connections but so it was mostly a matter of me having misunderstood exactly how to use the hyper api to do streams over a single connection so when we when i tried to do more streams over the same connection well the life there was a lifetime setup issue i don't remember the exact details but it turned out to basically i used it wrong and stored the data in the wrong place in curl so i had to sort of back that out and and sort of we have to come back to that and i have to sort of dive into it and understand it again how the hyper api actually because i thought i used it correctly but apparently i had misunderstood it so i need to, go back and figure out exactly what they what the requirements are for the api to do streams streams properly for it to be too. Yeah and i guess this symbiosis helps both sides see rust curl hyper like the other crates everyone profits in the end. I think so too as long as we just cooperate and you know we're open and, for i mean and just make sure that we try to move forward and ask each other how how things supposed to work how do you think how do you do this in your project and sure i think it's that is just a matter of time. We will get there eventually. Overall, would you say your Rust experience was good so far? Would you make the same decisions again? Were you happy? Well, I think I'm content. I'm happy. And I think from my point of view, I think I'm happy with my way of solving, or not solving, rust into curl by making sure that we can build with rust components, rust backends, or replace parts of curl with rust-written alternatives. And I imagine that going forward into the future, we could do so with other components of curl, possibly replace other backends or introduce new ways to do backends that could be written in Rust. So over time, maybe we could have even more Rusts in a future curl binary. Do you have a favorite talk or blog post on the topic that people who want to learn more can learn more about? I think the best way is to just find my YouTube channel and search for Rust. I think I've talked about it a few times. We will definitely add it to the show notes. And yeah, I also think that this would be a great resource. Source i also like your blog posts so anyone who's listening do check out the blog posts sometimes they are even hilariously funny i remember the one i guess you have an entire blog post about you to like something like i could write curl in three days or something like this or on a long. Weekend yes yeah that's one of my favorite quotes. And you collect those posts i think this is great also great marketing. Thank you yeah i think it's fun and for some reason it's so easy for people to just be a little bit dismissive and say well it's just an HTTP client i could rewrite i could write that easily in a weekend and yeah i think i think people actually honestly think they can and maybe they can in you know in some very limited restricted way you can make an HTTP client in a weekend, depending on conditions and things. But I think Curly is a little bit more than that. I've been doing this for 27 years now. And I don't think that there's a reason to stop now. No, no. I mean, it's nowhere near an end or a closure. It's rather the opposite. It's more More activity, more development, and more speed in everything now than ever before. How can people support the work? If they wanted to chime in, what could they do? Well, there is everything. It's just wherever you want to participate, right? It could be working with Rust stuff. It could be polishing documentation. It's understanding the build or working with the CI or just figuring out sort of how to make sure that the test suite runs better. So that's really where I often, just try to encourage people to join in and do what they think is fun or where they think they find a problem where they want it to work better and go with that because having fun is important, when you want to get something done. So if you're interested, since this is a Rust podcast, so if you're interested in Rust and one of these Rust components and getting it to run better in curl, that's an excellent opportunity Because we're certainly not overstaffed when it comes to people working on the Rust backends in curl. So there's, as we mentioned, there are test cases to make sure that they work better. And they're sort of, we need to upgrade to the latest Rust, rustls API. And I'm sure that there are flaws to work on in Rust related things in all of these backends. If anyone is interested, check out the show notes. We will make sure to add a few links there to get you started. And if a pull request or two comes out of this, that would make me extremely happy. Now, it has become sort of a tradition around here to ask one final question, which is, what would be your message to the Rust community? If you could say anything that is on your mind, if you wanted to dedicate a message to everyone that uses Rust or is considering to use Rust, what would it be? I think I've just reframed from that. Join in, make things happen, have fun. This is the one thing which is my takeaway from your last FOSDEM talk. On every slide, you had the words, have fun. And as someone that also maintains some open source code, not as popular as yours, but I understand that sentiment. This is, I guess, the core part because if you have to do something for months and years if you don't have fun doing it you can't get there. No no i'm absolutely sure that as long if it's a spare time project or something you don't do full time if it's not fun it's not going to be done right then you're just going to not then you're going to do something else that is fun so yeah i think it's important and that's why i always try to encourage people when they want to participate join you know what What should I do when I join and do something open source or do something Rust? Do whatever you want to do. Do something that you think is fun, you think needs to be done or could be fun to have happen or you're missing or you think would be cool. That's where you should at least start out with. Daniel, I have to thank you for your time, for taking the time to do the interview, and for being such an approachable person someone that is so well known and popular you're still down to earth someone that you can talk to and approach and you it is very very fun to interact with you so i had a lot of fun and i have to thank you for that. Well thank you it was fun being here. Rust in Production is a podcast by corrode and hosted by me Matthias Endler for show notes transcripts and to learn more about how i can help your company make the most of Rust, visit corode.dev. Thanks for listening to Rust in Production.