Here's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. I'm Matthias Endler from corrode, and today we talk to Alexey Milovidov and Austin Bonander from ClickHouse about oxidizing databases with Rust. Thanks to both of you for taking the time for the interview. Can you briefly introduce yourself and ClickHouse, the company we'll be talking about today? I'm Alexey, co-founder and CTO of ClickHouse, the company behind the best analytical database, open-source analytical database, that is also named ClickHouse. The company does the service for this database. It has thousands of customers and millions of lines of code, both in C++ and Rust and in many other languages. Hi, my name is Austin Bonander. I've been a Rust community contributor since before Rust 1.0, circa 2015. I created SQLx, co-authored SQLx with my launch badge, coworkers, Ryan Leckye, etc. And since September, I've been working with ClickHouse on their integrations team, building their Rust client. Thanks for taking the time, you both. and I don't think that we need a huge introduction, for ClickHouse, but maybe can you set the stage a little bit? Why is it such a big deal? Why do people use it in production? Why do they trust their most valuable data to it? The story of ClickHouse started in 2009 when I was working in a different company, and there was a task to build a system similar to Google Analytics. The system that can take the traffic from the internet, from a lot of websites, process this traffic in real-time, generate reports. It's not unusual, there are probably a hundred of similar systems. Not many work at scale, at similar scale. I tested many systems, but nothing worked. it. And I was just an engineer, just IC engineer in the company, but I was responsible for data structures, for data processing, and I decided maybe I can make a prototype of columnar and data structure just to generate these reports. I made this prototype in 2009, we put it in production, but it was not ClickHouse. It was very tempting to try to generalize this prototype to make something bigger, out of that so maybe I can experiment maybe I can try more and I started writing code, of this new database and after a few years we put it in production for just a single system it was in 2012, then it was used in many other projects in the company and in 2016, I convinced the company to make it open source. Now, it was a success, obviously, because otherwise we wouldn't be here today. And I wonder how much of that success was thanks to data structures and clever design, or thanks to C++, the language that ClickHouse was implemented in, or thanks to economic factors or environmental factors that were sort of outside of your control, like it was the right time for such a database. What would you say in hindsight was the factor that made ClickHouse so much more successful in comparison to the competitors? You said environmental factors. And yes, this is the main point. And actually the main point is production. The needs of, specific production workloads. If we developed a new database isolated in a vacuum, it never would be as successful as ClickHouse, and, we had to optimize it. We did not optimize it just because I like to write fast code. Yes, I do like writing just code, but I would not do it to that extent if there, was no need for particular production workloads. And if you look at different trade-offs inside ClickHouse, inside its language, dialect, inside the way how it operates, you will notice that all of these trade-offs, they are very practical. All of these are from practical needs. What was the first component that piqued your interest in using Rust at ClickHouse? Initially, it was a question more like, what if? What if we don't take it as a plan, as something that we have to do, we don't have to introduce Rust in ClickHouse, but maybe if we will need something, if we will need some library, maybe we can start and try to see what will happen. And we decided to take a very isolated library, just one hash function, and look what it will take from the build system perspective, to make it integrated, to make it built together with C++. We did it, we found a way and we found a lot of trouble. There are different ways to integrate rust into an existing application one would be to make rust the driver and go through cargo and i don't think that you, picked this route because if you have a large c++ code base the reverse is actually often preferred which is using cmake and calling into cargo how did that story go how did you, integrate the build process of rust Initially. If I recall correctly, we used Cargo, and we used external repositories, plus CMake config files that just call this Cargo and take wrapper code and link all of this together. But there are a few difficulties. One difficulty is that. We have to make it compile locally and we also did it with small modifications there is a local mode if i recall correctly i could be mistaken but we used something like that, but then we also needed to compile the code in different build configurations, and some of these build configurations were not if, again maybe i will miss some details but And some of these configurations were not available, like, we had to compile Rust code also with Memory Sanitizer, just to track which memory was initialized, although maybe it was not necessary. But back then it required a nightly toolchain for Rust. And also we had to patch a few libraries to remove, replace Panic to something else in certain libraries we did these patches and, all of our code is compiled from the source so we also made it for us for reproducible builds and to for offline builds, and to keep this possibility to rebuild everything as on demand as we need. I remember that i saw a talk where you, talked about this briefly and the tool that you used back then was called Corrosion and it's not affiliated with corrode but is that the tool that you used in the beginning to bridge the roster in C++, build time? Yeah and I think it is the same today. I can check the repository. I think we still use it. Yeah it's still used. I'm looking at the CMake files right now. Well, if it's still used, then it means it's served you well. Did it stop, or did you run into any issues? It's probably been a while, but if there were any gnarly bugs, then you would probably remember, right? We had to update it a few times, but I don't recall what was the reason. There was some trouble. We updated it, and the problem solved. Now, the other thing that often becomes a problem, And you kind of alluded to it earlier is, the fact that C++ uses exceptions and Rust uses basic error handling and panics as well. So how did you make sure that the Rust code was sort of safe? And on the other side, what do you think about exceptions versus errors in general? I know that a lot of C++ developers are fans of exceptions, and I would like to hear what they think about Rust's error handling. First of all, we don't guarantee that Rust libraries don't do panics, in the same way as we cannot guarantee that a library in C or C++, don't use an assert in a place that's actually reachable at runtime. Because Asserts and similarly Panics are mostly for places that will indicate a bug in the code but if code is bug-free, they just should not happen, but they do happen. And to find all these cases, we use fuzzing, and we already employed a lot of fuzzing when we started with Rust, so it helped us immensely to find many of these cases. Then we just write to the library author and they fix it, or we just patch and don't wait. So not a big problem. Do you run assertions in production? Yeah, we do a lot of assertions and also so there are three different types of error handling. First, exceptions for something that is normal but does not happen often and does not happen in a happy path. For example, For example, connection loss or even something like not enough memory It all goes to exceptions, So exceptions propagate in the stack and provide a high-level error to the user. With full stack trace, with a lot of information. I think it's a great tool. I like exceptions. Another way of error handling is just assertions. When we check invariants in the code that should not be broken, we use assertions. And we run release plus debug and sanitizer plus debug plus assertions in all our test suites in CI. So we find these cases and fix them. And there is also a third case when we, use exceptions but designate them as logical errors and modify the build. So in the debug build, these logical errors behave in the same way as assertions. They terminate the application but in the release build they give us a chance to catch this error so they are not disabled, But we look at the reports and fix this error in the same way as we fix fired assertions. So in regard to panics, I remember when Alexey was soliciting feedback about his P99 slides and it got to the topic of panic safety. I remember this blog post I saw called NoPanicRust, a nice technique for systems programming that talks about using linker tricks to actually statically guarantee, that panic is never called. I don't think we've tried it yet, and it would be really interesting if it was somehow integrated into the compiler, but it's potentially something we could try. I'm afraid it could be non-practical, because how do you do debug tracks in the code otherwise? And these traps are not necessarily supported by the type system or borrow checker or everything else. They could require complex mathematical proof and you just assert, a certain fact in the code just for yourself, so the linker will see that panic could happen, but you just do it because you want to catch if this mathematical statement, logical statement about the code does not hold. And I think that is fair because this does rely on the optimizer eliminating that call chain entirely. So if it can't statically eliminate it, then the trick's not very useful. I think it's possible that, you know, maybe we could use it for certain components where we're building, you know, self-contained dynamic libraries like the Delta kernel RS. But, yeah, I think it's just more a question of, you know, how do we want to handle asserts across an FFI? And it does seem like the story is not quite there yet. What does the Delta kernel do? I think they still have asserts planics. And I don't mind if a library crash, if it's a certain bug that is just discovered, it's a new bug and we have to fix it. You mentioned Delta Kernel RS. What's that for? What does it do? It's a library to access tables in the Delta Lake format, which is a format for data lakes, tables in external storage accessible for different query engines. And is that part of the ClickHouse dependency graph, or is that an external component? It's part of the graph, it is linked, and part of the distribution, part of our static binary, we do static linking. Right, did you ever run into any issues with that linking part? Did you run into problems with the Rust FFI boundary? We do, still. And we still have some problems we have to fix. And I greatly appreciate help from Austin, although we did not yet fix that particular problem. It's about the way how our code and how So the library in Rust find OpenSSL, So we use a particular version of OpenSSL, We link it statically, This particular version has to be that version Because, we have a build certified for FIPS compliance Particular compliance for cryptography and so on For some reason, when we link this Rust library, it also wants to find OpenSSL, but it finds it from the system. It tries to link it dynamically, but even if it does not link dynamically, we end up with different versions of OpenSSL. And Austin found why exactly this happens. Yeah, Austin, if you could explain, please. Yeah, so if I understand, this is about, I was asked through my chain of command to kind of take a look at this build issue they were having in the core server, because it's not normally part of my wheelhouse. And they were trying to enable Delta Lake testing with mammary sanitizer, and it was failing. I think, actually because it was trying to link symbols from ring specifically. And through the Delta Lake dependencies, the dependency on the object store crate, it uses the ring crate to, calculate RSA signatures for authentication to Google Cloud Storage. And I think at some other point, it also uses it for SHA-256 hashes. And because ring brings in compiled assembly symbols I don't understand exactly what the problem was as far as, why it was failing to build but it was essentially the hard dependency on ring was the problem and that code we've the code is flip-flopped back and forth between using ring and, the native TLS feature I think because of the OpenSSL linking issues. But the current blocker is that this hard dependency on ring is buried in a transitive dependency, so we then have to go patch that dependency. Yeah, it makes it clearer. So the problem looked that way. In our continuous integration system, tests under memory sanitizer produce an error use of uninitialized memory. And we think, okay, now we have heart bleed. We have to stop everything that we do and fix that error, because our code has to be memory sanitizer clean so we stop our work start to investigate and, we find that it originates from Rust and we think okay we have a memory safe language why do you have a heart bleed, then we figure out that it is a code not in Rust but in assembly. And obviously, memory sanitizer, to work, it has to annotate all the code at compile time Annotate all memory stores, So it knows where memory is initialized, where it is not initialized And when there is an assembly code, you have to instruct memory sanitizer that a particular memory produced by this code was actually initialized, okay so now we have to patch this ring library and problem solved but, then we start to think why this ring library is used in the first place if it does the stuff that in our code is already done by openssl, so it is not only a problem with annotation for memory sanitizer inside this library that could be fixed very easily. It's a problem of overall composability of components. You compose two components and some of their dependencies don't compose together. And so in fairness to the object, the maintainers of the object store crate, which actually is under the Apache Arrow project, if I understand correctly, they have an open pull request to make the crypto in that crate pluggable so that we can swap it out with an implementation based on our OpenSSL fork. It's just right now, as far as I understand, it's blocked on them preparing their next backwards incompatible release, minor release. And so there's not really a clear timeline on it, but we're just kind of blocked on that, not really sure where to go from there. Yeah, it's a very common issue in the Rust ecosystem, the fact that we have all of those smaller crates and then transient dependencies, which are in and of itself a good thing because you have, more variants, you have more diversity, you have more options. That's great but then again you have certain crates which are very central to the ecosystem right now which are sort of everywhere and it's it's hard to switch them off entirely or make sure that you have a feature toggle in every crate that you depend on, otherwise you end up with this big tree of dependencies and that makes things a lot harder both in terms of maintainability supply chain security, and reproducible builds even. So it feels like for an organization like ClickHouse, the challenge isn't just the language, but just the ecosystem. Yeah, that is true. And not a single ecosystem. It's the fact that you have to combine ecosystems together. And when you have two different cultures, I would say, it becomes more complex. You heavily rely on reproducible builds, do you? We have made our builds reproducible for the C++ part, and for some reason not yet for the Rust part, so only because of Rust our builds are not entirely reproducible, and I don't know why. I can say about problems with C++, not many, like hard for the past, compile time, date, usage, and so on. It seems like there's more work for you to be done, Austin. Austin speaking of which you are known for being one of the maintainers or maybe the maintainer of SQLx right now, and SQLx supports a lot of databases which databases does it support in particular and what about ClickHouse specifically. Yeah so SQLx i am the primary and sole maintainer of SQLx at this point, everyone else has just moved on to other things. I was the only one that kind of stuck with it. So we support right now MySQL, Postgres, and SQLite, and a lot of third-party databases that talk one of those protocols. We get a lot of issues for that, and we try to address it when we can, but it's a scalability issue. And so ClickHouse is a interesting and yet kind of weird, idea of adding it to SQLx because it is an analytical database so it has somewhat different concepts than MySQL and Postgres for example like row major data format versus column major, and the fact that ClickHouse doesn't really have transactions which are you know kind of baked into the SQLx API. Correct me if I'm wrong, Alexey. We do have transactions in beta, but I'm not sure where those are going or if they'll be generally available anytime soon. Yeah, that is true. We have transactional semantics for certain statements, but we don't have yet multi-statement transactions. It's in development. So in creating a ClickHouse driver for SQLx was actually one of the things i pitched in my cover letter in my application to ClickHouse this was one of one of the few jobs i actually, even applied ever applied to because the rest were cold offers, and you know one that i actually felt like i really should write a cover letter for and i kind of poured my heart out into it and one of the things was that i pitched ClickHouse support because funnily enough we have a, really old issue when we were kind of listing out you know databases we should support we just created tracking issues for them and ryan actually created one for ClickHouse and it's really funny because we have a, comment from Alexey in 2022 saying you know here's the protocol we'd love to see this, so it was really cool to see that story kind of come full circle very serendipitously with just you know me by chance seeing this job opening, so that's something we're really excited about and it's still currently in the planning stages but definitely something that's going to be happening sometime in the future. Transactions would certainly be one reason for making click support non-trivial in SQLx? Would there be anything else or is that the main blocker? So besides transactions, there is the difference, like I mentioned, between row major and column major data. ClickHouse does have row major formats, and that's actually what a lot of our clients use, because that's the most intuitive thing to map to a structure. You get a row, you map it to a structure, or you want to insert a structure, map it to a row, just send it to the server. But in talking with a lot of clients and users of the Rust library, we're finding that they are actually running into issues with the overhead of the row binary format. And so they would really like to do that transform to column major format on the client side and get all the optimizations that Alexey et al. Have put into the ClickHouse native format with the Delta compression, sparse serialization, and in general, providing the server with a data format that it can literally just shove straight to disk without a lot of song and dance. So figuring out how to design an API in SQLx for that is definitely a challenge. And it's starting to look like that maybe... There should be a whole new subset of the SQLx API or maybe a whole new crate, just designed for analytical databases. And then the final one is integration with the query macros. Right now, the ClickHouse API doesn't really have a way for us to send a query and get metadata back. It does through, to be fair, it does through the TCP protocol. But the Rust client currently uses the HTTP interface which are just regular Rust calls and through that there isn't really any way for that. So then there's kind of a bunch of blockers assuming we build the SQLx driver on top of the ClickHouse Rust client which makes a lot of sense, right? Then there's a lot of stuff that needs to be kind of upstreamed into the Rust client before we can start building the SQLx driver. The hard part of maintaining such central libraries is, making sure that abstractions are really tight and focused because, I mentioned briefly to you, Austin, that all of my clients use SQLx in production and that means it's a huge burden on the maintainers to make sure, SQLx is both supporting all of those use cases but that the abstractions of SQLx don't leak into other databases it supports it needs to support multiple things and if you add columnar database support, and row-based database support is already in there i can see some conflict if you don't know exactly what you're doing. Oh yeah absolutely i mean a lot of, the effort as as a maintainer of SQLx is kind of you know trying to explain to people like hey that's not, you know when they open pull requests cold all the time with bespoke api designs and i'm sitting there looking at it and going you know this doesn't really feel like it fits. And so having to sit there and explain that to them when they're just trying to resolve a blocker, and I understand that. But it's my responsibility to maintain a cohesive design and especially think about other use cases. Discoverability and guarding against misuse is a very Rust idiomatic thing and not something a lot of people think about. And one of those things is maintaining not just API stability, but behavior stability. There's a blog post I've been meaning to write for a long time that I have the title in mind, The Most Forgotten Law of Programming. And it's a law called Hyrum's Law. I'll just read it off. Puts succinctly, the observation is this, with a sufficient number of users of an API, It does not matter what you promise in the contract. All observable behaviors of your system will be depended upon by somebody. And that has proven time and time again to me maintaining SQLx. Because I'll make a change, a seemingly innocuous change that doesn't break, any APIs and put it out in a point release. And then someone will say, hey, this actually broke things because it turns out I was relying on the exact outputs of this API. Or, you know, you changed. One thing I was worried about recently was changing the spelling of a field on a span we emit in tracing. I think it was, there was a typo in it and I was like, is this a breaking change? Because someone will have put a filter somewhere in Google cloud looking for that field. And if I rename that field, even though it's not part of the API, if I rename that field, it's going to break that person's filter. And then they're going to come raise an issue because their, their monitoring doesn't work anymore. Um, so that's something I'm always thinking of. Thank you. That's insane. Yeah, it's named bug compatibility. And also, it means edge cases compatibility. And this is why in HTTP there is still a header named referrer with one R, or you can say two R. And everyone has to keep this mistake. Yeah. But you need to have a lot of foresight to think that far ahead. That's great. I remember this XKCD comic where someone complained that Emacs changed the way the spacebar worked and they found a bug which, would spin up the fans and then this person used that feature, quote-unquote, to keep themselves warm in winter and they complained about that when they fixed that bug. I remember a very interesting case when we optimized the performance of certain ClickHouse queries, and someone complained. And when we figured out what was the complaint, the problem was as follows. So they have a system, it makes a query to ClickHouse, and then it makes a query to some other system, like Mongo or something else. And we optimized queries in ClickHouse, and they did not have any schedule for these queries, they just did it in a loop, like do 10,000 queries in a loop, and that's it. But along with every ClickHouse query, they also queried another system like MongoDB. So when we optimized ClickHouse, their service just gets downtime. It's funny how these things sneak into production-grade systems, but yeah, that's the reality of our job. Now, the obvious elephant in the room is that there is another library for interfacing with ClickHouse, which is clickhouse-rs. And to a bystander, it might not be really obvious what's the huge difference between SQLx and clickhouse-rs. What's your take on this, Austin? How do they differ in API design? And what's maybe non-ideal about that situation and maybe about clickhouse-rs? Sure. So clickhouse-rs is the official Rust client for ClickHouse. It was not created internally. It was actually adopted from an outside contributor. And that happened before I joined. So I actually took over maintenance of it. And there are a few design decisions that immediately stood out to me that I would not have made the same choices. I can see why the author made those choices, but were... Somewhat dealing with the fallout of them. And one of the major ones is the reliance on serde for serialization and deserialization. And the problem there is that serde's data model is very much designed around JSON and similar self-describing formats. And it's also mainly designed around just serializing and deserializing regular plain old Rust structures without really any special data types. And one of the problems we've run into is when we want to serialize something like chronos datetime type or times offset datetime type. They have default serde serializations that are not compatible with our wire format. Because Time uses their own subset of ISO 8601. I forget what Chrono does. I think it's similar. But for ClickHouse, we want to serialize to our wire format, which I forget exactly what it is, but it's not ISO 8601. And so we end up having to define a bunch of adapter types and or adapter functions where you do, use the surday with attribute to map those and it becomes a huge learnability burden on the user and just, really frustrating to deal with so one of the big plans i have for it is to. Not rip it out initially because i imagine there's you know thousands tens of thousands of lines of code already built against that, but to supplant it with a bespoke trait system derives that are designed specifically, for ClickHouse that map directly to ClickHouse native data types. And that will provide a much better user experience because it's a similar decision we made really early on in SQLx because I can see why they did it. Why the author of the ClickHouse Crate did it is, you know, you seem it seems like you get so much out of the box for free. You get the derives, you get the traits and all that stuff. But there's so many baked in decisions to serde that, when you're mapping to such a sophisticated data model as a database that has all of its own type encodings, it doesn't really work. And so that's why in SQLx, we ended up going with our own bespoke trait system. And it's going to be basically the same thing for the ClickHouse crate. Right. And I would guess right now, serde is very much baked into clickhouse-rs. It's not easy to decouple those parts. Yeah, we can mostly just replace the entry points. There's only a few types that are directly dependent on serde's traits, so it's just going to be building a whole new API alongside that, and then maybe trying to deprecate and rename those methods to point users towards the ones using the bespoke traits. And then we'll keep the serde integration as a legacy code thing and drop it sometime in the future. That means essentially you rebuild the thing from the inside out. Absolutely, yeah. That's pretty cool that you can pull that off, it's off to you Alexey you must be really happy to have such a rust expert on the team and to me it sounds like rust is here to stay correct me if i'm wrong at ClickHouse, was it worth it did rust already measurably reduce defects at ClickHouse. Defects in production, I would say not yet. It's still mostly used for new features, for not the most important features, but it is very close to do, to serve this role. I have a few places in mind, a few places where just replacing a C, it is not even C++, a C library with Rust will solve a lot of problems. And in many cases, it is not a disadvantage of a language. It is more like a consequence of a cultural reset that Rust does for engineers. So just to provide some analogy, let's imagine that there is a library in C written 20 years ago, by grey-beard scientists. And if you use it in a slightly different way, not like this grey-beard scientist imagined, you get all sorts of segmentation faults, uninitialized memory grids, memory corruptions. Errors you cannot make any sense of, and everything like that. And now there are Rust engineers that make a lot of noise, and everyone hates this noise, this drama, and they say, we rewrite everything. And they actually do it, and they actually produce a higher quality code okay, back to our examples so there is a library named H3. For hexagonal grid on earth, to produce a coordinate system hexagonal coordinate system it's quite a good library but the first version was written by this sort of bugs it did not do bounds checking, there were pretty simple mistakes leading to memory corruption and we patched this library and then this library improved and now it is maybe perfect. But there is a new library named h3o so not h3 but h3o in rust, and also it is faster on benchmarks and it is faster not because rust is faster than c but because these new engineers, were capable to allocate more time for optimization optimizations and they thanks to rust it was possible to do these optimizations with less effort, more control, so more guarantees from the compiler, borrow checker, and so on. And I tried to replace H3 with H3O, and most likely in the nearest weeks it will be in ClickHouse, in the master branch, and then in the release. Wait, it almost felt hypothetical, but what you're saying is it will in fact be in master and it will be released. Yeah, so not yet. Currently we have a pull request. It passed the tests. Surprisingly, because we have such a huge CI that passing the tests for any new libraries, it almost never happens. Almost always we found a lot of bugs in every new dependency. But hopefully, next month or the next release after that, we will have it. And the assumption is, it was a good idea to rewrite that part in Rust, since you have reason to assume that this will be more robust than the prior version. Yeah, exactly. Maybe there will be no actual difference in bugs, just because the existing library already fixed it. all the bugs, but we will have a little bit more confidence. Yeah. The other part about confidence and maybe, trust in a language is that it's not evenly distributed in an organization some people, fully trust rust as a language already and maybe others they are just at the beginning of their journey, and knowledge sort of permeates across an organization over time it's not like everyone has the same information at all times, and and how does an organization deal with this when rust expertise is not spread evenly when do you make the jump when do you, help the team make the migration over to rust when would you restrain from that and maybe keep people from using rust and maybe keep them on the c++ side. For the clickhouse core team which develops the database server it is a blocker we have only a few people maybe just like five people, inside the departments department of 50 c++ engineers just five ten percent, can write rust and no Rust. And it is a blocker for Rust adoption. I still don't see any reasonable path, to actually migrate a large part of the codeways in Rust, but it is still a good idea to use it for dependencies, to allow such dependencies, and I think it will be proportional to this ratio. Maybe 10% of the volume. Not sure how to measure, but maybe it will be this way, like 10% in Rust, 19% in C++. For today, I don't see a path how to rewrite everything. Even with AI agents, it will be extremely expensive, extremely unlikely to do it for one and a half million splines of code. I get that and this is also one reason why rust is great because you don't have to rewrite everything you can start as you say, very simply with a smaller percentage of your code base written in rust and then maybe gradually improve or keep it that way if you feel like the majority of the bugs have vanished but, the truth of it and the reality of it is also that, it's sort of what you also mentioned earlier is a cultural reset, and with that also comes, a higher burden on things like compliance and supply chain security because if you bring on an entire ecosystem and a new tool chain, what does that change or from a supply chain perspective, speaking of SBOM files speaking of reproducible builds and there seems to be, a country-wide shift towards higher security memory safety memory safety security, how would you think about this would you say that's a soft problem in the rust and c++ world nowadays or would you say, well, you still need to be cautious and there are still issues? It is mostly a solved problem, but not because it is 100% solved, because both Rust and C++ have their own difficulties. So for C++, supply chain is often less of a problem, but for a bad reason, just because it is difficult to add a dependency, it's very difficult to... Sometimes it's difficult to update the library, and you tend to use very limited, number of dependencies. And in this situation, you just enumerate all of this, you don't receive a lot of updates. Often you set up some scanners, but And to be honest, many of these scanners don't work really well. For C++ codebase, we use some of them. Sometimes they do detect dependencies correctly, sometimes they miss versions, sometimes they found dependencies that we don't have. But anyway, and for us, we have to manage with a larger number of dependencies. We have to set up the same license scanning, the same monitoring for new versions. We have to update certain versions quicker. So it is mostly solved. And there are many dependencies not in the runtime code base also, in the build system in in the continuous integration infrastructure and it, it is often even a higher risk so in in the continuous integration we have dependencies on a lot of docker images and a lot of python code And we can say that, the particular version of the operating system and our infrastructure in cloud service providers, it is also a dependency, and it is much wider in the continuous integration, and also a lot of risk is there. Austin, what's your take on supply chain security? It is definitely a massive challenge in Rust because of how much has been punted to the ecosystem, with the Slim Standard Library and everyone having to invent their own solutions to things. And we end up with a ton of competing solutions and incompatibilities in the ecosystem. tokio versus Async Standard, Ring versus AWS, LCRS versus OpenSSL. And you don't see this problem in a lot of other languages because they have these very fundamental components as part of the standard library. And I, you know, I very well understand that that was a deliberate decision made early on to keep a slim standard library. But I think going forward, we could be a lot more proactive about identifying things that should be standardized, and making efforts to incorporate those either as blessed crates, or into the standard library. Yeah. Would that mean we have some extension of the standard library a, mega crate of sorts which just pulls in all of the blessed dependencies that we use in many different applications. I mean there's been attempts at that in the past a standard x i think was a big one. Yeah by Brian Anderson yeah Yeah. And those run into maintainer bandwidth issues, right? Or, you know, essentially holy wars about what should be blessed and what shouldn't. And that's a tough call. I don't really have a, you know, a pithy answer for that. But I think it is something that we should be looking at, because especially like we ran into this in SQLx with supporting multiple runtimes and not having, standard like async read and async write traits. You know, I was tacitly promised that those were coming soon back when we talked about removing async standard entirely. And that was like three or four years ago. So, you know, it does leave me wondering kind of where things are going. I think it's a very hard problem to make a standard library that will be good for everyone. Imagine you get a regular expression trade, not trade, great, whatever, regular expression library in the standard. And just a few days later, someone like bernsushii2 or whatever, whoever, will make a better regular expression library. And everyone will use this regular expression library. Someone decided to add regular expressions to the C++ standard And no one used these regular expressions Because another library, is just better And that library in the standard is unusable And also a provocative question Do you want, a library for XML in the standard? And what, type of parser will it use? and will it support something? I don't know. Whatever technology that was popular 20 years ago, but now mostly outdated. Probably not. So it's a very hard question. I'm actually happy when the standard library is slim. And to bridge the gap, we have technologies to get us part of the way with Cargo Audit and Cargo Vets and Cargo Geiger and so on. Those are tools which surface issues in critical libraries and everyone can contribute and vet those dependencies. That's cool. That's really nice. And all of these ideas kind of carry us into the future. And maybe in closing of this podcast, I wanted to talk about the future of Rust, not only at ClickHouse, but just in general. What's your gut feeling about it? And because the other part that we discussed across this entire interview was the fact that both Rust and C++, are here to stay and they need to work together. And a big part of it is FFI and tooling in that area. And there doesn't seem to be a single solution right now. We have C-Bind Chen, we have CXX and CrewBit and many other technologies. But that's all on the library level, not on the language level necessarily. What's your take on the future of build systems bridging Rust in C++? I don't worry about this future much, because today most of the boilerplate is solved just by a single prompt to your favorite agent. And it mostly works and for this boilerplate for this glue code wrappers it is reasonable to use this generated code so so far i'm quite confident. Yeah it would be i think it would be really cool if there was maybe more first party support you know you look at zig which is a i think a major competitor to Rust right now. And they just went ahead and built a whole C compiler into Zig. And one of the cross-compiling options for Rust is Cargo Zig Build to just go ahead and use Zig's C compiler, which is a little wild to me to just bring in a whole other third languages tool chain just because it has a higher quality approach. So something first party would be very cool. you know as Alexey, Alexey's perspective is a lot more about you know making things work right and, we have we have options i i just kind of you know consider them to be suboptimal i guess. There's so much more we could talk about but we have to come to an end and traditionally the last question is always what's your statement to the rust community. So yeah it seems like rust is definitely at a kind of a critical juncture where it has, it's positioned where it is in a very strong spot but it there's also the possibility that an app a language that's better designed for say applications versus systems could kind of come along and eat its lunch, and i think the biggest thing right now is it seems there is a definite maintainer bandwidth issue because you know we i hear about you know oh you know rust up doesn't have a dedicated maintainer, crates.io doesn't really have a dedicated maintainer because they don't have the budget, so i think really we need to see a lot more support of the rust project itself and try to resolve maybe some of these maintainer bandwidth issues, which would help push through all of this pending feature work that we're all kind of sitting here waiting for. And so I think support your Rust project is my sign-off. It's about the future of Rust. It has a great community today. It's quite trendy. But if we look at previously trendy languages, like C++ was very modern, popular, everyone was rewriting code from C in C++ everyone was happy. It was such a great language Java also solved a lot of problems many people were rewriting libraries to make them pure Java libraries because Java was the future, And it is inevitable that it will be something similar with Rust. It will accumulate design choices that are suboptimal, suboptimal libraries, obsolete things. It will accumulate its complexity, but it will remain a great language, just as Java and C++ are great languages, great tools for today's work. Let's hope that Rust will remain not too complex, it should not become like, I don't know, Scala or C++ It has already quite a bit of complexity, and I think it's enough it has a good niche. There is a certain risk that maybe it will be just a target language for AI-generated code, but I would not say it's a bad thing. It is kind of expected and it will also indicate its success, and I see it. It is already happening, and what happens in the future, I think it will be great. Austin, Alexey, thanks so much for taking the time to do the interview today. Thank you for inviting me, thank you for having me Yeah. Absolutely, it was my pleasure. Rust in Production is a podcast by corrode It is hosted by me, Matthias Endler and produced by Simon Brüggen For show notes, transcripts and to learn more about how we can help your company make the most of Rust, visit corrode.dev Thanks for listening to Rust in Production.