It's Rust in Production, a podcast about companies who use Rust to shape the future of infrastructure. My name is Matthias Endler from Corrode, and today we're talking to Anton Lazarev from Brave about building a high-performance ad blocker in Rust. Anton, thanks for joining. Can you introduce yourself? Yeah, I'm Anton. I live in San Francisco, and I work for Brave. Brave is a company that makes a web browser based on Chromium, and I specifically work on the Adblock engine. The Adblock engine that we use is completely written in Rust, and it's one of my favorite things about the company, and it works pretty well in my opinion. How did you even get started with Rust? What's your background? In high school, I was interested in 3D printing, and at one point I found a blog post that somebody wrote about Rust. And I was like, that looks interesting. Maybe I can use that to write some G-code, which is the control language that's used to talk to a 3D printer. I think I started by trying to make a parser for G-code and I was doing the standard string operations you would try to do in C or C++, taking indexes into the string to get letters and numbers out. That's clearly not the way to use Rust. And especially back then, And this was probably 2016. That was a bit of a mess back then. I wasn't too interested in Rust after that experience, but there was something about it that drew me back. And especially, I was also interested in electronic music. And so I wanted to build a VST plugin, which is basically an instrument that you can plug into audio software to make sounds and things like that. And I thought, maybe I'll give Rust another try. And I started looking into, let's see, what kind of UI libraries can I use for this, since most of VST plugins would have a visual interface. And the Rust UI space at the time was very underdeveloped. But one of the ones that I found was a library called Azul. And I started contributing a couple of things that were missing to Azul and the contributor was very helpful. He roasted my code a little bit, but that was super useful because I suddenly started to learn all the best practices. And eventually I felt pretty comfortable in Rust. I had sort of a working VST plugin. How did that shape your relationship to systems programming in general? Because I know that VST plugins are pretty low level. There are pretty harsh constraints. I think this especially was a pretty good project to do in Rust. Because for audio work in general, you need to have very fast code. Each audio sample has to be produced consistently with a low latency. And Rust is very good at that kind of code. So I really started to realize this is where Rust shines. And I started to be very interested in using it for every other kind of project as well. Can you even do allocations there? You could, but it's generally recommended not to if possible. So when the plugin starts, you get a chunk of memory and then you operate on that? Yeah, generally you want to leave allocation out of the audio thread as much as possible. You have very short time constraints for doing your computation and then the next loop starts. What are the implications of that? Yeah, the way it usually works is you get a buffer of 512 samples, let's say, and you fill that in. And that buffer size can be configured by whoever's using the plugin but you will get a fixed size buffer you fill it in and then you return it back with audio in it for the host to use those. Stories they take place i guess around 2017-18 this is where you learned a little more about rust you start your first project and then eventually you joined brave when was that. Yeah, so I started as an intern at Brave in 2019. And actually, I found the... Job posting on reddit on the rust subreddit of all places my my boss basically posted we're looking for an undergraduate intern who's willing to write an adblock engine in rust and it'll all be open source for brave browser and i saw that i was like whoa that looks like a lot of fun and so i did a little demo project as part of my interview and it seems like he liked it and i've been there ever since basically it. Looks like that job had your name on it because. They were. Looking for someone like you specifically an undergraduate as you said that's pretty cool why did they look for an undergraduate specifically why not just anyone with rust experience what was the background there. At the time the adblock engine was kind of the first component that was going to be built in rust for the browser and they weren't sure if it was actually even going to work well so i think they wanted to have a very low commitment let's see if this works if it doesn't no big deal that's. A very smart move from brave to say hold on it might fail we want to try it anyway but we give it to someone who maybe is fresh out of college or fresh out of university they maybe have some new ideas maybe they are enthusiastic about it and if it fails and the risk is pretty low, they might as well just learn a little bit about Brave and the company and so on. So we could repurpose them for something else if all else fails and it's a learning experience, right? Yep, exactly. I actually wasn't even the first intern to be working on ad blocking. There was one the previous year who did cosmetic filtering and they did it in JavaScript. So it was a pretty simple implementation. My initial project was to take that and move it into the block engine. In rust it. Reminds me of a story of mozilla they started to make the css parsing parallel and they needed three attempts they had two undergraduates okay so one undergraduate and one full-time employee that would try to make it parallel and failed and then the third one was successful it was the first one written in rust the first two attempts were in c++ as i remember. Yeah there's definitely something to be said about that approach. Okay so let's say it's your first week you started on this new project what now how do you even get started with such a task. Yeah so to be honest i hadn't really understood how adblocking worked prior to joining the the task that i did for my interview was to write a parser for adblock syntax and so i had to learn how to do that in the first place but it ended up being a lot of research on what is cosmetic filtering how do other ad blockers do it and so luckily we live in a world where pretty much every useful ad blocker is already open source so i could just dig into the code bases for those and take a look at what algorithms they used and it was a lot of reading some very convoluted javascript that made my head hurt but eventually I figured it out and I started translating that into Rust and I feel like the result was pretty nice. We were able to ship it while I was an intern and that was a super happy moment for me. Can you explain to the audience what cosmetic filtering is? Yeah, sure. So cosmetic filtering is basically anything that, is being hidden in the page or being modified about the page after the page has already been downloaded and. That is in contrast to what other sort of filtering. Yeah you would have network filtering where you could block network requests from happening in the first place but sometimes those network requests contain both good and bad things and you need to selectively choose which ones to hide that. Means your ad blocker at least the first iteration of it was purely user-facing, That's what you see, the rendered version of the page. You have a bunch of empty spaces on the page where ads used to be. But if you have cosmetic filtering, you can pretty easily get rid of those and make the page look a lot cleaner. But those two go in lockstep with one another. If you start to block one network request, you also need to update the cosmetic filtering. The thing about ad blocking is that there's a lot of variety in how pages structure themselves on the web. And so this all basically has to be done on a case-by-case basis. And the way we do that is with filter lists. Maybe people don't know, but all of ad blocking is based on tens to hundreds of thousands of lines of rules that say, on this website, block things that look like this in terms of the URL or in terms of what kind of a request it is, whether it's a script or an image or media or something like that. And hide things with the CSS selector, these specific sites. It's just a huge list of rules that are maintained by... People who do it for fun pretty much we actually employ one of them ryan brown at brave he's awesome but there's a lot of what the ad blocking world revolves around is filter list maintainers i. Would say he's a very brave person. Absolutely now. The way i envision it is you get those filter lists which i guess are selectors of some sort maybe is it a css selector or is it some other sort of selector variant. Yeah when it comes to cosmetic filtering those are usually css selectors there's a couple of more advanced ways to do it but in general css selectors and for network filtering you'll have like patterns that are part of a url you can have regexes you can have just basic string matching things like that. Yeah the other thing that i remember is xpath i'm not sure if that is used in any filter list? Yeah, it's pretty rare. It is a feature in Ublock Origin and AdGuard and as of recently in Brave as well. As of recently, that means it's the future or is it a format that you want to support for legacy reasons? It's just the more features we can support, the better compatibility we have with upstream lists. Not all of the list developers specifically target Brave as the AdBlock engine that they use. Sometimes, if you don't support all of the filter list features, things might break in unexpected ways. Correct me if I'm wrong, but in 2019, the Rust ecosystem was still pretty rough or a little immature for some areas. And I do wonder how much you could use from the Rust ecosystem and how much you had to write yourself. Yeah, it definitely was rougher than it is today. Thankfully, I didn't run into too many issues with things being unavailable. For the most part, what I was doing was pretty basic string operations and putting things into hash maps and VEX and things like that. And those were all pretty well supported by the time I was working at Brave. How did you do the selector parsing? What package did you use? What create? Yeah, so initially at the time, we didn't even parse the selectors. We just look for, there's a special token that's separating URLs that the filter should be applied on versus the filter itself, which is just a double hashtag, basically. And if you find one of those, then you can be pretty confident that it's cosmetic filter, and that's where you will... Cut the string in half and just store the css selector and eventually forward it onto the page once the page is loaded. So for every page you go through the list and you, probably you don't have to do that with every single request because as you said you build a hash map and the hash map key is a website url or what's it going to be and the values are the selectors that you need to apply yeah. The keys there's a couple of different ways that you can specify sometimes it's like on this exact subdomain sometimes it's on this kind of you can do google.star to say i want to block on google.com google.co.uk whatever else that they happen to own and you can also do negations to these and so there has to be some clever logic to combine all those together. But generally, that's how that works. How much of that did you have to write yourself? Pretty much all of it. Isn't it super hard to really cover all of the edge cases? Definitely. There's a lot of edge cases. There isn't really a well-defined spec for adblock syntax. So what I learned very early on is you have to write something, ship it, and eventually your users will tell you if it's incorrect. And luckily with ad blocking, it's usually done on a best effort kind of basis as well. If one particular page gets broken, that's usually not a huge deal because most of the other pages might still be fine. I'm kind of surprised that the Brave team itself was so enthusiastic about Rust. Was that... A very common sentiment, or was there also any pushback from the team? I think it was explicitly an experiment that they wanted to do. The research team at Brave has been really awesome. They put out a lot of cutting-edge work. Not all of it makes it into the browser, but a lot of it does. I think this was another example of kind of a moonshot project that if it works, it'll be awesome. If not, no big deal. Try something else. Did anyone suggest to use C++ for it? We actually did have a version in C++ at the very beginning. I wasn't there for that. That one basically blocked some network requests and that was it. One of the big reasons they wanted to write a new one was because that one was not very good. I can understand that but at the same time c++ was already available you had it in your build environment ci cd worked with c++ and then for the rust-based ad blocker you probably need to introduce a completely new tool chain into the environment did that already exist when you started or did you have to build the tool chain as well no. We basically had to figure out how do you compile a Rust target that can be included from C++ code. And that actually ended up being not too big of a problem at the time. Turns out you can just compile to a static library and then hook it into the rest of the build system and everything just works. It's essentially the same way that the rest of the Chromium build works. You create static libraries and link them together. Just this one happens to be written in rust. How would i get started with that what libraries would you recommend. So actually these days i would totally recommend using cxx that is a binding library for c++ to rust code but that didn't really exist at the time when we started we were writing all of the function signatures by hand and hooking them up to the appropriate places in c++ code and that also limits what kind of function signatures you can use you can't put like a vec you have to use a slice and length basically. Yeah a pointer and a length did that interface change over time or has it stayed the same mostly. It was mostly the same for a while once cxx became viable we jumped on that and it's been amazing ever since. Ah, so that means you transitioned to CXX? Yes. I think we have a couple of things which haven't transitioned to CXX at this time, but for the most part, we definitely wouldn't create a new Rust component that isn't using it. I use uBlock Origin. Did you compare uBlock with your solution? Yeah, I've definitely done a lot of looking at uBlock Origin. I think one of the benefits of having our ad blocker written in Rust is that it compiles to native code. And that gives us a lot of benefits, one of which is independence over the web extension API. One thing we've seen pretty recently is that the Chrome team has deprecated manifest V2. And now if you want to create a new ad blocker, it has to be using manifest V3, which is very limiting and prevents you from, optimizing your filter matching and using features that might not be supported by the web extension platform. One example of that is CNAME blocking. So Firefox does support a non-standard API for getting DNS records for a certain website. And uBlock Origin can use that on Firefox, but they can't use that on Chrome. So there is no way for uBlock Origin on Chrome to do CNAME uncloaking. But for us, since we have a native adblock engine and we can tie into it directly, we can basically implement features that are completely untethered from the web extension platform. So we have CNAME uncloaking in our adblocking. That's something we wouldn't be able to do in JavaScript. Why would i need cname uncloaking. Website hosts can make it look like a certain resource is coming from their own website when in reality it's being served by some tracking company and that allows the tracking company to update the script on the fly, and collect a lot of data about your browsing and send it to a data silo and that is one of the things that we generally try to avoid when we're using ad blockers right. Yeah thanks for explaining that because i only heard in passing that they deprecated manifest v2 but i didn't know what implications it had i only heard that it's bad news for ad blockers. Yeah it's a very controversial move on google's part now. What's the future going to be for chrome and ublock origin users. I'm not too sure about that. You might want to ask Raymond Hill about that. Raymond Hill is the uBlock origin developer. Oh, okay. You mean I should go straight to the origin? Yes, exactly. The puns are getting worse today. But I will say for Brave will not be affected by the manifest v2 deprecation. So our adblock engine continues to work without any restrictions, regardless of wherever the web extension platform goes. What about performance? Did you measure the potential performance improvements in comparison to JavaScript? Yeah. So benchmarking these things is pretty difficult and I'm not claiming to be an expert here, but I did actually do a little bit of basic timing measurement earlier. So you block origin, I measure it would block one particular type of thing, maybe 3.3 times faster. I'd block rust would be 2.5 times faster on another thing, something like that. I didn't really see any order of magnitude differences there. But one thing you have to keep in mind is that these are lookups that occur on the order of nanoseconds. So normally you have one lookup per network request on a page. So that's on the order of maybe 100 network requests. I think what's standing out here is the really nice thing is that I don't even have to think about performance when I'm writing the Rust code versus when I look at, say, the uBlock Origin code base. There's so much kind of convolution that happens as a way to squeeze out extra performance. And in Rust, I really just don't even have to think about that. And it still is competitive. What are some of the war crimes to make uBlock Origin fast? There's a lot of reuse of static containers, basically, to avoid triggering the garbage collection engine. And then a lot of indirection in terms of, you can't just have a string that says this is blocked. You have to use a number and then look up that number in a table somewhere to figure out what exactly happened to this request and things like that so it's when you add all of these up together, just taking a look through that code base makes it really hard to understand what's going on unless you're actually tracing through and stepping through the code the. Other part of performance that people always miss is predictable performance because yeah it helps if it's fast in general on a p50 but the p99 is also important like what are the outliers especially with javascript when the garbage collector triggers you might see very long response times. Totally yeah the the flip side also is that there probably is some room for optimization in network rust whereas i'm not even sure if there is any room if you block origin i think that might be squeezed to its maximum. Do you do any of these computations in parallel? Yeah, technically we could. It's a little bit challenging because we have to comply with the Chromium kind of threading model. So Adblock Rust in particular doesn't do any thread spawning or passing between threads or anything like that. Technically, if you wanted to use the block engine somewhere else, you could just access it from different threads. Tell me a little bit about this Chrome threading model. I haven't heard about it. Yeah, Chrome is very strict about where threads can be used and for what. So if you're ever doing any kind of file IO, that has to be on a very specific kind of thread executor. If you're doing rendering, obviously that has to be in the rendering thread. But you want to put as little as possible in the rendering thread because that is exactly what the user is looking at. They'll be frustrated if there is lag as a result of heavy computation occurring there. This is also what they recommend JavaScript developers to do to push stuff off the main thread. Yep. Okay, so we talked about the performance side, but the other thing that people mention a lot with Rust is that things tend to work once they compile. But I wonder if you can remember any particularly gnarly bug, any challenging bug that you solved in Rust when you built the adblocker. Yeah, it actually relates to build size rather than logic. So initially I embedded a version of the public suffix list in the adblock engine. And for those who aren't familiar, the public suffix list is basically a list that says, if you have a domain that ends in .com, then that .com part is a suffix and the same applies for .co.uk. So if you have example.co.uk, then the registered domain part is actually example and not co. Isn't that the same as TLD or what's the difference between the suffix and a TLD? It's a little bit complicated, suffice to say that this ends up being important when you're matching against domains by parts of them so we had an embedded version of the public suffix list in the adblock engine, and i had some complaints from a couple folks that it was too large in our compiled browser builds, and so I started looking into it and realized we do have a public suffix list in Chromium but if I remove it from Adblock Rust altogether then I won't be able to use it in other contexts outside of the browser which sometimes I do for either testing or some other browsers actually embed Adblock Rust as well and it's probably more convenient for them to have it this way. So what I ended up doing is making a feature for embedding the public suffix list or not. One of the neat things is that because there is a C++ FFI, we can just link it to a function and have that be the public suffix list implementation so that's what we ended up doing in chromium builds and then for external builds we just use the feature, everything is working and happy and there aren't duplicate versions of large binary data. So it's a bit like a proxy for the data something that you can mix and match so when you test locally what do you do you have your local checkout of that suffix list. We have a dependency which basically wraps around the public suffix list and embeds it downloaded at build time was. It really such a huge part of the build. There's going to be some people falling off of like the android install process and they just never open the app after it takes too long to download oh. Yeah for mobile i can totally understand do you also kick out features from chrome that you don't like or you don't need or just to trim some fat. Yeah we do at brave there's a lot of patching processes we have google in particular likes to push out features that are a little bit harmful to privacy and that kind of thing so we spend a lot of effort on making sure that those things don't hit our users and we have a team that's literally just devoted to, rebasing on top of chromium and keeping our patches up to date and they make sure that whenever there's a new chromium release we have our corresponding release in under 24 hours with the changes that we need. Not bad and while we are already at it we might as well address the elephant in the room the browser landscape because a lot of browsers recently or not so recently adopted rust in some form or another a couple examples are chrome which adopted rust at least partially they have it in the build process and then you got firefox of course they have parts of firefox written in rust nowadays mozilla being the creator of rust they at least do parts of their css rendering in rust and other parts now how is brave's approach to rust unique in the browser world and also what do you say about these other browsers and their rust usage. Chrome took a very measured approach in particular with adopting rust i think they've been interested in it for a while but they wanted to really stress test it and make sure that they have good integration with their build process, and that everything works and doesn't cause unexpected issues out in the wild because they have a huge user base, Things can go very wrong in ways that you have no idea about. So they were prototyping a way to include Rust code in their code base for a while. And we were watching that with a lot of interest because obviously we had our own build system integrating with Rust for a while. And we had ours pretty early on and we were pretty confident about it. And we just said, we'll try it. We'll see what happens. And as a result, we had it working pretty well. And I'm not sure if the Chrome team looked at us as a success story, but we were certainly available at the time. I can't even remember what they use it for. I only know that they put it into their build process, but not sure for what purpose. Yeah, their initial test was with the QR code decoder or encoder, one of those. Do they use it for anything else nowadays? Yeah they definitely have a lot of intention to replace things with rust where possible i think the benefits of rust are pretty clear in that regard the the memory safety the development velocity uh just confidence of your changes yeah. Yeah and don't get me wrong i don't want to sell them short i know that 70 of all chrome bucks are memory safety issues solving that problem alone is worth it. For them. I just don't know if they have any strategic approach as to where to focus on first. The approach they've been taking so far is to look for isolated components that have a pretty narrow API. And I think that's a pretty good idea in general. A lot of the complexity of integrating Rust into your build is around the foreign function interface where Rust becomes some other language. And so if you can keep that as limited as possible, that usually makes things quite a bit easier. Can you also contrast Brave's approach with Firefox? What do they do different? Firefox is very interested in replacing very core components with Rust. For the most part, at Brave, we don't try to rip out too much of the Chromium internals. We try to keep those functioning the way that users would expect Chromium to function. The more patches we add, the more burden there is to keep them up to date. So better for the upstream Chromium team to handle those, and we can benefit from those as they come along. So Mozilla obviously has to focus their engineering effort on maintenance of the existing C++ codebase. And I think they've realized rightfully that the more they can port to Rust, the better that it'll be in the long run. All of those browsers use Chrome to some extent, but there's many other languages in modern browsers nowadays. I can't even count them all, but I know it's certainly more than C++ and Rust. Do you know what languages get used? And what's your take on this? Where does each language shine and fall short? Yeah, if you look at just the Brave source tree alone, and you run the tokei tool in there, which counts lines of code in different languages, there's 60 lines of output there. And then if you take that to the Chromium tree, actually, it takes forever to finish and eventually outputs 162 different lines. So there are a ton of languages being used. A lot of it has to do with the fact that Chromium is supported on Android and all of the desktop platforms and iOS to some extent as well. There are a lot of platform-specific APIs that need to be called from certain languages. And there's a lot of configuration files and make files and build systems and things like that. And these all kind of get bundled together into one gigantic code base with millions of lines. In the Chromium tree, there's ABNF, ADA, Alex, Arduino, C++, ASCIIDOC, Assembly, GNU-style Assembly, AutoConf, AutoMake, Bash, C, Cheader, CMake, C#, CodeQL, CoffeeScript, and the list goes on and on. I haven't heard that one in a while. I wonder what they use it for. Same. So if there's any popular language out there, a browser probably uses it somehow. Do you think that some of the more obscure languages get replaced with Rust? I totally think that's the intention. I don't know if the nature of software is it's really hard to get rid of some of these. They're probably there as regression tests or something, but C++ might be high performance, but it's a high performance mess, in my opinion. Rust is certainly the way to go there. like JavaScript and TypeScript. Obviously, it's easy to get started with those, but they fall apart at scale to some extent. And TypeScript has its own problems with type system holes. Those are just the main ones that I interact with usually. And Rust, I think, does a very good job of filling the shortcomings of those ones. And honestly, just having consistent standard library APIs goes a really long way towards making the language more pleasant to use and, a little bit safer and better yeah. There's two new browsers on on the horizon i guess one is servo and the other one is ladybird and i think ladybird in particular is interesting because they are starting to port their browser engine from c++ to swift what's your take on this. Yeah i'm pretty excited to see more diversity in the browser space personally i do think a lot of people get caught up in the browser wars and they'll say oh you shouldn't use this particular browser because i disagree with the funding model it uses or i disagree with this opinion that the founder has or something like that i'm not trying to call out anyone in particular i've seen it across the spectrum but honestly i think if you're using an open source browser You have nothing to be worried about. Open source is like the best insurance we have against some company doing something not nice. If you don't like the browser, probably there are other people out there who don't and you're welcome to make a fork of it and start charging money for it or find some other business model there. But that's something you can do with open source browsers that you really can't do with closed ones. So I'd say as long as you're using an open source browser, things are great. Well the other part i would be worried about would be ladybird's adoption of swift being a bit of roadblock in the future because i'm not sure how the swift ecosystem works but i don't know if they support that many targets at least in the past used to be very macOS specific i know that they open it up now and i guess they became a little more independent from apple but That would be one of my concerns from a browser engine perspective. Yeah, it is an unusual choice for sure. We have seen other examples of browser vendors choosing unconventional languages. If you look at, for example, Arc Browser, they also made a similar choice and had to reimplement a Swift target for Windows, I believe. Interestingly ladybird also evaluated rust and one of the reasons for dismissing rust was that apparently according to the core maintainers the rust community was toxic and to be honest that took me by surprise did you encounter that yourself did you encounter any toxicity in the rust community i've. Never seen that personally i think the rust community is very welcoming, at least if you compare it to C++. Which tends to do a lot of gatekeeping. The only maybe negative thing I can say about the Rust community is that they want to rewrite everything in Rust, which could also be a positive thing, depending on your perspective. Okay, but let's try to stay clear from that a bit. It's certainly an interesting time for browsers again. No matter where we're going with this, it's nice to see some competition again, because in the past, that wasn't the case. I was a bit controversial lately when i wrote a blog post about the dying web my core statement was that people keep using derivatives of chrome or chromium and mozilla was out there as the only real independent competing engine because apple safari would have been an option too but it's also backed by a trillion dollar company and what's your take on this as someone that works on Brave? Really, as long as you're using an open source browser, things are pretty good because a lot of people are worried about the kind of browser monoculture if everyone uses Chromium, but Brave uses Chromium and we get rid of the parts that, are not user-friendly or user-hostile or things like that. Because the browser is open source, If there's something you don't like, you can go and patch it out, and this kind of solves the issue right there. But my main point of criticism from the article was that you end up with a single implementation of the spec. And if you make a mistake, which inevitably happens, we're all humans, if you have such a huge gravity in the ecosystem, your bug becomes the spec. So in order to avoid that you would have to have multiple competing browser engines which all have the same weight in order to balance it out and maybe fix the spec or fix bugs but it feels like some developers even stopped testing on other browsers this. Is definitely true to some extent but i guess the other thing to consider is that brave participates in standards meetings in W3C and IETF, and we actively push back on things that we believe are harmful to the web platform, whether it's privacy or security or other concerns. And that's something that we might not be able to have the bandwidth to do if we were starting from a different browser engine. This kind of diversity on top of Chromium is still also valuable to the space, in my opinion. Absolutely I fully agree even if you use the same browser engine having different stakeholders and maybe different variants of that is helping you, I think it could go a little beyond that, even past that, with new implementations of the spec. Because one thing that I heard from the Ladybird maintainers was that if you follow the spec, it's actually working. You can start a browser engine, even in 2024, by following the spec and literally implementing it almost verbatim. And you get pretty far with it, which is surprising to me. Yeah that sounds about right the specs are ginormous. And that kind of brings us to servo2 which is mozilla's new hope of a browser engine i guess but it's separated from mozilla now i don't know how involved they are nowadays what's your take on servo i've. Been seeing them post updates and it looks promising i still wouldn't use it as a daily driver but i am excited to see it especially being used in say embedded contexts where normally you would use chrome for that kind of thing and now there is a viable alternative in that space as well. Might take a couple years but maybe they get there in the end but i want to come back to brave now the first rust integration of your ad blocker is done what were some of the real world challenges and triumphs to make that happen? A lot of the challenges that we run into are things that we implement a version of. We realize, oh, other ad blockers do it a little bit differently. We can tweak it and then we figure it out from there. With regards to Rust specifically, I think it is very useful to have all of the Rust code in kind of an isolated module because we have that rebasing team working consistently to make the browser code stay up to date with Chromium. And sometimes that involves replacing all of the instances of the optional type with a different optional type or replacing all raw pointers in struct fields with a class called raw. And these things are a little bit annoying from a development process, but you have to deal with them and make sure that your changes are staying up to date in real time. And luckily when I work in Rust, I don't have to worry about that at all. In the past, we were using our own kind of hand rolled build system. To integrate with the rest of the broader GN build system that Chromium uses. Chromium was testing in their own branch, a tool that they call GNRT, which does a couple of things, including committing all of the source for all of the Rust dependencies into the rest of the Chromium source tree and also running the build process. Eventually they said, okay, we're ready to take this live in the master branch of Chromium. And at the time it wasn't quite ready for embedders to extend so we had to scramble to make sure that our hand-rolled rust build system would work together with this thing and so for a couple months our process of modifying dependencies was to basically download the crate tarballs directly from crates.io and put them into a numbered directory indicating the version of the crate and then update a ton of build files by hand to point to the right directory, which was quite painful. But eventually we got that sorted out. The Chromium team made a way to add additional directories where Rust files are located. And now a lot of this kind of version updating stuff is automated to some extent. Is that something they did anyway, or was that a specific ask from you? It was definitely something we were bugging them about. So you're in touch with them? Yeah, we do have access to their Slack channel, and sometimes we can post on their bug tracker as well. What were some other innovations of this build process? What's the before and after? One interesting thing is that GNRT doesn't actually use cargo at all. They call Rusty directly and wrap around GNRT. To pass the correct compiler flags and linking arguments and things like that. I think there's a hesitation to use Cargo just because it's kind of external build automation tool and the Chromium team likes to have a lot of control over that kind of thing. I think they use Bazel or Bazel, however you want to pronounce it, or did they move away from that? I think they use every build system in existence, to be honest. The main one is called GN, which is, I'm not even sure if any other project uses it, but it's a light wrapper around Ninja, as far as I'm aware, and it has its own Python-based domain-specific language. What percentage of the build time is because of Rust? So Rust builds are actually a pretty small percentage of the Chromium build process, but that's mostly because there's a ton of C++ code and not because Rust is particularly fast. For reference, compiling Chromium from scratch probably takes me about an hour on my desktop, which is not exactly an underpowered machine. Okay. And what about incremental builds? Yeah, luckily incremental builds work. If they didn't, I don't know if I'd be able to put up with this kind of thing. But yeah, for the most part, if you make a change, You only have to recompile that particular part of it and not the entire browser. What's your proudest achievement using Rust at Brave? I'm really proud of the community that's been rallying around Adblock Rust. We've made the first adblock engine that compiles to native code and is useful. I think because of that, it's been adopted in several other open source projects. So just to name a couple, there's AngelFish browser, which is used on a couple of Linux distributions and especially by the growing mobile Linux ecosystem. There's qutebrowser, which is a browser that's based on having Vim key bindings. There's privaxy, which is man in the middle proxy, which has adblocking built in. And there's a bunch of others too and i've noticed a couple of closed source projects even, bundling adblock rust in it and trying to be shy about it but it is possible to tell so it's pretty interesting to see and all these downstream users tend to contribute back fixes and improvements and bug reports i think just using rust naturally tends to attract like talented developers who are interested in building awesome software yeah. That's a very nice take we hear that a lot from different angles so maybe there's some truth to it and at the same time we also hear a lot that code sharing is something that is very much done and encouraged in the rust community your code might end up in places that you haven't anticipated before. Definitely definitely and the The other cool thing is that Rust builds on all kinds of targets. So we actually have a WebAssembly dashboard that embeds adblock Rust. We often use that to debug, like, okay, how exactly is this filter being parsed? Would this filter block this particular request? And you just get to type in an input field and have that work. We talked about adblocking a lot. I think that's the core feature at the moment. But are there any other features that you use Rust for? Yeah. So after we adopted Rust, we started to notice that it's a pretty natural fit for a bunch of other things. In particular, a lot of cryptocurrencies have their SDKs written in Rust. So we can integrate those with the wallet built into Brave pretty easily. That would be like Solana or Filecoin. We also have a feature called SpeedReader, which kind of distills the DOM of a page into a very minimal appearance, avoids downloading unnecessary resources as well and we wrote that in Rust as well. I have a use case for that. How does that part work, actually? Incredibly fast. Because it basically avoids all of the stylesheet parsing and application and a lot of the scripts that don't end up being necessary in a page. And it just is able to identify exactly where the text is in a document and put that out in just a very readable format. But so it's a bit like read mode in Firefox, which gives you a very clean version of the page. That's interesting. Does it remove HTML text, for example? It basically only extracts what is useful to the reading. How reliable is it? Because I have this project. It's made around a couple of years ago, actually. It's not really maintained anymore, but it was called readable. it was a service where you could enter a url and you will get a clean text version of it you can check the show notes if you're interested anyhow it wasn't really reliable because it wouldn't always pick up the main content of the website is yours more reliable yeah. It's actually been pretty reliable surprisingly so i i feel like the number of reports we get of it not working or pretty small in comparison to other things. So that's been good. And if you're curious about using it, I'm sure you can go and check out the Brave source code and pull it out directly from there. Since it's a Rust library, it should be pretty modular. Where else do you see potential for Rust in Brave? Kind of everywhere, especially if there's a way of making interfaces to the threading model or allowing us to use file input output those are things that we've had to be very careful about not using because they're not working with the chromium model just yet i think it's mostly just the chromium team needs to figure out what that might look like in the future now. I know that a lot of listeners are always curious about using rust in a major production, environment. You went through this experience with Brave and, What are some of the insights that you gained from that? Yeah, so I think the Rust in Brave is probably the part of our code with the highest velocity. Like when it comes to writing a totally new feature that would span across the whole stack, I'd say I probably write most of the code in Rust, but I would spend most of the time developing on like debugging silly C++ foot guns after I've integrated the Rust part. So i think rust is super powerful in that regard in terms of metrics and user impact i think brave has about 70 million monthly active users at this point so the ad blocking is one of the things that is working right in the background as soon as the user navigates to their first web page and it's one of those things where you hear about it the moment something stops working so the less you hear about it the better and for the most part things are just pretty smooth sailing so i think we definitely have a lot of benefit there did. You have any production crashes because of the rust part. There's been a couple i think usually it comes around to the code that i've written around it in c++ but for the most part it's pretty stable it's. A nice take on c++. You got issues with it because you call into c++ or from c++ into rust but not necessarily because of rust. Exactly What's your message to rust developers considering using rust for a large scale project. I would just say go for it it's totally worth it and i think especially if you start small you'll start to realize how much potential there is in the project for, upgrading to a memory safe implementation, upgrading to something that maybe has more performance than what you were previously writing in. And what excites you the most about the future of Rust in, say, a browser environment, in browser development? Yeah, I'm excited to see more and more memory safe code being added to the browser. And I think especially lowering the barrier to entry for new contributors c++ is pretty scary and the more rust code i think will make it easier for newcomers to take a look at and say oh maybe i can actually make a difference here and improve this feature or fix this bug something like that. Do you personally use rust outside of brave nowadays. Yeah, I do actually. So one of my passion projects is I'm building a keyboard from scratch and the firmware is going to be in Rust. And one of the cool features is that there's a screen and eight gigabytes of persistent memory. And so what I'm trying to build in there is a Unicode database so that you can look up Unicode characters on the fly and type those directly to your computer, have a password manager built in so that you can have your passwords with you wherever you have your keyboard, have a hardware 2FA key implementation in there so you can use it instead of a YubiKey and not have to plug another thing in. And all of this is going to be open source, including the PCB design and the case design. So I haven't announced anything there officially yet, But I do have a kind of teaser web page and email list in case anyone's interested. We can put that in the show notes. You have announced it now. I guess so. There's no way you have to finish it. The other thing I immediately thought of when you said you have 8 gigabytes of storage was macros. Because you could have things that are very repetitive and can store them on the keyboard. I know that as a Vim user, the macro feature is pretty nice if you want to record certain movements and inputs. Yeah, I usually use Vim for development. And so I feel like I personally get less value out of macros built into my keyboard. But certainly for software where macros aren't built in as a feature, the other interesting possibility is because I have persistent storage in the keyboard, I can start to collect metrics on what kind of keystrokes are most commonly executed together. Maybe there's some patterns that can be turned into repeatable motions automatically. And some room for optimizing your keymap on the fly. You could even track the number of typos because you know how many times you press the backspace button. Exactly. That's pretty cool. So you could have a list of the most commonly mistyped words and have a personal typing tutor. That's nice. Let me know once it hits the stores. I'm interested. Yeah, I will do. I'm interested in a lot of your projects. So I will also check out parts that you wrote in Brave. And also I'm interested in the readability feature. Now, traditionally, the final question is, do you have a statement to the Rust community? Do you have anything that you would like to share with all of us? Yeah, I guess I just feel that Rust solves so many of our problems in software development that we've run into. And it really just works everywhere from embedded to desktop to mobile to even on the web with WebAssembly. I guess my take is that you better have a really good reason not to use Rust as the go-to language for your next project. Yeah i agree where can people learn more about brave yeah so brave.com has all of the information about where you can download it we have a search engine as well you're welcome to use it from there and we can have those linked in the show notes as well. Awesome good luck with the keyboard and with future developments and hope to see you soon awesome thank you Rust in Production is a podcast by Corrode it is hosted by me Matthias Endler and produced by Simon Brüggen for show notes transcripts and to learn more about how we can help your company make the most of Rust visit corrode.dev thanks for listening to Rust in Production.