Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: My name is Matthias Endler from Corrode, and today we're talking to Anton Lazarev
Matthias: from Brave about building a high-performance ad blocker in Rust.
Matthias: Anton, thanks for joining. Can you introduce yourself?
Anton: Yeah, I'm Anton. I live in San Francisco, and I work for Brave.
Anton: Brave is a company that makes a web browser based on Chromium,
Anton: and I specifically work on the Adblock engine.
Anton: The Adblock engine that we use is completely written in Rust,
Anton: and it's one of my favorite things about the company, and it works pretty well in my opinion.
Matthias: How did you even get started with Rust? What's your background?
Anton: In high school, I was interested in 3D printing, and at one point I found a
Anton: blog post that somebody wrote about Rust.
Anton: And I was like, that looks interesting. Maybe I can use that to write some G-code,
Anton: which is the control language that's used to talk to a 3D printer.
Anton: I think I started by trying to make a parser for G-code and I was doing the
Anton: standard string operations you would try to do in C or C++, taking indexes into
Anton: the string to get letters and numbers out.
Anton: That's clearly not the way to use Rust. And especially back then,
Anton: And this was probably 2016.
Anton: That was a bit of a mess back then. I wasn't too interested in Rust after that
Anton: experience, but there was something about it that drew me back.
Anton: And especially, I was also interested in electronic music.
Anton: And so I wanted to build a VST plugin, which is basically an instrument that
Anton: you can plug into audio software to make sounds and things like that.
Anton: And I thought, maybe I'll give Rust another try. And I started looking into,
Anton: let's see, what kind of UI libraries can I use for this, since most of VST plugins
Anton: would have a visual interface.
Anton: And the Rust UI space at the time was very underdeveloped. But one of the ones
Anton: that I found was a library called Azul.
Anton: And I started contributing a couple of things that were missing to Azul and
Anton: the contributor was very helpful.
Anton: He roasted my code a little bit, but that was super useful because I suddenly
Anton: started to learn all the best practices.
Anton: And eventually I felt pretty comfortable in Rust.
Anton: I had sort of a working VST plugin.
Matthias: How did that shape your relationship to systems programming in general?
Matthias: Because I know that VST plugins are pretty low level.
Matthias: There are pretty harsh constraints.
Anton: I think this especially was a pretty good project to do in Rust.
Anton: Because for audio work in general, you need to have very fast code.
Anton: Each audio sample has to be produced consistently with a low latency.
Anton: And Rust is very good at that kind of code. So I really started to realize this is where Rust shines.
Anton: And I started to be very interested in using it for every other kind of project as well.
Matthias: Can you even do allocations there?
Anton: You could, but it's generally recommended not to if possible.
Matthias: So when the plugin starts, you get a chunk of memory and then you operate on that?
Anton: Yeah, generally you want to leave allocation out of the audio thread as much as possible.
Matthias: You have very short time constraints for doing your computation and then the
Matthias: next loop starts. What are the implications of that?
Anton: Yeah, the way it usually works is you get a buffer of 512 samples,
Anton: let's say, and you fill that in.
Anton: And that buffer size can be configured by whoever's using the plugin but you
Anton: will get a fixed size buffer you fill it in and then you return it back with
Anton: audio in it for the host to use those.
Matthias: Stories they take place i guess around 2017-18 this is where you learned a little
Matthias: more about rust you start your first project and then eventually you joined brave when was that.
Anton: Yeah, so I started as an intern at Brave in 2019. And actually, I found the...
Anton: Job posting on reddit on the rust
Anton: subreddit of all places my my boss
Anton: basically posted we're looking for an undergraduate intern
Anton: who's willing to write an adblock engine in rust and it'll all be open source
Anton: for brave browser and i saw that i was like whoa that looks like a lot of fun
Anton: and so i did a little demo project as part of my interview and it seems like
Anton: he liked it and i've been there ever since basically it.
Matthias: Looks like that job had your name on it because.
Anton: They were.
Matthias: Looking for someone like you specifically an undergraduate as you said that's
Matthias: pretty cool why did they look for an undergraduate specifically why not just
Matthias: anyone with rust experience what was the background there.
Anton: At the time the adblock engine was kind of the first component that was going
Anton: to be built in rust for the browser and they weren't sure if it was actually
Anton: even going to work well so i think they wanted to have a very low commitment
Anton: let's see if this works if it doesn't no big deal that's.
Matthias: A very smart move from brave to say hold
Matthias: on it might fail we want to try it anyway but we give it to someone who maybe
Matthias: is fresh out of college or fresh out of university they maybe have some new
Matthias: ideas maybe they are enthusiastic about it and if it fails and the risk is pretty low,
Matthias: they might as well just learn a little bit about Brave and the company and so on.
Matthias: So we could repurpose them for
Matthias: something else if all else fails and it's a learning experience, right?
Anton: Yep, exactly. I actually wasn't even the first intern to be working on ad blocking.
Anton: There was one the previous year who did cosmetic filtering and they did it in JavaScript.
Anton: So it was a pretty simple implementation.
Anton: My initial project was to take that and move it into the block engine.
Anton: In rust it.
Matthias: Reminds me of a story of mozilla they started to
Matthias: make the css parsing parallel and
Matthias: they needed three attempts they had two undergraduates okay so one undergraduate
Matthias: and one full-time employee that would try to make it parallel and failed and
Matthias: then the third one was successful it was the first one written in rust the first
Matthias: two attempts were in c++ as i remember.
Anton: Yeah there's definitely something to be said about that approach.
Matthias: Okay so let's say it's your first week you started on this new project what
Matthias: now how do you even get started with such a task.
Anton: Yeah so to be
Anton: honest i hadn't really understood how adblocking worked
Anton: prior to joining the the task that
Anton: i did for my interview was to write a parser for adblock
Anton: syntax and so i had to learn how to do that in the first
Anton: place but it ended up being a lot of
Anton: research on what is cosmetic filtering how do
Anton: other ad blockers do it and so luckily we
Anton: live in a world where pretty much every useful
Anton: ad blocker is already open source so i could just dig
Anton: into the code bases for those and take
Anton: a look at what algorithms they used and it was a lot of reading some very convoluted
Anton: javascript that made my head hurt but eventually I figured it out and I started
Anton: translating that into Rust and I feel like the result was pretty nice.
Anton: We were able to ship it while I was an intern and that was a super happy moment for me.
Matthias: Can you explain to the audience what cosmetic filtering is?
Anton: Yeah, sure. So cosmetic filtering is basically anything that,
Anton: is being hidden in the page or being modified about the page after the page
Anton: has already been downloaded and.
Matthias: That is in contrast to what other sort of filtering.
Anton: Yeah you would have network filtering where you could block network requests
Anton: from happening in the first place but sometimes those network requests contain
Anton: both good and bad things and you need to selectively choose which ones to hide that.
Matthias: Means your ad blocker at least the first iteration of it was purely user-facing,
Matthias: That's what you see, the rendered version of the page.
Anton: You have a bunch of empty spaces on the page where ads used to be.
Anton: But if you have cosmetic filtering, you can pretty easily get rid of those and
Anton: make the page look a lot cleaner.
Matthias: But those two go in lockstep with one another.
Matthias: If you start to block one network request, you also need to update the cosmetic filtering.
Anton: The thing about ad blocking is that there's a lot of variety in how pages structure
Anton: themselves on the web. And so this all basically has to be done on a case-by-case basis.
Anton: And the way we do that is with filter lists. Maybe people don't know,
Anton: but all of ad blocking is based on tens to hundreds of thousands of lines of
Anton: rules that say, on this website,
Anton: block things that look like this in terms of the URL or in terms of what kind
Anton: of a request it is, whether it's a script or an image or media or something like that.
Anton: And hide things with the CSS selector, these specific sites.
Anton: It's just a huge list of rules that are maintained by...
Anton: People who do it for fun pretty much we actually employ one of them ryan brown
Anton: at brave he's awesome but there's a lot of what the ad blocking world revolves
Anton: around is filter list maintainers i.
Matthias: Would say he's a very brave person.
Anton: Absolutely now.
Matthias: The way i envision it is you get those filter lists which i guess are selectors
Matthias: of some sort maybe is it a css selector or is it some other sort of selector variant.
Anton: Yeah when it comes to cosmetic filtering those are usually css selectors there's
Anton: a couple of more advanced ways to do it but in general css selectors and for
Anton: network filtering you'll have like patterns that are part of a url you can have
Anton: regexes you can have just basic string matching things like that.
Matthias: Yeah the other thing that i remember is xpath i'm not sure if that is used in any filter list?
Anton: Yeah, it's pretty rare. It is a feature in Ublock Origin and AdGuard and as
Anton: of recently in Brave as well.
Matthias: As of recently, that means it's the future or is it a format that you want to
Matthias: support for legacy reasons?
Anton: It's just the more features we can support, the better compatibility we have with upstream lists.
Anton: Not all of the list developers specifically target Brave as the AdBlock engine
Anton: that they use. Sometimes, if you don't support all of the filter list features,
Anton: things might break in unexpected ways.
Matthias: Correct me if I'm wrong, but in 2019, the Rust ecosystem was still pretty rough
Matthias: or a little immature for some areas.
Matthias: And I do wonder how much you could use from the Rust ecosystem and how much
Matthias: you had to write yourself.
Anton: Yeah, it definitely was rougher than it is today. Thankfully,
Anton: I didn't run into too many issues with things being unavailable.
Anton: For the most part, what I was doing was pretty basic string operations and putting
Anton: things into hash maps and VEX and things like that.
Anton: And those were all pretty well supported by the time I was working at Brave.
Matthias: How did you do the selector parsing? What package did you use? What create?
Anton: Yeah, so initially at the time, we didn't even parse the selectors.
Anton: We just look for, there's a special token that's separating URLs that the filter
Anton: should be applied on versus the filter itself, which is just a double hashtag, basically.
Anton: And if you find one of those, then you can be pretty confident that it's cosmetic
Anton: filter, and that's where you will...
Anton: Cut the string in half and just store the css selector and eventually forward
Anton: it onto the page once the page is loaded.
Matthias: So for every page you go through the list and you,
Matthias: probably you don't have to do that with every single request because as you
Matthias: said you build a hash map and the hash map key is a website url or what's it
Matthias: going to be and the values are the selectors that you need to apply yeah.
Anton: The keys there's a couple of different ways that you can specify sometimes
Anton: it's like on this exact subdomain sometimes
Anton: it's on this kind of you can do google.star to say i want to block on google.com
Anton: google.co.uk whatever else that they happen to own and you can also do negations
Anton: to these and so there has to be some clever logic to combine all those together.
Anton: But generally, that's how that works.
Matthias: How much of that did you have to write yourself?
Anton: Pretty much all of it.
Matthias: Isn't it super hard to really cover all of the edge cases?
Anton: Definitely. There's a lot of edge cases. There isn't really a well-defined spec for adblock syntax.
Anton: So what I learned very early on is you have to write something,
Anton: ship it, and eventually your users will tell you if it's incorrect.
Anton: And luckily with ad blocking, it's usually done on a best effort kind of basis as well.
Anton: If one particular page gets broken, that's usually not a huge deal because most
Anton: of the other pages might still be fine.
Matthias: I'm kind of surprised that the Brave team itself was so enthusiastic about Rust. Was that...
Matthias: A very common sentiment, or was there also any pushback from the team?
Anton: I think it was explicitly an experiment that they wanted to do.
Anton: The research team at Brave has been really awesome. They put out a lot of cutting-edge work.
Anton: Not all of it makes it into the browser, but a lot of it does.
Anton: I think this was another example of kind of a moonshot project that if it works, it'll be awesome.
Anton: If not, no big deal. Try something else.
Matthias: Did anyone suggest to use C++ for it?
Anton: We actually did have a version in C++ at the very beginning. I wasn't there for that.
Anton: That one basically blocked some network requests and that was it.
Anton: One of the big reasons they wanted to write a new one was because that one was not very good.
Matthias: I can understand that but at the same time c++
Matthias: was already available you had it in your build
Matthias: environment ci cd worked with c++ and then for the rust-based ad blocker you
Matthias: probably need to introduce a completely new tool chain into the environment
Matthias: did that already exist when you started or did you have to build the tool chain as well no.
Anton: We basically had to figure out how do you compile a Rust target that can be included from C++ code.
Anton: And that actually ended up being not too big of a problem at the time.
Anton: Turns out you can just compile to a static library and then hook it into the
Anton: rest of the build system and everything just works.
Anton: It's essentially the same way that the rest of the Chromium build works.
Anton: You create static libraries and link them together.
Anton: Just this one happens to be written in rust.
Matthias: How would i get started with that what libraries would you recommend.
Anton: So actually these days i would totally recommend using
Anton: cxx that is a binding library for
Anton: c++ to rust code but that didn't really
Anton: exist at the time when we started we were writing all of
Anton: the function signatures by hand and hooking them up to the appropriate places
Anton: in c++ code and that also limits what kind of function signatures you can use
Anton: you can't put like a vec you have to use a slice and length basically.
Matthias: Yeah a pointer and a length did that interface change over time or has it stayed the same mostly.
Anton: It was mostly the same for a while once cxx became viable we jumped on that
Anton: and it's been amazing ever since.
Matthias: Ah, so that means you transitioned to CXX?
Anton: Yes. I think we have a couple of things which haven't transitioned to CXX at
Anton: this time, but for the most part,
Anton: we definitely wouldn't create a new Rust component that isn't using it.
Matthias: I use uBlock Origin. Did you compare uBlock with your solution?
Anton: Yeah, I've definitely done a lot of looking at uBlock Origin.
Anton: I think one of the benefits of having our ad blocker written in Rust is that
Anton: it compiles to native code.
Anton: And that gives us a lot of benefits, one of which is independence over the web extension API.
Anton: One thing we've seen pretty recently is that the Chrome team has deprecated manifest V2.
Anton: And now if you want to create a new ad blocker, it has to be using manifest
Anton: V3, which is very limiting and prevents you from,
Anton: optimizing your filter matching and using features that might not be supported
Anton: by the web extension platform.
Anton: One example of that is CNAME blocking. So Firefox does support a non-standard
Anton: API for getting DNS records for a certain website.
Anton: And uBlock Origin can use that on Firefox, but they can't use that on Chrome.
Anton: So there is no way for uBlock Origin on Chrome to do CNAME uncloaking.
Anton: But for us, since we have a native adblock engine and we can tie into it directly,
Anton: we can basically implement features that are completely untethered from the
Anton: web extension platform.
Anton: So we have CNAME uncloaking in our adblocking.
Anton: That's something we wouldn't be able to do in JavaScript.
Matthias: Why would i need cname uncloaking.
Anton: Website hosts can make it look like a certain resource is coming from their
Anton: own website when in reality it's being served by some tracking company and that
Anton: allows the tracking company to update the script on the fly,
Anton: and collect a lot of data about your browsing and send it to a data silo and
Anton: that is one of the things that we generally try to avoid when we're using ad blockers right.
Matthias: Yeah thanks for explaining that because i only heard in passing that they deprecated
Matthias: manifest v2 but i didn't know what implications it had i only heard that it's bad news for ad blockers.
Anton: Yeah it's a very controversial move on google's part now.
Matthias: What's the future going to be for chrome and ublock origin users.
Anton: I'm not too sure about that. You might want to ask Raymond Hill about that.
Anton: Raymond Hill is the uBlock origin developer.
Matthias: Oh, okay. You mean I should go straight to the origin?
Anton: Yes, exactly.
Matthias: The puns are getting worse today.
Anton: But I will say for Brave will not be affected by the manifest v2 deprecation.
Anton: So our adblock engine continues to work without any restrictions,
Anton: regardless of wherever the web extension platform goes.
Matthias: What about performance? Did you measure the potential performance improvements
Matthias: in comparison to JavaScript?
Anton: Yeah. So benchmarking these things is pretty difficult and I'm not claiming
Anton: to be an expert here, but I did actually do a little bit of basic timing measurement earlier.
Anton: So you block origin, I measure it would block one particular type of thing, maybe 3.3 times faster.
Anton: I'd block rust would be 2.5 times faster on another thing, something like that.
Anton: I didn't really see any order of magnitude differences there.
Anton: But one thing you have to keep in mind is that these are lookups that occur
Anton: on the order of nanoseconds.
Anton: So normally you have one lookup per network request on a page.
Anton: So that's on the order of maybe 100 network requests.
Anton: I think what's standing out here is the really nice thing is that I don't even
Anton: have to think about performance when I'm writing the Rust code versus when I
Anton: look at, say, the uBlock Origin code base.
Anton: There's so much kind of convolution that happens as a way to squeeze out extra performance.
Anton: And in Rust, I really just don't even have to think about that.
Anton: And it still is competitive.
Matthias: What are some of the war crimes to make uBlock Origin fast?
Anton: There's a lot of reuse of static containers, basically, to avoid triggering
Anton: the garbage collection engine.
Anton: And then a lot of indirection in terms of, you can't just have a string that says this is blocked.
Anton: You have to use a number and then look up that number in a table somewhere to
Anton: figure out what exactly happened to this request and things like that so it's
Anton: when you add all of these up together,
Anton: just taking a look through that code base makes it really hard to understand
Anton: what's going on unless you're actually tracing through and stepping through the code the.
Matthias: Other part of performance that people always miss is predictable
Matthias: performance because yeah it helps if it's fast in general on a p50 but the p99
Matthias: is also important like what are the outliers especially with javascript when
Matthias: the garbage collector triggers you might see very long response times.
Anton: Totally yeah the the flip side also is that there probably is some room for
Anton: optimization in network rust whereas i'm not even sure if there is any room
Anton: if you block origin i think that might be squeezed to its maximum.
Matthias: Do you do any of these computations in parallel?
Anton: Yeah, technically we could. It's a little bit challenging because we have to
Anton: comply with the Chromium kind of threading model.
Anton: So Adblock Rust in particular doesn't do any thread spawning or passing between
Anton: threads or anything like that.
Anton: Technically, if you wanted to use the block engine somewhere else,
Anton: you could just access it from different threads.
Matthias: Tell me a little bit about this Chrome threading model. I haven't heard about it.
Anton: Yeah, Chrome is very strict about where threads can be used and for what.
Anton: So if you're ever doing any kind of file IO, that has to be on a very specific
Anton: kind of thread executor.
Anton: If you're doing rendering, obviously that has to be in the rendering thread.
Anton: But you want to put as little as possible in the rendering thread because that
Anton: is exactly what the user is looking at.
Anton: They'll be frustrated if there is lag as a result of heavy computation occurring there.
Matthias: This is also what they recommend JavaScript developers to do to push stuff off the main thread.
Anton: Yep.
Matthias: Okay, so we talked about the performance side, but the other thing that people
Matthias: mention a lot with Rust is that things tend to work once they compile.
Matthias: But I wonder if you can remember any particularly gnarly bug,
Matthias: any challenging bug that you solved in Rust when you built the adblocker.
Anton: Yeah, it actually relates to build size rather than logic.
Anton: So initially I embedded a version of the public suffix list in the adblock engine.
Anton: And for those who aren't familiar, the public suffix list is basically a list
Anton: that says, if you have a domain that ends in .com, then that .com part is a
Anton: suffix and the same applies for .co.uk.
Anton: So if you have example.co.uk, then the registered domain part is actually example and not co.
Matthias: Isn't that the same as TLD or what's the difference between the suffix and a TLD?
Anton: It's a little bit complicated,
Anton: suffice to say that this ends up being important when you're matching against
Anton: domains by parts of them so we had an embedded version of the public suffix
Anton: list in the adblock engine,
Anton: and i had some complaints from a couple folks that it was too large in our compiled browser builds,
Anton: and so I started looking into it and realized we
Anton: do have a public suffix list in Chromium but if I remove it from Adblock Rust
Anton: altogether then I won't be able to use it in other contexts outside of the browser
Anton: which sometimes I do for either testing or some other browsers actually embed
Anton: Adblock Rust as well and it's probably more convenient for them to have it this way.
Anton: So what I ended up doing is making a feature for embedding the public suffix list or not.
Anton: One of the neat things is that because there is a C++ FFI,
Anton: we can just link it to a function and have that be the public suffix list implementation
Anton: so that's what we ended up doing
Anton: in chromium builds and then for external builds we just use the feature,
Anton: everything is working and happy and there aren't duplicate versions of large binary data.
Matthias: So it's a bit like a proxy for the data something that you can mix and match
Matthias: so when you test locally what do you do you have your local checkout of that suffix list.
Anton: We have a dependency which basically wraps around the public suffix list and
Anton: embeds it downloaded at build time was.
Matthias: It really such a huge part of the build.
Anton: There's going to be some people falling off of like the android install process
Anton: and they just never open the app after it takes too long to download oh.
Matthias: Yeah for mobile i can totally understand do you also kick out features from
Matthias: chrome that you don't like or you don't need or just to trim some fat.
Anton: Yeah we do at brave there's a lot of patching
Anton: processes we have google in particular likes to push out features that are a
Anton: little bit harmful to privacy and that kind of thing so we spend a lot of effort
Anton: on making sure that those things don't hit our users and we have a team that's
Anton: literally just devoted to,
Anton: rebasing on top of chromium and keeping our patches up to date and they make
Anton: sure that whenever there's a new chromium release we have our corresponding
Anton: release in under 24 hours with the changes that we need.
Matthias: Not bad and while
Matthias: we are already at it we might as well address the
Matthias: elephant in the room the browser landscape because
Matthias: a lot of browsers recently or not
Matthias: so recently adopted rust in some form
Matthias: or another a couple examples are chrome which
Matthias: adopted rust at least partially they
Matthias: have it in the build process and then
Matthias: you got firefox of course they have
Matthias: parts of firefox written in rust nowadays mozilla being
Matthias: the creator of rust they at least do parts of their css rendering in rust and
Matthias: other parts now how is brave's approach to rust unique in the browser world
Matthias: and also what do you say about these other browsers and their rust usage.
Anton: Chrome took a very measured approach in particular with adopting rust i think
Anton: they've been interested in it for a while but they wanted to really stress test
Anton: it and make sure that they have good integration with their build process,
Anton: and that everything works and doesn't cause unexpected issues out in the wild
Anton: because they have a huge user base,
Anton: Things can go very wrong in ways that you have no idea about.
Anton: So they were prototyping a way to include Rust code in their code base for a while.
Anton: And we were watching that with a lot of interest because obviously we had our
Anton: own build system integrating with Rust for a while.
Anton: And we had ours pretty early on and we were pretty confident about it.
Anton: And we just said, we'll try it. We'll see what happens.
Anton: And as a result, we had it working pretty well.
Anton: And I'm not sure if the Chrome team looked at us as a success story,
Anton: but we were certainly available at the time.
Matthias: I can't even remember what they use it for. I only know that they put it into
Matthias: their build process, but not sure for what purpose.
Anton: Yeah, their initial test was with the QR code decoder or encoder, one of those.
Matthias: Do they use it for anything else nowadays?
Anton: Yeah they definitely have a lot of intention to replace things with rust where
Anton: possible i think the benefits of rust are pretty clear in that regard the the
Anton: memory safety the development velocity uh just confidence of your changes yeah.
Matthias: Yeah and don't get me wrong i don't want to sell them short i know that 70 of
Matthias: all chrome bucks are memory safety issues solving that problem alone is worth it.
Matthias: For them.
Matthias: I just don't know if they have any strategic approach as to where to focus on first.
Anton: The approach they've been taking so far is to look for isolated components that
Anton: have a pretty narrow API.
Anton: And I think that's a pretty good idea in general. A lot of the complexity of
Anton: integrating Rust into your build is around the foreign function interface where
Anton: Rust becomes some other language.
Anton: And so if you can keep that as limited as possible, that usually makes things quite a bit easier.
Matthias: Can you also contrast Brave's approach with Firefox? What do they do different?
Anton: Firefox is very interested in replacing very core components with Rust.
Anton: For the most part, at Brave, we don't try to rip out too much of the Chromium internals.
Anton: We try to keep those functioning the way that users would expect Chromium to function.
Anton: The more patches we add, the more burden there is to keep them up to date.
Anton: So better for the upstream Chromium team to handle those, and we can benefit
Anton: from those as they come along. So Mozilla obviously has to focus their engineering
Anton: effort on maintenance of the existing C++ codebase.
Anton: And I think they've realized rightfully that the more they can port to Rust,
Anton: the better that it'll be in the long run.
Matthias: All of those browsers use Chrome to some extent, but there's many other languages
Matthias: in modern browsers nowadays.
Matthias: I can't even count them all, but I know it's certainly more than C++ and Rust. Do
Matthias: you know what languages get used? And what's your take on this?
Matthias: Where does each language shine and fall short?
Anton: Yeah, if you look at just the Brave source tree alone, and you run the tokei
Anton: tool in there, which counts lines of code in different languages,
Anton: there's 60 lines of output there.
Anton: And then if you take that to the Chromium tree, actually, it takes forever to
Anton: finish and eventually outputs 162 different lines.
Anton: So there are a ton of languages being used. A lot of it has to do with the fact
Anton: that Chromium is supported on
Anton: Android and all of the desktop platforms and iOS to some extent as well.
Anton: There are a lot of platform-specific APIs that need to be called from certain languages.
Anton: And there's a lot of configuration files and make files and build systems and things like that.
Anton: And these all kind of get bundled together into one gigantic code base with millions of lines.
Anton: In the Chromium tree, there's ABNF, ADA, Alex, Arduino, C++,
Anton: ASCIIDOC, Assembly, GNU-style Assembly, AutoConf, AutoMake,
Anton: Bash, C, Cheader, CMake, C#, CodeQL, CoffeeScript,
Anton: and the list goes on and on.
Matthias: I haven't heard that one in a while. I wonder what they use it for.
Anton: Same.
Matthias: So if there's any popular language out there, a browser probably uses it somehow.
Matthias: Do you think that some of the more obscure languages get replaced with Rust?
Anton: I totally think that's the intention. I don't know if the nature of software
Anton: is it's really hard to get rid of some of these.
Anton: They're probably there as regression tests or something, but C++ might be high
Anton: performance, but it's a high performance mess, in my opinion.
Anton: Rust is certainly the way to go there. like JavaScript and TypeScript.
Anton: Obviously, it's easy to get started with those, but they fall apart at scale to some extent.
Anton: And TypeScript has its own problems with type system holes. Those are just the
Anton: main ones that I interact with usually.
Anton: And Rust, I think, does a very good job of filling the shortcomings of those ones.
Anton: And honestly, just having consistent standard library APIs goes a really long
Anton: way towards making the language more pleasant to use and,
Anton: a little bit safer and better yeah.
Matthias: There's two new browsers on on
Matthias: the horizon i guess one is servo and the other one is ladybird and i think ladybird
Matthias: in particular is interesting because they are starting to port their browser
Matthias: engine from c++ to swift what's your take on this.
Anton: Yeah i'm pretty excited to see more
Anton: diversity in the browser space personally i do think a
Anton: lot of people get caught up in the browser wars and they'll
Anton: say oh you shouldn't use this particular browser because
Anton: i disagree with the funding model it uses or i disagree with this opinion that
Anton: the founder has or something like that i'm not trying to call out anyone in
Anton: particular i've seen it across the spectrum but honestly i think if you're using
Anton: an open source browser You have nothing to be worried about.
Anton: Open source is like the best insurance we have against some company doing something not nice.
Anton: If you don't like the browser, probably there are other people out there who
Anton: don't and you're welcome to make a fork of it and start charging money for it
Anton: or find some other business model there.
Anton: But that's something you can
Anton: do with open source browsers that you really can't do with closed ones.
Anton: So I'd say as long as you're using an open source browser, things are great.
Matthias: Well the other part i would be worried about would be ladybird's
Matthias: adoption of swift being a bit
Matthias: of roadblock in the future because i'm not sure how
Matthias: the swift ecosystem works but i don't know if they support that many targets
Matthias: at least in the past used to be very macOS specific i know that they open it
Matthias: up now and i guess they became a little more independent from apple but That
Matthias: would be one of my concerns from a browser engine perspective.
Anton: Yeah, it is an unusual choice for sure. We have seen other examples of browser
Anton: vendors choosing unconventional languages.
Anton: If you look at, for example, Arc Browser, they also made a similar choice and
Anton: had to reimplement a Swift target for Windows, I believe.
Matthias: Interestingly ladybird also evaluated rust
Matthias: and one of the reasons for dismissing
Matthias: rust was that apparently according to the core maintainers the rust community
Matthias: was toxic and to be honest that took me by surprise did you encounter that yourself
Matthias: did you encounter any toxicity in the rust community i've.
Anton: Never seen that personally i think the rust community is very welcoming,
Anton: at least if you compare it to C++.
Anton: Which tends to do a lot of gatekeeping. The only maybe negative thing I can
Anton: say about the Rust community is that they want to rewrite everything in Rust,
Anton: which could also be a positive thing, depending on your perspective.
Matthias: Okay, but let's try to stay clear from that a bit. It's certainly an interesting
Matthias: time for browsers again.
Matthias: No matter where we're going with this, it's nice to see some competition again,
Matthias: because in the past, that wasn't the case.
Matthias: I was a bit controversial lately when
Matthias: i wrote a blog post about the dying web my core
Matthias: statement was that people keep using derivatives
Matthias: of chrome or chromium and mozilla was out there as the only real independent
Matthias: competing engine because apple safari would have been an option too but it's
Matthias: also backed by a trillion dollar
Matthias: company and what's your take on this as someone that works on Brave?
Anton: Really, as long as you're using an open source browser, things are pretty good
Anton: because a lot of people are worried about the kind of browser monoculture if
Anton: everyone uses Chromium,
Anton: but Brave uses Chromium and we get rid of the parts that,
Anton: are not user-friendly or user-hostile or things like that.
Anton: Because the browser is open source, If there's something you don't like,
Anton: you can go and patch it out, and this kind of solves the issue right there.
Matthias: But my main point of criticism from the article was that you end up with a single
Matthias: implementation of the spec.
Matthias: And if you make a mistake, which inevitably happens, we're all humans,
Matthias: if you have such a huge gravity in the ecosystem, your bug becomes the spec.
Matthias: So in order to avoid that you would have to have multiple competing browser
Matthias: engines which all have the same weight in order to balance it out and maybe
Matthias: fix the spec or fix bugs but it feels like some developers even stopped testing on other browsers this.
Anton: Is definitely true to some extent but i guess the other thing to consider is
Anton: that brave participates in standards meetings in W3C and IETF,
Anton: and we actively push back on things that we believe are harmful to the web platform,
Anton: whether it's privacy or security or other concerns.
Anton: And that's something that we might not be able to have the bandwidth to do if
Anton: we were starting from a different browser engine.
Anton: This kind of diversity on top of Chromium is still also valuable to the space, in my opinion.
Matthias: Absolutely I fully agree even if you use the same browser engine having different
Matthias: stakeholders and maybe different variants of that is helping you,
Matthias: I think it could go a little beyond that, even past that, with new implementations of the spec.
Matthias: Because one thing that I heard from the Ladybird maintainers was that if you
Matthias: follow the spec, it's actually working. You can start a browser engine,
Matthias: even in 2024, by following the spec and literally implementing it almost verbatim.
Matthias: And you get pretty far with it, which is surprising to me.
Anton: Yeah that sounds about right the specs are ginormous.
Matthias: And that kind of brings us to servo2 which is mozilla's new hope of a browser
Matthias: engine i guess but it's separated from mozilla now i don't know how involved
Matthias: they are nowadays what's your take on servo i've.
Anton: Been seeing them post updates and it looks promising i still wouldn't use it
Anton: as a daily driver but i am excited to see it especially being used in say embedded
Anton: contexts where normally you would use chrome for that kind of thing and now
Anton: there is a viable alternative in that space as well.
Matthias: Might take a couple years but maybe they get there in the end but i want to
Matthias: come back to brave now the first rust integration of your ad blocker is done
Matthias: what were some of the real world challenges and triumphs to make that happen?
Anton: A lot of the challenges that we run into are things that we implement a version of.
Anton: We realize, oh, other ad blockers do it a little bit differently.
Anton: We can tweak it and then we figure it out from there.
Anton: With regards to Rust specifically, I think it is very useful to have all of
Anton: the Rust code in kind of an isolated module because we have that rebasing team
Anton: working consistently to make the browser code stay up to date with Chromium.
Anton: And sometimes that involves replacing all of the instances of the optional type
Anton: with a different optional type or replacing all raw pointers in struct fields
Anton: with a class called raw.
Anton: And these things are a little bit annoying from a development process,
Anton: but you have to deal with them and make sure that your changes are staying up to date in real time.
Anton: And luckily when I work in Rust, I don't have to worry about that at all.
Anton: In the past, we were using our own kind of hand rolled build system.
Anton: To integrate with the rest of the broader GN build system that Chromium uses.
Anton: Chromium was testing in their own branch, a tool that they call GNRT,
Anton: which does a couple of things, including committing all of the source for all
Anton: of the Rust dependencies into the rest of the Chromium source tree and also
Anton: running the build process.
Anton: Eventually they said, okay, we're ready to take this live in the master branch of Chromium.
Anton: And at the time it wasn't quite ready for embedders
Anton: to extend so we had
Anton: to scramble to make sure that our hand-rolled rust build
Anton: system would work together with this thing and
Anton: so for a couple months our process of modifying dependencies was to basically
Anton: download the crate tarballs directly from crates.io and put them into a numbered
Anton: directory indicating the version of the crate and then update a ton of build
Anton: files by hand to point to the right directory, which was quite painful.
Anton: But eventually we got that sorted out.
Anton: The Chromium team made a way to add additional directories where Rust files are located.
Anton: And now a lot of this kind of version updating stuff is automated to some extent.
Matthias: Is that something they did anyway, or was that a specific ask from you?
Anton: It was definitely something we were bugging them about.
Matthias: So you're in touch with them?
Anton: Yeah, we do have access to their Slack channel, and sometimes we can post on
Anton: their bug tracker as well.
Matthias: What were some other innovations of this build process? What's the before and after?
Anton: One interesting thing is that GNRT doesn't actually use cargo at all.
Anton: They call Rusty directly and wrap around GNRT.
Anton: To pass the correct compiler flags and linking arguments and things like that.
Anton: I think there's a hesitation to use Cargo just because it's kind of external
Anton: build automation tool and the Chromium team likes to have a lot of control over that kind of thing.
Matthias: I think they use Bazel or Bazel, however you want to pronounce it,
Matthias: or did they move away from that?
Anton: I think they use every build system in existence, to be honest.
Anton: The main one is called GN, which is, I'm not even sure if any other project
Anton: uses it, but it's a light wrapper around Ninja, as far as I'm aware,
Anton: and it has its own Python-based domain-specific language.
Matthias: What percentage of the build time is because of Rust?
Anton: So Rust builds are actually a pretty small percentage of the Chromium build
Anton: process, but that's mostly because there's a ton of C++ code and not because
Anton: Rust is particularly fast.
Anton: For reference, compiling Chromium from scratch probably takes me about an hour
Anton: on my desktop, which is not exactly an underpowered machine.
Matthias: Okay. And what about incremental builds?
Anton: Yeah, luckily incremental builds work. If they didn't, I don't know if I'd be
Anton: able to put up with this kind of thing.
Anton: But yeah, for the most part, if you make a change, You only have to recompile
Anton: that particular part of it and not the entire browser.
Matthias: What's your proudest achievement using Rust at Brave?
Anton: I'm really proud of the community that's been rallying around Adblock Rust.
Anton: We've made the first adblock engine that compiles to native code and is useful.
Anton: I think because of that, it's been adopted in several other open source projects.
Anton: So just to name a couple, there's AngelFish browser, which is used on a couple
Anton: of Linux distributions and especially by the growing mobile Linux ecosystem.
Anton: There's qutebrowser, which is a browser that's based on having Vim key bindings.
Anton: There's privaxy, which is man in the middle proxy, which has adblocking built in.
Anton: And there's a bunch of others too and i've noticed a couple of closed source projects even,
Anton: bundling adblock rust in it and trying to
Anton: be shy about it but it is possible to tell so it's pretty interesting to see
Anton: and all these downstream users tend to contribute back fixes and improvements
Anton: and bug reports i think just using rust naturally tends to attract like talented
Anton: developers who are interested in building awesome software yeah.
Matthias: That's a very nice take we hear that a lot from different angles so maybe there's
Matthias: some truth to it and at the same time we also hear a lot that code sharing is
Matthias: something that is very much done and encouraged in the rust community your code
Matthias: might end up in places that you haven't anticipated before.
Anton: Definitely definitely and the The other cool thing is that Rust builds on all kinds of targets.
Anton: So we actually have a WebAssembly dashboard that embeds adblock Rust.
Anton: We often use that to debug, like, okay, how exactly is this filter being parsed?
Anton: Would this filter block this particular request?
Anton: And you just get to type in an input field and have that work.
Matthias: We talked about adblocking a lot. I think that's the core feature at the moment.
Matthias: But are there any other features that you use Rust for?
Anton: Yeah. So after we adopted Rust, we started to notice that it's a pretty natural
Anton: fit for a bunch of other things.
Anton: In particular, a lot of cryptocurrencies have their SDKs written in Rust.
Anton: So we can integrate those with the wallet built into Brave pretty easily.
Anton: That would be like Solana or Filecoin.
Anton: We also have a feature called SpeedReader, which kind of distills the DOM of
Anton: a page into a very minimal appearance,
Anton: avoids downloading unnecessary resources as well and we wrote that in Rust as well.
Matthias: I have a use case for that. How does that part work, actually?
Anton: Incredibly fast. Because it basically avoids all of the stylesheet parsing and
Anton: application and a lot of the scripts that don't end up being necessary in a page.
Anton: And it just is able to identify exactly where the text is in a document and
Anton: put that out in just a very readable format.
Matthias: But so it's a bit like read mode in Firefox, which gives you a very clean version of the page.
Matthias: That's interesting. Does it remove HTML text, for example?
Anton: It basically only extracts what is useful to the reading.
Matthias: How reliable is it? Because I have this project. It's made around a couple of years ago, actually.
Matthias: It's not really maintained anymore, but it was called readable. it
Matthias: was a service where you could enter a url and you will get a clean text version
Matthias: of it you can check the show notes if you're interested anyhow it wasn't really
Matthias: reliable because it wouldn't always pick up the main content of the website
Matthias: is yours more reliable yeah.
Anton: It's actually been pretty reliable surprisingly so i i feel like the number
Anton: of reports we get of it not working or pretty small in comparison to other things.
Anton: So that's been good. And if you're curious about using it, I'm sure you can
Anton: go and check out the Brave source code and pull it out directly from there.
Anton: Since it's a Rust library, it should be pretty modular.
Matthias: Where else do you see potential for Rust in Brave?
Anton: Kind of everywhere, especially if there's a
Anton: way of making interfaces to
Anton: the threading model or allowing us to use file input output those are things
Anton: that we've had to be very careful about not using because they're not working
Anton: with the chromium model just yet i think it's mostly just the chromium team
Anton: needs to figure out what that might look like in the future now.
Matthias: I know that a lot of listeners are always curious about using rust in a major
Matthias: production, environment.
Matthias: You went through this experience with Brave and,
Matthias: What are some of the insights that you gained from that?
Anton: Yeah, so I think the Rust in Brave is probably the part of our code with the highest velocity.
Anton: Like when it comes to writing a totally new feature that would span across the
Anton: whole stack, I'd say I probably write most of the code in Rust,
Anton: but I would spend most of the time developing on like debugging silly C++ foot
Anton: guns after I've integrated the Rust part.
Anton: So i think rust is super powerful in that
Anton: regard in terms of metrics and
Anton: user impact i think brave has
Anton: about 70 million monthly active users at this point so
Anton: the ad blocking is one of the things
Anton: that is working right in the background as soon
Anton: as the user navigates to their first web page and it's one of those things where
Anton: you hear about it the moment something stops working so the less you hear about
Anton: it the better and for the most part things are just pretty smooth sailing so
Anton: i think we definitely have a lot of benefit there did.
Matthias: You have any production crashes because of the rust part.
Anton: There's been a couple i think usually it comes around to the code that i've
Anton: written around it in c++ but for the most part it's pretty stable it's.
Matthias: A nice take on c++.
Matthias: You got issues
Matthias: with it because you call into c++ or from c++ into rust but not necessarily because of rust.
Anton: Exactly
Matthias: What's
Matthias: your message to rust developers considering using rust for a large scale project.
Anton: I would just say go for it it's totally worth it and i think especially if you
Anton: start small you'll start to realize how much potential there is in the project for,
Anton: upgrading to a memory safe implementation, upgrading to something that maybe
Anton: has more performance than what you were previously writing in.
Matthias: And what excites you the most about the future of Rust in, say,
Matthias: a browser environment, in browser development?
Anton: Yeah, I'm excited to see more and more memory safe code being added to the browser.
Anton: And I think especially lowering the barrier to
Anton: entry for new contributors c++ is pretty scary and the more rust code i think
Anton: will make it easier for newcomers to take a look at and say oh maybe i can actually
Anton: make a difference here and improve this feature or fix this bug something like that.
Matthias: Do you personally use rust outside of brave nowadays.
Anton: Yeah, I do actually. So one of my passion projects is I'm building a keyboard
Anton: from scratch and the firmware is going to be in Rust.
Anton: And one of the cool features is that there's a screen and eight gigabytes of persistent memory.
Anton: And so what I'm trying to build in there is a Unicode database so that you can
Anton: look up Unicode characters on the fly and type those directly to your computer,
Anton: have a password manager built in so that you can have your passwords with you
Anton: wherever you have your keyboard, have a hardware 2FA key implementation in there
Anton: so you can use it instead of a YubiKey and not have to plug another thing in.
Anton: And all of this is going to be open source, including the PCB design and the case design.
Anton: So I haven't announced anything there officially yet, But I do have a kind of
Anton: teaser web page and email list in case anyone's interested. We can put that in the show notes.
Matthias: You have announced it now.
Anton: I guess so.
Matthias: There's no way you have to finish it. The other thing I immediately thought
Matthias: of when you said you have 8 gigabytes of storage was macros.
Matthias: Because you could have things that are very repetitive and can store them on the keyboard.
Matthias: I know that as a Vim user, the macro feature is pretty nice if you want to record
Matthias: certain movements and inputs.
Anton: Yeah, I usually use Vim for development. And so I feel like I personally get
Anton: less value out of macros built into my keyboard.
Anton: But certainly for software where macros aren't built in as a feature,
Anton: the other interesting possibility is because I have persistent storage in the
Anton: keyboard, I can start to collect metrics on what kind of keystrokes are most
Anton: commonly executed together.
Anton: Maybe there's some patterns that can be turned into repeatable motions automatically.
Anton: And some room for optimizing your keymap on the fly.
Matthias: You could even track the number of typos because you know how many times you
Matthias: press the backspace button.
Anton: Exactly.
Matthias: That's pretty cool. So you could have a list of the most commonly mistyped words
Matthias: and have a personal typing tutor.
Matthias: That's nice. Let me know once it hits the stores. I'm interested.
Matthias: Yeah, I will do. I'm interested in a lot of your projects. So I will also check
Matthias: out parts that you wrote in Brave. And also I'm interested in the readability feature.
Matthias: Now, traditionally, the final question is, do you have a statement to the Rust community?
Matthias: Do you have anything that you would like to share with all of us?
Anton: Yeah, I guess I just feel that Rust solves so many of our problems in software
Anton: development that we've run into.
Anton: And it really just works everywhere from embedded to desktop to mobile to even
Anton: on the web with WebAssembly.
Anton: I guess my take is that you better have a really good reason not to use Rust
Anton: as the go-to language for your next project.
Matthias: Yeah i agree where can people learn more about brave
Anton: yeah so brave.com
Anton: has all of the information about where you can download it we have a search
Anton: engine as well you're welcome to
Anton: use it from there and we can have those linked in the show notes as well.
Matthias: Awesome good luck with the keyboard and with future developments and hope to
Matthias: see you soon awesome
Anton: thank you
Matthias: Rust in Production is a podcast by Corrode it is hosted by me Matthias Endler
Matthias: and produced by Simon Brüggen for show notes transcripts and to learn more about
Matthias: how we can help your company make the most of Rust visit corrode.dev thanks for
Matthias: listening to Rust in Production.