Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: My name is Matthias Endler from corrode and today we talk to Kevin Guthrie and
Matthias: Edward Wang from Cloudflare about handling 90 million web requests per second with Rust.
Matthias: Kevin and Edward, thanks so much for taking the time. Can you introduce yourselves
Matthias: and Cloudflare, the company you work for?
Kevin: Sure. I'll go first. My name is Kevin Guthrie.
Kevin: I'm Principal Software Engineer or Systems Engineer at Cloudflare.
Kevin: I've been here about a year and a half. I've been a Rust developer for about
Kevin: four-ish years, on and off professionally.
Kevin: I've done some side projects, some games, some really stupid projects,
Kevin: some really complex projects.
Kevin: I just love Rust language. and it's hard to do anything else what about you Edward.
Edward: Yeah hey my name is Edward
Edward: i'm also a systems engineer here at
Edward: cloudflare and i've been working
Edward: on rust i mean since i joined the
Edward: company essentially about almost five years ago now at this point and i've been
Edward: working on well we're going to talk about the Pingora framework today previously
Edward: i was working at a game studio so working on internet plumbing essentially was
Edward: a pretty big difference now.
Matthias: Yes we will talk about Pingora today but i'm not sure if people are aware of
Matthias: the scale that cloudflare is at can you share some numbers just to fill everyone in.
Kevin: Yeah, okay. So we have some changing data.
Kevin: This is just based on public data. All the things we're going to share today
Kevin: are things that are publicly available.
Kevin: We have about 20% of the internet goes through Cloudflare.
Kevin: The reference here is from a tweet from one of our engineers.
Kevin: This is up from a couple years ago when you had,
Kevin: Steve Klabnik was on and talked about how Cloudflare had 10% of the world's
Kevin: internet, or 10% of the internet. So we're a little bit up from that.
Kevin: Currently, from the internal Pingora side, we handle about 90 million requests
Kevin: per second worldwide, occasionally going up above 100 million requests per second.
Matthias: That is crazy. I guess beyond comprehension for most people.
Matthias: That would mean that probably a huge majority of the traffic goes through Cloudflare,
Matthias: and maybe to some extent through Rust. We will talk about that today.
Matthias: But what is the setup internally to handle that scale, to handle that amount of requests?
Edward: Yeah, I think we have a,
Edward: I mean, if you're not familiar with Cloudflare to begin with,
Edward: Cloudflare operates a global network rate of more than 300 points of presence around the globe.
Edward: There are all these points of presence data centers in many different countries,
Edward: and traffic is routed to them via these Anycast addresses, something that we've
Edward: talked about on the blog before.
Edward: So there are all sorts of setups, both on the layer 4 side and layer 7 side,
Edward: to be able to load balance, distribute traffic and capacity accordingly.
Edward: Internally, we operate one of the,
Edward: and by we, I mean our team operates one of the services that your request travels
Edward: through in order to get served in a response.
Edward: And those are the services that are using our Angular framework, which is our team.
Edward: But internally, yes, there's a bunch of different mechanisms to balance and
Edward: distribute the traffic outside of data centers.
Edward: External to the data centers, routing to the data centers, and then within our
Edward: data centers themselves throughout the life of a request, as we call it.
Edward: Traveling through a few different CDN services or proxies.
Kevin: Some rust some not.
Matthias: A lot of people who are in the rust community for
Matthias: quite a while know that cloudflare was one
Matthias: of the earliest adopters of frost at a
Matthias: larger scale but what was rarely talked about was the reasoning behind why cloudflare
Matthias: chose rust can you shed some light on that was it for performance reasons was
Matthias: it for memory safety reasons what was the big driver behind Rust adoption at Cloudflare?
Edward: I think neither Kevin and I were around for the very beginning of this, right?
Edward: But we know that our teams have the content delivery network,
Edward: you know, compute, workers compute teams, etc.
Edward: They've all been eyeing Rust for a long time, as you mentioned.
Edward: And I think it was really, yeah, All of the, like, compile time...
Edward: Checks that you'd be able to do, all of the classes of bugs that you would essentially
Edward: eliminate from production, right?
Edward: You would be, I mean, on the content delivery side,
Edward: it's no secret that a lot of our company was built on these proxy services using
Edward: NGINX, right, which is based in C code, as well as Lua business logic on top of that.
Edward: The amount of, let's just say that there were certainly a number of core dumps
Edward: and invalid memory accesses associated with us perhaps making changes within our NGINX fork.
Edward: Over time, we've had to implement more and more complicated features within NGINX internally.
Edward: And these core dumps are really impactful, obviously.
Edward: When a core dump happens on a worker process in NGINX, that drops like thousands of requests.
Edward: So we had executive, honestly, we had executive visibility and support on that.
Edward: But something that our team has talked about before was that at least in the
Edward: earlier years, prior to Rust adoption,
Edward: for each core dump, I recall that our former CTO, John Graham Cumming,
Edward: would actually get an email for each of those.
Edward: These crashes were very much top of mind for folks.
Edward: So when you have that kind of, if you're able to build something to leverage
Edward: all of those advantages of,
Edward: hey, I can just completely erase, eliminate these classes of errors,
Edward: then you're definitely going to be pursuing something like that.
Edward: And Cloudflare is certainly we are not shy to consider every new technology
Edward: an advantage we can come by.
Matthias: The one thing that I noticed when you explained the reasoning behind Rust was
Matthias: that a big chunk of the business logic was written in Lua.
Matthias: And I wondered immediately, couldn't you just use another language with a static type system?
Matthias: Like, I don't know, Go, for example, wouldn't that have been easier to integrate?
Edward: Sure, but we were using NGINX, which was relatively new at the time that we were adopting.
Edward: It, I believe, we had built a lot of our features, the firewall,
Edward: etc., and DDoS features on top of filters that would run in NGINX.
Edward: So the OpenResty, I believe, as it's called, the OpenResty is a framework set of.
Edward: It allows you to implement business logic that you can plug into each of the
Edward: NGINX filters that run across the life of a request without necessarily touching all of the very,
Edward: perhaps arcane to a lot of folks, C code.
Edward: So in order to do something like integrate Go and stuff, there might be certain
Edward: similar efforts to do that.
Edward: But I think none have been as mature as OpenResty and its Lua logic and Lua filters.
Edward: I think for a while, some of the OpenResty folks were working with us on our CDN teams.
Edward: So generally speaking i
Edward: think go was one of the possibilities i
Edward: believe when we were evaluating other languages right to go to to switch to
Edward: but i think rust is the definite forerunner for all the reasons that i mentioned
Edward: a you know zero cost abstractions for great performance and And obviously,
Edward: most importantly, I think,
Edward: eliminating all sorts of memory safety issues and bugs that can arise from memory safety issues.
Kevin: One other thing that impacted our decision to go to Rust, I think.
Kevin: Like I said, I've only been here a year and a half, so I was definitely not around when the
Kevin: decision was being made.
Kevin: But we had a lot of the forerunners, like the celebrities and the Rust community,
Kevin: were working at Cloudflare at various points in time.
Kevin: Like Steve Klabnik himself worked at Cloudflare. Ashley Williams also worked here.
Kevin: So, I mean, there was a lot of popularity of the Rust language in Cloudflare to begin with.
Matthias: When you want to integrate a language like Go into an existing infrastructure
Matthias: that runs on NGINX, that would probably be a little harder because Go has a
Matthias: small but not negligible runtime.
Matthias: It has a garbage collector and so on. Whereas with Rust, you could integrate
Matthias: very deeply with basic C FFI.
Matthias: Did that also play a big role? And also, did you end up integrating Rust into
Matthias: your NGINX server for a while before you moved on to build your own solution?
Edward: Yeah, I think this speaks to the matters of how do you migrate and switch over
Edward: pieces of your infrastructure gradually to Rust as well, right?
Edward: So all of the, as I mentioned, a lot of the core business logic historically
Edward: has been built on Lua via OpenResty.
Edward: All of that, you know, business logic built up over time.
Edward: So I think there were initially some notions of how do you integrate,
Edward: how do you maybe integrate,
Edward: change those filters to how do you extract the business logic, right?
Edward: To use, to be using your Rust-based logic instead.
Edward: And there, I think we're varying approaches to this for, there are a lot of
Edward: teams that work on the CDN in addition to us.
Edward: Some of these, some of the logic you can kind of extract into different services,
Edward: either in-band with the request processing or out-of-band.
Edward: You can make calls to other services.
Edward: For example, the approach that we ended up choosing was to extract a specific, on a high level,
Edward: extract a particular responsibility of one of our NGINX proxies into a separate service.
Edward: That was what we were doing when we were first developing Pingora,
Edward: which was at the time, NGINX would reach out to make origin connections directly
Edward: and make origin requests directly.
Edward: We decided, hey, what if we situated an in-band with a request proxy that sits
Edward: just behind that NGINX proxy? and routed requests to that instead.
Edward: And then that service would decide how to make origin requests to which origins.
Edward: Handling all of the origin communication responsibility.
Edward: And then you're able to do something like divert traffic to that selectively
Edward: depending on how ready that service is to handle certain classes of requests.
Edward: I think this is generally the strategy that has been working out pretty well
Edward: for various services at Cloudflare,
Edward: as long as you're able to have something that has some sort of control plane that sits in front and,
Edward: decides, you know, the routing in band of that request.
Kevin: It's the unsurprising answer of how do you solve a problem at a proxy company
Kevin: is by adding more proxies.
Edward: Yeah. And certainly at first, I think we were a little concerned about,
Edward: I think whenever you're adding another hop, it's another proxy hop at injecting
Edward: another service into it.
Edward: You're worried about complexity. You're worried about performance regressions
Edward: and things like that, latency, obviously, right?
Edward: Generally speaking, what we had noticed was that adding another service hop is
Edward: certainly unconditionally going to add some amount of latency.
Edward: Thankfully the feature the
Edward: new logic that we were adding on top
Edward: of that was generally able to offset a lot of any those detriments that you
Edward: would face the the example that we tended to point out in our in our blog a
Edward: while ago was how our Pingora service i don't know how much we want to get into,
Edward: the why exactly in terms
Edward: of like NGINX versus Pingora architecture and stuff
Edward: but Pingora was a lot more
Edward: efficient this was definitely top of mind for us
Edward: a lot more efficient in terms of how it was making and
Edward: reusing origin connections and so
Edward: something like that generally brings down your you
Edward: know the the the latency of making an origin
Edward: request significantly if you can skip all of the TLS handshake etc latency fortunately
Edward: for us as well when it comes to replacing an origin facing proxy the the cost
Edward: of the cost of origin latency significantly dwarfs any additional proxy hop latency you have so.
Matthias: Okay, but was the project already
Matthias: called Pingora back then, or was it some sort of intermediate step?
Edward: I guess I have to shout out some folks who were working on Pingora.
Edward: I say I was working on it, but in reality, it was Yuchen, Yuchen Wu,
Edward: as well as Andrew Houck, who were kind of the primary and first drivers of Pingora.
Edward: And at first it was called OpenRusty, I think.
Edward: You still see this term in some of the old tests because it was very much meant
Edward: to replace OpenResty and NGINX itself and be a, I don't want to say a drop-in replacement,
Edward: but do all the things and model a lot of its logic off of NGINX and OpenResty.
Edward: Because honestly, that worked for us. and NGINX's logic models and the way that
Edward: it thought about request processing worked for us.
Edward: So we wanted to do a lot of things pretty similarly to NGINX.
Matthias: I really like the name, OpenRusty, but of course.
Kevin: I don't remember why they didn't go with that name in the end.
Kevin: I think it partially was sort of pejorative, but also could have been confused as a typo.
Kevin: I think Pingora is a much better name. The name, I think, came from the manager
Kevin: of the team who almost slipped and died off of the mountain,
Kevin: the literal mountain that's actually called Pingora.
Edward: I believe the story is that a particular trip to the Pingora Mountain almost cost him his life.
Edward: And now we've been ascending that summit ever since.
Matthias: Sounds like negative foreshadowing, but in reality it worked out well.
Matthias: But the one thing that I wondered was that, let's say you add another hop.
Matthias: You add proxy behind a proxy.
Matthias: And then you have a ton of requests coming in and then you want to switch to
Matthias: a new version you just kind of want to do a release basically wouldn't that
Matthias: be an easy source for dropping connections and dropping requests it.
Kevin: Is so it's something we have to do very carefully and we have since we handle
Kevin: things like web sockets we have lots of long-running requests so upgrades updates
Kevin: are something we don't do very often now or then.
Kevin: The way Pingora does this is a really slick system where when you bring up a
Kevin: new update, the process that you want to move everything to, it can start.
Kevin: It can know about the old instance of Pingora that's currently running.
Kevin: And that old instance can gracefully hand over the socket to start listening
Kevin: for new connections on the new instance of Pingora while old requests finish
Kevin: out on the old one and then the old instance can handle all of its requests
Kevin: and then gracefully shut down
Kevin: whereas the new one is bringing up any new connections and handling those.
Matthias: Is that safe or does that happen between processes? I wonder if you can even make it safe.
Kevin: It is. I mean, I don't know. I'm sure in Rust this is classified as some form
Kevin: of unsafe code because you're passing around raw file descriptors for sockets.
Kevin: But it is also a really common thing. This is something I first heard of at Facebook,
Kevin: where their networking, their HTTP servers do the same exact thing or even their
Kevin: load balancing system i think it's a very common process but i never worked
Kevin: with the actual code to do it until working on the Pingora project.
Edward: Yeah there's actually so yeah there's there's this process of transferring these
Edward: listen file descriptors i think it's one of the few places,
Edward: i i could be wrong but i think it's one of the few places where yes because
Edward: we're dealing with those raw file descriptors, there's a bit of unsafe code there.
Edward: There's actually also a crate that I believe we've put out, not us ourselves, but I mentioned,
Edward: or maybe I haven't mentioned yet that Cloudflare is not in a monolith when it
Edward: comes to Rust, and we are not the only folks,
Edward: developing in the Rust ecosystem.
Edward: So the folks who are working on another proxy
Edward: framework called oxy have actually open sourced a
Edward: crate that is specifically for these kinds of graceful process restarts and
Edward: it's called shellflip so that it uses a very you know similar mechanism of you
Edward: know transferring while descriptors and doing that handover of of yes doing that handover.
Matthias: ShellFlip sounds like another really cool name. You have a way with names, I guess, at Cloudflare.
Edward: I think it was taken from the TableFlipGo package.
Edward: And I think some of our engineers decided to...
Kevin: That's a good name.
Edward: I don't know why it's Shell in particular, but maybe it has to do with crabs.
Matthias: Maybe, yeah. But do you share a lot of code with other teams at Cloudflare?
Matthias: Rust creates, that is. You mentioned that you have Pingora and you have Oxy,
Matthias: but there's probably more stuff at Cloudflare which uses Rust.
Matthias: How does code sharing look like?
Matthias: Because from another company, or actually from a few, I heard that sort of by
Matthias: serendipity, they start to use different crates in completely different contexts.
Matthias: And it kind of happens very naturally to share code.
Kevin: Yeah, that's true. We have a sort of a haphazard way of sharing code.
Kevin: We do have our own internal repository for uploading crates,
Kevin: like it's an internal copy of crates.io.
Edward: Internal registry.
Kevin: Sorry, internal registry. That's the right terminology. But for a lot of it,
Kevin: it's done through referencing crates through Git URLs.
Kevin: So it's a little bit on the Go side of things. So you have a crate that you
Kevin: want to share with other people in Cloudflare.
Kevin: It's up on our internal Git server. You can write a blog post about it.
Kevin: Anybody can just include it.
Kevin: Putting it on the internal registry should be a more common thing to do,
Kevin: but I have literally never done it.
Kevin: I've shared a couple of crates with different teams for various stupid things,
Kevin: but most of those are just incorporated either through one way,
Kevin: which is making the project open source and then having people consume it just
Kevin: from the open internet or from the actual crates.io or consuming it internally
Kevin: from an internal Git repo.
Edward: I will say that I think usage of the internal registries is pretty common now.
Edward: The whole point of the registry, One of the major points was to avoid that the
Edward: kit commits kind of references that Argo allows you to do.
Edward: It's still used in some cases, right?
Edward: But yeah, more and more, I think the ecosystem around Rust has become a lot
Edward: more shared maybe in the past few years. In our code, we use both approaches.
Matthias: And when you publish code, do you have a formal process for the publication?
Matthias: Do you run any cargo tools to make sure that the code quality is on par with the rest?
Kevin: Uh we use the i mean really we
Kevin: use the standard open source tools we use
Kevin: clippy we use the auditing tool
Kevin: name i can't think of right now make sure we are not publishing anything within
Kevin: secure code cargo audit yes cargo audit
Kevin: yeah exactly but that's that's really
Kevin: about it we're very stringent on our internal code reviews
Kevin: so like i said we have open source projects all of the open source contributions
Kevin: that come in go through internal external review as well as internal review
Kevin: before they go into the the main branch for Pingora but as far as automated
Kevin: tools yeah it's really just clippy a clippy in testing now.
Matthias: Let's come back to Pingora for a while we established that we have a system
Matthias: called open rusty it's behind NGINX that's the current place we're at when was
Matthias: that roundabout like the year that you had that system running in production.
Edward: Oh boy so the blog came about 2022 i want to say the first forays into Pingora started,
Edward: around 2020 or a little before that i would need to look into the exact dates
Edward: for production But I want to say that the service,
Edward: I think it was around,
Edward: it wasn't long after 2020 that these services,
Edward: that the Pingora services first started to get used and deployed.
Matthias: Yeah. And pretty early on, you saw some advantages. I guess,
Matthias: Edward, you also mentioned that.
Matthias: The additional hop didn't really make a big difference because the connection
Matthias: to the origin was the bottleneck and the new Rust-based system was already pretty fast.
Matthias: But then one could argue NGINX was already plenty fast.
Matthias: What were some of the other NGINX limitations that you ran into which kind of
Matthias: triggered you to find a different approach other than, say,
Matthias: the lack of the type system, for example, that you had with the lure solution in the past.
Edward: Yeah i can definitely speak to that i think i was working on one particular
Edward: feature that was a bit hard to.
Edward: It was i mentioned that every time
Edward: we have we i mentioned
Edward: that we have an internal NGINX fork that we've added
Edward: more and more complexity into for developing our
Edward: own internal features whenever we want to futz around with
Edward: how NGINX does its request
Edward: processing and response serving right over time
Edward: and eventually there was
Edward: a moment at which the the straw
Edward: on the camel's back broke where
Edward: we were implementing we were
Edward: trying to implement more complicated logic on top
Edward: of the things we are already doing for example i
Edward: think there are we've blogged about concurrent
Edward: streaming acceleration which is a fancy name for we're serving your cached request
Edward: cached response body as it gets pulled from the origin it those changes are
Edward: pretty intrusive see changes as we iterated on top of that.
Edward: Any feature of decent complexity would cause core dumps.
Edward: And as I mentioned before, that was highly visible to leadership.
Edward: So if we were to make significant progress at all, we would usually be debugging
Edward: what sort of invariant we were violating inside of NGINX.
Edward: NGINX is great in a lot of ways.
Edward: And it, like the developers themselves are experts in what is valid, you know,
Edward: to access when and what can you do asynchronously from the, from the lifetime of the main request,
Edward: for example, and what is not safe to do so.
Edward: But those things are not necessarily, I mean, they're not as enforced within
Edward: the code strictly, right?
Edward: The way that you can encapsulate those exact kinds of lifetime and memory restrictions in Rust.
Edward: So that was the
Edward: point at which we said
Edward: we were already developing Pingora and then we said actually for any feature
Edward: of significant complexity we need to start moving it into the new system we
Edward: need to start migrating to the new to the new proxy system and developing features
Edward: there as much as possible instead of NGINX itself.
Kevin: One thing we want to make clear is we definitely are not here to complain about
Kevin: NGINX or to bash NGINX in any way.
Kevin: NGINX is like the foundation of Cloudflare. And the actual NGINX and OpenResty
Kevin: projects are amazing and stable and used in millions, billions of places. I don't actually know.
Kevin: But the modifications we were doing were not as stable and leading to the core dumps.
Kevin: As someone who came to this, came
Kevin: to Cloudflare not having done internet plumbing, just similar to Edward.
Kevin: Seeing the C code for NGINX, which is asynchronous, it's not written in an async
Kevin: await kind of way like you're used to with Rust or TypeScript or anything.
Kevin: It is literally, it's async code, but you're working with it in the time domain,
Kevin: like you are managing the state literally as you go through these,
Kevin: waiting for different files, waiting for sockets to open, close.
Kevin: So it is a very complex thing and almost impossible to debug.
Edward: Yeah, for sure. the
Edward: the honestly developer ergonomics
Edward: and developer velocity on top of
Edward: the you know the the the
Edward: classes of bugs that you're able to avoid and not
Edward: worry about that avoiding whole
Edward: classes of bugs speeds up your productivity where you
Edward: don't have to worry about introducing those things this is actually why a lot
Edward: of our business logic was written in lua filters as well because you don't you're
Edward: the the amount you're not going to seg vaults from manipulating lua objects right but that,
Edward: often comes at a performance cost with with lua vm and lua runtime even if you lua jit so this.
Edward: The other, like, main primary advantage of switching to Angular and Rust was
Edward: honestly just, like, as Kevin had mentioned,
Edward: the expressiveness of async Rust is extremely powerful.
Edward: And when you're
Edward: looking at especially for onboarding new engineers learning
Edward: NGINX and how it manually handles
Edward: the event loop events right because it when when you're going in when a request
Edward: comes into NGINX it needs it is it is handling those equal events and propagating
Edward: it to the request event handlers,
Edward: and then it needs to decide what comes next.
Edward: Assign the next handlers once your header is done, then you assign the event
Edward: handler for the body, etc.
Edward: There's a lot of manual mental effort involved with that kind of coding model
Edward: where you are both handling the HTTP processing logic in tandem with handling the event loop.
Edward: And with async await constructs, all of that logic then becomes linear,
Edward: actually. You can very much see...
Edward: After this, you're going to do this next in the life of a request.
Edward: And that, I think, I believe, has been really helpful for folks who are,
Edward: for onboarding new engineers, for learning the code base, etc.
Edward: I think it's that that was extremely those ergonomics were just as important
Edward: to us honestly because we need to ship things fast here.
Matthias: Sounds super crazy because not many people will be familiar with how engine
Matthias: x or to be more specific c handles asynchronous execution,
Matthias: That's a thing that was sort of a selling point for NGINX in the beginning.
Matthias: It was event-driven in comparison to Apache, which was not very much event-driven.
Matthias: It was more or less process-driven, and NGINX kind of changed that model,
Matthias: but you kind of need to shoehorn your logic into that.
Matthias: But is it similar to the state machine that gets converted to something more
Matthias: maintainable on the Rust side?
Matthias: So on Rust, we don't really need to write a big state machine ourselves.
Matthias: We just use async Rust as we do, and then the compiler will just generate the state machine for us.
Matthias: Is the code similar on the C side, or is it completely different?
Edward: Got it, yeah. Yeah, I would say NGINX,
Edward: I guess a lot of that is hand unrolled, the way we were talking about, right?
Edward: Where the events and the next state that you're going to go to for the next
Edward: event that you encounter are manually defined within NGINX.
Edward: And then you were also talking about how NGINX was...
Edward: Really revolutionary in terms of
Edward: how it was doing the asynchronous event driven model and that kind of touches
Edward: on a it's it's not exactly related
Edward: to how you know async rust gets kind of converted into a state machine,
Edward: but it is something that it's a big explanation for why NGINX is already a great performer,
Edward: right? Does really well in benchmarks already.
Edward: The underlying mechanisms for how that works, I talked about something called ePoll before.
Edward: The underlying mechanisms of NGINX
Edward: are such that there's an event
Edward: loop that an NGINX worker process goes
Edward: through and it's able to
Edward: with the help of operating system utilities
Edward: like epol determine when io
Edward: events are ready on certain file descriptors
Edward: and otherwise block if if
Edward: there are no events ready there but then so
Edward: you can think of it as essentially processing all of
Edward: these events as they become ready
Edward: as as io events come in on in
Edward: this you know literal loop and the equivalent like that that's the that was
Edward: a really powerful and greatly you know efficient way to to do all of this and respond to,
Edward: network and file IO, right?
Edward: We got to cheat with Pingora because we are able to reap the benefits of tokio
Edward: that does something really similar.
Edward: It has a, and I am not a tokio expert by any means, I would say,
Edward: but it is doing something really similar also with the help of Mio underlying, right,
Edward: Metal.io, where it's also handling the file descriptors and this event loop, etc.
Edward: Within its reactor that is using a lot of the same operating system mechanisms,
Edward: be it EPOL or KQ or what have you, to listen to these IO events and then propagate them,
Edward: awake the corresponding tokio tasks that are relying on them, right?
Edward: So all of a sudden, we are able to look at that as Bangora and we have a great abstraction,
Edward: you know, from the actual underlying event handling systems that we can build upon.
Edward: And so a lot of, I would say that, like.
Edward: Great big success when it comes
Edward: to why we were able to develop Pingora relatively quickly, I would say,
Edward: is because we were able to build upon the success of tokio and all of its great
Edward: performance considerations and mechanisms, right?
Edward: Because tokio also has a bunch
Edward: of internal optimizations for especially
Edward: when it comes to well we can get into how
Edward: it does things between threads etc and and tries to load balance load balance
Edward: tasks and work between threads but it's basically we were able to do so much
Edward: on top of because we already had a great underlying async runtime and event handling mechanism.
Kevin: I don't know if you sold, you said you weren't a tokio expert.
Kevin: I don't know if that came across because it sounds like you are quite the tokio expert now.
Kevin: I can say as a definite non-tokio expert that you don't really even need to
Kevin: know all this to use tokio as an async code.
Kevin: You can come into it as a TypeScript developer and be like, oh yeah,
Kevin: it's async, wait, I get it.
Edward: We have an actual tokio maintainer on our team now, Noah Kennedy,
Edward: who is actually a tokio expert.
Edward: So I don't generally say I'm an expert on things unless I really feel like I
Edward: know them very, very, very well.
Matthias: Well also the error messages have
Matthias: gotten a lot better in recent years i can still
Matthias: remember the early days when you got
Matthias: a page full of you know
Matthias: gibberish about the type system and so on but recently they
Matthias: improved it a lot thanks to compiler internals
Matthias: which helped infer what really went wrong and also present the information in
Matthias: a more consumable way and also just new mechanisms inside of the components
Matthias: of like Impultrate which allow you to focus more on the core issue at hand.
Matthias: I don't know if you saw any of the older error messages before but.
Kevin: Definitely yeah yeah so i
Kevin: i did some work with uh with tokio a long
Kevin: time ago like before before like well before i went
Kevin: 1.0 so things were a little bit different than in terms of error messages so
Kevin: like it was basically back in the old days of dealing with java error messages
Kevin: or other languages error messages where you can almost ignore them just look
Kevin: at where in the code it was pointing and go there and try to figure things out yourself,
Kevin: as opposed to right now, where the error messages in async are practically as
Kevin: good as they are in synchronous Rust code, which is to say really good.
Matthias: Did you ever run into problems with tokio, like, for example,
Matthias: starving the executor on threads when you block too many futures or async cancellation
Matthias: where you had a sub-request that you didn't want to kill when you killed main
Matthias: task, for example, and you wanted to keep it running?
Matthias: And the Rust Futures ecosystem was sort of getting into your way?
Kevin: At least a little bit. There was one scenario that I ran into within the past
Kevin: month where I was downloading a file, downloading a large file,
Kevin: running on a hyperscaler VM, so really excellent internet connection,
Kevin: and just the process of downloading that file on a small VM with a limited number
Kevin: of threads and a small number of tokio workers was enough to starve every other
Kevin: connection because I wasn't being smart. I wasn't using budgeting.
Kevin: I was just letting this one task take over the entire runtime and blocking everything else.
Kevin: Like no other async things were being taken care of. It was just happily downloading
Kevin: this one giant file and using the entire CPU to do that.
Matthias: Yeah, what happens a lot during testing is that people have these multi-core
Matthias: machines, they have, I don't know, 16, 32,
Matthias: 64 cores, what have you, and they run that system in their test laptop in their
Matthias: development environment.
Matthias: And everything works because you have plenty of threads.
Matthias: By default, tokio just spawns as many background threads as you have cores.
Matthias: But then you move to production where maybe you have to make do with two cores
Matthias: and then suddenly you have two blocking threads,
Matthias: tasks and then you ran
Matthias: out of threats and kind of the
Matthias: threat pool got exhausted that's a thing at least that i
Matthias: saw with some clients and you running
Matthias: 20 of the internet i did wonder if you ever run and ran into these sort of problems
Matthias: where these might be hard to troubleshoot as well because you look at a dashboard
Matthias: or so and everything looks normal as if it was sort of working but it doesn't
Matthias: do any work it doesn't make any progress on the futures.
Kevin: That's true we have recently at least our team recently the ping core team has
Kevin: incorporated tokio internal metrics into our dashboards to give us visibility
Kevin: into these sort of things but you're right that's something before like the
Kevin: past couple months that we didn't have visibility into,
Kevin: and if we were running into that problem we wouldn't have known since then we've
Kevin: encountered a few problems in production that we've seen in in our like measurements
Kevin: of runtime queue size and,
Kevin: i've got the other metrics but basically how much is the the scheduling system
Kevin: getting backed up by all of the all of the threads being busy i.
Edward: Think we were also running into issues where we
Edward: had certain file io operations that were taking while and that for for those
Edward: tokio actually has like a separate blocking thread pool that usually doesn't
Edward: get saturated because it's really large, but may, right?
Edward: And on the note of, I would say you also brought up async cancellation as well.
Edward: I think it's really easy to mess up.
Edward: Async, you know, thoughts about cancel safety and things.
Edward: It's really easy to mess up a while tokio select loop so that you,
Edward: if any of the branches aren't necessarily cancel safe, that's not something
Edward: that you, that's purely an async Rust problem that you're, that generally I
Edward: think doesn't get introduced very well for other people who are entering Rust.
Edward: There was a great Rust talk actually by rain
Edward: from oxide that i would love to shout out because
Edward: it was really helpful in thinking about why exactly
Edward: async cancellation is is hard to reason about it's
Edward: because there's i i there are a lot of ways in which cleaning up async rust
Edward: being mindful of cancel safety being mindful of when a future is canceled and
Edward: thus cancels everything else under it, right?
Edward: That's both really, really useful in async Rust, but also very,
Edward: very easy to mess up. And it's a problem.
Edward: They had introduced this concept, which I was really...
Edward: Which helped me think about this, right? It is a problem that you can't determine
Edward: whether or not something is safe to cancel just within the function itself.
Edward: You have to look at every other child future under it and determine what else
Edward: is going to get canceled when I decide to cancel this.
Edward: And so it becomes a really hard problem to think about because suddenly you
Edward: have to think about everything else, you know, all of the global context.
Kevin: This is structured futures?
Edward: I think it's just because when a future, like child futures,
Edward: that will also get canceled, right?
Edward: When you cancel the parent future, I believe.
Edward: So it's tough.
Edward: And so async Rust definitely has its sharp edges.
Edward: That's not necessarily a tokio-specific thing, but the last bit,
Edward: the very last bit, is that actually there is, to shout out another Cloudflare
Edward: crate that we are not yet using in Pingora either,
Edward: but other services at Cloudflare are.
Edward: There's a crate called Foundations that I think helps you export a lot of these
Edward: tokio metrics and such out of the box.
Edward: So it has a lot of nice functionality to be able to do that.
Edward: And it should be a pretty minimal, again,
Edward: foundation layer for folks if you're interested in more easily exposing a lot
Edward: of those runtime operational concerns and getting observability into that.
Kevin: One thing to build on that, the one thing that Pingora does to help you avoid
Kevin: the problem of going from one machine with many cores to another machine with
Kevin: few cores, the problem about unexpected number of tokio tasks,
Kevin: is to make you be explicit.
Kevin: Like well Pingora uses tokio under
Kevin: the hood it doesn't really expose the it
Kevin: doesn't really expose tokio to the the caller instead
Kevin: it talks about things in terms of backends how many how
Kevin: many threads do you want to use on this back end and we don't do
Kevin: a default number we don't default to the
Kevin: number of cores we make you be explicit to say okay you want
Kevin: to run this tell us how many tasks you want to run and we do a lot of things
Kevin: like isolating services to a certain subset of tasks it's not like one giant
Kevin: tokio runtime which you would get if you were just running tokio main it's a
Kevin: good way of isolating business critical things from things that need to run
Kevin: in the background that can take a little extra time.
Matthias: Yeah and since we're on the
Matthias: topic of using a certain number of
Matthias: cores in rust code i always wonder why everyone sort
Matthias: of defaulted to the number of cores that you have on your system
Matthias: because if every dependency if every library does that
Matthias: you end up with a multiple a multiple of
Matthias: the number of cores you have and i'm just
Matthias: saying this that so that people are mindful
Matthias: about the resources that they request from a system and speaking of resources
Matthias: that's the other part about performance or efficiency let's say that i wondered
Matthias: about when you compare NGINX with Pingora did you or were you able to squeeze out even more,
Matthias: requests per server now that you switched over to Pingora because NGINX must
Matthias: have already handled a ton of requests I'm assuming because it was written in C.
Edward: I don't know if we were able to squeeze out more necessarily,
Edward: like people squeeze out all of the resources from us with the amount of requests
Edward: per second that they drop on our network.
Edward: But I think, again, I think the resources that we are really saving,
Edward: as far as I can recall, because
Edward: NGINX you know just bare request
Edward: processing without you know extra compute
Edward: futzing with the request processing with
Edward: lua filters or whatnot is generally already pretty efficient and
Edward: just trying to limit what it does to being pipes we we have similar goals right
Edward: We want to be as minimal as we can and just ferrying the bytes through and making
Edward: the necessary modifications on the layer seven stuff.
Edward: Now, the things that we were saving, I had mentioned earlier that we were like.
Edward: More efficient at reusing origin
Edward: connections for example you can in theory
Edward: i guess squeeze out yeah you
Edward: can you can squeeze out and save compute
Edward: on making for both yourself and the
Edward: origin right if you're if you have better
Edward: origin connection reuse when
Edward: you're trying to make requests upstream the reason
Edward: why we there was
Edward: such a fundamental difference i think that in in
Edward: the blog i think we had mentioned that we had reduced we had
Edward: lowered it by like
Edward: two-thirds the amount of origin connections we
Edward: were making the fundamental reason for that was like a fundamental
Edward: architecture reason which was
Edward: that NGINX worker processes right
Edward: because their individual processes weren't able
Edward: to share a connection pool unlike the thread-based model that we have in in
Edward: Pingora so that and where where we have an upstream connection pool that that
Edward: all the threats can on a particular server can share from.
Edward: Except in those particular design, fundamental architecture ways that we were
Edward: really conscious of when we were first optimizing for Pinguara because our team
Edward: making origin connections, that's a big deal for us.
Edward: I think generally we would expect performance to be pretty much on par, right?
Edward: With what Entirenex is doing. And that's the promise of Rust, right? You can do that.
Edward: And be just as expressive and easy to understand.
Kevin: Yeah, a lot of this comes down to physical limits. So NGINX is optimized to
Kevin: the max to the level of what you can do on a network card and what you can do
Kevin: reading files from a disk. We are limited by the same physical constraints.
Kevin: We are reading from the same network, reading from the same disk effectively.
Kevin: The place where Rust excels here is an ability to make it easy to read and easy
Kevin: to write and easy to onboard as opposed to requiring a PhD to unroll C code.
Edward: And when it comes to like implementing new,
Edward: playing with the shiny new tools that the kernel allows you to,
Edward: like uring and stuff, that's perhaps a lot easier to,
Edward: like I would certainly want to be doing that within our framework instead of
Edward: trying to roll that into NGINX, right?
Edward: That's at this point i think it's just we we are a lot more comfortable working with our ecosystem.
Matthias: Which brings us to today and just to wrap up the part about Pingora can you
Matthias: share some numbers about the project where are we at today and maybe about cloudflare in general.
Kevin: Sure. So, I mean, the first thing that I always tell people about Pingora or
Kevin: Cloudflare in general is that the teams are really small,
Kevin: surprisingly small, especially if you look at teams at other big companies like
Kevin: Amazon or Facebook or Google, there's only between six and eight people,
Kevin: depending on the time of day on the Pingora team.
Kevin: So this team that handles a large 20% of the internet traffic in the world,
Kevin: handled by six or seven people, most of whom are asleep at the same time.
Kevin: In terms of lines of code, we, for some reason, are not giving out the official
Kevin: number of lines of code in Cloudflare that is written in Rust.
Kevin: But for Pingora, even on the open source side, there's about 130,000 lines of code.
Edward: To be clear we're not the only content
Edward: delivery network team not the
Edward: only a proxy service through which you know
Edward: these these requests are passing through there there are lots of other folks
Edward: but yeah i think that where it makes sense the the engineering teams are are
Edward: definitely we have a lot of autonomy each of us as engineers and a lot certainly
Edward: a lot of responsibility and we're kind of each, you know,
Edward: driven to do what we want within the team.
Edward: So each of us carries, I think, a lot of load without trying not to stress the
Edward: bus factor, though, in that case.
Kevin: True. Yeah, the reason the team size fluctuates is because we, as a company,
Kevin: are open to working cross teams to the extent that for the past,
Kevin: I don't know, three or four months, I've not been working on the Pingora team
Kevin: and been working on the Speed team for other undisclosed yet projects,
Kevin: but are also written in Rust, more on the core side than the edge side,
Kevin: but still very interesting and all async Rust, just like Pingora.
Matthias: Amazing.
Edward: Though I don't think we can share across all of Cloudflare how many lines of
Edward: Rust code there are, I will say that Rust, we mentioned that Rust has been of
Edward: interest to Cloudflare for a long time.
Edward: Pretty much every new service on the edge is written in Rust,
Edward: I believe, unless there's some significant reason not to.
Edward: I think all of the services that are running are proof enough that it provides
Edward: significant value, especially in our performance-critical, like,
Edward: segfault-avoidant environment.
Kevin: Yeah, there are at least a few requests that go through Cloudflare that touch only Rust.
Kevin: It's not the majority yet, but it is a significant number.
Matthias: Speaking of which, have we talked about the number of requests per second that
Matthias: Pingora handles right now?
Kevin: Oh, yeah, I mentioned it briefly in a ramble. But yeah, so Pingora itself,
Kevin: there are multiple Pingora projects.
Kevin: But the one most prevalent that talks to upstream origins handles on average
Kevin: about 90 million requests a second.
Kevin: There was a blog post that came out that the Primagen read out loud.
Kevin: And he got to that number and said, wow, is that a billy? no that's a trillion
Kevin: that's a trillion requests per day.
Matthias: Wow that's crazy that's a lot of requests.
Kevin: Yeah.
Matthias: Does the Rust ecosystem cover everything that you need right now?
Matthias: Are there any crates that you wanted to mention that are amazing,
Matthias: that are invaluable for you?
Matthias: And are there any things lacking in the ecosystem right now?
Kevin: One crate that I think is sort of underutilized is the valuable crate.
Kevin: As part of the tokio project, it ties in really nicely with tokio tracing.
Kevin: It allows you to basically give a controllable summary of objects that you want
Kevin: to show up in your traces.
Kevin: It's got a usage pattern that's similar to serde. You annotate your structs
Kevin: that you want to be able to display.
Kevin: It's got some great new features that allow you to omit fields if you don't
Kevin: want PII to show up in your traces. It's really well written.
Kevin: We've added our own features on top of it that allow you to do things like when
Kevin: you have structures from external crates that you don't have access to add annotations to them.
Kevin: You can give it a special valuable annotation to instead of giving a full object
Kevin: representation of this structure, you can give it the debug representation or
Kevin: the display representation and have that show up in your logs.
Kevin: It's just a really simple way of avoiding the boilerplate that comes up with
Kevin: wanting to give a summary of an entire object structure, which I've seen in
Kevin: lots and lots of places, especially in other languages.
Kevin: You want an object to show up in multiple ways, but you can't interfere with
Kevin: how it's serialized to json so you have to go through all the boiler plate of
Kevin: writing okay this field goes in oh no skip this field it's got ip addresses,
Kevin: it's having a crate that's designed to do this and also is thoughtful,
Kevin: Because it doesn't add overhead of implementing. You're implementing it as a
Kevin: blanket trait implementation, but it's done in a dynamic way.
Kevin: So it doesn't add even a lot of monomorphism.
Kevin: It just gives you one implementation for anything that implements display or is valuable.
Kevin: It's a great crate. I can't get enough of it.
Matthias: One could say it's invaluable.
Kevin: It is invaluable, yes.
Edward: No, yeah, I'm glad you mentioned that because I feel like I'm cheating.
Edward: Everything that comes to mind is a core dependency, right?
Edward: Like, tokio, obviously, has so many great utilities for us to express,
Edward: like message passing and in async fashion, etc.
Edward: And obviously we've extolled its, I think we've sung its praises.
Edward: And other things that come to mind seem really foundational,
Edward: just like, you know, reference counted bytes, byte buffers with the bytes crate.
Edward: Very foundational. dash map
Edward: how do you get a concurrent hash map
Edward: with as little lock contention as
Edward: possible something like dash map with a bunch of shards it's it's great oh the
Edward: the other things i've already mentioned as well which is our you know our cloudflare
Edward: crates that i've already mentioned before shellflip when it comes to process restart,
Edward: graceful process restarts and foundations for various telemetry and observability
Edward: things among other operational service things.
Edward: I don't know if we wanted to shout out like some community work on top of Pingora 2.
Kevin: Oh, yeah.
Edward: Like there are originally, I think we had been working with some folks within
Edward: the Proximo memory safety org on a more batteries included actual drop in NGINX
Edward: replacement called River.
Edward: I do believe that a lot of that work maybe is on pause right now, though.
Edward: But there are a lot of other great community folks who come in,
Edward: report issues, etc., contribute, who are working on, I think,
Edward: there's this ping gap crate as well, is one of the most significant and popular,
Edward: where they've also implemented all of other, dealt with our more arcane APIs
Edward: around caching and stuff.
Edward: So definitely, that's a tremendous effort.
Edward: And I think we have been so flattered and excited by the community engagement with Pingora.
Edward: It was monumental and humbling.
Matthias: What does River do?
Edward: It is both of these projects that I mentioned, River and PingGap,
Edward: are meant to be more batteries included, NGINX, like actual binary deployment.
Edward: So Pingora is meant to be a library,
Edward: and it can be a bit difficult to work with if all you're trying to do is use
Edward: it as a drop-in for NGINX, right?
Edward: You have to actually implement all of, you know, define the proxy service and
Edward: things like that in code.
Edward: And it is
Edward: not a batteries included like plug-and-play sort of deployment where you can
Edward: just versus something like one of these other projects where you can in theory
Edward: just build it and run it as if it were an NGINX binary.
Edward: So we were really trying to build the foundations of a lot of this, of a proxy framework.
Edward: And allow the community to expand on it since we don't necessarily haven't yet
Edward: necessarily needed that generalized solution ourselves with the amount of heavy
Edward: customization and heavy like...
Edward: You know, surface fiddly bits that we do ourselves in spinning up a Pingora service.
Kevin: Yeah. And as we mentioned, we're only six or seven people. We don't have so
Kevin: much time to add additional features. I mean, we love adding features to Pingora.
Kevin: The River project, when it was envisioned, was supposed to have things like
Kevin: WebAssembly integration. So you can do all these things, but expose them as WebAssembly.
Kevin: That was one of those things that I would love to implement myself.
Kevin: But, you know, there's just, there's enough Cloudflare work to go around,
Kevin: and also it's a significant project to take on.
Kevin: The community has been really good at putting things into Pingora directly, though.
Kevin: Some notable ones that come to mind are Russell's integration.
Kevin: We internally use OpenSSL.
Kevin: The Russell's integration was a huge undertaking that one person did themselves,
Kevin: and we're very grateful to that. Harold, if you're listening, thank you very much.
Kevin: There's another similar integration for another TLS implementation, I think the AWS 2SN TLS.
Kevin: That one is still yet to be reviewed, obviously is assigned to me.
Kevin: I'm slacking off on my open source job there.
Edward: We really try to, we are really trying to stay on top of open source,
Edward: but there's only sometimes, I wish I just had more.
Edward: I think we all wish we had just more.
Kevin: Open source time. Yeah, the open source stuff is so fun.
Matthias: Yeah there's never enough time speaking of
Matthias: which we have to conclude as well because we ran out of time but it was amazing
Matthias: to talk to both of you likewise if you could phrase a statement to the rust
Matthias: community anything that you always wanted to share what would it be yeah.
Edward: No like we there's a bunch of http ecosystem things that there's There's a great
Edward: maintainer for all of it is like open source, the HTTP,
Edward: literally HTTP crate, h2, you know, a lot of those are our core dependencies for Pingora as well.
Edward: And the maintainer, Sean, is incredible at what he does.
Matthias: Yeah, I agree. Shout out to Sean.
Kevin: Yeah, the thing I was going to thank the Rust community for is for being so coherent,
Kevin: especially around HTTP things like the hyper ecosystem, the h2,
Kevin: all of those things are so ubiquitous that it makes integrating with existing projects much easier.
Kevin: Specifically, I was working with a ClickHouse client that is an official ClickHouse
Kevin: client that the ClickHouse team puts out, but I needed to add a new feature
Kevin: for rotating MTLS certificates, which obviously their client does not support.
Kevin: But because they expose access to the hyper HTTP client under the hood,
Kevin: it made it an easy thing to do.
Kevin: It's just such a good experience to come to.
Kevin: Like if you need a feature you already have the tools necessary to add functionality
Kevin: to tools that are published by other people in a coherent way something that
Kevin: you don't get in java something that you i don't know if you get in go that's
Kevin: not my ecosystem but as a former and recovering java programmer it's very nice yeah.
Matthias: That's a very nice closing statement as well Edward anything that you want to add.
Edward: I'm glad you had a specific answer because really, I am mainly just thankful
Edward: for, I mean, it is true that the ecosystem,
Edward: though I'm sure there are gaps from time to time, generally,
Edward: if you are looking for a particular pattern or thing,
Edward: you will either find out that it is hard to do so, or that someone else has
Edward: already tried to at least some extent to do it and has a working very much like
Edward: you know if not production ready nearly production ready implementation of it,
Edward: so the rust ecosystem in general has
Edward: has just been kind of the amount of excitement
Edward: that folks have within the
Edward: community is is a great sign
Edward: of sign of promise and i mean obviously i
Edward: think rust has already eaten up a lot
Edward: of the internet if we are
Edward: anything to if we are a good example but no we're we're just so once again just
Edward: so thankful that people are interested in what we do and are patient with us
Edward: and are you know are great contributors so.
Matthias: Kevin and Edward thanks so much for taking time for the interview today.
Kevin: Thanks Matthias we appreciate it.
Edward: Thank you yes yeah.
Kevin: I mean thanks for putting on this podcast Yes, I cannot believe it took five
Kevin: seasons for me to catch on.
Matthias: It's never too late. Rust in Production is a podcast by corrode.
Matthias: It is hosted by me, Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.