Stephen: Today we got steven with us steven can you introduce yourself
Stephen: and what do you do hi there i'm steven cto
Stephen: at pub nub and we work on building an edge net messaging network that we've
Stephen: invested a lot into and we actually have a lot of you know what it should have
Stephen: brought into us the number of patents that we have that run against this because
Stephen: it's really easy to connect your servers to a message bus.
Stephen: You might be familiar with like Kafka or RabbitMQ or JMS.
Stephen: These are fantastic message buses. But once you hit the scale of the web where
Stephen: there's mobile devices and laptops that need to connect in, it's a different story.
Stephen: And that's where we have some of PubNub's scale. We have over a billion devices
Stephen: connected to our network, which is not a small feat to be able to do that.
Stephen: And if you've ever pressed and dialed a call, you ever noticed that it might
Stephen: take a little while for the other phone to ring, that's a signal.
Stephen: It's actually a signal that's being delivered to the device to make that phone ring.
Stephen: And what we've done is we are able to provide a signal that happens the moment you click the button.
Stephen: We can make that signal deliver, and we can do that at the scale of billions.
Stephen: And this is something that's been very challenging because all of a sudden,
Stephen: thousands of devices, they might need to send and receive a lot of messages
Stephen: all at at once. So we have this non-homogeneous workload.
Stephen: And to be able to scale something like this has been a high challenge.
Stephen: And Rust is part of the picture. So I'm excited to talk about Rust in that.
Matthias: Amazing. That's why I wanted to talk to you because in your videos,
Matthias: you have this passion for what you do.
Matthias: And I guess you can transport it as well.
Matthias: And at the same time, PubNum seems to be a company with a lot of scale and something
Matthias: that you don't usually find.
Matthias: And it's a pleasure to talk to you about maybe scalability things and also how
Matthias: latency is important to you because it feels like latency is also a big part
Matthias: of that equation is that correct.
Stephen: Yeah, it's huge, especially when we're, you know, we're in a prospect conversation.
Stephen: We've got customers that are
Stephen: evaluating us and speed tends to play a nice little bonus on top of it.
Stephen: Now, you might not always think that, hey, some use cases don't necessarily
Stephen: need to be at light speed, though it is that much better when you have the experience
Stephen: to say, oh, I clicked the button.
Stephen: Like, I didn't even see it. It already happened. And so you have to really pay,
Stephen: it's a great experience.
Stephen: And so performance is a huge aspect that we're, so that's how we architected.
Stephen: We had to go a step further.
Stephen: So PubNub is not just, you know, like a server sitting in somewhere and devices
Stephen: connect into, we're globally distributed. We run an AWS.
Stephen: We have Kubernetes clusters spread across all of Amazon's zones and using GeoDNS,
Stephen: you you connect to the closest region.
Stephen: And what that allows you to do is have a really good latency experience running on Amazon's network.
Stephen: You can have users that are nearby have the best millisecond latency,
Stephen: sub-millisecond latency.
Stephen: It's amazing the kind of performance that you can get. And then when users span
Stephen: across continents, you're running on Amazon's network then at that point.
Stephen: So you still have a fantastic latency even across the world.
Stephen: What sort of background do you need to do all of
Stephen: that maybe we haven't really properly introduced yourself
Stephen: and your background so maybe we can tie this in
Stephen: somehow yeah it's a good
Stephen: question it you know here here's the deal like all the efforts that you jump
Stephen: into over the years and have these these experiences i think there's some level
Stephen: of value to that i think really the biggest biggest value that anyone can have
Stephen: is what's accomplishable today.
Stephen: Like what are you able to achieve today? And really it's all about putting your effort into it.
Stephen: So whether you have no experience or a little experience or a lot,
Stephen: it really comes down to what is going to happen today because you could have
Stephen: this huge amount of experience and then just sort of sit back and not do anything.
Stephen: You could just relax. And then at that point, what's the point of the the value.
Stephen: You're bringing not much to the table and you need to be able to really dive in and bring value in.
Stephen: It really doesn't matter about that level of experience.
Stephen: And so being able to build something like PubNub required that level of diving.
Stephen: You really have to stand in, read the Linux documentation, which kernel APIs do I need to worry about?
Stephen: How does the buffering work over over here in the network layer.
Stephen: You need to just stack all that up and then write it down and then build code
Stephen: that runs on Linux, what we have, EPOL. We're running an EPOL.
Stephen: It's the primary speed performance boost that we have.
Matthias: And so- Can you explain what an EPOL is and why it's important for PubNup?
Stephen: So for PubNub, when you have millions of connections coming in, one server can't do it.
Stephen: However, the more connections you can have on a server, the better your margins
Stephen: and the higher scale you can offer and offer a better price to your customers.
Stephen: So in order to track those connections,
Stephen: you don't want a process that's actively looking at each of those connections
Stephen: on a box because that will burst your CPU. You'll immediately run out of CPU. for you.
Stephen: What you need is an interrupt that the kernel provides through APIs,
Stephen: through different polling interfaces,
Stephen: which is exactly how all the devices on your motherboard work when it's communicating
Stephen: to your devices. It's all through interrupts in the kernel.
Stephen: So you press a button like the different keyboard presses on your keyboard.
Stephen: Those are interrupts that come through, but there's nothing that's actively checking.
Stephen: Did you press the H key? Did you press the J key? it
Stephen: only goes through interrupts and that allows you to have an
Stephen: efficient communication through the hardware on the
Stephen: system and so this is what epol is
Stephen: it is an interface that allows us to receive those
Stephen: interrupts through the os bind them onto network
Stephen: events and then be able to act on them or emit data over over the bus and get
Stephen: all sorts of scale it's it's night and day if we didn't have this it would be
Stephen: a lot more expensive to run pub nub from from what you tell me it,
Stephen: sounds like you've always been interested in performance interested in these
Stephen: technical problems and and how to solve them if did us if that is correct when
Stephen: did you first get in touch with rust and how did it feel like and also what
Stephen: did you do before rust so today pub nubs written written in C,
Stephen: which is been, you know, there's there are challenges with that.
Stephen: We've got all sorts of stability over the years that we've nailed down,
Stephen: and we've got fantastic performance.
Stephen: Rust came into the picture when one of our senior engineers was like,
Stephen: hey, check this thing out over here. There's this thing called Rust.
Stephen: And this was maybe, I don't know, five years plus, a long time ago.
Stephen: And we were all like, yeah, yeah, that's Rust, sure.
Stephen: What is this thing? No one really knew. But it turns out the capabilities of
Stephen: Rust with the memory safety and the performance that we get near C speeds is
Stephen: really attractive to us.
Stephen: That level of capability is huge, especially with the compiler being able to
Stephen: detect some common problems.
Stephen: That level of attractiveness at our scale brings in the reason why we brought Rust in, basically.
Matthias: Okay and before we talk about
Matthias: rust i'm curious about the c code
Matthias: as well because when you run such a platform you're
Matthias: bound to run into runtime problems as well
Matthias: and you probably had your fair share of maybe outages or maybe runtime issues
Matthias: maybe can you tell us a few stories that you you encountered through all these
Matthias: years with your Rust and with your C experience?
Stephen: Yeah, absolutely. So you run into these things called segmentation faults all the time.
Stephen: And you get these things that are called core dump files, which is all the memory
Stephen: that the current application at the time had.
Stephen: And then that saves that to dist so you can go in and debug it.
Stephen: You don't want to be in that scenario because that means that you've potentially
Stephen: damaged or data can be lost. There's problems in that scenario.
Matthias: And how does it work then? Did you have an on-call rotation?
Matthias: Did you have a lot of outages or a lot of incidents?
Matthias: And how has that changed since then?
Stephen: Yeah, a lot of problems over the years. Yeah, absolutely. Absolutely.
Stephen: Hitting scale, running into memory pressure, hitting.
Stephen: So with PubNub, you can multiplex data on a single connection,
Stephen: which means that you need to subscribe to certain topics or channels.
Stephen: We call them channels in PubNub. You can also apply filters to those channels
Stephen: and say, I only want to receive messages that have this tag or that are beyond
Stephen: this threshold based on this floating point value. So you can make those subscriptions.
Stephen: The problem comes in, how do you
Stephen: manage the memory when you're streaming this
Stephen: data over through a network across systems in a distributed
Stephen: fashion and then multiplex that data
Stephen: in but then still you have to deal with memory management and cleanup afterwards
Stephen: and so it's really easy to free some data or free a pointer or something but
Stephen: you really needed to keep that alive for a little while and did you have the The opposite problem,
Stephen: too, where you were leaking data over a longer period of time. Oh, yeah, that's right.
Stephen: We had that problem, too. We absolutely had that problem. We were using memory
Stephen: allocators that were performing like JMALEC.
Stephen: Even in that case, we were in situations where memory just kept growing and...
Stephen: It's it's a tough situation because you run things like valgrind
Stephen: and you like dive in and you want to find where
Stephen: those leaks are because you know it's causing slow it's it's expensive you don't
Stephen: want that because it's memory is not cheap and so in order to stay up and operating
Stephen: you have to buy more memory to outpace the growth that leak and then that buys
Stephen: you time to find the problem.
Matthias: So on one side, you have a very performant language like C, which saves you
Matthias: hardware costs, I guess, because it's really efficient.
Matthias: And on the other side, you have engineering costs because of these issues.
Matthias: And I wonder how you balance it out, if it evened out at the end,
Matthias: or if you noticed that the engineering costs were significantly higher than,
Matthias: say, the hardware savings. And what's your take on this?
Stephen: If you run such a platform at scale yeah right yeah it's a good question so,
Stephen: it's it's i because we've been
Stephen: doing this for 14 years we've we
Stephen: have time and experience under our belt and so
Stephen: we've been able to create a stable system over the
Stephen: years that is the most powerful most
Stephen: popular most heavily used api in
Stephen: our current suite of apis how did you hire
Stephen: for such positions did you go to hardcore network
Stephen: admins did you go to fang companies
Stephen: that had staff engineers with that experience how did you find such people even
Stephen: yeah it's all the recruiting patterns that we possibly could take advantage
Stephen: of because finding c experts today even back uh even back 10 years ago was a
Stephen: challenge because even if you were a C expert.
Stephen: You don't really might want not want write C anymore because you know all the
Stephen: challenges and problems that come with it. And especially in a system like the
Stephen: web where you need to have something that's active and always alive resilient
Stephen: and it's you know we can rely on it,
Stephen: that's a scary proposition. It's like okay so we're gonna have to
Stephen: write some C that's super stable and any bit
Stephen: of latency is going to be noticed. So it's like
Stephen: the worst possible situation.
Stephen: Did you ever
Stephen: have that situation where someone came into a job
Stephen: interview applying for a role as a C
Stephen: developer and actively saying by themselves that they don't want to use the
Stephen: language anymore for such problems and they wanted to try something else or
Stephen: was it more like they learned on the job that C at scale was hard. They probably
Stephen: knew but it before so.
Stephen: Yeah they knew
Stephen: ahead of time yeah. They had the experience because
Stephen: it's really I think even if you're getting started as a C
Stephen: developer you'll immediately run into the segmentation fault
Stephen: or something else like it is a it's a rite of passage something that will happen
Matthias:
Matthias: Yeah the question is not if but when that's correct. The
Stephen: it's like guaranteed.
Matthias: other thing I wondered about was kafka is written in Java and then there's this
Matthias: new competitor I guess it's called NATS which is written in Go.
Matthias: And did you ever consider to use any of these languages?
Stephen: Java wasn't on the list, mostly because that was mostly an academic language,
Stephen: at least for me at the time.
Stephen: Right. And that was, it seemed not like the right language if I wanted to get
Stephen: the best possible speed and performance for exactly what I was looking for.
Stephen: So I wanted something that was, you know, robust and powerful.
Matthias: And Go? What about Go?
Stephen: You know, Go wasn't, I didn't know about Go back then. So that didn't even make
Stephen: the list because I didn't know what it was.
Matthias: I guess by now you know Go and maybe you have a couple of thoughts about it,
Matthias: especially comparing it with Rust, for example. What's your take on this?
Stephen: I like the compiling speed of Go.
Matthias: It's true.
Stephen: Yeah, it's boom. It's almost immediate compared to like Rust, for example.
Stephen: Example so it but otherwise we still we so we have a service in production with
Stephen: go we actually have a couple services in production with go but you still run
Stephen: into runtime errors with go like that was something that's like oh it's a compiled
Stephen: language and it's more modern you know it's a little bit of pythonic and you
Stephen: know the the syntax is a little more streamlined but,
Stephen: there's it's still easy to get verbose with the language it's easy to over abstract a
Stephen: nd it still runs into runtime errors.
Stephen: The performance is okay in production, but it has these GC pauses that we noticed.
Stephen: We did attempt to rewrite part of our PubSub bus in Go, and it couldn't even
Stephen: come close, not even close in performance.
Matthias: Did you have a one-to-one comparison? Did you run a benchmark?
Matthias: And how did that look like?
Stephen: Yeah, latency was immediately 10x slower just out the gate, even at low scale.
Stephen: And latency was you know it's it kind of tails up into you know how much further
Stephen: we can push the thing a messages per second were like was it 5x less right,
Stephen: it's yeah and then there were the gc pauses so just suddenly latency would spike periodically.
Stephen: And that was not great either right so
Stephen: we've got this thing that's over here we want to read and we actually the
Stephen: question you would ask is like why would we choose to attempt
Stephen: to rewrite something that we have that's robust and see and it's
Stephen: stable rewrite it and go and the
Stephen: answer is maintainability because we're scared to touch it it's
Stephen: working right now and if we make any adjustments you
Stephen: know that's the danger zone and how did
Stephen: you go about like even or how
Stephen: would you go about rewriting parts
Stephen: of that in rust if that is even an option for you because
Stephen: certainly for companies it is
Stephen: just a cost to rewrite anything and
Stephen: it's also a risk you need training you need adoption you
Stephen: need trust in what you
Stephen: build and what you deploy does it even make
Stephen: sense for you right now and if so how
Stephen: would you address such a topic yeah it's
Stephen: it's as we go so right now rust is the
Stephen: most popular language at PubNub by far
Stephen: all of our new services are typically elected
Stephen: to be written in rust everything going forward will be rust and this is due
Stephen: to our scale and you know we've seen fantastic results from it we actually i
Stephen: actually have some real world numbers as we have replaced some of our services
Stephen: with Rust and it has been fantastic.
Matthias: Can you share some of these numbers? I would love to hear them.
Stephen: Yeah. Yeah. Okay. So some of our services were written in Python.
Stephen: Has some performance considerations in general with Python.
Stephen: We have gotten past some of those was using something called pi
Stephen: pi which is you know what is it more about
Stephen: memory route routing and pathing but
Stephen: at runtime and so like a just-in-time compiler
Stephen: for python exactly yeah and there's
Stephen: a trade-off though because it takes up a lot more memory
Stephen: okay like a lot like it can
Stephen: go gigabytes per process but you
Stephen: know the cpu is you know five times faster which is in cpus are are expensive
Stephen: right you pay around four pennies per hour for cpu on amazon for cpu and then
Stephen: memory is what it's like half that price right based on so it's like a good
Stephen: trade-off in terms of cost,
Stephen: when when we replace that service with rust we saw another 5x boost not only
Stephen: in memory but in performance as well on top of it that means you use one-fifth
Stephen: of the memory in rust compared to of PyPy? Well, actually, way, way less.
Stephen: It's more like 30 megabytes or something like that in Rust versus a gigabyte.
Stephen: Because PyPy, it likes the memory. It keeps, it's a memory fan.
Matthias: Would you say over-provisioning is fine for some of your problems?
Matthias: Where, let's say, you have a big event and you know exactly that this is upcoming
Matthias: and then you need to prepare the infrastructure for it.
Matthias: Do you usually over-provision for such a case? And in this case,
Matthias: you scale it up to whatever memory it needs, times two, for example.
Matthias: Or would you say, no, this is exactly what ruins our costs.
Matthias: And I guess we want to be as close to what we need as possible.
Matthias: Where do you draw the line?
Stephen: Oh, it's a business question. Yeah, that's a good one.
Stephen: Because, you know, we want to make sure that we're making profit.
Stephen: We want to keep pushing forward, continue innovating. And that's where our source
Stephen: of funds allows us to do that.
Stephen: And if we're spending all of our, if we're getting like, here,
Stephen: Amazon, take all the money, give all of it over to Amazon, then we won't have
Stephen: any to continue to grow the business.
Stephen: So yeah, that's, that is, that is a trade off, right? That's a balance.
Stephen: How far do you go? Something that we took heavily in investment was Kubernetes
Stephen: and auto scaling with HPAs.
Stephen: So these days now, PubNub, we don't have to do any extra beforehand readiness.
Matthias: What is HPA?
Stephen: Horizontal Pod Autoscaling.
Matthias: And how does it help you in this case?
Stephen: It auto scales for us okay based on
Stephen: memory usage or cpu usage or do we have any
Stephen: yeah it could be both and custom metrics too so based
Stephen: on network load based on connectivity number connections based
Stephen: on the cpu usage we can
Stephen: target our scale tart our
Stephen: desired amount of resources that we purchase
Stephen: and this is really cool and now the way
Stephen: we spend like you're talking about our budgeting how much do we pre-purchase we
Stephen: don't need to do that ahead of time say if there's a big event
Stephen: what we do is we just think okay how much
Stephen: it's a throttle it's a yaml config file for us
Stephen: we use helm for helm charts we just say okay let's bring it up to 50 or bring
Stephen: it down to 20 like it depends on this on which service we're we're optimizing
Stephen: and then that allows the system like okay when When it hits this threshold across
Stephen: the pool of pods as an average, I will add more.
Stephen: Or if it dips below, then I will subtract some. So that way it's always at this particular level.
Stephen: And so now we don't have to pre-scale. We just tune with levers.
Matthias: So you got predictable performance, especially for the new services,
Matthias: which going forward will be written in Rust. Rust.
Matthias: And coming back to the original question, how would you integrate a new service
Matthias: into your current infrastructure?
Matthias: Say you have a Rust service and then you have all of the previous services that
Matthias: you have. Do you connect with them on the network layer?
Matthias: Do you integrate them with FFI, like the foreign function interface and start to...
Matthias: Yeah compile things and maybe use language
Matthias: bindings or or what's your take on this do you
Matthias: do it through maybe your own
Matthias: channel and you send messages and you do some sort of event sourcing like is
Matthias: it a hybrid of these models how does it look like in the back end.
Matthias: Yeah that's
Matthias: that's that's a strong question depends on the the situation right so there's
Matthias: two kinds of typical models that that you see in the industry,
Matthias:
Matthias: and we leverage both. One model is, you know, a request and response based through an API.
Stephen: That's our primary, like offering. And so if we were using when we're using
Stephen: Rust, for example, we use axiom as a API framework.
Stephen: And then for our, our event driven, which is the other approach, right?
Stephen: So you've got these other ones, we've got web and event. for event we run off
Stephen: of msk through kafka with which is like amazon's competitor to confluent and
Stephen: we run off of that as an event source and that allows us to process that data,
Stephen: asynchronously out of band from the system right so you've got these inline
Stephen: api calls and then you've got these asynchronous background workers that work
Stephen: in the event sourcing and how do the the workers communicate internally?
Matthias: Do you use JSON or Protobuf or Cap'n Proto or something else?
Stephen: Yeah, it's Protobuf. For some others, it's JSON, right? So it's, but it's, yeah.
Matthias: I would assume that you use Protobuf for some of the more, let's say,
Matthias: high traffic topics or maybe the ones that need the most stability.
Matthias: That's also an important factor, right? Back at my previous job,
Matthias: we used JSON and then moved to Protobuf.
Matthias: And what we found was that the number of errors would go down because you would
Matthias: handle all of the edge cases more or less at compile time because you knew the binary format.
Stephen: It and there were way less surprises at runtime did you see the same what's your take on protobuf,
Stephen: yeah that's that's a good question usually the project that
Stephen: we had we didn't migrate like you did so i don't have a comparison between the
Stephen: two um i can we can see both sides and they're both doing what they do protobuf
Stephen: is more more performant with us pubnub our primary api is json based for for our customers,
Stephen: as you're connecting to devices and sending data between them.
Stephen: Right. So even if you're connected up through an IOT device,
Stephen: like August lock is one of our customers, you can send a message from your phone
Stephen: to trigger your door to open to unlock.
Stephen: That is, yeah, that that's JSON, right?
Stephen: You can choose to also make it
Stephen: a string or basics 64 encode it to have maybe more you know
Stephen: shorten that down but for us
Stephen: since everything's in json and it's a lot of its text of most of its text protobuf
Stephen: doesn't change much of that because all text based but things like maybe a boolean
Stephen: where it would be the best right because you've got these strings like true
Stephen: or flase right those are big character strings that can be,
Stephen: Protobuf can really shrink those down.
Matthias: Yeah, yeah. But at the same time, you probably have compression enabled anyway, so you use Gzip.
Matthias: And then since in big payloads you have some repetition, you probably benefit
Matthias: from this as well. So it's not...
Matthias: I can see a lot of advantages on both sides. It's very true.
Matthias: Speaking of advantages and coming back to Rust for a little bit,
Matthias: I wonder if you could pick only one trade of Rust, whether it might be performance
Matthias: or stability slash robustness.
Matthias: So on one side, you have raw performance.
Matthias: On the other side, you have operational robustness.
Matthias: What would you pick? It could only be one of these.
Stephen: I'm leaning towards performance. I think performance I mean yeah because obviously
Stephen: we chose C at the algo right and then most of the internet still runs on C today
Stephen: that because of the scale that we're at performance is paramount.
Stephen: Yeah and and then,
Stephen: the first endeavor you had with Rust what was it what was the first project
Stephen: you tried at PubNub and you obviously thought it was a success because you moved forward with it.
Matthias: Was it like typical CLI app or was it a service that you started or how did it work?
Stephen: Yeah, good question. First, Rust at PubNub was talking about gRPC.
Stephen: We were offering and looking to offer a gRPC endpoint that you can communicate
Stephen: with devices outside the network on like things on cell towers and things like that.
Stephen: We added that. that we built grpc.publim.com with Rust.
Stephen: You can connect in, yeah.
Matthias: And that was pretty early in this gRPC world for Rust because the ecosystem
Matthias: probably back in the day wasn't as mature as it is now, if you want to call it that.
Matthias: And gRPC, just for the people that don't know, it's, I guess,
Matthias: remote procedure calls, and it comes from Google.
Matthias: I think this is what the G stands for. Correct me if I'm wrong.
Matthias: But it's basically protobuf over HTTPS and HTTP2, I guess. Is that correct?
Matthias: Yep. And you provided that service. And why did you pick Rust for this specific task to begin with?
Stephen: That was an engineer I was talking about earlier. The senior engineer was like,
Stephen: hey, did you hear about this thing called Rust?
Stephen: And then two years later, after, there's like, okay, let's build the service
Stephen: in Rust. Let's just do it. Let's see how it goes. And we did it and it worked.
Matthias: Were there any setbacks or
Matthias: was there any pushback from the team to maybe move
Matthias: away from an existed tried and tested technology and
Matthias: moving towards rust were they skeptical
Matthias: was there any resistance so to say there is
Matthias: a surprising amount of excitement i would
Matthias: say yeah and and just eagerness to
Matthias: move forward with the next generation technology
Matthias: which super endearing like
Matthias: and did you set any goals or
Matthias: expectations before the project for example do you say oh yeah all the tests
Matthias: need to pass or it needs to be integrated with our cicd pipeline or did you
Matthias: say well it's such a new technology anyway you cannot really set those standards
Matthias: because you need to try and experiment And if you set two stringing goals,
Matthias: then what you end up with is maybe the same solution that you had before.
Matthias: So let's just see what it takes and run free.
Stephen: Yeah, that's a lot more of my philosophy is the latter there is like,
Stephen: hey, let's see how far we can push this.
Stephen: Let's see where we can go. Let's be pioneers here as well.
Stephen: And, you know, it turns out Rust, I think, and I believe is that next,
Stephen: it is the, you know, multi-billion dollar language that if you take advantage
Stephen: of it, you will profit from it.
Stephen: Like that is, it's no doubt in my mind, we're already seeing it now.
Stephen: It's fantastic. fantastic would you say that
Stephen: is true for things outside of infrastructure or
Stephen: for infrastructure specifically oh okay
Stephen: so all right let's think
Stephen: about like devices in the world that might be running rust right like maybe
Stephen: your car or maybe some there i mean we need to consider binary sizes and other
Stephen: kinds of hardware limitation that comes into the picture and obviously we're
Stephen: in you know almost 2024 now.
Stephen: And we've got supercomputers of all sorts of shapes and sizes and everything's really fast and small.
Stephen: We're in the future, though it's still...
Stephen: We might as well take the best advantage that we can. And, you know,
Stephen: I haven't tried, how small can we get rust?
Stephen: Can it go pretty small? I've seen usually pretty big binaries that kind of look
Stephen: like they're scary, a little scary to have on like IoT devices.
Matthias: And would that be a use case for PopNub to deploy to IoT devices?
Matthias: Is it something that you do or have on your roadmap?
Stephen: No i'm just thinking like
Stephen: how far can we stretch the rust like how far can it go now
Stephen: obviously we can it can basically run everywhere i would say any device and
Stephen: it can be fine and you know maybe we can optimize its footprint over time but
Stephen: for pub nub and our api service being a communications company where we need
Stephen: to have that reliability and of speed and performance,
Stephen: no doubt Rust is our top choice going forward.
Matthias: Maybe let's turn it around and say, do some of your customers use it for IoT devices already?
Stephen: They do. Yeah. Actually, now that you mention it, we do have a customer and
Stephen: they've actually been pushing us for years for Rust SDK because they have their
Stephen: devices in IoT device that's deployed with Rust.
Stephen: You know, I didn't even think of that. and they chose rust because it's
Stephen: got that stability reliability and yeah
Stephen: we we did we delivered them a rust sdk so
Stephen: it can communicate because they had to do it by hand before
Stephen: and now with the rust sdk it's even
Stephen: cooler with rust and you can integrate that into existing c applications right
Stephen: so you're mentioning ffi and things like that so this is a neat strategy that
Stephen: that we're looking into to make sure that our existing customers that are C-based,
Stephen: which we have several that are still in C, they could actually just rely more
Stephen: heavily on our Rust SDK that we spend more time on.
Stephen: Because I like that. Now we can start putting... Where else can we put Rust? How far can it go?
Stephen: Let's bring it everywhere.
Matthias: True. It seems like since this is open source, And yeah, I take this way because
Matthias: I read the blog post about it, but it is open source.
Matthias: And the beauty of it is that people could look at it and integrate it into their projects.
Matthias: And it could be some sort of standard or protocol that you use to communicate with them.
Matthias: And I guess developing such standards in the open is a huge advantage as well
Matthias: for you from a company perspective.
Matthias: And we talked a lot about the upsides of Rust.
Matthias: Of course, there are still downsides.
Matthias: There are things that are just trade-offs we need to be aware of.
Matthias: A couple of things that I can think of, for example, might be if you're too
Matthias: much on the bleeding edge, then you might at least risk instabilities or you
Matthias: might risk some churn from time to time.
Matthias: Because not every introduction of a new technology runs very smoothly.
Matthias: But I wonder what what your take is on this is that really a huge problem in
Matthias: Rust's case did you have other transitions which were easier or harder and also
Matthias: would you even say that there are,
Matthias: downsides or what would be the downsides of Rust that's a
Matthias: lot of questions yeah yeah i'm really
Matthias: thinking about this you know i was reading your reading your "
Matthias: Why Rust" article and the part that
Matthias: that stood out to me was how NPM migrated
Matthias: to Rust yeah what I
Matthias: mean their whole thing is JavaScript like why that was so great is that as they
Matthias: really did that that's been they did that in 2019 and one thing that i was extremely
Matthias: surprised by was they do 1.3.
Stephen: Billion javascript package downloads
Stephen: per day which is incredible yeah
Stephen: that's amazing and okay rust
Stephen: makes yeah rust the win for sure yeah that's
Stephen: another win for rust and what are the losses what are the downsides yeah are
Stephen: there any downsides i mean it the compiler does take a while but you know when
Stephen: you're doing iterative compilations when you're locally developing it's fine
Stephen: it's like a couple seconds here or there so that that's that no worries on that.
Stephen: From a stability perspective, the compiler itself, even if you're bringing in
Stephen: like alpha software or alpha packages, like with a 0.0.1, like version number,
Stephen: it's the compiler is still doing a good job there.
Stephen: Right and so it's very likely that as
Stephen: long as you're just you know leaning on the rust
Stephen: native primitives for concurrency then these
Stephen: i think that it's it's wild west at that point you
Stephen: don't really have to worry about it because you're going to have that compiler safety
Stephen: so i i think that's also okay right
Stephen: i mean tell me i'm wrong like i think that's i fully
Stephen: agree 100 what about the the ecosystem did
Stephen: you miss any packages anything that's like could
Stephen: be better from an ecosystem perspective you mentioned
Stephen: pre 1.0 packages is that
Stephen: a huge problem for you or would you say no actually you're pretty much covered
Stephen: getting rd kafka lib rd kafka i would say that one's usually a pain to get to
Stephen: work every time a developer tries to set up a dev environment or get it to deploy
Stephen: into production or integration,
Stephen: they have trouble.
Stephen: I do have trouble too. It's like, oh, great, libRDK. Okay.
Stephen: So there are rough edges into getting things to work.
Stephen: But as soon as you get all the right settings tweaked and dependencies ready,
Stephen: it does exactly what it needs to do.
Matthias: I can see why libRDK might be a problem. I like it,
Matthias: but as far as I can remember, Remember, it has bindings to other C libraries
Matthias: like OpenSSL, and this makes it extremely hard sometimes to pin the correct version in LibRD Kafka,
Matthias: which you will use from the Rust side at some point.
Matthias: And then you need to kind of transitively talk to the C compiler to do the right
Matthias: thing and to link against the correct libraries. Yeah.
Matthias: Yeah, it can be a bit of a pain, I guess. Yeah.
Stephen: So there's those challenges. Oh, okay, I got another good one for you.
Stephen: I'm a proponent of simplicity and keeping things as simple as possible.
Stephen: And even, you know, as we're writing code, abstractions typically,
Stephen: though they sound nice and fancy, they are, they add this debt over time.
Stephen: And I would say it's better to have repeated code and not have super abstractions.
Stephen: But rust comes with
Stephen: all sorts of fancy bells and whistles doesn't it what
Stephen: are these generics and traits and all these fun
Stephen: things and it's really easy to be attracted to
Stephen: those because they look appealing and they do bring some benefits
Stephen: they add a huge amount of overhead for
Stephen: bringing new people into the team and as we're
Stephen: training on the job that level of complexity just
Stephen: adds a huge barrier to entry and rust is massive for
Stephen: that that's probably the biggest downside can i have
Stephen: like an easy rust i want like
Stephen: a simple rust that's what i want well there
Stephen: is one guy david mccloyd i'm not
Stephen: sure if i pronounced the name correctly but he recently wrote a book about simple
Stephen: rust And I guess the idea was to go and maybe learn Rust up to a point where
Stephen: it's usable for solving your problem, but not go too far.
Matthias: And certainly, I agree that there is an issue with too many abstractions.
Matthias: And it feels like Rust attracts people that like abstraction.
Matthias: And then at least their second Rust program is barely readable.
Matthias: And the ad trades and so many abstractions until they come back to the essentials
Matthias: and work on the core of the idea again.
Stephen: Yeah, that's much better. Just keeping things simple, it's maintainable.
Stephen: Anyone can come in and read it.
Stephen: There's not this giant barrier to entry.
Matthias: Did you notice that, from your perspective, is asyncrust another such dialect
Matthias: which makes the language harder to understand?
Matthias: Did you get in touch with asyncrust and what's your opinion on that?
Stephen: You know, I would say async helps helps because having to deal with futures
Stephen: was always very annoying.
Stephen: And I remember early days, even now, when I was running the compiler and anything
Stephen: to do with the future, the compiler, it always tries to be helpful.
Stephen: But if it had a future involved, it would just print out pages of unintelligible,
Stephen: like there's no way to debug it, like even close.
Stephen: Have they fixed that? I don't know. Is that better with futures now?
Stephen: Because I haven't run into it myself personally recently.
Matthias: Yeah it is gone it has gotten much
Matthias: better of course you don't have to write futures yourself
Matthias: anymore i'm not sure if you ever did that but it's not fun now the language
Matthias: helps you a lot more and the error messages are still not ideal but they get
Matthias: better yeah i love it they're like hey there's this thing do this right here
Matthias: and that'll fix your problem i'm like okay i'll go and type type it in and do it.
Stephen: And it's like, error. Nope, you did it wrong. You did it totally wrong.
Stephen: And it's like the compiler is yelling at me, but I just did what you told me to do.
Stephen: Why isn't it working?
Stephen: And so that still happens today with myself. I still run into that problem.
Matthias: How do you teach people on Rust? How do you train them?
Matthias: And how do you spread knowledge across your company?
Stephen: Good question. So it's all on the job training. We also have community practices
Stephen: where we jump in and say, Hey, learn these cool things about rust.
Stephen: And we try, we try to keep that going. We also have a dedicated Slack channel
Stephen: where we post all sorts of things.
Stephen: We've been posted links to one of your articles.
Stephen: One of our engineers jumped in there. It's like, Hey, look, there's this article
Stephen: over here. Check it out. And it ended up being Matthias's article.
Stephen: Like, Whoa, check it out.
Matthias: Thank you for that.
Stephen: It yeah so it's it's a lot of knowledge sharing and that sort of thing was it always the case,
Stephen: did this first engineer that wrote
Stephen: the first rust service also share their experiences with
Stephen: the team and was it something that kind of
Stephen: came naturally and they talked about it or was it
Stephen: enforced by the organization to keep spreading
Stephen: knowledge it was encouraged mostly informal
Stephen: so it started informal and organically
Stephen: and then more heavily encouraged you
Stephen: know mostly i'm a big champion so i
Stephen: wanted to push that as much as possible and encourage rust where at all possible
Stephen: because it's there's there's profit there there's gold the rest is like this
Stephen: nice uh from a business perspective perspective you would only choose rust for
Stephen: everything basically all scenarios including ai,
Stephen: but it's all python these days right yeah yeah and do you see future where ai
Stephen: and rust come together and and maybe the infrastructure or the platform that
Stephen: ai runs on is written in rust.
Stephen: So today it's written mostly in C, right?
Stephen: So even if you're running Python, you're probably using LibTorch,
Stephen: which is Facebook's framework that is written in C and it communicates the GPU.
Stephen: And most of the latency is in the round trip between the system bus and the
Stephen: memory that you have for those matrices, having to pipe that over to the GPU
Stephen: to run some multiplications.
Stephen: And so that latency is the main bottleneck these days.
Stephen: And through various pipelining techniques and texture memory buffers,
Stephen: you can provide some major improvements there.
Stephen: Even with Python, which is the main language these days, you're still mostly
Stephen: using C libraries at that point, right?
Stephen: And so my thought is, how much faster can we make it if we do this in Rust?
Stephen: Can we bring it even further? And there are some wins there,
Stephen: because as you're doing data processing, data cleaning, and all sorts of other
Stephen: things that need to go through the pipeline, those are non-GPU tasks for the part.
Stephen: And so if we start getting more rust into the ecosystem, do data cleaning,
Stephen: data processing, embeddings, right?
Stephen: Those things are non-GPU tasks, they're CPU tasks because we need to convert
Stephen: various organic data, which is going to be like text or images or pixels.
Stephen: Pixels, we need to convert that over into embedding format, which is,
Stephen: you know, dense matricy that we can then pipe over to a GPU.
Matthias: Yeah.
Stephen: This is where Rust can start shining beyond the existing pipelines with Python.
Matthias: Because it would reduce the memory that is required for these transformations
Matthias: or for some other reason?
Stephen: Memory performance, more organic performance there.
Stephen: And you know we do see was it numpy and
Stephen: some other python libraries are also c-based so
Stephen: you're still getting some c-based performance there but there's
Stephen: still the overhead of the communication between you know the python vm and the
Stephen: c library dependencies so i think there's just a better chance to go more native
Stephen: rust in terms of performance especially as we're starting to get bigger right
Stephen: so nvidia just announced their h2s which are almost double the amount of giga
Stephen: It's like 141 gigabytes for GPU.
Stephen: It's like a huge amount of memory.
Stephen: That just means a lot of data. And the more we can save in performance,
Stephen: Rust, I think, can take us that much closer.
Matthias: Time to order one. I know very little about machine learning, to be honest.
Matthias: But one thing I heard about is that in the data science area,
Matthias: there are a lot of improvements being made by people that use Rust for the first
Matthias: time for some of the lower level libraries.
Matthias: I might be wrong here, but I guess Parquet is written in Rust now or parts of it.
Matthias: And Parquet is this data format, which is, I guess, column and row based.
Matthias: And it's kind of efficient if you want to do very typical data science transformations.
Matthias: You see, my knowledge is pretty limited here, but it feels like Rust is getting
Matthias: into these areas as well.
Stephen: Well that's a good way to get rust into things is make it as a library that
Stephen: you can import into other languages so it becomes a foundational technology
Stephen: that you cannot escape from anymore i like that it's like you're getting rust
Stephen: whether you like it or not,
Stephen: it's all rust in the end will you say there's still room for improvement for
Stephen: tooling in general I'm talking about debuggers or profilers and things that
Stephen: might help you in your day-to-day.
Stephen: I think the biggest one would be, so you can set up a linter,
Stephen: you can set up your linting configuration to have certain standards and be more
Stephen: strict about certain things, right?
Stephen: Like guarantee no ability to use the word unsafe anywhere ever, right? Yes.
Stephen: That's a really good one. Make sure we have that.
Matthias: That but also i want a linter that more
Matthias: strictly enforces like maybe a lego version
Matthias: of rust a simple like a python version where it's
Matthias: just really super duper simple and i think you know
Matthias: that's something that might already be available to us today i just hadn't really
Matthias: jumped in to investigate it much but i
Matthias: think something like that where we just have like easy simple
Matthias: rust and the linter will check that to
Matthias: make sure hey you're using something that's a little too complex go this
Matthias: direction instead using something more simple even if
Matthias: there's a little bit of repetition i think it's it's a
Matthias: lot more worthwhile this is a
Matthias: brilliant idea i never thought about that but it
Matthias: makes total sense the tool
Matthias: would simplify your code it would be like
Matthias: a refactoring tool which instead of making
Matthias: code more abstract it will make it more readable
Matthias: more maintainable and more testable
Matthias: and maybe give you advice i wonder
Matthias: how such a tool could look like but actually these
Matthias: are things that i personally believe that
Matthias: ai might help us with to see those patterns and say oh yeah maybe this has higher
Matthias: complexity actually maybe you don't even need ai for this because you could
Matthias: just look at complexity numbers themselves there are cyclic complexity numbers and all that stuff.
Matthias: And this has been research that has been going on for decades now.
Matthias: I guess we don't really use that knowledge, which is out there.
Matthias: To put it into practice and maybe simplify our code. I can totally see where you're coming from.
Matthias: And is that something that you would...
Matthias: Where would you even look for such a tool? Where would you even go and maybe ask for such a tool?
Matthias: And how do you find out about these new developments in general around Rust
Matthias: or just about new technologies? What are your resources?
Stephen: Yeah, it's tough, right? So we go to the R Rust for Reddit and there's other
Stephen: places we can go to jump around, maybe even ask on various channels.
Stephen: Probably Reddit would be a way to go. and you know
Stephen: adding these things to cicd as a
Stephen: gating requirement is is
Stephen: that i think it's a good way to get that going so
Stephen: we do have linting and like it
Stephen: must be rust has to have passed the
Stephen: formatter check so you can have because you can do a
Stephen: lot of interesting fun things with rust and put
Stephen: things all over the place but you know we want rust fmt
Stephen: as a you know a basic readability so that's
Stephen: that's in there but if we could add one more thing to
Stephen: simplify rust and not have these
Stephen: advanced language features that are really fancy
Stephen: that you know at the end of the day it's an over optimization because how often
Stephen: do we really go back to that code especially since it's rust we don't need to
Stephen: go back and modify it that often at at the rate that it justifies us us creating such a complex,
Stephen: fun, fancy feature edition.
Matthias: It's a really cool idea, for sure. Probably it's something between Go and Rust
Matthias: and arrowing more towards the Rust end with ownership and borrowing and no runtime,
Matthias: but very simple abstractions. I like it.
Stephen: Yeah.
Matthias: I wonder what you could throw out from Rust Core to even keep the Rust spirit, if you know what I mean.
Matthias: Maybe there are a couple of things that might not be needed.
Matthias: And I also wondered about your approach towards using the right paradigm for
Matthias: the job, because what you build is also essentially very purely functional in nature.
Matthias: If you like to build it this way, right, you could model something in a very
Matthias: functional pipeline where you take input, you transform it, and then you send
Matthias: it somewhere else or to multiple places without any side effects.
Stephen: Objects but at the same time you have object-oriented programming in rust as
Stephen: well which some people use for the same problem and it's also totally fine to
Stephen: do so but i wonder where you find the balance where do you draw the line here
Stephen: yeah that's that's a really good question and i,
Stephen: think we just have to you know rust comes with you can you can do modules and
Stephen: keep things you you know, in files and then access those functions from those
Stephen: files using, you know, Rust scope.
Stephen: I think that could be the line to draw from there.
Stephen: And then as we're importing or leveraging those functions, we don't need to
Stephen: go any much further in terms of abstraction beyond that.
Matthias: That means you use object-oriented programming on a larger scale to structure
Matthias: the code. and inside of functions, you could use functional paradigms like map
Matthias: and filter and collect, for example.
Matthias: Okay, I see. That makes a lot of sense. This is also how I think about it.
Matthias: Well, I guess we're getting closer towards the end.
Matthias: And I have two questions, one that I kind of forgot to ask and one that is a
Matthias: sort of tradition around here. The first question I forgot to ask is about onboarding.
Matthias: Let's say you need to find a Rust engineer or someone that maintains a larger
Matthias: part of your stack that is written in Rust.
Matthias: Would you say you're looking for someone that already
Matthias: has Rust experience and maybe is proficient
Matthias: with the Rust paradigms such that they can go
Matthias: in and hit the ground running and know
Matthias: the principles and maybe also avoid what we talked about before where they maybe
Matthias: add too many abstractions and they have some production experience or would
Matthias: you say you would rather hire someone that comes from a different language and
Matthias: is smart and can get things done and you train them on the job?
Stephen: Yeah, it's both. Both is the answer. The pool of Rust engineers is also one
Stephen: of the downsides with Rust today.
Stephen: Having that Rust expertise is not as commonplace as say like JavaScript,
Stephen: for example, right? The most popular language.
Stephen: So we need to expand that pool by allowing folks who are interested in writing
Stephen: in Rust as their day job, who might be excited to learn about it, even on the job.
Matthias: And if they wanted to pursue a career at PubNub, or they wanted to learn more
Matthias: about PubNub, could you give the listeners some hints, some resources that they can go and check out?
Matthias: Any videos, any tutorials, or maybe also free tiers that they can check out?
Stephen: Oh, yeah. So the first thing to do definitely is to go to PubNub.com and click
Stephen: sign up. So you can see firsthand that we are a communications company,
Stephen: an edge messaging technology that allows devices to communicate.
Stephen: We're in the top 10 apps in the App Store for communication,
Stephen: including something hot that needs extreme latency, such as games, right?
Stephen: Games need to have that really, when you're doing multiplayer,
Stephen: latency is really critical, because you're going to have, you're there to have fun.
Stephen: And any blips or dots in that experience, they take away from that,
Stephen: that fun, right? And we're here to have fun.
Stephen: And so that level of performance and capability and scale is something that we offer.
Stephen: And you can get that firsthand experience by signing up and giving it a try out.
Matthias: Right and as far
Matthias: as i remember you are also pretty active on linkedin
Matthias: you post regular videos and they
Matthias: are very entertaining you explain various concepts
Matthias: and i think this would be one other resource that
Matthias: i could recommend i personally follow that and as
Matthias: i said lastly it has become a tradition around
Matthias: here to ask this final question which is do you
Matthias: have a message to the broader rust community something
Matthias: that you want to share an opinion maybe a
Matthias: message for the future of rust you already
Matthias: touched on many many different points that you could tie
Matthias: into but i just want to give you the microphone and you can say whatever you
Matthias: want to the rust community now yeah so this is this is an easy one so we chatted
Matthias: a little bit about this earlier let's have the lego version of rust please I
Matthias: want a really simple version of Rust and just have it baked in.
Stephen: For example, maybe some sort of switch. And it should be a standard or it needs
Stephen: to be sort of implied that, hey, this is the accepted approach for simple Rust.
Stephen: If we had something like that and the community was on board,
Stephen: I think it would be a lot easier to get the adoption of Rust.
Stephen: Why is Python, why is JavaScript so popular?
Stephen: Because the entry point is so much easier.
Stephen: With Rust, it is a major challenge, and it is very difficult to really dive
Stephen: in, especially having to deal with some of the more heavily abstracted scenarios.
Stephen: If we could have simple mode, like easy mode, that will just go even further
Stephen: in terms of the adoptions within organizations.
Stephen: I think that will be a huge bonus, but also make me a lot happier too,
Stephen: because now we have this really simple Rust that allows our developers to jump
Stephen: in more easily, and we we can, other teams can deal with it.
Stephen: So I think that will help us make Rust more portable.
Matthias: I love it. It's a really great idea. And if it's not a compiler feature,
Matthias: it could still be some sort of course or training material to maybe have a 0.5
Matthias: step or like 50% of the way.
Matthias: And yes, it has been a pleasure to talk to you, Stephen.
Matthias: I wish you all the best with PopNub. i see a
Matthias: lot of promising alleys for your
Matthias: future and for using rust in production and
Matthias: i really like the energy as well this is something also
Matthias: i wanted to share it's always a pleasure to talk to you because as i said you're
Matthias: an entrepreneur and engineer at heart and it's a very nice combination you have
Matthias: going there and i guess it just
Matthias: shows so thanks a lot yeah it was a great Great talking with you today.
Stephen: Thank you so much for the time. And we'll chat again.
Matthias: Amazing. See you.