Matthias: Hello and welcome to Season 6 of Rust in Production, a podcast about companies
Matthias: who use Rust to shape the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today I chat with Cian Butler from
Matthias: Cloudsmith about oxidizing Python backends with Rust.
Matthias: Cian, thanks so much for taking the time for the interview today.
Matthias: Can you say a few words about yourself?
Cian: Yep. I'm a performance engineer and SRE at Cloudsmith.
Cian: I've been doing Rust in some form or another for the last 10 years,
Cian: mostly as side projects, but I have been doing it professionally for nearly three years now.
Cian: Working at Cloudsmith, trying to build on the Edge team, where we work on our
Cian: CDN and all that fun networking stuff.
Cian: Cloudsmith, we're a package management company. So we do package management as a SaaS.
Cian: We support like 36 different formats of packages for Node, Cargo,
Cian: Python, all the big ones.
Cian: We do public repositories, private repositories and open source repositories.
Cian: We're going pretty fast. We've got some big customers that I don't know who
Cian: I can mention. So I won't mention anyone just in case.
Cian: Because of that, we process about 110 million API requests daily,
Cian: That equates to petabytes of packages downloaded every day.
Cian: A lot of that is done in Python right now.
Cian: We have a very old Django monolith that we've had since day one, which is 10 years ago.
Cian: It's grown. And as we attempt to scale it, we needed to find new ways to scale it.
Cian: So we started looking at Rust as a way of making it faster and more efficient.
Matthias: Great. That means the monolith is exactly as old as your Rust experience was long.
Matthias: So it's 10 years for the monolith and 10 years of Rust for you.
Cian: Yeah, I hadn't even thought about it, but yeah, it's a nice little commonality there.
Matthias: And I could imagine you want to use Cloudsmith in a situation where you have
Matthias: an organization that manages a bunch of packages,
Matthias: maybe a bunch of packages in different ecosystems, and you want to have hosted
Matthias: version of that that is secure and safe, like we're talking about supply chain
Matthias: security, or are there any other reasons for using Cloudsmith?
Cian: Oh, 100%. Supply chain security is one of those things we're very big on, very focused on.
Cian: It's not just, though, security. So if you run different, you could run multiple
Cian: different formats of packages or just one format.
Cian: You'd use us to be a proxy to your upstreams. So you could say,
Cian: pull all your packages through Cloudsmith, and that gets you better caching
Cian: on them because you get our access to our CDN, and then you can apply security posture on it.
Cian: So don't download any packages that have these vulnerabilities or CSVs published
Cian: on them, which are decision engines for that kind of tooling.
Cian: But as well, you might just publish your own packages for internal use.
Cian: So if you are a big company that's building lots of packages that you use internally
Cian: for other services, so you could be having, let's say, a logging library with your custom logs.
Cian: You push it up there, it gets pulled in by all your microservices or CLIs and they can get built.
Cian: That's much more the traditional way of like people have private packages they
Cian: don't want to put on the internet and they don't want to have the insane tooling
Cian: of putting all their packages in one repo.
Cian: So they have lots of, so they have private repository.
Cian: A lot more focus now in the industry, that's supply chain security.
Cian: So that's where you see a lot of our development happening right now in like
Cian: securing different supply chains.
Cian: I won't say I'm an expert on that side of it. We have people who are a lot smarter
Cian: about that, who focus on that.
Cian: I mostly focus on the low-level networking stuff and data processing side of it all.
Matthias: I realize that you might not have been around, but can you maybe,
Matthias: from conversations with other employees, remember why Python was chosen to start
Matthias: a project in the first place?
Cian: I think it's a comfort situation. We had two founders who started it.
Cian: Their story is not one I will be the best person to repeat, so I won't repeat it.
Cian: If you want to look it up, we've definitely done some posts on it.
Cian: One of our CTO likes to talk about his history.
Cian: Lee Skillen, if you want to look up anything from his blog and his LinkedIn. in.
Cian: But the reason we chose Python is that it's just familiarity.
Cian: Like, it's a really good language. Like, it's powerful in that you can write so much code so easily.
Cian: It's very business friendly. It's not overly verbose.
Cian: So it leads to rapid prototyping very quickly.
Cian: Same is true with Django. Django makes it so easy to spin up a web server and
Cian: hook it up to a database and start playing around and getting your proof of
Cian: concepts ready, getting your POC built.
Cian: I think that, no, I don't think, I know.
Cian: I know that we wouldn't have scaled as fast as we did without Django and Python
Cian: because we wouldn't be able to roll features out as quickly as we did.
Cian: They definitely helped us scale the company and get to where we are today.
Cian: Saying that, after 10 years of code being written, I think I said there's like
Cian: 200,000 lines of code in our thing, somewhere over 20,000 files in our monolith.
Cian: That's a lot of code. It's a lot of.
Cian: Code that not everyone understands and we're
Cian: constantly like going back reading it
Cian: trying to figure out how it works and if you've
Cian: even this morning i was trying to read a set of
Cian: mixins trying to figure out what request what path a
Cian: request goes through as we have multiple different layers
Cian: of python code to process it it adds up over time it so what made it really
Cian: good for scaling on day one has kind of like caught up and made it really difficult
Cian: to understand and handle now.
Cian: So double-edged sword of Python there.
Matthias: Yeah, and a lot of people might say, let's just remove everything,
Matthias: start from scratch, rewrite it in Rust.
Matthias: But what people forget is that those 200,000 lines of code, they contain a lot
Matthias: of business logic and a lot of value.
Matthias: I'm assuming right now, please correct me if I'm wrong here,
Matthias: is that a lot of the logic is also about handling different package manager formats,
Matthias: file formats lots of parsing lots of error handling and so on can you talk a
Matthias: little bit about what's in there what's the bread and butter for you to make
Matthias: that infrastructure work even.
Cian: Yeah yeah it's it's that it's all that kind of stuff i said each package format is its own distinct.
Cian: Concept like we have they all have a lot of
Cian: similarities under the hood the data types are all very
Cian: similar in our infrastructure but each
Cian: one has its own idiosyncratic ways of
Cian: being of being handled and request flows we can like the flow for uploading
Cian: a package is under the hood is we take a binary and we store it somewhere but
Cian: the handshake you do with that and the metadata you store and that differs in each package,
Cian: which means that you could go into our code base and go into the slash packages
Cian: folder and then you'll just see 36 different code bases in there that are similar.
Cian: They have shared bits of code for logging and for metadata processing and tracking
Cian: of events used internally and all that kind of business logic that's shared,
Cian: but each format is different and their code paths are different.So
Cian: we'll never, like, we could, like, sit down and very quickly scaffold out
Cian: a brand new service in Go or Rust that hits those same things.
Cian: But we then have the weird edge case of, like,
Cian: how do you, how does that interact with our processing of, our processing of SBOMs generation?
Cian: And and then we need to store that in this in the in a way that's can be queried
Cian: by our api to be displayed in our ui or and we also need to track all those
Cian: data all those bytes you care about how many bytes are being downloaded we need
Cian: to ensure that all that data is being tracked correctly,
Cian: we have we're still we're in that scale-up
Cian: phase of startup life so we're we're hiring we're bringing on
Cian: new engineers but we're still a small enough team so let
Cian: if you brought in you bring in me you we bring
Cian: in me lee our cto made the joke of one day he's going
Cian: to wake up and everything's going to be rust after hiring me and we all we all
Cian: we laugh and it's funny but we know it's not really going to happen we're going
Cian: to have some core bits that are rust and but they're still going to be that
Cian: core python code that's not changing because everyone in our shop knows python,
Cian: we have a couple people who know Go.
Cian: We have a couple people, we have me who knows Rust.
Cian: We have some people willing to learn and who have tried Rust and Go at different
Cian: times, but they're not like,
Cian: ready to jump in on a project and start developing today or tomorrow right.
Matthias: But also even if you were let's say
Matthias: an expert in go it would be harder to
Matthias: integrate go into the project because go
Matthias: has its own runtime it has a garbage collector and you could do so by using
Matthias: the network boundary but not necessarily integrating it into the existing project
Matthias: as you could do with for example PyO3 or so.
Cian: 100 we and we have actually experimented with go and that's where it ended up
Cian: so we've moved logic for doing specific things out into a go microservice previously
Cian: nothing core to that business it was specifically supporting for one format
Cian: and for scaling that format,
Cian: and yeah we couldn't you can't it's nice it works it's there and it's solid
Cian: but it is a separate microservice and it goes against that belief we have that
Cian: everything should be in the monolith this is one of those core tenets we have
Cian: that we should scale our monolith we should,
Cian: focus on making sure code is in the monolith interesting.
Matthias: Point because at least in the last decade or so monoliths were sort of frowned
Matthias: upon weirdly enough and now it feels like the industry is circling back on that
Matthias: idea can you maybe explain from your perspective what's so great about having a monolith.
Cian: Yeah yeah no i i think i
Cian: came into the industry when we were just heading
Cian: for that peak of microservices or on our way up to it and
Cian: i've never liked them i'm like a big hater
Cian: on them i've always hated them now maybe i got cut very
Cian: early on and i've never like recovered from it but i
Cian: think the thing about scaling microservices is
Cian: it seems like it's a really easy thing
Cian: to do you can just throw a little service at it and everything
Cian: works and it's like it's i
Cian: just have this i call that service and it gives me a response and that's
Cian: great and when you're running like one box with
Cian: talking to another box that does scale pretty nicely and when
Cian: you have a small bit of traffic that scales real it scales
Cian: really nicely because you have a small bit of traffic but in the real world
Cian: it's never that simple you deploy 10 services for your microservice it says
Cian: it's got 10 it's got you've 10 replicas and you have 20 replicas of your other
Cian: service let's say you need to ensure that you're properly load balancing across those 10 services.
Cian: You need to account for the network delay in your one service as it waits on the other.
Cian: You start running into issues about managing connection pools and blocking IO resources.
Cian: Like this is one of those things that we actually ran into a lot in our monolith.
Cian: The way Python blocks...
Cian: Can be quite problematic because it doesn't just like go to sleep and pull,
Cian: it could just like sit there and wait.
Cian: And then you just have resources that are blocked waiting for that.
Cian: You need to know how to sleep and how to, and pick up more work in the background
Cian: while you wait on resources to fill up.
Cian: But if you never have to call across that network boundary,
Cian: if you have all your logic in a monolith you don't you can avoid the overhead
Cian: of a network you have a much simpler cognitive design that you can account for right.
Matthias: I fully agree and also refactoring across microservices is never fun.
Cian: No no and yeah and
Cian: this is a problem we're running into i say
Cian: i keep saying this problem running into we don't
Cian: have microservices but we do have a cdn and
Cian: we and how we roll code out
Cian: to that cdn versus how it interacts with the
Cian: monolith is a core part of what we do on the edge team and you need to really
Cian: ensure that you have that two-factor step of like we add a feature in the monolith
Cian: in the cdn so it can start using it and then you add you enable the feature
Cian: in the monolith and then we can remove the old legacy.
Cian: So you have that like three step deploy phase and it, you'd think it's, it's such a,
Cian: a hassle is the is the only way to say it of like remembering that and if you
Cian: don't do that you end up with all these dead code paths which we have we have
Cian: hundreds of line of dead code paths in our in our edge because we just didn't go back and clean it up.
Matthias: Yeah it's such a bespoke process
Matthias: to make releases across microservices all
Matthias: the ceremony the adding a
Matthias: feature but also putting it behind a feature flag making
Matthias: sure that the other one is bumped up
Matthias: to the correct version and then
Matthias: slowly migrating over and whereas if you have a monolith you can just make all
Matthias: of those changes in one pull request and then review all of those changes and
Matthias: your debugger still works and your linter still works and and all of those niceties yep.
Cian: That's and i think that's the it's It's the debugger that still works.
Cian: I think it's one of the nicest ones as well.
Cian: I'm not a big debugger fan myself, but I know that a lot of people in our industries love debuggers.
Cian: And it's the fact that you don't have to pull out something like Jaeger or Datadog
Cian: to do that debugging because you're calling across different services.
Cian: Like tracing is great. I love tracing tooling, like all the open telemetry kind of stuff.
Cian: It's great. But when you need to run a dedicated open telemetry stack for debugging
Cian: one simple request, that's a lot of overkill on my laptop.
Cian: And like, I have a nice laptop, but I don't know I need to be running a data
Cian: center on my laptop just to do a little bit debugging.
Matthias: Coming back to Rust, because that's kind of what I want to talk about.
Matthias: It's nicer in Rust. Yes, you can integrate Rust with PyO3, but I'm not sure
Matthias: how that process went for you.
Matthias: Did you even use PyO3 for that work, or did you decide on doing it a different way?
Cian: So let's step back a little. I came into Cloudsmith last year as performance engineer.
Cian: We decided as a company, we wanted to focus on building and scaling.
Cian: And it was known that I was a Rust developer coming from a Rust shop.
Cian: So there was a known value that I was probably going to write some Rust at some
Cian: point. But we didn't sit down and say, how can we bring Rust in to scale this service?
Cian: I sat down and I just started looking at those traces.
Cian: Started looking at Datadog, started looking at where the bottlenecks in our
Cian: service were. We had load tests running.
Cian: We were getting information back about what was slow, what were our slowest
Cian: endpoints, all that kind of stuff.
Cian: So the things that came out when you
Cian: look at that data was we would sit waiting
Cian: on io we would and it would be serialization these were two of our biggest things
Cian: the io was two different types of io our database we we queried the database
Cian: a lot probably too much but we do it uh,
Cian: eats up a lot of resources. The other side is the network.
Cian: So we call out to upstreams like PyPy and Cargo to pull in information.
Cian: And then we have the inbound requests. So that's requests from our customers to us.
Cian: So and how many requests per second can we process from the pull in from the
Cian: network and process concurrently.
Cian: The other bits being serialization, that's serializing large JSON payloads,
Cian: large XML payloads, and that kind of stuff.
Cian: So we sat down and said, how can we go about fixing this?
Cian: And it wasn't a one shot of like, we need to fix it all at once,
Cian: or we need to roll everything out, switch everything up at once,
Cian: or let's build it ourselves.
Cian: We try not to be a shop that suffers from not built here kind of thing we like
Cian: to use open source software where
Cian: possible or use sasses where possible because there's only so many people we
Cian: have so we so i started googling because i knew a solution to the json serialization already,
Cian: uh back in in two jobs ago back when i worked in video games we worked we had
Cian: a very large logging pipeline where we would serialize everything to JSON across the whole fleet.
Cian: And so we were also a Python shop,
Cian: and I was working on the metrics team, and we rolled out a logging change that
Cian: switched how we serialize JSON in all of our microservices with a Rust library called orjson.
Matthias: Oh, yeah.
Cian: It's a great library. Well, it's a Rust library and a Python library.
Cian: It's written in Rust, and it's
Cian: got nice Python bindings that look similar
Cian: enough to the normal ones the normal
Cian: json python bindings so i knew
Cian: from then that the speed up varies somewhere
Cian: between 7 and 10x depending
Cian: on what you're doing and what it looks
Cian: like and i know that when we did the change in that company i saw about a one
Cian: to two percent change of cpu usage across our data center over a couple weeks
Cian: it takes time for changes to go out but we definitely saw improvements,
Cian: and at that scale it was really important to kind of like you get a lot of you get the,
Cian: those small gains they really add up over time so i reached for that library
Cian: because i had such success with it before and when we went to reach for it it
Cian: turns out django already has a wrapper It was even easier than that.
Cian: So we installed the Django or JSON serialization library, and it swapped out our...
Cian: JSON serialization, which is just the normal Python JSON serialization with a Rust-based one.
Cian: We then had to go through all the code base and find every place we imported
Cian: JSON and replace it with orjson.
Cian: And then we did these each incremental steps. We didn't like flip the switch.
Cian: We flipped the switch on
Cian: updating all the json files one at a time until i think one day i just got very
Cian: bored sat down and had a train ride and just banged through every single i just
Cian: grabbed everywhere we imported json library and just iterated through those
Cian: files making sure they were all correct nice.
Matthias: It's always the train rides right.
Cian: Yeah it's.
Matthias: Always that's when we get the work done but.
Cian: Yeah also.
Matthias: What i find particularly interesting about that story is,
Matthias: If you didn't know it was written in Rust, you might not even have cared about
Matthias: it because it was yet another Python package that you just integrate into your workflow.
Matthias: And it was a drop-in replacement. But I wonder how many organizations out there
Matthias: run Rust without even knowing it this way because Orchason happens to be written in Rust.
Cian: I think there's probably so
Cian: many places. Like, if you asked my previous employer if they're on Rust,
Cian: they would say, nope we have no rust and i know for a fact there's rust
Cian: in every service because of the fact i put
Cian: it there through that python library and i
Cian: think that's actually like a nice thing it's it's
Cian: also true of the cryptography library in python it's
Cian: rust based now it the there is
Cian: rust and there's rust in linux now
Cian: like rust is everywhere it's getting rolled out
Cian: everywhere but it's those
Cian: nice places like orjson where someone has sat down and said how can i make this
Cian: faster without breaking the api
Cian: or in such a way that it doesn't take a massive lift to switch it out yes.
Matthias: But also as some sort of counter argument
Matthias: to that someone might listen to it and think well json is sort of a nice easy
Matthias: interface to integrate with there because there's a nice api surface but how
Matthias: often does that happen in practice that you can just use a drop-in replacement
Matthias: what would you say to that.
Cian: Yeah, not as often as I would like. It's totally not as often as I'd like. We've managed to get...
Cian: I talked about this before in previous talks at FOSDEM about our experience at it.
Cian: We switched or JSON first, and it worked great.
Cian: Well, it worked great. One customer broke because they were parsing JSON with
Cian: bash and grep and seds and all those things. Don't do that.
Cian: It's bad. they realized it was bad and they moved on
Cian: so ordreson great drop replacement
Cian: after the success of ordreson i i
Cian: didn't want to like i knew pyotree existed so i sat down and said where could
Cian: pyotree be used next my other the one that i wanted to look at was xml parsing
Cian: not parsing serialization We have to serialize very large XML payloads.
Cian: So I was interested in seeing how could I come up with a more efficient way
Cian: of doing this for our use case using Rust and Pyotree.
Cian: But I got distracted when I went on to Pyotree's docs and I noticed...
Cian: That you they had a json schema
Cian: library and i thought oh cool i wonder
Cian: if this is faster than our json schema library so i
Cian: went to see our usage of json schema library and found out we were already using
Cian: the rust one but also we were using the python one we had both installed and
Cian: we're using them both at different parts in the code and i kind of just looked
Cian: at myself going how do we What happened here?
Cian: Did someone just not look at our folder and say, do we have a JSON schema library already?
Cian: Or were we planning to do the migration?
Matthias: My suggestion would be to use Cloudsmith because they handle package management for you.
Matthias: And this is how you could avoid the problem.
Cian: Yes, yeah, totally. Well, I think you could at least catch it sooner.
Cian: Maybe we wouldn't have been running the two things for so long.
Cian: But saying that, it gave another opportunity for us to like continue the rollout
Cian: of like switching to Rust because we clearly knew it worked for us.
Cian: We had success already. So all I needed to do that one was again,
Cian: just switching the imports in all the other ones and removing the pure Python implementation.
Cian: And we rolled it out and it was smooth as butter.
Cian: Like that, it wasn't, I didn't change any code. I just changed the import statements.
Cian: So there definitely is like that ability to do those drop-in replacements that
Cian: work so well right there.
Matthias: What I find cool about that story is that these initial quick wins gave you
Matthias: a lot of confidence into integrating Rust in the stack without really requiring
Matthias: a lot of backing from the entire organization.
Matthias: You can just go step by step and you can see the success right away.
Matthias: But then eventually you might have hit a wall where this is no longer possible
Matthias: because all of the quick wins are gone so i wonder how you transitioned from
Matthias: there to maybe introducing more rust because well obviously it was kind of a success yeah.
Cian: Like i said i was playing around with pyotree and different ideas and when i
Cian: started looking at our bottleneck for the network I started thinking about how
Cian: we manage work in the service.
Cian: And the way our request model worked was,
Cian: We were using WSGI, W-S-G-I, and effectively where processing requests come
Cian: in, we'd give them to a Python worker,
Cian: and it would do the request to completion and then hand the response back.
Cian: So for Rust developers, they might look at that, and the model is very similar
Cian: to a Tokyo service that we had, and that's my instant thought about it.
Cian: I looked at it and said, that looks like a Tokyo service that has one event
Cian: loop that does some processing,
Cian: hands it off to a background task, and then it waits for the task to complete
Cian: and get back the results onto the main event loop and throws it back over the wire.
Cian: Of course, it doesn't use serialization to bytes or any of that kind of stuff, but it looks like it.
Cian: One of the bottlenecks I found was we were wasting cycles doing work for connections
Cian: that had already closed.
Matthias: Oh, wow.
Cian: Yeah.
Matthias: Why is that?
Cian: It's a little to do with our queuing model and a little bit to do with request
Cian: management in uWSGI, the process we were using.
Cian: Effectively, if a request had sat in the queue for too long,
Cian: it would be handed over to uWSGI.
Cian: uWSGI would do it and it would time out in the upstream because it had been
Cian: processing for longer than a minute.
Cian: But there's no way to cancel the request once it's in flow.
Cian: It would we would benefit from
Cian: it because we do all the work and cache the result so another
Cian: request would be so the request would have been retried and would be in the
Cian: queue and by the time it gets to the front of the queue it's all its results
Cian: are cached so it was a nasty flow but it kind of we kind of optimized for it
Cian: yeah but i thought to myself this feels insane there's no.
Cian: Is the first is going to be cached or it's
Cian: going to be re-driven like most of the time it
Cian: is we're dealing with some some of the i think someone
Cian: recently described some of our clients as some of the best and worst
Cian: clients in the world because they're designed for public
Cian: infrastructure since they're all package management clients so
Cian: they have a lot of retries but they have a lot of weird formats so
Cian: we're dealing with some of the best and worst clients so we
Cian: know a lot of things are going to be like retried and attempted again
Cian: but it's also not a
Cian: perfect cache because some of our caches are in memory and
Cian: some of them are memcache so things that were in memcache those were quick but
Cian: if it was in memory cache unless you hit the exact same node again that in memory
Cian: cache is useless and like i said we're running lots of replicas so there's no
Cian: real guarantee on that yeah.
Matthias: That's a thing that i heard a couple times already is
Matthias: that if you think about a
Matthias: highly performance service that does not waste a
Matthias: lot of cpu cycles then you need less of
Matthias: those which means you have higher cache locality if
Matthias: you have a service that is not as fast you need more instances so you lose the
Matthias: ability to have things in your in-memory cache so that's kind of another way
Matthias: on how more performant languages or more performant code is effective.
Cian: And.
Matthias: Helps with performance.
Cian: Yeah like those i'm a big believer
Cian: that in-memory caches are are only
Cian: good when you can have a small footprint because they're
Cian: effectively they build up in that small footprint and if you need to have lots
Cian: of replicas for whatever reason be that be that like budgetary or a limiting
Cian: of like only having one CPU per map to a process or something like that,
Cian: you end up with these very disparate caches that have different information
Cian: and your load kind of ends up going all over the place.
Matthias: But wouldn't you have been able to query the in-memory cache and then,
Matthias: if that fails, go to memcache right away?
Cian: Yes, you would think that. But the issue isn't that we were,
Cian: it was, it's not that we have one caching mechanism, it's that we have different caching mechanisms.
Cian: So we were using the Python caching library for in-memory cache.
Cian: And then we were using our memcache with our database to cache responses from
Cian: the database. So these are actually two different caches.
Cian: The memcache one is just, could we stop ourselves from going to database?
Cian: And we would totally check that on every request. So if we had done a very expensive
Cian: DB query, it should be in that memcache.
Cian: So on the retry, it would come from the memcache.
Cian: What wasn't being cached were those pure functions we were running inside the
Cian: monolith that were in the Python cache.
Matthias: Got it.
Cian: Yeah.
Matthias: So the new bottleneck right now is between the network layer,
Matthias: which was your uWSGI, and the Django monolith.
Matthias: There's where you lose a lot of the performance now.
Cian: Yeah. And my goal was something we're still working on, was I wanted to be able
Cian: to do request cancellation.
Cian: So I wanted to be able to say, that's timed out upstream, I want to cancel it.
Cian: Something i had previously done in a tokyo service so kind of was like totally
Cian: let's do this so i sat down to try and figure out how i could map a tokyo request
Cian: managed service to our WSGI app and,
Cian: it was and i was reading pyotree docs and i was playing around with a library called,
Cian: RustImport, which lets you very quickly write PyoTree bindings for your Rust libraries.
Cian: You can get a very rough and ready code in 20 lines with some macros.
Cian: And you can have this very rough importing of Rust code directly into your Python
Cian: code without a lot of overhead.
Cian: Great for prototyping.
Cian: I had found some places where I thought I would probably change this if I wanted
Cian: to bring it to prod and just use Pyotree for creating the interface exactly as I wanted to.
Cian: But it was definitely great for prototyping.
Cian: But saying that while prototyping i started looking at prior art and i had found
Cian: someone had this idea already which is i want to say the best thing about like
Cian: open source is sometimes you go and look and say someone has someone already
Cian: had this idea and more often not someone has so.
Matthias: Yeah and also you could have gone and
Matthias: completely ignored that and not have done
Matthias: any more research and you would have that liability on your side whereas now
Matthias: you looked at prior art as you said and you found a thing that someone else
Matthias: worked on before so that also shows that you took a very level-headed approach to that.
Cian: Yeah, 100%. The project we found, it was called Granian, or I might mispronounce
Cian: it a handful of times because I got so used to call it Granian at one point.
Cian: But it's effectively a replacement for that WSGI service we were using that is written 100% in Rust.
Cian: It's a Tokyo event loop that hands off to Python processes for doing the actual processing of the code.
Cian: So all your business logic runs there and it just ensures that all the network
Cian: logic is done inside Rust.
Cian: This was really cool for me because I was like, cool, here's a project that
Cian: does exactly what I wanted to do.
Cian: And I started reading the code and I learned that the concept of request cancellation,
Cian: the thing I was doing all of this to was not
Cian: possible in new WSGI at all like there was never going to be a chance of
Cian: doing it in WSGI because it's just not supported by a protocol Gradian does
Cian: only supports it if you're using ASGI which is an async version of SGI that's
Cian: more like a traditional event loop style of async await mm-hmm.
Matthias: Similar to an io_uring or so.
Cian: Exactly.
Matthias: Completion-based.
Cian: It's exactly the same kind of design.
Cian: And you get to reuse all that kind of code that's designed for those IOU loops.
Cian: But we sat down and we'd already started looking at it.
Cian: So it saved me a lot of time in that concept of prior art walking down paths
Cian: that we could have lost so much time if I had spent working on it.
Cian: But it did have a feature that I loved, and that was it had a built-in queue
Cian: for managing the requests.
Cian: So right now, to this day and at the time, we were running HAProxy in front
Cian: of uWSGI to allow us to scale.
Cian: HA Proxy was effectively doing the queuing for us, managing work in a queue,
Cian: and then handing it off to a uWSGI process that would hand it off to a Django
Cian: process and do the request.
Cian: And for reasons that elude me of why an engineer decided to do this,
Cian: we also are running an Nginx in front of the HA Proxy to do very light routing
Cian: control and optimizations.
Cian: Nothing that couldn't have been done in HAProxy, but it was just being done
Cian: in Nginx for some reason.
Cian: And there was a ticket on a backlog
Cian: for years of merge HAProxy and Nginx together and just have HAProxy.
Matthias: It's interesting that you make that decision. One could have made the decision to go with Nginx.
Matthias: Personally I find the nginx config to be easier to read and write in comparison
Matthias: to the HAproxy config maybe that was the reason for nginx you're.
Cian: Probably right it's like I,
Cian: nginx is a really nice config it's super like
Cian: readable and simple and probably of all the
Cian: tools i'm going to talk about it's just it's the easiest to
Cian: work with and was pretty bulletproof and in doing some amazing things for us
Cian: but the ha proxy one was i think ha proxy is just a better queuing tool or at
Cian: least my experience of using of doing request management in ha proxy has been better,
Cian: but i think when no one knew which
Cian: way we wanted to actually go the idea of like let's replace them
Cian: both let's replace one with the other was this
Cian: was the idea and when i
Cian: found granian i looked at it and said oh this
Cian: can not only replace our WSGI management interface
Cian: but it can also replace ha-proxy because it
Cian: can do that queuing internally and it
Cian: has dials for tuning that queuing as we needed it to work so it there was and
Cian: we also had this intense dislike of the of uWSGI because uWSGI is quite
Cian: difficult to tune uWSGI being the tool we use for managing WSGI requests.
Cian: So...
Cian: So I started chatting to a principal and said, have a look at this.
Cian: What do you think about this?
Cian: And I got the thumbs up of, ah, sure, let's try it out and see what happens,
Cian: which is a very Irish way of going, let's run a load test and see how it performs.
Cian: So we threw up a version into our load test environment that replaced uWSGI
Cian: with Granian. Granian. Granian.
Matthias: One of these.
Cian: One of these two. So we threw up a version into our load test environment that
Cian: replaced uWSGI with Granian and we began load testing it.
Cian: We just started throwing lots of different types of requests.
Cian: We have some nice load testing tooling that simulates some request flows.
Cian: So we just had it run. and the
Cian: numbers we got back were marginally better
Cian: it wasn't like a night and day like oh my
Cian: god this thing is going to save us we found
Cian: the savior of scaling no nothing like that but what it did say was it changed
Cian: the numbers in our in our percent in our p50s and our p90s our p50s went down
Cian: and our p90s went up which meant we just had a lot more outliers and our Averages were better,
Cian: which was enough of a signal for us to sit down and go, there's something here.
Cian: Don't, it could just be a better tool for us to be able to tune.
Cian: It could just be more cues is helping us scale in some way or another.
Cian: But it was definitely, it was a signal that we said, we need to test a little
Cian: bit more with this. This isn't something we need to just walk away from.
Matthias: Right, because if you see that your P50 is better, that means the outliers are now more prominent.
Matthias: So there might be things in your business logic or timeouts with upstream,
Matthias: which mean that they drive up the P50.
Matthias: P90 or p95 signal but overall this is also a thing that you see a lot with replacing
Matthias: code with faster code on on the back end side is if you do it right then the
Matthias: outliers become more prominent.
Cian: Yeah no 100 we were definitely seeing that where it was these very slow paths
Cian: that were blocking us were still the slow ones.
Cian: But the very quick paths, they just became quicker.
Cian: And there's a lot of differences in how uWSGI and Gradian were configured in
Cian: those early load tests that I now know were silently masking different things about.
Cian: They were handling switching contexts differently, how tread management worked.
Cian: So the memory footprint was,
Cian: little more stable in one
Cian: while it correlated to workload better
Cian: in the other that's got good and bad it meant that previously we would have
Cian: like the memory which and cpu would stay flat but now like as requests went
Cian: up you could actually see the cpu was going up and down because we were doing
Cian: more work and we're like that's a good signal for us scaling now we could use that to do some
Cian: where previously we couldn't do that auto-scaling.
Matthias: Yeah, because you could never go down to zero.
Cian: Exactly, yeah. So we sat down and we drew up a testing scenario,
Cian: like some numbers we wanted to see, some testing we wanted to do.
Cian: Which parts of the stack could we try removing now that we just,
Cian: and could we just replace it with Gradian?
Cian: So we did a lot of different load tests to the point we actually managed to
Cian: bottleneck in the load test tooling.
Cian: We hadn't scaled the load test tooling up high enough that it could push enough
Cian: throughput in one of our tests that we needed to step back and change the load test tooling out.
Cian: We were previously using Locust, which is a fantastic load test tool where you
Cian: write your load test in Python,
Cian: and then you spin up lots of Python workers that are managed and it does the
Cian: load test from different places.
Cian: But those workers were becoming our bottleneck. So, well, they're not really a bottleneck.
Cian: How much money we were willing to spend on those workers became the bottleneck.
Cian: Like how many workers could you spin up for a load test was the bottleneck.
Cian: So we switched out for a tool called Goose, which was a reimagining of that in Rust.
Cian: Managed to push the same amount of workers, we were able to push more requests, like,
Cian: I think 100 or 1,000 X more requests per worker, which meant that bottleneck was out the window.
Matthias: It's somewhat funny that in the process of oxidization, you also have to swap
Matthias: out the load testing tool.
Cian: I think that was the biggest signal of we can push more was when we had to swap
Cian: out the load testing tool because that was what was being saturated.
Cian: Yeah. Yeah, and it was really good.
Cian: At the end of it all, we had a test scenario that showed we were able to push about 2x,
Cian: per compute resources than we previously were.
Cian: And there's a lot of reasons for that. One is we were running less intermediate services.
Cian: We weren't running Nginx after this. We weren't running HAProxy.
Cian: And Gradian was effectively doing all of that for us in a nice Rust event loop
Cian: and handing it off to background processes in Python.
Cian: And the Python was that original P50 gain was adding up along with all these
Cian: less resources having to be run.
Matthias: It's great because yeah as
Matthias: a first step you could say you handled twice
Matthias: the load which means you could have half
Matthias: the servers if you wanted to but then on top of
Matthias: it you have better memory locality now
Matthias: so maybe you even need less
Matthias: cache servers if you had those and on top of it even before the request even
Matthias: hits your monolith you can also optimize a lot because now you don't need nginx
Matthias: and ha proxy you could replace all of that with one service and.
Cian: I think that was the biggest one for us management saw that i what we're saying
Cian: we could squeeze more requests out of what we're already paying that's we could
Cian: scale We said we could scale down, but we knew scaling down was not going to
Cian: be what we were going to do.
Cian: We're signing customers on every day. We're growing every day. We're scaling up.
Cian: So the idea of scaling down,
Cian: of compressing the amount of work we can do in compute is big for us we it got
Cian: us the time to experiment more and continue our testing and see what's next
Cian: for us what's what can we improve and.
Matthias: You need that time because i'm assuming that there are differences between the
Matthias: old stack and the new stack especially if you deal with a lot of real world http traffic.
Cian: Yeah yeah there was two big differences for us that caused two annoying outages for us as well,
Cian: the one that's gonna is burned into my brain was
Cian: to do with docker we we have so docker has a lot of interesting clients is the
Cian: best way i can describe it and it's a it's a standard of how you do stuff But
Cian: every client can kind of implement,
Cian: do the implementation slightly differently and handles edge cases slightly differently than each other.
Cian: So for scaling reasons of our
Cian: cdn we would often respond with trio
Cian: sevens and say and say the resources
Cian: over in this other location for storage go get it and and you download it yourself
Cian: rather than me downloading it for you and handing it off like you don't want
Cian: to be you don't want a python service doing a download and sending it back over
Cian: the wire you want something that's built to scale and serve those requests.
Cian: So it's our CDN out of the edge.
Matthias: The Docker clients that you meant are things like the implementations of things on your local machine.
Matthias: Like if you do Docker pull or you use Podman or...
Cian: No, yeah, it's... When I say Docker clients, I mean Podman versus Docker versus BuildX versus...
Matthias: OrbStack.
Cian: OrbStack, yeah. And there's hundreds more multiple...
Cian: You work in a company, you'll be running different versions and different developer machines sometimes.
Cian: And you'll be, so one developer is doing one thing and that could be different
Cian: to prod because you're not running in prod, you're actually running Kubernetes,
Cian: which is different again to Docker.
Cian: Like the Docker clients are all different and unique and there's many of them
Cian: with different edge cases. right.
Matthias: So back to your story we were at a point where you don't want to handle the
Matthias: requests for the clients instead you tell them look elsewhere for the resource
Matthias: that you're trying to pull yeah.
Cian: Exactly so we'd give them a nice 307 to
Cian: our cdn location and they respect it
Cian: and they pull it it's it's part of the protocol that
Cian: they can do that but for reasons
Cian: that are very legacy and to do
Cian: it how go implemented it's a
Cian: first HTTP client they were
Cian: accepting bodies they accepted
Cian: trio sevens with content lengths
Cian: that were not zero and because
Cian: of that they would have the docker
Cian: client for some reason used that first content
Cian: length it saw as the metadata as
Cian: the content size of the image it
Cian: was eventually going to be and it would so
Cian: it would look at the response and say cool i have a content length of two of
Cian: 200 megabytes going to put that in the met in the metadata for my for my docker
Cian: my eventual docker image so it then follows the 307 and goes and grabs all the
Cian: other layers and it says,
Cian: and then it signs it and says, here you go, this is your built image.
Cian: The issue came in when we were getting, so we're sending back this 307 saying
Cian: it's got a content-like length of 200 megabytes, let's say.
Cian: The ALB we were using, the load balancer we were using, started to have errors
Cian: on this. It started saying, nope, that's an invalid request.
Cian: I don't remember exactly what error it started returning, but it started throwing
Cian: random errors that were not the correct error as well.
Cian: So it was processing something internally and it broke its serialization.
Cian: It's kind of scary when I kind of start saying it internally because these were SaaS products.
Cian: We didn't have proper logs for them. We just had metrics of error rates going up and down.
Cian: So we sat down, started digging in, and we managed to map the error rates to the Docker requests.
Cian: And we decided we needed to flip some, we needed to move some stuff around and try some stuff out.
Cian: So we started encoding, we said, oh, this is encoding 307s as,
Cian: it's saying these 307s have a content length of 200 megabytes or whatever the
Cian: eventual image size is going to be. Mm-hmm.
Cian: Let's not do that. That's what's breaking this. Let's respond with actual valid
Cian: HTTP and say the content length is zero.
Cian: So we did that and Docker freaked out.
Cian: It started, well, actually we ran tests and they were working.
Cian: Like, we're like, great, our end-to-end tests are still working in this, this is fine.
Cian: And then one of our developers came in and said, hey, I can't get my local dev to start.
Cian: So we started debugging it. and it turned out that their local dev was getting the wrong metadata.
Cian: And my local dev was working completely fine.
Cian: And that's where it became really weird. I was using BuildX and they were not
Cian: using BuildX for building their Docker images and running their Docker images.
Cian: And that's when we realized it was very specific clients were doing stuff differently.
Cian: Some of them were checking the metadata data from the header,
Cian: and some of them were doing the maths themselves and putting it in there.
Cian: We rolled back the change of the header, and we moved the logic around.
Cian: We moved the validation of the content length out to the edge network so we
Cian: could do some like after our load balancers had done all their work and hyper had changed.
Cian: Nginx was just handling that 307 completely
Cian: differently and arguably incorrectly
Cian: it was doing it was massaging it
Cian: into a way that the load balancer was accepting it yeah and
Cian: we needed to work around all of all of
Cian: those kind of weird edge cases that we had previously
Cian: just got nginx working on nginx was just doing
Cian: stuff in we moved it out to our cdn
Cian: layer our so our request processing was at the
Cian: edge then and it works like once you got once you move those things around you
Cian: can see the that it does work but like there's so many weird edge cases in in
Cian: hp that i that i can't like say this is a drop in replacement yeah it's one
Cian: of those you really have to test them.
Matthias: Yeah, I remember that in one of our earlier episodes with the maintainer of cURL, Daniel Stenberg,
Matthias: he mentioned a very similar problem, which is that Hyper was very strict about
Matthias: certain ways HTTP traffic should be handled.
Matthias: And cURL needs to be extremely permissive because people expect it.
Matthias: That's kind of the API of the command line tool.
Matthias: And he needed people to go in and either soften the edges on hyper or make parts
Matthias: of that transition layer a bit more permissive on the cURL side.
Matthias: But that was a tough job for them.
Matthias: And eventually they removed the Rust backend because of that.
Matthias: So because they couldn't fix it or there were not enough people who wanted to put in the work.
Matthias: And you hit this because, I just want to reemphasize that, you hit that because
Matthias: Granian, the WSGI server, uses hyper correct.
Cian: Yeah it's exactly it's the exact same stuff hyper
Cian: was doing everything technically correct the
Cian: the fun sentence of everything is technically correct granian uses hyper and
Cian: tokyo and pyotree it's just core libraries so it it was using hyper to serialized
Cian: a response and it was just and.
Matthias: Do you believe that we need more permissive libraries more permissive rust crates
Matthias: for real world hdp usage or other areas where things have historically grown
Matthias: to make those you know rust adaptations easier for people,
Matthias: Or would you rather say, well, no, instead we should work with better standards
Matthias: and maybe fix our code? Yeah.
Cian: As I've noted, I work with some of the best and worst clients.
Cian: They do retries, they expect really good responses, but I don't own the API contract on them.
Cian: I have to just follow the API contract.
Cian: I would love to say that we as an industry should be following the standards
Cian: being so strict to them and I can totally see that if I look back at me five
Cian: years ago I would be there shouting no no follow the standards we should make
Cian: everyone who doesn't follow the standards feel the pain,
Cian: the issue is there that's a
Cian: lot of people that's a lot of pain and it's not
Cian: something you can fix overnight like i
Cian: think we i know because i work in a package company a lot
Cian: of people run a lot of different versions of the same software
Cian: so even if like we started making tools stricter every everyone on december
Cian: on february 28th decided to do one launch where everything switched to strict
Cian: mode the in every library we then have to get that rolled out to every version of that software,
Cian: it's not going to be, it's going to be a painful rollout.
Cian: You need to have a level of permissiveness in the clients.
Cian: Saying that, I don't want the default to be permissive. The default should be perfect.
Cian: It should be the best way a client should run.
Cian: The client should have timeouts. It should have sane defaults and should follow the standard.
Cian: But when you run a legacy system, you're going to have a lot of weird legacy
Cian: issues. And you need to be able to flip those switches off to mean that you
Cian: can enable these things.
Matthias: Otherwise.
Cian: You're going to end up with a lot of duct tape around your very strict system
Cian: to flip those switches off.
Matthias: Yeah be very strict initially and then lower the guard yeah exactly now when
Matthias: you look back on the project what would you say were your key learnings i'm
Matthias: talking about things that you would have done differently but also things where
Matthias: you believe rust is a good fit,
Matthias: how did that project go?
Matthias: Maybe you can summarize it in a few sentences.
Cian: The project could have gone a lot better. It's still underway.
Cian: We're using it in specific environments now.
Cian: We haven't rolled out 100% everywhere because of these weird edge cases we found with Docker.
Cian: And the other issue we found was about connection management to our database.
Cian: It's a big problem. We need to do some upgrades, which means we've held off
Cian: and we haven't got there.
Cian: That's and that was the biggest things about the project
Cian: that was the unknown unknowns we
Cian: sat we sat down and i keep saying we there was maybe me a principal to review
Cian: my work and a manager to like sign off on it and and set out like would you
Cian: we'd leverage our end-to-end test to do stuff.
Cian: We'd use our load tests to validate our request throughput and that kind of stuff.
Cian: But we never had a plan. And we had rollback and rollout plans.
Cian: We had rollout plans that were like, well, canary in lower environments, raise them up.
Cian: We'll do it in off regions in quiet times.
Cian: Following the SRE handbook of how do you roll out changes safely.
Cian: But we we had issues
Cian: with like that that business logic like at the start of all this we started
Cian: pulling in rust tools to speed up python because we didn't want to do a full
Cian: rewrite for many different reasons we wanted to use.
Cian: Small bits of rust in our stack to speed it
Cian: up or small bits of sea as well if that was not if that was going to be there
Cian: as well we were very much just looking for faster ways to do what we were currently
Cian: doing but what we were currently doing wasn't well wasn't understood enough by myself and,
Cian: others because we have that 10 years of legacy
Cian: there's edge cases where the person
Cian: who worked done it has come and gone or that
Cian: same person has come and gone to the company three times he's
Cian: my principal and he's like this is kind of reminding me of an outage i had five
Cian: years ago and he and he's trying to remember it and we're trying to fix it these
Cian: are the things i would have loved to know beforehand i would have loved to have
Cian: known that we were going to run into these,
Cian: weird edge cases and i don't know
Cian: how i would have known how i would have got there how much
Cian: more time researching could we have come up with how much
Cian: more testing could we have done were these things we're
Cian: only going to find in prod probably but i wish there was better we we had a
Cian: better way of validating these things like a better test suite for like hp testing
Cian: better test suite for different clients.
Matthias: I guess the question I would ask to myself is, had I known all of these things
Matthias: before would I have made any difference?
Matthias: Would I have made a different choice? And maybe the outcome was kind of still worth it?
Cian: Yeah. No, we retroed. Like I said, we're still in process, but we do regular retros.
Cian: And that question came up. Was this the right choice?
Cian: Should we have made a different choice? And I said, and we all agreed this is the right choice.
Cian: There is something here that is worth testing. It's worth using.
Cian: If we're not moving as fast as we want to and we've just introduced a new thing,
Cian: we still know where we're going. We all agreed on a roadmap.
Cian: The roadmap was just a lot longer than we thought we'd agreed upon.
Cian: But it's still worth it. The speed increases we're seeing and they're totally worth it.
Matthias: Now, looking back at your 10-year of Rust experience,
Matthias: three of them professionally how has
Matthias: your perception of rust changed over time
Matthias: remember that maybe when
Matthias: you started you might have been enthusiastic about the language just trying
Matthias: to explore what's there but now that you use it professionally what would you
Matthias: say has shaped your perception on rust in in the last couple years.
Cian: Rust has changed a lot over 10 years. Like, I can remember a time when you'd
Cian: get a clippy warning that would tell you, don't do that, do this.
Cian: You'd do that, and it would produce a different clippy warning.
Cian: And you could be 10 clippy warnings deep before you had the working code.
Cian: Rust today is a lot different than that was. Like, you get one clippy warning, and then you're fixed.
Cian: Or maybe you get one clippy warning, and your fix is ever so slightly different
Cian: because you didn't turn on pedantic mode or something like that.
Cian: But like rust is a lot friendlier now than it used to be and when i started
Cian: writing rust i was very much just looking for at the cool hip language i think
Cian: i think i first found rust out of fosdom going like going full circle in my life.
Matthias: Me too,
Matthias: by the way.
Cian: Yeah it's like Mozilla was so big on it and it seemed so interesting and I had
Cian: just come off learning Go and I had and Google,
Cian: was going Go is great, Go is great and I was talking to SREs who were like this
Cian: Go thing seems really cool but I was like I was just there was some idiosyncratic
Cian: things about Go that I was never a big fan of so I so that's why I started learning Rust and I,
Cian: It's, I think we, it's got some rough edges still that are not fully sanded out or fully well.
Cian: The story's not there yet. Like, when to choose a framework is still like an
Cian: interesting problem you have in Rust.
Cian: You have the issue of, do you use Hyper or do you use Axum or do you use Rocket?
Cian: And I'm not even sure is Rocket still a thing. Like, I remember when that came
Cian: out and it was really good, but I've never used it professionally.
Cian: I think I've always reached for Axum and Hyper professionally.
Cian: Because you the smaller projects don't
Cian: move as fast in the
Cian: right in sometimes but saying that hyper hyper moves
Cian: very slowly hyper was only went v1 a
Cian: year and a bit ago it took a
Cian: long time to get to v1 and v1 was a big change
Cian: as well so that was a that switch was like was almost
Cian: a full rewrite of services and i
Cian: think that's the thing i appreciate though about rust we took
Cian: our time to get to get an api that was
Cian: going to be stable that wasn't going to change a
Cian: lot and was and it's going and you can
Cian: work against but when you
Cian: look but how many of the projects have never
Cian: hit v1 is scary i look
Cian: at my my cargo lock file
Cian: or my cargo toml and a lot of
Cian: my projects are still 0.8 0.7 0.1239
Cian: like these values that i'm like they could break at any time but i need to keep
Cian: track of these things because i work at a package management company where people
Cian: need to track we We need to track stable and secure versions constantly.
Matthias: I would say...
Matthias: Yes, a lot of versions are still not 1.0, or there are a lot of unstable crates out there.
Matthias: But at least from my experience, they break less often than in other ecosystems.
Matthias: Even though there might be a feature release bump or so, rarely do I need to
Matthias: go in and make any bigger sweeping changes.
Matthias: It's rather just minor things, or sometimes I'm not even affected by that.
Matthias: And so like i don't want to devalue your point but it's just to give more people
Matthias: some perspective maybe people that don't work with rust a lot it it might not
Matthias: be the biggest problem right now in the ecosystem.
Cian: No i i'd say i don't think it's the biggest problem
Cian: in the ecosystem right now i think it's i think
Cian: it's one of those problems that is a bit of a perception problem
Cian: and it's very much one that you might
Cian: see newbies might interrupt might
Cian: feel a lot more like i i say
Cian: that i said that i'm like bumping crates and i
Cian: i'm maybe being apprehensive but like
Cian: i have the same experience you have i rarely have to go
Cian: in and actually change an api i i
Cian: don't i did a very big bump on some stuff recently and
Cian: we're using a new framework for
Cian: we're testing a new framework for writing parts of our edge code and
Cian: rust and it went from version
Cian: 0.6 to 0.7 and i think
Cian: it just added a lot of optional args to a lot of stuff so we had
Cian: to try to read the docs and add those optional arcs that was that was not a
Cian: big change it took maybe an hour of my time to just do that and that was fine
Cian: and at saying that it came with improvements it came with cache improvements
Cian: and all that kind of stuff so taking in those changes was good it was obviously
Cian: feature changes is not bug fixes and that.
Cian: So I'm like happy to take that stuff in. But when I...
Cian: We're trying out more Rust and I'm bringing more people in to look at Rust.
Cian: They, who are coming from a Python world and coming from different worlds.
Cian: And they look at a lock file and they say, why are none of these things stable?
Cian: I have to have that conversation with them about why we're still using pre-release
Cian: software and why it might be years before that pre-release software comes in.
Cian: And I don't think it's a problem you need to fix, but maybe it's a problem of education.
Cian: And how do we talk about the v0 of packages to make people understand that this
Cian: is, should this be production or should this not be production?
Cian: It's not, a v1 isn't a signal that this should be production or not.
Cian: It's just a signal of stability of the API.
Matthias: Do you think you will use Rust in 10 years?
Cian: I hope so. Like, there's an answer of, I hope so.
Cian: I think languages change a lot, and the language ecosystem change a lot.
Cian: I didn't think 10 years ago I'd be still writing Python or JavaScript,
Cian: but I'm still writing Python and JavaScript.
Cian: But you look at them, and they're a lot different to the Python and JavaScript you wrote 10 years ago.
Cian: So I think Rust is here to stay.
Cian: It's, I said earlier, it's in the Linux kernel now.
Cian: It's in low-level libraries for Python. It's in UV. It's in ty.
Cian: It's becoming a core part of our industry.
Cian: But how will I be writing it? Or will someone else be writing it? I don't know.
Cian: Maybe we'll have got to a point where we have saturated the amount of rust we
Cian: need to write. And we can use...
Cian: Higher level tooling built on top of that rust could
Cian: we have a language that's less verbose than
Cian: rust that is gives us
Cian: the same memory safety could we take the
Cian: lessons we learned from the borrow checker and apply
Cian: that to an a language that looks something like a python for business logic
Cian: and call in and out of it and maybe that's better for us maybe that's actually
Cian: what i want is a language that takes all the learnings from Rust and takes the stability from Rust,
Cian: but is a little friendlier for newcomers or a little easier for people fresh,
Cian: for graduates fresh out of college to get started with without feeling like
Cian: they're writing a systems language. Because that's something you always hear.
Cian: Rust is a systems language. It's for systems programming. It's for systems problems,
Cian: which isn't true. You can write anything. you.
Cian: Rust is a language. It's a tool. You can do whatever you want with that tool.
Cian: I've written business APIs in it. I've written load balancers in it.
Cian: I've written CLIs in it. It's great for all of those things.
Cian: And we've learned a lot from it that we could apply to other places.
Cian: So will I be writing Rust? I hope so.
Cian: Will everyone be writing Rust? Probably not. Will there be a new language that
Cian: hopefully isn't inspired by Rust?
Cian: Probably. Will there be a new language? Definitely.
Matthias: And finally what's your message to the rust community as a whole.
Cian: I think it's gonna start with a thanks because i
Cian: wouldn't be doing what i enjoy right now without the rust community like
Cian: they that knowledge sharing the
Cian: rust books people willing to talk about it
Cian: and whenever someone's willing to talk about it it's always that very enthusiastic
Cian: talking about it like i
Cian: was lucky enough at for them to go for dinner with a lot
Cian: of the other speakers of for the rust room the enthusiasm
Cian: people have about their projects and not the language it's
Cian: great and people are always willing to have a
Cian: very open conversation and talk about different things
Cian: lessons learned and all that kind of stuff and i gotta say that's such a great
Cian: thing and we need to keep that so it's so important i don't think i would have
Cian: got into rust without that because it's what led it's But reading the Rust book
Cian: is what let me learn Rust.
Cian: It's such a nice way to learn. And I think we have to keep focusing on ways
Cian: to make it easy to get new people into learning the language,
Cian: to make it a better language, and to make people not think of it as a fad or
Cian: a systems programming language.
Cian: We have to focus on that path for beginners.
Cian: Tools like Clippy have done massive improvements there. like
Cian: that it's it's more than just a linter
Cian: it's a tool for helping you learn how to
Cian: write good an idiomatic rust like i and
Cian: when we focus on tooling that's natural to humans i think we just come up with
Cian: a better language and i think we have to keep that in mind when we develop rust
Cian: is it's tooling to make you as a human enjoy writing rust and make sure it's not a pain Where.
Matthias: Can people learn more about Cloudsmith?
Cian: So cloudsmith.com is our website. You can, if you want to use Cloudsmith or
Cian: think that you need better package management, check it out.
Cian: If you are interested in joining us, we are always hiring.
Cian: My team is experimenting with Rust. So if you're a Rust developer and want to
Cian: write some Rust in production, reach out, reach out to me.
Cian: I'll get my email dropped in the show notes so people can reach out.
Cian: And if they want to just talk about Cloudsmith, or package management or Rust, you can also reach out.
Matthias: Amazing. Cian, thanks so much for taking the time for the interview today.
Cian: Thank you. It's been a very pleasurable chat.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.