Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: My name is Matthias Endler from corrode and today I talk to Jon Gjengset from
Matthias: Helsing about keeping critical infrastructure secure and resilient with Rust.
Matthias: Unless you've been living under a rock, you know the guest of the show,
Matthias: but I will let him introduce himself.
Matthias: Jon, happy to have you.
Jon: Thanks, Matthias. So I'm Jon Gjengset. You may know me as @jonhoo online,
Jon: otherwise on the various channels, although YouTube I think is the primary one.
Jon: And I guess, who am I?
Jon: So professionally, I work at a company called Helsing, which I guess we'll talk
Jon: more about today. I work as a principal engineer, which means I end up jumping
Jon: all over the stack, owning wherever the biggest areas of attention are and where
Jon: I can be the highest leverage.
Jon: So I'm not necessarily in one particular team persistently over time,
Jon: but rather seeking out where I'm needed.
Jon: I've worked there for the past three years. Before that, I worked at Amazon.
Jon: I maintained and built their Rust build infrastructure.
Jon: And then before that, I did a PhD at MIT where I built a sort of novel distributed
Jon: systems database project that has since turned into a startup called ReadySet.
Jon: That's sort of the professional side of my career. And then the thing I'm more
Jon: widely known for in the Rust community is the Rust educational materials that
Jon: I make, primarily in the forms of videos on YouTube,
Jon: where I do long form content on building things from scratch in Rust and showing
Jon: people the actual code as we develop it and trying to give you know a realistic
Jon: view of the the the actual development workflow like what does it look like
Jon: when you go from zero to one on a code base in rust.
Matthias: Yes and that's exactly the reason why i
Matthias: was super excited to have you on the show to do
Matthias: this interview and i guess i can speak for everyone
Matthias: listening right now when i say thanks so much for all the content for educating
Matthias: all of us you're probably one of the most famous rustaceans out there and
Matthias: overall from what i can tell an amazing human being always happy to share
Jon: I
Jon: appreciate that thank you!
Matthias: And a lot of people know you from this educational context, but maybe not everyone
Matthias: knows that you're a principal engineer at Helsing.
Matthias: So talk a little bit about your role.
Matthias: You mentioned it already, but maybe you can get into the details.
Matthias: What do you do there and what is Helsing's responsibility right now?
Jon: Sure. So Helsing is a defense company based in Europe.
Jon: They started fairly recently. So it's only, I want to say, four or five years old.
Jon: I forget the exact start date, but around there. I joined in...
Jon: Sort of towards the end of, or middle to end of 2023, after I moved back to Europe.
Jon: And Helsing operates in sort of across the entire defense spectrum in Europe,
Jon: but they're specifically focused on software-enabled defense.
Jon: So how can we use software as the primary driver for, you know,
Jon: keeping up the deterrence capabilities of especially democracies and especially
Jon: focused on on Europe, given that's where the company is based as well.
Jon: It's already become quite a large company. So in the sense that,
Jon: you know, we're now somewhere in the vicinity of a thousand employees we operate
Jon: in, or we have offices in, I think, six different countries now.
Jon: So we have offices in Estonia and Poland, in the UK, Germany, France.
Jon: Am I missing any? I think that those are the sort of primary office locations we have.
Jon: And then obviously, you know, I work remotely from Norway for the company and
Jon: we have other people working elsewhere as well. But those are the sort of primary locations.
Jon: But for a company that's only four to five years old, that's quite the growth.
Jon: And the, you know, the reality is that that is partially because of the world
Jon: situation and the way things are developing in Europe, where a pretty,
Jon: you know, a pretty severe investment into European defense has been needed.
Jon: And Helsing has sort of seen fit to try to fill at least some of that space.
Jon: And so we operate across land, air, maritime.
Jon: We recently announced some work in space. And so the observation is that by
Jon: focusing on the software,
Jon: there's a lot we can bring to the table and we can bring it to the table pretty
Jon: rapidly because software in
Jon: general has more rapid development cycles than traditional hardware does.
Jon: And so, you know, I, in my work at Helsing, I've been primarily working in the air domain.
Jon: So working with, for example, the capability upgrades for the Eurofighter,
Jon: which is a European jet fighter program.
Jon: And currently I'm working more on the CA-1, which is our recently introduced
Jon: product that is essentially an autonomous UAV.
Jon: And so I'm working on building a lot of the software that's going to underpin that entire stack.
Matthias: If you compare Helsing's usage of Rust with AWS, can you see any differences?
Jon: Yeah, I mean, I think there are a number of differences, actually.
Jon: One of them is that Helsing was...
Jon: A sort of rust first company so the it
Jon: was very early on decided that the the entire stack
Jon: there should be rust based and and
Jon: should be you know wherever possible rust would be the language of choice and
Jon: we get we can get into exactly why that was but but that has sort of shaped
Jon: a lot of the software engineering it shaped a lot of the software that we've
Jon: built and the way that we built software whereas at amazon you know amazon was
Jon: originally a sort of java company and,
Jon: some healthy amounts of Perl in there.
Jon: And then over time, it's sort of grown into this polyglot company where there
Jon: are lots of different languages in use in different parts of the company.
Jon: And Rust is obviously one that's, I think it's beyond up-and-coming now.
Jon: It's actually being adopted in pretty serious use cases across AWS in particular.
Jon: But Rust was the sort of up-and-coming competitor.
Jon: It was not the incumbent. and that changes
Jon: how you adopt it right because it means that at amazon there's
Jon: a lot of infrastructure there's a lot of tooling that's not built for rust that
Jon: was built with other languages in mind and where rust now has to make inroads
Jon: into that ecosystem and integrate with all the things that people are used to
Jon: working with whereas the tell thing we can build everything specifically for
Jon: using rust because that is the the primary target language yeah.
Matthias: What i get from this is aws had infrastructure before and it was sort of a brownfield
Matthias: adoption of rust whereas at Helsing it more or less might have been a greenfield
Matthias: adoption i'm not sure if that is true.
Jon: Yeah i think that's right i mean it was a
Jon: is a pretty principled choice from the beginning of the company to say you know
Jon: we're going to be building technology where you know a lot of the technology
Jon: is critical right it's like it it it ends up having implications for life and
Jon: death decisions it ends up having implications for you know,
Jon: the correctness in in the military
Jon: domain like messing things up here is
Jon: very very costly and i don't mean in terms of monetary
Jon: cost right it's just costly in like a human cost and so as a result you need
Jon: to make sure that you build systems that are highly resilient highly reliable
Jon: highly predictable and and robust and rust is one of the mechanisms that we
Jon: wanted to use from very early on to make that be the case.
Jon: And, you know, because the company is relatively young,
Jon: we could also then say we're going to take the attitude here of everything is
Jon: going to be Rust from the get-go rather than say, you know, let's just try out
Jon: different languages and over time figure out what to choose.
Jon: We're just going to say that is what the stack is going to be built in.
Jon: So it's very much a greenfield approach in that way.
Matthias: And yet, even if you base everything on Rust, you still need to interface with
Matthias: existing libraries that might not be written in Rust.
Matthias: You might drive controllers or other hardware that maybe has firmware that isn't
Matthias: written in Rust. How is that story like?
Jon: Yeah, and I actually think that's one of the reasons why Rust was able to gain
Jon: such adoption in such a wide set of areas is because its interoperability story is decently good.
Jon: There are things I would still like to see Rust grow on here for sure,
Jon: but it gives you low enough control to be able to write firmware,
Jon: operating systems, embedded devices in it.
Jon: But it also makes it relatively easy to plug Rust into existing code bases,
Jon: whether that is underneath something like Python.
Jon: Python cryptography is a good example here where they are now using Rust under
Jon: the hood for some of the native components,
Jon: but also being able to put Rust above other things where you have an existing
Jon: C code base or even C++ code base, and you want to be able to interface with it from Rust.
Jon: Well, you can also do that. So you can sandwich Rust into wherever in the stack
Jon: you feel like it's appropriate, and then it can grow out from there.
Jon: And that's much harder with some other languages. Like if you have a runtime,
Jon: for example, and the being able to do foreign function interfaces either above
Jon: or below you tends to be more painful.
Matthias: Did you see cases where vendors provide Rust SDKs now that it's becoming more popular?
Jon: It's a mix. There are some domains where we're seeing more interest in supporting
Jon: Rust from industry vendors.
Jon: And then there's others where it's still, you know, all like all C++ or you'll
Jon: see places like the, if you look at the Terraform Kubernetes ecosystem,
Jon: a lot of that is Go and they're not really planning or they're not.
Jon: They don't seem particularly interested in saying we're going to also provide Rust bindings now.
Jon: And then you have, you know, more embedded ecosystems where maybe they already
Jon: have an existing sort of stack for development that's C-based maybe,
Jon: and they don't necessarily want to change that.
Jon: But I do think we're seeing vendors now think more about maybe a Rust SDK is
Jon: also something we want to provide.
Jon: I mean, Amazon is a good example where they, you know, now have a Rust SDK.
Jon: And it was because there was sufficient demand for, you know,
Jon: we want to build things that use AWS from Rust. So please let us do that.
Matthias: Maybe for some additional context, could you list the things where Rust is front
Matthias: and center at Helsing? What makes the stack?
Jon: So Rust is used throughout our entire stack, essentially.
Jon: So we use it for obviously anything that's backend services and the like,
Jon: but we also use it for anything that's close to edge devices.
Jon: So if you are writing code that's going to run on a UAV, on a drone,
Jon: on an underwater autonomous submersible, whatever it might be,
Jon: realistically, you have to use a language there where you have fairly tight
Jon: control over things like power usage, performance, predictable performance over
Jon: time, low overhead compute, and hardware control.
Jon: And so we use it a lot there in sort of embedded or bare metal systems,
Jon: but also things that are almost embedded or bare metal, right?
Jon: So things like, you know, essentially single application, but still runs on
Jon: Linux, but on a very particular compute board would still be something where
Jon: we write those applications in Rust.
Jon: We use it for a lot of our tooling is built in Rust a lot of our networking
Jon: technologies are built in Rust in fact it might be easier to sort of list the
Jon: things that are not done in Rust which I'd say,
Jon: there's primarily three categories so there's.
Jon: Web frontends, we tend to build in TypeScript instead, because you can do some
Jon: of it in WebAssembly, but realistically, TypeScript is where you're going to
Jon: get the most mileage here.
Jon: And then for AI research, like for the people actually working on the underlying
Jon: machine learning algorithms and training and the like, there we try to enable
Jon: people to use Python because that is where they're the most productive.
Jon: That's where a lot of the state-of-the-art research happens.
Jon: And that's where a lot of the existing tooling and libraries and such exist.
Jon: And so we don't really want to move all of that to Rust, even though we think
Jon: it's probably feasible.
Jon: It's not clear that it gives the bang for the buck in that area.
Jon: And then the last is we have some things around infrastructure that's in Go
Jon: I mean I mentioned Terraform and Kubernetes and stuff so there's like some stuff like that where,
Jon: Go has the best support ecosystem for writing things in that environment but
Jon: I'd say that's a that's a pretty small minority I'd say sort of in in order
Jon: of in order of adoption percentage I think it's Rust by far and then Python
Jon: and then TypeScript and then there's like a tiny bit of Go there and then there's like
Jon: There's always Bash because CI and everything, but I'd say those are the primary languages.
Matthias: According to our internal statistics, a large percentage of our listeners do
Matthias: use Rust in production in some way.
Matthias: But what does it feel like to work in a codebase where Rust is truly the answer
Matthias: everywhere, not just in one layer?
Jon: It's very convenient, right? Because it means that whenever I go to any codebase
Jon: across the company, chances are I know how to read that codebase.
Jon: Chances are it's written in Rust.
Jon: It's not just about being able to read the code. It's also the structure is
Jon: kind of predictable because they're all cargo projects.
Jon: They all have, you know, the layout you would expect from that.
Jon: It also means we can write tooling that is specifically designed for Rust projects
Jon: and then, you know, add support for Python, for TypeScript.
Jon: But we can build things specifically for, as an example, we have an internal
Jon: linter that we use to catch things that are more company preferences rather.
Jon: So we run Clippy, and then we also run this other thing that tries to lint for
Jon: things that are preferences we have for software engineering in Rust.
Jon: And so this can be things like looking for preferred libraries we have for things
Jon: like logging, for example.
Jon: Or it can be things like how we believe you should take internal dependencies,
Jon: like how they should be expressed in your Cargo.toml.
Jon: Or it can be things like, you know, how we think you should be using expect,
Jon: like the sentence structure we believe you should have inside of expect statements.
Jon: Like there's a bunch of that sort of stuff that you can do linting that I don't
Jon: think necessarily makes sense for the, like, it wouldn't make sense for upstreaming
Jon: into Clippy, but it does make sense for enforcing things internally.
Jon: And the fewer languages you have, the more you can build tooling specifically
Jon: for those languages to encourage the sort of software engineering excellence
Jon: practices you want to have for that language.
Matthias: Yeah, I'm sure a lot of people will be interested in taking a peek at that.
Matthias: So please open source it if you can, even if you don't want to upstream it.
Jon: I think it's mostly uninteresting in the sense that Clippy already has a relatively
Jon: straightforward mechanism where you can basically implement your own lints.
Jon: And so our internal tool is just that.
Jon: It's just a Clippy, but where all the lints are replaced with things we decided we wanted to look for.
Jon: And where there are things that we think are actually useful to Rust users more
Jon: broadly, then we would obviously open source those or upstream those into Clippy itself.
Jon: But for a lot of the others that are just like encoding of our software quality
Jon: standards, it's like, I don't actually think they're all that interesting outside
Jon: of the company. And there's certainly nothing about the linting engine itself
Jon: that's interesting, right?
Jon: Because it is just the clippy tooling that exists that we've added our own rules to, if you will.
Matthias: Which sort of begs the question, do you use that tool across the stack?
Matthias: Or do you make differences between embedded developers and backend developers?
Matthias: Do they use Rust differently?
Jon: No, that tool is used across the stack. It's not currently mandated.
Jon: So it's a thing that you can choose to add to your CI. And then we are strongly
Jon: encouraging everyone to use it in their CI.
Jon: But most of the rules here are around things that are good practice,
Jon: no matter where in the stack you operate.
Jon: This can be things like if you have an error type that is, let's say,
Jon: error result, and you are propagating that error type using the question mark
Jon: operator, you should have a call to dot context before it.
Jon: Right right and that makes sense for for any code base you're operating in the
Jon: same thing like we can include encode rules like always prefer dot context over
Jon: dot wrap error or vice versa right but we can at least encode that we believe
Jon: it should always be one or always be the other,
Jon: and so it's like those kinds of things more so
Jon: than you know it's harder to write a lint for
Jon: i don't know how you should handle back pressure in an application which is
Jon: the kind of thing that would differ between embedded development and cloud development
Jon: you can't really write a lint for that at the clippy level it's more it's almost
Jon: more architectural and so those things are not things we lint for do.
Matthias: You have an example for when dot context is helpful.
Jon: Oh i mean i love dot context i use it everywhere
Jon: the the thing that context gives you is the
Jon: ability to you know when you propagate an error up
Jon: through the stack that error might just be
Jon: something like permission denied or not found or something
Jon: and if you just propagate it with question mark then at
Jon: the point where you emit the error you know in your main in
Jon: your binary or something you would just get an
Jon: error printed that says file not found or permission denied and that's completely
Jon: useless and if by adding context you can include not just information about
Jon: like which file was being accessed but why was that file being accessed so imagine
Jon: things like you have a imagine you have config files that have include directives.
Jon: And so you might be like three levels down in an include hierarchy,
Jon: and then you reach a file you're not allowed to read.
Jon: Well, then the question is, which file included the one that said that you should read this file?
Jon: So even if you had the file name without the context, the file name might be
Jon: like, well, I don't know why foo.bar is being included.
Jon: That's not any of the files that are listed in my top level config.
Jon: And so the context allows you to give.
Jon: Both things like data context, but also programmatic context,
Jon: like why are we loading a config file here in the first place?
Jon: Or in which part of the application did we load this configuration?
Jon: Like was it, you know, I think that's loaded at startup, or maybe it's like
Jon: the code that handles configuration changes at runtime.
Jon: They both end up reading the config files, but which of them failed with this error?
Jon: And so in general, adding context like this is very, very helpful for making
Jon: your errors actually be actionable where they are where they're emitted yeah.
Matthias: And it also works across different crates even in the same namespace or outside.
Jon: Um well i mean context is just
Jon: the the way this is set up is if you use anyhow or you use eyre or you use miette
Jon: and you have this sort of opaque error type dot context allows you to a take
Jon: such an opaque error type and turn it into another opaque error that has the
Jon: original one plus this additional context as a part of the chain.
Jon: But it also lets you take an error that is not one of these opaque errors,
Jon: like one that comes out of some external library, and then chain this context
Jon: on top of that error and turn it into one of the opaque errors.
Jon: It obviously doesn't work so well if you have enumerated errors.
Jon: So if you have ones that are like, I have an error enum, and these are the variants,
Jon: then you can't easily call dot context on that unless you're willing to erase
Jon: the the concrete type of that error and turn it into an opaque error like an anyhow error.
Matthias: How
Matthias: high is the code reuse at Helsing
Jon: I think it depends on where in the stack you look, right?
Jon: So we have projects that span from relatively new sort of experimental,
Jon: we're trying something out,
Jon: to this is in a product and has been in development for several years.
Jon: And the amount of code reuse you do in one versus the other is pretty significant.
Jon: It's also the amount of code reuse you do within the domain versus across domains varies.
Jon: So there are more commonalities between something like a UAV and a strike drone.
Jon: Those have a lot more similar components than, let's say, a radar analysis system and a submersible.
Jon: Those don't share as much in common. So it's hard to give you one number for the amount of reuse.
Jon: But we do try to lean pretty heavily into if you've built something that is
Jon: useful elsewhere at the company or indeed outside of the company,
Jon: then build it as a reusable library.
Jon: And so, you know, you've already seen some of this come out on the public side of Helsing.
Jon: So we have some open source repositories like we have a tool called buffrs,
Jon: which is basically a package manager for protobuf files.
Jon: So it lets you take a collection of protobuf files,
Jon: create basically a proto.toml, which is equivalent to a cargo.toml that says, this is the version,
Jon: this is the package name, and you can take dependencies on other protos and
Jon: it resolves those and runs proto and gives you the flattened set of all the
Jon: files and resolves the dependencies between them.
Jon: We have a tool called Sguaba, which is a library for doing spatial math,
Jon: like rigid body dynamics or rigid body transformations in a type-safe manner.
Jon: And this is something that we use across a lot of our different code bases where
Jon: it's a real pain to get them right once, and you don't want every team to have
Jon: to get them right again and again and again.
Jon: So we build it once, and then we make that be a central library,
Jon: a central utility that every team can make use of, and then we also open source it.
Jon: And then we have other versions of this internally, like we have some tooling
Jon: for the Avro ecosystem where anyone who uses that internally now has access to our tooling.
Jon: And some of that we might open source, some of it we've already open sourced.
Jon: And so I'd say there's a pretty broad swath of reuse that's pretty intentional.
Jon: It's something where we see that there's a lot of cost to telling every team to reinvent the wheel.
Jon: And it's cost not just in terms of engineering time, but also in terms of correctness, right?
Jon: It means that you only manifest the bug once, you only fix it once,
Jon: rather than every team having to wrangle with the same complexities and the same bugs over time.
Matthias: Yeah, that's also what I like about Rust in general, because undefined behavior
Matthias: is a thing that you can centralize and then fix once, and then the entire ecosystem profits from the fix.
Jon: Yeah and i mean this is partially a property of just having a
Jon: good packaging system right like having a package manager like cargo means it's
Jon: pretty easy to turn something into a library it's pretty easy to take a dependency
Jon: on it and so that incentivizes doing this kind of of sharing which would be
Jon: more annoying and at least some other language ecosystems yeah.
Matthias: Also, thanks a lot. I always like it when companies open source their work. It's super amazing.
Matthias: And Squabble is certainly an amazing library too. You gave a talk about it.
Matthias: We will link to it in the show notes.
Matthias: And can you allude a little bit more to buffers?
Matthias: Why is it important to have a package manager for protobuf definitions?
Matthias: You must really like protobuf definitions and probably use them everywhere.
Jon: Well, so it comes as a pretty natural outcome of not having a monorepo, right?
Jon: Because you end up with different teams having protobuf files for their configuration
Jon: or their data exchange or whatever it might be.
Jon: And then you have some other team that also wants to make use of that team's protobuf files,
Jon: well then either you need to like be in the same repository as them or you need
Jon: to copy paste the files or you need to have a mechanism for publishing your
Jon: protofiles and then getting them into another codebase,
Jon: at which point you need versioning because you're not tagging a particular commit
Jon: of it, you need the version of it.
Jon: And thus you start getting transitive dependencies where maybe there's a shared
Jon: definition for maybe just data types even, so not full structure of gRPCs,
Jon: just some of the core data types.
Jon: And you want those to be shared across two different repositories inside of one team.
Jon: And then they both take your dependencies on that. And then anyone who takes
Jon: a dependency on either now also needs the transitive dependency.
Jon: And so you very quickly run into the situation of we basically need a package
Jon: manager here and we need versioning, we need packages.
Jon: And that's what Buffers was built to solve.
Matthias: What do you use protobuf for internally?
Jon: So we tend to like having sort of textual code representations of protocols
Jon: because it makes it a lot easier to decouple the two sides of any given communication pattern.
Jon: And, you know, protobuf is one way to do that. Avro is another,
Jon: and there exist many others as well.
Jon: It also means that it's easier to,
Jon: Once you create that protocol divide, and if you encode the protocol separately
Jon: from any given code base,
Jon: it now makes it easier as well to change the technology choices on either side,
Jon: or just completely reinvent one side by rebuilding it from scratch,
Jon: but being able to reuse the definitions.
Jon: Why Protobuffer specifically? Specifically, it's a pretty mature toolchain and
Jon: ecosystem, and it has a proven track record of working well,
Jon: being efficient, having support for many languages.
Jon: And for that reason, I think it's a pretty obvious default choice.
Jon: But as you'll see, or as you might have seen from my streams,
Jon: for example, there are things where we use Avro instead of protobuf.
Jon: And the rationale for that is there are some features that Avro has that Protobuf
Jon: does not, where if for your particular use case, those features are worthwhile,
Jon: well, then you should pick the technology that has those.
Jon: So as an example, Avro support for this thing called logical types,
Jon: where you can annotate particular fields of a type with, you know, this is a F64,
Jon: but I'm going to annotate it with the logical type velocity in meters per second.
Jon: And protobuf doesn't really have that
Jon: mechanism for adding sort of richness to the the types
Jon: of fields there's like you you
Jon: can kind of hack your way there in protobuf but in abro that's just directly
Jon: supported by the tooling it also has better support for streams of blobs in
Jon: protobuf you don't really have that in protobuf you have it in gRPC but we don't
Jon: necessarily use these protocol definitions for RPC mechanisms.
Jon: We often use them to represent just the data format that's being exchanged in
Jon: various different formats and protocols.
Jon: And so just because we're using protobuf does not mean we're using gRPC everywhere.
Jon: Same thing for Avro. We might be using the Avro IDL for expressing data types
Jon: for protocols. We're not necessarily using the RPC mechanisms that are built on top of Avro.
Matthias: Is it time to write an Avro package manager now?
Jon: Maybe. It's not impossible. I mean, running a package manager,
Jon: if it is as simple as, you know, assign a version, create a bundle,
Jon: upload it somewhere, is not that bad.
Jon: And I mean, if you look at buffers, you know, there is some amount of complexity
Jon: there, but it's not, you know, monumental. And we could probably pretty easily
Jon: take buffers and create an Avro version of buffers.
Jon: The thing where it starts to get really gnarly is when you want sophisticated
Jon: semantic versioning resolution, for example.
Jon: There's been some really cool work on...
Jon: There's this effort called PubGrub, which is essentially trying to write a version
Jon: resolver that can be reused across packaging ecosystems,
Jon: across different types of both specifier semantics in your dependency thing, thing,
Jon: but also write it in such a way that it's reusable.
Jon: So you could use it in NPM, you could use it in cargo, you could use it in whatever
Jon: package manager you dream of, and just get really rich resolution of package
Jon: dependencies, because it turns out that's a fairly complicated problem.
Jon: And then PubGrab also tries to give you good error messages from resolver failures,
Jon: which tends to be something that many package managers struggle with,
Jon: because they build version resolution in this kind of ad hoc way.
Jon: And with PubGrub,
Jon: Cargo hasn't adopted PubGrub yet, but it's sort of on the long-term roadmap.
Jon: And it has been for a while, so don't hold your breath.
Jon: But it does mean that, for example, buffers as of today only has exact version lookups.
Jon: So if you take a dependency on 1.2.3 and 1.2.4 is released, then buffers will not pick it up.
Jon: But if we integrated PubGrub into buffers, we would get this version resolution
Jon: behavior that we would say, the default should be semantic version like caret
Jon: matching like what Cargo does.
Jon: And at that point, moving to say Buffer's version that supports Avro feels like
Jon: it shouldn't be that bad because there's not too much that is protobuf specific inside of Buffer's.
Jon: It is more like a collection of files with some metadata that annotates version
Jon: and dependencies and then being
Jon: able to bundle those up into tarballs that you can publish somewhere.
Jon: And if you squint at it, that's like most of what many package managers are.
Matthias: Okay, so we understand that management of those protobuf definitions is really
Matthias: important for housing. But what do you use it in the first place?
Jon: So we do use gRPC for some parts of the tech stack, although gRPC tends to be
Jon: best suited for environments where communication is fairly reliable and predictable.
Jon: And you have like TCP, you have stable IP networks and the like.
Jon: So there we use gRPC, and then protobuf is a good way to make use of it.
Jon: But protobuf is also useful for just describing data formats.
Jon: Data formats is the wrong word, but like...
Jon: The structure of messages that are going to go over a network or in some cases
Jon: that will go to disk, although protobuf tends to be less well-suited for that,
Jon: but certainly for anything that
Jon: has to go over a network, but it doesn't necessarily need to go over gRPC.
Jon: So for example, there are especially closer to the edge networks that we have
Jon: where you have drones flying around where there's radios with very limited bandwidth,
Jon: connections that come and go you get jammed so
Jon: you lose connectivity or your bandwidth gets severely reduced because
Jon: you're flying through a zone where there's a lot of interference whatever
Jon: it might be in those environments gRPC is not going to work like realistically
Jon: you can't use tcp it will like the the network is simply too poor and too dynamic
Jon: for that to work plus you want to make use of things like you know if you have
Jon: a radio network it's inherently broadcast and so you send one packet,
Jon: you want to make it useful to as many other peers in your network as possible.
Jon: And so that's an example where we're not using gRPC, we're not using TCP.
Jon: And in fact, we've built our own stack that is reliant on CRDTs,
Jon: on conflict-free replicated data types,
Jon: that tries to build a distributed network where you can still get reliable exchange
Jon: of information, even in the presence of severe packet loss,
Jon: network reordering, packet reordering,
Jon: node sending, frequent updates over time, and you want to make sure you only
Jon: accumulate the updates that are newer, that old data gets erased when new data
Jon: replaces it, even if you get the packets out of order.
Jon: Like this sort of very...
Jon: Traditional distributed systems when they're not in a cloud environment setting.
Jon: But there, for the CRDT stuff we've built, we're still using protobuf for the
Jon: actual data definitions.
Jon: So the definitions of the schema, effectively, of what an application,
Jon: what data an application might write, might read, what gets sort of stored and
Jon: sent, all of that is still in protobuf files because, again,
Jon: it's something that people know.
Jon: The tooling is good around it. You know, it has editor support and all of that stuff.
Jon: We have buffers for being able to give you versioning over those schemas.
Jon: And so it makes a lot of sense to reuse that ability to describe your protocols,
Jon: your data definitions, even though the underlying exchange technology is very different.
Matthias: And what are reasons for using CRDTs specifically?
Matthias: You mentioned a few things already, but what's the bird's eye view?
Matthias: When would you use it? when wouldn't you use it and maybe what is it in the first place.
Jon: Yeah so so crdts
Jon: are at the the very basic level
Jon: algorithms or it's
Jon: a it's a an exchange data type so it's a it's a data type that comes with some
Jon: algorithms that define what to do when these messages are exchanged over a network
Jon: or between peers in whatever way they are, such that if.
Jon: Imagine you have set up a distributed system where you have,
Jon: let's say, three nodes in that system, nodes A, B, and C, and node A and B concurrently
Jon: make edits to some underlying document or whatever.
Jon: Let's use document as a good example. The shared document between A,
Jon: B, and C, A is making edits to the document, B is making edits to the document,
Jon: and they can't talk to each other, but they're able to send packets to C.
Jon: C is now going to observe the changes from both A and B. how does it reconcile the changes from A and B?
Jon: And CRDTs are the data type plus algorithms that make C able to reconcile those
Jon: changes in such a way that if A and B's edits were not conflicting with each other,
Jon: then there is no conflict observed by C. It just observed both edits.
Jon: And if they do conflict with each other, then C has a conflict resolution strategy
Jon: in the sense that it can detect that there was a conflict and it can decide
Jon: what to do about that conflict in such a way that afterwards there is no conflict.
Jon: And so imagine for example that your document is like a key value
Jon: store if a writes to key foo and b writes to
Jon: key bar then all the edits to foo and all the edits to bar should just come
Jon: into c and there should be no problem if a deletes foo and b updates foo so
Jon: the same key and then they send to c then c is now going to have to decide what
Jon: to do when it observes a delete and an update at the same time.
Jon: And the CRDT would be something like, you know, it both informs you about the
Jon: metadata you need to add to the packets to detect that these are the same key and they happened
Jon: concurrently with each other rather than, let's say, the update happened first
Jon: and the delete happened after, in which case you should always take the delete.
Jon: But if they actually happen concurrently, which the CRDT metadata will tell you,
Jon: then the CRDT algorithm will tell C whether it should prefer keeping the updated
Jon: value or prefer the delete and the sort of baked into the data type itself,
Jon: how to resolve that conflict.
Matthias: And is that resolution always the same, or does it depend on the use case?
Matthias: Can you configure the resolution?
Jon: Yes, you can choose different CRDTs depending on the outcomes that you want.
Jon: So for maps, for example, the most common way to construct a map is something
Jon: called an observe removed map, where you can only remove items or updates that you have observed.
Jon: So in the case before of A, updates foo, and B, deletes foo,
Jon: I think I said it in reverse, but it doesn't matter, then because B did not
Jon: observe the update to A, it is not allowed to remove the update to A.
Jon: And therefore, the update to A will win out. And the result will be that foo
Jon: will not be deleted in the resulting resolution.
Jon: And the protocol and the algorithms and the metadata ensure that this is always the case.
Jon: But you can choose, there's a different CRDT that is not the observed removed
Jon: map that allows you, I forget the name of this one, but there's a CRDT for maps
Jon: that is specifically a, like removes wins,
Jon: where you are guaranteed that if you have a remove and an update,
Jon: the update will be removed.
Jon: And so these are different semantics you can choose by choosing the appropriate CRDTs.
Jon: And the CRDTs tend to also be composable. So you can say that,
Jon: you know, you have a key value map where the values themselves are also CRDTs of a particular type.
Jon: And you can structure them in this way where you choose the
Jon: semantics at every layer by choosing the appropriate crdt mechanism
Jon: and the the rule of crdts is
Jon: that as long as you observe all the same operations as another node in the system
Jon: you agree on the final state so they're guaranteed to be sort of commutative
Jon: and associative to the point where eventually everyone has the same state as
Jon: long as they get to exchange all messages.
Matthias: I sort of have immediate follow-up questions, two of them. The first one would be...
Matthias: How do you decide on such data structures? Is there a team meeting and then
Matthias: someone kind of knows algorithms really well, algorithms and data structures and proposes that?
Matthias: And maybe there might be competing algorithms that maybe you considered.
Matthias: And have you been there when the decision was made?
Jon: Yeah, so our CRDT distributed system I built.
Jon: And part of that was out of a conviction that.
Jon: GRPC is not something you can run
Jon: on the edge network so we need something else and the
Jon: question then becomes what is the something else and you
Jon: know i happen to have a decent amount of background in building distributed
Jon: systems and so i've had a decent idea of what the different options were and
Jon: crdts felt like they fit this particular set of use cases or at least the use
Jon: cases we could predict we would we're going to have quite well there are other
Jon: designs you could come up with here, but they come with a different set of trade-offs.
Jon: I think at the time, I went into it with a conviction that this is the right way.
Jon: As you build more and more complex and compounding stacks, you start having
Jon: to document these decisions as well so that the motivation and the rationale
Jon: for making the choice is not lost to time.
Jon: And that's where you end up writing architecture decision records like ADRs,
Jon: where you write down, here's the problem statement, Here's the decision we made
Jon: for what algorithm to use.
Jon: Here are the options we considered and why we decided to discard them.
Jon: So that for the future, people have insight into the decisions you made.
Jon: And also, you know, the process of writing this document becomes the way you
Jon: take the decision is you convince the group as part of writing this document
Jon: that all the options should be discarded except for the one that you choose.
Matthias: And I'm assuming there's still ongoing research on CRDTs.
Matthias: Would you amend any of the prior decisions now if you had the chance?
Jon: I don't think so. I think the CRDT design has actually worked out very, very well.
Jon: And, you know, when we implemented the CRDTs in the first place,
Jon: it was built off of a very recently released paper.
Jon: So we started doing that implementation in...
Jon: Shortly after I joined, actually, so this is sort of end of 2023,
Jon: and we were implementing it based on a paper called DSON, which was released, I want to say 2022.
Jon: So it was very much sort of bleeding edge technology already when we started implementing it.
Jon: And then as part of implementing that set of data structures and algorithms
Jon: and protocols, we also effectively extended the research into things that were
Jon: needed for the operational use cases and production use cases we had in mind.
Matthias: Sounds like a decent paper and.
Jon: Maybe i mean we did actually go a
Jon: lot of back and forth with the authors of the paper we've also open sourced
Jon: the the core of that implementation so there's a on crates.io there's a crate
Jon: called decent dson that is the core of that crdt and that compound set of crdts
Jon: precisely because we think this might be useful to other people and because
Jon: we think we made, you know,
Jon: innovations in the space that, you know, were not in the decent paper.
Jon: And some of this was purely because in order to put this into production,
Jon: we had to do a lot of both optimization, but also debugging to make sure it's
Jon: extremely reliable and fast.
Jon: And as part of that found corner cases that like in the paper were either handled in, you know,
Jon: suboptimal ways, or in fact, we found some bugs, at least in their prototypical
Jon: implementation, not necessarily in the algorithms that we wanted to correct.
Jon: And therefore also wanted to publish.
Matthias: And when you did the implementation and you found those bugs,
Matthias: did the Rust type system surface them easily?
Jon: Yes and no. There were some where the Rust type system was helpful.
Jon: So the original DSON paper was accompanied by a research prototype written in JavaScript.
Jon: And when writing the sort of rust
Jon: encoding of that same algorithm there were
Jon: definitely places where it pointed out that they had relied
Jon: on the javascript type system
Jon: being fairly forgiving where they were just sort of interchangeably using two
Jon: different types that just like really should not be mixed because you can very
Jon: easily get into bugs that way and i think we found like one or two bugs in the
Jon: in the research prototype again not necessarily in the algorithm but in the
Jon: research prototype that were because of this.
Jon: But I think decent amount of the sort of nuances of the algorithms,
Jon: especially when it comes to performance, for example,
Jon: are not things that would be caused by the Rust type system as much as they're
Jon: caught by a lot of testing,
Jon: like property-based testing, fuss testing,
Jon: and doing a lot of performance benchmarks and tracking down the root causes.
Jon: They're like things that are hard to catch in the type system.
Matthias: Now when you look at all of these things that you've implemented so far and
Matthias: there's certainly a lot of amazing things in there I also want to check out
Matthias: your CRDT implementation now,
Matthias: what would you say was the biggest learning since you started modeling logic
Matthias: in Rust how do you model your types nowadays things that people can learn and apply in their own work.
Jon: I think there's a trade-off that you learn over time of where is it worthwhile
Jon: introducing more types and where is it not?
Jon: Where is it worthwhile making things generic versus where is it not?
Jon: I don't think there's a hard and fast rule that you can just always follow.
Jon: But over time, you develop a sort of intuition for this doesn't feel like a
Jon: good use of a type or this feels like it would be dangerous if I don't add a type.
Jon: Like you can start to sort of predict the bugs that people will make if you
Jon: don't introduce a new type.
Jon: And you also, whenever you introduce a new type, you like feel a tingling in
Jon: your hands about the amount of pain you just introduced to people using it because
Jon: now there's an extra type or previously that it didn't need to be.
Jon: And so I think the thing I've learned, and I think this is not a,
Jon: it's not a moment in time learning. It's sort of a lesson over time is to better tune that trade-off.
Jon: And I think one of the observations I've come to is.
Jon: Representing things in the type system is extremely valuable,
Jon: and people don't do it enough.
Jon: But you have to balance it against the pain of using the library that you've
Jon: developed that has all of these types.
Jon: And you need to really keep in mind what am I actually gaining
Jon: in terms of the safety I add to the system when I introduce all
Jon: these types and what is the cost of the people using it
Jon: and the way you do that is by making sure that when you
Jon: write for example a library like Sguaba for example for the rigid body transformations
Jon: also write code that uses that library while you are writing this type states
Jon: library because it will immediately show you just how painful the consuming
Jon: code ends up and that will help guide your way into,
Jon: okay, maybe I don't need a dedicated type for this.
Jon: So an example here maybe is in Sguaba, we have a type for WGS84,
Jon: which is basically GPS coordinates.
Jon: So latitude, longitude, and altitude. Now it turns out that WGS84 is actually a moving standard.
Jon: It's a moving standard because the parameters.
Jon: Earth changes over the course of
Jon: time like you get you have you know plate drift
Jon: and drift of the magnetic north pole and like they also update
Jon: the reference ellipsoid sometimes when they get better estimates of like the
Jon: roundness of the earth and so as a result there's not actually one wgs84 there's
Jon: like multiple over time and if you capture a coordinate right now and then in within 10 years.
Jon: You try to take the same coordinate and plot it on a map, then it wouldn't be
Jon: in the same place as where the original one was.
Jon: So if you read out the GPS coordinates of the Eiffel Tower, the actual GPS coordinates
Jon: of the Eiffel Tower will change over time.
Jon: But they'll change very little, usually.
Jon: But it doesn't mean that technically, like
Jon: you kind of want the type system to represent the
Jon: point in time at which this measurement was
Jon: taken the more extreme case here is so
Jon: you have this thing called local tangent planes which is basically imagine
Jon: a plane is flying and it has a some gps coordinate
Jon: and then you want to know you know the the relative
Jon: location to the plane so something like
Jon: either in front right down coordinates or north east
Jon: down coordinates so it's like this thing is one
Jon: kilometer north three kilometers east and 500
Jon: meters up from me then that is a it's a local tangent plane to the current location
Jon: of the plane so i record that coordinate but now the plane moves then you kind
Jon: of want to represent the fact that this was a coordinate relative to that plane's
Jon: position at this point in time.
Jon: The reality is if we actually tried to encode that information in the type system
Jon: of Sguaba, it would be impossible to use because every type would be distinct.
Jon: You just like, there would be no easy way to move between different coordinate systems.
Jon: You would end up with these like, WGS84 would have like three different generic
Jon: type parameters that are like involve time. And then how does that translate
Jon: into other coordinate systems that involve time?
Jon: And it would be more accurate, it, but it also would be much more painful to
Jon: use, and it's not entirely clear that it eliminates very many classes of bugs.
Jon: Quite to the contrary, it might introduce more, because now if you get the wrong
Jon: time bases, then nothing compiles, and then you pull shortcuts to try to get
Jon: out of the mess of errors you end up with.
Jon: And so for Sguaba, I made the explicit choice to say, this library will not
Jon: represent time in the type system.
Jon: And that does reduce the safety you get from the type system,
Jon: but it also makes the library much more pleasant to use, which increases the
Jon: number of people who will use it,
Jon: and therefore overall increases safety compared to if I had put it in.
Matthias: Is that an example of the conflict between ergonomics and correctness?
Jon: I think that's right.
Jon: And, you know, it's like, I don't actually think it's sacrificing correctness,
Jon: it's sacrificing precision.
Jon: And precision can be useful for correctness, but you can also have correctness
Jon: without that precision, right? So you can write correct programs that don't
Jon: have time represented in the type system.
Jon: It just means that there's some classes of bugs that you don't get to eliminate
Jon: through the type system. but that doesn't mean you end up not having correctness yeah.
Matthias: So from your lens it's still correct because it encodes everything that you
Matthias: want the type system to encode but you just explicitly leave out a thing that
Matthias: you don't want to be so precise on yeah.
Jon: And where where i think you know i'm going to leave it to the users of this
Jon: library to get that part correct.
Matthias: Do you make a difference between library code and binary code?
Matthias: Do you write different code if you had to write application-level Rust versus library-level Rust?
Jon: Yeah, I think I do in the sense that when I write library-level Rust,
Jon: I think a lot more about the programmatic API that I present.
Jon: So that includes not just documentation, right?
Jon: Obviously, you need to write good documentation for a library to be useful,
Jon: but also the structure of that API.
Jon: What are the backwards compatibility hazards? Where do I think I might put myself
Jon: into a trap when it comes to breaking changes down the line?
Jon: So I want to be conservative about what things I expose in the public API because
Jon: those are things I can't change later unless I do a breaking release.
Jon: The way that you propagate errors might be different because you might want
Jon: library consumers to have a better ability to deconstruct the error and figure out the origin.
Jon: Whereas in a binary, usually what you want is to present a chain of errors to
Jon: the user that results in something actionable on their end to fix the problem.
Jon: And so I do think the design ends up somewhat different.
Jon: I don't think it changes the internal writing of the code very much,
Jon: but it changes how much focus you put on the external API.
Jon: But for a binary, of course, you have to think about what does the command line
Jon: interface look like for that binary and so that also requires thought but it's
Jon: it's a different kind of design process earlier.
Matthias: You mentioned that you also want to write the application level code that goes
Matthias: along with the library code or actually vice versa does that mean you start
Matthias: with a main.rs model out your types and then gradually move them into library crate.
Jon: It can i do do that sometimes as well but more commonly it means i already have
Jon: at least two code bases that,
Jon: have a need for this library. And so I'm going to create the library and see
Jon: how it affects those two codebases.
Jon: Because then I have a real set of use cases that I can test out how the library feels.
Jon: And I mean, this was the case for Sguaba, for instance, was we had codebases
Jon: internally that had to do this kind of spatial math.
Jon: And they already had code for it, like they were working code bases but
Jon: the that code was like hard to
Jon: review brittle had a bunch of like magic constants in
Jon: there and so it didn't it felt like
Jon: you know we've had to solve this problem at
Jon: least twice so we should turn it into a reusable library that is you know well
Jon: designed well tested and because it would be recommended going forward and so
Jon: that is what informs the the the design of the library is the the the evident
Jon: need from the things you've you've already built when.
Matthias: You build squab up what was your testing strategy was it based on unit tests
Matthias: integration tests or or property-based testing or fuzzing.
Jon: It's it's all of the above so the there's both a bunch of unit tests in there
Jon: there's also a lot of equivalence tests to other crates that implement some
Jon: subset of the functionality.
Jon: So for example, there's a crate called NavTypes that implements,
Jon: for example, conversion between WGS84 and ECEF, which is another Earth-based coordinate system.
Jon: And so there's a property-based test inside of Sguaba that basically generates
Jon: random points on Earth and then converts them back and forth using Sguaba,
Jon: converts them back and forth using NavTypes, and then checks that the results are near each other.
Jon: And then there's also just general property-based testing that does things like,
Jon: you know, if you pick a random coordinate on Earth,
Jon: run it through like back and forth through WGS84 and ECEF, which is a lossy
Jon: conversion, run it through that 10 times and see how much degradation you get.
Jon: And you get guarantees about, you know, you get probabilistic guarantees about
Jon: how much deterioration will you see over time.
Jon: And then we also have a bunch of tests around the enforcement of the type system, right?
Jon: So both tests to make sure that you can express the correct computations,
Jon: but also compile fail tests that say you cannot try to use a coordinate from
Jon: one coordinate system as a coordinate in a different one without an explicit conversion.
Matthias: You put in so much work into Sguaba, similar libraries, and then you decide
Matthias: to open source that work. that's a big gift why do you do that why open source those libraries i.
Jon: Think it's a it's a combination of factors one
Jon: of them is the the the traditional you know by having more people being able
Jon: to look at a thing you're more confident that it's correct and i think that
Jon: applies to to open sourcing software in this context too and and i think there's
Jon: a related point to that which is,
Jon: you know, we operate in the defense sector.
Jon: And so the systems that we build, we want to have as much confidence as we can is correct.
Jon: But we also want to sort of, to the extent that we can, give people the ability
Jon: to look at how we build software for whether they think we are building software in a responsible way.
Jon: And obviously we can't open source like the actual products we're developing,
Jon: but at least one stepping stone is to open source some of the techniques,
Jon: some of the tools that we use in order to produce this software to the level
Jon: of sort of reliability that we want to give.
Jon: And so open sourcing these kinds of libraries, I think,
Jon: gives hopefully both some feeling of transparency on that part,
Jon: but also inspires some amount of confidence that we are building this software,
Jon: at least at a technical level with care and so I think it matters to demonstrate
Jon: that and then I think there's a there's a.
Jon: Sort of a wanting to give back kind of feeling, right?
Jon: Of we get a lot from the Rust community.
Jon: And I mean, we are sponsors of the Rust Foundation partially for this reason to give back.
Jon: But the other way to give back is to make sure that when we build things that
Jon: we think are useful to other people than us, that we make them useful to other people than us.
Jon: And then I think that there's obviously a cynical angle too,
Jon: right, which is you put things out there so that other people get to look at
Jon: interesting things you've built and then go, I also want to work on those things, right?
Jon: Do you get to actually show some of your code, show some of your development
Jon: styles, show some of the problems you're working on and hopefully get other
Jon: people interested as a result?
Matthias: Yeah. In preparation for this interview, I read through some of the blog posts
Matthias: on the Helsing Tech blog.
Matthias: And I have to say, it's astonishing and certainly enticing to know what sort
Matthias: of problems you're working on.
Matthias: And i think it attracts a certain group of people who are interested in solving
Matthias: hard problems and working with rust because they know that rust is the right choice.
Jon: I i think that's true and if you
Jon: think about the flip side right imagine we didn't open source anything we
Jon: didn't write any technical blog posts then i think the the
Jon: first question would be well why not what do you have to hide right but
Jon: the the other observation is how do you hire like
Jon: especially you know talented engineers who are curious
Jon: about technical depth if the only
Jon: thing they see is sort of the the product side of
Jon: things externally like they don't have direct access
Jon: to the engineers we have internally so you kind of you would have to apply go
Jon: through interviews and then get to talk to the engineers which is a lot to ask
Jon: of someone who's still like deciding whether they might want to join the company
Jon: and so by opening the doors a little bit and showing some of the work we work
Jon: on and the way that we actually do engineering,
Jon: You give people more insight and therefore hopefully more to go on when deciding
Jon: whether this is a place where they would want to work.
Matthias: Do you get any contributions, pull requests, people creating issues?
Jon: We do. It varies between the different projects, right?
Jon: So the different things we've open sourced are varying degrees of useful to
Jon: other, like the Dson Crate, for example.
Jon: I'm not expecting lots of people to make use of because it's a very,
Jon: you know, you need to have a very particular use case for those to be the most useful to you.
Jon: And then other things like the Avro tooling, for example, I actually expect
Jon: could become quite popular because a lot of people use Avro and this thing gives
Jon: you faster and better error.
Jon: Like it's faster than the upstream version and it gives you better error messages.
Jon: So we might get a bunch of people using it, but it's fairly new.
Jon: Squab, I expect, would probably be quite popular.
Jon: And that's also the one we've seen the most interest in, actually,
Jon: of people wanting to find issues, reuse it, contribute to it,
Jon: and we take those contributions seriously.
Jon: Buffers has been a bit of a mix where people have had interest,
Jon: but I think the companies where this becomes the most relevant,
Jon: many of them have monorepos and therefore don't need this particular tech.
Jon: As you need to both have a need to do versioning and packaging of extended dependency
Jon: chains of buffers, of protobuf files, and also not have a monorepo.
Jon: And that combination, I think it's somewhat rare, although not unique.
Jon: But in general, we do see interest on the projects we put out there.
Matthias: And I believe one other angle is that a lot of people might just be interested
Matthias: in knowing how a Rust expert structures a library.
Matthias: And there's very little material out there outside of maybe a handful of popular
Matthias: crates and maybe a bunch of blog posts on how to write advanced Rust code.
Matthias: And a lot of people want to learn by osmosis, by reading what other people have
Matthias: written that they deem to be Rust experts.
Jon: Yes, I think that's also true.
Matthias: After all those years, do you still see yourself as an educator and do you do
Matthias: Rust education outside of Helsing or also within Helsing?
Jon: Yeah, I very much see my job as an educator. And it's something that I have,
Jon: you know, I think a pretty deep passion for.
Jon: I really enjoy, you know, that moment where you can experience someone else
Jon: understanding something. That makes me very happy.
Jon: And so, you know, I continue to do education outside of Helsing.
Jon: And it's the same thing I did at Amazon as well, where, you know,
Jon: a lot of my live streams and stuff, that all continued while I worked there.
Jon: And it's the same thing as Helsing.
Jon: I do some amount of education internally at Helsing as well.
Jon: Although internally, it has more of a sort of reactive nature, right?
Jon: Where people will poke me and be like, hey, Jon, why doesn't this work?
Jon: Or how should we do this? Or because like we do some amount of office hours
Jon: internally, but also more of that, you know, we have like a Rust help channel
Jon: where people ask and then occasionally I'll get, you know,
Jon: poked explicitly and be like, I think Jon wrote a blog post about this,
Jon: or I think Jon implemented something along those lines.
Jon: But I actually think the most amount of education I do that has value internally
Jon: is actually the external education that I do.
Jon: So I know that a lot of the engineers we have at Helsing have learned or partially
Jon: learned Rust through my public educational resources, right?
Jon: And that is also how some of them continue to learn new concepts in Rust,
Jon: is to observe the same teaching resources that I put out publicly.
Jon: And this is also why I think Helsing is quite supportive of me continuing to
Jon: do my public education, because not only is it sort of a, cynically speaking,
Jon: like a sales thing, right?
Jon: Like it's valuable to the company to have someone who's seen as a Rust expert,
Jon: both publicly be operating as a Rust expert and be employed by them.
Jon: But I think more meaningfully, it also means that more people are able to learn
Jon: Rust through the abilities or through the teaching that I do,
Jon: which means that there's a bigger hiring pool for Helsing to draw from.
Jon: But also, the people that we hire, hopefully, have then also learned more things
Jon: about Rust because I've produced those intermediate teaching resources.
Jon: And the people at the company can continue to improve their skills by me continuing
Jon: to do teaching. So it ends up being a virtual cycle in a way where it's good
Jon: for the company, which is good for me, which is good for the company, which is good for me.
Jon: And I do think there's also some amount of recognition internally at the company that...
Jon: You know, if I started just building internal teaching resources,
Jon: it would be seen as a bit of a shame, right?
Jon: It would be like, why are we not making this material public when it could be?
Jon: There's nothing secret about it. It's just how do you build good Rust code?
Jon: How do you engineer high quality Rust products?
Jon: Then that feels like something we should be sharing back to the community because
Jon: those aren't, they're not industry secrets, right? They are just things that
Jon: are beneficial to everyone using Rust. And I think it's also the case that,
Jon: If we make the Rust community better, we benefit as a result,
Jon: not just through hiring, but also because the quality of the crates that exist
Jon: in the open source ecosystem will be better.
Jon: The tooling will be better. Everything gets better if the community improves.
Jon: And so there's just a lot of positive externalities here and positive feedback
Jon: loops that mean that me continuing to do the public part of education is valuable.
Matthias: And with 200 plus hours of video material out there of you teaching rust i sort
Matthias: of think it's inevitable that people share videos of you internally without you knowing.
Jon: Oh that that definitely happens i
Jon: mean this happened at amazon too where the moment
Jon: i got on the inside i kept finding places where
Jon: people had referred to either things that like videos
Jon: i'd made or blog posts i'd written or crates i've published
Jon: being like Jon go look at this thing or like you should read Jon's thing about
Jon: this and that is always fun it's the same thing when when we do hiring a decent
Jon: number of the people that go through the interview process say that either they
Jon: learned rust through me or they heard about the company through me and that
Jon: is a weird feeling for sure do.
Matthias: You see people taking it to the extreme sometimes where maybe they they learn
Matthias: about an advanced concept and they want to apply it at work and during code review you find well,
Matthias: it's expressive it's certainly concise but maybe not maintainable by a larger
Matthias: team and where do you draw the line.
Jon: Yeah, I actually think this is not just a like intermediate Rust programmer thing.
Jon: I think this is pretty common across Rust, even from the early days,
Jon: is that people see all of these tools and techniques that are possible in Rust,
Jon: and then they immediately want to make use of them.
Jon: And I think the new type pattern is a big one, right? Like I can define custom
Jon: types for everything, and then everything is type safe.
Jon: And people start using that pattern to the extreme. And to the discussion we
Jon: had earlier around the trade-off space here, they just navigate the trade-off
Jon: space by always picking the most type-safe thing.
Jon: And we see the same when you look at the hesitance to use locks,
Jon: the hesitance to use RC and ARC.
Jon: So people try to use lock-free algorithms and use references with lifetimes
Jon: everywhere. And you end up with multiple lifetime annotations.
Jon: And no one wants to clone anything. and everything has to be monomorphized so
Jon: there's no dynamic dispatch.
Jon: And people really lean into every possible feature that Rust gives you.
Jon: And it makes it really painful to program in the language. It makes it painful to review the code.
Jon: It means that you end up constructing suboptimal software architecture because
Jon: you can't express the architecture you want with the borrow checker,
Jon: with the type checker, or even just with your current knowledge of the language.
Jon: And so I do tend to see.
Jon: Especially in people who haven't built production code in Rust very much,
Jon: they tend to lean overly much into some of these patterns. And then you kind
Jon: of have to pull them back and be like...
Jon: It's okay to clone here. It's okay to put this thing behind a mutex.
Jon: It's okay to not have a new type for this particular string representation of an email, right?
Jon: And over time, people learn that distinction.
Jon: But that is part of the education you kind of have to do on the job is see the
Jon: code that people write and then course correct as you do for where they're maybe
Jon: overzealous about the use of some of Rust's features.
Matthias: Yeah, it's certainly a bit of a rite of passage.
Jon: Yes, I think so.
Matthias: Do you think the key to idiomatic Rust is keeping it simple and then maybe making
Matthias: it right where it matters?
Matthias: So finding that balance between on one side simplicity and maybe ease of maintenance
Matthias: versus correctness for things that really are important?
Matthias: Or what's your working definition of idiomatic, Rust?
Jon: I think it's very hard to give a sort of general definition.
Jon: I tend to start from, I don't think you should start with the simplest possible
Jon: thing, but I also don't think you should start with the most complicated thing.
Jon: And this is where, and I don't like using this, but I think experience matters here, right?
Jon: Where like over time, you just get a feel for where the balance should lie.
Jon: You start writing the code and you go, it's okay to clone here,
Jon: and it's not okay to clone here.
Jon: And it's hard for me to distill what the principles are for making those choices.
Jon: I would say in general, it is very useful to have a running system.
Jon: Once you have a running system, you can then, like, refactoring with Rust tends
Jon: to be a lot easier than in other languages because you have the type system
Jon: and the solid compiler and type checker and borrow checker to rely on.
Jon: And so I would tend to err on the side of where the type safe thing is easy,
Jon: then do the type safe thing.
Jon: And where the type safe thing gets in the way of you building the actual application
Jon: to the end, build the application to the end first, and then mark it with a to do to come back to.
Jon: And some of those to do would be very painful to fix later.
Jon: But the reality is, if you don't finish the whole thing, you're never going
Jon: to come back to the to do's because you didn't work in the first.
Jon: Place so so it's useful to like use that that as a forcing function for making
Jon: you make a suboptimal choice is like well i need to at least get to a thing
Jon: that runs otherwise this whole thing is irrelevant.
Matthias: Rust is a huge language, and while I'm pretty sure that you know more than most
Matthias: people about Rust, what is one thing that you would personally want to spend more time on?
Matthias: If you had three months to focus on one subject that was Rust-related, what would it be?
Jon: I think there are two categories for me.
Jon: One of them is around WebAssembly. So I've done very little WebAssembly in Rust,
Jon: and I think it's both a cool technology, And it's something that I think there's
Jon: a bunch of use cases for it that I think we haven't fully explored.
Jon: And I would love to fiddle around with it to see what I can make of it.
Jon: But also, I think it's a very useful skill or set of knowledge to have in your toolbox.
Jon: And same thing for, you know, I'm thinking of writing another sort of version
Jon: or not version, but a second iteration of Rust for Rustaceans.
Jon: And, you know, Rust Rustaceans doesn't have a chapter on WebAssembly,
Jon: in part because I hadn't done very much WebAssembly at the time when I wrote
Jon: the book and I didn't feel like I could be an authority on that subject.
Jon: And I still don't think I could be. And so that is something I would want to do more of.
Jon: I think the other category would be sort of deep embedded development.
Jon: I've done some embedded development in Rust and I've certainly written,
Jon: you know, crates for no-std and everything.
Jon: But to really write something low level on a microcontroller where, like, you need to, like,
Jon: You need to initialize the CPUs in the right way, and you need to handle the
Jon: interrupts, and you need to write some inline assembly.
Jon: Code like that is really fun to write, and I haven't written lots of it in Rust,
Jon: but I would like to, because I want to see, you know, what does it feel like
Jon: when I push the language in that direction a little bit?
Jon: And like, where are the sharp edges, and what can I do to make those sharp edges
Jon: be more ergonomic, right?
Jon: Like, this could be building tooling, building libraries to make that experience better.
Matthias: I wanted to briefly touch on supply chain security because I believe it's kind
Matthias: of important for housing.
Matthias: You write a lot of code, you maintain a lot of code yourself,
Matthias: but you still need to depend on a lot of crates that are out there that we sort of take for granted.
Matthias: And there's been some recent challenges around some packages in Rust.
Matthias: I don't want to mention any names.
Matthias: And Cargo itself also had some sort of exploit because of the tar crate just a couple days ago.
Matthias: I wonder what's Helsing's stance on that?
Matthias: And what's the state of the Rust ecosystem in regard to supply chain security?
Jon: I think Rust is not in a worse place than other ecosystems here.
Jon: I think it's a bit similar to other ecosystems where when you take a lot of
Jon: third-party dependencies, there's some amount of inherent risk there.
Jon: And I don't think Rust's tooling is worse or Rust's risk is higher.
Jon: And I think the question, as with any language like this or any project that
Jon: has to take third-party dependencies, the question becomes, what do you do about those risks?
Jon: And you think, you know, the reality is you have to build in defense in depth
Jon: against these things, right?
Jon: There's not going to be one silver bullet that just solves all your supply chain security problems.
Jon: Instead, you have to have sort of a collection of processes and tools that make
Jon: sort of as many parts of it as secure as you can.
Jon: And then you layer them on top of each other to get coverage across your whole pipeline.
Jon: So this includes everything from, you know, being judicious about your selection
Jon: of dependencies in the first place.
Jon: Like, don't take a dependency on some tiny, barely maintained project if you
Jon: can easily just replicate the functionality yourself.
Jon: Like the reason why it's worthwhile
Jon: to take dependencies is if the maintenance cost
Jon: of the code in that dependency is large but if
Jon: the maintenance cost is actually pretty small it might not be worth taking the
Jon: dependency and introducing that risk the same thing with if you have the choice
Jon: between multiple dependencies then look at the different dependencies not just
Jon: in terms of the the quality of them in terms of like the you know the api the
Jon: documentation the current code maturity,
Jon: but also look at the maintenance of that package.
Jon: Who maintains it? How many people? Do they have CI?
Jon: What kind of testing strategy do they have? Do they have a security disclosure policy?
Jon: There's a bunch of things you can look for here that indicate something about the.
Jon: The sustainability of that package and of taking a dependency on it.
Jon: And then there's also the sort of ongoing monitoring part, right?
Jon: So obviously you want to monitor all of the security vulnerability databases
Jon: to make sure that if you run into a problem, or rather if a problem is discovered
Jon: with some version of some dependency,
Jon: you A, are notified, and B, that you internally have the infrastructure to find
Jon: all the places where the impacted dependency are used.
Jon: And so this includes being able to track provenance information for all the builds that you do,
Jon: provenance for all your deployments, and this is where you get into things like
Jon: generating SBOMs, like software bill of materials that list all of the dependencies
Jon: that went into a given artifact,
Jon: tracking which software releases are released to what customers,
Jon: at what time, in what products, in what physical devices, and keeping track
Jon: of that whole graph structure.
Jon: And being able to do analysis over that graph over time as you learn about new vulnerabilities.
Jon: And then, of course, there's also work here on security scanning.
Jon: So this would mean both doing scanning of our code for insecure patterns and
Jon: the like, but also running proactive scanning on dependencies that we take.
Jon: The first time they're brought into the company, anytime there's a new version
Jon: and so on, to actually scan them for.
Jon: Is this a dependency that we want to take? And some of that could be human review.
Jon: Some of that can be AI-assisted review. And there's a combination of these that also could work.
Jon: We wrote a blog post recently about using AI for assisted vetting of software packages.
Jon: I'll send you the link and you can put it in the show notes.
Jon: And so that contains some more thoughts about how you can not necessarily replace
Jon: the human review here, but at least make it more efficient for humans to review
Jon: those dependencies that you take.
Jon: And so there's like a whole host of techniques where you kind of need to do
Jon: all of them because they end up giving you, so each one gives you sort of partial
Jon: coverage of the stack that you have.
Jon: And only when you combine all of them do you get the defenses that you need.
Jon: But even then, you know, taking dependency ultimately is a risk.
Jon: And so you have to take the calculator risk of is the upside of taking this
Jon: dependency worth the potential risk that you're introducing.
Jon: But I do think that there's a genuine security case for taking dependencies
Jon: because the alternative, if you build everything in-house, is that you will not have the people,
Jon: especially sort of subject matter experts, to maintain those internal implementations over time.
Jon: So if we internally implement it, I don't know.
Jon: I mean, crypto is the obvious example of like the old adage of you should not roll your own crypto.
Jon: If we personally like implemented all of our own crypto libraries,
Jon: I'd be deeply uncomfortable with that because we don't have enough, you know,
Jon: cryptography specialists and analysts and engineers to A, build it in the first
Jon: place and then B, maintain it over time.
Jon: So I would much rather that be a publicly vetted, widely used.
Jon: Continuously handled by a large number of security experts, and then we take
Jon: a dependency on it, is a much better and more secure decision for your dependency
Jon: chain than in-housing everything.
Jon: And the question really becomes that risk-reward trade-off of at what point
Jon: does it become better to just in-house that dependency so that you don't take
Jon: an external dependency on it because the upkeep is not that bad or the upkeep
Jon: is not likely to have security implications compared to the third-party dependency.
Matthias: There's a common trope that people use, which is Rust's package ecosystem is similar to NPMs.
Matthias: We have a lot of smaller packages, and that exposes us to bigger risk.
Matthias: What's your take on that?
Jon: Yes and no. It is true that Rust tends to have more dependencies than Java or C++, for example.
Jon: It tends to have more but smaller dependencies.
Jon: I think the jury's still out on whether that's a good thing or a bad thing,
Jon: because the downside with taking a large dependency is that A,
Jon: the large dependency means the maintainer of that project is maintaining way more code.
Jon: And chances are they're not an expert in the entirety of that code.
Jon: And so the likelihood that any part of it is under maintained or under vetted
Jon: or underdeveloped is much higher.
Jon: And then the other is the breaking changes sort of update cadence tends to be
Jon: worse because if you have one giant dependency that you have to,
Jon: like now they make a breaking change, that might be a lot harder for you to
Jon: adopt because you have to adopt all or nothing of the entire dependency.
Jon: Whereas if you have many smaller dependencies, fewer of them make breaking changes
Jon: at any given point in time.
Jon: So more of them will be fully up to date because they don't need to align on
Jon: like a single breaking change schedule.
Jon: But ultimately, you know, there are some things where taking a large dependency
Jon: is probably worthwhile. And we do see this in the Rust ecosystem too.
Jon: If you look at things like Bevy, for example, right? Bevy is effectively one
Jon: big dependency. Tauri is another one.
Jon: And so Rust doesn't preclude you from doing this.
Jon: It's more that I think it often makes sense to have smaller dependencies of
Jon: this kind. And I don't think that's inherently a, it's not obvious to me that
Jon: it's a guaranteed security risk compared to the alternative.
Jon: You also have the downside with large dependencies that you could end up with
Jon: because it's so large, you need to have many maintainers.
Jon: And so you have a large number of maintainers that,
Jon: Would it be better if that same number of maintainers each maintained a smaller
Jon: library that was a subset of the overall thing?
Jon: I think that might be better. I'm not sure.
Jon: But yeah, it's not clear to me that Rust is in more of a danger because it has
Jon: this coarse or this finer granularity of packages.
Matthias: I really like your take on Rust's great ecosystem and also contrasting it with
Matthias: whatever Node and NPM provide.
Matthias: What about unsafe code, though? Because this is very unique to Rust and to how
Matthias: we think about safety and code.
Matthias: If you look at this from a perspective of supply chain security,
Matthias: aren't we exposing ourselves to a lot of risk by taking on a lot of unsafe code?
Matthias: And also how would you vet for that.
Jon: Well it's complicated because unsafe code is not inherently less safe even though
Jon: the name kind of implies that right because if you look at java if you look
Jon: at go if you look at node.js and and certainly if you look at c and c plus plus.
Jon: Those languages have no guardrail for what is safe and what is unsafe.
Jon: You know, if you look at Java, you have unsafe operations in Java as well,
Jon: where you do direct pointer manipulation and you can do really bad things there
Jon: and you can violate memory safety and you can do all these things.
Jon: In C++, like all the guardrails are just off. Even if you turn on a lot of the
Jon: compiler validation, like you can just do these things.
Jon: It's just that in Rust, it's more obvious when you do these things than it is in those languages.
Jon: And so I think in Rust, the reality
Jon: is that safe and unsafe is more of a communication mechanism to say,
Jon: this part of the code, you should look at more carefully because it needs to
Jon: sort of uphold things that are not checked by the compiler.
Jon: And so it's places where you get fewer of the benefits of safety from Rust,
Jon: but they're not places that are inherently less safe than general third-party software.
Jon: I do think actually that, you know, you should think a little bit more about
Jon: taking a dependency that includes unsafe than one that does not.
Jon: But I don't think it should be a thing that, you know, excludes you from taking
Jon: that dependency or see it as significantly more risky.
Jon: Where I worry more is when you have crates that have no business having unsafe
Jon: code, but they do anyway.
Jon: Usually this is for like performance optimizations reasons or just they want
Jon: to work around the borrow checker.
Jon: That kind of use I'm more skeptical of, but it's not really a binary of is it unsafe or is it not?
Matthias: Do you think unsafe, the term, was a misnomer?
Jon: Yeah, I think in a way it is, because there are two uses of unsafe in Rust.
Jon: One of them is on function definitions, and the other is on blocks inside of code.
Jon: On blocks inside of code, it really should be described as safe.
Jon: The goal of that annotation is to claim that the code between this curly bracket
Jon: and this curly bracket is not subject to the standard compiler checks or it's
Jon: allowed to break some of the rules or it's not checked that it doesn't break the rules.
Jon: But trust me, I have checked that it is safe.
Jon: That's what you're asserting by putting unsafe around a block.
Jon: And so it's not to say this code is unsafe. It's actually to say this code is
Jon: safe. It's just that it's checked by me, not by the compiler.
Jon: And when it's put on a function definition, unsafe is more of an appropriate
Jon: term because it's saying,
Jon: this function is unsafe to call unless the following is true yeah but so i i
Jon: do i do wish that there was a different name for for the the unsafe in a block.
Matthias: I like that you said assert in that context because yeah it feels like an assertion
Matthias: maybe it could be called assert safe.
Jon: Yeah that is the the challenge right is that asserts we think of as something
Jon: that can fail and the assertion here can't fail like the the program can't fail
Jon: to run as a result of that assert it doesn't actually check anything it's like
Jon: a signature that i just trust me i've checked.
Matthias: What's next for Helsing and for you.
Jon: For Helsing, I think it's continued development of the main product lines that
Jon: we currently have, all of which are fairly ambitious and have some really cool tech in them.
Jon: And so this includes the CA-1 is the project I work on, which is basically an
Jon: effort to try to build a self-flying jet fighter in two years.
Jon: So it's a very ambitious timeline. We're building the whole thing,
Jon: hardware and software, from scratch.
Jon: It's no joke, but it is really interesting work.
Jon: And then we have the SG-1, which is an underwater, also autonomous vehicle that
Jon: aims to do business to look for things like submarine traffic in large underwater areas.
Jon: Think like the Baltic Sea, for example, where you need these things that can
Jon: stay underwater for very long periods of time, have very minimal compute,
Jon: very limited ability to, you know, very constrained budgeting for power.
Jon: They can't have any moving parts in them because then they're too easy to detect
Jon: and then you still need to write software on there that does sonar analysis
Jon: for example so they need to run ML models how do you do that when you have extremely
Jon: constrained battery power.
Jon: And so that's some interesting stuff and then obviously we do
Jon: a lot of work with the HX-2 for use in Ukraine for instance where that is a lot
Jon: of work on being close to the sharp end of defense but also systems where safety
Jon: criticality is enormously important.
Jon: And it might sound self-contradictory to talk about safety critical in a system
Jon: that explodes, but the reality is that it is extremely important that it explodes
Jon: in the right place for the right reasons at the right time.
Jon: And that's where the safety criticality comes in. And the cost of getting that
Jon: wrong can be catastrophic. as a working on these systems, I think is,
Jon: you know, it's, I don't like to describe it as fun because it feels incorrect, but it is challenging.
Jon: It's interesting. It is...
Jon: I think it's meaningful. I think it's important. And I think,
Jon: you know, in terms of the future of Helsing, a lot of that becomes continuing
Jon: to work on these products and other ones to see how far we can go with,
Jon: you know, using software to build really good deterrence capabilities for Europe.
Matthias: And for you, which streams can we expect in the near future?
Jon: I actually tend to not plan my streams very far ahead.
Jon: Instead, what I do is I look for things that I want to exist,
Jon: either at work or in my personal life, and then I build those and then I turn
Jon: on the camera while building them.
Jon: Because if I try to do streams that are where the content is...
Jon: Hand-picked for streaming. It sometimes works if I pick a topic that I know
Jon: is particularly deep and gnarly in there, but it's much more compelling when
Jon: I can say what my use case is and what I'm building towards.
Jon: It means that I can build a solution that actually solves a problem and that
Jon: therefore comes with additional constraints on the implementation,
Jon: a less generic goal, because I think that's where a lot of really good software
Jon: engineering practices come in are when you're not just writing code,
Jon: but you're writing code for a purpose.
Jon: And so that's why my streams tend to be when I have a thing to build,
Jon: then I do a stream on that topic, which was the case recently,
Jon: for example, for the Avro IDL converter was I needed one of those, so I built it.
Jon: And I think increasingly we'll see other examples of this of I need a thing
Jon: either for work or for my personal life, and I will build them.
Jon: That's how I've done my streams until now and how they will continue.
Matthias: And so far, your intuition hasn't failed you around that.
Jon: That's true. I have some ideas for other forms of streams or other forms of
Jon: educational material that I have sort of half-baked in the back of my head.
Jon: I think some of them could be really good, that the challenge is always,
Jon: when will I find the time?
Jon: And so this could be things like, I've had an idea for a sort of Rust intermediate
Jon: course that would actually be a structured course rather than just a sort of ad hoc streams.
Jon: I've had ideas for more of a,
Jon: casual chat beginner's introduction to Rust that comes in shorter snippets rather
Jon: than like 10-hour streams.
Jon: I think that could be really cool, but finding the time is always the hard part.
Matthias: There's a ton more that we could discuss, but we have to get to the end.
Matthias: And traditionally, our final question is, what's your message to the Rust community?
Jon: I think my message to the Rust community is twofold. I think the first half
Jon: is Rust is now a language that is actively in use in important systems across the world.
Jon: And that's a good thing, right? This is what a programming language aims for,
Jon: is to be successful to the point where it's adopted for real use cases that
Jon: makes a real difference.
Jon: But the result of this is that companies care about what happens to the language
Jon: and the direction the language goes in. And I think this is sort of,
Jon: a little complicated for the Rust community, which traditionally has had very
Jon: strong opinions on not just how the language is used, but also sort of there's
Jon: been an association, I think,
Jon: with like a value judgment on what the language is used for.
Jon: And increasingly, I think the Rust community will have to come to terms with
Jon: the fact that it's being used in production in ways that we don't control.
Jon: And I think we need to decide how to interface with that.
Jon: A good example of this is like sponsorship of Rust conferences, right?
Jon: Or in general, like sponsorship of the Rust Foundation, for example,
Jon: like putting money into the Rust ecosystem, where this to me is like a good
Jon: thing for the Rust community. I want that to happen.
Jon: I want the community to get the language of the funding that it needs to continue
Jon: to grow and continue to develop.
Jon: But that comes at the cost that the language needs to prove that that influx of money is worthwhile.
Jon: Because in general, if you want the funds to continue to come,
Jon: there needs to be something that you get back for the money you put in.
Jon: And that's not to claim that I have the answer for this, but I think there's
Jon: been a sort of allergic reaction in the Rust community to industry involvement in the language.
Jon: And I understand why in many ways, but I do think that it is an allergic reaction.
Jon: We have to figure out how to mitigate because otherwise the funding for the
Jon: language dries up and we end up with a language that doesn't have the funding to keep growing.
Jon: And then I think that the second half of my takeaway from the Rust community
Jon: is we've built something that's really good.
Jon: Like the Rust language is really good. The tooling is really good.
Jon: The community is really good.
Jon: And I think that the ecosystem as well is really good.
Jon: But I also think I've observed a sort of stagnation in the,
Jon: I don't know how to describe it, like the creativity of the use of the language
Jon: compared to some of the early days.
Jon: And that might be because we've solved many of the problems, right?
Jon: But it does feel like there are still cool things you could do with the language
Jon: that can change how the ecosystem works.
Jon: serde was a good example of this from back in the day where it was a different way of doing things.
Jon: And then it made us have things like derived procedural macros and the ability
Jon: to do serialization in a really efficient and cross-language ecosystem way.
Jon: And I want to see more of those kinds of ambitious, let's build something that
Jon: is different, is better, is cool, new, innovative in the language, in the ecosystem.
Jon: I'm seeing less of a hunger for that than I think I saw in the early days.
Jon: And that makes me a little sad.
Matthias: Well here's hoping that people like you who are ambassadors of the language
Matthias: will bring some of that passion back and i certainly hope to see you on the
Matthias: stream i will follow along Jon thanks so much for taking the time and for being
Matthias: part of this community no.
Jon: Thanks for having me it was a fun conversation.
Matthias: Rust in production is a podcast by corrode it is hosted by me Matthias Endler
Matthias: and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.