Matthias: Nicolas, can you say a few words about yourself and the company Apollo that you work for?
Nicolas: Yeah, so first of all, Matthias, thank you very much for having me today.
Nicolas: So I am a staff software engineer at Apollo GraphQL.
Nicolas: Just before we get started, just a quick introduction on Apollo GraphQL.
Nicolas: We're a company that specializes in GraphQL API technologies,
Nicolas: as the name indicates. So we have a bunch of quite well-known open-source libraries
Nicolas: for clients to reach out to GraphQL APIs, but also server-side technologies.
Nicolas: And what's probably more interesting for this podcast about Rust is we have
Nicolas: two main things that are mainly written in Rust.
Nicolas: One is the Apollo router, depending on what you prefer.
Nicolas: So the idea with the router is that when you start to have multiple microservices
Nicolas: that all are using GraphQL for the API technology,
Nicolas: you want to combine that into a single API that will group all of them in one go.
Nicolas: And so what the router will do is it will take an incoming request and be able
Nicolas: to redirect the traffic to the right microservices.
Nicolas: And in my case, I'm part of the Apollo GraphOS cloud team, and we are responsible
Nicolas: for manager offering of the router.
Nicolas: So we host this on behalf of our customers, and then they can then send the
Nicolas: traffic to their own microservices from there.
Matthias: I guess a lot of listeners already have heard of GraphQL, but maybe they don't know the specifics.
Matthias: Maybe you can give us a quick primer on what it is and how it's different from, say, REST.
Nicolas: Sure thing. So with GraphQL APIs, you define a schema that's
Nicolas: basically your entities, your objects, and the fields that you have on.
Nicolas: So you might have, for example, products with names, description, pricing, and so on.
Nicolas: And one thing that's very different compared to REST APIs is that the client's
Nicolas: able to say exactly which fields,
Nicolas: So with REST APIs, you often have problem with either under-fetching or over-fetching.
Nicolas: So over-fetching, you're fetching too much data compared to what you need.
Nicolas: Under-fetching, you don't fetch enough, and then you need to make subsequent API calls.
Nicolas: With GraphQL, you're actually able to make a single call for all the data you
Nicolas: need by specifying exactly which information are useful for you.
Nicolas: In the case of federation, which is important for the Apollo router,
Nicolas: the idea there is that we can combine multiple microservices
Nicolas: apis into a single one and then from your client you
Nicolas: can make one call that will actually go to all these different
Nicolas: microservices and bring you a single response back
Matthias: when you explained that i thought wow this is a massive problem it
Matthias: sounds like a hard GraphQL problem maybe an
Matthias: np complete problem even and you
Matthias: make something very complicated very simple or very
Matthias: easy for front ends do you agree
Matthias: and also if you you agree what is
Matthias: your perspective on this and how complex is
Matthias: it in the back end
Nicolas: so yeah yes i really agree with this especially on the on
Nicolas: the front end stuff it's really nice to just be able to make one query you don't
Nicolas: have to think about the back-end infrastructure you don't have to think about
Nicolas: how many microservices and all the different turnpoints just make one query
Nicolas: fetch data you need and indeed the routing part is it's a solved problem right
Nicolas: i mean otherwise the apollo which would not exist.
Nicolas: But this is still a relatively hard problem to decide was the correct order operation.
Nicolas: So we call that making a query plan. So we plan what the queries we need to
Nicolas: make to each microservices on the backend.
Nicolas: We call them subgraphs. And then that federated API is called a supergraph.
Nicolas: And then, you know, making all the queries in an optimized way to ensure that
Nicolas: you get a response as fast as possible and exactly with the data that you need and nothing more.
Matthias: When you started with GraphQL at Apollo, was Rust already that mature that you
Matthias: could start with Rust or did you start with a different language and evolved from there?
Nicolas: So to give a bit of history on Apollo itself, I mean, Apollo is over 10 years
Nicolas: old, right? So our adventure with Rust is relatively recent.
Nicolas: It's a question of a few years now.
Nicolas: But also, a lot of what Apollo brings to the table and a lot of what people
Nicolas: know about Apollo today is technologies that are specific to certain environments.
Nicolas: So on the client side, we have JavaScript libraries, libraries for iOS,
Nicolas: for Android. And then on the server side, we have the Apollo server, which is in JavaScript.
Nicolas: The first version of a federated API we made, it's called the Apollo Gateway,
Nicolas: was actually written in JavaScript.
Nicolas: So the Apollo router is mostly a rewrite of this, but into Rust,
Nicolas: which was really made for performance reasons at first.
Nicolas: The main reason why we started on this journey was to deliver the kind of performance
Nicolas: that you would expect from an API at scale that's fronting all your traffic.
Matthias: It feels like you're hinting towards...
Matthias: Other advantages of Rust other than performance in the long run?
Nicolas: Yes, there is. So in the two use cases we have, where we are heavily using Rust
Nicolas: today at Apollo, there's one where performance is really critical.
Nicolas: There's a second one where safety is very critical, right?
Nicolas: So on the cloud side of things, every time we manage infrastructure,
Nicolas: we update resources on behalf of customers, we basically have distributed transactions.
Nicolas: Right safety, order of operation, execution, making sure that everything is
Nicolas: happening in the right order with the right information, that nothing is lost,
Nicolas: is very important in this context.
Nicolas: So I would say these are the kind of main two drivers for that.
Nicolas: A third thing that's important too, GraphQL is a strongly typed form of API
Nicolas: and Rust is a strongly typed language.
Nicolas: So these two are actually quite interesting together as well to make sure that
Nicolas: we align from that perspective.
Matthias: It feels like Rust in general aligns with this problem really well because if
Matthias: I can just quickly touch on what you said, maybe stability is also a core part
Matthias: of that because the least thing you want to go down is your API.
Matthias: It always needs to be up. And I don't know if you have any SLAs or something
Matthias: around this, but probably customers expect uptime. and for that you need stability
Matthias: in your code base and this is where Rust can help, right?
Nicolas: Yeah, this is certainly a very important thing for us if I'm thinking about
Nicolas: our managed service, right?
Nicolas: So our DrivePress cloud offerings, our managed cloud router,
Nicolas: this is something we hear a lot from customers around SLAs and reliability.
Nicolas: And the most important thing in terms of maintaining your SLA is not necessarily
Nicolas: the underlying infrastructure.
Nicolas: I mean, you need to build something on top of solid foundation,
Nicolas: on top of solid cloud infrastructure that's going to work no matter what. But...
Nicolas: Most of outages come from application-level changes, right? Not necessarily
Nicolas: the underlying infrastructure.
Nicolas: So changes are risky, and asking people to trust us with all that traffic being the API,
Nicolas: the front-end API that's going to ingest all the traffic and then route it to
Nicolas: their microservices is a huge responsibility.
Nicolas: Safety and reliability is extremely important for us for that reason.
Matthias: On top of that if the application code
Matthias: causes most outages i think this is a
Matthias: good space to be in because your customers
Matthias: make changes it's very normal that from time to time they deploy something which
Matthias: maybe isn't perfect but the stability of the platform itself is always guaranteed
Matthias: so at the risk of getting ahead of ourselves i was really curious about on call
Matthias: as well because did Did you see a change when you moved to Rust,
Matthias: when you transitioned from JavaScript, say, to Rust for your router?
Matthias: Did you see an improvement with regards to on-call and outages?
Nicolas: So there's a few different things here. In terms of the timelines,
Nicolas: it's kind of hard because we really started our cloud journey when we already
Nicolas: had the Apollo router, so we already went into our Rust journey.
Nicolas: But now, if I kind of take a step back and I'm thinking about what Rust made
Nicolas: us think about as we were starting.
Nicolas: So we started the cloud journey about a year ago, right? A bit over a year ago now.
Nicolas: When we started on this cloud journey and all of the control planes,
Nicolas: so the control plane is the part that manages all the customer resources,
Nicolas: so all the managed routers and so on, this is also all written in Rust.
Nicolas: And Rust makes you think about safety a lot, even though the language itself is designed for safety.
Nicolas: So there are lots of ways to not shoot yourself in the foot just because you're already using Rust.
Nicolas: So all the safety guarantees around
Nicolas: mutability around access to data and so on
Nicolas: that are built into the language are great but it also shaped the way you think
Nicolas: and then you start to think way more in term of safety features that are important
Nicolas: but that aren't necessarily built into the language but language makes it easy
Nicolas: to then implement them so i
Nicolas: mentioned for example right ride safety being extremely important for us.
Nicolas: So to create a cloud router for a customer, we need to provision a bunch of different resources.
Nicolas: And we need to avoid creating the same thing twice. We need to keep track of where everything is.
Nicolas: And so when we are making changes on our own code base, we are making all these
Nicolas: changes there, we need to make sure that we are not accidentally breaking things.
Nicolas: So we put a lot of fun phases on the ride safety safety and making sure that
Nicolas: the state of our databases, the state of what we know exists for in terms of
Nicolas: customer resources, was always correct.
Nicolas: And so we've encapsulated a lot of write safety into the struct and variants that we use in Rust.
Nicolas: So we actually get, by using the strong typing of Rust and the way we structure
Nicolas: that, we actually get a lot of safety guarantee from that perspective,
Nicolas: which is not something that that Rust enforces, but Rust made us think about.
Matthias: Does that mean you would get a lot of errors at compile time that you would
Matthias: otherwise get at runtime?
Nicolas: Yeah, exactly. We're trying to push a lot of these, kind of check at compile
Nicolas: time as much as possible.
Nicolas: But it's also just a thing that...
Nicolas: Okay, let's talk a bit about domain model and DDD, right?
Nicolas: And we had this question at some point, hey, should we have semantic types or
Nicolas: should we have safe types? because these are not always the same.
Nicolas: So I'm going to take back my example of a product from earlier.
Nicolas: The idea of you have a product, it has a name, description, price, and so on.
Nicolas: Well, a product might have a state. Is it for sale or not?
Nicolas: And if your product is not for sale, it should not have a price.
Nicolas: So semantic representation, you might just represent them, all these fields as next to each other.
Nicolas: You would have a status type for your product. you would have a price field,
Nicolas: you know, you would have a status field, a price field, and so on.
Nicolas: But what you can do is actually encapsulate the price of your product in the
Nicolas: state, which is not semantic, you know, what's the relationship between a status and a price.
Nicolas: But now it's safe, because you only have a price if this product is for sale.
Nicolas: That's just one example right here, which is, but we have these kind of things.
Nicolas: So if a Cloud Router workload is deployed in a certain state,
Nicolas: then it must have this information, for example.
Matthias: How would that be encoded in the type system? I imagine maybe you have something like a,
Matthias: normal product and then you have a product for sale or
Matthias: something you have two different types for for guaranteeing that
Matthias: you have a price for one and not for the other or is it completely different
Nicolas: oh no so so the way you would do
Nicolas: that is you would have a product struct right but
Nicolas: then in this product status type you would
Nicolas: have two that that would be an enum with a
Nicolas: not for sale and then for sale but
Nicolas: for sale would contain a field with the
Nicolas: price for example
Matthias: ah okay so it's a very rich type with an enum yeah to encapsulate
Matthias: that and on the front and when you retrieve that you
Matthias: probably don't even have to feel available so you can't even query it or would you get a null
Nicolas: yeah in that case because then it's it's matter
Nicolas: of how different languages think about the type system them right what we would
Nicolas: typically do for the simplicity of the front end would just be yeah the price
Nicolas: would be null if the product is not for sale so you would then remap it into GraphQL native objects
Matthias: which brings me to my next question about the interaction
Matthias: between the front end and the back end so I guess on the front end you might
Matthias: have something like TypeScript and a lot of people use TypeScript in combination
Matthias: with Rust and it seems like a good collaboration if I may I say so. Do you agree?
Nicolas: Oh, yeah, no. These are certainly languages that have a lot of things in common.
Nicolas: So even if you look at strongly typed languages, not all of them have exactly
Nicolas: the same features and the same things.
Nicolas: There are certainly things that don't necessarily map exactly one to one,
Nicolas: and you have to do a bit of conversion between the two if you think about the type system.
Nicolas: And I'm just addressing to people that might be learning Rust, for example,
Nicolas: and you might come from typescript you already know a lot if
Nicolas: you understand the typescript type system you already understand a lot
Nicolas: of how rust will behave right right yeah i'm
Nicolas: not a front-end person and my experience with typescript is really great i cannot
Nicolas: complain i think it makes a lot of those little rough edges in javascript much
Nicolas: much easier to handle yes yes totally it's i mean,
Nicolas: I'll be honest I am very spoiled with Rust I used to be mostly a Python dev
Nicolas: before I started really doing Rust seriously I've been doing Rust for a bit for three years now.
Nicolas: So this was still at the beginning of Python having actual type hints and these kind of things.
Nicolas: Nowadays, I've looked, and there's so much clever stuff you can do in Python,
Nicolas: but still rely on tools to tell you, like, hey, this is right, this is wrong.
Nicolas: But Rust's type system has really spoiled me, and
Nicolas: it's really hard for me to think about going to a language
Nicolas: that doesn't have any kind kind of strong type anymore because
Nicolas: it's just so easy in terms
Nicolas: of cognitive load i don't have to think about and then as
Nicolas: i mentioned earlier you can encapsulate so much clever stuff
Nicolas: in types just for your own safety or to
Nicolas: match whatever need you have for example yeah
Matthias: the other thing i briefly wanted to touch on was the control
Matthias: plane is it so that you deploy
Matthias: one completely completely independent instance for
Matthias: each customer or do you have shared
Matthias: infrastructure as well for all the customers maybe to make it a bit more price
Matthias: efficient or where do you find the balance here
Nicolas: yeah so it depends so we have two tiers at the moment we have the first one we started one year ago which
Nicolas: is the serverless tier which use way more shared infrastructure infrastructure.
Nicolas: We still have strong isolation of workloads and so on.
Nicolas: But then we have our dedicated tier, which we announced in private preview a few weeks ago.
Nicolas: That tier then has way more dedicated resources.
Nicolas: I mean, that's the point of that
Nicolas: tier is to actually give you dedicated resources only assigned to you.
Nicolas: There's still going to be some level, if you think about about cloud providers
Nicolas: and so on, there's still going to be some level of shared infrastructure.
Nicolas: It doesn't make sense, for example, to have one AWS account per workload or one VPC per workload.
Nicolas: There are things you can do to save on the number of resources you have to manage,
Nicolas: because every time you add more resources, that's a lot of things to manage
Nicolas: in the end and make sure that everything is working fine.
Nicolas: But then when we talk about the underlying resources that are actually running
Nicolas: that workload, then in that case, they would be dedicated to a single workload.
Matthias: And what about this serverless platform? How does it work?
Matthias: Is it actually using Lambdas or is that an internal name for a platform that
Matthias: you don't have to maintain yourself?
Nicolas: So serverless there is more in terms of how is the customer experience,
Nicolas: right? So it's not using Lambda functions under the hood.
Nicolas: But it's based on the pricing model where we are closer to a price-per-request
Nicolas: model like you would have with serverless services like Lambda.
Nicolas: While the dedicated tier, you pay for provision resources that are always available
Nicolas: for you when you have predictable performance, basically.
Matthias: I see. So to recap, the router is in Rust.
Matthias: Probably parts of the GraphQL parser is in Rust, I assume.
Matthias: GraphOS is proprietary, but it is also written in Rust.
Nicolas: So this one is, because GraphOS is a bit more than just, there is GraphOS Cloud,
Nicolas: which is our managed offering where we run the router for you.
Nicolas: But then there is Apollo GraphOS as a whole, which is a management platform for your own resources.
Nicolas: For example, let's say I'm a microservice developer. I make an update to my GraphQL API.
Nicolas: API, you then publish that change with Apollo GraphOS, and then we will create
Nicolas: a new schema for the federated API, so for the composition of all these microservice APIs into one.
Nicolas: We need to update the schema and then push it to your router,
Nicolas: whether it's self-hosted or managed by us.
Nicolas: That part is mostly in Kotlin, actually.
Matthias: And how do you decide which language to use for a new code? Where do you strike the balance here?
Nicolas: So, I mean, Kotlin has been, once again, a lot of it comes to history and what
Nicolas: we've been using before.
Nicolas: So a lot of the management plane platform was written in Kotlin and is written in Kotlin.
Nicolas: And if you make change to this, it doesn't make sense to rewrite it in Rust,
Nicolas: because this would be a lot of efforts to rewrite everything in Rust.
Nicolas: That's not necessarily a compelling case today.
Nicolas: And now when we talk about new pieces of software, so new functionalities and so on.
Nicolas: It kind of depends a bit. I've seen both. I've seen new services,
Nicolas: new internal services that are then written in Rust because we are at the point
Nicolas: now as a company where we have plenty of Rust developers internally.
Nicolas: We've built experience.
Nicolas: We understand the road ahead in
Nicolas: terms of what does it take to run a Rust service and we can just do that.
Nicolas: So some new services are going to be written in Rust, but
Nicolas: then we might also write some new codes in Kotlin in
Nicolas: to to stay consistent because we might need some
Nicolas: libraries we've written and so on right so it's it's very
Nicolas: much a case-by-case basis even though i'm seeing more
Matthias: and more rust adoption over time another tool that i saw that is written in
Matthias: rust apparently is rover which is a cli tool yes can you quickly say what it
Matthias: is and what you do with it
Nicolas: yeah sure so i gave you this example where I'm a graph developer, right?
Nicolas: I have my microservice, I'm changing the API, and I want to publish that change.
Nicolas: So Rover is a tool, it's a CLI tool to interact with Apollo Graph REST.
Nicolas: I mean, it's more than that because
Nicolas: it also contains features to help with local developments and so on.
Nicolas: So it's meant to be a one-stop tool for all your GraphQL, Apollo GraphQL-related needs, basically.
Nicolas: So you can use that to quick start a new service, a new microservice.
Nicolas: You can use that to publish the changes you've just made.
Nicolas: So you can integrate that into your CI, CD pipeline whenever you've updated
Nicolas: your microservice, then you tell us like, hey, I've made an API change.
Nicolas: Another feature we have on our management planes, the ability to do checks to
Nicolas: make sure that, okay, you're making a change.
Nicolas: Will it actually work? Because if we have two microservices with the same field
Nicolas: name, like which one has precedence, which one is the right one.
Nicolas: Or there might be conventions, so you might have lane checks and so on.
Nicolas: So Rover can help you with all this kind of running these checks in CI, for example.
Matthias: And Rust is pretty great for CLI tools, so probably it was a natural decision to write that in Rust.
Nicolas: Oh, yes, yes. It's a very different experience because the different components...
Nicolas: I mean, Rust is a very wide ecosystem.
Nicolas: Rust is something that can be used in so many different domains.
Nicolas: The way you write Rust for CLI too is very different from how you write Rust for...
Nicolas: You know the the control plane services or because
Nicolas: then you use a lot of async stuff and so on that
Nicolas: you might not need in CLI tool right but yes rust is really great from that
Nicolas: and you have all the compatibility that we need to add features quite easily which is nice
Matthias: do you have a situation in the command line tool where you try
Matthias: to access two resources asynchronously somehow i can imagine for For example,
Matthias: you want to look up a schema and maybe you need to fetch information from one
Matthias: resource and one from the other and you could do that concurrently somehow?
Nicolas: I'm not the most familiar with the Rover code base. So my knowledge might be
Nicolas: rusty because last time I checked that was not necessarily...
Nicolas: I mean, if it was the case, it would be done sequentially.
Nicolas: But yes, totally. This is something that you could... I mean,
Nicolas: okay, let's take a step back actually because then we get into the ball of GraphQL
Nicolas: because, well, all APIs is GraphQL, shockingly enough.
Nicolas: So you can also make optimized queries.
Nicolas: So instead of having to do two queries, you can make one query that will fetch
Nicolas: all the information in a lot of cases, right?
Nicolas: So then the question of doing it sequentially is not necessarily the most relevant thing.
Matthias: Yeah. And async Rust is not always something that you want to have in a CLI tool
Matthias: necessarily. It may be not even necessary most of the time.
Matthias: And I'm wondering, what are your experiences with AsyncRust,
Matthias: especially on the control plane, especially on the backend side with routing?
Matthias: Does it really help you solve a problem? Do you see that without async Rust,
Matthias: it would be way harder to build some of these things?
Matthias: Or would you say it's kind of syntactic sugar on top?
Nicolas: So on that end, the asynchronous part simplifies a lot of things.
Nicolas: I will actually go back a step and talk about what I've seen in terms of Rust
Nicolas: developer experience, and that will be quite important for this, I promise.
Nicolas: So what I've found, if you look at a lot of Rust developers' journey,
Nicolas: there's kind of three main phases.
Nicolas: There's the first phase where you're starting out, you're still a bit confused
Nicolas: by the borrow checker and other things from the compiler, and you clone a lot of things and so on.
Nicolas: Then there's a point when you start to be comfortable with a lot of these systems
Nicolas: and so on, and you might start using AsyncRust and you're able to be quite productive.
Nicolas: And then there's a third phase when you start to have mastery over some of the
Nicolas: more intricate systems, like in-depth knowledge of how Tokyo works,
Nicolas: for example, and how some of the more complex Async stuff work.
Nicolas: And so if you're in the second category, if you're people that are able to be
Nicolas: proficient in Rust, but you don't necessarily know all the details of the libraries
Nicolas: and so on, Async Rust is already a great tool there.
Nicolas: Because you're just able to make something that works, it's going to be highly
Nicolas: optimized, is going to be able to handle a lot of traffic without you having
Nicolas: to put much thought into making it work nicely.
Nicolas: Even in the first category, even if you clone a lot of data,
Nicolas: it's still working great.
Nicolas: There's so many great libraries nowadays that are just available, have a great API.
Nicolas: So you can get started in AsyncRust and just already, just by using AsyncRust,
Nicolas: you're going to have systems that are going to be very performant when you think
Nicolas: about microservices that need
Nicolas: to handle potentially thousands of RPS or traffic or something like this.
Nicolas: But then when you get to do the third tier and you start to dive really deep
Nicolas: into asynchronous, then you can start to do some pretty clever optimizations,
Nicolas: some pretty clever thinking in terms of how to optimize your asynchronous communication,
Nicolas: to avoid cross-system race conditions,
Nicolas: you know, distributed race conditions, to make sure that you actually fetch
Nicolas: multiple data systems, multiple pieces of data at the same time, and so on, all in one go.
Nicolas: So there's kind of this journey of growth into the kind of Rust async ecosystem,
Nicolas: where you already get a lot of benefit just being able to make async calls and
Nicolas: having libraries like Axum and so on to just write a server that's already going
Nicolas: to be able to handle a lot of traffic without you having to think too hard about it.
Nicolas: On the third tier, year how would tokyo or async rust in general prevent race
Nicolas: conditions on multiple systems,
Nicolas: okay so we had a case like this
Nicolas: i'm trying to remember the details i've been i was off last week
Nicolas: so so i'm still kind of going back into my work mind but we we had a problem
Nicolas: the order of operation with a distributed system was very important because
Nicolas: if we were to call it first to fetch the state and then run a certain operation,
Nicolas: there might be a timing issue in the very, very, very narrow gap between the two pieces.
Nicolas: We're talking microseconds or even less. I mean, more like milliseconds.
Nicolas: Network is not that cheap.
Nicolas: But we're talking about millisecond gap or sub-millisecond gap here where there
Nicolas: might be a race condition where we query the state and then start doing an action.
Nicolas: What we could do is actually, if we can start the action first, then query the state.
Nicolas: And because we benefit from things being async, we can actually optimize things quite well there.
Nicolas: And then we could interrupt the future if we realize there's actually no work to do, for example.
Matthias: Kind of a good reminder that Rust does not prevent you from running into race conditions.
Matthias: Even though the type system is very sophisticated and a lot of things around
Matthias: async Rust is maybe wrapped in a mutex or an arc,
Matthias: doesn't necessarily mean that you can avoid these cases, especially if the system
Matthias: becomes more and more complex, right?
Nicolas: Yeah, yeah, it is. You have to think still about the kind of primitives and
Nicolas: what you're actually doing.
Nicolas: So Rust will do a lot to already prevent yourself from shooting yourself in the foot.
Nicolas: But the most beneficial part, when you get into the second tier,
Nicolas: third tier of your Rust journey, it's how Rust changes your mind and you start
Nicolas: thinking about, Well, I need to think about these things ahead of time.
Nicolas: I need to really think about those before I implement them.
Nicolas: And sometimes there are also weird side effects that you didn't think about
Nicolas: that you suddenly realize exist.
Nicolas: So one thing I'm a big fan of doing, we were doing some load testing when we,
Nicolas: a few months ago, as we were building our dedicated tier,
Nicolas: because dedicated, the idea is that you provision a certain number of graph
Nicolas: compute units, and these compute units basically guarantee you a certain amount of traffic.
Nicolas: So we had to make sure that, hey, you can actually do what you want to do, right?
Nicolas: That we give you the right amounts. So I did a lot of load tests.
Nicolas: And one thing that's important when you do load tests is to go way past the
Nicolas: point of good put, way past what you actually should expect.
Nicolas: And we found some odd behavior of tokio, for example, when you really go into,
Nicolas: I'm talking about sending 50X the traffic we were reasonably expecting.
Nicolas: But then you start to see weird behavior of Tokyo you're aware oh
Nicolas: some of the internals will actually behave like a queue
Nicolas: that you didn't expect right so there are
Nicolas: still things there that you need to be careful of and that you need
Nicolas: to test yourself and you need to still write tests right nothing is
Nicolas: magical but trust really helped us think a lot about a lot of what could go wrong ahead of time
Matthias: i once heard i'm not sure if it's entirely correct but i
Matthias: heard that at oogle they usually plan for 10x the traffic whenever they
Matthias: build a new system and they want it to reach 100x,
Matthias: which is kind of the stretch goal whenever they build something new.
Matthias: And it feels like you're roughly in the same ballpark here where you mentioned 50x and,
Matthias: It feels like you're planning for growth already, which is great.
Nicolas: Yeah, it's, I mean, I used to be, I worked for five years at AWS as a solution architect there.
Nicolas: And there is, if you don't know about it, it's a really great resource if you're
Nicolas: building kind of distributed systems. It's called the Amazon Builders Library.
Nicolas: Not very specific at all, but there is this article by David Yanacek,
Nicolas: who was at the time on the AWS Lambda team, around testing what you expect in terms of good puts.
Nicolas: And yeah, the general recommendation, as you can see from Google,
Nicolas: from AWS, is go way past what you think is expected amount of traffic to make
Nicolas: sure that your system can actually handle that and will not crash and burn.
Matthias: Did you run into any issues with cancellation of futures?
Matthias: Did you test any edge cases where you test negative paths for example you have a query that,
Matthias: triggers a lot of sub queries and you want to stop that is there something in
Matthias: the system or maybe even inside rust in in tokio maybe that prevents you from running into edge cases,
Matthias: yeah that's part of the thing where you know we get into this kind of third
Matthias: tier stuff where you You have to think about what's happening.
Nicolas: What happens when you cancel and what's the cancel safety of the futures that
Nicolas: might depend on what you're doing and so on.
Nicolas: So it's certainly, in our case, it was fine. In that specific case, it was fine.
Nicolas: But this is certainly something that you need to think about and explore to
Nicolas: make sure that there's no go-touch or test to make sure there's no go-touch
Nicolas: because you update your library that might change a certain behavior internally.
Nicolas: And this is the kind of thing where, yeah, There might be gotchas as we get into that space.
Matthias: We went really down this rabbit hole and went a little technical here.
Matthias: But it's sometimes unavoidable if you talk about these very challenging projects.
Matthias: And I want to take a step back and think about the team and what Rust brings
Matthias: to the team and also the Apollo team in general. roles.
Matthias: So can you give us a quick snapshot of what the team looks like right now?
Matthias: Maybe just roughly how big the engineering team is, how many people are ROS
Matthias: developers, and also if people freely move between languages or if they are
Matthias: more focused on one specific stack?
Nicolas: Yeah, so I don't have the details in my mind on how many developers we have.
Nicolas: And because of what we do as a company, it's quite spread out.
Nicolas: We have a sizable portion of our developers working on open-source technologies,
Nicolas: so all the libraries we're making and so on, and these are going to be very language-specific.
Nicolas: And then we have people working on the management plane, on GraphOS Cloud.
Nicolas: I would say there's quite a strong affinity to languages in general.
Nicolas: I mean, it doesn't mean that you never do any other languages, right?
Nicolas: So in my personal case, for example, I mostly write Rust, but I had to fix bugs
Nicolas: in a Terraform provider, which is in Go. So, you know, you go into Go, right?
Nicolas: So, but in general, there's quite a strong affinity,
Nicolas: to whatever language you're using, except, as I mentioned, there's a strong
Nicolas: part of Kotlin where now some people are starting to do more and more Rust,
Nicolas: which is exciting to see.
Nicolas: Now, if I look a bit at the two teams that are really 100% involved with Rust,
Nicolas: so we have the Apollo Router team, where it's 100% Rust, and then my team,
Nicolas: the GraphOS Cloud team, where we're also doing that.
Nicolas: The background is actually quite different, and that's also because the preoccupations
Nicolas: we have are quite different, right?
Nicolas: So the team that's actually writing the router itself, they have a lot of concern
Nicolas: around performance to make sure that the router itself is going to be as fast as possible.
Nicolas: While for us, because we care a lot more about provisioning and managing resources,
Nicolas: performance is not necessarily the most important piece.
Nicolas: So this kind of very high level, the very complex optimization with async rusts
Nicolas: are not necessarily the first thing we go to.
Nicolas: But then we have way more concern about safety.
Nicolas: In our case, most of the people on my team actually were not using Rust before coming to Apollo.
Matthias: Interesting. So they learned Rust on the job.
Matthias: Did you use any training material or can you recommend any resources for people starting with Rust?
Nicolas: I think it kind of really depends. It's something I always say.
Nicolas: I've helped quite a few people learning Rust And I'm generally happy to kind
Nicolas: of give pointers to people when possible.
Nicolas: But it really depends on how do you learn best.
Nicolas: In my case, I'm an experiential learner. I learn best by actually just doing
Nicolas: and writing code myself and doing experiments.
Nicolas: And the first thing I'd like to say on that, by the way, when you're starting
Nicolas: your Rust journey and you're starting to write code, it's okay.
Nicolas: I mentioned the tier one stuff where you do a lot of clone, where the broad
Nicolas: checker is still confusing, and sometimes you get a lot of compilation error
Nicolas: that you don't necessarily understand.
Nicolas: It's fine. It's totally fine to get there. By the way, even if you clone excessively
Nicolas: resources, you might still be way better in terms of performance as what you
Nicolas: would have if you're using JavaScript or Python.
Nicolas: I mean, coming from Python, Python does a lot of cloning of values under the
Nicolas: hood that you don't realize.
Nicolas: Rust just makes things explicit, right? So some people also learn better via
Nicolas: books, via tutorials and so on.
Nicolas: And there's so much great resources all over the board.
Nicolas: It's really hard for me to kind of point to one. In my case,
Nicolas: I use the official resources that you find on a Rustlang website.
Nicolas: So there is the Rust book.
Nicolas: I think there's the Rust from examples, I think it's called.
Nicolas: I don't really exactly remember now.
Matthias: Rust by example, yeah.
Nicolas: Yeah, yeah, yeah. These ones already helped me so much.
Nicolas: And then I was just writing, starting with CLIs, writing some libraries and so on.
Nicolas: I had a mentor, which when I started
Nicolas: using Rust before at AWS and I had a mentor there was very helpful.
Nicolas: We also have, because now we have a sizable number of Rust engineers,
Nicolas: we have a Rust Slack channel where everybody can ask questions,
Nicolas: but there's also the official Rust Discord.
Nicolas: People are very friendly in the Rust community, so don't hesitate to reach out
Nicolas: and ask questions to people.
Nicolas: They will usually point you in the right direction.
Nicolas: Then as you're getting more experience, I would highly recommend the book Rust for Rustacean.
Nicolas: If you feel like you're getting more comfortable, you rarely get compiler nowadays
Nicolas: or something like this, go with the Rust for Rustacean. It's a great book to
Nicolas: really level up your Rust knowledge.
Nicolas: So that's kind of my, yeah, like six to 12-month recommendation journey if you're
Nicolas: looking at learning Rust or where you might feel yourself to be in terms of Rust experience.
Nicolas: But to be honest, like a lot of people on our team were able to make PRs in
Nicolas: Rust after a few weeks, right?
Nicolas: Back when I was at AWS, as I was a solution architect, my job was to talk with
Nicolas: a lot of companies and I had people reaching out to me saying,
Nicolas: hey, that company wants to start using Rust, but they're a bit afraid.
Nicolas: To be honest, they're a bit afraid of starting, of not finding the talents and so on.
Nicolas: Well, that's fine. You can train people to learn Rust and they can start to
Nicolas: be productive in a few weeks.
Nicolas: It's not that scary of a language.
Matthias: book is a book by Jon
Matthias:
Matthias: The Rust for
Matthias: , I guess,
Matthias: and it's about the internals that you mentioned.
Matthias: I really liked your analogy with the three tiers. I think this is really on point.
Matthias: And it helps you from getting from tier one to tier two or from getting from
Matthias: tier two to tier three. What do you think?
Nicolas: Yeah, I think it's really helped you cement your knowledge in tier two because
Nicolas: it's all about best practice for library design, you know, an API,
Nicolas: but like rust API design,
Nicolas: Getting to Tier 3 is getting more complex because it goes a way it's going to
Nicolas: be way more situational.
Nicolas: I do async Rust. I don't do that much embedded Rust.
Nicolas: And the journey, I wouldn't consider myself to be extremely proficient in embedded Rust.
Nicolas: I can code some stuff, and I've done some side projects like this,
Nicolas: but this is not my area of expertise.
Nicolas: And this is the part where what you're actually building starts to matter way
Nicolas: more, and then looking into whatever resources are interesting.
Nicolas: So the book from Mara Boss around atomics and locks,
Nicolas: for example, was quite interesting for me to understand some of the inner behavior
Nicolas: when we talk about some aspect of performance optimization for understanding
Nicolas: how mutex works and the hoods, understanding how...
Nicolas: It also helped me understanding how I can implement fairness better that are
Nicolas: into the systems we have to achieve this kind of,
Nicolas: you know, ensure that the system is going to continue to be able to deliver
Nicolas: good puts when we send it 100x the amount of traffic which it should be able to handle, for example.
Matthias: Quite a few sophisticated resources you have there. It's pretty cool.
Matthias: I guess a lot of people won't even reach that
Matthias: point because they are happy with Rust and
Matthias: just what it provides out out of the box you probably have
Matthias: to work on a distributed system or something
Matthias: more specific to get into these areas right
Nicolas: yeah no it's it's you very likely won't need that and it's good it's actually a very
Nicolas: good testament to the rust language itself like that you don't need these things
Nicolas: it just it just already gives you so much out of the box in terms of performance
Nicolas: just i mentioned like like T1,
Nicolas: just doing the cloning stuff and so on, depending on which programming language you come from,
Nicolas: you might already see performance boost just by doing that.
Nicolas: And you might see safety boost because, oh, well, now you have to deal with
Nicolas: options and results that you were not thinking about before.
Nicolas: So you were not thinking about some error classes and now you have to think about.
Nicolas: So just even there, you're already getting value from using Rust.
Matthias: Can you see differences between the different types of developers that start to adopt Rust?
Matthias: So do you see that this person comes from a JavaScript backend or from a Kotlin
Matthias: backend or from a Python backend, or is it very hard to discern?
Nicolas: Oh, there are some. I've seen a lot of people coming, for example,
Nicolas: from Java to Rust, and you see the kind of thinking of object-oriented programming,
Nicolas: some patterns that come from Java.
Nicolas: Java is a language...
Nicolas: With a lot of literature in terms of architecture patterns, software paradigms,
Nicolas: and so on, that people are just used to adopting and sometimes want to bring
Nicolas: to Rust in ways that don't always map one-to-one.
Nicolas: The trade system is not hierarchy, right? It's not inheritance.
Nicolas: So these are the kind of things that you see people trying to force certain
Nicolas: software architecture patterns into Rust that don't necessarily fit one-to-one.
Nicolas: And as I said, Java is probably the one that I see the most,
Nicolas: but that's probably because I don't know Java myself.
Nicolas: And there are so many Java developers out there. That's probably the one that surprises me the most.
Matthias: Is the trait system underused, in your opinion?
Nicolas: In a way, yes, but it's also a very dangerous slope if you don't understand
Nicolas: necessarily what you're doing.
Nicolas: It's like, in a lot of cases, I try to use traits where it makes sense,
Nicolas: but it can also bring a lot of complexity, and especially when you're doing
Nicolas: asynchronous, you might end up with very complicated trait signatures.
Nicolas: I worked on the Rust runtime for AWS Lambda in the past, which has to be extremely
Nicolas: flexible, because it's all about you bringing your own codes that does what
Nicolas: you want, and the library just wraps that so it can communicate with the Lambda internal API. API.
Nicolas: Yeah, the trait bounds were getting very wide.
Nicolas: And then you have to use a lot of clever stuff to do conversion and so on.
Nicolas: So there's lots you can do with the trait system.
Nicolas: I think it's underutilized, but it can also bring a lot of complexity to your code base.
Nicolas: There's also static versus dynamic dispatch. I won't go too much into the details,
Nicolas: but that also has implications that you might need to understand.
Nicolas: It's all about finding this balance or finding calibration
Nicolas: there yeah yeah but there's one thing about
Nicolas: books really
Nicolas: traits and i think this is something that the Rust for
Nicolas: helped is understand what are some of the common traits that are available in
Nicolas: the system for example i seen quite a lot of people doing a from string implementation
Nicolas: for force for for type while you
Nicolas: the from str that's way more generic and way more flexible and so on so,
Nicolas: there are things to or display for example right there are lots of traits that
Nicolas: are built into the standard library that you might want to kind of explore and get to know better.
Matthias: Earlier, you mentioned domain-driven design and design patterns in general.
Matthias: Are there any pointers to people that are in tier two that want to learn more
Matthias: about how to build proper rust code, idiomatic rust code with proper,
Matthias: let's say, design patterns?
Matthias: Did you learn anything along your journey with regards to design patterns?
Matthias: And where did you learn that?
Nicolas: I mentioned it. I'm an experiential learner.
Nicolas: I learn by doing and creating, and I find some cool ideas, and I'm thinking
Nicolas: to myself, hey, how can I use that?
Nicolas: It's kind of how I do things a lot. But there's also a lot of good literature
Nicolas: for that beyond just the Rust community.
Nicolas: One thing about patterns, patterns aren't a panacea. It's not like a silver
Nicolas: bullet. Nothing is a silver bullet.
Nicolas: You don't have to stick to exactly the right terms that this book guides you
Nicolas: to do. but there are lots of really good resources in terms of software architecture
Nicolas: patterns. I'm just trying to see if...
Nicolas: I have one on my bookshelf. I don't remember the name, but it's a very good one.
Nicolas: Yeah, so things like 97 things every software architect should know,
Nicolas: for example, are good things.
Nicolas: We all do a bit of software architecture as software engineers,
Nicolas: whether we know it or not.
Nicolas: So understanding these kind of things are quite good to have in terms of just
Nicolas: looking a bit at everything that's possible and so on.
Nicolas: Now that not everything will be applicable one-to-one to Rust,
Nicolas: because a lot of these books are written by people that use Java or C Sharp.
Nicolas: But still, there are lots of interesting things for that.
Nicolas: So, for example, just the question about, I mentioned earlier,
Nicolas: semantic versus safe types, right?
Nicolas: It's a distinction that we had to make that you don't really find often on when we talk about DDD.
Matthias: And how functional should Rust code be?
Nicolas: Ooh.
Matthias: Dangerous question.
Nicolas: Dangerous question, because I like it a lot, but it's also like a foot gun sometimes,
Nicolas: because there's so much power in the composability of Rust, all the map,
Nicolas: map or, and so on, which are very, very nice.
Nicolas: And I use them a lot because I really, really like them.
Nicolas: But then the question is, how much readable is it?
Nicolas: We get back to these kind of basic things around maintainability of code.
Nicolas: The code file is going to optimize a lot of things in the hood for you, right?
Nicolas: So the question is, if you think about your team, if you think about all the
Nicolas: people around you, are people going to understand what this piece of code does?
Nicolas: So the map methods are some that I use a lot, for example, for mapping a return
Nicolas: type from a result, things like this, right?
Nicolas: All the iterators are really awesome, and I really like them a lot.
Nicolas: Once again, all the map stuff in iterators are really awesome and I really like them a lot.
Nicolas: But you can also go too far to the point it's really hard to understand.
Nicolas: If you start to have a chain of calls that's 30 lines long, ask yourself,
Nicolas: hey, can anyone except me understand what's going on here?
Nicolas: And will me in six months remember what this does
Matthias: if you have a chain of 30 calls in a functional way maybe that hints at a missing.
Matthias: Abstraction maybe a missing struct or some sort of trait or something that is
Matthias: in between that helps you break down the chain
Nicolas: yeah yeah yeah i mean Rust will let you do that but yeah it's,
Nicolas: you might i mean yes you might just do like for example multiple maps that you
Nicolas: could could just do it to one function or something like this.
Nicolas: And remember, when you need to pass a closure, you can actually just pass a function.
Nicolas: So you can also decouple that by writing a name function that you can then test in isolation.
Nicolas: Because we haven't touched on test, but test has been very important in Rust.
Matthias: And with regards to structuring your code in general,
Matthias: would you agree that what you described is using functional patterns in,
Matthias: let's say, smaller scopes and structuring your code in a more object-oriented
Matthias: way in a bigger scope, or would you turn it around?
Nicolas: It kind of depends on what you do. I found that there are two very big ways.
Nicolas: If you think about async Rust, I found there are two very big ways to do it,
Nicolas: depending on where you end up.
Nicolas: If you are building a network service,
Nicolas: for example, something like the router, it's a GraphQL API that works like a
Nicolas: pipeline that makes sub-requests to microservices and so on.
Nicolas: Then we are way more leaning into the tower ecosystem and the service traits
Nicolas: and basically composing service together into a pipeline.
Nicolas: But now if I'm thinking about our control plane where we need to interact with 10 different services,
Nicolas: all with their own SDKs and so on, then we might need to have some kind of struct
Nicolas: that contains references to these different SDKs and these different clients
Nicolas: so we can actually make the calls more efficient.
Nicolas: And so these are two very opposed architecture patterns of AsyncRust, for example.
Nicolas: And they have their pros and cons. It's a matter of knowing when to use watch.
Nicolas: But the thing like composability of total service is really, really great as well.
Nicolas: Just to avoid repeating the same thing over and over again.
Nicolas: So it's a matter of finding kind of what's the right method based on what you're
Nicolas: doing and what you can reasonably manage.
Matthias: That's one of the things I like the most about Rust that all of these crates,
Matthias: they develop outside of the standard library.
Matthias: And if you look at error handling, for example, we had multiple iterations of
Matthias: different error crates and each one improved on what came before.
Matthias: And I think this is really, really powerful. I see the same happening in Async
Matthias: Rust and in network services and web services and so on. and I think we should keep that up.
Nicolas: Oh, yes, yes. Going back to the question about traits, it's really where traits
Nicolas: are really, really powerful.
Nicolas: So, total service is great. You can build something on top of that.
Nicolas: Or the question about the error trait or...
Nicolas: Actually, I had a case like this, speaking of traits, in the way the hyperlibrary is composed,
Nicolas: where it's under the hood, it's using trait, and I was able to do that to actually
Nicolas: make a mock tester, for example, that actually makes calls, but just these calls
Nicolas: don't go to the internet, just because I was able to plug into this trait system.
Nicolas: Or if you do embedded programming in Rust, they also made awesome thing with traits, right?
Nicolas: Where no matter what your actual specific hardware, where you have this abstraction
Nicolas: layer, which is zero cost as well.
Matthias: Great. I guess we're getting towards the end. The remaining questions would
Matthias: be about the broader Rust community and also taking a bet on Rust from a business perspective.
Matthias: Regarding the community, would you say that you benefited from existing crates
Matthias: or did you have to mostly write everything from scratch for yourself?
Nicolas: So I have a mixed opinion on the crates.
Nicolas: One is, yeah, there's so much stuff I could just use from the community.
Nicolas: But small asterisk in terms of my personal philosophy in terms of dependency
Nicolas: management is that every time you add a dependency, well, you own that from a vulnerability,
Nicolas: from a security point of view, right? So still be careful.
Nicolas: And that's actually where it gets into my downside. is if you're new to Rust,
Nicolas: it's kind of hard to know which crates are actively maintained,
Nicolas: which crates are the right ones to use.
Nicolas: If you're looking at something a bit niche.
Nicolas: Then that might get also a bit tricky to find, hey, which one's actually actively maintained?
Nicolas: I was working on a small side project for myself with a very specific algorithm,
Nicolas: and there were two implementations with two different crates.
Nicolas: And I had to do a bit of research, you know like which one's actually actively
Nicolas: maintained and which one hasn't been updated in five years right this
Nicolas: is a part of the thing that's a bit hard to do if you
Nicolas: just search for crates by itself trying to find
Nicolas: which one's actually doing going to do what you need so that you don't have
Nicolas: to reinvent the wheel yourself every time so it's a double edged sword it goes
Nicolas: both ways yeah like that there are things where it will be fine you know tokio
Nicolas: i mentioned tower are we We're using thiserror, for example, for error management.
Nicolas: A lot of these crates are really well maintained and so on.
Nicolas: But then when you get to the more deceptive stuff, or if you don't really pay
Nicolas: attention, you might end up with something that's not maintained anymore.
Matthias: Do you think that Rust community is underfunded?
Matthias: Or what could be the reason for some maintenance issues
Nicolas: i mean i think it's it's there's
Nicolas: lots of things that come from enthusiasts which is
Nicolas: great you know it's still a fairly young language or things
Nicolas: considered right so there are lots of craze that
Nicolas: are made by enthusiasts and then you know enthusiasm doesn't necessarily
Nicolas: last forever you can't be a single person maintaining the
Nicolas: same project for 10 years right that's not really reasonable to
Nicolas: expect that so things that have of actual
Nicolas: funding things that are backed by companies are
Nicolas: great where developers from
Nicolas: a certain company can actually contribute to a
Nicolas: certain project for example is really great and that's the story of open source
Nicolas: in general right you need you need people that can maintain it and if these
Nicolas: people can make a living out of it these people can be you know compensated
Nicolas: for for their works and efforts. It's a really great thing.
Nicolas: But there's lots of things in Rust that where people are just trying things
Nicolas: out, learning, and so on, and then they just don't have the energy,
Nicolas: the time, or something to maintain it anymore. And that's kind of the tricky part there.
Nicolas: So there are some islands of very well-funded projects, but then there's lots
Nicolas: of things that are not so well-funded.
Matthias: That means we have to find a transition from enthusiasm to long-term maintenance,
Matthias: and it's an ongoing process.
Nicolas: Yes, it is.
Matthias: And if you wanted to address a company that was on the verge of trying Rust
Matthias: for their own use case, for their own product, with your experience,
Matthias: what would you tell them?
Nicolas: So I'm going to start very businessy on this. but define a business case,
Nicolas: define clear objectives and results you want to achieve is the first thing.
Nicolas: It's not something that's just specific to Rust, but it's something I've seen
Nicolas: a lot of companies kind of struggle with and say like, oh, we need to go with
Nicolas: Rust, it's going to be more performant and so on.
Nicolas: But then it's still going to be a project in itself.
Nicolas: Try to find something small where you can have a higher value impact from what
Nicolas: your expected returns are from there. So if you think like, oh,
Nicolas: I'm going to give you two examples that might be relatable to some folks.
Nicolas: So the first one is, let's say we have a performance-critical system,
Nicolas: and we need to improve its performance.
Nicolas: And we think, okay, if we rewrite it in Rust, we will get better performance.
Nicolas: First step, is there some part
Nicolas: of this that you can actually rewrite in Rust, you know, in isolation?
Nicolas: Either a service itself or nowadays
Nicolas: there's so many great interrupt capabilities between other
Nicolas: languages so wasm is one you have
Nicolas: bio3 for python you have all the things like this that allow you to write a
Nicolas: library in rust or something that's for example very slow or if you're using
Nicolas: lambda function on aws or any kind of things like this typically you can rewrite
Nicolas: just one function So trying to find something small you can do to measure the impact,
Nicolas: gain the experience as well of building that, because if it's for lots of people,
Nicolas: the first project they're doing in Rust, they also need to learn.
Nicolas: And so that would be a good way to kind of go forward. Another example I found
Nicolas: is people using Rust for, once again, leveraging the ability to interrupt with different languages.
Nicolas: Using like Wasm, for example, to create one library that you can use across
Nicolas: web, iOS, Android, for example.
Nicolas: So same here, you have a clear objective of what you want to achieve.
Nicolas: And then from there you can go build
Nicolas: what would be the minimum implementation we can actually
Nicolas: see the return on how are we going to measure that
Nicolas: you know well how do we define success so we can then
Nicolas: how are we going to capture the learnings what was tough what
Nicolas: was hard you know so you can actually have some solid
Nicolas: base to say like hey do we really want to go with rust from then
Nicolas: on right where can people learn more about
Nicolas: apollo i would say it's very likely on
Nicolas: our website you're going to learn learn a lot we have odyssey courses so these
Nicolas: are courses to help you learn about graphical technologies in general but also
Nicolas: what we offer at Apollo and how this can help you if you want to use graphical
Nicolas: technologies awesome and finally it's become a little tradition around here.
Nicolas: To let the guest share something with
Nicolas: the broader rust community so if you have anything on
Nicolas: your your mind that you want to convey something that
Nicolas: maybe you want to always always wanted to share
Nicolas: with the rust community something that we need to be aware of as a community
Nicolas: that would be your chance yeah it's so one thing i really liked with the rust
Nicolas: community is the people in general i found a lot of inclusiveness from from
Nicolas: rust people especially towards people that are neurodivergent.
Nicolas: Which is really awesome rust is is a programming language that help a lot when
Nicolas: you have trouble dealing with high level of cognitive cognitive laws like me
Nicolas: for example and that really helped me personally that is a really awesome trait
Nicolas: of of the rust community is the fact that we are so.
Nicolas: Accepting other others welcoming here to help each other's there is this i always
Nicolas: remember when we We talk about these kind of things about Esteban
Nicolas: Kuber, talking about ballads,
Nicolas: the importance of ballads in the Rust programming language.
Nicolas: So that's the one thing I would say to the Rust community as a whole. Keep that aspect.
Nicolas: This is something that I haven't really experienced in many other communities,
Nicolas: this level of inclusiveness and people just being there, being awesome and helping each other.
Nicolas: So yeah, stay like this.
Matthias: You said it perfectly. I have nothing else to add. Nicolas, thanks so much for being on the show.
Matthias: I hope you enjoyed it as well. And yeah, thanks a lot.
Nicolas: Yeah, thank you for having me.