Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from Corode, and today we talk to Arpad Borsos from
Matthias: Sentry about enhancing a massive Python platform with Rust.
Matthias: Welcome to the show.
Arpad: Hi, thanks for having me.
Matthias: , can you quickly introduce yourself and Sentry, the company you work for?
Arpad: Yeah, absolutely. Well, I'm Arpad and I've been with Sentry for four years and
Arpad: Sentry is in the application monitoring business.
Arpad: So we do error handling or rather error tracking.
Arpad: We do performance monitoring, we do profiling, and we are also about to,
Arpad: well, in the future, launch a metrics product as well.
Matthias: And i know you from a GitHub
Matthias: repository that i personally use in a lot of rust
Matthias: projects because you are the maintainer of rust cache and your handle is swattingham
Matthias: and yeah it is very very popular i can tell so first off thanks for this GitHub action yeah.
Arpad: How d
Arpad: My pleasure.
Matthias: id you get started with that action before we go into the main content?
Arpad: Well, Sentry itself uses GitHub. And for people who don't know,
Arpad: Sentry is fully open source, depending on the definition of open source you want to use.
Arpad: So parts of Sentry are completely liberally licensed. Other parts use the user …
Arpad: well, I would say open source license, but other people might call it the source available license.
Arpad: But anyway all of the source is available on GitHub
Arpad: and Sentry is using GitHub and GitHub actions
Arpad: for all of the ci and well back
Arpad: in those days before i actually started writing my my GitHub action there was
Arpad: some recommendation about using the well GitHub's own caching action but you
Arpad: had to configure it in a weird way you had to define a cache key that well depending
Arpad: on On the operating system you run,
Arpad: depending on the Rust version you run, you have to actually make sure to bump
Arpad: this cache key manually.
Arpad: And also the recommended way of which files to cache was rather unoptimal for Rust.
Arpad: So my action had basically two goals was to make it extremely simple to use.
Arpad: So no manually defining cache keys by default.
Arpad: If people want to do it, they can, of course. and also to kind of find the right
Arpad: balance of how many files to cache, right? Because you have a limited...
Arpad: Space for for the cache itself and of course there's also download time extraction
Arpad: time so it it tries to strike the right balance between the overhead of the
Arpad: cache itself versus the gains you get from caching.
Matthias: an tell it is quite a sophisticated project by now and it is is very well received
Matthias: so far?
Matthias: that's all i can say. For how long have you been working at
Arpad: It's my fourth anniversary already so i'm four plus years here and i'm very
Arpad: lucky to be writing rusts as my day job since almost basically the whole time and.
Matthias: You also write articles on your blog about rust right.
Arpad: Yeah exactly so every time I discover some interesting fun fun facts or some
Arpad: problems and challenges that I face, then I tend to write about it.
Arpad: And the good thing here is that Sentry is on GitHub.
Arpad: It's open source. So I can actually write about all of this stuff and point
Arpad: directly to the code that has been causing issues.
Arpad: I can point directly to the fixes that I've applied.
Arpad: So people can really take a hard look at stuff.
Matthias: One of the benefits of working in the open, I guess.
Arpad: Exactly, yeah.
Matthias: Has Sentry always been open source?
Arpad: Yeah, as far as I know, Sentry has been open source since the very beginning.
Matthias: Amazing. Now, going back to the days when you started at Sentry,
Matthias: can you take us back and explain to us what the infrastructure looked like,
Matthias: what the application looked like,
Matthias: which languages you used, and all those things? Can you maybe just give us some context?
Arpad: Yeah, back when I joined Sentry, there was already quite a bit of Rust in production.
Arpad: And I initially started rewriting the native SDK.
Arpad: So I spent lots of time in C code before then moving on to the Rust SDK,
Arpad: which already existed back then and revamped that a little bit.
Arpad: There are things that already existed back in those days were Symbolicator,
Arpad: which is is part of Sentry's processing pipeline,
Arpad: which deals with native events, well, processing native events,
Arpad: and then, well, applying all of the debug files to make stack traces actually readable,
Arpad: and also Relay, which is first part, the ingestion part where events actually come in from customers.
Arpad: So those two pieces have already, since more than four years, been written in Rust.
Arpad: I started out with the native SDK, moved on to the Rust SDK,
Arpad: and then eventually took over the maintenance of the processing pipeline and
Arpad: specifically Symbolicator, which then later on also gained the ability to process JavaScript events.
Matthias: You have a strong background in C?
Arpad: Sentry
Arpad: See so before starting at Sentry i
Arpad: Sentry
Arpad: was actually a JavaScript and TypeScript developer i did
Arpad: contribute to c and also to
Arpad: rust before so i had some some hobby
Arpad: projects or whatever in rust and i've also contributed to c code one of the
Arpad: the contributions i'm actually quite proud of was contributing to Firefox and
Arpad: Mozilla so there's some contributions in Firefox things things that are touching C code,
Arpad: things that have to do with the SpiderMonkey JavaScript engine.
Matthias: How did you get started with that? I imagine it is quite a sophisticated piece
Matthias: of technology, and it sometimes can be intimidating.
Matthias: How did you even dare to touch the Firefox codebase?
Arpad: Yeah, it actually is quite intimidating. and I can remember that I was actually
Arpad: shivering a little bit opening my first code contribution to Firefox.
Arpad: And I would say I always felt quite welcome to do so.
Arpad: Back in those days, Mozilla had this Bugzilla bug tracker and there was tons of stuff in there.
Arpad: Some things might have been labeled as good first bugs for people to pick up
Arpad: and i just chose one of those i tried my hands at it and then one thing led to another.
Matthias: Firefox is a tool that is used by millions of users and back in the day probably it
Matthias: was more in the hundreds of millions of users. I would personally be scared that i introduce a,
Matthias: soundness bug in the code so parsing
Matthias: or handling input from various sources could lead to dangling pointers and access
Matthias: to memory that maybe should not be accessed were you scared of that were there
Matthias: eah t
Matthias: any safety hatches in the code base.
Arpad: his was always in the back of my mind And I believe I might have caused some
Arpad: issues as well, especially the contributions to the JavaScript engine.
Arpad: There were things that were discovered with fuzzing and other fuzzers.
Arpad: Otherwise, well, it's not only me as someone who comes to the project as a new contributor.
Arpad: It's also people who have been working with the project for a long time.
Arpad: They can also make mistakes, right? it and either you have like a really good
Arpad: test suit that covers this you might have fuzzing that,
Arpad: discovers things and and memory corruptions or whatever that no one would have
Arpad: thought of and now luckily we have the rust compiler who hopefully prevents
Arpad: But b
Arpad: most if not all of these issues.
Matthias: efore we get to that take us back to this moment when you
Matthias: started working on the
Matthias: c sdk and explain to us what you felt when you
Matthias: looked at the code and the patterns was it well tested was there any fuzzing
Matthias: going on did you have ways to check
Matthias: that for example the type system would check for bugs what was it like.
Arpad: Hmm i would say there was a
Arpad: test suit but it wasn't as extensive and some
Arpad: things i also pushed for quite heavily was to introduce things like fuzzing
Arpad: which then found quite a few bugs in in the c json parser for example and i
Arpad: also pushed for introducing more things like static code code analyzers,
Arpad: things like CodeCoverage and
Arpad: then Clang Analyzer and all these kinds of things that are in the C world,
Arpad: which are quite difficult to use, I would say, in comparison to things like
Arpad: Clippy or Miri, which are just a breeze in comparison.
Arpad: So, yeah, it was quite interesting working on it. And there was also lots of
Arpad: stuff that was going wrong.
Arpad: So as I was saying, with fuzzing, we found some memory corruption related to the JSON parser.
Arpad: There was also a very serious bug that might have caused data loss on customers'
Arpad: computers related to some problems in the C SDK.
Matthias: What did an error report look like back in the day?
Matthias: Did you really get the entire context of what happened? Did you have to take a lot of guesses?
Matthias: Could you kind of locate the issue in the code base? Was it a pleasant experience?
Arpad: It was complicated. So it was often either a GitHub issue where people reported
Arpad: these issues or some internal customer communications.
Arpad: And figuring this out, it all depends on the issue report, right?
Arpad: So if someone opens an issue on GitHub, it should have enough context for the
Arpad: maintainers to actually figure out what's happening.
Arpad: Oftentimes, the context might be very good and you find the issues immediately
Arpad: or you actually have to dig deeper.
Arpad: You have to figure out how can I reproduce this?
Arpad: And if you reproduce this, then comes the way of actually figuring out where
Arpad: is the bug and how can I fix it?
Matthias: And at the same time you look over your shoulder and your colleague is working
Matthias: on the rust sdk did that spur some interest from you or were you even maybe
Matthias: a little envious that they used this new modern language um.
Arpad: Yeah a little bit so when i started it was always you know advertised to me
Arpad: that I might be able to work with Rust.
Arpad: And I was really eager to actually move on from C to writing more Rust code as well.
Matthias: Was there a reason why you applied as well?
Arpad: Yes, it was definitely a very big factor.
Arpad: So like I said earlier, I used to be JavaScript and TypeScript developer.
Arpad: And actually, I intentionally made this move and cut in my my career to move
Arpad: to rust and this was a very good decision and very liberating as well does.
Matthias: It mean at some point you looked for companies that used rust in production
Matthias: so that you could do more rust in your day job.
Arpad: Not specifically actually it was
Arpad: how your Austrians would say Freunderlwirtschaft
Arpad: so i was basically
Arpad: referred from from a friend so
Arpad: a previous colleague who i worked with joined
Arpad: sentry first and she immediately referred me because
Arpad: she found out oh sentry was working with rust and she knows me so she immediately
Arpad: referred me as well and that's how i got to sentry eventually so so i did not
Arpad: actually search specifically for it but it kind of happened and through connections yeah.
Matthias: A very common pattern.
Arpad: Yeah
Arpad: Referrals between g
Matthias: ood colleagues yeah. Makes sense. But eventually you moved on and you moved over
Matthias: to rust that means you were the maintainer of the rust sdk is that correct or d
Matthias:
Arpad: id you share that responsibility with others?
Arpad: Yes i still am mostly the maintainer
Arpad: of the Sentry rust sdk there isn't too much work going on with it it's quite
Arpad: stable there there isn't much that we need to add or change right now,
Arpad: if i had the chance i would do one major rewrite yet again so there is some
Arpad: things that we did back when i first kind of rewrote the Rust sdk which i
Arpad: would do differently today day what.
Matthias: Would that be and how would you tackle that rewrite?
Arpad: So one thing I did early on was to split the Rust SDK into different sub crates.
Arpad: And I faintly remember that tokio early on did the same thing.
Arpad: And they moved back to actually a single crate with a bunch of feature flags.
Arpad: And I would probably do the same thing to instead of have like 10 different
Arpad: sub crates for each integration, like sentry anyhow, sentry tracing,
Arpad: sentry blah, blah, blah, whatever.
Arpad: I would rather move back to a single crate for a variety of reasons.
Arpad: So, one is maintaining it should be easier.
Arpad: And the other reason is that people get confused using this.
Arpad: So we had a couple of problems reported to GitHub that people had version mismatches, right?
Arpad: So they had one Sentry crate in one version, but another Sentry crate in a different
Arpad: version, and they're incompatible with each other.
Arpad: Because one good thing about Rust is that you can have a crate in different
Arpad: versions compiled in the same program.
Arpad: So it's a good thing that it works but for
Arpad: sentry the problem is that you configure sentry version i don't know version
Arpad: x to actually send things to sentry but then the sentry that actually captures
Arpad: panics is a different version which has never been configured to actually send events upstream.
Arpad: So people, I've come across this GitHub issue a couple of times,
Arpad: and then this happens to people and they are confused why panics never land in Sentry. Well.
Matthias: Couldn't you fix that retroactively by merging all of them and releasing a breaking change?
Arpad: We could definitely do it. And one of the problems with the Rust SDK is that
Arpad: we have breaking change releases all of the time,
Arpad: because our types are a bit too detailed.
Arpad: And Sentry itself as a product is changing also quite a bit.
Arpad: So when I joined, we only had the error reporting.
Arpad: And then we also added performance tracing. So the tracing integration,
Arpad: which is, by the way, really awesome. And I can highly recommend this to people.
Arpad: So you're just tracing instruments, a function, you just slap the attribute on there.
Arpad: And these things will be shown on Sentry then.
Arpad: So adding this to the SDK means we have new types, which back in those days,
Arpad: you didn't even had this non-exhaustive attribute.
Arpad: So this was added later. So at one point, we also added non-exhaustive to a
Arpad: lot of enums and lots of types that we have.
Matthias: Just for context, before you continue, non-exhaustive for the uninitiated is a feature where you...
Matthias: Add an attribute to a type like an enum which prevents your type from getting matched,
Matthias: on a call site for example if you have an enum with two
Matthias: variants you cannot match exhaustively on both
Matthias: variants and be sure that you covered all the cases and that is a safety or
Matthias: an escape hatch for adding further variants down the line and that feature was
Matthias: not available back then and I assume it would be quite hard to recreate such
Matthias: behavior because it's outside of the standard library, right?
Arpad: Mm-hmm, mm-hmm.
Matthias: It could have potentially been done with external crates, but then you would
Matthias: have to know about these crates first.
Arpad: Yeah, I mean, I think especially for enums, there used to be this pattern where
Arpad: you have a non-documented variant variant that never shows up in any kind of
Arpad: publicly visible documentation.
Arpad: But for the type checker, it exists, right? So that was back then,
Arpad: I believe, the main escape hatch to make something like this possible.
Arpad: And for structures as well, if you have a single private member,
Arpad: then you cannot use struct literals, for example.
Arpad: And the main thing with non-exhaustive is also you cannot use a struct literal,
Arpad: you You have to use a constructor, right?
Matthias: A struct literally is when you don't use a constructor, but you explicitly name
Matthias: the fields in the struct when you initialize it, right?
Arpad: Exactly, exactly. So in this case, you have to define every field separately,
Arpad: or you just do dot dot default.
Arpad: And then basically you get all the default values. You define the ones you want to override. And...
Arpad: One of the things I wish Rust could support was actually to combine this ..default
Arpad: struct literals with structs that are non-exhaustive.
Arpad: Because right now, it's not supported. So the Rust compiler will scream at you.
Matthias: Interesting. I did not know that that would have been a possibility,
Matthias: but yeah, it makes sense.
Matthias: Now moving on from the early rust days for you at sentry and moving on from
Matthias: the rust sdk which i assume is in good shape now because you took care of it,
Matthias: what was the next step for you in this rust journey at sentry.
Arpad: Yeah the next step was taking over ownership of Symbolicator which is a big
Arpad: part of the the whole processing pipeline in Sentry.
Arpad: And I remember one of the big early challenges was updating Symbolicator from
Arpad: things like the very, very old Actix web and the tokio 0.1 days.
Arpad: And then to update this to tokio 0.2. And then afterwards, we also went to tokio 1.0.
Arpad: And since then, it's stable. So that's been very good. But those early days, they were a bit rough.
Arpad: Right. And back then, Actix Web had really good support and kind of an out of
Arpad: the box solution for both having an HTTP client and also an HTTP server.
Arpad: Which I believe was one of the reasons people were such big fans of Actix Web back in those days.
Arpad: And moving from this to tokio was quite a challenge.
Arpad: So it was actually finding an HTTP client and we chose requests,
Arpad: which I believe lots of people in the Rust community use.
Arpad: But it did have some differences in behavior to how the Actix web clients used to work.
Arpad: So that was a challenge making it compatible and having the exact same behavior that we were relying on.
Arpad: And then also to change all of the internals from the Actix model over to the tokio model.
Matthias: For some context, just because people might be wondering, Actix is a web framework
Matthias: and tokio is an async runtime.
Matthias: But back in the day, or even to this day, I'm not sure, Actix was built on an actor framework.
Matthias: So that was a different model of running asynchronous code, I assume.
Matthias: And i think this is what you meant moving away from this actor model that actix
Matthias: imposed towards tokio plus.
Arpad: Mm-hmm
Arpad:
Arpad: a more
Matthias: Lightweight http client is that correct.
Arpad: Exactly yeah and it was also back then writing async rust wasn't really as comfortable
Arpad: as today so back then you you didn't even have syntax for async function.
Arpad: You just had future types from the futures crate and it was the futures crate 0.1.
Arpad: You actually had to chain each step onto the other using and then and then and
Arpad: then and then sharing state between those always involve things like arcs and maybe arc,
Arpad: And the way we write async Rust today with just async function and the compiler
Arpad: just takes care of everything.
Arpad: And you can have local references, you can have self-referential structs,
Arpad: you can have mute references and all of these nice things.
Matthias: Although if you use the default tokio runtime you still need to use arc and mutex you.
Arpad: Mm-hmm
Arpad: could use a local e
Arpad:
Matthias: xecutor but the default would be to still lock things, correct?
Arpad: Yes absolutely. But
Arpad: what i rather mean is you can write an
Arpad: async function just as you would a normal function. S
Arpad: o you can have mut reference
Arpad: and holding this across await points so this is perfectly possible with an async
Arpad: function but back then when you had to chain future well and then together this
Arpad: eah.
Arpad: was really really challenging.
Matthias: Essentially what the compiler does for us automatically now,
Matthias: we had to do by hand writing generator or continuation-like syntax.
Matthias: Oh, yes.
Matthias: A state machine in the end. Yeah. I'm glad that these days are mostly over.
Arpad: Oh, yes.
Matthias: Do you have any guidelines for that inside of Sentry on how to write Rust code,
Matthias: how to write idiomatic Rust code, how to maybe also improve your skills and
Matthias: improve your Rust prowess over time?
Arpad: So we do have Rust development guideline at Sentry, but I would say it's mostly
Arpad: for new contributors to the Rust code bases we have.
Arpad: Me, for example, or other team members who have been writing Rust for a very
Arpad: long time at Sentry, we kind of predate these guidelines in a way.
Arpad: But things that are written down, I would definitely say there are very good
Arpad: guidelines also about how do you write things like panic safe code and all of these kinds of things.
Arpad: So things I would definitely say that I would recommend to everyone is to be
Arpad: weary of slicing syntax.
Matthias: Can you elaborate?
Arpad: Well, slicing syntax, if you're out of bounds, you're panicking.
Arpad: And also, if you're dealing with strings and you happen to index into a Unicode
Arpad: boundary, then you're also panicking.
Arpad: If you use the get function call, you can specifically check for this because you get an option.
Matthias: Is there a rule inside Sentry which prevents or discourages slice syntax?
Arpad: I wouldn't say it's a hard rule. rule it rather
Arpad: depends on the reviewer reviewing the code and on
Arpad: the circumstances so especially because we're
Arpad: dealing with lots of untrusted data we tend to index or slice into things where
Arpad: the index isn't trusted and in those cases i would definitely flag this in reviews
Arpad: as hey please use get instead of slicing into this but.
Matthias: Sometimes you might miss a case and maybe there is a tool for that.
Matthias: I don't know if Clippy supports you with that. I don't know.
Arpad: I think Clippy supports you with that, yes. But the tool is called Sentry.
Arpad: And we capture those panics that happen in production.
Matthias: Nice segue. Are there generally any rules on how you review Rust code?
Matthias: Things that you look out for?
Matthias: Things that you keep in mind? Things that you usually check,
Matthias: which people often get wrong? You mentioned the slice syntax,
Matthias: for example. Are there any other things?
Matthias: How do you review Rust code?
Arpad: That's an interesting question. I mean, there are a couple of things I tend
Arpad: to be very cautious about, mostly because it's things that happened in production.
Arpad: For example, what I said, slicing into Unicode and getting an error or getting
Arpad: a panic because you slice on a Unicode boundary.
Arpad: Those things happened to us and they panicked in production and that's how you learn.
Arpad: I would say there are certain things I tend to focus on, things like unwraps, things like slicing.
Arpad: Maybe in some cases it might be worth using checked math, especially if you're
Arpad: dealing with untrusted data, but those are rare cases.
Arpad: Other things I try to review, especially now that we have a lot more engineers
Arpad: onboarding to Rust code,
Arpad: is to maybe look at things like clones a little bit more,
Arpad: especially if you have vectors or if you have strings cloning is actually o
Arpad: of n and it allocates and it might be better to just avoid it by using string slice what.
Matthias: I found was that if you use clones a lot then usually that indicates that you
Matthias: have a problem with your architecture or with your your design,
Matthias: there can be situations when you cannot fully lean into the Rust,
Matthias: borrow checker because of a fundamental issue with your code base.
Matthias: And if you try to remove those clones, you will find the root cause of it.
Matthias: And maybe you can improve the design of the entire application to not work against
Matthias: the borrow checker all the time, because it is certainly a warning sign if you
Matthias: have a lot of clones, at least to me.
Matthias: It's not bad unless you are not aware of it, of what's going on and maybe how
Matthias: you can improve the situation.
Arpad: I think you're right. There's definitely lots of cases where clones can show
Arpad: you that you're doing something wrong.
Arpad: For us, though, one of the problems might be that in lots of cases,
Arpad: we actually want to have things that have the static lifetime.
Arpad: So, or to phrase it differently, we want to have owned types without any lifetime, right?
Arpad: So for example, as you were, as we were discussing tokio previously,
Arpad: if you want to tokio spawn something, it better be, or it better have the static lifetime, right?
Arpad: Because tokio takes it. And now tokio is responsible for it. tokio will move it across threads.
Arpad: Well, and, and tokio will just you know drop it once it's finished and…
Matthias: Isn't that also one thing that,
Matthias: you see as well where you
Matthias: have normal set let's say synchronous rust
Matthias: code and you freely borrow things
Matthias: and you can move things around and all of a sudden you move into
Matthias: an async ecosystem system where maybe the
Matthias: runtime imposes that you think about such things like allocations or maybe ownership
Matthias: as well and and then some of the patterns don't work anymore so you need to
Matthias: clone where before you would have to you would be able to move for example.
Arpad: I think the very interesting thing here is that you actually have a choice and
Arpad: what i mean by this is async doesn't doesn't automatically mean that you cannot
Arpad: have references or that everything needs to be Sent and Static and all these kinds of things.
Arpad: Because you can have.
Arpad: Structured concurrency, I believe that's the term people use,
Arpad: is you can join all or you can create a bunch of futures and you can use join all.
Arpad: But join all means that those futures still execute within, well, your async function.
Arpad: But it's just spawn where you actually have to have something that sends sync and static, right?
Arpad: And there's different trade-offs here, right? Right.
Arpad: So if you use structured concurrency with join all or with, well,
Arpad: if you join two futures or if you join a vector of futures, then those run concurrently,
Arpad: but they do not run in parallel.
Arpad: Right. So it's still a single CPU at a time working all of these futures.
Arpad: But you have the advantage that you can actually have references without any kind of problems.
Arpad: But on the other hand though if you use tokio spawn you do have parallelism
Arpad: because individual cpu cores can execute all of this code but you have the disadvantage
Arpad: that you have to use things like arcs and mutexes all over.
Arpad: ery t
Matthias: rue. Although as a small caveat there are other runtimes which don't depend
Matthias: on work stealing where you could potentially have multiple things running concurrently
Matthias: on different cores like thread per core models where you don't need synchronization.
Arpad: You might not need synchronization but you definitely have to make these things
Arpad: static because they might potentially outlive the place you actually spawn the future right.
Matthias: Yes, yes so you might get rid of the mutex but not necessarily the arc.
Arpad: Mhm.
Matthias: Well it's kind of an interesting discussion we kind of diverged a little bit
Matthias: let's come back to the business side of things a little you mentioned that you
Matthias: worked on symbolication,
Matthias: and you also touched on relay real quick maybe can we start with symbolication
Matthias: what is it about what does it do, how is it built, and so on.
Arpad: Symbolication, to put it in simple terms, we make stack traces readable,
Arpad: which means right now symbolication is done for native events and it's done for JavaScript events.
Arpad: For native events, you might know also from Rust codes that if you have a panic
Arpad: in release mode built without debug symbols,
Arpad: you see a lot less detail as if you are panicking in a debug build, right?
Arpad: With Sentry, you build your release code with debug symbols,
Arpad: but then you split it apart.
Arpad: One thing you ship to production, the other thing with all the debug data you upload to Sentry.
Arpad: And we put those two parts together again to give you a readable stack trace.
Arpad: That's what the processing pipeline does.
Arpad: For JavaScript, it's a bit similar, but it's called source maps.
Arpad: And JavaScript, TypeScript developers might be familiar with them.
Arpad: Then in JavaScript, you also have your minified JavaScript that runs somewhere
Arpad: in some browser or maybe also in the cloud.
Arpad: And then you have a source map, which actually turns the stack traces into something
Arpad: that's readable. And that's also what we're doing in the processing pipeline.
Matthias: Which languages are supported for these stack traces?
Arpad: It's JavaScript and everything that boils down to native code.
Arpad: And also .NET to a certain degree.
Matthias: That's a lot of supported platforms.
Matthias: For example, that would also support Rust, I guess, and Go and C?
Arpad: Absolutely, yeah.
Matthias: And I only vaguely know that you have a concept called demangling in some of these languages,
Matthias: where you have certain identifiers for functions and types,
Matthias: and they get mangled in some form by the compiler.
Matthias: Hider and is that also something that you do you kind of reverse that process
Matthias: to get underlying symbol or type that you had in your code.
Arpad: Yes exactly so demangling is especially relevant for all the native platforms
Arpad: so as you mentioned the symbols that might be shipped in your or executable are compressed using,
Arpad: well, a certain definition or a certain way of how these symbols are compressed,
Arpad: the names of these symbols are compressed.
Arpad: And demangling basically reverses this and gives you a readable function name
Arpad: that also has things like the generic parameters and all these kinds of things in it.
Matthias: And how do you even build such a platform? walk us through the components which
Matthias: take some input and then in the end you have some output like what are the steps required the.
Arpad: First step is you get to events and i would say even the most difficult part
Arpad: then is finding the corresponding debug files,
Arpad: and for maybe on the.
Matthias: Server shouldn't they be right there.
Arpad: They are right there but you need to know which files to use and so there are many files.
Arpad: There needs to be some kind of identifier for it right and this is also something
Arpad: I talked about on on my blog a little bit is that the files need to be or
Arpad: need to have some kind of unique identifier identifier,
Arpad: because we're talking about a pair of files, right?
Arpad: So you have always one file that's with executable code that runs in production.
Arpad: And then you have a different file that has all of the debug information in it, right?
Arpad: And they have to share some kind of unique identifier, right?
Arpad: And if a crash or a panic happens in production, we attach this unique identifier
Arpad: to the event we send to Sentry.
Arpad: And with this unique identifier, we can then look up the corresponding debug file.
Arpad: And there's also things like system libraries, and there is a whole symbol server
Arpad: specification that I believe was pioneered by Microsoft.
Arpad: And for example, all of the Microsoft NT DLL32 or however they are
Arpad: called, they are accessible on Microsoft's official symbol server.
Arpad: And they all have a unique ID with which we can download it from there.
Arpad: And then we can show you the function name in the Windows system call.
Matthias: Do you also sometimes have to guess if you're not exactly sure what symbol might
Matthias: be? or do you have to reverse engineer this process or do you say the specification
Matthias: for symbols is sufficient?
Arpad: I would say, well, it's a tricky question because specification,
Arpad: are things really specified well enough?
Arpad: Sometimes maybe yes, sometimes maybe no.
Arpad: So the symbol server specification, where we know depending on things like unique
Arpad: identifiers and file names, how to download corresponding debug files,
Arpad: that's quite well done. And we can use it quite well.
Arpad: Things like demangling, they are so-so, I would say.
Arpad: It is kind of specified, but the specification lags behind reality a little bit.
Arpad: Because especially in the C++ world, Clang tends to add support for newer C++
Arpad: language features, but those are not really specified in the demangling scheme yet.
Arpad: On the plus side.
Matthias: You'll learn a lot about new C++ language features like that.
Matthias: Absolutely, yeah.
Arpad: And also, what has been quite challenging since half a year or so,
Arpad: I believe it's a year or half a year roughly since we added support for source maps here.
Arpad: And source maps are especially difficult in this regard.
Arpad: Because JavaScript files do not have a unique identifier.
Arpad: And we are kind of championing this idea to give javascript files and their
Arpad: corresponding source maps a shared unique identifier because usually a javascript
Arpad: file just has a source map reference or not even this,
Arpad: so it only refers to its source map by name but as you might be aware in the
Arpad: web if you redeploy and the file names don't change they have different versions right so,
Arpad: don't know if you are using the right source map
Arpad: or maybe it's an outdated source map so you're applying the wrong debug info
Arpad: and you get the wrong results so especially in javascript the process of actually
Arpad: finding the correct source map for a javascript file is the most challenging thing this.
Matthias: This source map analysis and the demangling part were completely separate Were they separate code bases?
Matthias: Did you later integrate SourceMap support? How did that process work?
Arpad: Symbolicator
Arpad: It's not completely separate. So it's all built into Symbolicator.
Arpad: And Symbolicator internally has lots of shared code.
Arpad: Things like we do caching, very sophisticated caching on different kinds of layers. layers.
Arpad: And the JavaScript and source map processing definitely took lots of these things and extended on them.
Arpad: And afterwards, we also tried splitting up Symbolicator into a workspace with
Arpad: different crates as well.
Arpad: So we have a crate for native processing, we have a crate for source map processing,
Arpad: and then we have some other shared codes that's used by all of those.
Matthias: And where do you deploy the code? What cloud provider do you use?
Arpad: We are using Google, and we have a whole team responsible for all of these operations
Arpad: topics, and they use Kubernetes under the hoods.
Arpad: And for me personally, I sometimes take a look at all of these Kubernetes definition
Arpad: files, and I'm quite overwhelmed.
Arpad: I try to do some modifications here and there, but oftentimes I don't know what I'm doing.
Matthias: It's funny that you mentioned that because it feels like rust is easier for
Matthias: you to grasp or it feels more natural to you than yes that domain,
Matthias: and all of these parts can be scaled horizontally let's say you had a lot of
Matthias: traffic probably there was some way to scale it up to to handle the traffic right Mhm.
Arpad: Absolutely. So we do have horizontal scalability. So we have load balancers
Arpad: and behind the load balancers, there's a couple of servers that are deployed
Arpad: with a Symbolicator Rust code.
Arpad: And the load balancers just take care of distributing this to all of the nodes running the Rust code.
Matthias: Now the other project that you mentioned is relay and can you quickly talk about
Matthias: when it comes to the architecture
Matthias: the differences between relay and the
Matthias: of those two applications.
Arpad: So relay is really the first entry point when an event reaches
Arpad: so a
Arpad: client application communication panics, it sends a report, and it goes to Relay first.
Arpad: Relay handles a lot, a lot, a lot of traffic.
Arpad: And Relay does a couple of things. First, it actually checks that is your DSN correct.
Arpad: And just as a side note, DSN is kind of basically your token with which you send events to Sentry.
Arpad: So Relay validates that there is actually a project that exists.
Arpad: It validates that the project has quota to
Arpad: actually ingest those events relay parses the
Arpad: initial json payloads or depending on
Arpad: the architecture it's json or for native oftentimes it's also mini dumps so
Arpad: relay validates them that they have the correct format it rejects stuff that's
Arpad: invalid it does things like like spike protection all these kinds of things things.
Arpad: So Relay definitely has to handle a ton more traffic than Symbolicator does.
Arpad: So Relay is the big filter in front of the whole rest of the Sentry pipeline.
Matthias: And since it has to handle all of that traffic and it feels like it's rather
Matthias: monolithic correct me if i'm wrong here does that mean it runs on a big box
Matthias: and you have one or two instances in front like a proxy or is it also horizontally
Matthias: scalable somehow.
Arpad: It is also horizontally scalable but it's also vertically scalable so there is
Arpad: different data centers. And the thing that Sentry has invested a lot of energy
Arpad: recently was to open a new data center location in the EU.
Arpad: I'm not quite sure if it's available yet for customers, but it's definitely on the horizon.
Arpad: And these are called points of presence. We have relays running horizontally scaled there.
Arpad: And they do like what I talked about, this initial filtering.
Arpad: And then they forward stuff to other relays, which then do a little bit more
Arpad: processing of the event.
Arpad: And then they forward things into Kafka, which is kind of the source of truth
Arpad: when it comes to ingesting customer traffic or customer events.
Matthias: And these other relays, are they the same application, just different configurations?
Arpad: Exactly. Yeah. So it's the same application.
Arpad: It can run either in proxy mode or
Arpad: in processing mode and in
Arpad: proxy mode it just checks spike protection it checks
Arpad: that the token is valid that customer has
Arpad: quota and if those things match it's forwarded and otherwise it's rejected so
Arpad: that's just a relay in proxy mode and relay in in i think processing mode does
Arpad: more things like it it might split up certain events into different parts.
Arpad: It splits up Sentry envelopes into different payloads.
Arpad: And as a short explanation here, Sentry envelopes are things or is a custom
Arpad: data format to group different things together.
Arpad: So an envelope can have some metadata for the event itself.
Arpad: It can have attachments. For example, you tell the SDK to,
Arpad: hey, please attach this log file every time a panic happens,
Arpad: then that's a different payload and relay takes this whole envelope that contains
Arpad: everything and then it splits it up into the different components it validates
Arpad: those it applies like quota and and all of these kinds of things and then throws them into kafka okay.
Matthias: If you if you allow me this interruption. Early on you also mentioned mini dumps
Matthias: what are these because i've never heard that term either.
Arpad: Mini dumps are a way to ingest native crashes.
Arpad: And I believe it has been pioneered by Windows or by Microsoft because it's part of the Windows API.
Arpad: But there's also been some open source projects to generate the same format also on Linux and on Mac.
Arpad: And mini dumps basically contain the CPU context for all of the threads that
Arpad: are running. plus maybe memory regions for all of the stack memory.
Arpad: Maybe if you want, you can have other memory regions and also loaded libraries
Arpad: that are being loaded into the process.
Arpad: All of this is contained within this mini dump.
Arpad: And the processing pipeline can then take all of this information,
Arpad: the list of loaded libraries, which contain, by the way, the unique IDs we talked about previously.
Arpad: And the CPU context, and the stack memory.
Arpad: And from there, we actually extract the stack trace on our processing end.
Arpad: So that's different from a panic where, for example,
Arpad: the Rust itself or then the Rust SDK already has a stack trace at runtime and
Arpad: then just puts it into JSON payload to send it to Sentry. So that's the big difference.
Matthias: Okay a bit like a superset of a stack
Matthias: trace with much more context and you
Matthias: kind of have wrappers around the application or
Matthias: business logic or maybe the the error that happened and you process it at various
Matthias: stages you have the stack traces mini domes and then you have envelopes and
Matthias: at some point you need to go through each of these layers in order to route
Matthias: and process the errors that you encounter.
Arpad: Mm-hmm.
Matthias: In layman's terms. Yeah. Now comparing the architecture of the Symbolicator
Matthias: and Relay, are there any differences?
Matthias: Do you also use tokio in Relay? Do you use something else? What are the differences?
Arpad: Like Symbolicator, Relay has also made this change from Actix to tokio and also
Arpad: changing the front-end HTTP server to Axum right now.
Arpad: And also the client HTTP server that sends things further to request.
Arpad: So this change has also been done in Relay.
Arpad: But one difference, I believe, is that Relay still uses the actor model in some parts of its code base.
Arpad: And the actor model has then been appended with some niceties around it.
Arpad: It looks and feels like it's just an async function, but internally it's actually
Arpad: sending messages to actors and receiving replies and all these kinds of things one.
Matthias: Thing i heard about actor models
Matthias: in parallel processing which
Matthias: some people say you need to be cautious of
Matthias: is the queue or the message handling so
Matthias: every actor has an input queue or
Matthias: some sort of inbox and this
Matthias: could grow and depending on your settings it could either grow indefinitely
Matthias: or it could reach a limit where it stops to accept messages anymore did you
Matthias: ever run into such issues with relay or is there anything that you say you need
Matthias: to be cautious of when you use the actor model in your context.
Arpad: I Believe one of the big reasons that relay still still uses the actor model is
Arpad: exactly this point that you do have these inboxes and queues.
Arpad: But in a positive sense, this also gives you the possibility to apply back pressure
Arpad: and to react to back pressure a little bit better as if you just had async code.
Arpad: So I believe that is one of the main reasons Relay is still using this internally.
Arpad: But otherwise i'm not sure i can i can really answer this question.
Matthias: How would you handle back pressure with tokio?
Arpad: Ah, that's a good question i believe you gotta have some kind of queue or semaphore
Arpad: or something somewhere at.
Matthias: At the end you build your own inbox of some sort and you need to.
Arpad: Yeah basically.
Matthias: Yeah. I know that from previous projects, there were a lot of queues in such systems.
Matthias: And of course, there were also a lot of constraints and definitions of these queues.
Matthias: So how long they can be and what sort of traffic they handle and so on.
Matthias: It's a very, very tough problem, I guess. Interesting problem space, though.
Matthias: Now with regards to that sentry handles a lot of traffic and massive volume
Matthias: of events and transactions daily how would you say from your perspective has
Matthias: rust contributed to handling this
Matthias: high throughput and maybe also what performance gains have you observed.
Arpad: I would say Rust actually enables us to handle such volumes of traffic in lots of cases.
Arpad: So the whole processing pipeline that we talked extensively about that I maintain,
Arpad: it doesn't handle as much traffic.
Arpad: It's just about 2000, 3000 events per second.
Arpad: But here, actually processing them is the hard part or the expensive part in this case.
Arpad: Other parts of the pipeline they handle a lot lot more and i'm actually not
Arpad: quite sure about the exact number of total events that we get also considering
Arpad: things that are just rate limited and thrown away immediately versus things
Arpad: that actually get ingested into sentry but it,
Arpad: having this rust code and having this high performance is actually what enables
Arpad: us to to build new products that handle a lot more traffic.
Arpad: And maybe I've hinted on this previously that Sentry is also working on a metrics product.
Arpad: And metrics especially wouldn't be possible without having Rust go in between
Arpad: to handle the amounts of traffic.
Arpad: And to give you some numbers as well, there's been a couple of initiatives internally
Arpad: internally to rewrite some parts of the whole pipeline.
Arpad: The whole pipeline is enormous.
Arpad: It has lots of moving parts.
Arpad: And there's tons of Python code still there.
Arpad: And we try to move little bits of this to Rust, things that are especially critical
Arpad: or things that are costing a lot in terms of cloud costs.
Arpad: And here, one team has rewritten one critical part of the infrastructure in Rust.
Arpad: And they've observed 10 to 20x improvement in throughput and eventually in cloud cost as well.
Arpad: Because you can then just scale down the number of horizontally scaled workers
Arpad: and the CPU and memory allocations they get.
Arpad: And that has got 10 to 20x improvements here.
Matthias: And what is the amount of traffic that this new service gets?
Arpad: Depending on the workloads, there is one specific use case where it has more
Arpad: than 100k Kafka messages in this case per second.
Matthias: So that's massive.
Arpad: Yeah. Those are being handled by Rust code. They are being batched up and then processed further.
Matthias: Pretty impressive. What are,
Matthias: in your opinion, some of the more unique challenges that you faced when integrating
Matthias: Rust with other languages, which is given that Sentry has a very homogenous
Matthias: stack, but also multiple languages in the stack, like Python as well.
Matthias: What are some things that you encountered that maybe are very specific to your problem domain here?
Arpad: It's very interesting. So Sentry itself is a really big Python application and
Arpad: we are interfacing Python and Rust on different layers.
Arpad: So we have some extension modules written in Rust that are directly called from Python.
Arpad: We also have things like Symbolicator, for example, which expose an HTTP interface.
Arpad: And the HTTP interface is a completely different topic.
Arpad: It could be something else, but it's a completely different service,
Arpad: which is scaled differently, which is stateful in the sense that it's long running
Arpad: and it has some internal caching that is long lived.
Arpad: But the other way we interface to Python is via extension modules.
Arpad: So you write an extension module with Rust.
Arpad: And Sentry has been doing this for a long time already.
Arpad: And just recently, one of the things I did specifically was to pioneer PyO3.
Arpad: So we created a new Python package written using PyO3 and Maturin.
Arpad: And I'm actually really, really satisfied with how this turned out to look eventually.
Arpad: There were some really arcane ways of how to write Python extension modules
Arpad: previously where you had to use bindgen to generate C headers and then from there use,
Arpad: I don't know what the Python module is that actually consumes C headers and
Arpad: C FFI bindings and then just have tons more Python glue code on top of it.
Arpad: And PyO3 just does all of this for you.
Arpad: And that's really liberating. And I believe doing this one project now and actually
Arpad: these last couple of weeks, we've demonstrated that this actually works and
Arpad: it works really well and it's productive.
Arpad: And you can use this to write even more code in Rust that's been used from Python directly.
Matthias: And can you give me an example for what such a Python extension might do?
Matthias: What such a Python module might do for you for Sentry?
Arpad: There's different things we use this for. Things we use existing Python extension
Arpad: modules right now is for validating native debug files.
Arpad: So we talked extensively about debug files that they need to have a unique ID
Arpad: and all these kinds of things.
Arpad: And when customers upload them to Sentry,
Arpad: they are validated from within
Arpad: python code that calls rust
Arpad: code to do this validation and there's other parts of the processing pipeline
Arpad: that are not moved into symbolicator as like its own stateful service written
Arpad: in rust but which are still written in rust but called from python So,
Arpad: an example here is ProGuard processing.
Arpad: And for people who don't know, ProGuard processing is basically a similar de-mangling
Arpad: or de-obfuscation that happens for Java and for Android.
Arpad: So Java code is being obfuscated in a certain way and you have a ProGuard file
Arpad: which acts acts as some form of debug information where you can turn this back
Arpad: into readable function names and stack traces.
Arpad: And handling these ProGuard files and parsing this ProGuard format,
Arpad: and it's a text-based format and we do this in Rust, but the way we drive it is still via Python.
Arpad: So Python says, please parse this file, please de-obfuscate this function name.
Arpad: But the actual logic to do so, that's written in Rust.
Matthias: Yeah, and you get that input from users and it could be whatever, it could be anything.
Matthias: And I would imagine that Rust helps you treat such unsafe input like untyped
Matthias: JSON as something that maybe...
Matthias: Will turn out to be safe from the python ecosystem from the python's perspective is that correct.
Arpad: Yes yes so one thing that rust definitely helps with is dealing with untrusted user inputs.
Matthias: Yeah and with regards to that given that the rust ecosystem is still maturing
Matthias: and And it's probably still a little rough around the edges for very specific use cases.
Matthias: Have you encountered any gaps or missing features that you had to work around
Matthias: or contribute back to the community?
Arpad: In Rust specifically, what kind of missing features?
Arpad: There's different things I would say. One part, as you asked specifically about
Arpad: parsing things and handling files,
Arpad: one thing I miss from Rust as it exists today is self-referential data types.
Arpad: And we have some workarounds for this.
Arpad: So you can write unsafe codes to have like a wrapper that actually owns a buffer,
Arpad: but also holds a type that references this buffer. Right.
Arpad: And it's an owns type. It has a static lifetime on the outside,
Arpad: even though it has self references on the inside. Right.
Arpad: But for this, you still have to do some arcane trickery and unsafe and i would
Arpad: love this to be eventually possible and i hope it will be someday,
Arpad: the other thing from the rust ecosystem point of view more on the side of sentry as,
Arpad: product would be the whole thing about rust error handling and stack traces
Arpad: in particular particular.
Matthias: Isn't that one of the big advantages of rust?
Arpad: Um it is it is but it's also it's both an advantage and also disadvantage so
Arpad: it's it's an advantage in a way that it's very explicit,
Arpad: and i believe also this this split between panics that in theory you can catch
Arpad: but in practice you you don't versus results and error as a type that you return
Arpad: and you have this question mark operator that just propagates this.
Arpad: So from users perspective, this is really, really nice.
Arpad: It's a lot better to reason about than exceptions, where you never know which types you get.
Arpad: So Rust makes it extremely clear that if you have a result, you know which types
Arpad: to expect, even though in reality, no one really matches on the error type.
Arpad: The cases in which you really, really match on the error type are extremely rare.
Arpad: Oftentimes you just question mark and propagate the results.
Arpad: And from Sentry as a product perspective, what I would like here is really better
Arpad: integration for stack traces.
Arpad: And this is happening. There is an initiative in the Rust community to bring stack traces to errors.
Arpad: It's still very experimental and there's a lot of progress that still needs to be made.
Arpad: But one issue with this still, even if we had a way to have stack traces from
Arpad: Rust errors, it's a problem that where do you capture the stack trace, right? Right.
Arpad: So Rust is a language which makes it extremely clear what the costs of certain operations are.
Arpad: Right. So you do know what the cost of certain operations is and capturing a
Arpad: stack trace is quite expensive.
Arpad: Yeah so you you do not want to capture a stack trace and return it with every
Arpad: result in a tight loop.
Matthias: Because you would have to allocate things on the heap and that could potentially be
Matthias: expensive or almost impossible on some platforms.
Arpad: Yeah exactly so allocations for example is one thing but also capturing the
Arpad: stack trace is quite an expensive process.
Matthias: Yes.
Arpad: But it has advantages and disadvantages. The advantage is that Rust results or errors,
Arpad: for example, if you're using this error crate to just define an enum that has
Arpad: a bunch of variants, it's easy to do.
Arpad: And oftentimes it's quite cheap in the sense that it's just a couple of bytes.
Arpad: Those things can balloon quite a bit if you wrap an error in an error in an
Arpad: error but the problem with stack traces in particular is that.
Arpad: The location where you initially
Arpad: return an error versus the
Arpad: location where you capture it or where you
Arpad: attach a stack trace to it might be like
Arpad: completely different right from developers
Arpad: perspective i want to know what is
Arpad: the original location where an error
Arpad: was first returned but oftentimes
Arpad: i only get this wherever
Arpad: i unwrap or wherever i capture it
Arpad: in sentry or if you use anyhow with the backtrace feature whenever you convert
Arpad: a more detailed error type into an anyhow error and anyhow attaches a stack
Arpad: trace to it that's kind of the code location you get and oftentimes it's not
Arpad: detailed enough right that.
Matthias: Would be one area where you or
Matthias: more specifically would want to see some
Matthias: improvements maybe in the standard library or maybe also outside the standard
Matthias: library in some some crates.
Matthias: How would you try to improve the situation here?
Arpad: Mh-mhh.
Arpad: I would say anyhow, as a crate, or there's also alternatives to it,
Arpad: of course, but I'm just using it as one example,
Arpad: is already very good because it's quite a generic error type that you can throw
Arpad: or return from anywhere.
Arpad: And depending on feature flags, it does have a stack trace on it. So that's very good.
Arpad: So some ecosystem support here exists. The big question is,
Arpad: basically the question of anyhow versus this error.
Arpad: Do you want to have like an opaque error type that can be anything that,
Arpad: by the way, is also cheap in the sense of it's just a pointer to some heap allocated
Arpad: thing versus do I want to have a really detailed error that I can match on,
Arpad: but which might be an enum with sub-enums and which balloons to 200 bytes error
Arpad: case that also has some other costs.
Arpad: So there is this difference between these two approaches.
Arpad: And I'm not quite sure what the best way here is.
Arpad: I believe in the Rust community, it's often said that if you're writing a crate
Arpad: that you publish to crates.io, oh, you better have some detailed error enums.
Arpad: But on the other hand, if you have an application and you never match on those
Arpad: enums anyway, or just in like, I don't know, tiny percentage of cases,
Arpad: then you better use Anyhow for this.
Arpad: So there's these two walls and they have pros and cons.
Matthias: I agree. One other strategy that I heard about was that if you build a library
Matthias: that other people depend on, you might want to have an explicit error type with this error.
Matthias: Whereas if you build an application, you can use anyhow, as you explained.
Matthias: I mean, this is more or less anyway what you said, but I think it's a reasonable guideline.
Matthias: Line but certainly it's great to have the choice
Matthias: and to say oh yeah i need more control over
Matthias: my errors or no in fact i don't really
Matthias: care i want to have the option to handle
Matthias: this error if i want to in the future i can search for
Matthias: it i can grab for it but i don't have to do it right away and
Matthias: kind of deferring this error handling
Matthias: mechanism and maybe making a decision in
Matthias: the future is also really great that I miss from
Matthias: other languages where things happen and
Matthias: then an exception gets thrown and maybe it's
Matthias: not exactly apparent from the code whereas in Rust
Matthias: it's almost like a stop sign that makes you think about the code for a second
Matthias: even if you decide not to handle it at this point and with that I wonder if
Matthias: you have any final message to the Rust community anything that you want to mention,
Matthias: things that you say would be important going forward with Rust 2024.
Matthias: The Rust 2024 edition or anything else in the ecosystem? The stage is yours.
Arpad: So I'm definitely very excited about Rust 2024.
Arpad: There is a bunch of proposals here. There's lots of compiler internal work.
Arpad: There's work to teach the borrow checker new things with Polonius,
Arpad: which finally hopefully gets into the compiler, and I hope it makes it to Rust 2024.
Arpad: Other things here are a rewrite of the trait system, which might be just details
Arpad: in the compiler itself, but I'm also very excited about it.
Arpad: And all of these things hopefully enable enable support for self-referential
Arpad: data types, which I mentioned before.
Arpad: Right now you can't do it, but you need arcane unsafe code to do it.
Arpad: And I hope this might be built in at some point in the language.
Arpad: Another thing that's slowly being built and I'm following along the commits
Arpad: to the compiler every now and then.
Arpad: And one thing I notice is being worked on actively right now is support for
Arpad: first-class generator syntax.
Arpad: So things like writing an iterator, but with generator syntax.
Arpad: So that's also definitely something I'm looking forward to.
Matthias: Yes, and the generator syntax is also very close to my heart, coming from Python.
Matthias: I really, really use a lot of generators. And yes, it is something that I also
Matthias: would like to see in Rust.
Matthias: And it's always fun to have something to look forward to in this language.
Matthias: Which it's pretty exciting.
Matthias: Yeah, I hope that Rust 2024 will happen.
Matthias: And if so, I will upgrade immediately.
Arpad: I mean, it will definitely happen. The question is what's included.
Matthias: Yeah, yeah.
Arpad: But I'm definitely hopeful. And for other companies trying to adopt Rust,
Arpad: I would say just try it, give it a shot.
Arpad: But take a project, a part of the code base, which is quite isolated,
Arpad: which you can just experiment writing Rust in it.
Arpad: And Rust, by the way, as we talked about, we use it from Python.
Arpad: And from Python, I can really recommend Py03. It's a game changer from the ways we did it before.
Arpad: So it's actually very easy to use Rust from within other languages.
Arpad: Languages and i believe there's also PyO3 equivalent in the javascript and
Arpad: node ecosystem as well i'm not quite sure what it is exactly.
Matthias: It's called neon yeah or maybe there might be multiple but the one that i know
Matthias: is called neon and i don't know if it is used for any,
Matthias: bigger project but it exists yes there it's it's one of the great advantages
Matthias: of rust that we have have such nice ways to write wrappers for other languages.
Arpad: Yeah.
Matthias: And with that, we're getting very close to the end.
Matthias: Arpad, it was an honor to talk to you today to learn more about Sentry internals,
Matthias: monitoring, error handling, observability.
Matthias: I think if anyone out there wants to give Sentry a try or wants to integrate
Matthias: it into their Rust application, where can they go?
Arpad: They can just head to sentry.io or to docs.rs and take a look at the Sentry SDK.
Arpad: Both are fine, I would say.
Arpad: And especially in Rust, we have support for just capturing panics,
Arpad: which I believe is the most important thing.
Arpad: But as I mentioned previously, we also have integration for tracing.
Arpad: So if you do tracing error, the tracing error macro, or if you just instrument
Arpad: things with tracing instruments, we capture those.
Arpad: We give you a great overview over traces and we also have support for distributed tracing.
Arpad: So if you want to integrate a Rust backend with JavaScript frontend,
Arpad: that's covered end-to-end with the various, various language ecosystem SDKs that we have.
Arpad: So I would say just give it a shot.
Matthias: You heard it there. Head over to Sentry check it out. Arpad, thanks again for your
Matthias: time and see you next time yeah.
Arpad: Thanks for having me. Bye bye.
Matthias: Ciao Rust in Production is a podcast by corrode and hosted by me, Matthias Endler. For show notes,
Matthias: transcripts and to learn more about how I can help your company make the most
Matthias: of Rust, visit corrode.dev. Thanks for listening to Rust in Production.