Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we are talking to Micah Wylde
Matthias: from Arroyo about how they simplified stream processing for data engineers with Rust.
Matthias: Micah, welcome to the show. Can you tell us a few words about yourself and Arroyo,
Matthias: the company you founded?
Micah: Thanks so much for having me. Yeah. So I am a Rust engineer and the creator
Micah: of the Arroyo streaming engine.
Micah: So Arroyo is a real-time data processing engine that allows you to write SQL
Micah: queries with Rust user-defined functions on top of streaming data.
Micah: For example, data you might might have in Kafka or another streaming system.
Micah: And I come to that problem and company after spending five years leading streaming
Micah: teams at companies like Splunk and Lyft, which is a rideshare company in the US.
Micah: And more broadly, I've been in the big data space working on data systems for
Micah: pretty much my entire career starting out in ad tech, working on real-time ad bidding systems,
Micah: and then leading data teams and building data systems so yeah that's that's
Micah: a brief background about me at.
Matthias: Splunk you were a principal engineer and you were the team lead of the streaming
Matthias: compute team, so that makes you somewhat of an expert in stream processing I would say.
Matthias: Maybe for the uninitiated, could you give us a just very quick,
Matthias: very brief introduction of what stream processing is in your own words?
Micah: Yeah. So traditionally, when people have wanted to process data,
Micah: we do it in what's called batch mode, which means you take all the data in through
Micah: whatever data sources those are, whether it's coming from logs you're reading,
Micah: from API requests that are ending up somewhere, or wherever that data is coming,
Micah: it all kind of filters through your system and eventually lands in traditionally
Micah: a database or today maybe like a data lake or a data warehouse.
Micah: And then once all that data is there, you run a really big data processing job
Micah: on top of all of that rest, that data at rest.
Micah: Often this means you wait, you know, an hour or a day for all the data to land
Micah: before you can kind of analyze it or learn anything about it.
Micah: Stream processing, in contrast, does this data processing as the data actually
Micah: arrives in your system. So in real time.
Micah: And the advantages there, obviously, latency is much better.
Micah: You can process the data within milliseconds or seconds instead of waiting hours or days.
Micah: But it also can give you a much kind of easier way to build these like end-to-end
Micah: data systems where you need to consider like different properties around like
Micah: timeliness and completeness in order to kind of build your higher level analytics or data products.
Micah: And for kind of real-time companies like at Lyft, this becomes really crucial
Micah: to be able to basically know things about your world really quickly.
Micah: In rideshare, you need to understand kind of where your users are, where your drivers are.
Micah: You need to understand traffic speeds in order to do routing.
Micah: You need to be able to do dynamic pricing based on supply and demand.
Micah: And all of this stuff really demands that you be able to do complex analysis
Micah: on data really quickly instead of waiting, you know, a day for it all to land in your data warehouse.
Micah: So that's kind of like a high level of the problem stream processing and solving or kind of how it fits.
Matthias: Yeah, so stream processing has existed before.
Matthias: There were other companies that did a lot of groundwork. You mentioned at some
Matthias: point Hadoop, you mentioned BigQuery in your seminal article that we will get to in a second.
Matthias: But I think maybe you can just quickly explain what makes Arroyo special in
Matthias: this case and also what the competitors are lacking right now that maybe is
Matthias: a nice niche for Arroyo. Yeah.
Micah: So, well, so BigQuery and Hadoop are both kind of like in that batch paradigm,
Micah: where you let all the data collect, and then you do a big data processing job over that data at rest.
Micah: In the streaming world, traditionally, the most popular system has been one called Apache Flink.
Micah: This is about a decade old
Micah: but was really the first system that found a good programming model for streaming
Micah: and really i would say made it work at a level of correctness that that allowed
Micah: it to be applied to a lot of these problems before flank we really had very
Micah: simple systems that were,
Micah: couldn't couldn't really guarantee anything about correctness or completeness
Micah: and we're sort of just orchestration systems around your own logic.
Micah: So I spent my year, my career in streaming working on Flink.
Micah: And I think that's true of most of the other people who are kind of doing new things now.
Micah: And for all of us, we kind of have this perspective on Flink that it solved
Micah: this problem really well for people who are able to invest a ton of energy into
Micah: becoming experts in Flink.
Micah: So, at the companies I've worked for, that meant staffing up teams that were 10 to 30 people,
Micah: full of people working on Flink, building infrastructure and tooling around
Micah: it, and then especially supporting end users who were actually building these streaming pipelines.
Micah: And I think while Flink was really successful allowing sophisticated companies
Micah: to roll out this technology in a way that would have been dramatically harder a few years earlier,
Micah: it never really got to that point of ease of use where you could hand Flink off to a data scientist,
Micah: to a data engineer, or a product engineer, and allow them to be successful building
Micah: these real-time pipelines on their own.
Micah: We always needed a lot of hands-on support from the Flink experts of the company.
Micah: And that's really what we're trying to innovate around in Arroyo.
Micah: We're trying to build a system that is easy enough for any engineer or data
Micah: scientist at your company to kind of pick up and build these correct,
Micah: reliable, performant real-time data pipelines.
Matthias: So how do you see the relationship between stream processing on one end and
Matthias: these new workflow engines that pop up nowadays like Windmill,
Matthias: which is coincidentally also written in Rust? Do you see an overlap?
Matthias: Do you see the industry converge to something that maybe encompasses both?
Matthias: Or would you say these are fundamentally different areas of expertise?
Micah: Yeah, I think they're very different systems and they are good at different
Micah: kinds of problems. problems.
Micah: So workflow engines are really excellent at these very long-running tasks.
Micah: We have a bunch of things we need to do based on fairly simple criteria over the course of a day.
Micah: For example, a user signs up, we need to send them this email.
Micah: Depending on what they do in response to that, we need to do this other sequence of events.
Micah: And that's the sort of think actually streaming engines like Flink or Arroyo are pretty bad at.
Micah: It's hard to specify that type of logic, that kind of conditional logic over
Micah: all of these different states.
Micah: And they also architecturally are kind of way overpowered to do that kind of stuff.
Micah: I think these systems actually work together quite well because streaming systems,
Micah: stream processors are really good at data oriented problems.
Micah: So often, this will mean you put your like really big feed of data,
Micah: your millions of events per second feed into your streaming engine.
Micah: And that produces features or
Micah: events that can then be consumed by the much lower scale workflow system.
Micah: So that that's actually a pretty common pattern for these to kind of work together.
Micah: But at least in the near future, I don't see them as being kind of in the same space at all.
Matthias: Mm-hmm on your website you have a very
Matthias: nice example where you take a Kafka stream and then you write some i think it's SQL
Matthias: might be Apache Arrow or some other syntax but it's similar to sql to pipe
Matthias: events through your system and then see the results in real time and this was
Matthias: pretty pretty impressive demo.
Matthias: So is Apache Arrow SQL-like, or is it more than that?
Matthias: Is it different? If so, in what sense?
Micah: So the main way you program Arroyo is through SQL.
Micah: We have slightly our own dialect, but we aim to be pretty Postgres compatible.
Micah: To do real-time SQL, you do need to extend it in some way. There's different approaches for this.
Micah: But SQL SQL, as originally defined, was really designed for these batch computations
Micah: to do like a group by or an aggregate or a join.
Micah: You really need all of the data to be available. Otherwise, how do you know,
Micah: you know, in a join, there might be more data coming in on one side or the other
Micah: in the future, so you can't ever return that result. result.
Micah: So different streaming systems that use SQL have come up with different answers
Micah: for basically how can we decide that we're done, we're able to return a result for these expressions.
Micah: In Arroyo, that looks like.
Micah: We introduce these time-oriented window functions, like a tumbling window and
Micah: a sliding window and a session window.
Micah: And these rely on a notion of what's called watermarking, which is this concept
Micah: of basically estimated completeness.
Micah: Watermark is a special value that flows through the data flow of the pipeline
Micah: and tells all of the operators that we have seen all of the data,
Micah: or we believe we've seen all the data from before a certain time.
Micah: And that tells us if we have like a window that closes at time t and we get
Micah: a watermark that is after t, it tells us that we can close that window,
Micah: that we've seen all the data that will be in that window and we're able to process
Micah: it and return the results to the user.
Micah: So this is a common pattern in certain types of string processors like Flink and Arroyo.
Micah: There's other approaches to this, which Arroyo and other systems like Materialize
Micah: also support, which is based around a more incremental style of computation,
Micah: where we actually decide we're never going to be done.
Micah: We never know that we have all the data for a particular time period.
Micah: So every time an event comes in, we're going to update the state of that window
Micah: and emit the new result there.
Micah: So depending on the kind of problem, you may kind of want one style of SQL or the other style of SQL.
Micah: But yeah in any case it's all it's all sequel you.
Matthias: Wrote an article called rust is the best language for data infra which is kind
Matthias: of a catchy title and i read the article and and one thing i wondered about was okay.
Matthias: Was Rust your first choice when you started? Have you looked into,
Matthias: for example, the solutions that came before you?
Matthias: And also, was it around the time where Zig also became popular?
Matthias: And where do you see yourself in this space?
Matthias: Would you say, okay, Rust was just there at the right point in time?
Matthias: Or would you also say, well, there would be alternative realities,
Matthias: so to say, where Arroyo was written in C++ or maybe Zig in a different world.
Micah: Yeah, so I mean, kind of setting this historically, the very earliest systems
Micah: of this kind of in this space,
Micah: like the original Google systems that establish a lot of how we think about
Micah: big data today, like MapReduce and BigTable and GFS,
Micah: those were all written in C++.
Micah: And then we had a long history of writing systems in Java, like Hadoop and HBase.
Micah: And Flink itself is written in originally Scala, and then rewritten in Java.
Micah: And then we had a whole period of doing Go, like CockroachDB,
Micah: and a handful of other big data systems.
Micah: And yeah, I think now, definitely, we would not have chosen Java or Go for Arroyo.
Micah: I think in many ways, the current era of systems is a reaction to the previous
Micah: era of writing these systems in Java.
Micah: A lot of people are finding that you can get much better performance,
Micah: much easier operations, literally just by rewriting these systems in a non-managed
Micah: language like C++ or Rust.
Micah: So we're kind of following in the footsteps of projects like Red Panda,
Micah: which did this with Kafka, and ScyllaDB that did this with Cassandra.
Micah: So I think we could have done some of the things we're trying to do in Java
Micah: or Go, but it would have been much harder to accomplish our goals.
Micah: So in a world without Rust, I think we probably would have ended up choosing C++.
Micah: But I'm very grateful that we are in a world with Rust.
Micah: It has definitely made our lives
Micah: much easier than it would have been if we had to choose C++ for this.
Matthias: Especially i assume to optimize
Matthias: the platform you would have to avoid a
Matthias: lot of copies and in c++ passing references
Matthias: around can be a bit of a nightmare sometimes
Matthias: if you don't know exactly what you're doing and even if you do there can be
Matthias: issues with that and i just wondered do you have a lot of lifetimes in your
Matthias: code as well or is that something that the rust compiler completely completely
Matthias: elides and you don't even have to think about lifetimes at all.
Micah: So the most memory-oriented or lifetime-oriented part of our system is the storage layer.
Micah: So maybe to give a little bit of architectural insight here,
Micah: the way these systems look, they are these directed acyclic graphs of data flow.
Micah: Take a SQL statement, you compile it into a SQL plan, and then eventually optimize
Micah: that into this data flow graph.
Micah: Each node of this graph is some kind of potentially stateful operator.
Micah: So for example, doing a filter or a map or a stateful function like a window or a join.
Micah: And between these operators, the events and process data flow over queues or over network sockets.
Micah: So within these stateful operators, we potentially have to store data for long periods of time.
Micah: So if you imagine you have like a 30-day sliding window, we need to store some
Micah: representation of that data for 30 days.
Micah: And we do that in a mix of offline S3 storage and local disk cache and then in-memory cache.
Micah: And managing that in-memory cache brings into issue these lifetime concerns,
Micah: managing the data as it flows from that cache into the processor in order to be used.
Micah: Fortunately, in these systems, the architecture constrains that problem somewhat.
Micah: So at the semantic layer, you're kind of processing one event at a time in each of these operators.
Micah: So you don't have to deal with really concurrency issues at the direct processing layer. layer.
Micah: And that ends up simplifying the kind of lifetime management that you might
Micah: have in a more traditional database where you're kind of dealing with a bunch
Micah: of different requests to the same data.
Matthias: So in Rust-lingua, that would be your types are not sync, or they don't have to be?
Micah: That's correct. Yeah, we're always accessing a particular...
Micah: You can think logically, each of these operators is single-threaded.
Micah: This is all implemented in tokio, so what's happening under the hood is much
Micah: more complicated than that.
Micah: But as a programmer, you can really think of it as synchronous processing, not a single thread. red.
Matthias: Speaking of tokio it feels like
Matthias: this is an ideal use case for it because
Matthias: you're kind of leaning into things that are inherently concurrent
Matthias: they don't really have to be sequential they can at least parts of them can
Matthias: be executed concurrently sometimes maybe even in parallel but i wonder what
Matthias: you think about tokio your experiences with the framework the ergonomics of it,
Matthias: and also the recent discussion about async-rust being a send,
Matthias: sync, and so on, and work-stealing,
Matthias: schedulers, all of that stuff.
Micah: Yeah, so at a high level, a system like Arroyo doesn't really need a complex scheduler like tokio.
Micah: As I mentioned, each of these operators essentially acts as a single thread.
Micah: It receives one event, it does all the processing for that event,
Micah: and then it sends it on to to its next destination.
Micah: And all this has to happen in order to uphold the correctness guarantees of the system.
Micah: And because of that, the first version of Arroyo actually was built around threads and thread processing.
Micah: At some point, it migrated to tokio and AsyncRust, actually pretty early on.
Micah: And the core reason for that was that so much of the ecosystem is in AsyncRust
Micah: at this point, that if you want to use common network libraries or database
Micah: drivers or almost anything from kind of network programming ecosystem, ecosystem,
Micah: you do have to deal with async at some point.
Micah: And at some point, it's easier just to move your whole system over to async.
Micah: And that was definitely a challenging migration. Actually, for me,
Micah: I had never worked with async REST before.
Micah: So it involves a lot of learning, a lot of time on the tokio Discord channel,
Micah: which is extremely helpful.
Micah: But in the end, actually, the surprise was that it ended up being a lot faster.
Micah: Just purely doing that migration made the system like 30% faster,
Micah: which was not my expectation at all.
Micah: But it turns out that the tokio scheduler is really, really effective at this
Micah: class of problems, where even though it looks at a high level,
Micah: like all this processing is single-threaded,
Micah: there's a lot more going on under the hood, a lot more work that has to be coordinated.
Micah: Between like, you actually have threads, like in our case, like talking to S3
Micah: or talking to other systems.
Micah: We have a lot of queues involved.
Micah: So even though we have only a smallish number of actual processing threads,
Micah: there's a lot of network exchange happening on other threads, talking to...
Micah: Coordination system over gRPC. And tokio is really good at, at,
Micah: you know, organizing all of this work efficiently and really maxing out the use of your core.
Micah: I think that the most surprising thing for us is that we're able to run the
Micah: system at extremely high utilization,
Micah: like above 95% CPU utilization,
Micah: and everything remains responsive and reactive and is able to,
Micah: to really work effectively at that extremely high level of CPU thrash,
Micah: which has never been my experience with the systems written in other paradigms.
Micah: And then in terms of, I guess, kind of how I think about the async Rust,
Micah: I guess, drama, if we want to say that word, I think the Rust community has
Micah: a higher level of drama in general, and I don't fully understand why that is.
Micah: But I think maybe the technology just works so well that we can't,
Micah: you know we sort of have to invent other stuff to be upset about
Micah: but i will say async rust
Micah: definitely has a learning curve it took me like a
Micah: month coming from from being like i
Micah: would say a pretty strong rust programmer already it took
Micah: maybe a month to really be an effective async rust programmer and
Micah: it's definitely been the edge of the system that other people who contributed
Micah: to it have the most trouble with this the requirement that that values passing
Micah: over weight descend send definitely can be frustrating if you aren't experienced
Micah: in the strategies for dealing with that.
Micah: And the sometimes bad error messages in the compiler don't help with that either.
Micah: It can make it really hard to figure out where exactly that problem is introduced
Micah: in a large amount of code.
Micah: But I'd say overall, tokio has been a huge boon to us.
Micah: And it's really remarkable what, you know, what allows us to do in terms of
Micah: just not having to think very much about how we schedule work.
Micah: It just does a really remarkably good job on its own.
Matthias: Well, Rust protects you from memory or safety problems. It does not protect
Matthias: you from race conditions.
Matthias: So I wonder if you, as someone that uses Rust and tokio at scale,
Matthias: has run into any sort of data races or things that you encountered at runtime,
Matthias: which maybe were a bit of an issue for your platform, or did you never ever
Matthias: have any outages in production?
Micah: So not specifically from race conditions which
Micah: again like the architecture of the
Micah: this of our system makes the
Micah: high level concurrency pretty straightforward the although a lot more complexity
Micah: creeps in in the details especially when you try to get to the like that next
Micah: level of performance so for example like the the storage system is extremely
Micah: complex and has has a lot of concurrency.
Micah: But Rust does really help a lot with managing that complexity.
Micah: In terms of issues in production.
Micah: The issues we've seen are much more around the high level,
Micah: the ways that all the different pieces of this distributed system interact with
Micah: each other, and about wrong assumptions in different pieces about what other things are doing.
Micah: Which unfortunately rust definitely
Micah: does not fix distributed systems issues but
Micah: in terms of like the kind of micro level
Micah: it's remarkable how well things work once you get them to compile an example
Micah: i brought up in that blog post but still kind of blows my mind is that i wrote
Micah: the entire network stack like
Micah: the the piece of software that allows this system to to be distributed.
Micah: I wrote that in like a two-day push, basically like two 12-hour days.
Micah: And basically just coded that straight. And then at the end,
Micah: spent maybe an hour trying to get it to compile.
Micah: And from there, it just worked perfectly the very first time.
Micah: I made a single node system, a distributed system without any testing, any iteration on that.
Micah: And it basically hasn't changed since that initial implementation.
Micah: I've definitely never experienced that writing network software in C++ or even Java for that matter.
Matthias: That's pretty impressive, yes. Pretty awesome that you could pull that off.
Matthias: And it's a testament to the Rust type system, which is helpful,
Matthias: and also the borrow checker and all of the things that make Rust development
Matthias: and the developer ergonomics pretty awesome. Awesome.
Matthias: I wondered though, although maybe you didn't have that many runtime problems,
Matthias: I wondered if you had any compile time problems in a sense that maybe parts
Matthias: of the ecosystem were not aligned,
Matthias: like compatibility issues with, say, different versions of tokio or maybe different
Matthias: libraries that were sometimes more mature, sometimes less.
Micah: Yeah, it's never been a huge issue. And just the Rust-created ecosystem in general
Micah: has been such a boon to us from a productivity perspective, compared to the
Micah: C++ world where using dependencies is so challenging.
Micah: And you don't have this incredible, rich ecosystem that we already have in Rust
Micah: after such a relatively small amount of time.
Micah: So there are occasional compatibility issues we've had to fork a few open source
Micah: projects we rely on but i would say it's a very small part of my day dealing with any of that stuff.
Matthias: All of the things that you mentioned are at least to an average developer pretty
Matthias: low level or at least you need a lot of expertise on how to structure such systems
Matthias: or architect such such systems in order to perform well.
Matthias: And I wondered, what do you think, how much does Rust guide you towards an idiomatic
Matthias: solution? And what is your own expertise?
Micah: Yeah, I think Rust definitely guides you towards a correct solution.
Micah: I don't know, it always helps you that much with like being idiomatic.
Micah: Although the tooling around it is very helpful. Like Cargo Clippy is really
Micah: helpful and was very helpful.
Micah: So my co-founder had never used Rust before.
Micah: Working on this project. I really experienced distributed systems engineer and
Micah: has worked on a bunch of query systems, but was new to Rust.
Micah: And tools like Clippy really helped him pick up the idiomatic style of Rust pretty quickly.
Micah: Beyond that, I think the Rust community is also really helpful.
Micah: I mentioned already the tokio Discord, which was super useful when I was trying
Micah: to get up to speed with AsyncRust.
Micah: But in general, the Rust community is extremely useful in helping you solve
Micah: problems or figure out why some weird compile issue is happening.
Matthias: Did you use any resources outside of the official Rust book and maybe the community
Matthias: to help you get started with Rust?
Matthias: Or did you start on a project and learn on the job? So.
Micah: I've actually been using Rust since like 2014. I've never convinced a company
Micah: we should do like a major project in Rust until now.
Micah: It was always a big uphill battle trying to introduce Rust into a large organization.
Micah: But I've been using it for all of my kind of like personal projects for a really long time.
Micah: I've been a fan of the language since basically I first learned about it.
Micah: But so in terms of my own development, there's been a lot of resources like over that time.
Micah: The first version of the Rust book, which I have on my bookshelf back there, that was very helpful.
Micah: But also it just changed so much in the early days that it was a full-time job
Micah: just kind of keeping track of the updates to the language.
Micah: Today, it's much easier. It's been pretty stable for a number of years.
Micah: And I think the quality and quantity of resources has also increased a lot.
Micah: But I know there's a really good book actually on Rust in production that I've
Micah: looked at a fair bit for the more –,
Micah: kind of how do you actually run rust details like what what does logging look like in Rust what does,
Micah: how do we do metrics like these these kinds of things that aren't necessarily
Micah: like part of the intro to what book is that it's called Rust in Production.
Micah: h by Luca Palmieri.
Matthias: Awesome and you mentioned that it's a bit tricky sometimes to convince bigger
Matthias: companies and and organizations to move towards Rust and introduce Rust at these companies?
Matthias: Why is that in your experience?
Micah: Yeah, I think large companies tend not to be that ambitious in their technical choices.
Micah: A lot of it is built around minimizing risk rather than maximizing reward.
Micah: And Rust definitely seems risky to a CTO today.
Micah: They worry, will it be too hard for engineers to learn how to do Rust?
Micah: Will we be able to, if we restructure teams, will we be able to pass off this
Micah: project to another team? Will they have to figure out how to use it?
Micah: Will we be able to hire enough REST engineers?
Micah: And if you're Google and you need to hire 10,000 engineers, I think you should
Micah: be rightly concerned about hiring 10,000 REST engineers.
Micah: I doubt there are that many REST engineers in the world.
Micah: But for a smaller company, that's not an issue at all, right?
Micah: Hiring three Rust engineers is pretty easy.
Micah: And I think especially for a small company, it's an advantage in a way that
Micah: it maybe isn't for a big company to be using Rust.
Micah: Because as a small company, you can attract people because they want to work in Rust.
Micah: And that's a big incentive to work for you.
Micah: And I think that's working in maybe a slightly obscure language.
Micah: You get those people who are really excited about it. And that can be a big boon to you.
Micah: But for big companies, they kind of just see the risk side of that equation.
Matthias: How do you hire Rust engineers? Do you reach out in your network or do you post
Matthias: job announcements somewhere?
Micah: Yeah, well, I guess actually for us initially, we've been hiring more on the
Micah: streaming expertise side.
Micah: There's actually maybe more overlap there now than there was maybe two years ago.
Micah: A lot of the newer streaming systems are also in Rust. But historically,
Micah: as I mentioned, streaming systems have been largely in Java.
Micah: So that's where most people have expertise.
Micah: But I definitely anticipate as we try to hire more broadly that hiring from
Micah: that pool of REST engineers will
Micah: be pretty productive, especially as a non-cryptocurrency REST company.
Micah: There's, I think, a lot of demand for those jobs right now. So we'll be able to tap into that.
Matthias: Very true. What sort of other crates do you use to get your job done?
Matthias: I guess in the blog post, you mentioned Data Fusion. Maybe that's one that you
Matthias: can talk about, but feel free also to talk about any other crate that you like.
Micah: Yeah, so Data Fusion is probably the most critical one to us.
Micah: Data Fusion is a number of things. This comes from the arrow-rs ecosystem.
Micah: We use it primarily as a SQL parser. So it takes SQL text and turns it into an AST and then planner.
Micah: So taking that AST and turning it into a graph-oriented plan that describes
Micah: what that SQL is supposed to do.
Micah: SQL is an extremely complex language with like 30 years of history and a bunch
Micah: of different equivalent ways to express stuff.
Micah: So having a library that deals with a lot of that complexity for you is extremely
Micah: helpful when you're building a SQL engine. we get a nice clean plan out of that
Micah: that we're able to then optimize in our own way and compile into our own set of operators.
Micah: So that data fusion has been extremely critical to us being able to build this
Micah: thing as quickly as we have. Beyond that.
Micah: I guess I'll also call out maybe a little bit lower level or higher level,
Micah: but really appreciate the actually Rust web ecosystem.
Micah: So we rely on Axum and SQLX, which is a really great SQL library.
Micah: This is not like the core of our product at all. This is like to power our API and our web interface.
Micah: Face but it's remarkable that even a domain that maybe rust isn't natively as
Micah: well suited to we still have these incredibly high quality libraries that make
Micah: it actually really easy to to build good products in that domain so that's that's
Micah: been an impressive discovery for us.
Matthias: The grades that you mentioned i cannot speak about data fusion but definitely
Matthias: the other ones are top of class in any language literally i would say at least
Matthias: from my experience I used Axum and SQLX before, and I think they are really awesome.
Matthias: But I wonder about the future of this ecosystem.
Matthias: Do you see that we kind of reached a point where crates are starting to more
Matthias: or less stabilize and there's one go-to crate that you pick for your job?
Matthias: Or would you say the ecosystem is still so young that I can see myself switching,
Matthias: let's say, to a different web framework in a year or maybe a different parser
Matthias: or whatever, if if it comes up.
Micah: Yeah, I mean, I think it's probably too early to say that things have stabilized.
Micah: A year ago, your choices in a lot of these areas would have been different.
Micah: Definitely three years ago, none of these crates existed.
Micah: Axum itself is still changing quite a lot from release to release.
Micah: So I think even these crates are not directly stabilized.
Micah: But I think we will be hitting more of a period of stability,
Micah: especially with async rust becoming more feature complete.
Micah: A lot of these libraries have had to work around limitations in the async ecosystem
Micah: and implementation, like missing the ability to use async functions and traits.
Micah: Which has just landed or is about to land.
Matthias: In 1.74, yeah.
Micah: Yeah. So I think that will allow things to stabilize their APIs in a way that
Micah: has been challenging so far.
Micah: And I do expect more kind of stability going forward and more obvious choices
Micah: around which crate we use to solve different problems.
Micah: And something that's been impressive to me about the Rust ecosystem is that
Micah: there maybe were opportunities to stabilize earlier.
Micah: Just to give you a random example, for logging, we had the log crate that was
Micah: like the obvious crate for a long time to use for logging.
Micah: And we could have just decided that was good enough. But actually,
Micah: it turns out there was a better option and a better design.
Micah: And we ended up with the tracing crate instead. dead.
Micah: And the ecosystem was able to move to kind of like this better option,
Micah: rather than getting bogged down in kind of like a local optimum.
Micah: And you've seen this in a lot of different areas where like there was like an
Micah: early consensus around a crate as the solution to this class of problems.
Micah: But the ecosystem was able to move on to something that solved it better.
Micah: And I think that that's not a property you have in all ecosystems.
Micah: And something I really appreciate about the rest community were able to move
Micah: fairly quickly and also in a pretty consensus driven way.
Micah: To better options in the ecosystem.
Micah: So I think we'll continue to see that happening. I don't know if Axum,
Micah: for example, is like the end term of like Rust web programming.
Micah: I think we'll continue to see iteration happening.
Matthias: What about Rust itself, the standard library? What about stabilization of the Rust core?
Matthias: Would you say this is already in a very satisfactory state?
Matthias: Or would you say that for your use case, there would be things that you would wish for?
Micah: I think everyone has their own wish list of RFCs that we would hope finally get merged.
Micah: I think for me personally, the lack of completeness around async has been the biggest frustration.
Micah: Missing async functions and traits, for example, has required a lot of somewhat
Micah: ugly workarounds for us.
Micah: And even the version of this that's going to be stabilized isn't quite complete
Micah: enough for all of our use cases.
Micah: But I appreciate that Rust takes time to get these solutions right.
Micah: And I think we've seen that process play out with async.
Micah: Overall, I think the Rust programming language is in a really good place.
Micah: And I think it has stabilized over the past couple of years compared to the previous five years.
Micah: And we'll continue to see that stabilization with hopefully a few nice improvements,
Micah: like the work we're getting out of GADTs or the improvements to async we're
Micah: seeing right now.
Matthias: I fully agree. Where I see some issues is on the edges of the Rust standard library.
Matthias: So where you talk to other languages with FFI or where you load code dynamically.
Matthias: And I guess for a streaming platform, that is also an interesting use case,
Matthias: maybe where you can hook stuff into your engine at runtime.
Matthias: And of course, there are technologies like WebAssembly and that sort of stuff
Matthias: getting pushed forward.
Matthias: I wonder if you already experimented with that and what's your impression on
Matthias: the current state of the ecosystem around that?
Micah: Yeah, actually, maybe I should have called that out. I called that out in my
Micah: blog post as a frustration.
Micah: Rust does not have a stable ABI, Application Binary Interface,
Micah: which is challenging if you're trying to build anything that looks like a plugin system.
Micah: So if you want to compile, like in our case, we support user defined functions.
Micah: So users can write REST code that then gets loaded into the engine at runtime.
Micah: If you're writing this in C or C++, there's a stable C API that you're able
Micah: to use to basically dynamically link software at runtime. time.
Micah: Rust doesn't have this. So if you want to compile a library and a host application,
Micah: and link them, you have to do that with the exact same version of the Rust compiler.
Micah: And in many cases, like the same settings for those compilers.
Micah: So it makes it really hard to distribute basically binary software separately
Micah: from the thing that is consuming that library.
Micah: So today, the solution you basically have to use is to use the C API,
Micah: which means giving up a lot of like the features and power of Rust,
Micah: at least at your interfaces.
Micah: You also mentioned Wasm, which is another class of solutions to this problem.
Micah: In some ways, this is even worse from like interface perspective.
Micah: Because there's no real standard way to interact between like hosts and plugins in the Wasm ecosystem.
Micah: So every application sort of has to figure this out for themselves.
Micah: We have explored Wasm as a solution to kind of this class of problems.
Micah: We actually have have an integration with Wasm time, which is a great Rust Wasm runtime.
Micah: And I think for systems like ours, that probably will be the direction that we take going forward.
Micah: Particularly great for integrating with other language ecosystems
Micah: and there's a lot of energy in
Micah: the wasm world to kind of figure
Micah: out these integration problems like how how does
Micah: a rust program talk to a python program over shared
Micah: wasm memory how how can we build these kind of like unified interfaces that
Micah: mean that individual projects like ours don't have to keep solving this class
Micah: of problems over and over again but it would be really nice if rest were better
Micah: at this this kind of interacting dynamically with other compiled code.
Matthias: Are you aware of any RFCs that propose the stabilization of the Rust ABI?
Micah: Yeah, there are a couple RFCs in this area with different approaches,
Micah: but I haven't seen a lot of progress in the past couple of years or real interest
Micah: in solving this problem.
Micah: I think it is somewhat niche. Most REST projects are distributed as source code.
Micah: Most libraries are compiled at build time.
Micah: But for projects like ours, or anything that's dealing with plugin ecosystems,
Micah: yeah, that's not really good enough.
Matthias: I guess we covered a lot of the technicalities of the project.
Matthias: I also wanted to touch on a few things that were a bit more business-related.
Matthias: I guess the first question I had
Matthias: along these lines would be Arroyo is backed by Y Combinator as far as I'm aware.
Matthias: Did investors ever care about your choice of programming language or was it never for debate?
Matthias: Or maybe it was even a good thing and maybe they encouraged you to use Rust.
Micah: Yeah, I would say Rust has only helped us talking to investors. Yeah.
Micah: There are a number of systems that have come before us that have proved Rust can work commercially.
Micah: Investors know it's like the hot language in the data space.
Micah: And so you definitely seem more attractive in that sense if you're using Rust.
Micah: But honestly, most investors do not care about your language choice.
Micah: That's just not the level that they operate on.
Micah: And if you come in as an expert in this area and you say, like,
Micah: I think this is the right technical choice, investors are not going to second
Micah: guess you on that they're much more interested on the like commercial side of
Micah: the question how are you going to sell this who are your users,
Micah: you know why why are they going to choose you over a more established company in this space,
Micah: they're definitely not grilling you about like why are you using tokio or async-io or whatever.
Matthias: Yeah what would you recommend to people that are in the same space and are considering
Matthias: to use rust maybe they dabbled into it but they are not sure if they should
Matthias: fully commit to it for their next project.
Micah: Yeah well i think in this space rust is just the obvious choice today it's you
Micah: know we We went through this whole era of building these systems in Java or Go or whatever.
Micah: But today, especially in the current macroeconomic environment,
Micah: companies are much more cost-conscious.
Micah: And when you can write something in Rust that takes half the resources or a
Micah: quarter of the resources of the Java version of it, that's a huge, huge selling point.
Micah: And it's really hard to compete with these much slower Java systems.
Micah: And I mean the Java systems are responding by like rewriting core pieces in
Micah: like C++ like we saw Spark rewrote their core engine in C++.
Micah: Confluent, the Kafka people have been rewriting stuff as well so I think it's
Micah: just really hard to compete if you're not in either C++ or Rust and I think
Micah: you're going to find even though there's maybe a larger pool of C++ developers
Micah: today it's much easier to teach someone to become a good Rust programmer than
Micah: to teach them to be a good C++ programmer.
Micah: And the Rust compiler helps you so much with people who aren't really experienced
Micah: dealing with memory management.
Micah: It makes it much harder to make these classes of mistakes.
Micah: So I think it is very much the obvious choice.
Micah: Maybe there's some newer languages that you could explore. You mentioned ZIG earlier. earlier.
Micah: But all these kind of like new, I guess, rust replacements are so much less
Micah: mature today that you really would have to be very ambitious to,
Micah: to like experiment with that.
Micah: So yeah, so I think either C++ or Rust and really, unless you have a strong
Micah: reason to use C++, I think Rust is just the default choice today.
Matthias: Taking the example of Confluent, they took parts of their code base and rewrote
Matthias: it in C++, if I understood correctly.
Matthias: And I wonder why they chose C++ instead of Rust, because maybe that's already
Matthias: a very mature alternative.
Matthias: Why didn't they pick Rust then? or was it before Rust even became that mature?
Micah: Yeah, maybe I'll speak of Spark. I maybe have a little more background on...
Micah: But yeah, so Spark historically was written in Scala and then mostly written in Java.
Micah: And then Databricks rewrote their core engine in C++.
Micah: And that's something that they've kept closed source. I think it was just this
Micah: project started like six years ago when Rust was much less mature than it is today.
Matthias: Do you know of any other companies that are currently planning to rewrite parts
Matthias: of their codebase in Rust in that space?
Micah: Well, yeah, a great example is InfluxDB, which was originally written in Java,
Micah: then they rewrote in Go, and have just completed a major rewrite of their core storage engine in Rust.
Micah: And actually, we've benefited a lot from that because they're big supporters
Micah: of Data Fusion and the Arrow project.
Micah: TiKV, I'm not exactly sure how you say that, another example where they started
Micah: in Go and rewrote their core engine in Rust.
Micah: So that's, I think, been a pretty common trend in recent years.
Matthias: If you're curious about InfluxDB's usage of Rust, then you should check out
Matthias: episode number one where we had Paul Nix on the show.
Matthias: And yeah, he talks a lot about the reasoning for InfluxDB moving to Rust.
Matthias: And I think this wasn't planned, but it's a nice segue into promoting
Matthias: this other episode if people are interested very interesting
Matthias: i think looking forward maybe in
Matthias: the next three four or five years and looking at the projects that might get
Matthias: started along the way and the things that are existing and are evolving over
Matthias: time what is your perspective what is your vision for the future where do you see the industry move.
Micah: Yeah i i definitely see yeah i feel like I'm a broken record but like i think
Micah: for people starting new data systems or new like large-scale systems most people
Micah: are going to choose rust going forward,
Micah: there's still people starting new c++ systems but just looking in my own space
Micah: three quarters of the new systems are rust and one quarter of them are in are
Micah: in c++ and i think that trend is going going to only increase as Rust becomes
Micah: less risky from a technical perspective and from a hiring perspective.
Micah: I mean, you know, maybe we'll see disruption from these other newer languages
Micah: that are able to become more mature and start attracting projects.
Micah: At some point, I'm sure Rust will become boring and people will want to use
Micah: more exciting languages.
Micah: But that would be, I think, an extremely successful outcome for the Rust project.
Micah: For now, we have not regretted our technical choice at all.
Micah: We're about a little over a year into this, and Rust has proved an extremely
Micah: successful technology choice.
Micah: I think maybe an interesting question is how much REST adoption happens in the
Micah: more application space.
Micah: So there's kind of a divide here between infrastructural software and application software.
Micah: So infrastructure software, like the stuff we're working on,
Micah: or a database, for example, is something that's written by a small team and
Micah: then run by a much larger group of people.
Micah: So definitely, it makes sense to...
Micah: Put a lot of effort into making it really efficient and fast because it's going
Micah: to run on so many CPU cores over its lifetime.
Micah: For application software where the development costs are much closer or much
Micah: greater than the runtime costs, you don't necessarily have that same financial
Micah: pressure to make it really efficient.
Micah: And today, I think Rust is a much harder sell in that space space because the
Micah: the additional complexity of writing stuff in rest and additional like difficulty,
Micah: you know hiring rest engineers or training
Micah: people in rest so i wonder you know
Micah: how much rest will kind of grow in that
Micah: space through just more
Micah: maturity in the the language and ecosystem system and maybe
Micah: a growing set of like people who use rest or want to use rest but to me that's
Micah: like a big open kind of area for a language to expand or a new language to come
Micah: and move into that that space because i think we can do better than java and
Micah: go for for kind of application level programming.
Micah: There's so much of the ergonomics of rest that are great for that but dealing
Micah: with some of the the sharper corners of rust around the memory management and
Micah: lifetime issues, where it just feels like,
Micah: if you don't care that much about performance, you shouldn't need to kind of
Micah: deal with this for that class of problems.
Matthias: If you look at a related field like data science, it feels like they are also
Matthias: starting to experiment with some ideas from the Rust world,
Matthias: if not even rewrite parts of their libraries in Rust to use it in less performant,
Matthias: higher-level languages like Python.
Matthias: You have Parquet files, and then you have parses around that,
Matthias: and you have pandas, and all of this is inherently...
Matthias: An interesting space for Rust because it's a mixture between analysis where
Matthias: performance is also relevant, right? Do you agree in general?
Micah: Yeah, so I think that's definitely a term that will continue.
Micah: And this is really taking a Rust core and wrapping it in a higher level language like Python.
Micah: And that's been really, really successful. We see that in the Java ecosystem
Micah: system as well, where a lot of these Java tools have rewritten their cores in
Micah: Rust and gotten 10x or more performance out of that.
Micah: I'm not personally a Python person, but people obviously really love it.
Micah: And it's very hard to convince data scientists to use anything besides Python.
Micah: So if you do want to give them better performance, I think this approach of
Micah: writing the core and Rust and wrapping it in your higher-level language is something
Micah: that has been really successful.
Micah: I guess the other fascinating approach, I don't know if you're familiar with
Micah: Mojo. This is Chris Lattner's new language that the creators Swift.
Micah: This is creating a Python-like language that actually compiles into LLVM and
Micah: MLIR and aims to provide C++ performance.
Micah: Python somewhat python compatibility and that to me it is extremely ambitious
Micah: given the kind of semantics of python and how hard that is to optimize but like
Micah: if you don't want to to kind of take this rest approach like that's the only
Micah: other way you can really get acceptable performance,
Micah: with with these python apis so for us we're since we're starting with sql you
Micah: really it's very easy to optimize SQL into whatever,
Micah: implementation you want and that gives
Micah: us a lot of advantages for providing really really high performance because
Micah: SQL is kind of declarative you're able to rewrite the expressions in ways that
Micah: make it much faster to actually execute but when you have something like Python
Micah: you're much more limited in how much you can really optimize that even with a Rust core,
Micah: so i think it'll be interesting to see like how data science
Micah: as our data volumes
Micah: increase and the complexity of the processing we're doing increases
Micah: how that will kind of financial pressures will push people into high performance
Micah: paradigms but yeah for now i think the polar's approach of of kind of the rest
Micah: core is something we'll see in in a lot of these data science ecosystems systems yeah.
Matthias: I agree i think we're
Matthias: getting towards the end and it has been somewhat a tradition around here to
Matthias: ask this final question to people if there was one thing that you could say
Matthias: to the rust community as a whole one of a statement a message that you have
Matthias: to the community what would it be i.
Micah: Think my my message to the rest community would be like chill out a little bit,
Micah: Rust is an incredible language, an incredible ecosystem and community.
Micah: And yet we seem to have 10 times the drama of any other language community I've been part of.
Micah: And I don't really understand why or where it all comes from.
Micah: But I think that that level of drama can only hurt adoption when people look
Micah: at the Rust Reddit and are like, this is a shit show. Why would I want to be part of this?
Micah: So yeah, I think hopefully we can look back at the past year and just say,
Micah: we all just need to calm down a little bit and figure out how to work with everyone
Micah: else and stop driving people out of the community.
Matthias: That's a great final statement. Really love it. Micah, it has been a pleasure to have you on the show.
Matthias: Where can people learn more about you, about Arroyo? How can they get started
Matthias: with the platform? form?
Micah: Yeah. So I think we have some pretty good docs.
Micah: If you head to our website, arroyo.dev, we link to those there.
Micah: We have a Docker image, super easy to run it and play around,
Micah: get a nice web UI where you can write SQL, you can talk to like WebSocket APIs
Micah: and HTTP APIs, and it's easy to play around with publicly available streaming data.
Micah: And then we have a really friendly Discord community so if you head to our website
Micah: we have a link to that and you can join.
Matthias: eard it, there's nothing more to say. Again thanks a lot Micah for coming into
Matthias: the show and yeah thank you.
Micah: Thanks so much for having me! This was great.
Matthias: Rust in production is a podcast by corrode and hosted by me Matthias Endler
Matthias: for show notes transcripts and to learn more about how I can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.