Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we are talking to Debadyuti Roy Chowdhury
Matthias: from InfinyOn about building a distributed streaming engine in Rust.
Matthias: Deb, so happy to have you. Can you say a few words about yourself?
Deb: Oh yeah, absolutely. I am talking to you from Toronto area. I've worked in healthcare,
Deb: automotive manufacturing, surveillance tech, e-commerce, and so on.
Deb: So I'm like what you would call a technical product manager.
Deb: So that's really my background.
Deb: And that background has everything to do with my current role as the vice president of product at InfinyOn.
Matthias: Before we go into that, I did wonder, how do you go from health to data?
Matthias: Where's the connection here?
Deb: Oh, that's a good clarification question. As more and more healthcare systems got digitized,
Deb: there were quality of care questions, which were analytical in nature,
Deb: which would be around population health and quality of care,
Deb: which has expanded into telemedicine and other things that you see like matching
Deb: patients to proper care and so on today.
Matthias: Yeah, that makes sense. So essentially, a lot of those healthcare businesses
Matthias: are nowadays also tech companies.
Matthias: Somehow, they need to manage a lot of data. And this is where data came in and
Matthias: where you maybe found a way into more of the data pipeline world, I guess.
Deb: Yeah, yeah, yeah. Initially, it was just mostly descriptive statistics.
Deb: We were trying to figure out what are the average age of people with certain
Deb: conditions. And those patterns then lead to something proactive in terms of
Deb: health practices, right?
Deb: And obviously, you need to first collect the data, build the system to have
Deb: all of that information go in.
Deb: And if you look at it, it has happened pretty recently, probably in the last 15 years or so.
Deb: Nothing existed before that. Everything was paper and pen. And I remember working
Deb: in scanned faxes and things like that, which were forms filled out.
Deb: And then you have to... It's like human assisted. that some part of it is automated,
Deb: some part of it is manual, and you still need to make sure that all of these
Deb: things add up and so on. Yeah.
Matthias: You would love it here in Germany. We still have that system.
Matthias: So once you found your way into data, what was the first real job working with lots of data?
Deb: So initially, I was working at a smaller firm, which was dealing with insurance providers in the US.
Deb: So Medicare, Medicaid, and other health plans.
Deb: Then I moved into a software company which sold software to hospitals across
Deb: the United States and in Canada.
Deb: And there were practices in Australia and across the globe in some sense.
Deb: We had a SQL Server implementation of the core database that supported the application.
Deb: And that had 4,200 tables.
Deb: So it's a pretty complex database. And then I worked in the analytics team.
Deb: So the analytics stack was actually built by one of the bigger software companies
Deb: using what was the state-of-the-art Microsoft stack, if you will.
Deb: But it was incredibly slow and complicated to process any of these things.
Deb: So someone is trying to run a report for what they did for one year,
Deb: and they're trying to run it for a few hundred doctors in their practice.
Deb: And it's taking days. and if it breaks and there is some problem in the data and some mapping,
Deb: those are all basically running stored procedures or extremely large SQL queries,
Deb: which are like dissertation paper where you could probably, if you printed them
Deb: on a Word doc, that would probably take up like five, seven pages.
Deb: And you're trying to troubleshoot in all the joint tables, you know,
Deb: a few hundred at times, where are things going wrong?
Deb: And none of it is optimized. So like during that period, I would pretty much
Deb: always have a profiler running.
Deb: And then you learn how not to write queries to make or break the system,
Deb: whether it's an issue with the schema or mapping some wrong data set,
Deb: wrong data point with another data point, or just the performance of queries
Deb: that are accurate, but how can you optimize it so that it doesn't take three
Deb: days for people to find a report and they'd say, there's something wrong with the data,
Deb: rerun this, because it's just not, it's like a fool's errand, right?
Deb: We're actually working on moving to an MPP store, still like an enterprise solution,
Deb: moving from Microsoft to Teradata for performance reasons.
Deb: But then it was like, okay, we have this quote unquote, better way in the big
Deb: data realm with Hadoop and Spark and Flume and Hive and all of this entire ecosystem, if you will,
Deb: of replicated storage and parallel processing.
Deb: And that's how I got into distributed systems.
Deb: And it was a bunch of Java, Scala, and making sure these systems work.
Matthias: So you came from a mostly proprietary stack with some Microsoft stuff in there.
Matthias: And then you ended up in an open source ecosystem with lots of new tools.
Matthias: And some of those tools I still remember.
Matthias: At least to me.
Matthias: They were a revelation because eventually you had a system that was composable.
Matthias: You had multiple different very well thought out components a lot of those things
Matthias: were really well done you can remember Hadoop and then Spark and so on was that
Matthias: the same to you or was it also lacking to some extent still all.
Deb: The papers that you're reading coming from Google or big tech companies seem
Deb: like a revelation and there's like an incredible halo effect of those.
Matthias: Shiny objects.
Deb: Because fundamentally, if you separate or compartmentalize the storage layer
Deb: and the processing layer and the infrastructure layer, and you're able to access
Deb: via neat levels of abstraction, different aspects that you need to,
Deb: it actually, at least in theory,
Deb: enables more teams to work together and collaborate more effectively.
Deb: And then that was actually predicated on the promise of the JVM,
Deb: and obviously Java was huge, and Hadoop is how I got into Java,
Deb: and the promise was that write-ones run anywhere, right?
Deb: That was one of the key expectations, which became, in reality,
Deb: write-ones debug everywhere.
Deb: So that was the disillusionment, but I think initially, the theory was airtight,
Deb: and there is certainly promise to a composable system, which interoperates.
Deb: And then the learnings where, you know, what does it take to manage that infrastructure
Deb: and integrations and what have you.
Deb: And the other thing is like the human behavior of this preference.
Deb: We were always using databases, whether it's Oracle or SQL server or whatever.
Deb: Everyone wants to write SQL queries.
Deb: Now you're all of a sudden telling all of those people that you have to write
Deb: some Java or Spark, which is not something that they are mostly doing.
Deb: I think the challenge was that the foundation layer being rock solid and the
Deb: right ones run everywhere did not really work as well as anticipated.
Deb: And the performance challenges made it a little bit esoteric whereby certain people,
Deb: like the people who have built Kafka and Flink, they know how to wield the garbage
Deb: collector code in Java better than the average Java developer,
Deb: and that makes it a little bit challenging for everyone else to use it,
Deb: right? You're stuck with the API.
Deb: What essentially happened, as is true for any programming paradigm that gains
Deb: maturity and popularity, is that you'll get a lot of projects which does similar things.
Deb: There were projects to manage the infrastructure, projects to manage jobs and
Deb: orchestration and failures and things like that.
Deb: You're happy to compose with a bunch of different tools and make it work.
Deb: But the flip side of it is shoving a root problem under the rug,
Deb: which requires you to do some hard work of optimization.
Deb: If you're working at the level of abstraction of the API,
Deb: then that actually hamstrings you to how much optimization you can do or what
Deb: your infrastructure management challenges are going to be.
Matthias: When i think of hadoop the
Matthias: one word that comes to mind is it's complex it's
Matthias: a very bespoke black box so to
Matthias: say i could look at all of the components i could look at hdfs and i might be
Matthias: able to understand it but it's complex at least you need to understand that
Matthias: there's a runtime behind it and there's various things which could fail or it
Matthias: could be flaky and that's a little tedious. Yeah.
Deb: And really every distributed system at the end of the day, you need to really
Deb: think about a couple of things.
Deb: One is all of the points of failures, right? Because you want it to be reliable,
Deb: highly available and computing the right way.
Deb: But the other piece of fault tolerance is, okay, you can have failures.
Deb: And most of the errors that you would find in that paradigm is out of memory
Deb: exceptions, which is the typical error that you see.
Deb: You're processing a large amount of data and for whatever reason,
Deb: you have to troubleshoot it. Like that's the paradigm. And it's quite onerous, right?
Deb: Cognitively burdensome to like even doing the troubleshooting,
Deb: even if you might have the code solution to the problem, just troubleshooting
Deb: and finding the problem where the problem is and where it broke.
Deb: It's pretty time-consuming. If you're playing at the API abstraction level,
Deb: all you can do is run the query again and run the job again and thereby incur
Deb: more costs of using cloud resources if you're cloud hosting.
Deb: In order to troubleshoot, you have to go like double-click a couple of times,
Deb: and that's not an option, right?
Matthias: So that was 2014. We had batch processing. We had big boxes.
Matthias: We had all of that heavy machinery. When I think of data processing in 2024,
Matthias: I think of stream processing.
Matthias: And maybe some lambdas, some smaller things that can be combined,
Matthias: more of a Unix-y sort of way of thinking about things and also smaller machines.
Matthias: But then again, I want to hear it from you because you're still working in data.
Matthias: So there's still hope, I guess.
Matthias: What has changed in the last decade and how did we end up with Rust in data pipelines?
Deb: I came across it because in my last rule, we were aggregating data from,
Deb: let's say, billion products on Amazon.
Deb: And we would scrape those at a certain frequency, anywhere from a few hours
Deb: for very popular high-selling products to maybe a few days to a few weeks,
Deb: depending on the sales frequency and so on.
Deb: And based on that data, we would make calls to a few other APIs to enrich it.
Deb: Basically, we wanted a time series of price history, time series of rank history,
Deb: and so on to see how the changes in those values relate to sales.
Deb: So when we went in and when I joined this role.
Deb: Very quickly, as a product person in data,
Deb: we have four or five teams in data that are doing collection,
Deb: pre-processing, building machine learning models, and the serving infrastructure
Deb: and so on, maintaining this entire system.
Deb: I built my roadmap for the data infrastructure team.
Deb: And then within a couple of months, everything was like, okay,
Deb: forget whatever you did in your first few weeks, because our cost of onboarding
Deb: every customer is more than the customer is willing to pay us.
Deb: For every dollar they paid us, we were actually spending maybe $1.60 or so.
Deb: Everyone wants insights, but data processing is not cheap.
Deb: And the reason why it's not cheap is because you need to use five different systems.
Deb: You're using a hard data store a data warehouse a
Deb: lake multiple regions in a lake and people are like
Deb: oh storage is cheap it's like the complexity of troubleshooting and
Deb: all of that all of the cost adds up like none of this is free you can only go
Deb: so far by decommissioning old data and putting them in cold storage and maintaining
Deb: just the basic hygiene of it right so then an architect friend of mine a little
Deb: more than two years back,
Deb: we're like, okay, we are doing the diagramming of this is what we are going to build.
Deb: And basically what I was trying to build is, hey, here is an abstraction to
Deb: put a connection to your data source.
Deb: Get it in. Some magic happens where you choose what are your columns and stuff like that.
Deb: And I give you one, not three tiers, four solutions or whatever,
Deb: but I give you one system that gives you standard data on the other end,
Deb: which you can immediately visualize.
Deb: And that's how I came across Fluvio and InfinyOn.
Deb: I had heard of Rust, but I was just in my happy place of Python,
Deb: Golang, and that level of optimization and whatever machine learning stack is
Deb: there, PyTorch, TensorFlow, SageMaker.
Matthias: It's interesting because Python and Go are usually mentioned when people talk about data pipelines.
Matthias: What was your first impression when you saw Rust?
Deb: So I just went into, okay, let me find out what is this entire thing about Rust
Deb: and WebAssembly, right?
Deb: So I just like dove in a little bit into the book at a conceptual level,
Deb: try to figure out, build a calculator and look at how things work.
Deb: And I'm like, okay, this is actually amazing. I don't think people will start
Deb: writing data pipelines in Rust just because most of the data people go back
Deb: to the inertia of preferring SQL over writing Java.
Deb: They have adopted Python after a lot of work. So I don't think it is so easy
Deb: to convert all these people to start using Rust. So I didn't look at it that way.
Deb: I actually remember reading that, oh, Async support is coming.
Deb: They're working on it. And then I looked at Wasm and OK, Wasm seemed to certainly
Deb: be saying the same thing that the JVM promised in some sense.
Deb: But the browser is way more standardized. Yeah.
Deb: In like modern day user experience where so many mobile devices,
Deb: so many compute devices, everyone is browsing web pages.
Deb: So it seemed like, okay, conceptually, this seems like there is a very,
Deb: very strong probability that this would be a safer system, a faster system,
Deb: all the benchmarks are showing that.
Deb: And at the same time, if WASM evolves, there was a big asterisk,
Deb: if WASM evolves to be robust enough, the security and other things are there,
Deb: but it It becomes functional,
Deb: that it can be general purpose for everyone.
Deb: Then I thought this would be like a system that would be miles ahead of any
Deb: existing system there is, right?
Deb: And when I looked at Fluvio, I was like, okay, this seems very interesting.
Deb: And then I spoke with A.J. and Sehyo, right?
Deb: They've been around in the Silicon Valley building products,
Deb: building software for decades, three or four now.
Deb: And they built a management layer, if you will, a control plane for Docker containers
Deb: before Kubernetes existed.
Deb: They got acquired by Nginx and they were running service mesh.
Deb: So large scale microservices, you can think of travel booking sites where they're
Deb: running all these services to aggregate all the tickets and the prices and the
Deb: availability and the inventory and users who are trying to book from all over
Deb: fairly complicated systems.
Deb: And that's how they came across using Kafka and Flink, because you need to do
Deb: all of this processing real time.
Deb: And five years ago, they started Fluvio because they had endured maybe a few
Deb: years trying to make Kafka and Flink work, not functionally.
Deb: Functionally, it was like, fine, you could write your Scala code or your Java
Deb: code or your Python code and make it all work.
Deb: But the infrastructure layer that they needed to do for maintaining this stuff
Deb: was very hard. So that's how they built Fluvio.
Deb: They were trying to build a solution that will serve as the backend system for
Deb: processing data to enable intelligent applications.
Deb: And I'm like, okay, this actually makes a lot of sense. I still know that people
Deb: in the Python and SQL world would struggle to just come into Rust.
Deb: Although in the last couple of years, there's been a lot more hype with Rust in the data space.
Deb: But I was skeptical of that. But I was convinced that this is a bet that I want
Deb: to place because it seems like this is the solution we need.
Matthias: If I understand you correctly, using Python and Go is a very fine choice for data pipelines.
Matthias: However, there are things that we can learn from using Rust as a platform.
Matthias: In your own words, what is the big idea behind InfinyOn now?
Deb: What people are trying to build today for any intelligent application is basically
Deb: get data from different sources,
Deb: enrich it, count and present materialized results and add some machine learning
Deb: or AI sprinkles in between.
Deb: Right now we believe that what we need precisely is an infrastructure that is modular and composable,
Deb: and an infrastructure that is lean meaning you don't need five seven different
Deb: tools you use a few so if you think about all of the data integration streaming
Deb: and processing events you should not need three or four you should be able to
Deb: do it with one and that's really our core proposition here,
Deb: that InfinyOn is trying to build the single system for end-to-end event stream processing.
Deb: And so we do that by Fluvio, which is a distributed streaming engine.
Deb: We do that with Stateful Data Flow, which is a framework for building end-to-end
Deb: stream processing pipelines.
Deb: So you connect with your data source, you can send data anywhere,
Deb: you can process it the way you need.
Deb: And in the consumer layer, you serve it to the application that you need to, right?
Deb: Which is different in the sense that today you would have a three-tier application
Deb: with your front and backend database.
Deb: Then you change data capture, write it to a warehouse, do the standard raw standard
Deb: medallion architecture on your data lake or lake house or whatever.
Deb: Store it into an optimized format so that you can run fast queries to reverse ETL.
Deb: Now, all of a sudden, you've got multiple tools before you can get value of your data.
Deb: If I have to interact with five, seven systems, it's going to cost me more.
Deb: It's going to be hard to maintain.
Deb: And I'll have infrastructure overhead, my dev velocity will be slower or it
Deb: would require a lot more people.
Deb: Our thesis is that people need to compute data on demand,
Deb: It doesn't necessarily need to be real-time, but on-demand concurrently from
Deb: different sources, enrich it and materialize that data.
Deb: And you need a simple infrastructure to do that, to implement event-driven architecture.
Deb: And that's what we are building.
Matthias: And when you explained that, I wondered, how would you scale that?
Matthias: Am I responsible for adding new worker nodes, or do you do that on your end? Yeah.
Deb: So cool thing is that, again, with Rust and with Wasm, and this is where Rust and Wasm shines, right?
Deb: Like our core project, Fluvio, is a little over 120,000 lines of code,
Deb: compiles to a 37 MB binary, which I can compile to an ARM v7 Raspberry Pi.
Deb: This is the benefit that we get from Rust and Wasm that it enables running it
Deb: in a compatible device, which could be tiny, which could be resource constrained.
Deb: So why do I say that? Because you can scale up quite a bit when you're running
Deb: it, let's say, on a MacBook Pro M1 or M2 style machine.
Deb: It would scale up a lot more than a
Deb: current java-based system would and then
Deb: scaling out the workers you essentially allocate
Deb: the system right when you're running a service you initialize a worker and you
Deb: initialize the topic and we can partition and scale the partition of the topics
Deb: vertically and horizontally we can scale the workers that is processing different
Deb: service operations on a topic,
Deb: both vertically and horizontally as well. We take care of that.
Matthias: And the other question I have is, which language can I use to write stream processors?
Deb: Yeah, so Fluvio currently supports in the core streaming engine,
Deb: three clients, so Rust, Python, and JavaScript, or Node.
Deb: And it generates all the boilerplate code or the dependencies.
Deb: And you just need to focus on writing the business logic in an infrastructure as code type pattern.
Matthias: Let's say I wanted to use JavaScript. How would I even interact with the incoming data?
Matthias: Do I import some dependency that you provide, some sort of standard library?
Matthias: Or do I get an object that has a certain name into my scope automatically somewhat globally?
Matthias: And then I pass it on to the next layer. How does that work?
Deb: What you need to define is the schema with the data types.
Matthias: So you define that.
Deb: And we have primitives, which are like maps, filters, groups,
Deb: plates, join, whatever, those primitives.
Deb: That's all that you need to do. The system actually brings in all of the relevant dependencies.
Matthias: And how about sharing state between those different stages?
Matthias: Let's say I want to store some value globally so that some other component can access it.
Matthias: Do you have any dependency on, say, a key value store in there or something like a Redis?
Deb: When I joined, we had some tight integrations with etcd as the key value store.
Deb: But that actually proved to be a blocker because people wanted to deploy on Docker,
Deb: deno nomad right so
Deb: you're like okay fine we decoupled our
Deb: dependency from etcd which had another
Deb: impact because the binary when i joined the fluvio binary
Deb: was like maybe 150 200 mb or something and
Deb: we were like no we want to just give you a small
Deb: tiny binary and we we worked
Deb: significant period of time i think maybe about
Deb: six months developer months if
Deb: you will to decouple etcd so we
Deb: have our own key value system it's not a full-fledged database but everything
Deb: that is required to manage offsets time stamps anything you can think of in
Deb: streaming topics we actually built that and and then it became like a 14 meg
Deb: binary and now we have added on top of that and now it's 37 megs with all of
Deb: the bells and whistles of deduplication at multiple levels.
Deb: Watermarking offset management all the things that you need to be certain that
Deb: you have control over your time and completeness in the API.
Deb: Yeah, long story short, we don't have any dependency on any key value store
Deb: and we interface with etcd when we are doing cloud deployment if we need to,
Deb: but we have most of the functionality that we have built ourselves because of that.
Deb: The ideal experience that we are going for is you,
Deb: Run the cluster locally, which is just download a binary, say,
Deb: fluvio cluster, start, it starts.
Deb: And SDF, or stateful data flow, you write your business logic and your operators
Deb: with your data model, and you iteratively build packages.
Deb: Now, you should just be able to allocate a worker and deploy it.
Deb: You don't have to worry about anything else on the cloud.
Deb: You just initialize a cluster on the cloud, you authenticate,
Deb: and you deploy it. That's pretty
Deb: much all you need to do. The rest of the stuff we take care of. earlier.
Matthias: You mentioned that when you first looked into rust there was no async rust.
Deb: We were actually doing some async stuff that
Deb: was built in so we wrote some custom code to do async stuff which then when
Deb: async standard came up there are a bunch of async runtimes that were coming
Deb: up at the time then we went on to async standard because it's easier right and
Deb: then And I think over a period of time,
Deb: what happened is Tokio became the go-to,
Deb: and you can see both Async Standard and Tokio as a dependency.
Deb: That's because we essentially first developed a few pieces of our own,
Deb: which was pretty easy to port to Async Standard.
Deb: And then we figured the API that Async Standard had, the interfaces and abstractions
Deb: was working well, and Tokio was more functional.
Deb: So essentially, today we became async standard interface and Tokio was the runtime
Deb: for most of our workload.
Deb: And that has worked out pretty well. I think Tokio has developed really neatly.
Deb: And this is one of those things like whenever libraries are coming up,
Deb: these decisions are not easy.
Deb: It's like an art and a science. You look at community and sometimes things could go one way or the other.
Deb: But yeah it's worked out pretty well and we are pretty big
Deb: on async because we are wanting to support all
Deb: of these services async if you want to make an api call out and you have to
Deb: wait for the result to come back then it has it becomes synchronous and you
Deb: don't want that right you want it to run record by record through the log so
Deb: it's a hard problem to solve but async standard and tokio has worked out pretty well for us do.
Matthias: You still support both async std and tokio or how does that work internally
Matthias: do you have a shim on top of tokio that is like the old async std layer or you
Matthias: have both as dependencies how.
Deb: Does that work we don't use the async standard runtime all that much anymore
Deb: there are still some parts which we are essentially in the,
Deb: passes of, let's say, migrating over to Tokio. But we have still very much maintained
Deb: the async standard interface.
Deb: So I would say most of our asynchronous runtime stuff is being handled by Tokio.
Matthias: How does it feel as a product person, someone that needs to market a tool that
Matthias: is based on Rust and maybe a fast-moving target as well?
Deb: The buyer doesn't really care. As long as they get their
Deb: data guaranteed they are okay right and in
Deb: the technical buyer space they will anchor you to a
Deb: kafka or a flink and then say oh do you have dead letter do
Deb: you have this xyz connector do you have this do you have that it's
Deb: always hard to compete against mature tooling
Deb: because the cost of migration is pretty high right i think the biggest problem
Deb: is essentially shaping up or right sizing a proof of concept which shows them
Deb: that hey this is a system that actually eliminates it's your need to run at
Deb: least three or four other systems,
Deb: which means that you're going to have better development velocity.
Deb: You're going to have less infrastructure overhead, and you're going to be able to innovate faster.
Deb: It's not going to need as much resources. So it's also going to cost you less
Deb: because you're doing more with less, not figuratively, quite literally.
Deb: And so the conversations there
Deb: becomes how quickly can we show proof of concept or a proof of value that,
Deb: with examples, tutorials, and contextualize, like, hey, let's scope out a POC and build something.
Deb: And then in the top layer, if you will, we communicate some other things around
Deb: the security and the delivery guarantees, which comes obviously leaning on to Rust.
Deb: And also the polyglot thing, which is not completely a realized dream,
Deb: but the Wasm write ones run everywhere with the component model allows us to
Deb: support natively other programming languages,
Deb: which obviously then means you don't have to all of a sudden become a Rust shop
Deb: in order to use our product, right?
Deb: So all of those things put together makes a decent proposal.
Deb: And I haven't really had any conversations with people who hate the idea.
Deb: It just becomes a thing of, okay, what is the value equation?
Deb: And when do we have the capacity to go for it?
Matthias: What's the biggest argument for Fluvio and maybe to a larger extent for Rust
Matthias: when you talk to customers?
Matthias: Do they care about security, safety, scalability, reducing costs? What is it?
Deb: In the current market landscape, a lot of people would think of cost,
Deb: which was also my motivation to look at a leaner system.
Deb: Now, cost is obviously directly tied with efficiency.
Deb: You can never really sell performance because the proverbial notion is,
Deb: hey, you don't necessarily need
Deb: real time if you're making decisions at a different cadence or whatever.
Deb: But when you're doing system-to-system services that does not have a human in
Deb: the loop, then it has to be asynchronous and real-time.
Deb: Most people that actually are active on the platform today,
Deb: they are coming to us because they want an intuitive interface that allows them
Deb: to build 80% of the use cases without having to manage three or five different tools.
Deb: And that's really the core thing that they tell us because I don't tell them.
Deb: They're like, yeah, I currently use A, B, C, D, and they will list out a few
Deb: of our competitors and so on.
Deb: And they will invariably list out three to five tools, including a streaming
Deb: engine or message queue and key value store and old app store or database.
Deb: And okay, then we've gotten into five, six different things.
Deb: And they appreciate the ability to do 80% of that stuff on demand,
Deb: the application in one place.
Matthias: And you said that they also come for the user interface because apparently it's
Matthias: nicer than what the competitors offer.
Matthias: What does the user interface look like?
Deb: Yeah, the user experience, which is broader than the interface,
Deb: is around transparency and debuggability and things like that.
Deb: So for a developer, the user experience is all about control and debuggability
Deb: and ease of finding issues and so on. So if you need to debug today, let's say,
Deb: a fling job and you find an error then you have to look at
Deb: the different workers that failed and then you find the memory exception and
Deb: then you go to your kafka partitions figure out the partitions went in balance
Deb: and this is still to a certain extent the pattern right and then you find out
Deb: okay it's run out of memory we have some obvious suspects if you will now that
Deb: is a non-starter in our scenario because you're not needing to deal with the
Deb: impedance mismatch i would call it across different tools,
Deb: whereby, oh, they're applying this mode of processing at the low level,
Deb: serialization, deserialization, and the interfaces now are struggling to integrate with each other.
Deb: So we give you an intuitive CLI where you can just run the commands and over
Deb: a period of time, developers like, okay, I'll just write this command and I get it.
Deb: And inside of the different CLIs, the Fluvio CLI, SDF CLI, you have ways to
Deb: look at the metrics, look at the logs, and if things are not working,
Deb: it gives you a good debugging experience. And then on top of that,
Deb: we have built out a ui for sdf so.
Matthias: Dag is a directed acyclic graph what is an sdf.
Deb: A stateful data flow and.
Matthias: In regard to the technology that you use for the front end is that also written in rust.
Deb: No so yeah of course it is written in rust i was gonna say no yeah we actually
Deb: wrote it was not a library is what i was thinking we actually had our own framework
Deb: i asked we called it heaven We built our own web framework to do visualization.
Deb: So on InfinyOn Cloud, when people log in, there is a web UI,
Deb: which when you're running a connector,
Deb: applying transforms, it gives you a continuously updating real-time graph of
Deb: how many data points you're getting, how many are you processing,
Deb: what is the back pressure, and so on.
Deb: But we moved recently from heaven actually
Deb: it's been a little while from heaven to leptos because leptos
Deb: became the more popular go-to
Deb: framework or library you want to call whatever you want
Deb: to call for the web paradigm so it is in rust but
Deb: we also have some examples which are using let's
Deb: say something like apache e-charts which is like a real-time visualization library
Deb: and that runs like a JavaScript server to read from WebSocket to consume the
Deb: data and the topics that is materialized and to just show some visualization
Deb: and so on to show other ways of doing it.
Deb: Whatever we are building as components, obviously Rust, we are a predominantly Rust shop.
Matthias: Why did you move away from your own UI library?
Deb: Oh, it was like, we are a small team, 12 or so people.
Deb: The overall repository has maybe 65 or so contributors over the years,
Deb: it was just like taking up a lot of capacity to build out the components that are reusable.
Deb: So it was like a build versus reuse in this case decision.
Deb: And Leptos had matured and a couple of developers who work a lot on the middle
Deb: layer and the front end, they do some infrastructure server side,
Deb: but a lot of front end contributions are from a couple of developers in our team.
Deb: And they really loved Leptos, the way it works. And it would have been faster
Deb: for us to iterate and build.
Deb: Actually, if you were to build what we are building as the SDF UI in heaven,
Deb: I believe we would have probably taken a lot longer.
Deb: It gave us speed and allowed us to focus on the infrastructure and the distributed
Deb: sides of things, because those are obviously more complicated problems to solve.
Deb: Yeah, and we have really appreciated using Leptos so far.
Matthias: Yeah, sounds like a smart decision to focus on the core part of the product.
Matthias: Which data formats does Fluvio support?
Deb: When you're actually reading events if
Deb: you're reading from an api you're typically thinking an application you're
Deb: going to probably get json for the most part you
Deb: can obviously support anything binary if you're getting data from iot sensors
Deb: and so on you can load that payload if you have raw binary format that you want
Deb: to capture doable different customers or prospects they'll come and say okay
Deb: can you do avro can you do protobuf where the schema you you take it from the file format itself.
Deb: Most of our examples are JSON, but we can read from S3, CSV, XML, whatever.
Deb: And we have smart modules on our connectors, which you can then use to convert
Deb: that to JSON or if you wanted something else.
Deb: And then on the materialization side, things become a little more interesting
Deb: because typically when you are materializing data, you're saying,
Deb: okay, here's the payload.
Deb: You need it to be in some form of table or data frame, if you will.
Deb: And there are some aggregates or the records for whatever it is that you want to do.
Deb: So in that, we actually integrated Apache Arrow in the materialization side.
Deb: Arrow is pretty popular. Many projects around it.
Deb: In-memory data frame library is how I would describe it in memory data frame,
Deb: which is like fast and works very well with real time.
Deb: And then we found that there was an interface which was built on top of Arrow
Deb: called Polars, which is a very popular library as well. We actually also played
Deb: around with other engines there.
Deb: We have a DuckDB connector because materializing data, people want to do it in different ways.
Deb: We looked at graphite to visualize using Grafana and stuff like that.
Deb: So we've got a few connectors there.
Deb: These are probably like 500 lines of code, 2,000 lines of code each and separate repositories.
Deb: And we were actually for stateful data flow, all of the materialization piece,
Deb: using a fair bit of Polaris as the interface, the data frame interface to interact with Apache Arrow.
Deb: Your topic data comes in, you run your operations, and then when it becomes
Deb: a table of top five, top 10 aggregates,
Deb: whatever it is that you want to visualize, that goes into Arrow,
Deb: the Stateful DataFrame, and then we were using the Polars interface on top of that.
Deb: We had to since build our own data frame layer.
Deb: When we were using Arrow and Polars and Stateful Data Flow, we built it out.
Deb: One of the nagging problems were, and we are using M2 MacBooks or M1 MacBooks right across our team.
Deb: We are trying to build Stateful Data Flow as is with Arrow and Polars.
Deb: And we found that interface was actually making our build time or our compile
Deb: time was around two and a half, three minutes on my laptop.
Deb: And it's a 64 gig RAM M2, I think, MacBook Pro.
Deb: And we were like, okay, these are pretty expensive laptops. We can't expect
Deb: every developer to be working with that.
Deb: What about the person working with the Replit Repel or want to do this thing
Deb: on a Raspberry Pi or something? So we tried it on an orange Pi.
Deb: I think SDF took maybe 45 minutes or so to just compile. To run, it doesn't take as much.
Deb: This is no good. There was like this nagging problem that we had.
Deb: So there was no way to solve it.
Deb: But Polars is great so you're like okay what can we do
Deb: it was just like this hcd type thing like we don't want
Deb: this dependency if this bloats up our binary and
Deb: increases the build time every time we try to use it so
Deb: we basically spent again a couple of developer months to build our own data
Deb: frame api abstraction to interact with apache arrow and that made our product
Deb: a little more over-engineered little more feature-rich little more independent
Deb: or what have you and now on my machine the build time has dropped to 15 seconds does.
Matthias: It mean you still support apache arrow and polars yeah.
Deb: Yeah, we are also very deliberately minimalistic because we don't want to become
Deb: yet another iteration of what looked like the big data ecosystem.
Deb: So we are trying to be very deliberate about what is the most necessary.
Deb: And what is most necessary is perhaps support for Python because everyone in
Deb: data knows Python, support for SQL because everyone knows SQL.
Deb: We support the serialization formats on the in. So, you know,
Deb: you do JSON, you do binary, you do protobuf, you do Avro.
Deb: Obviously, that will help people not have to think about how they can load data into Fluvio.
Deb: And then on the materialization side, I believe right now all the evidence suggests
Deb: Apache Arrow is something that has matured to the point where large companies
Deb: want to build on top of it and so on.
Deb: So we kept Arrow and Polar support. And on the materialization side or the consumption
Deb: side, we will enable it to be as low level or as high level as you want.
Deb: The thing that you don't want to do is to put too many different tools to confuse
Deb: people. Like, I don't want five tools to do the same thing.
Deb: But Polars as a data frame is an iteration or evolution on top of pandas to
Deb: give them a fast data frame.
Deb: And all data scientists were already used to pandas and they want the Polars interface.
Deb: So if they want the Polars interface, have at it.
Matthias: My impression is that you integrate with a lot of different technologies.
Matthias: And the one thing I would wonder about as a Rust developer is how mature is
Matthias: the ecosystem to interface with all of these systems?
Matthias: Because you're putting the company on the line here by being...
Matthias: Maybe confident enough to say that you can provide a certain level of quality
Matthias: with all of these different integrations.
Matthias: And that's not always a given because the ecosystem is still very young and
Matthias: people expect a certain quality of code when they trust you with their data.
Deb: Absolutely. Different things that we are integrating with, even though they
Deb: are a small enough list for my liking, because if you look at the current state
Deb: of the data stack, there are probably like 2,500 tools there that you can integrate with.
Deb: But there are parts where in critical places where we integrate with mature technology.
Deb: And again, like I mentioned, so Avro, Protobuf, Arrow, Polarsis,
Deb: not as mature as Arrow is, let's say, but these are considered to be mature.
Deb: Now, the rest of the ecosystem, let's say the Wasm component model,
Deb: for example, is, you know, Dev Preview 3 running right now.
Deb: And, you know, there is a lot of work that goes into managing the infrastructure
Deb: and making sure that there is reliability and so on.
Deb: This is where the engineering experience of the entire team comes in.
Deb: Like our head of engineering was working with, you know, embedded aerospace,
Deb: autonomous aircraft and storage, disk storage systems and stuff like that.
Deb: So we've got a lot of very solid systems people to make sure that we,
Deb: whatever we are building is robust. we are not building it in a loose way.
Deb: Our CI, CD and our integration testing costs are probably as much as our infrastructure
Deb: costs are running things.
Deb: So we obviously make sure that we build a robust solution. Functionally.
Deb: You cannot do whatever you're depending on.
Deb: So for example, we were using WASM initially for smart modules,
Deb: and you could only do transformations using smart modules on streams of data.
Deb: And it was really difficult to do time-bound window operations like the stateful computation.
Deb: You could do it. But then WASM was enhanced, if you will, because WASI is very much an extension of it.
Deb: So now we have all these new interfaces that allows us to do more with it.
Deb: And then came the component model, right? So in this paradigm,
Deb: functionally, we are constrained, let's say, to deliver whatever the leading
Deb: edge of technology can deliver.
Deb: But that's actually quite a bit of critical mass because we are working with
Deb: IoT companies which are embedding Fluvio on edge devices, let's say,
Deb: flow meters that are monitoring flow rate, right?
Deb: And then using Fluvio on the cloud and stateful data flow to compute and run
Deb: their analytics, right?
Deb: And it takes care of your use cases around
Deb: predictive maintenance detecting anomalies fraud
Deb: and stuff like that and we have customers that
Deb: have now come on board to start implementing which is not the case a couple
Deb: of years back so we obviously it's been a five-year journey to try and get something
Deb: over the line and all databases take like decades to mature you can see like
Deb: all of the challenges in the ecosystem there.
Deb: So certainly everything is not figured out, right? It's really figuring out
Deb: how to make the mature and opinionated,
Deb: components, jive well with the somewhat newer radioactive components,
Deb: and ensure that you don't pass on your tech debt or the challenges of this particular
Deb: synergy, if you will, to the customer or the user.
Deb: So that's why we exist to make the system.
Matthias: Did you hit any runtime bugs with Rust?
Deb: Since we launched the beta, it's typically been something around,
Deb: oh, you're using a certain version of Python or a certain version of Node at
Deb: the interface level, and so the client is struggling to give you this version.
Deb: And so typically updating the user side to the latest version has solved it,
Deb: which is what you would ideally want, right?
Deb: Or you have to give them an environment, okay, these are all the requirements
Deb: or whatever, but it's been pretty smooth.
Matthias: If you talk to a company about using Rust, for example, as part of their startup,
Matthias: what advice would you give them?
Deb: In the context of you are a non-vendor company, like you're not a company like
Deb: InfinyOn trying to build a streaming tool.
Deb: You are a healthcare company or an e-commerce company or whatever,
Deb: a data team there, and you're trying to build a better, more performance system using Rust.
Deb: You have to iterate and find the parts of your system which requires most concurrency,
Deb: See most asynchronous pieces and piece by piece,
Deb: look at existing repositories that can help you and build it for any other company
Deb: that are trying to build Rust-based tooling. Right. Startups like us.
Deb: I think one of the things you recognize, especially in the data infrastructure
Deb: space, is that it is a long game.
Deb: You cannot come in here to like, oh, think, okay, let's just like put a wrapper.
Deb: A wrapper would not be as valuable because there's already a Kafka wire wrapper
Deb: and a few like those things like you can do code gens.
Deb: We were doing this stuff where we would use PyO3 to generate Python code and
Deb: generate the bindings and run.
Deb: Now, those things are great for POC type scenarios, but in production,
Deb: you want some robustness to the solution that you build, right?
Deb: So that makes the build process a little bit longer.
Deb: And finally, I would say like it's well worth it. The co-founders,
Deb: they actually were initially debating on whether to use Golang or Rust to build Fluvio five years ago.
Deb: And it was a fairly contentious decision. And the CTO won that debate and Rust was selected.
Deb: They had done a bunch of Golang work with server side and Golang is pretty fast, right?
Deb: But the things that are most difficult to learn in Rust for beginners,
Deb: like borrower, checker, like async traits, are the ones that guarantee safety
Deb: and security and guarantee that you get your data the way you want it.
Deb: And this is why we made, or the CTO and the CEO, when they were making the decision,
Deb: made the choice to write the Fluvio project in Rust.
Matthias: In Season 1, we had Micah Wylde from Arroyo as a guest.
Matthias: Do you see them as competition?
Deb: We know Micah. Micah is on our Discord.
Deb: They built a Fluvio connector or integration before we even knew about Arroyo.
Deb: I think their play is really amazing.
Deb: I don't see the entire streaming ecosystem as a competitor because everyone has their angle.
Deb: The market then decides what they want to do.
Deb: There is room for many players to take a slice of the market.
Deb: And it's not time for consolidation in this part of the data market. Micah is from Liften.
Deb: They were running large-scale flink operations. And so they recognize,
Deb: again, the ability to run these data flows, which are stateful,
Deb: managed state, time versus completeness trade-offs, and give a SQL interface.
Deb: I think they are also building on top of Arrow. So I'd say in that sense, we are friends.
Deb: The difference between Arroyo and our thing would be that most engines that
Deb: you would come across, their thing would start from a distributed streaming
Deb: engine, right? Like they will start at, oh, here is a Kafka topic.
Deb: Now we are going to do the processing, right? But we built the Kafka alternative
Deb: first, which is extremely hard to penetrate into the market because Kafka has
Deb: a pretty solid stronghold there. Right.
Deb: We have a slightly different proposition. It's like one system which does a
Deb: bunch of different things.
Deb: And most people that are adopting us are really bought in on that unification
Deb: and the lean proposition.
Deb: Like they are willing to write some Rust. And a lot of our current users are actually Rust shops.
Deb: And if they are not, they have at least a few developers who are picking up and writing Rust code.
Deb: And so that's slightly different. In our case, giving inline SQL or Python is
Deb: like the icing on the cake and the cherry on top. It's not the core, right?
Deb: But I believe like Arrow's core proposition is, hey, SQL, making it easy.
Deb: And I really applaud that project. I think it's a solid project.
Deb: I don't see them as a competitor necessarily, although we might have customers
Deb: that we may provide something together. We have actually had open source users
Deb: who have come and said, oh, you know what? We are building this before we had SDF.
Deb: We have Fluvio and they have the, Arroyo has the Fluvio integration to get data from topics.
Deb: We are building this and we have helped a few open source projects like that
Deb: to put something up the ground.
Deb: So I at least see it as more collaborative and not necessarily competitive.
Matthias: What about other products like Nets and ATS that seems to be thrown around quite
Matthias: a lot lately and that's written in Go? What are your thoughts on that?
Deb: Yeah, we have found a bunch of people who are NATS users. We've had on the open
Deb: source project people who have asked for the NATS connector and things like that.
Deb: We had a NATS connector built, but it was not like the full functional latest version API and so on.
Deb: And if I remember correctly, there are a couple of other products on top of NATS.
Deb: I think there is another one called memphis.dev, which was NATS plus, you can say.
Deb: You can call that as a competitor in some sense. But Nats, again,
Deb: any of the competitor, right, like just smaller, newer project is going after
Deb: the incumbent that is Kafka, right?
Deb: And that they take is to first create an integration, which we did as well.
Deb: And then the integration becomes tighter and then you have diversity of options and workflows.
Deb: I think that the Nats core product is pretty solid. and the buying decision
Deb: with the user would become one of really preference.
Deb: There is no way to convince one or the other.
Deb: I think we exist in a parallel ecosystem. I would be perhaps a tiny bit arrogant
Deb: and say that I don't think the alternatives.
Deb: Which are in the distributed streaming space, not the processing part,
Deb: the alternatives would struggle to be as lean and nimble as us to do the IoT
Deb: use cases with the API callouts and enrichment.
Deb: I think of the ecosystem, by the way, in a slightly different way whereby it's okay.
Deb: What are the other systems that I can integrate with, which could create meaningful
Deb: technology partnerships? I'm looking at,
Deb: Rust-based data systems that we have synergy with.
Deb: So for example, like Arroyo is a Rust-based system and they built a Fluvio connector.
Deb: We didn't even ask them. And then, okay, great. We built a relationship with
Deb: Micah and we're happy to collaborate and move the community towards this.
Deb: Similarly, there are a bunch of Rust-based vector databases which are for Gen AI stuff.
Deb: And we are working with a couple to create something there. The team at Quadrant,
Deb: the vector database, they built a Fluvio connector.
Deb: This is another thing that came as a pleasant surprise. That's okay, we have built this.
Deb: Do you want to integrate it and put it on your hub? We didn't know about it.
Deb: And they have done pretty well. OpenAI is using them. Twitter is using them, right?
Deb: That is a big enough vision, I guess, where multiple players would need to come
Deb: together to fulfill that need.
Deb: And I don't really have much time to worry about competition in that case.
Deb: Make sure that we are giving the features that the developers need to implement
Deb: and give them a reliable infrastructure that works for them, right?
Deb: There are many companies in the low-code ETL space, which we don't play in,
Deb: but people tend to ask us like, oh, are you like another ETL?
Deb: They give you like out of the box 500 connectors, but those connectors are at
Deb: a level of abstraction where you cannot do much with it other than connect,
Deb: authenticate, and have open access data.
Deb: So you can do EL with it. you cannot really do much of a T because T requires
Deb: you to know the model, data types, and all this complexity come in immediately.
Deb: And that's the thing that the data world is stuck in, that it's similar to batch processing.
Deb: Another parallel mindset like SQL is ETL, or everyone wants that out of the box, give me something.
Deb: And we're like, no, it's a secure thing that belongs to you.
Deb: I'd rather teach you how to fish rather than give you a fish.
Deb: It will not take you a long
Deb: time to build and maintain a connector that you do
Deb: because it's core to your data operation like you you want to own that and you
Deb: want to have let's alleviate the infrastructure but you own anything to do with
Deb: your secrets your authentication your connections to your data i talk to clients who.
Matthias: Run their pipelines in those no code or low code platforms and their problem
Matthias: is that it works really well in the beginning but then after a while when the
Matthias: company grows they feel locked into this platform and then the runtime is not
Matthias: controlled by them they cannot really,
Matthias: work their way around it and then even if they
Matthias: talk to the companies that they integrate with it doesn't mean that necessarily
Matthias: they have same interests as these growing companies and then eventually you
Matthias: find yourself somehow locked into these platforms and then yeah they look for
Matthias: alternatives so that seems to be a very real problem yeah.
Deb: Yeah someone on our discord said that this like a golden cage that they're locked
Deb: in like they can't escape the golden cage and fly.
Matthias: I think we're getting close to the end,
Matthias: but one thing I was really curious about was when you explained how you can
Matthias: visualize the different stream processors and how the data would flow through
Matthias: your engine and you as a user could see that,
Matthias: that automatically rang a bell because I thought of logging and tracing.
Matthias: And that seems to be one other area where people wanted to have more quality
Matthias: of service, I guess, with open telemetry and with observability in general.
Matthias: What is Fluvio's story there?
Deb: We have logging at multiple levels where you can troubleshoot and debug your system.
Deb: And we also bake in fairly exhaustive amount of metrics that can be piped into
Deb: their own topics for monitoring and observability.
Deb: And then if you want to do more with that, like you wanted to do an open telemetry
Deb: connector to then pipe and have a graph on our dashboard of how that system is running.
Deb: We have prototypes, design, stuff like that.
Deb: Most of the things that we already thought of needs to be monitored and visualized.
Deb: We have already built into it.
Deb: And anything additional, we have the integration capabilities that will allow
Deb: you to build a connector and pipe data out and take it to visualize it if you
Deb: wanted to visualize it with other data and so on.
Deb: We don't have an open telemetry connector yet, but we have had that come up
Deb: enough to say, okay, a few more people asked if we would build it.
Matthias: What would be your message to the Rust community?
Deb: I really loved the enthusiasm of the Rust community overall.
Deb: And I would say like all the conferences in your talk, even that's how we connected
Deb: the four horsemen of bad Rust code, if I remember that correctly.
Deb: Yeah, I think the community has done really a lot of amazing work.
Deb: And yeah, if you're looking for a distributed streaming system,
Deb: a messaging system, enterprise service bus, whatever it may be,
Deb: different people call it different things, I would say check out Fluvio.
Deb: We are happy to grow this and integrate it with other solutions.
Deb: Like I mentioned, a few that we have had the opportunity to reintegrate with.
Deb: So I'm really looking forward to partaking in the journey with the rest of the Rust community.
Matthias: Deb, thank you so much for being a guest. And I hope to talk to you soon.
Deb: Yeah, thank you for having me.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.