Matthias: And without any further ado, Paul, can you introduce yourself and the company InfluxDB?
Paul: Yeah, so I'm Paul Dix. I'm the co-founder and CTO of InfluxData,
Paul: which is the company that makes InfluxDB.
Paul: So InfluxDB is an open source time series database, so developers can use it
Paul: for tracking what's going on in their servers, their applications.
Paul: Sensor data is a very big use case for us.
Paul: We created the project in 2013 and had the 1.0 release in 2016,
Paul: the 2.0 release in I think 2020, and we have now 3.0 available,
Paul: not in open source yet, only under our commercial offerings as a cloud offering
Paul: or as an on-premise software.
Paul: And we'll have a 3.0 open source release hopefully an alpha by the end of this year or early next year.
Matthias: Awesome. So take us back to 2013. What was the ecosystem of time series databases back then?
Matthias: What was your personal need to start the project?
Matthias: And also how did InfluxDB and InfluxData start in general?
Paul: Yeah, so obviously the world of time series databases was very different in
Paul: 2013. So, I started the company with my co-founder actually in 2012,
Paul: and we started originally as a totally different thing, right?
Paul: It was an application called Airplane, like E-R-R-P-L-A-N-E,
Paul: and we were building like a SaaS application for doing real-time metrics and
Paul: monitoring and application exceptions.
Paul: So, you know, like in the same vein as like Datadog or New Relic or any one
Paul: of these dozens of other companies.
Paul: And to build that application,
Paul: originally my goal there was I have a bit of a background in machine learning
Paul: and my thought was I can apply machine learning techniques to all this monitoring
Paul: data and do things like predictive analytics and anomaly detection and stuff.
Paul: First, we had to build the application to get people to actually send us their data.
Paul: And essentially, I had to build a time series API for storing all of this data
Paul: and working with it at scale.
Paul: And the first thing I built was essentially web services written in Scala using
Paul: Cassandra as the backend database, and Redis as basically a real-time indexing
Paul: layer and last value store.
Paul: So I created that. We got into Y Combinator in the winter 13 batch.
Paul: And around this time, I actually rewrote that back end in Go using LevelDB,
Paul: which is a storage engine that was open sourced originally at Google.
Paul: And I built like this basically all in one thing, and that thing became the
Paul: back end for this SaaS application.
Paul: We went through the program, I raised the seed round of funding,
Paul: I was the founding CEO of the company, and then by the fall of 2013,
Paul: we realized that that company just wasn't gonna work.
Paul: Like we had maybe a dozen or so customers, they weren't even paying us enough
Paul: to like pair infrastructure bills.
Paul: Thankfully, at that time, we were still using free Amazon credits.
Paul: And we realized that that wasn't going to take off.
Paul: But we thought there was something there with the infrastructure that we had built, right?
Paul: We could see a bunch of other companies kind of trying to roll their own,
Paul: like create this from scratch, right?
Paul: Every other server monitoring company, every user analytics company,
Paul: every analytics company of any kind was kind of creating their own time series database,
Paul: either from scratch or with a bunch of application code on top of a more general purpose database.
Paul: And what I saw in large companies is that they were trying to solve this problem with open source tools.
Paul: And at the time, the only thing available in 2013 was essentially graphite. So graphite or RRD.
Paul: And graphite in 2013, it's an open source project written in Python.
Paul: It's a metrics data store. It was originally built in 2008 by a team at Orbitz.
Paul: And it hadn't had an actual release in over a year. Basically,
Paul: it was a project that was largely orphaned, but it had a surprising amount of
Paul: adoption in large companies that were trying to build their own monitoring stack
Paul: using open source tools.
Paul: The other thing I saw was I had actually had, prior to this in 2010,
Paul: I was working for a FinTech startup.
Paul: For that FinTech startup, I had to build a solution for working with time series data.
Paul: This was like financial market data and stuff like that. And the set of tools
Paul: that I used was the same set of tools I used to build the first back end for Airplane.
Paul: So it could see this idea of time series, not just being about server monitoring
Paul: data, but financial market data,
Paul: user analytics data, real-time analytics data, and sensor data,
Paul: which in 2013 wasn't really a big thing, but it was starting to become obvious
Paul: that it probably would be.
Paul: So, essentially what we did was we said, okay, well, the application airplane isn't taking off.
Paul: So, let's maybe do a quick spike real quick.
Paul: We'll take the same tools that we had built the backend of airplane with,
Paul: take some of that code and pull it out and repurpose it as an open source project.
Paul: And we built that and we basically took, I think, five or six weeks to build
Paul: a prototype of this thing. And then in early 2013, in November of 2013,
Paul: I arranged to give some talks at some meetups and I announced that we were working on it.
Paul: We had like a basic documentation website that landed on the front page of Hacker News.
Paul: And it was, you know, there all day and a bunch of people said like they were interested in it.
Paul: And when I gave these talks, it was obvious pretty quickly that people were
Paul: really interested in what we were building.
Paul: It looked like we were trying to address a need that a lot of people had that
Paul: nobody else was kind of paying attention to at that time.
Matthias: If you say that was around 2013, that was still pretty early for Go.
Matthias: So I wonder if it was one of the first major projects for Go outside of Google, even.
Paul: So yeah, Go 1.0 came out in I think March of 2012, which is when,
Paul: like basically in the fall of 2012 is when I started using it pretty heavily.
Paul: Docker was written in Go and that came out in early 2013.
Paul: And then right around the same time in 2013, 2014, HashiCorp started creating
Paul: their first set of tools in Go.
Paul: So there was like this whole like
Paul: set of projects around that time that started, you know, started in Go.
Paul: But we definitely early on in, you know, late 2013, all of 2014 and 2015,
Paul: we definitely got a lot of interest from developers because of the fact that we were written in Go.
Matthias: Yeah, I can guess that would be the case. And maybe you also worked a little
Matthias: bit with that hype and also generated some traction for your product early on
Matthias: because Go had a focus on maybe web scale or maybe even infrastructure products
Matthias: and that is still the case, of course.
Matthias: So that was sort of the most sensible tooling for this job, right?
Paul: Yeah, I mean, at the time when we started the project, the debate we had internally
Paul: was, do we write the database in C, C++, or Go?
Paul: And the reason we were hesitant at first to write it in Go was because of the garbage collector.
Paul: A database is usually a very performance-sensitive piece of system software,
Paul: generally you want to have total control over what happens.
Paul: The thing is, there were other databases that were written in Java,
Paul: which obviously has garbage collector.
Paul: So I thought, well, we could just use the same techniques that those databases
Paul: use if we end up running into problems with Go's garbage collector, right?
Paul: Their techniques are basically hide the memory away from the garbage collector
Paul: and manage it yourself in some place, right? In some unsafe place.
Paul: And our bet at that time was, we knew Go more than we knew either C or C++.
Paul: And we just thought, you know, it was me, my co-founder, and one other person.
Paul: And we just thought, you know, our biggest problem right now is trying to make
Paul: software that anybody will want to use.
Paul: So we figured we could do more faster with Go and that the language,
Paul: you know, in 2013, our bet was the language will just keep improving and we'll
Paul: get the benefits of those improvements along the way.
Paul: And that certainly ended up being the case over the, you know, over the next 10 years.
Paul: I mean, most, I would say, you know, 3.0 is written in Rust.
Paul: So we could obviously, we'll get to that. But most of what we run right now
Paul: for our customers and in production is Go code, right?
Paul: We have yet to transition a lot of the business over to the Rust code base that we now have.
Matthias: Yeah. I got in touch with InfluxDB around 1.0.
Matthias: It was even before that, it was somewhere around 0.8.
Matthias: That was the first version we used in production at Trivago.
Matthias: And InfluxDB was killing it.
Matthias: And it was so performant in comparison to what we had before,
Matthias: which was, of course, graphite and something custom.
Matthias: And then all of a sudden, you had access to something that scaled to all of our servers.
Matthias: We ran this thing for about a year on a single machine, and it handled the entire traffic of Trivago.
Matthias: Only later, we went with Influx Cluster and so on.
Matthias: But But the point that I'm trying to make is, what you built was purely magical,
Matthias: I would even say, because something like that didn't exist before.
Matthias: And you bargained from storage engines like LevelDB, or I guess later you switched to BulkDB as well.
Matthias: Maybe you can talk a little bit about this and say, why did you move away from
Matthias: LevelDB and why did you look for alternatives in the end?
Matthias: And just in general, what were some of the main problems that you faced building
Matthias: something like this in Go, or any other language for that matter?
Paul: Yeah, so LevelDB is essentially a key value store where the keyspace is ordered.
Paul: So you can do range scans on the keyspace and get the values.
Paul: It's written in C++, which if you're bringing that into a Go program,
Paul: you know you pay a price for for for making for for the C-Go bridge,
Paul: So, the way, so one of the common things in a time series use case is people
Paul: want to, people like data is super valuable,
Paul: like around the time when it's ingested and generated and the value of the data
Paul: quickly falls off a cliff, right?
Paul: Particularly in like monitoring and stuff like that. A lot of times like people
Paul: don't need the raw high precision data longer than say 10 days, right?
Paul: So one of the, and the thing is the volume of data that you're ingesting,
Paul: the number of individual records that you're ingesting is way,
Paul: way higher than you see in, you know, traditional like transactional database workloads, right?
Paul: If you have a customer database, you're not, you don't generally have like a
Paul: billion records a day that you're ingesting, right?
Paul: So one of the problems in, in the time series space is figuring out how to manage
Paul: the lifecycle of that data and a lot of times how to basically just evict that
Paul: data from the database as it ages out, right?
Paul: So people wanted to say, I want to keep my data around for seven days or I want
Paul: to keep my data around for 90 days and I want it to automatically go away when that's done.
Paul: Now, in a traditional database, if you did this the naive way,
Paul: which would be you issue a delete for every record 90 days after you you insert
Paul: it, your database would just completely fall over.
Paul: It's not designed to delete literally every single record that you feed into the database.
Paul: It just causes the performance to completely fall off the cliff.
Paul: So generally, if you're using a traditional transactional database,
Paul: the way you organize the data is you say, I'm going to create an actual table for each day of data.
Paul: And then when the day ages out, I will drop the table. Dropping a table is cheap
Paul: because you're literally just like, you know, getting rid of the files.
Paul: So in level DB, essentially what we would have to do is for each period of time.
Paul: And for us, I think at that time we had a period of time be seven days.
Paul: We would create a new level DB database. Now level DB database is actually,
Paul: you know, thousands and thousands thousands of small individual little files
Paul: that it manages there, right?
Paul: So one of the early problems that users had with InfluxDB was they would hit
Paul: file limit, open file handle limits, right?
Paul: Literally we had to keep telling people like, oh, you need to adjust these up,
Paul: which is I guess more standard now, but I can't tell you how many bug reports
Paul: we got because people didn't have their file handle limits high enough.
Paul: So essentially, we created a level DB database for each one of these things.
Paul: And then, you know, we would drop the entire the entire database when that when it aged out.
Paul: And it wasn't, we felt like it wasn't an ideal structure. And that and like
Paul: the Segoe bridge wasn't ideal either.
Paul: And there are other things like doing backups was really, really hard at that
Paul: time. Like, I don't know that we ever even had backups, technically for version
Paul: 0.8, which still used level DB.
Paul: So essentially we wanted to solve some of those problems.
Paul: So when we created the next version, 0.9, we thought, oh, we'll use BoltDB,
Paul: which is also a key value store.
Paul: It's written in Go, but it's a different structure.
Paul: So level DB is what's called a log structured merge tree, which is actually
Paul: something that's fairly good for this kind of use case.
Paul: BoltDB is what's called a copy-on-write B plus tree, which is fantastic if you
Paul: have a high read workload.
Paul: It's kind of ruinous if you have a super high insert workload, which we do.
Paul: We didn't realize this early on, so we built around it.
Paul: And actually, the 0.9 series of releases was essentially two things that we
Paul: were getting done at that time.
Paul: One, we were changing the data model to the data model that you see today in
Paul: InfluxDB, which is you have a measurement,
Paul: you have tags, which are key value pairs where the values are strings,
Paul: and then you have fields where the key is, the field key is the name of the
Paul: thing and then the value is whatever kind of value you're measuring,
Paul: which could be a float, an int, a Boolean, or a string.
Paul: So with version 0.9, what we did is we introduced this data model and we basically
Paul: split the database out into two separate things.
Paul: One, which is an inverted index that maps metadata to an underlying time series,
Paul: and the other is time series data, which in 0.9 we were trying to keep in Bolty B.
Paul: We worked on that for probably about eight months until we realized that Bolty
Paul: B just wasn't going to work for what we were trying to do.
Paul: So in over, what, early September, over like a holiday weekend, I kind of like,
Paul: prototyped out this idea for what I was thinking of as the time-structured merge
Paul: tree, which is the storage engine we ended up building.
Paul: It's basically a storage engine that we designed that's kind of like a log-structured
Paul: merge tree, but optimized for this time series use case, how we saw data flowing
Paul: in and all this other stuff.
Paul: So this was in 2015 that we were doing all of this work.
Paul: September of 2015, I prototyped this out.
Paul: It takes a few months and we realized okay, this is probably actually going to be really good.
Paul: So by March of 2016, we had a release of InfluxDB, which I think was the 0.11 release,
Paul: where we had the first version of that storage engine paired with this new data
Paul: model and with the same old query language that we had, and we saw at that stage
Paul: performance that was really, really exceptional, right?
Paul: We had much better performance than we ever had with previous versions of InfluxDB,
Paul: and the data model kind of mapped with how people were thinking about their data.
Paul: So in version 0.8, we had a data model where you had like a table or a measurement
Paul: name, whatever you want to call it, and then a bunch of columns.
Paul: And it was really easy in 0.8 for people to create a schema that gave them very, very poor performance.
Paul: And again, how this would come through is people would log bugs and they'd be
Paul: like, oh, the performance sucks after I insert 10 million rows into the database.
Paul: And we're like, well, that's because your schema doesn't match how you're trying to query your data.
Paul: So when we introduced the tags concept, the goal there was to create.
Paul: Drive users into modeling their data in such a way that they would get good
Paul: query performance out of the box.
Paul: We released version 1.0 in September 2016.
Paul: With that custom storage engine that we had built.
Paul: And that was looking good. But even with that release, as I mentioned, it's two data stores.
Paul: It's an inverted index for the metadata and then the time series data store.
Paul: In that version, the time series data was kept on disk and the inverted index
Paul: was built only in memory.
Paul: So basically, when the database server booted up, it would build the inverted
Paul: index on the fly from the underlying time series data. And what that meant was
Paul: as the database got bigger and bigger, you got more and more time series data,
Paul: the boot up process would take longer.
Paul: So for the next, I'd say probably for the rest of 2016 and the first half of 2017,
Paul: we put a ton of work into adding basically the time series index,
Paul: which was essentially an inverted index on disk that we could use.
Paul: And when the combination of those two, TSM and TSI, formed the core storage
Paul: engine that actually underlies 1.0 and also InfluxDB 2.0, it's the same storage engine.
Paul: And, yeah.
Matthias: So while you explained all this, I wondered how much of it was informed by your
Matthias: choice of programming language and how much was just very general,
Matthias: algorithms work or architecture work, and maybe you can shed some light on this.
Matthias: Would you say that Go impacted the design, or would you say that was completely
Matthias: agnostic of Go, especially with regards to garbage collector times or high cardinality?
Paul: I think the only way I think Go impacted the design, it didn't impact the architectural
Paul: design. again, most of the work we're doing here would have been the same stuff
Paul: we would have had to do regardless of the language.
Paul: If we had written it C, C++, or then later Rust. There's still these core things
Paul: that you have to figure out.
Paul: I will say that the choice of Go
Paul: as a language definitely impacted what we decided to use as dependencies.
Paul: If we had written the entire thing in C++, we almost certainly would have just
Paul: taken an existing storage engine from somewhere else and used it, right?
Paul: Because that's way, way easier. And another part of this is essentially like
Paul: a learning process as we went, right?
Paul: Well, knowing what I know now, I would not build a storage engine.
Paul: Like that's not something I would spend my time doing.
Paul: You can basically pick something else up and get a lot more mileage just by
Paul: using, you know, stuff that people already have, but at the same time.
Paul: 10, anywhere from eight to 10 years ago. And you there were a lot fewer options
Paul: in terms of the different, you know, technologies you could use, right?
Paul: Most storage engines in 2013, 2014, 2015, were again designed for transactional workloads.
Paul: And the time series use cases in my mind, it's not a transactional workload,
Paul: like it's insert heavy with very large range scans, and you know,
Paul: very large data evictions.
Paul: And that kind of like makes it very different than other database workloads.
Paul: So, you know, I don't know that Go as a language really impacted all that stuff.
Paul: But again, like, it's hard to say.
Matthias: Was it still at a time where you had some parts of your code written in C++?
Matthias: Or would you say that was already beyond C++ for InfluxDB?
Paul: No, so the only thing we ever had in C++ was, well, the first versions of InfluxDB
Paul: used LevelDB as the storage engine, which is in C++.
Paul: And we had like a YACC parser for the query language, which was originally in C,
Paul: but then with 0.9 and then the releases after that, we built a parser for the query language in Go.
Paul: So it was a pure Go parser and 0.9 in the releases after where the one had no
Paul: C++ code whatsoever, right? They're all just pure Go codebases.
Matthias: That was around the time of Rust 1.0.
Matthias: I wonder when was the first time you heard about Rust?
Paul: I certainly knew about Rust pretty early on.
Paul: I definitely knew about it in by like 2015. By 1.0, I definitely knew about Rust.
Paul: But I had had hesitation to pick
Paul: up the language because I, so I had actually in 2010 and half of 2011,
Paul: I spent my time writing, actually, well, yeah, I spent my time writing Scala code.
Paul: So I picked up Scala and I became a big Scala enthusiast for like a year or so.
Paul: And then after working with it for a year, I realized that I actually didn't like language.
Paul: I did personally didn't like it because I thought it was like too complex.
Paul: There were too many things in Scala. It was basically trying to be all things
Paul: to all people was my feeling about it. If you're a Scala fan, I apologize.
Paul: It's just my personal, I didn't feel like great love for the language after
Paul: I'd worked with it for over a year and a half.
Paul: And my early impressions of Rust were that it was a very complex language with
Paul: a very large like footprint in terms of like syntax and all this other stuff.
Paul: And I had heard it was difficult to learn.
Paul: And to me, those were all markers of a language that wouldn't get any adoption.
Paul: I loved Go because you could learn the language in a day at the time, in 2013.
Paul: You could read through the guide and anybody who had worked with a few programming
Paul: languages before could pick up Go pretty easily.
Paul: And I thought that was a great strength of Go as a language,
Paul: I still do, in addition to great compile times, which I very much miss.
Paul: So basically, Rust, I kind of ignored it for the first, basically,
Paul: until, probably about 2018.
Paul: I knew about it, and I just was like, I'm not going to... One, obviously
Paul: I was completely fully invested in Go.
Paul: Everything we did was in Go, at least on the back end.
Paul: You know, on the front end, obviously, there's TypeScript and JavaScript and all that.
Paul: So, I didn't bother. And then in 2018, I basically thought, well.
Paul: Maybe there's something here.
Paul: Honestly, the thing that attracted me to Rust initially was just seeing reports
Paul: of people just building highly performance systems.
Paul: I was like, that's what lured me in to begin with.
Paul: I was just like, you know what? I haven't given this language a fair chance.
Paul: I've been kind of judgmental based on no actual knowledge.
Paul: So, I'm going to, like, commit to actually really trying to learn it.
Paul: So, I wanted to work through this book about programming interpreters.
Paul: It's Torsten Ball, I think his name. And he wrote it for with Go as the language that he used.
Paul: And it's basically programming, like, you know, creating a whole programming
Paul: language from scratch, right, right.
Paul: A lexer, a parser, an interpreter, and then a VM and all this other stuff.
Paul: I was like, I'll work my way through this book, but instead of doing the examples
Paul: in Go, I'll do the examples in Rust.
Paul: And I, you know, I read my way through the free online book,
Paul: Programming Rust or whatever, I can't remember the exact title of it.
Paul: And it did, it still wasn't clicking, you know, I'd basically started,
Paul: started it and got a certain way.
Paul: And then, you know, I kept getting tripped up by the borrow checker and stuff like that.
Paul: I was like, okay, obviously I still don't get it enough to do anything useful.
Paul: So then I got the O'Reilly book on programming Rust, and this again, like 2018.
Paul: And once I worked through a good chunk of that book, I started actually being
Paul: able to create something.
Paul: So I probably wrote, you know, I did this project, I probably wrote maybe like 2,500 lines of Rust.
Paul: None of it, I didn't use any crates, I didn't use, there was no multi-threaded
Paul: piece, there was no network programming piece, right?
Paul: So it was merely like kind of constrained problem space, but it was enough for
Paul: me to learn some of the key parts of the language.
Paul: And at that point, I just became super excited about Rust.
Paul: I was like, yeah, it's really hard to learn, but actually there are some things
Paul: here that I think are super compelling as far as the language goes.
Paul: But the thing is, I didn't have any reason to use it.
Paul: So I wrote a blog post about my experience and said I was excited about it, and that was that.
Paul: And then in, I think it was in the fall of 2019, the async await stuff finally landed.
Paul: Obviously that stuff had been in the works for a while, but it landed in a way
Paul: that you could actually use it for real.
Paul: At least I thought. People who were in Rust probably had already been doing it for a while.
Paul: But fall of 2019 is when I remember saying, oh, now that that's landed,
Paul: Rust is going to become basically maybe the language of first choice for building
Paul: high-performance server-side software, right? System software of all kinds.
Paul: And we had some big things we had to do with InfluxDB, and I thought,
Paul: well, maybe I should try writing, you know, this next, at least this core of
Paul: the database piece in Rust, which I can talk about as well.
Matthias: I would describe this learning process as hardcore mode.
Matthias: It feels like you are a very analytical person and a very thorough person in
Matthias: your thinking and learning process.
Matthias: Maybe that's just one observation, but I wonder how much of this is because
Matthias: of your background and maybe your way of thinking in systems,
Matthias: or has it always been like this, the learning process for new technologies?
Paul: No, I don't. I don't think so. I don't know. I'd like, I think I took a little
Paul: bit more of a deliberate approach with rust than I have with other things,
Paul: though, if that's true, either.
Paul: Because then, again, like with go like I, you know, I, when I picked up go,
Paul: I had a very specific project I was trying to do, which was build this like
Paul: back end, you know, database, the back end for airplane, right?
Paul: Which essentially is what became InfluxDB.
Paul: So I had this very specific project. I was like, I know I'm going to use Go.
Paul: I knew about LevelDB, so I need to figure out how to pull that in.
Paul: I used that little project as a way to like initially learn the language.
Paul: And again, like I worked through all the reading materials online,
Paul: like a book and stuff like that.
Paul: Generally, if I'm going to learn a language, I like to have a book.
Paul: I don't need a physical book, but I like to have a book to read through as well
Paul: as online documentation and stuff like that, but I prefer the structure of a book.
Paul: But even then, I guess the last language, well, Scala I had learned,
Paul: but again, that was around a specific project that I was doing.
Paul: In 2005, I was a C-sharp.net programmer, and I was no longer doing that.
Paul: And I quit work to go back to school. And I was like, okay, I can literally
Paul: pick up any new language I want.
Paul: So I went shopping around for languages. I was like, okay, there are two contenders here in 2005.
Paul: One was Python and one was Ruby. And I basically learned the basics of both of them.
Paul: And I ended up picking Ruby specifically because of Ruby on Rails.
Paul: I ended up being a Ruby programmer.
Paul: But again, I think the learning process with both of those was again, like pick up a book,
Paul: read through it, try and have, I mean, generally like if I'm trying to learn
Paul: a new thing, a new programming language or something like that like the best
Paul: way for me to learn it is to actually put it to use in something.
Paul: So I pick some sort of project. It doesn't have to be a project that is like
Paul: ever going to be used in any meaningful way, but it's something right.
Paul: So again, like when I learned rust it was all I'll implement this,
Paul: you know, this language the that that he that Torsen they've created in it for his book.
Paul: I'll implement that in rust and of course like all that code Like the code's
Paul: online, it lives, but nobody's using it.
Paul: It's something I created just to go into the trash.
Matthias: And the other thing I observed, which is also really cool, is that I saw a bit of a pattern.
Matthias: When you discovered Go, you went ahead and built the core of what would become
Matthias: new engine for InfluxDB in Go, and you kind of did the same in Rust, right?
Matthias: You locked yourself in for a while, you built this thing, and then you had a
Matthias: project and you gained more confidence by doing that.
Paul: Yeah, for sure. I mean, the so talking about the, you know, the rust implementation
Paul: of InfluxDB in, in late 2019, essentially, like, we were still very,
Paul: we were, we had all of our effort going into version 2.0, both the open source
Paul: and our commercial products.
Paul: And with 2.0, we tried to create, you know, branch out beyond just the database,
Paul: we tried to create this whole platform for working with time series data.
Paul: So it's like the database, all these other pieces of tasks.
Paul: And we also, at the same time, created a new programming language,
Paul: a scripting language called Flux for this, because we wanted people to basically
Paul: be able to inject code into the platform and execute arbitrary logic inside
Paul: of it, in addition to just declarative style queries against time series data.
Paul: And the other thing we did was we decided to go to shift our company to be a cloud-first company.
Paul: Previously with version 1 and beyond all the version 1 releases of InfluxDB,
Paul: we had a cloud-hosted platform, which is single-tenant, and we had an on-premise
Paul: commercial product, right?
Paul: And what we did was we shipped that commercial product, it's like an enterprise
Paul: software release cycle, right? We'd ship that like two to four times a year,
Paul: but really only like two feature-varying releases a year.
Paul: And then that would get rolled out to the single-tenant.
Paul: With 2.0, what I told the engineering team is like,
Paul: I want engineers to be able to ship code to production every business day,
Paul: which is very, very different than six month release cadence on a shipped package
Paul: software product, right?
Paul: So we shifted to that and as part of that, created this like multi-tenant platform
Paul: with all this other stuff.
Paul: So in late 2019, we were doing all this stuff And none of it was essentially
Paul: the core of the database.
Paul: The core of the database was still exactly the same as what it was for version one.
Paul: And we're doing all this work and we can see, I mean, customers are basically
Paul: asking us for a set of features that we can't deliver on because everybody else
Paul: is focused on all these other things, right?
Paul: The features they're asking for is infinite cardinality, right?
Paul: Cardinality is essentially like, I mentioned you have tag key value pairs.
Paul: Those are kind of like the dimensions that describe the data,
Paul: like a taggy value pair could be like what region or a server name or a process ID or like whatever.
Paul: So they want as the as the number of unique values for the tag values goes up,
Paul: that creates higher and higher cardinality in the implementation of InplexDB
Paul: at that time and many other time series databases.
Paul: High cardinality is a very big problem, right? the performance falls over,
Paul: it creates all sorts of problems.
Paul: So people wanted to be able to not worry about cardinality. They also wanted
Paul: to your data storage, right?
Paul: They wanted to be able to keep their historical data in cheap object storage
Paul: as opposed to on locally attached SSDs with provisioned IOPS and all this other stuff.
Paul: And a lot of people wanted, you know, so well, they wanted these like core database
Paul: features, they want better query performance as well.
Paul: A lot of people had been asking us for years if we were going to support SQL, right?
Paul: Our language before 2.0 is InfluxQL, which looks kind of like SQL, but it isn't.
Paul: And for somebody who's really, really familiar with SQL, the ways in which it's
Paul: not SQL can be quite frustrating.
Paul: Like, it's really powerful and useful for creating a basic time series query,
Paul: like you can express a basic time series query in InfluxQL faster and easier,
Paul: I think, than you can in SQL.
Paul: But when you want to do more advanced analytical things, InfluxQL doesn't have
Paul: the capabilities and we couldn't figure out how to bring that into the language.
Paul: And people kept asking us for all these features.
Paul: So in late 2019, early 2020, 20, I realized I was like, okay,
Paul: everybody's still focused on the 2.0 stuff and they need to do that.
Paul: But I need to like, start thinking about the new like the how we're actually
Paul: going to get the core of the database to deliver these things.
Paul: Infinite cardinality, tiered data storage, and better query performance.
Paul: I thought, well, that's basically totally a totally different database than
Paul: the database we have, right?
Paul: Like the entire database architecture
Paul: we had is around this inverted index and time series data store.
Paul: I was like, well, maybe I'll start prototyping some ideas and I'll do it in Rust.
Paul: So it was me and another engineer in the beginning, just for a couple of months.
Paul: And we start prototyping some
Paul: ideas. Another engineer joins us about three months in to this effort.
Paul: And at this stage, it's three of us and the engineering team at Influx at that
Paul: point is probably like 80 people.
Paul: So three people is just not like... We're not making a huge commitment here.
Paul: We're really just playing around in the lab trying to see if there's something interesting here.
Paul: And along the way, I saw some other things.
Paul: Now, at this stage, the goal is to solve those problems.
Paul: But I had some guesses about some tools that we'd want to use.
Paul: So one was like, I wanted to use as much existing open source software as we could.
Paul: So initially, my intention was that we'd actually take the query engine and
Paul: potentially storage engine from an existing open source project written in likely C++ or C.
Paul: So I was like, we're going to have a huge dependency on a big mountain of C++ code.
Paul: If we want a SQL engine, we're probably going to want to pull in an existing
Paul: engine, right? Which again, is going to be in C or C++.
Paul: So I was pretty sure we were going to have that.
Paul: And I also knew that I wanted the other guarantees that Rust provides.
Paul: So one, you can pull in C++ code and you don't to pay a performance penalty,
Paul: right? It's not like the Segoe bridge.
Paul: And to like I wanted, you know, all the good Rust stuff, you know,
Paul: like error handling, crates, concurrency, no garbage collector,
Paul: right? The borrow checker, all that stuff.
Paul: And over the course of 2020, we did more and more research.
Paul: And actually, we ended up deciding not to use a big mountain of and C++ code,
Paul: we picked up an existing project called Data Fusion, which is under the Apache Foundation.
Paul: It's actually a sub-project of Apache Arrow, and it's written in pure Rust.
Paul: And it's essentially a SQL parser, planner, optimizer, and execution engine. So, yeah.
Matthias: Early on when we talked about Rust, you mentioned Async Rust.
Matthias: And that felt like a pivotal moment for you when you realized,
Matthias: oh yeah, seems like the ecosystem, the community is moving towards Async Rust.
Matthias: And when you described the features that you wanted to build with Rust for InfluxDB,
Matthias: you didn't mention Async Rust.
Matthias: So I wonder, was it still on your radar or would you have changed something
Matthias: in the design of the new InfluxDB rewrite in Rust, if async didn't become a thing,
Matthias: or was it something extra on top, like a little cherry on top?
Paul: I mean, it definitely would have changed the way the code looks and all that other stuff.
Paul: But as I mentioned, like using object stores, the historical data store was one part.
Paul: So like, we need to be able to make many requests at the same time to object
Paul: storage and be able to pull back that data as it comes in, right?
Paul: Both sending data to it and reading data from it. But ultimately,
Paul: like, you know, InfluxDB 3 is a distributed system.
Paul: So, you we need a way to talk to a bunch of different servers at the same time
Paul: to make the whole thing work.
Paul: So when I saw Async Rust, I thought, okay, Async Rust is the thing that's going
Paul: to make it so that people can build server software that has to communicate
Paul: over the network to a lot of other systems more efficiently.
Paul: At least that was my thought. I don't know if that is true, but whatever.
Paul: That's what I was thinking.
Paul: So, you know, obviously we use the Tokyo runtime inside the server and async
Paul: is littered everywhere in our Rust code base.
Paul: So, it definitely like without async Rust, we would have changed what we had to build.
Paul: I'm thinking we probably would have had to write more code to deal with all
Paul: the network programming and all that other stuff. Whereas like using async rust,
Paul: we're able to pull in, you know, a variety of other crates and existing libraries to do those things.
Paul: But some of the things we've created, right? So like the object store create
Paul: crate that we use is one we created.
Paul: And then after we basically wanted, we knew we had had to work in,
Paul: you know, all the major cloud platforms with their versions of object store. server.
Paul: So Azure GCP AWS.
Paul: So we created this crate that kind of like has one API and behind that underlying
Paul: implementations for those things.
Paul: And we actually ended up donating that crate to the Apache Foundation.
Paul: So that's now part of I think it's a sub project in Arrow maybe or I don't I
Paul: don't know where it is, but it's it lives on its own separately. So.
Matthias: Correct me if I'm wrong, but it sounds like InfluxDB is fundamentally an I-O bound system.
Matthias: So you have a lot of I-O that you need to do either from the disk or over the
Matthias: network, and this is where Async Rust shines.
Matthias: Or would you also say, actually, it's also a CPU bound problem because we need
Matthias: to do quite a lot of computation to even know what query we are going to execute
Matthias: and where we're going to fetch the data and how we're going to aggregate it
Matthias: before we send it back to the client.
Paul: Yeah, so it's it's kind of both, unfortunately. I mean,
Paul: so the way the way it's organized is like data is organized into a logical database
Paul: and then below that a table and then below that a partition and beneath the
Paul: partition you have individual parquet files.
Paul: We use parquet is the persistence format for 3.0.
Paul: Now, the model of it is essentially when a query comes in, it evaluates the
Paul: query to find what partitions and what individual parquet files it needs to
Paul: execute against for the query.
Paul: But once it finds those, it just does a brute force query computation on that.
Paul: And it's like a best-in-class columnar database.
Paul: So when you think about columnar databases, This is generally their performance
Paul: you expect is about, you know, you can get through about a billion records per core per second.
Paul: So basically, but that's if you're churning through a lot of CPU.
Paul: And if you want, like to get through more, you have to figure out a way to spread
Paul: it out over multiple cores.
Paul: So it can become I mean, depending on the query that's being executed,
Paul: it can become quite CPU intensive.
Paul: But I will say, you know, we're running three dot O in production now. Right.
Paul: And the the design of it, it's not it's not like monolithic database.
Paul: Right. we've actually taken it and separated it out into different components.
Paul: So there's an ingestion tier, there's a compaction piece, and then there's the query tier.
Paul: So when a query comes in, it has to ask the ingestion tier for a buffer of data,
Paul: and then it has to either work with parquet data that's already in cache,
Paul: or it has to go to object storage to get it.
Paul: So if we look at traces for individual query response times,
Paul: most of the time, the query response time is dominated by network calls and
Paul: not by not by actual computation.
Paul: But again, I think that's kind of an artifact of how people are using the database
Paul: right now today for the kinds of queries they're executing.
Paul: I think as as we pick up workloads, there are more real time analytics workloads,
Paul: not just like metric data workloads that are querying, you know,
Paul: the last five minutes of data or the last hour of data.
Paul: People are going to get into larger, you know, more computationally intensive
Paul: queries and eventually maybe,
Paul: you know, more some more like data warehouse and OLAP workloads as well,
Paul: at which point I think there'll be a lot of compute stuff.
Matthias: The way I see it, you have this amazing little piece of technology now,
Matthias: you have your prototype, it seems to work as expected.
Matthias: Did you see the benefits, but at the same time, you said that you already have
Matthias: quite a big engineering team.
Matthias: Of course, granted, they work on different projects, so you don't have to bring
Matthias: everyone in at the same time, but I can imagine that it might take some convincing
Matthias: at least to do this buy-in, even though it's maybe not a risky one.
Matthias: But at least you want to mention the trade-offs to the team and maybe get their
Matthias: acceptance, get their buy-in before you start deploying Rust across InfluxDB
Matthias: and maybe also InfluxData?
Paul: Yeah, so this was a multi-year development effort, right?
Paul: So for 2020, essentially, it was a research project with me and one other person
Paul: initially, and then a third person.
Paul: And it was basically us creating it. And I definitely had to convince.
Paul: The first guy to work on it, I had to convince him that Rust was a good idea
Paul: and he was willing to give it a shot.
Paul: And the other guy coming in, he was actually a new hire, but his experience was in C++.
Paul: Recently he had been doing a bunch of Python stuff, but his database experience
Paul: was in C++ and he had had no Rust experience coming in.
Paul: So he had to learn Rust once he got hired. And initially, you know,
Paul: he wasn't sure, but after a year, he was definitely all in.
Paul: He's like, yes, Rust is obviously the best choice for this kind of a project.
Paul: So now he's a very large fan of the language. But I announced that we were working
Paul: on this new core of the database in November of 2020 in a talk I did.
Paul: And I said we were hiring. And basically, we got a bunch of inbound interest
Paul: because of the fact that it was written in Rust.
Paul: So at that point, we got some people who already knew the language,
Paul: definitely people who knew the language better than me, which I guess isn't super high bar.
Paul: But so we ended up hiring, we ended up building out the team to nine people
Paul: by essentially the beginning of March of 2021.
Paul: So those nine people worked on the database.
Paul: At that point, we'd selected all the tools we were going to use.
Paul: And we're like, at that stage, it was like creating a bunch of code and iterating
Paul: on the architectural design of the database.
Paul: And they worked on that for, you know, all of 2021 and basically up until probably August of 2022.
Paul: At which point we started trying to loop in a lot more of the engineering team.
Paul: By then, we saw that we really desperately needed to get this new core of the
Paul: database released to solve all these problems that our customers were asking us to solve.
Paul: And we needed to loop in more of the engineers to be able to get it deployed
Paul: meaningfully into production.
Paul: But at this stage, we still have a split in our engineering organization of
Paul: the people who are the rust programmers and the people who are the go programmers
Paul: and generally they're not there.
Paul: There's some people on the engineering team who like flip-flop between the two,
Paul: but for the most part it's like you're either in one camp or the other and we
Paul: I mean we're going to continue to run go code in production for for years.
Paul: So that's not that's not like going away.
Paul: So I imagine you know if I were to talk to you three years from now,
Paul: we're still going to be running some Go code in production.
Paul: We're still going to have some Go programmers on staff, but probably more Rust
Paul: programmers than Go programmers at that stage.
Matthias: That means initially, you hired people that already knew Rust,
Matthias: but then later on, you needed to train people on the job.
Matthias: The existing team that previously did most of the Go work also had to learn
Matthias: Rust, or at least the people that wanted to.
Matthias: I wonder how that went. Did you just test the experienced people to train the
Matthias: non-experienced people or did you get any outside help?
Matthias: Did you use any workshops, any training material? How was the process?
Paul: Yeah. So, so initially, you know, the three people working on the project knew nothing about Rust.
Paul: So, I mean, yeah, we were learning it as we went.
Paul: And then, you know, once we hired people in, I'd say like three of the new hires
Paul: that we made knew Rust well, and everybody else like had to learn it.
Paul: So we ended up bringing in Integer32, which is a consulting company,
Paul: Jake and Carol, to help us out.
Paul: One with like code, you know, writing code and doing code reviews and like helping us out there.
Paul: But we've also had them, they run like a multi-day class training session.
Paul: And we've run, had them run people through that a few times,
Paul: like different groups of engineers within our company through that.
Paul: So that, I think that's really, really helpful.
Paul: Like what the experience I didn't have in the first like six months I was personally
Paul: writing Rust was I had nobody I could ask really basic, dumb questions to you,
Paul: which I think would have helped my learning process a lot more.
Paul: You know, if like ChatGPT existed back then, I would have asked it all sorts
Paul: of dumb Rust questions that probably would have like shortened my learning time just like immensely.
Matthias: That's great that they came in and helped you out with that and probably focused
Matthias: on on the more intricate parts and just generally taking questions from the team.
Matthias: And I think this is how you should do it. But at the same time,
Matthias: I wonder, did you have confidence in the code base?
Matthias: Did you think that the code was idiomatic from the beginning?
Matthias: Did you do any pair programming or reviews, code reviews to ensure that the
Matthias: code was idiomatic, that you could extend it and it was kind of rustic, quote unquote?
Paul: Yeah, so we didn't do pair programming, but we always do code reviews, right?
Paul: So pull requests with code reviews to change stuff.
Paul: And we brought Carol and Jake in pretty early to do that specifically because,
Paul: I knew that we didn't have the expertise in-house and I was like,
Paul: I want to make sure that what we're creating isn't a complete mess.
Paul: I think I saw, yeah, I would worry about whether or not something was idiomatic
Paul: and it was like rustic, but at the same time, like I wouldn't worry about whether
Paul: or not the code was correct, right?
Paul: Like I feel like the compiler gives you a lot of confidence in terms of being
Paul: able to write code that, you know, is going to be relatively performant relatively correct.
Paul: Of course, like, mileage may vary, but yeah.
Matthias: Since you already touched on it, let's speak about performance.
Matthias: Do you see any big performance improvements with regards to Rust in comparison to Go?
Matthias: And if so, where?
Paul: Yeah, so I can't really, I can't really speak to that because we didn't,
Paul: we didn't take something in Go and create the exact same thing in Rust, right?
Paul: We totally changed the database architecture.
Paul: Nothing like it is the same. I guess one thing we created, which I don't know
Paul: if we've actually performance benchmarked this,
Paul: is InfluxDB has a line protocol, which is like a text protocol for writing in data.
Paul: And we did create a Rust parser for this LIME protocol data.
Paul: So we could probably do a benchmark of that versus the Go LIME protocol parser.
Paul: I'm not sure which would come out ahead, to be totally honest,
Paul: because the Go LIME protocol parser has been heavily, heavily optimized.
Paul: And that's one of those things where...
Paul: Yeah, I don't know. But the performance of version 3 of the database,
Paul: we've seen really, really great performance wins on, we can ingest far more data using fewer CPUs.
Paul: We get a compression win because we changed the file format to parquet.
Paul: We get better compression than we did before.
Paul: And depending on the query, the thing that it's generally not the actual compute,
Paul: it's how you, what you filter the query down to, right?
Paul: So in InfluxDB 1 and 2, if you had a query that touched, you know,
Paul: a million time series, that either wouldn't run or it'd just run incredibly slowly.
Paul: In version three, because the way we organize the data, you can execute a query
Paul: that touches a million time series and, you know, execute in milliseconds, it's fine.
Paul: But, at the same time, there are queries in version 3 that are slower.
Paul: So a query that asks for one specific time series, version 1 and 2 of InfluxDB
Paul: is optimized for that kind of use case, so that query is exceptionally fast.
Paul: Because what it does is it checks the inverted index, it finds the time series
Paul: ID, and then the way the data is laid out on disk is actually sorted by that time series and time.
Paul: So they can basically jump right to the file right to where the data is and read it.
Paul: So it's super super fast in three.
Paul: Identifies these large chunks of data and then it has to brute force an execution
Paul: path to pull out the specific individual time series from it.
Paul: That's generally fast enough, but we're definitely seeing cases where we need
Paul: to add like further either customization or optimizations so that people can
Paul: kind of like design a schema for that specific use case.
Paul: But again, like But none of this is really a Rust versus Go comparison,
Paul: because we didn't rewrite the same thing.
Paul: We created a totally different thing. It just happens to share the same name.
Paul: But as a database, it looks completely different under the hood.
Matthias: That means you chose Rust not solely because of performance improvements or
Matthias: because of the potential for performance improvements. There might be things
Matthias: that go beyond that, which brought you to rust.
Matthias: Maybe you can talk about this. I don't want to put words into your mouth or make any assumptions.
Matthias: So I want to hear it from you. What is the main benefit other than performance?
Paul: Yeah, so I mentioned this before, I think like the fearless concurrency, right?
Paul: Having, you know, eliminating data races essentially, which we had before,
Paul: like these really gnarly bugs in version 1 of Influx due to stuff like that,
Paul: which we just wouldn't have in Rust.
Paul: I think Crates is a great system for sharing libraries and sharing dependencies.
Paul: It's been a long time since I've actually been a Go programmer.
Paul: My Go experience predates modern Go package management, but I'll say from like
Paul: 2013, 2012 to 2016, when I was writing Go code.
Paul: The package management situation was a nightmare. It was terrible.
Paul: Again, my personal opinion. Maybe not everybody agrees.
Paul: But I viewed crates as like a really a great strength of Rust as a language having that there.
Paul: What else? The error handling, the way error handling works in Rust.
Paul: Again, like coming from Go, like checking for nil and like, you know,
Paul: we got bit by that so many times too.
Paul: And again, that's just something you just don't have to worry about it in Rust,
Paul: right? Check error types and whatever.
Paul: So, yeah, the error handling is something that was appealing to me. What else?
Matthias: Yeah. Sorry, did you have any problems with deadlocks in the old Go version?
Matthias: I know that Rust doesn't protect you from deadlocks necessarily,
Matthias: but I wonder if you see a difference here, or if that never was a problem with Go.
Paul: I mean deadlocks are always a problem, I don't care what language you're in, I guess.
Paul: So I don't think there's any more or less of a problem in Rust than they have in Go. Yeah.
Matthias: Nice. So, overall, I would say it
Matthias: was a great transition, a happy customer experience, a good case for it.
Matthias: And I wonder how you could use that language now, now that you have this tool
Matthias: in your hands to build something that goes beyond what you have right now, even.
Matthias: So what would InfluxDB look like in 2024, 2025, moving in? And what sort of
Matthias: features are you planning to build?
Paul: Yeah, so, I mean, right now, the version 3 of the database is actually just
Paul: the core of the database.
Paul: We still have the same, we brought over the version 1 API, so you can write
Paul: data to it as though it's a version 1 database.
Paul: You can query data from it as though it's a version 1 database,
Paul: right? You can have the same API, same query language, InfluxQL.
Paul: And we have the version 2 write interface. So the only new query interface we have now is Flight SQL.
Paul: It's a SQL-based query interface using Apache Aero Flight.
Paul: Now, with Parquet as the storage format, Parquet has support for structured nested data.
Paul: Basically, I can see us creating a new version 3.0 Write API that supports a
Paul: much richer data model than we do now.
Paul: It's being able to specify the units of something that's measured or specify
Paul: rates, an actual histogram type that is built in,
Paul: an underlying histogram type, and nested types like structs,
Paul: enums, arrays, those kinds of things.
Paul: So I see us adding those kinds of features on the right side and then on the
Paul: query side, enabling people to pull that stuff out.
Paul: And then one of the other things I really want to add in is essentially an embedded VM, right?
Paul: And at this stage, I'm thinking either Python or JavaScript.
Paul: I'm not sure which, but I really like to have an embedded VM in the actual database
Paul: so that you can do... Essentially, there are three different cases.
Paul: So one is essentially something like a database trigger where when you write
Paul: data in, it would execute a script that you specify to transform the write or
Paul: do something when you have a write come in.
Paul: Another is on persist. So basically, data gets buffered up in the database and
Paul: that gets persisted as a parquet file.
Paul: So I'd like to open an event there so that people could attach a script in the database.
Paul: They could transform the data, they could do things with it.
Paul: Either before it gets written out to modify it, or basically as a result of
Paul: it getting written out, you can build monitoring and alerting.
Paul: And then the other is essentially like a periodic system, like Cron essentially
Paul: in the database, but being able to specify that so that you can both...
Paul: Ideally, at that point, you're putting a script inside the database and you
Paul: can query data out of the database and have like a very, you know,
Paul: fast performing experience because you have this data locality.
Paul: So features around that I think and then enabling having essentially like a
Paul: shared service where people can share like scripts and stuff like that that
Paul: that are useful within the database.
Paul: So those are things I can see us doing. And then obviously like we are we're
Paul: big contributors to Apache Aero Rust, the Rust Arrow RS, as well as Data Fusion,
Paul: and I see us just contributing, continuing to contribute to that.
Paul: I really think like some of those building blocks, those Rust building blocks are going to be.
Paul: Are going to form the basis of a lot of like really interesting,
Paul: you know, large-scale data processing systems over the next,
Paul: you know, five to ten years, right?
Paul: Things that take place, you know, a lot of stuff that's currently in Java, right?
Paul: For doing large scale data analytics, analytical processing, stream processing.
Paul: And I really see, I mean, I think that's what's gonna be interesting to watch
Paul: over the next like 10 years is what new Rust projects are formed to start taking,
Paul: you know, market share away from some of these larger, older Java native projects.
Matthias: When you said you want a query language, the one thing that came to mind was WebAssembly.
Matthias: I wonder if you could support both languages, both Python and JavaScript,
Matthias: with something like WebAssembly. Have you considered that?
Paul: Yeah, my concern is that it's just not going to be performant enough because
Paul: you need to copy the data into the WebAssembly VM and all this other stuff and then copy it out.
Paul: I'm just worried that that bridge is going to be too expensive.
Paul: So it's an idea. I don't know. I'm not going to create the VM.
Paul: Whatever it is, it's going to be something that's already written.
Paul: We're not going to create the VM. We're not going to create the language.
Paul: We want to just use existing tooling that's out there and let people adopt,
Paul: use a language that they're already familiar with.
Matthias: Very exciting because it feels like you move away from pure time series data
Matthias: towards an analytical engine,
Matthias: if you like, or at least some sort of marketplace so that people can share scripts
Matthias: or share ideas or think about different ways how to use the data and make it
Matthias: work for their business and their use case.
Matthias: And I don't know any other players that are in that realm.
Matthias: And also, if there's any collaboration between those different companies,
Matthias: or if you would say no, inherently, this is such a special problem,
Matthias: you're working with very specific data here.
Matthias: It doesn't make any sense to even share scripts across databases or across these engines.
Paul: No, I mean, I think…,
Paul: I mean, so to me, the way I think of time series is not just like metric data,
Paul: really like the vision for InfluxDB has kind of always been that it would be
Paul: like the place for observational data of all kinds,
Paul: whether it's metric data, event data,
Paul: you know, logs and traces I view as time series data too, just like different kinds of structures.
Paul: So I think there, and And then sensor data, again, like depending on the sensor,
Paul: they're all different kinds of sensors.
Paul: So people's specific data problems are probably unique to them,
Paul: but the kinds of things you want to do against these, there are patterns that pop up, right?
Paul: So a common pattern in like monitoring alerting is like what's called a dead man alert, right?
Paul: Like if you have a sensor where you expect a reading every minute and it stops
Paul: reporting, that triggers an alert condition, right?
Paul: I think there are little bits of code, ideally, that people will be able to
Paul: share across these different things, these places where they identify,
Paul: oh, this is a common transformation that you need to make on time series data
Paul: to do this kind of analysis or whatever.
Paul: I think most of the data and analytics companies, real-time analytics companies
Paul: and all these other things, if they're collaborating at all,
Paul: it's around some underlying libraries. right?
Paul: So data fusion, for example, is used, you know, and developed not just by us,
Paul: but by a bunch of other companies, some of which we may end up competing with,
Paul: you know, at some point in the future.
Paul: And so there's like the actual underlying libraries and then they're kind of like the standards.
Paul: So within analytics, I think one of the standards that's starting to get traction,
Paul: is something called Apache Iceberg.
Paul: And it's basically a spec for a catalog of data in object storage.
Paul: So tabular data and object storage. It was originally created at Netflix and
Paul: then open source and put in the Apache Foundation and it looks like you know
Paul: some of the analytic companies are starting to pick up support we're certainly
Paul: going to support it as well.
Paul: It's a snowflake data bricks a bunch of mothers.
Matthias: Yeah as you can imagine there might be a few people who listen to this podcast
Matthias: who might get excited to either try rust or influx to be or a combination of
Matthias: those And I wonder if you want to give them any hints on how they can get started,
Matthias: how they can start contributing,
Matthias: the sorts of things that you would like to have help with, or even just feedback from the community.
Paul: Yeah, so for InfluxDB, right now the only way to experience InfluxDB 3 is through
Paul: our commercial products, right? our multi-tenant platform or our dedicated single
Paul: tenant platform or our on-premise software.
Paul: Because we don't have an open source release yet.
Paul: So for us, the best contributions we see are in the data fusion projects.
Paul: Although that can be kind of a hard project to get into because it's,
Paul: you know, query planner and optimizer and execution engine.
Paul: So sometimes that can be kind of an opaque project to pick up and learn about.
Paul: I think, you know, for people getting started with Rust, the best thing to do
Paul: is, you know, read through, read through a book, pick a project that you want
Paul: to actually implement with it and, and go with that.
Paul: And then if there are open source projects that you know of in Rust that are
Paul: interesting, it's always useful to look through other people's code and read
Paul: other people's code to see how they structure things.
Paul: So, yeah.
Matthias: And finally, is there anything that you would like to mention,
Matthias: something that you would like to promote or maybe something that that you want
Matthias: to share with the Rust community, this is your moment.
Paul: We're hiring Rust programmers. So yeah, no, and hopefully, like I said, soon, you know,
Paul: end of this year, early next year, we'll have an actual open source release
Paul: of InfluxDB 3 that was written in Rust.
Paul: And I think that'll be very, very exciting time. So if you want to contribute
Paul: to it and get paid for it, please reach out and paul at influxdata.com.
Matthias: Amazing. Yes. I love that you're so approachable and I love that you're such
Matthias: a nice conversation partner.
Matthias: It was a really great interview. Thanks a lot for taking the time, Paul.
Paul: Yeah. Yeah. Thank you for inviting me on.