Rust in Production Cover

Rust in Production

Matthias Endler
Since 12/2023 19 episodes

InfluxData with Paul Dix

Paul Dix, CTO of InfluxDB, talks about the open-source time series database's development, the decision to use Go and Rust, challenges of managing high data volumes, performance improvements, future plans, and the value of hands-on learning.

2023-12-14 69 min

Description & Show Notes

For our very first episode, we welcome a special guest, Paul Dix, the CTO of InfluxData.

He starts by giving us an overview of InfluxDB, an open source time series database used by developers to track server and application data. He takes us back to the early days of InfluxDB and explains how it came into existence, starting with the challenges they faced with their initial SaaS application and how they made the decision to repurpose their infrastructure and create this open source database. Paul also sheds light on the popularity of the programming language Go, which had a significant influence on their decision to use it for their project.
He takes us through the journey of InfluxDB's development and the improvements that have been made over the years. He emphasizes the enhancements made in versions 0.11 and 1.0 to improve performance and query capabilities. Moreover, he shares their decision to explore using Rust for certain parts of the project and the positive impact it has had. Moving forward, the conversation delves into the challenges of managing high volumes of data in time series databases.

Paul talks about the solutions they implemented, such as using BoltDB and developing the time-structured merge tree storage engine. We then dive into the decision to rewrite InfluxDB in Rust and the benefits it offers. He explains the improved performance, concurrency, and error handling that Rust brings to the table. Paul goes on to discuss the development process and how the engineering team has embraced Rust across their projects.

As the conversation progresses, we touch on the performance improvements in InfluxDB 3 and the future plans for the database. Paul shares their vision of incorporating additional features and integrating with other tools and languages. He also mentions InfluxDB's involvement in open-source projects like Apache Aero Rust and Data Fusion, highlighting their ambition to extend beyond metric data. Paul concludes the conversation by discussing the standards and libraries in analytics, the role of Apache Iceberg, and the collaboration among data and analytics companies. He provides advice for getting started with Rust and InfluxDB, urging listeners to engage in hands-on projects and learn from books and online documentation.

Thank you, Paul, for sharing your insights and expertise.

About InfluxData
InfluxData is the creator of InfluxDB, the leading open source time series database. They offer a cloud service, InfluxDB Cloud, and a commercial on-premise product, InfluxDB Enterprise (https://www.influxdata.com/products/influxdb-enterprise/).

About Paul Dix
Paul Dix is the founder and CTO of InfluxData (https://www.influxdata.com/). He has helped build software for startups, large companies and organizations like Microsoft, Google, McAfee, Thomson Reuters, and Air Force Space Command. He is the series editor for Addison Wesley's Data & Analytics book and video series (https://www.informit.com/imprint/series_detail.aspx?ser=4255387). In 2010 Paul wrote the book "Service Oriented Design with Ruby and Rails" (https://www.oreilly.com/library/view/service-oriented-design-with/9780321700124/) for Addison Wesley's Professional Ruby Series. In 2009 he started the NYC Machine Learning Meetup (https://www.meetup.com/nyc-machine-learning/), which now has over 13,000 members. Paul holds a degree in computer science from Columbia University. You can find Paul on Twitter (https://twitter.com/pauldix) and GitHub (https://github.com/pauldix).

Links
- InfluxData: https://www.influxdata.com/
- Careers at InfluxData: https://www.influxdata.com/careers/
- Blog post: Meet the Founders Who Rewrote in Rust: https://www.influxdata.com/blog/meet-founders-who-rewrote-in-rust/
- Reddit: Details and discussion on the Rust rewrite: https://www.reddit.com/r/rust/comments/16v13l5/influxdb_officially_made_the_switch_from_go_rust/
- Blog post: The Plan for InfluxDB 3.0 Open Source: https://www.influxdata.com/blog/the-plan-for-influxdb-3-0-open-source/

Transcript

Matthias
00:00:23
And without any further ado, Paul, can you introduce yourself and the company InfluxDB?
Paul
00:00:29
Yeah, so I'm Paul Dix. I'm the co-founder and CTO of InfluxData, which is the company that makes InfluxDB. So InfluxDB is an open source time series database, so developers can use it for tracking what's going on in their servers, their applications. Sensor data is a very big use case for us. We created the project in 2013 and had the 1.0 release in 2016, the 2.0 release in I think 2020, and we have now 3.0 available, not in open source yet, only under our commercial offerings as a cloud offering or as an on-premise software. And we'll have a 3.0 open source release hopefully an alpha by the end of this year or early next year.
Matthias
00:01:19
Awesome. So take us back to 2013. What was the ecosystem of time series databases back then? What was your personal need to start the project? And also how did InfluxDB and InfluxData start in general?
Paul
00:01:35
Yeah, so obviously the world of time series databases was very different in 2013. So, I started the company with my co-founder actually in 2012, and we started originally as a totally different thing, right? It was an application called Airplane, like E-R-R-P-L-A-N-E, and we were building like a SaaS application for doing real-time metrics and monitoring and application exceptions. So, you know, like in the same vein as like Datadog or New Relic or any one of these dozens of other companies. And to build that application, originally my goal there was I have a bit of a background in machine learning and my thought was I can apply machine learning techniques to all this monitoring data and do things like predictive analytics and anomaly detection and stuff. First, we had to build the application to get people to actually send us their data. And essentially, I had to build a time series API for storing all of this data and working with it at scale. And the first thing I built was essentially web services written in Scala using Cassandra as the backend database, and Redis as basically a real-time indexing layer and last value store. So I created that. We got into Y Combinator in the winter 13 batch. And around this time, I actually rewrote that back end in Go using LevelDB, which is a storage engine that was open sourced originally at Google. And I built like this basically all in one thing, and that thing became the back end for this SaaS application. We went through the program, I raised the seed round of funding, I was the founding CEO of the company, and then by the fall of 2013, we realized that that company just wasn't gonna work. Like we had maybe a dozen or so customers, they weren't even paying us enough to like pair infrastructure bills. Thankfully, at that time, we were still using free Amazon credits. And we realized that that wasn't going to take off. But we thought there was something there with the infrastructure that we had built, right? We could see a bunch of other companies kind of trying to roll their own, like create this from scratch, right? Every other server monitoring company, every user analytics company, every analytics company of any kind was kind of creating their own time series database, either from scratch or with a bunch of application code on top of a more general purpose database. And what I saw in large companies is that they were trying to solve this problem with open source tools. And at the time, the only thing available in 2013 was essentially graphite. So graphite or RRD. And graphite in 2013, it's an open source project written in Python. It's a metrics data store. It was originally built in 2008 by a team at Orbitz. And it hadn't had an actual release in over a year. Basically, it was a project that was largely orphaned, but it had a surprising amount of adoption in large companies that were trying to build their own monitoring stack using open source tools. The other thing I saw was I had actually had, prior to this in 2010, I was working for a FinTech startup. For that FinTech startup, I had to build a solution for working with time series data. This was like financial market data and stuff like that. And the set of tools that I used was the same set of tools I used to build the first back end for Airplane. So it could see this idea of time series, not just being about server monitoring data, but financial market data, user analytics data, real-time analytics data, and sensor data, which in 2013 wasn't really a big thing, but it was starting to become obvious that it probably would be. So, essentially what we did was we said, okay, well, the application airplane isn't taking off. So, let's maybe do a quick spike real quick. We'll take the same tools that we had built the backend of airplane with, take some of that code and pull it out and repurpose it as an open source project. And we built that and we basically took, I think, five or six weeks to build a prototype of this thing. And then in early 2013, in November of 2013, I arranged to give some talks at some meetups and I announced that we were working on it. We had like a basic documentation website that landed on the front page of Hacker News. And it was, you know, there all day and a bunch of people said like they were interested in it. And when I gave these talks, it was obvious pretty quickly that people were really interested in what we were building. It looked like we were trying to address a need that a lot of people had that nobody else was kind of paying attention to at that time.
Matthias
00:06:29
If you say that was around 2013, that was still pretty early for Go. So I wonder if it was one of the first major projects for Go outside of Google, even.
Paul
00:06:43
So yeah, Go 1.0 came out in I think March of 2012, which is when, like basically in the fall of 2012 is when I started using it pretty heavily. Docker was written in Go and that came out in early 2013. And then right around the same time in 2013, 2014, HashiCorp started creating their first set of tools in Go. So there was like this whole like set of projects around that time that started, you know, started in Go. But we definitely early on in, you know, late 2013, all of 2014 and 2015, we definitely got a lot of interest from developers because of the fact that we were written in Go.
Matthias
00:07:30
Yeah, I can guess that would be the case. And maybe you also worked a little bit with that hype and also generated some traction for your product early on because Go had a focus on maybe web scale or maybe even infrastructure products and that is still the case, of course. So that was sort of the most sensible tooling for this job, right?
Paul
00:07:57
Yeah, I mean, at the time when we started the project, the debate we had internally was, do we write the database in C, C++, or Go? And the reason we were hesitant at first to write it in Go was because of the garbage collector. A database is usually a very performance-sensitive piece of system software, generally you want to have total control over what happens. The thing is, there were other databases that were written in Java, which obviously has garbage collector. So I thought, well, we could just use the same techniques that those databases use if we end up running into problems with Go's garbage collector, right? Their techniques are basically hide the memory away from the garbage collector and manage it yourself in some place, right? In some unsafe place. And our bet at that time was, we knew Go more than we knew either C or C++. And we just thought, you know, it was me, my co-founder, and one other person. And we just thought, you know, our biggest problem right now is trying to make software that anybody will want to use. So we figured we could do more faster with Go and that the language, you know, in 2013, our bet was the language will just keep improving and we'll get the benefits of those improvements along the way. And that certainly ended up being the case over the, you know, over the next 10 years. I mean, most, I would say, you know, 3.0 is written in Rust. So we could obviously, we'll get to that. But most of what we run right now for our customers and in production is Go code, right? We have yet to transition a lot of the business over to the Rust code base that we now have.
Matthias
00:09:44
Yeah. I got in touch with InfluxDB around 1.0. It was even before that, it was somewhere around 0.8. That was the first version we used in production at Trivago. And InfluxDB was killing it. And it was so performant in comparison to what we had before, which was, of course, graphite and something custom. And then all of a sudden, you had access to something that scaled to all of our servers. We ran this thing for about a year on a single machine, and it handled the entire traffic of Trivago. Only later, we went with Influx Cluster and so on. But But the point that I'm trying to make is, what you built was purely magical, I would even say, because something like that didn't exist before. And you bargained from storage engines like LevelDB, or I guess later you switched to BulkDB as well. Maybe you can talk a little bit about this and say, why did you move away from LevelDB and why did you look for alternatives in the end? And just in general, what were some of the main problems that you faced building something like this in Go, or any other language for that matter?
Paul
00:11:10
Yeah, so LevelDB is essentially a key value store where the keyspace is ordered. So you can do range scans on the keyspace and get the values. It's written in C++, which if you're bringing that into a Go program, you know you pay a price for for for making for for the C-Go bridge, So, the way, so one of the common things in a time series use case is people want to, people like data is super valuable, like around the time when it's ingested and generated and the value of the data quickly falls off a cliff, right? Particularly in like monitoring and stuff like that. A lot of times like people don't need the raw high precision data longer than say 10 days, right? So one of the, and the thing is the volume of data that you're ingesting, the number of individual records that you're ingesting is way, way higher than you see in, you know, traditional like transactional database workloads, right? If you have a customer database, you're not, you don't generally have like a billion records a day that you're ingesting, right? So one of the problems in, in the time series space is figuring out how to manage the lifecycle of that data and a lot of times how to basically just evict that data from the database as it ages out, right? So people wanted to say, I want to keep my data around for seven days or I want to keep my data around for 90 days and I want it to automatically go away when that's done. Now, in a traditional database, if you did this the naive way, which would be you issue a delete for every record 90 days after you you insert it, your database would just completely fall over. It's not designed to delete literally every single record that you feed into the database. It just causes the performance to completely fall off the cliff. So generally, if you're using a traditional transactional database, the way you organize the data is you say, I'm going to create an actual table for each day of data. And then when the day ages out, I will drop the table. Dropping a table is cheap because you're literally just like, you know, getting rid of the files. So in level DB, essentially what we would have to do is for each period of time. And for us, I think at that time we had a period of time be seven days. We would create a new level DB database. Now level DB database is actually, you know, thousands and thousands thousands of small individual little files that it manages there, right? So one of the early problems that users had with InfluxDB was they would hit file limit, open file handle limits, right? Literally we had to keep telling people like, oh, you need to adjust these up, which is I guess more standard now, but I can't tell you how many bug reports we got because people didn't have their file handle limits high enough. So essentially, we created a level DB database for each one of these things. And then, you know, we would drop the entire the entire database when that when it aged out. And it wasn't, we felt like it wasn't an ideal structure. And that and like the Segoe bridge wasn't ideal either. And there are other things like doing backups was really, really hard at that time. Like, I don't know that we ever even had backups, technically for version 0.8, which still used level DB. So essentially we wanted to solve some of those problems. So when we created the next version, 0.9, we thought, oh, we'll use BoltDB, which is also a key value store. It's written in Go, but it's a different structure. So level DB is what's called a log structured merge tree, which is actually something that's fairly good for this kind of use case. BoltDB is what's called a copy-on-write B plus tree, which is fantastic if you have a high read workload. It's kind of ruinous if you have a super high insert workload, which we do. We didn't realize this early on, so we built around it. And actually, the 0.9 series of releases was essentially two things that we were getting done at that time. One, we were changing the data model to the data model that you see today in InfluxDB, which is you have a measurement, you have tags, which are key value pairs where the values are strings, and then you have fields where the key is, the field key is the name of the thing and then the value is whatever kind of value you're measuring, which could be a float, an int, a Boolean, or a string. So with version 0.9, what we did is we introduced this data model and we basically split the database out into two separate things. One, which is an inverted index that maps metadata to an underlying time series, and the other is time series data, which in 0.9 we were trying to keep in Bolty B. We worked on that for probably about eight months until we realized that Bolty B just wasn't going to work for what we were trying to do. So in over, what, early September, over like a holiday weekend, I kind of like, prototyped out this idea for what I was thinking of as the time-structured merge tree, which is the storage engine we ended up building. It's basically a storage engine that we designed that's kind of like a log-structured merge tree, but optimized for this time series use case, how we saw data flowing in and all this other stuff. So this was in 2015 that we were doing all of this work. September of 2015, I prototyped this out. It takes a few months and we realized okay, this is probably actually going to be really good. So by March of 2016, we had a release of InfluxDB, which I think was the 0.11 release, where we had the first version of that storage engine paired with this new data model and with the same old query language that we had, and we saw at that stage performance that was really, really exceptional, right? We had much better performance than we ever had with previous versions of InfluxDB, and the data model kind of mapped with how people were thinking about their data. So in version 0.8, we had a data model where you had like a table or a measurement name, whatever you want to call it, and then a bunch of columns. And it was really easy in 0.8 for people to create a schema that gave them very, very poor performance. And again, how this would come through is people would log bugs and they'd be like, oh, the performance sucks after I insert 10 million rows into the database. And we're like, well, that's because your schema doesn't match how you're trying to query your data. So when we introduced the tags concept, the goal there was to create. Drive users into modeling their data in such a way that they would get good query performance out of the box. We released version 1.0 in September 2016. With that custom storage engine that we had built. And that was looking good. But even with that release, as I mentioned, it's two data stores. It's an inverted index for the metadata and then the time series data store. In that version, the time series data was kept on disk and the inverted index was built only in memory. So basically, when the database server booted up, it would build the inverted index on the fly from the underlying time series data. And what that meant was as the database got bigger and bigger, you got more and more time series data, the boot up process would take longer. So for the next, I'd say probably for the rest of 2016 and the first half of 2017, we put a ton of work into adding basically the time series index, which was essentially an inverted index on disk that we could use. And when the combination of those two, TSM and TSI, formed the core storage engine that actually underlies 1.0 and also InfluxDB 2.0, it's the same storage engine. And, yeah.
Matthias
00:19:33
So while you explained all this, I wondered how much of it was informed by your choice of programming language and how much was just very general, algorithms work or architecture work, and maybe you can shed some light on this. Would you say that Go impacted the design, or would you say that was completely agnostic of Go, especially with regards to garbage collector times or high cardinality?
Paul
00:20:06
I think the only way I think Go impacted the design, it didn't impact the architectural design. again, most of the work we're doing here would have been the same stuff we would have had to do regardless of the language. If we had written it C, C++, or then later Rust. There's still these core things that you have to figure out. I will say that the choice of Go as a language definitely impacted what we decided to use as dependencies. If we had written the entire thing in C++, we almost certainly would have just taken an existing storage engine from somewhere else and used it, right? Because that's way, way easier. And another part of this is essentially like a learning process as we went, right? Well, knowing what I know now, I would not build a storage engine. Like that's not something I would spend my time doing. You can basically pick something else up and get a lot more mileage just by using, you know, stuff that people already have, but at the same time. 10, anywhere from eight to 10 years ago. And you there were a lot fewer options in terms of the different, you know, technologies you could use, right? Most storage engines in 2013, 2014, 2015, were again designed for transactional workloads. And the time series use cases in my mind, it's not a transactional workload, like it's insert heavy with very large range scans, and you know, very large data evictions. And that kind of like makes it very different than other database workloads. So, you know, I don't know that Go as a language really impacted all that stuff. But again, like, it's hard to say.
Matthias
00:21:56
Was it still at a time where you had some parts of your code written in C++? Or would you say that was already beyond C++ for InfluxDB?
Paul
00:22:07
No, so the only thing we ever had in C++ was, well, the first versions of InfluxDB used LevelDB as the storage engine, which is in C++. And we had like a YACC parser for the query language, which was originally in C, but then with 0.9 and then the releases after that, we built a parser for the query language in Go. So it was a pure Go parser and 0.9 in the releases after where the one had no C++ code whatsoever, right? They're all just pure Go codebases.
Matthias
00:22:45
That was around the time of Rust 1.0. I wonder when was the first time you heard about Rust?
Paul
00:22:54
I certainly knew about Rust pretty early on. I definitely knew about it in by like 2015. By 1.0, I definitely knew about Rust. But I had had hesitation to pick up the language because I, so I had actually in 2010 and half of 2011, I spent my time writing, actually, well, yeah, I spent my time writing Scala code. So I picked up Scala and I became a big Scala enthusiast for like a year or so. And then after working with it for a year, I realized that I actually didn't like language. I did personally didn't like it because I thought it was like too complex. There were too many things in Scala. It was basically trying to be all things to all people was my feeling about it. If you're a Scala fan, I apologize. It's just my personal, I didn't feel like great love for the language after I'd worked with it for over a year and a half. And my early impressions of Rust were that it was a very complex language with a very large like footprint in terms of like syntax and all this other stuff. And I had heard it was difficult to learn. And to me, those were all markers of a language that wouldn't get any adoption. I loved Go because you could learn the language in a day at the time, in 2013. You could read through the guide and anybody who had worked with a few programming languages before could pick up Go pretty easily. And I thought that was a great strength of Go as a language, I still do, in addition to great compile times, which I very much miss. So basically, Rust, I kind of ignored it for the first, basically, until, probably about 2018. I knew about it, and I just was like, I'm not going to... One, obviously I was completely fully invested in Go. Everything we did was in Go, at least on the back end. You know, on the front end, obviously, there's TypeScript and JavaScript and all that. So, I didn't bother. And then in 2018, I basically thought, well. Maybe there's something here. Honestly, the thing that attracted me to Rust initially was just seeing reports of people just building highly performance systems. I was like, that's what lured me in to begin with. I was just like, you know what? I haven't given this language a fair chance. I've been kind of judgmental based on no actual knowledge. So, I'm going to, like, commit to actually really trying to learn it. So, I wanted to work through this book about programming interpreters. It's Torsten Ball, I think his name. And he wrote it for with Go as the language that he used. And it's basically programming, like, you know, creating a whole programming language from scratch, right, right. A lexer, a parser, an interpreter, and then a VM and all this other stuff. I was like, I'll work my way through this book, but instead of doing the examples in Go, I'll do the examples in Rust. And I, you know, I read my way through the free online book, Programming Rust or whatever, I can't remember the exact title of it. And it did, it still wasn't clicking, you know, I'd basically started, started it and got a certain way. And then, you know, I kept getting tripped up by the borrow checker and stuff like that. I was like, okay, obviously I still don't get it enough to do anything useful. So then I got the O'Reilly book on programming Rust, and this again, like 2018. And once I worked through a good chunk of that book, I started actually being able to create something. So I probably wrote, you know, I did this project, I probably wrote maybe like 2,500 lines of Rust. None of it, I didn't use any crates, I didn't use, there was no multi-threaded piece, there was no network programming piece, right? So it was merely like kind of constrained problem space, but it was enough for me to learn some of the key parts of the language. And at that point, I just became super excited about Rust. I was like, yeah, it's really hard to learn, but actually there are some things here that I think are super compelling as far as the language goes. But the thing is, I didn't have any reason to use it. So I wrote a blog post about my experience and said I was excited about it, and that was that. And then in, I think it was in the fall of 2019, the async await stuff finally landed. Obviously that stuff had been in the works for a while, but it landed in a way that you could actually use it for real. At least I thought. People who were in Rust probably had already been doing it for a while. But fall of 2019 is when I remember saying, oh, now that that's landed, Rust is going to become basically maybe the language of first choice for building high-performance server-side software, right? System software of all kinds. And we had some big things we had to do with InfluxDB, and I thought, well, maybe I should try writing, you know, this next, at least this core of the database piece in Rust, which I can talk about as well.
Matthias
00:28:22
I would describe this learning process as hardcore mode. It feels like you are a very analytical person and a very thorough person in your thinking and learning process. Maybe that's just one observation, but I wonder how much of this is because of your background and maybe your way of thinking in systems, or has it always been like this, the learning process for new technologies?
Paul
00:28:55
No, I don't. I don't think so. I don't know. I'd like, I think I took a little bit more of a deliberate approach with rust than I have with other things, though, if that's true, either. Because then, again, like with go like I, you know, I, when I picked up go, I had a very specific project I was trying to do, which was build this like back end, you know, database, the back end for airplane, right? Which essentially is what became InfluxDB. So I had this very specific project. I was like, I know I'm going to use Go. I knew about LevelDB, so I need to figure out how to pull that in. I used that little project as a way to like initially learn the language. And again, like I worked through all the reading materials online, like a book and stuff like that. Generally, if I'm going to learn a language, I like to have a book. I don't need a physical book, but I like to have a book to read through as well as online documentation and stuff like that, but I prefer the structure of a book. But even then, I guess the last language, well, Scala I had learned, but again, that was around a specific project that I was doing. In 2005, I was a C-sharp.net programmer, and I was no longer doing that. And I quit work to go back to school. And I was like, okay, I can literally pick up any new language I want. So I went shopping around for languages. I was like, okay, there are two contenders here in 2005. One was Python and one was Ruby. And I basically learned the basics of both of them. And I ended up picking Ruby specifically because of Ruby on Rails. I ended up being a Ruby programmer. But again, I think the learning process with both of those was again, like pick up a book, read through it, try and have, I mean, generally like if I'm trying to learn a new thing, a new programming language or something like that like the best way for me to learn it is to actually put it to use in something. So I pick some sort of project. It doesn't have to be a project that is like ever going to be used in any meaningful way, but it's something right. So again, like when I learned rust it was all I'll implement this, you know, this language the that that he that Torsen they've created in it for his book. I'll implement that in rust and of course like all that code Like the code's online, it lives, but nobody's using it. It's something I created just to go into the trash.
Matthias
00:31:27
And the other thing I observed, which is also really cool, is that I saw a bit of a pattern. When you discovered Go, you went ahead and built the core of what would become new engine for InfluxDB in Go, and you kind of did the same in Rust, right? You locked yourself in for a while, you built this thing, and then you had a project and you gained more confidence by doing that.
Paul
00:31:55
Yeah, for sure. I mean, the so talking about the, you know, the rust implementation of InfluxDB in, in late 2019, essentially, like, we were still very, we were, we had all of our effort going into version 2.0, both the open source and our commercial products. And with 2.0, we tried to create, you know, branch out beyond just the database, we tried to create this whole platform for working with time series data. So it's like the database, all these other pieces of tasks. And we also, at the same time, created a new programming language, a scripting language called Flux for this, because we wanted people to basically be able to inject code into the platform and execute arbitrary logic inside of it, in addition to just declarative style queries against time series data. And the other thing we did was we decided to go to shift our company to be a cloud-first company. Previously with version 1 and beyond all the version 1 releases of InfluxDB, we had a cloud-hosted platform, which is single-tenant, and we had an on-premise commercial product, right? And what we did was we shipped that commercial product, it's like an enterprise software release cycle, right? We'd ship that like two to four times a year, but really only like two feature-varying releases a year. And then that would get rolled out to the single-tenant. With 2.0, what I told the engineering team is like, I want engineers to be able to ship code to production every business day, which is very, very different than six month release cadence on a shipped package software product, right? So we shifted to that and as part of that, created this like multi-tenant platform with all this other stuff. So in late 2019, we were doing all this stuff And none of it was essentially the core of the database. The core of the database was still exactly the same as what it was for version one. And we're doing all this work and we can see, I mean, customers are basically asking us for a set of features that we can't deliver on because everybody else is focused on all these other things, right? The features they're asking for is infinite cardinality, right? Cardinality is essentially like, I mentioned you have tag key value pairs. Those are kind of like the dimensions that describe the data, like a taggy value pair could be like what region or a server name or a process ID or like whatever. So they want as the as the number of unique values for the tag values goes up, that creates higher and higher cardinality in the implementation of InplexDB at that time and many other time series databases. High cardinality is a very big problem, right? the performance falls over, it creates all sorts of problems. So people wanted to be able to not worry about cardinality. They also wanted to your data storage, right? They wanted to be able to keep their historical data in cheap object storage as opposed to on locally attached SSDs with provisioned IOPS and all this other stuff. And a lot of people wanted, you know, so well, they wanted these like core database features, they want better query performance as well. A lot of people had been asking us for years if we were going to support SQL, right? Our language before 2.0 is InfluxQL, which looks kind of like SQL, but it isn't. And for somebody who's really, really familiar with SQL, the ways in which it's not SQL can be quite frustrating. Like, it's really powerful and useful for creating a basic time series query, like you can express a basic time series query in InfluxQL faster and easier, I think, than you can in SQL. But when you want to do more advanced analytical things, InfluxQL doesn't have the capabilities and we couldn't figure out how to bring that into the language. And people kept asking us for all these features. So in late 2019, early 2020, 20, I realized I was like, okay, everybody's still focused on the 2.0 stuff and they need to do that. But I need to like, start thinking about the new like the how we're actually going to get the core of the database to deliver these things. Infinite cardinality, tiered data storage, and better query performance. I thought, well, that's basically totally a totally different database than the database we have, right? Like the entire database architecture we had is around this inverted index and time series data store. I was like, well, maybe I'll start prototyping some ideas and I'll do it in Rust. So it was me and another engineer in the beginning, just for a couple of months. And we start prototyping some ideas. Another engineer joins us about three months in to this effort. And at this stage, it's three of us and the engineering team at Influx at that point is probably like 80 people. So three people is just not like... We're not making a huge commitment here. We're really just playing around in the lab trying to see if there's something interesting here. And along the way, I saw some other things. Now, at this stage, the goal is to solve those problems. But I had some guesses about some tools that we'd want to use. So one was like, I wanted to use as much existing open source software as we could. So initially, my intention was that we'd actually take the query engine and potentially storage engine from an existing open source project written in likely C++ or C. So I was like, we're going to have a huge dependency on a big mountain of C++ code. If we want a SQL engine, we're probably going to want to pull in an existing engine, right? Which again, is going to be in C or C++. So I was pretty sure we were going to have that. And I also knew that I wanted the other guarantees that Rust provides. So one, you can pull in C++ code and you don't to pay a performance penalty, right? It's not like the Segoe bridge. And to like I wanted, you know, all the good Rust stuff, you know, like error handling, crates, concurrency, no garbage collector, right? The borrow checker, all that stuff. And over the course of 2020, we did more and more research. And actually, we ended up deciding not to use a big mountain of and C++ code, we picked up an existing project called Data Fusion, which is under the Apache Foundation. It's actually a sub-project of Apache Arrow, and it's written in pure Rust. And it's essentially a SQL parser, planner, optimizer, and execution engine. So, yeah.
Matthias
00:38:59
Early on when we talked about Rust, you mentioned Async Rust. And that felt like a pivotal moment for you when you realized, oh yeah, seems like the ecosystem, the community is moving towards Async Rust. And when you described the features that you wanted to build with Rust for InfluxDB, you didn't mention Async Rust. So I wonder, was it still on your radar or would you have changed something in the design of the new InfluxDB rewrite in Rust, if async didn't become a thing, or was it something extra on top, like a little cherry on top?
Paul
00:39:39
I mean, it definitely would have changed the way the code looks and all that other stuff. But as I mentioned, like using object stores, the historical data store was one part. So like, we need to be able to make many requests at the same time to object storage and be able to pull back that data as it comes in, right? Both sending data to it and reading data from it. But ultimately, like, you know, InfluxDB 3 is a distributed system. So, you we need a way to talk to a bunch of different servers at the same time to make the whole thing work. So when I saw Async Rust, I thought, okay, Async Rust is the thing that's going to make it so that people can build server software that has to communicate over the network to a lot of other systems more efficiently. At least that was my thought. I don't know if that is true, but whatever. That's what I was thinking. So, you know, obviously we use the Tokyo runtime inside the server and async is littered everywhere in our Rust code base. So, it definitely like without async Rust, we would have changed what we had to build. I'm thinking we probably would have had to write more code to deal with all the network programming and all that other stuff. Whereas like using async rust, we're able to pull in, you know, a variety of other crates and existing libraries to do those things. But some of the things we've created, right? So like the object store create crate that we use is one we created. And then after we basically wanted, we knew we had had to work in, you know, all the major cloud platforms with their versions of object store. server. So Azure GCP AWS. So we created this crate that kind of like has one API and behind that underlying implementations for those things. And we actually ended up donating that crate to the Apache Foundation. So that's now part of I think it's a sub project in Arrow maybe or I don't I don't know where it is, but it's it lives on its own separately. So.
Matthias
00:41:51
Correct me if I'm wrong, but it sounds like InfluxDB is fundamentally an I-O bound system. So you have a lot of I-O that you need to do either from the disk or over the network, and this is where Async Rust shines. Or would you also say, actually, it's also a CPU bound problem because we need to do quite a lot of computation to even know what query we are going to execute and where we're going to fetch the data and how we're going to aggregate it before we send it back to the client.
Paul
00:42:25
Yeah, so it's it's kind of both, unfortunately. I mean, so the way the way it's organized is like data is organized into a logical database and then below that a table and then below that a partition and beneath the partition you have individual parquet files. We use parquet is the persistence format for 3.0. Now, the model of it is essentially when a query comes in, it evaluates the query to find what partitions and what individual parquet files it needs to execute against for the query. But once it finds those, it just does a brute force query computation on that. And it's like a best-in-class columnar database. So when you think about columnar databases, This is generally their performance you expect is about, you know, you can get through about a billion records per core per second. So basically, but that's if you're churning through a lot of CPU. And if you want, like to get through more, you have to figure out a way to spread it out over multiple cores. So it can become I mean, depending on the query that's being executed, it can become quite CPU intensive. But I will say, you know, we're running three dot O in production now. Right. And the the design of it, it's not it's not like monolithic database. Right. we've actually taken it and separated it out into different components. So there's an ingestion tier, there's a compaction piece, and then there's the query tier. So when a query comes in, it has to ask the ingestion tier for a buffer of data, and then it has to either work with parquet data that's already in cache, or it has to go to object storage to get it. So if we look at traces for individual query response times, most of the time, the query response time is dominated by network calls and not by not by actual computation. But again, I think that's kind of an artifact of how people are using the database right now today for the kinds of queries they're executing. I think as as we pick up workloads, there are more real time analytics workloads, not just like metric data workloads that are querying, you know, the last five minutes of data or the last hour of data. People are going to get into larger, you know, more computationally intensive queries and eventually maybe, you know, more some more like data warehouse and OLAP workloads as well, at which point I think there'll be a lot of compute stuff.
Matthias
00:45:03
The way I see it, you have this amazing little piece of technology now, you have your prototype, it seems to work as expected. Did you see the benefits, but at the same time, you said that you already have quite a big engineering team. Of course, granted, they work on different projects, so you don't have to bring everyone in at the same time, but I can imagine that it might take some convincing at least to do this buy-in, even though it's maybe not a risky one. But at least you want to mention the trade-offs to the team and maybe get their acceptance, get their buy-in before you start deploying Rust across InfluxDB and maybe also InfluxData?
Paul
00:45:44
Yeah, so this was a multi-year development effort, right? So for 2020, essentially, it was a research project with me and one other person initially, and then a third person. And it was basically us creating it. And I definitely had to convince. The first guy to work on it, I had to convince him that Rust was a good idea and he was willing to give it a shot. And the other guy coming in, he was actually a new hire, but his experience was in C++. Recently he had been doing a bunch of Python stuff, but his database experience was in C++ and he had had no Rust experience coming in. So he had to learn Rust once he got hired. And initially, you know, he wasn't sure, but after a year, he was definitely all in. He's like, yes, Rust is obviously the best choice for this kind of a project. So now he's a very large fan of the language. But I announced that we were working on this new core of the database in November of 2020 in a talk I did. And I said we were hiring. And basically, we got a bunch of inbound interest because of the fact that it was written in Rust. So at that point, we got some people who already knew the language, definitely people who knew the language better than me, which I guess isn't super high bar. But so we ended up hiring, we ended up building out the team to nine people by essentially the beginning of March of 2021. So those nine people worked on the database. At that point, we'd selected all the tools we were going to use. And we're like, at that stage, it was like creating a bunch of code and iterating on the architectural design of the database. And they worked on that for, you know, all of 2021 and basically up until probably August of 2022. At which point we started trying to loop in a lot more of the engineering team. By then, we saw that we really desperately needed to get this new core of the database released to solve all these problems that our customers were asking us to solve. And we needed to loop in more of the engineers to be able to get it deployed meaningfully into production. But at this stage, we still have a split in our engineering organization of the people who are the rust programmers and the people who are the go programmers and generally they're not there. There's some people on the engineering team who like flip-flop between the two, but for the most part it's like you're either in one camp or the other and we I mean we're going to continue to run go code in production for for years. So that's not that's not like going away. So I imagine you know if I were to talk to you three years from now, we're still going to be running some Go code in production. We're still going to have some Go programmers on staff, but probably more Rust programmers than Go programmers at that stage.
Matthias
00:48:46
That means initially, you hired people that already knew Rust, but then later on, you needed to train people on the job. The existing team that previously did most of the Go work also had to learn Rust, or at least the people that wanted to. I wonder how that went. Did you just test the experienced people to train the non-experienced people or did you get any outside help? Did you use any workshops, any training material? How was the process?
Paul
00:49:17
Yeah. So, so initially, you know, the three people working on the project knew nothing about Rust. So, I mean, yeah, we were learning it as we went. And then, you know, once we hired people in, I'd say like three of the new hires that we made knew Rust well, and everybody else like had to learn it. So we ended up bringing in Integer32, which is a consulting company, Jake and Carol, to help us out. One with like code, you know, writing code and doing code reviews and like helping us out there. But we've also had them, they run like a multi-day class training session. And we've run, had them run people through that a few times, like different groups of engineers within our company through that. So that, I think that's really, really helpful. Like what the experience I didn't have in the first like six months I was personally writing Rust was I had nobody I could ask really basic, dumb questions to you, which I think would have helped my learning process a lot more. You know, if like ChatGPT existed back then, I would have asked it all sorts of dumb Rust questions that probably would have like shortened my learning time just like immensely.
Matthias
00:50:40
That's great that they came in and helped you out with that and probably focused on on the more intricate parts and just generally taking questions from the team. And I think this is how you should do it. But at the same time, I wonder, did you have confidence in the code base? Did you think that the code was idiomatic from the beginning? Did you do any pair programming or reviews, code reviews to ensure that the code was idiomatic, that you could extend it and it was kind of rustic, quote unquote?
Paul
00:51:14
Yeah, so we didn't do pair programming, but we always do code reviews, right? So pull requests with code reviews to change stuff. And we brought Carol and Jake in pretty early to do that specifically because, I knew that we didn't have the expertise in-house and I was like, I want to make sure that what we're creating isn't a complete mess. I think I saw, yeah, I would worry about whether or not something was idiomatic and it was like rustic, but at the same time, like I wouldn't worry about whether or not the code was correct, right? Like I feel like the compiler gives you a lot of confidence in terms of being able to write code that, you know, is going to be relatively performant relatively correct. Of course, like, mileage may vary, but yeah.
Matthias
00:52:00
Since you already touched on it, let's speak about performance. Do you see any big performance improvements with regards to Rust in comparison to Go? And if so, where?
Paul
00:52:12
Yeah, so I can't really, I can't really speak to that because we didn't, we didn't take something in Go and create the exact same thing in Rust, right? We totally changed the database architecture. Nothing like it is the same. I guess one thing we created, which I don't know if we've actually performance benchmarked this, is InfluxDB has a line protocol, which is like a text protocol for writing in data. And we did create a Rust parser for this LIME protocol data. So we could probably do a benchmark of that versus the Go LIME protocol parser. I'm not sure which would come out ahead, to be totally honest, because the Go LIME protocol parser has been heavily, heavily optimized. And that's one of those things where... Yeah, I don't know. But the performance of version 3 of the database, we've seen really, really great performance wins on, we can ingest far more data using fewer CPUs. We get a compression win because we changed the file format to parquet. We get better compression than we did before. And depending on the query, the thing that it's generally not the actual compute, it's how you, what you filter the query down to, right? So in InfluxDB 1 and 2, if you had a query that touched, you know, a million time series, that either wouldn't run or it'd just run incredibly slowly. In version three, because the way we organize the data, you can execute a query that touches a million time series and, you know, execute in milliseconds, it's fine. But, at the same time, there are queries in version 3 that are slower. So a query that asks for one specific time series, version 1 and 2 of InfluxDB is optimized for that kind of use case, so that query is exceptionally fast. Because what it does is it checks the inverted index, it finds the time series ID, and then the way the data is laid out on disk is actually sorted by that time series and time. So they can basically jump right to the file right to where the data is and read it. So it's super super fast in three. Identifies these large chunks of data and then it has to brute force an execution path to pull out the specific individual time series from it. That's generally fast enough, but we're definitely seeing cases where we need to add like further either customization or optimizations so that people can kind of like design a schema for that specific use case. But again, like But none of this is really a Rust versus Go comparison, because we didn't rewrite the same thing. We created a totally different thing. It just happens to share the same name. But as a database, it looks completely different under the hood.
Matthias
00:55:15
That means you chose Rust not solely because of performance improvements or because of the potential for performance improvements. There might be things that go beyond that, which brought you to rust. Maybe you can talk about this. I don't want to put words into your mouth or make any assumptions. So I want to hear it from you. What is the main benefit other than performance?
Paul
00:55:40
Yeah, so I mentioned this before, I think like the fearless concurrency, right? Having, you know, eliminating data races essentially, which we had before, like these really gnarly bugs in version 1 of Influx due to stuff like that, which we just wouldn't have in Rust. I think Crates is a great system for sharing libraries and sharing dependencies. It's been a long time since I've actually been a Go programmer. My Go experience predates modern Go package management, but I'll say from like 2013, 2012 to 2016, when I was writing Go code. The package management situation was a nightmare. It was terrible. Again, my personal opinion. Maybe not everybody agrees. But I viewed crates as like a really a great strength of Rust as a language having that there. What else? The error handling, the way error handling works in Rust. Again, like coming from Go, like checking for nil and like, you know, we got bit by that so many times too. And again, that's just something you just don't have to worry about it in Rust, right? Check error types and whatever. So, yeah, the error handling is something that was appealing to me. What else?
Matthias
00:57:01
Yeah. Sorry, did you have any problems with deadlocks in the old Go version? I know that Rust doesn't protect you from deadlocks necessarily, but I wonder if you see a difference here, or if that never was a problem with Go.
Paul
00:57:18
I mean deadlocks are always a problem, I don't care what language you're in, I guess. So I don't think there's any more or less of a problem in Rust than they have in Go. Yeah.
Matthias
00:57:31
Nice. So, overall, I would say it was a great transition, a happy customer experience, a good case for it. And I wonder how you could use that language now, now that you have this tool in your hands to build something that goes beyond what you have right now, even. So what would InfluxDB look like in 2024, 2025, moving in? And what sort of features are you planning to build?
Paul
00:58:02
Yeah, so, I mean, right now, the version 3 of the database is actually just the core of the database. We still have the same, we brought over the version 1 API, so you can write data to it as though it's a version 1 database. You can query data from it as though it's a version 1 database, right? You can have the same API, same query language, InfluxQL. And we have the version 2 write interface. So the only new query interface we have now is Flight SQL. It's a SQL-based query interface using Apache Aero Flight. Now, with Parquet as the storage format, Parquet has support for structured nested data. Basically, I can see us creating a new version 3.0 Write API that supports a much richer data model than we do now. It's being able to specify the units of something that's measured or specify rates, an actual histogram type that is built in, an underlying histogram type, and nested types like structs, enums, arrays, those kinds of things. So I see us adding those kinds of features on the right side and then on the query side, enabling people to pull that stuff out. And then one of the other things I really want to add in is essentially an embedded VM, right? And at this stage, I'm thinking either Python or JavaScript. I'm not sure which, but I really like to have an embedded VM in the actual database so that you can do... Essentially, there are three different cases. So one is essentially something like a database trigger where when you write data in, it would execute a script that you specify to transform the write or do something when you have a write come in. Another is on persist. So basically, data gets buffered up in the database and that gets persisted as a parquet file. So I'd like to open an event there so that people could attach a script in the database. They could transform the data, they could do things with it. Either before it gets written out to modify it, or basically as a result of it getting written out, you can build monitoring and alerting. And then the other is essentially like a periodic system, like Cron essentially in the database, but being able to specify that so that you can both... Ideally, at that point, you're putting a script inside the database and you can query data out of the database and have like a very, you know, fast performing experience because you have this data locality. So features around that I think and then enabling having essentially like a shared service where people can share like scripts and stuff like that that that are useful within the database. So those are things I can see us doing. And then obviously like we are we're big contributors to Apache Aero Rust, the Rust Arrow RS, as well as Data Fusion, and I see us just contributing, continuing to contribute to that. I really think like some of those building blocks, those Rust building blocks are going to be. Are going to form the basis of a lot of like really interesting, you know, large-scale data processing systems over the next, you know, five to ten years, right? Things that take place, you know, a lot of stuff that's currently in Java, right? For doing large scale data analytics, analytical processing, stream processing. And I really see, I mean, I think that's what's gonna be interesting to watch over the next like 10 years is what new Rust projects are formed to start taking, you know, market share away from some of these larger, older Java native projects.
Matthias
01:01:59
When you said you want a query language, the one thing that came to mind was WebAssembly. I wonder if you could support both languages, both Python and JavaScript, with something like WebAssembly. Have you considered that?
Paul
01:02:13
Yeah, my concern is that it's just not going to be performant enough because you need to copy the data into the WebAssembly VM and all this other stuff and then copy it out. I'm just worried that that bridge is going to be too expensive. So it's an idea. I don't know. I'm not going to create the VM. Whatever it is, it's going to be something that's already written. We're not going to create the VM. We're not going to create the language. We want to just use existing tooling that's out there and let people adopt, use a language that they're already familiar with.
Matthias
01:02:45
Very exciting because it feels like you move away from pure time series data towards an analytical engine, if you like, or at least some sort of marketplace so that people can share scripts or share ideas or think about different ways how to use the data and make it work for their business and their use case. And I don't know any other players that are in that realm. And also, if there's any collaboration between those different companies, or if you would say no, inherently, this is such a special problem, you're working with very specific data here. It doesn't make any sense to even share scripts across databases or across these engines.
Paul
01:03:31
No, I mean, I think…, I mean, so to me, the way I think of time series is not just like metric data, really like the vision for InfluxDB has kind of always been that it would be like the place for observational data of all kinds, whether it's metric data, event data, you know, logs and traces I view as time series data too, just like different kinds of structures. So I think there, and And then sensor data, again, like depending on the sensor, they're all different kinds of sensors. So people's specific data problems are probably unique to them, but the kinds of things you want to do against these, there are patterns that pop up, right? So a common pattern in like monitoring alerting is like what's called a dead man alert, right? Like if you have a sensor where you expect a reading every minute and it stops reporting, that triggers an alert condition, right? I think there are little bits of code, ideally, that people will be able to share across these different things, these places where they identify, oh, this is a common transformation that you need to make on time series data to do this kind of analysis or whatever. I think most of the data and analytics companies, real-time analytics companies and all these other things, if they're collaborating at all, it's around some underlying libraries. right? So data fusion, for example, is used, you know, and developed not just by us, but by a bunch of other companies, some of which we may end up competing with, you know, at some point in the future. And so there's like the actual underlying libraries and then they're kind of like the standards. So within analytics, I think one of the standards that's starting to get traction, is something called Apache Iceberg. And it's basically a spec for a catalog of data in object storage. So tabular data and object storage. It was originally created at Netflix and then open source and put in the Apache Foundation and it looks like you know some of the analytic companies are starting to pick up support we're certainly going to support it as well. It's a snowflake data bricks a bunch of mothers.
Matthias
01:05:46
Yeah as you can imagine there might be a few people who listen to this podcast who might get excited to either try rust or influx to be or a combination of those And I wonder if you want to give them any hints on how they can get started, how they can start contributing, the sorts of things that you would like to have help with, or even just feedback from the community.
Paul
01:06:12
Yeah, so for InfluxDB, right now the only way to experience InfluxDB 3 is through our commercial products, right? our multi-tenant platform or our dedicated single tenant platform or our on-premise software. Because we don't have an open source release yet. So for us, the best contributions we see are in the data fusion projects. Although that can be kind of a hard project to get into because it's, you know, query planner and optimizer and execution engine. So sometimes that can be kind of an opaque project to pick up and learn about. I think, you know, for people getting started with Rust, the best thing to do is, you know, read through, read through a book, pick a project that you want to actually implement with it and, and go with that. And then if there are open source projects that you know of in Rust that are interesting, it's always useful to look through other people's code and read other people's code to see how they structure things. So, yeah.
Matthias
01:07:20
And finally, is there anything that you would like to mention, something that you would like to promote or maybe something that that you want to share with the Rust community, this is your moment.
Paul
01:07:34
We're hiring Rust programmers. So yeah, no, and hopefully, like I said, soon, you know, end of this year, early next year, we'll have an actual open source release of InfluxDB 3 that was written in Rust. And I think that'll be very, very exciting time. So if you want to contribute to it and get paid for it, please reach out and paul at influxdata.com.
Matthias
01:08:03
Amazing. Yes. I love that you're so approachable and I love that you're such a nice conversation partner. It was a really great interview. Thanks a lot for taking the time, Paul.
Paul
01:08:14
Yeah. Yeah. Thank you for inviting me on.

2024 - Matthias Endler

Hosted by LetsCast.fm
Create your own podcast