Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: It's Matthias Endler from corrode, and today's guest is Adam Hendel,
Matthias: founding engineer at Tembo.
Matthias: We talk about production-grade infrastructure on top of Postgres with Rust.
Matthias: Adam, thanks for joining us today. Can you introduce yourself and tell us about
Matthias: Tembo? What exactly are you building there?
Adam: Yeah, thanks for having me. I'm Adam Hendel
Adam: i am one of the first engineers hired
Adam: into Tembo and Tembo is a
Adam: managed Postgres platform you know
Adam: where you can click a few buttons and get a managed Postgres database pretty
Adam: quickly yeah and like on the Tembo platform one of our main products is what
Adam: we call stacks and that's where you can get Postgres pre-configured for certain
Adam: types of workloads for you with those same few clicks.
Adam: So if you're doing OLAP or search or doing something with AI and you need embeddings,
Adam: in that example, we have a vector DB stack that gives you everything you need
Adam: to go build your application.
Matthias: What is it about Rust companies and their love for Postgres?
Matthias: It seems like every other Rust startup I talk to is building on top of Postgres.
Matthias: What makes this 40-year-old database so appealing to modern developers?
Adam: I mean, I love Postgres because it just works. I learned to write SQL on Postgres.
Adam: And earlier in my career, I was doing a lot of stuff in data science and machine
Adam: learning, and I didn't know really what write looks like in terms of good SQL
Adam: queries or efficient ways to build an application.
Adam: And Postgres was just there so i just abused the
Adam: hell out of it and it always just kept working
Adam: for me no matter what i threw at it and over
Adam: my career you know i i got i got better and then
Adam: learned how to do things really way more efficiently and then Postgres became
Adam: to shine even more so you know for me it's just super flexible always been able
Adam: to just work for me don't have to hassle too much with tuning it or anything
Adam: it just it just works isn't.
Matthias: Postgres already production ready out of the box what exactly are you building
Matthias: on top that goes beyond just hosting it.
Adam: As you do get to having very specific types of workloads,
Adam: and you want to start to make some trade-offs to really get all the juice you
Adam: can out of Postgres, you have to start to make some decisions.
Adam: And that's where you don't have to touch the C code, but you might have to touch
Adam: some of the configuration.
Adam: For example, there are certain configs in Postgres like shared buffers that
Adam: kind of decides how much of your system's memory can you allocate to put the
Adam: working set from your database into memory.
Adam: So that's something that we can set dynamically
Adam: for you and then certain extensions
Adam: that can really improve and change your experience
Adam: with Postgres those can be quite a hassle to
Adam: get installed so we pre-install those
Adam: into Postgres for you and kind of
Adam: go through the go through the process of you know for for specific types of
Adam: workloads what are the best extensions to make Postgres the best that it can
Adam: be for for that workload so it's kind of configuration and getting the extensions
Adam: installed and providing a really good experience around that.
Matthias: So you're dealing with configuration extensions performance tuning when you
Matthias: started building all of this was rust the obvious choice or did you consider
Matthias: c c++ or python maybe as an alternative.
Adam: Well since Postgres is
Adam: written in c that was definitely something that
Adam: that came up the experience of
Adam: you know building software in c is not
Adam: you know it's not super modern
Adam: you know so for somebody like me i i think
Adam: the first programming language i learned outside of
Adam: like html and whatnot was like R
Adam: for like statistical programming and then Python and in Python you have stuff
Adam: like the Python package index where you want to do something well hey there's
Adam: like a library for that and you can just pull in that library and work on solving
Adam: the specific problem that you're trying to solve.
Adam: That kind of exists in C but it's not at all as quite as easy to just get up
Adam: and running building stuff in C so even though,
Adam: Postgres is written in C, and it's super performant and very stable.
Adam: It is hard to just come up on it and start to add functionality.
Adam: But in Rust, Rust does have cargo and crates.
Adam: And if you are trying to build software, you can get up and running pretty quickly
Adam: with Rust and using other libraries that are in crates.io and whatnot. okay.
Matthias: I get the c limitations like no modern tooling painful package management but
Matthias: you mentioned coming from python which has great abstractions and developer
Matthias: experience why not use python for this did you at least prototype in it before jumping to rust.
Adam: Well at at the time that at Tembo
Adam: that we were getting started building extensions there was
Adam: a framework that made it really easy to get
Adam: up and running with rust that framework's called pgrx
Adam: well at the time it was just called pgx they renamed
Adam: to pgrx at at some point but
Adam: to go from like nothing to a hello world example it was like an hour or less
Adam: to create an extension that you know you write some function in rust turn that
Adam: function into a sequel function that you could call that was like super easy so So it was kind of,
Adam: you know, like we didn't have to, we were up and running really quickly.
Adam: So we didn't really explore Python 2 seriously.
Matthias: Okay, that's pretty impressive. So PGRX made it super easy to get started.
Matthias: But I don't really understand how Postgres extensions actually work.
Matthias: You write a Rust function, somehow becomes a SQL function.
Matthias: What's happening under the hood? Is there some kind of plugin manager?
Matthias: How does your Rust code actually talk to Postgres?
Adam: Yeah, so you can write functions in pure SQL.
Adam: There's procedural languages like PL/pgSQL that you can kind of still just write,
Adam: you know, kind of SQL and easily create these functions.
Adam: So there's an object that lives in Postgres, and it's a function that's an object in Postgres.
Adam: And that function can kind of point to SQL code or one of the procedural languages,
Adam: or it could point to a shared object.
Adam: So now that shared object could be compiled in C or it could be compiled in Rust.
Adam: So in the case of building extensions with PGRX, you have this function and
Adam: it really just points to a shared object that's the code that you wrote in Rust.
Adam: So when you compile a PGRX extension, you might create some functions.
Adam: There's more complicated things you can do than just functions.
Adam: But in this case of a function, there's some code that's like,
Adam: hey, I'm creating a Postgres function and that function is pointing to your Rust shared object.
Matthias: But wait, if extensions are just functions pointing to shared objects, how do you handle state?
Matthias: Functions are supposed to be stateless, but any real application needs to manage state somehow.
Matthias: What happens when you need to maintain data across calls or coordinate between operations?
Adam: Yeah, so it does get complicated to manage state.
Adam: I think the default, like when you call that function
Adam: everything is starting to happen within a transaction so you
Adam: can you know you can start to store state just
Adam: in memory like as soon as you call that function you
Adam: imagine like you're gonna read some data from
Adam: a other table in Postgres and you
Adam: can pull that into memory and then go do whatever it is that
Adam: you want to do and ultimately you know
Adam: return some result set back to who's ever calling that
Adam: that function so you could store it in memory or you could store it in some
Adam: intermediate tables in back in Postgres but in those cases you're always you'll
Adam: be working through the Postgres spi and that's the server programming interface
Adam: so you have some tools available to you to kind of deal with your state there and.
Matthias: The other thing that bothered me while you explained this you're going
Matthias: through this spi this server programming interface dealing
Matthias: with ffi boundaries don't you lose all the conveniences and ergonomics of rust's
Matthias: type system i mean how would Postgres even know that a function is really available
Matthias: all of rust's advantages are in its type system and safety guarantees don't
Matthias: you lose all of that at the ffi interface.
Adam: In the like memory space where your rust code is written you still have all
Adam: of rust features available to you so you know as you interact with FFI, you get some,
Adam: All the FFI integrations, if you're using PGRX, like PGRX abstracts that for you mostly.
Adam: So you just write Rust code the same as, generally the same as you would if
Adam: you're, you know, writing a web application or, you know, some script in Rust.
Matthias: Okay, you convinced me. PGRX seems like the way to go if you want to start a new extension.
Matthias: But did you ever hit any limitations which might make you move away from PGRX
Matthias: or is everything pretty much covered?
Adam: A lot of stuff is covered. There are some limitations and I feel a little bad
Adam: in what I'm about to say because I wish I could have contributed back to PGRX.
Adam: But some of those SPI, the server programming interface in PGRX are just not fully optimized.
Adam: So if you have some SQL that you want to execute through the SPI from using PGRX...
Adam: You have to, data that kind of goes to and from Postgres through that SPI,
Adam: you have to serialize and deserialize all of it, even if you're not really going to touch it.
Adam: So there's a project, a message queue project that we worked on called PGMQ.
Adam: It was originally written in Rust using PGRX. And particularly for the batch
Adam: operations where we would, you know, send or read, you know, 100 messages at a time,
Adam: those would get real slow because we would have to iterate over everything,
Adam: every message and just deserialize it and then resealize it as we would,
Adam: you know, send it back to the user.
Adam: But in those, really in those cases, like there's not a ton of advantage to,
Adam: you know, just not be using just pure SQL for those cases.
Matthias: Does this mean the code base is now mostly SQL with a nice Rust wrapper on top?
Matthias: You've kind of inverted the pattern.
Matthias: Instead of writing a pure Rust Postgres extension, now it's SQL with a nice
Matthias: wrapper around it, sort of.
Matthias: Like, what are the boundaries between SQL and Rust in that case?
Adam: So PGMQ, I can just tell a little bit of background about that project.
Adam: So at Tembo, we needed a queue between our control plane and our data plane.
Adam: Data plane is like where we spin up people's databases, and control plan is,
Adam: you know, backend web applications.
Adam: So we were building this queue on Postgres and all of our tech was already in Rust.
Adam: So we basically had a bunch of like SQL strings like written in Rust code.
Adam: And that code would be duplicate across the control plane and then the same
Adam: service in the data plane would have like the same SQL statements for like sending
Adam: messages and reading messages.
Adam: And that was all Rust. So we first pulled it out into a crate and then installed
Adam: the crate all the places that it needed to be.
Adam: So it was all client side.
Adam: And, you know, since it was a crate, it was already kind of packaged up.
Adam: If we could turn the crate into a Postgres extension using PGRX with,
Adam: it was like, I don't know, a couple hours worth of work to take that crate and
Adam: use PGRX and create an extension out of it.
Adam: So then in that case, like there were all these shared objects for every function
Adam: in Postgres and, you know, it was great.
Adam: We had this extension and we could ship that extension around to different Postgres
Adam: instances and share it with the world.
Adam: And, you know, it was great. And then we started to run into those limitations
Adam: where, you know, it was really how we were using PGRX made some things slow.
Adam: So we actually was with the help of the community and we even got some help
Adam: from folks at Supabase to rewrite it from PGRX Rust into PLPG SQL.
Adam: And that was just for like the queue operations.
Adam: And you know there's still you know we still want
Adam: to use it use that extension at Tembo
Adam: and all our stuff is in rust and you know
Adam: tons of people on around the world are have their applications in rust and want
Adam: to use this extension too so there's a the rust client library for it and and
Adam: and that is to like give you the idiomatic rust experience for for using pgmq on on Postgres.
Matthias: It's very nice pattern you go from plain vanilla SQL extract it into a separate
Matthias: crate and make it a Postgres extension with ptrx and I really like these code
Matthias: usability stories I hear that a lot and I really like that about Rust and being
Matthias: able to kind of move things out into separate crate,
Matthias: so that was the transition if I understood correctly but
Matthias: then you run into serialization issues because when
Matthias: you try to get batches of messages over the wire it's probably
Matthias: an O of N operation to go through each and
Matthias: every message and deserialize it with 30 and that can be slow and eventually
Matthias: you want to go back a little bit and write parts of the more performant code
Matthias: in maybe a lower level abstraction or maybe not use pgrx for that case right yeah.
Adam: That's exactly it you know we talked to the the pgrx maintainers about what
Adam: we were doing and they're like hey,
Adam: You know, what you're doing doesn't really need to be in PGRX.
Adam: You're just executing SQL statements.
Adam: And yeah, there are some limitations right now with the maturity of the API
Adam: that PGRX had around just executing SQL statements.
Adam: So it was kind of a, yeah, we should, this should be SQL instead of, you know, in Rust.
Matthias: But couldn't you have kept a hybrid approach? ptrx for most things SQL for the
Matthias: performance critical parts what made you abandon ptrx entirely.
Adam: We could have but there's some additional like gotchas with extensions in Postgres,
Adam: and it's mostly around the like portability of
Adam: extensions so if you
Adam: want if you have an extension in Postgres extensions
Adam: have like the objects that we kind of talked about earlier and
Adam: they also have control files and migration
Adam: files and all these other things and for
Adam: it to be an extension those things have to be on like the
Adam: same host as Postgres but like
Adam: let's say you're running on rds or you're running in supabase or google cloud
Adam: somewhere if it's an extension then you have to deal with all these additional
Adam: things but if it's just sql then you can just take that sql with you and manage
Adam: it however you want to from.
Matthias: That perspective it makes total sense the pgmq client caught my attention because
Matthias: unlike NATS or Kafka it's just Postgres so no need for new tech in your tech
Matthias: stack but what impressed me the most was how rust like the api feels.
Adam: How did.
Matthias: You make a SQL-based queue feel so idiomatic to Rust developers?
Adam: Yeah. So, you know, my opinion, and I didn't write all the client-side code
Adam: here, so we had some help from the community who, you know, built some of these things.
Adam: But the things that I care about in the client are establishing a connection pool to Postgres,
Adam: And you can let the client do that for you, create a pool to Postgres, a connection pool.
Adam: But maybe you're doing a bunch of other stuff in your application too.
Adam: So we have the ability for you to create the pool yourself and like provide
Adam: the pool to the client and let PGMQ's client use your pool.
Adam: And so with in the
Adam: client if with Rust there's like some generics typing that
Adam: we could do with that to be like you know
Adam: how we execute the the Postgres
Adam: functions for the the queue you know
Adam: we just take in certain as long
Adam: as the objects have certain traits implemented we can execute that that sequel
Adam: and you know like if if it's in a different language i don't you know i don't
Adam: know if like it'd be way more complex but like since we can just make these
Adam: assumptions like hey pass in this thing as long as it implements these traits we're good to go you.
Matthias: Mentioned sqlx some people might
Matthias: not be familiar with it what makes sqlx special for this kind of work.
Adam: Yeah and in fact the pgmq client uses sqlx so yeah so sqlx i describe it as
Adam: like everything that i care about from an ORM, but it's not an ORM.
Adam: All the great things and none of the bad.
Adam: So SQLX, you know, it does a lot of things and I don't use all of its features,
Adam: but the thing I like the most is it gives me the compile time checks on the SQL that I write.
Adam: So if I have some insert statement in my application and it's just written in raw SQL.
Adam: I can rest assured that, you know, if I have some struct that I'm trying to, you know,
Adam: serialize and insert into a row, if like the types on my struct are not in compliance
Adam: with the table, SQLX will give me a compile error.
Adam: Yeah, I love SQLX.
Adam: It's like an ORM, but better. So what I like about ORMs is that they give me
Adam: this really nice interface between my application and the types in the database.
Adam: So if I have some struct, struct has attributes on it and those are typed,
Adam: an ORM would be like, yeah, there's this connection between this object and
Adam: the table or many tables.
Adam: But I actually really like to write SQL and
Adam: a lot of times I end up with SQL statements that don't
Adam: really fit in ORM super well so I end up with this code base with like some
Adam: SQL and some ORMs and it gets really messy but with SQLX I can just always write
Adam: SQL and I love that as a developer because I enjoy writing SQL,
Adam: but I still get these type checks because SQLX will look at my SQL statements
Adam: and then check in the database,
Adam: does the SQL statement and what you're trying to insert or what you're trying
Adam: to read and deserialize into some struct.
Adam: Does it is it going to work do the types match up and that that is huge it could
Adam: be a little frustrating at first until you kind of get the hang of it but that
Adam: piece right there is kind of what hooked me on sqlx now.
Matthias: I would guess the other important part is async support because all of these
Matthias: operations are io bound and you kind of don't want to block the client for concurrent queries,
Matthias: what did you end up using for async and was it straightforward to integrate into the existing code.
Adam: You know rewind way back to when
Adam: we were like first building first started building in
Adam: rust and it was like okay so for
Adam: async we like we kind of have to handle this
Adam: ourself and that was there's a
Adam: ton of complexity around that that like frankly we
Adam: like didn't want to spend our time like trying to implement and
Adam: so then there's a few different runtimes out there and Tokio was
Adam: like clearly the one that was the most most popular the most supported biggest
Adam: community around so we just use that and you know to this day like when i'm
Adam: building in rust i don't even i don't really think about which async runtime
Adam: to go with i just use Tokio right.
Matthias: But what about very complex things like supporting transactions and by transaction
Matthias: i mean multiple statements that are executed atomically on Postgres if you wanted
Matthias: to do that in an async way i would be afraid that it would get really complex.
Adam: Yeah i i don't exactly know how you would if you had like two separate threads
Adam: and you wanted to span some transaction across those i don't,
Adam: I don't know. I haven't actually tried that. So I don't know how it would work.
Adam: But kind of the way that we implemented the, I think earlier I mentioned like,
Adam: yeah, this, you know, you can bring your own connection pool or really your
Adam: own executor, as long as it has certain traits implemented on it, which is basically,
Adam: you know, we inherit that from SQL X.
Adam: But you can, you know, you can create, start a transaction, do whatever SQL you want.
Adam: You could your like connection or your
Adam: transaction object that you've created and if you pass that into pgmq
Adam: then you could go do the pgmq things through the
Adam: pgmq api and then finally commit your transaction
Adam: however you however you you want to yourself you know so so like we lean on
Adam: Tokio we lean on sqlx for that oh yeah the question about like to do that across
Adam: threads i don't that could get tricky to like pass that connection across threads
Adam: i don't know how you would do that it might be possible you.
Matthias: Could probably share the connections across threads with
Matthias: something like arc mutext and sqlx allows that kind of pattern but at the same
Matthias: time why would you because Postgres already handles that with transactions and
Matthias: that guarantees atomicity so maybe you just leave it as an implementation detail to the database.
Adam: Yeah from Postgres side you know that's just it's the feature transactions you
Adam: know we don't have to reinvent the wheel on that part some.
Matthias: People might hear this and think why do i need pgmq
Matthias: i can just write my own abstraction on top
Matthias: of Postgres how hard can it be can only be a couple lines of code right and
Matthias: we touched on transaction support and edge cases but maybe you can allude to
Matthias: a few more things that you probably don't want to do yourself things that people
Matthias: tend to forget when they build their own abstraction functions.
Adam: Yeah i mean with all this stuff you know it's just
Adam: code so you can you can always just go write write the code you know there's
Adam: these operations that you need to do if you're going to build a queue you need
Adam: to have a way to create a queue so how do you how do you define what a queue
Adam: is you could you can think through that and make it what you want we have a
Adam: function called create queue that,
Adam: does it a way that we think is right and there's an api around it you know sending
Adam: a message to a queue it's basically just an insert statement you know but that
Adam: insert is handled for you,
Adam: and if you're like if you're working in rust you have some message and that
Adam: message becomes like an attribute like in a column on a table so you need to
Adam: serialize that that message and get inserted into the table.
Adam: So if you're using the Rust client, you know, there's some helpers there to
Adam: do the serialization for you.
Adam: So, you know, you save a lot of time by, you know, having this stuff like it's
Adam: already built, it's already tested, you don't have to rewrite a bunch of things.
Adam: But, you know, I'm sure there's cases out there where somebody like it makes
Adam: sense for somebody to write their own.
Matthias: Even if you asked me to design a message queue table schema,
Matthias: I'd probably miss something important.
Matthias: I'd add a timestamp, maybe a byte array for the message, though I'm not sure
Matthias: if you support bytes or just UTF-8 strings.
Matthias: Then some kind of locking or ownership fields to prevent reading the same message twice.
Matthias: What actually goes into that table?
Adam: Yeah, I mean, you're pretty close there. So there's a concept that,
Adam: you know, we're inspired from the SQS, Simple
Adam: Queue service from aws the visibility
Adam: timeout which is kind of that that locking
Adam: feature a little bit so when you read a message
Adam: from a queue you have to specify how long
Adam: do you want that that message to be unavailable to yourself or if you were to
Adam: try to read again or to any other consumers of of that queue you know so let's
Adam: say you have like 10 threads all reading from the same queue that the first
Adam: time the message is read,
Adam: you could set it to say, hey, make this message invisible for five minutes.
Adam: And then during that five minutes, any of those other threads,
Adam: none of them will be able to get that exact same message until that message,
Adam: that time, the visibility time expires, and then that message becomes visible again.
Adam: So that that piece
Adam: i think it's super cool because without that
Adam: you'd have to have some additional worker process that
Adam: would watch the queue and look for messages that need
Adam: to be you know need to be flipped back available again but in this way it's
Adam: like completely stateless everything is just there's nothing that can really
Adam: break there you know there's no process that crashes and all of a sudden you
Adam: know messages get stuck they just automatically become visible again that's.
Matthias: Clever it's sort of a log free algorithm because you just update the timestamp
Matthias: when you read it and everyone knows the message is in flight.
Adam: Yeah the the most complicated part
Adam: of pgmq is the read statement uh it
Adam: is a super complicated query it's a it's
Adam: a for update statement so it it's
Adam: reading but it's updating the the how
Adam: many times it's been read so there's like a counter on that
Adam: on that table that keeps track of how many times this
Adam: message has been read there's that visibility time which you know it's some
Adam: time that you know anything after that time that message can be read yeah so
Adam: you know there's when you read the message you're actually it's an update statement
Adam: to the table and it's a select for update which actually creates a lock on that table.
Adam: And that makes it so that if two threads read at the exact same time,
Adam: Postgres would figure out, hey, who gets this record first?
Adam: So the for update just guarantees that only one person, one worker can get the message.
Adam: But that does create a lock on the table.
Adam: So we say for update, that creates a lock.
Adam: And then there's a another clause in there
Adam: skip locked which means any records
Adam: that are locked skip over them and go to the next one and that
Adam: makes it so that any other workers that are reading aren't sitting there waiting
Adam: for that lock to leave they can skip it and go to the next message and so that
Adam: lock is is pretty quick it's that lock only lasts for the duration of you know
Adam: that that transaction which is pretty quick and like the long-term locking is
Adam: handled by the visibility timeout.
Matthias: You probably came up with that on the first try, right?
Adam: No, I mean, there's a ton of resources out there.
Adam: If you just search, you know, message queue, Postgres, there's a lot of people
Adam: have written about for update and skipped locked. It's kind of the standard.
Matthias: But still, you can make a lot of mistakes when implementing that.
Matthias: And also, you might not know about this research in the first place.
Matthias: You might just naively implement it and do it the wrong way.
Adam: Yeah definitely like you know if you if you don't have the skipped lock on there
Adam: you could be like you know you have 10 threads reading,
Adam: one of them got a message and the other 10 are just sitting there that you like
Adam: they would just sit there forever you know until that initial lock was released
Adam: so yeah you could you could mess it up i.
Matthias: Really like these implementation details that's what i live for because that
Matthias: could really hamper your performance and in the worst case you don't realize
Matthias: until you're on the load and when you really need it the most it will kind of fail on you.
Adam: Yeah yeah that that's that's a
Adam: good you know i'm glad you mentioned that because that's kind of a you
Adam: know a reason why to use something that's
Adam: been pre-packaged you know because you can you
Adam: know there's a lot now today there's quite a few people using pgmq and
Adam: you know when people run into
Adam: issues like issue gets created on on the
Adam: project and then you know we or somebody in
Adam: the community resolves it but there's there's ways
Adam: that you know try to build your own queue like there's ways it can
Adam: go wrong so you know pgmq as a project has kind of started to be this like place
Adam: that people go to learn and just use the code of you know when they want to
Adam: do a queue on Postgres they can you know look at that project and either use
Adam: the code or modify it and do it their own way that's.
Matthias: Why i love open source in the first place because so much
Matthias: is out there and you can look at the actual implementation and then decide if
Matthias: you really want to go down that rabbit hole and implement it yourself because
Matthias: just sending in a pull request will fix a problem for every instance out there
Matthias: that has the latest version of course which is so so great.
Adam: Yeah how.
Matthias: Hard would it be to abstract that to other databases for example if there was
Matthias: a customer that needed MariaDB support or MySQL.
Adam: Support.
Matthias: How hard would that be if you had to support that?
Adam: I guess, you know, it would be, there's some SQL files in the PGMQ project,
Adam: and it would be like, hey, for every one of these statements,
Adam: what's the equivalent in the other database?
Adam: And if you translated that, then it, you know, theoretically should work.
Matthias: The cool thing is that you could use Rust's feature flex.
Matthias: So the project builds against the database that it depends on.
Matthias: I was wondering about Postgres Listen and Notify support.
Matthias: Where do you draw the line between using native PopSup versus needing a full message queue?
Adam: We've had some discussions of using those features from Postgres in PGM queue.
Adam: But I don't think they are exactly, like you can just replace PGM queue with alert and notify.
Adam: Mostly because you know if if you're going to send the
Adam: message across one of those notification channels like
Adam: if that message doesn't make it then it's kind
Adam: of gone you know and you have no record of it you know in pgmq it's every message
Adam: is a a row in a in a table and you can archive that row or just completely delete
Adam: it if you want so you still have like a complete audit log of every message if you want it,
Adam: if we're just to move it into one of those other channels and it's kind of gone,
Adam: you know, unless we build something to handle it.
Adam: But I do think it would be really useful to use it as a way to notify consumers
Adam: that there are messages available in AQ.
Adam: So right now, you know, if your application is reading messages from a queue,
Adam: you have to pull the queue,
Adam: you know, pull it once every second or have some back off logic,
Adam: you know, pull it every second and then back off to 10 seconds or something.
Adam: But it would be nice to be able to subscribe to the queue and have one of those
Adam: notifications come to you and say, hey, there are messages now, now pull.
Adam: And that could really help some
Adam: efficiency if you have really low latency requirements and you can't wait.
Adam: A one second pull interval is too long and you need to know right away.
Adam: You know that uh and you don't want to just pull every you know 10 milliseconds
Adam: or something less if we had something implemented on those features that could
Adam: it could help a lot i think also.
Matthias: On top of this you could use the Tokio stream abstraction and you could just
Matthias: iterate over the messages and the futures would resolve as quickly as messages come in and it.
Adam: Would all.
Matthias: Kind of beautifully happen under the hood where you say well you get a future
Matthias: ready and it contains the value and that is your message and i guess it would
Matthias: be kind of nice to have that how far away are you from that reality.
Adam: Well i think we would first need to to get the you know implement a way to like
Adam: set up those channels on any given queue and i don't think that would be super difficult,
Adam: there's probably a way that this could all be implemented purely
Adam: on the the rust client side too so if
Adam: i think of how like lib rd
Adam: kafka reads messages from a
Adam: kafka topic you know it pulls but it
Adam: is pulling messages in in batch to the client and then the consumer of lib rd
Adam: kafka you know iterates over those those messages and we could probably do something
Adam: similar in the rust client you know where you you have some pull interval but
Adam: we're handling that pull asynchronously and pulling messages back in batch.
Adam: And then your Rust application could iterate over those messages asynchronously if it wanted to.
Adam: Yeah, I always really liked how LibRD Kafka did it because as a user there,
Adam: all that stuff is abstracted from you, but it's super efficient because it is pulling for you.
Adam: And unless you really get into the weeds, you don't really know that it's pulling
Adam: and it's pulling messages in batch.
Adam: But we could probably do the exact same thing in the Rust client.
Matthias: What's your biggest deployment at the moment?
Adam: So I think I mentioned earlier this architecture with a control plane and a
Adam: data plane. We have several data planes.
Adam: There's a data plane in AWS, Azure, Google Cloud, and then some self-hosted ones as well.
Adam: But all those public clouds, they all read messages from a single Postgres instances,
Adam: and there's multiple queues within that.
Adam: And I think at peak, I think there's like 10,000 per minute,
Adam: maybe five minutes or something.
Adam: But really like the scalability of that it's all comes down to what can Postgres
Adam: handle how many inserts how many updates per second can Postgres handle is under
Adam: because that's really all that pgmq is doing is inserts and updates well inserts
Adam: updates and deletes okay.
Matthias: So if you use that for a larger deployment it certainly won't be the bottleneck
Matthias: in your application unless you are at web scale and then you always have bigger problems you.
Adam: Have bigger fish.
Matthias: To fry okay.
Adam: Yeah what's.
Matthias: Your setup for the production cluster right now.
Adam: It's a dedicated Postgres cluster for
Adam: the for the queue yeah and it's
Adam: those specs on it are exactly right yeah and
Adam: it it like never really gets over 20 cpu or memory memory utilization is kind
Adam: of just hovers around 25 probably because that's what shared buffers are set
Adam: to on that thing but i mean it is dedicated to the queue so you know it's kind
Adam: of it is i think it is over provision well.
Matthias: That's at least better than being on the provision what are some of the other
Matthias: use cases of Postgres that you see in production a lot.
Adam: Yeah you know full text search is it is built into Postgres so if if you're
Adam: trying to do that like it's pretty easy to get up and running we also have a
Adam: number of people using our VectorDB stack,
Adam: which that's primarily PG Vector, which is another open source project out there
Adam: that lets you kind of gives you the data type of a vector and then the operations
Adam: to do similarity search on top of it.
Adam: It is definitely the gold standard for,
Adam: working with embeddings within Postgres. And then we built kind of a wrapper
Adam: extension around it using PGRX and Rust.
Adam: It lets you, say you have a table with some text in it and you want to generate
Adam: embeddings from every row in that table.
Adam: And be like, hey, I have this table. I want to use OpenAI or I want to use Anthropic
Adam: or I have a self-hosted model. And I just want to get embeddings for this table.
Adam: So our wrapper extension, it's called PG Vectorize.
Adam: You just call a function on a table, tell it which model you want to use,
Adam: and then in the background, it's using PGM-Q to look at the table and be like,
Adam: hey, which columns do we need to get embeddings for?
Adam: Pull that data, call the transformer model, get the embeddings,
Adam: and insert those embeddings to another table or the same table. It's configurable.
Adam: But yeah, it helps you with the orchestration there.
Matthias: I like that you use your extensions in combination with other extensions and
Matthias: calling into them. It's kind of cool.
Matthias: How do you ensure safety between the boundary of Rust and Postgres?
Adam: That Vectorize extension, it runs in a background process that Postgres manages.
Adam: And we purposely don't use the SPI, that interface previously,
Adam: because it's kind of unnecessary.
Adam: So we just use SQLX. So Postgres spins up this background process that,
Adam: reads from that queue of jobs that it needs to create embeddings for.
Adam: And we just use SQLX for it. So it's kind of treated as just like a normal application internally.
Adam: So there is no like interacting with FFI or anything going on there.
Adam: We treat it as just after Postgres starts it, then it's like normal Rust application.
Adam: Since we built it that way, we have the option on our cloud platform to run
Adam: that background worker in a separate container.
Adam: So instead as a Postgres background worker we
Adam: just take that same rust binary and run it in a separate container next to Postgres
Adam: so that's an even even better way to get around that limitation of like memory
Adam: management well if it's not even on the same host as Postgres then you know
Adam: you can scale that thing independently with.
Matthias: Tembo you shipped your first real rust production application what took you
Matthias: the longest to wrap your head around in rust.
Adam: Yeah so today i love
Adam: working in rust it it is a
Adam: joy to build software using rust but
Adam: when i was getting started there was this huge hump
Adam: this hill to get over and it was
Adam: super frustrating early on one of those
Adam: things was just wrapping my head around
Adam: like null like they're like you
Adam: know there is no no you have like some and none and i
Adam: looked at that and i'm like this is so weird it i
Adam: came from python so you know you had a none type but there was no it was not
Adam: an enum and until i like learned that it like it's an enum with like data is
Adam: what you know how Rust handles null values so it's just like wrapper around your.
Adam: Data, and is it null or is it not?
Adam: That was, I don't know why, but it was really hard for me to grasp that.
Adam: And in that same, like, direction, error handling, where it's like, okay, or error.
Adam: I don't know why it just took me so long to wrap my head around.
Adam: Like, yeah, you basically wrap your data, your, if you have a function,
Adam: and it could error, well, you wrap it in this thing, a result,
Adam: you know, and it's an enum.
Adam: And it's the same for none, or like how Rust handles none.
Adam: And I don't know I think I think like not today I love those things it just makes it so clean,
Adam: like is there an error or not is it did
Adam: it return something or was it there nothing you know
Adam: it's super clean when I look at it today but when I was learning rust I'm like
Adam: what is this this I don't even understand so I don't know what would have like
Adam: made it easier for me to learn that early on but i remember it being super frustrating
Adam: and then all of a sudden i got it like oh this is this is awesome i love it.
Matthias: How long did it take you until you had a good grasp on the language.
Adam: I'd say so when i when i started i was like learning full-time basically when
Adam: i was learning Rust and i think it was two months you know maybe that's slow
Adam: for some people like like all day,
Adam: five days a week at least you know i was able to get up in like the building software,
Adam: and still not quite understanding error handling and none you know handling
Adam: of none types so but yeah i think it was two months for me of pain two months of pain i.
Matthias: Really like that framing I mean.
Adam: But the one thing, the one thing I loved like day one though,
Adam: was having cargo, like, you know.
Adam: I came from Python, and it was like, how do I get my environment set up to run my application?
Adam: At the time, it was like, do I use virtual env? Do I use poetry?
Adam: Do I just install everything with pip?
Adam: You know, it's just, it's a mess. Like, there's some projects that are making
Adam: it better in Python today.
Adam: Like, UV is doing a really great job cleaning up that, solving that problem.
Adam: But when I was learning Rust, it was like, hey, how do I run this?
Adam: How do I run this application?
Adam: Oh, Cargo Run. that's it you know how do
Adam: i test it cargo test you know it it was just so intuitive you know how do i
Adam: add a library from crates cargo ad you know and it was just all built into the
Adam: tool chain and i didn't have to go search and read a bunch of blogs to figure
Adam: out how to just get started,
Adam: so that from day one was probably what kept me there through those two months of pain oh.
Matthias: Yeah day one in python is strange because on one
Matthias: side the language is great but on the other side the tooling is
Matthias: not so great and you really get
Matthias: those mixed feelings on that note
Matthias: shout out to Charlie Marsh from Astral who was a guest in episode three of season
Matthias: four you might want to check out that one so they do great work but the experience
Matthias: before that was subpar the tooling was a bit all over the place and very difficult to use.
Adam: Yeah i have like nightmares of so when i was working at shipped i all of a sudden
Adam: they started issuing employees the macbooks with apple silicon so they all had arm architecture,
Adam: And all of our, like not every project out there had Python wheels for ARM.
Adam: So now like all of a sudden new employees coming in, like their local environments
Adam: would just not work with our internal libraries because we weren't building wheels for ARM.
Adam: Like we were using Kafka and librd or Python's Kafka library didn't have a
Adam: wheel for ARM for the longest time.
Adam: It was like, okay, we have to compile this stuff from source all of a sudden.
Adam: And it was I had nightmares from that everything fell apart.
Matthias: Well at least they did a great job on the migration from Python 2 to 3 that
Matthias: was extremely painless of course.
Adam: Just.
Matthias: Kidding, it was a nightmare.
Adam: Yeah the most obvious one there is just like trying to print something,
Adam: it's like the print API changed.
Matthias: You know yeah, thanks for bringing back that memory by the way,
Matthias: do you have to touch a lot of python nowadays.
Adam: Not not too much you know there's a lot of stuff in machine learning is still
Adam: you know there's lots of libraries out there for machine learning things in python and,
Adam: in the machine learning space python's mostly
Adam: used as a wrapper around some c library
Adam: that's super well optimized rust is i'd
Adam: say catching up and like hugging face has
Adam: built some rust libraries that kind of do the same thing
Adam: as the python equivalents so but yeah
Adam: most most of the time if i'm building a web server today
Adam: i'll just grab actix and and run with it you know if i'm working with data and
Adam: Postgres i'm going to grab sqlx 10 years ago it would have been like flask or
Adam: fast api or maybe not 10 years ago for fast api but you know i have the equivalent
Adam: of everything that I would,
Adam: you know, five years ago would have gone to something in Python. I have,
Adam: For me, I'd just pick Rust. It's just easier. It's a better experience.
Matthias: And on that very positive note, what's your final message to the Rust community?
Adam: Yeah. I mean, keep contributing to the project. The project doesn't live unless
Adam: people are working on it and making it better.
Adam: Today, there's really no such thing as finished software or a finished programming language.
Adam: It's not like software is distributed in the
Adam: mail where you burn stuff to a disc and mail it out and
Adam: it just runs that way forever software is living breathing organisms now so
Adam: things have to be constantly fed and for rust like if people stop contributing
Adam: to rust and you know like the project won't go on.
Adam: So I guess my biggest message would be thank you for building awesome stuff
Adam: and please keep doing it.
Matthias: Adam, thanks a lot for the interview.
Adam: Thank you.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.