Matthias: It's Rust in Production, a podcast about companies who use Rust to shape the
Matthias: future of infrastructure.
Matthias: I'm Matthias Endler from corrode, and today we talk to Andrew Burkhardt from
Matthias: 1Password about securing logins with Rust.
Matthias: Andrew, thanks so much for being a guest today. Can you say a few words about
Matthias: yourself and about 1Password?
Andrew: Yeah, definitely. Thank you for having me. So I'm Andrew Burkhardt.
Andrew: I'm a senior Rust developer at 1Password. I work on the product foundations
Andrew: team, or in the org, I should say, on the frameworks team.
Andrew: So we specifically focus mostly on async Rust frameworks.
Andrew: I've been here about three years now and working on data synchronization mostly,
Andrew: which has been pretty interesting as we grow.
Andrew: We spread kind of to other products that have come in. You know,
Andrew: the password manager has kind of historically been our core business,
Andrew: and that's what people have known us for for almost 20 years now.
Andrew: But, you know, over the last few years, especially, we've grown a lot and acquired
Andrew: a few companies and so forth and really spread ourselves across the security space,
Andrew: you know, with various things from the password manager to, you know,
Andrew: developer tools and a lot of like enterprise password management features.
Andrew: And we have things like connect servers and so forth that, you know,
Andrew: skim bridges that help with identity management, all these things.
Andrew: And then into the companies that we acquired, like Passage, which helps with
Andrew: Passkeys, Collide, which does on-device health checking, Metrolika,
Andrew: which handles provisioning and some things like that.
Andrew: So, yeah, so across all those various areas, you know, sync has been a really,
Andrew: really interesting use case to kind of combine all the various information we have in different ways.
Matthias: And by sync, you mean actually combining all of the information from devices
Matthias: or user profiles or what does that mean exactly?
Andrew: Yeah, so the sort of summary, I guess, would probably be, you know,
Andrew: federating or, you know, getting the data to be the same from clients to the
Andrew: Rust core to the server and all the way back around to any clients that are relevant.
Andrew: So that may be something as simple as, you know, you and I are on the same one
Andrew: password account in the password manager and I add an item that you now need.
Andrew: So we sync that data across, but it can also be other things like,
Andrew: you know, if you're in the enterprise space,
Andrew: just making sure that the data from one place talks to another and so forth,
Andrew: so that if there is relevant context from one area, that is available, area, things like that.
Matthias: Yeah i can totally relate to that because one of the worst
Matthias: developer experiences is if data that
Matthias: you expect to be there isn't there or is in
Matthias: a sort of limbo and i think sync needs to be really reliable and that's the
Matthias: core of the product if that is not reliable then well everything else is just
Matthias: nice user experience but it's not really like i would say user interface but
Matthias: not a great user experience.
Andrew: Yeah definitely and that's you know it's created
Andrew: a lot of you know patterns i guess you
Andrew: would say that we hold pretty strongly to like you know sync
Andrew: here conversations about this this morning actually sync here is
Andrew: always sort of a background task so anything
Andrew: that you're doing sync should essentially be a side effect
Andrew: of it and it should happen asynchronously in the background and you know that
Andrew: way you kind of always assume that the data you have is current because sync
Andrew: has been happening in the background and therefore you have the expectation
Andrew: that that data has already been brought locally to your device,
Andrew: which gives the feature developers a little bit of freedom because you don't
Andrew: have to go request things and wait for it and do this.
Andrew: I mean, there are cases where that has to happen, but in general,
Andrew: the vast majority of the data has already been brought locally and cached and so forth.
Andrew: So I can just act on that cache as though, you know, I'm the only one using
Andrew: this data or something, which is convenient in a lot of cases.
Matthias: It's sort of funny that you need async in order for something to be synced,
Matthias: but it does make sense if you think about it because, well, it happens in the background and,
Matthias: A lot of things need to happen for it to look synchronous, but it's not easy.
Andrew: Yeah, yeah, definitely. And you get into the nitty gritty details of like synchronous
Andrew: versus synchronized and so forth. And we definitely overload the word sync probably way too much.
Matthias: So take us back a couple of years ago when you started. You said you're at 1Password
Matthias: for three years now, I guess.
Matthias: What was your background back then? Where do you come from? And then what was
Matthias: the state of Rust at 1Password when you started?
Andrew: Yeah, so I had done mostly, my prior job had been a lot of Kotlin and Spring
Andrew: Boot, which, you know, is kind of like, it's a framework for Kotlin doing server side stuff.
Andrew: And I was doing a lot of that, like backend web programming.
Andrew: I also worked on the front end a little bit with TypeScript and the Vue framework for UI.
Andrew: But I wasn't really like a front end developer. I did, you know,
Andrew: it's referred to, I think, as like the BFF pattern where like,
Andrew: you have one server that manages a lot of different things.
Andrew: And then you have all these front ends that manage pieces of that,
Andrew: right? Like, for example, I worked on an app that did a lot of settings,
Andrew: like behind the scenes settings.
Andrew: And then we had a lot of other apps where actual end users would interact with
Andrew: it. And the one main server powered it all.
Andrew: But instead of muddying that server with, you know, all these either,
Andrew: you know, like you could use something like GraphQL, I suppose.
Andrew: But like instead of muddying it with all these APIs that supported vastly different
Andrew: use cases, we created these BFF servers where each front end has its own back
Andrew: end that then translates to the real sort of underlying back end.
Andrew: So I'd always done kind of that mid layer.
Andrew: Of development. And that was actually very helpful because it turns out,
Andrew: you know, getting into the state of Rust and 1Password is that's kind of how Rust is used here.
Andrew: So each of our apps, you know, obviously we have quite a few between iOS and
Andrew: Android and Mac and Windows and Linux and web and extension and all these different places.
Andrew: All of them are native clients.
Andrew: So they operate, you know, the iOS app is in Swift and so forth.
Andrew: And then they are bundled with kind of like an embedded backend,
Andrew: which is a Rust, we call it the core.
Andrew: So the Rust core is inside of all of these and that is doing as much of the
Andrew: business logic as we can to leave those native clients to just be display layer,
Andrew: essentially a presentation layer.
Andrew: And then that communicates to the actual cloud server and so forth.
Andrew: And so Rust got picked up here. I don't actually know when it was,
Andrew: but my understanding is it was originally picked up for the Windows app that was being built.
Andrew: And And over time, when we got from 1Password 7 to 1Password 8,
Andrew: I think it was, again, it was slightly before me, but I think that was when
Andrew: it was like, okay, we need more cross-platform functionality because we're building
Andrew: so many features that were not originally just password management concerns.
Andrew: We're building all these different things, and they need to work on all these
Andrew: platforms, and we're implementing them over and over and over again.
Andrew: And so when they decided they wanted to create sort of the singular core that
Andrew: was going to power all the apps, they had already started to use Rust on Windows
Andrew: and it had worked really well. And so a lot of it just got ported over.
Andrew: And so when I got here, we were just in this kind of big growth phase back in
Andrew: 2022 and a ton of new people coming in. And so Rust was sort of proliferating
Andrew: very quickly as the core was building, you know, all these developer features
Andrew: and these different things.
Andrew: And it was right about then we did a reorg of the development teams and created
Andrew: the team that I'm now on, which is the frameworks team,
Andrew: along with a few other teams that do, for example, like security specific development
Andrew: or platform specific development, like handling the nuances of Windows or Android or whatever it might be.
Andrew: And all of us kind of came together to design patterns and rules and so forth
Andrew: around Rust development to try to sort of wrangle all these different things that were happening.
Andrew: And that's kind of got us into the patterns that we have today,
Andrew: which have been really nice to keep this giant core as maintainable as possible.
Matthias: Nice. I want to get back to the patterns in a second. But before we do that,
Matthias: what did you use before Rust? Was it very platform dependent?
Matthias: Did you have this business logic in, for example, Kotlin and then Swift and so on?
Andrew: Yeah, exactly. So, like I said, that was all before me.
Andrew: But my understanding is that everything was basically built natively,
Andrew: which gives you some benefits, right? There are pros and cons of that.
Andrew: But at the scale that we're at now, it's just not feasible to have that much
Andrew: code and have that many different implementations. And obviously,
Andrew: just when you're building a security app, you don't, multiple implementations is a problem, right?
Andrew: Anytime you have to do that, there is now the risk that one of these implementations
Andrew: is vulnerable in some way that the other isn't.
Andrew: And we're trying to compare, you know, Swift code to Kotlin code to TypeScript
Andrew: to, you know, Go or whatever and so forth, and make sure that these are all
Andrew: doing the exact same thing.
Andrew: But those languages may have different underlying models for async or memory
Andrew: management or whatever it is. And so it just wasn't feasible to continue on like that.
Andrew: And so the REST core really helped us to, I mean, it just enabled features that
Andrew: we really weren't, that weren't possible before.
Matthias: In my idealized way to think about it,
Matthias: what I hope would happen is that once you merge in all of that code into one
Matthias: code base, into one Rust code base,
Matthias: that you see how different platforms or different languages solve these problems
Matthias: or different developers solve the same problem and some of them solve different edge cases.
Matthias: And then when you merge that together, you get a thing that is stronger than
Matthias: its parts. did that happen?
Andrew: Yeah I think so I think and you get the opposite too
Andrew: right where it's like you know we'll get one component and it's like what is
Andrew: this thing doing it's like oh we ported that from this other version like
Andrew: oh that was the worst version why did we pick that one you know and so you do
Andrew: see that but yeah I think in a lot of ways you definitely get the benefit of
Andrew: that and we see that having native clients I mean we've got developers who have
Andrew: done Swift and Mac development for you know 20 years or whatever it might be
Andrew: and so then they come at it and it's like okay hey hey, we need to build this feature.
Andrew: You know, I need some help with the Rust side of it. And you see something and
Andrew: you're like, that's not at all how I would have thought to do it.
Andrew: And they're like, well, here's how we do it, you know, in Mac land.
Andrew: And you start to get interesting new ideas, I think, about how you can handle,
Andrew: you know, how you can tackle various problems.
Andrew: So I think we get that just by nature of having all the different developers,
Andrew: but we definitely saw it in the growth of the Rust core and trying to pull things together.
Andrew: And now trying to add in some of the browser features that we had implemented
Andrew: natively in the browser, now that we're trying to pull them into Rust and compile
Andrew: them into Wasm, we're seeing some of those patterns again that we can learn
Andrew: from and improve throughout.
Matthias: Was the Rust ecosystem prepared for what 1Password asked it to do?
Matthias: Or were there any gaps in the Crate ecosystem, for example?
Andrew: I think there were probably some. I would say, in general, yes.
Andrew: I think the ecosystem was growing so fast and.
Andrew: There were so many people building interesting things that I think it gave us
Andrew: a lot of freedom to innovate on top of those.
Andrew: I think there were small gaps, which is where you've seen some of the crates that we've put out.
Andrew: Gaps like, for example, having those native clients, there's no type safety
Andrew: across the FFI necessarily.
Andrew: If I'm writing Swift and then I send something into the FFI and then into the
Andrew: Rust core, I'm just hoping the types line up.
Andrew: So we created the type share crate, which is, it's not giving you necessarily
Andrew: complete type safety, but it is at least making sure that you're not writing
Andrew: the types on the client side.
Andrew: They are generated from your Rust code so that if the Rust code changes,
Andrew: the client side types change. You're not having to manually keep those in sync.
Andrew: So I think that's been really helpful.
Andrew: And then there are other little things. Another one we released somewhat recently
Andrew: is called zeroizing alloc.
Andrew: Which is like a wrapper around the global allocator that makes sure,
Andrew: you know, if you're using this, whenever memory would normally be deallocated,
Andrew: which in that case, you know, it's sort of just dropped, right?
Andrew: It's free for the system to use for other things.
Andrew: We don't want to just let that sit around in case the system doesn't need it.
Andrew: And that memory sits there forever, right? We can't have that when you're dealing
Andrew: with sensitive data. So the zeroizing alloc crate will help to just flip those
Andrew: all back to zeros during that deallocation process.
Andrew: So there were little things like that, that obviously Rust as a language was
Andrew: ready for. There just hadn't been that use case necessarily.
Andrew: And so I think it created nice gaps for us to fill with some of those things.
Matthias: Yeah, I came across this zeroizing alloc crate.
Matthias: Actually before that, because the link was purple. So this is how I know.
Matthias: I'm surprised that no one else built such a crate before. Or did you look at
Matthias: some and didn't really like them?
Andrew: I don't know how we came across that. And I wonder to some degree,
Andrew: like if you look at the crate, it's not particularly complex,
Andrew: right? It's a fairly straightforward implementation.
Andrew: And I've seen this in other cases where...
Andrew: I think sometimes we look at a problem and we go, oh, it's so simple,
Andrew: like I don't need to make a crate to deal with this problem.
Andrew: You know, it's maybe a common problem, but it's like, I'll just throw this together.
Andrew: And, you know, maybe people don't want the dependency or something like that.
Andrew: I'll just kind of hand roll it very similarly or something along those lines.
Andrew: I think sometimes that happens, right?
Andrew: And people are obviously wary sometimes of dependencies, you know,
Andrew: especially ones that do smaller things, right? We saw, I think,
Andrew: in the TypeScript NPM world, there was like the left pad incident,
Andrew: right? where like all this thing is doing is handling this left padding.
Andrew: And it's so simple, but then when it's gone, now we have a big problem, you know.
Andrew: And so that could be the reason that some of those things just don't get created.
Andrew: It may not be, you know, I doubt we were the first people to think of it.
Andrew: You know, it's probably something someone else had solved before.
Andrew: They maybe just didn't make a crate for it or something like that.
Andrew: Or maybe there is one out there and I just haven't seen.
Matthias: What is the policy around creating crates at one password?
Andrew: You know, there's, this has been an interesting thing that I've seen here,
Andrew: right, is because one password, the password manager concept is 20 years old, right?
Andrew: And conceptually, a lot of it is similar to what it was then.
Andrew: And then the Rust app itself, you know, now is, you know, five,
Andrew: six, seven years old. I don't know exactly how old, but it's been around a while.
Andrew: You have these things that have grown so much and for the most part creating
Andrew: new crates internally it's just a matter of code architecture does it make sense,
Andrew: is it a necessary separation a lot of problems you deal with anywhere,
Andrew: making crates public is sometimes tricky because one you have to figure out
Andrew: does this make sense to make public like zeroizing alloc, we probably could
Andrew: have left that internal but we figured it might help some people,
Andrew: And it's not like a, you know, private, you know, secret thing that 1Password does.
Andrew: I think a lot of people know that, you know, we have to deal with memory this way.
Andrew: So you have some of those initial concerns about whether or not we should make this public.
Andrew: And then it's like untangling it from everything else you've built over the last few years, right?
Andrew: And, you know, there are things like our, you know, clipboard manager,
Andrew: it's called Rboard, that's made public, you know, and you got to go,
Andrew: okay, well, we had all these, you know, 1Password specific assumptions.
Andrew: Now we have to pull those out and make this just a dependency as though it were
Andrew: a third party dependency to us as well, you know, and so a lot of it is just,
Andrew: just small bits of process of untangling things.
Andrew: But, but as far as deciding to make it public, that's pretty case specific,
Andrew: I guess, or situation specific.
Andrew: And I think we've been trying to, to get more and more out there.
Andrew: You know, we released, for example, the passkey RS crate, which I think is only
Andrew: valuable to other password managers.
Andrew: Like, I don't know if there's anybody because it really just validates,
Andrew: like, does verification of passkeys. And so I don't know that anyone else would ever need it.
Andrew: But, you know, we released it because, one, it might help them.
Andrew: And we really want to see passkeys get adopted in more places,
Andrew: even if that's, you know, in a competing app.
Andrew: We want to make sure that that's available and so forth. And it's also just
Andrew: interesting, you know, rust that people might be able to learn from and so forth.
Andrew: So, yeah, it definitely depends on the situation, but it's a bit of a process either way.
Matthias: Maybe one of the listeners checks out that crate and builds a thing that we
Matthias: both haven't anticipated.
Matthias: It happens more often than I expect.
Matthias: And the other thing is open sourcing software.
Matthias: Whether it's Rust or not is a huge lift for every company, because what a lot
Matthias: of people forget is everything other than the code, versioning,
Matthias: documentation, testing, CI, communication, and so on.
Matthias: And also, to some extent, exposing how you build things at a certain company.
Matthias: And also, a lot of developers are afraid to share how they write code.
Matthias: And did you ever get any external contributions to make it worth it?
Andrew: Yeah, I mean, especially like with TypeShare, I think I'm trying to think of what language it was.
Andrew: I want to say like Scala or something like that. Like some, you know, we don't use Scala.
Andrew: And so we never built, you know, a TypeShare integration with Scala,
Andrew: but someone from the community did.
Andrew: And so now I want to say it's Scala. Now there is, you know,
Andrew: the capability to convert your Rust types into Scala and so forth.
Andrew: And I think those are nice. We don't, I don't know that we've ever really open
Andrew: sourced anything with the hope that people would build features that we need.
Andrew: You know, we generally are trying to do it sort of the other way around,
Andrew: like, you know, make sure that our work is helping people where we can.
Andrew: Like you said, it can be tricky, especially just from an operational standpoint, right?
Andrew: There's, I think I saw some, we have like 150,000 businesses that use 1Password
Andrew: or something like that or more than that.
Matthias: Wow.
Andrew: So when you have that number of people, the number of needs is pretty high,
Andrew: right? We have a lot of, this company needs this thing and this company needs this thing and so forth.
Andrew: And if you open source something, you lose the ability to make breaking changes quickly.
Andrew: You don't want to, I mean, I guess you technically could, but that's not a very
Andrew: good open source project if you're just breaking all the time.
Andrew: So that's one of the harder parts is when you're making that decision,
Andrew: is this something we can open source?
Andrew: And can we give, like you said, versioning, can we give any kind of guarantee
Andrew: that if we make version one, And version one is stable and safe and it's good.
Andrew: That's fine. But now if we need to make version two a breaking change because
Andrew: we need it for something we're trying to build, well, now version one kind of becomes unsupported.
Andrew: So if version two doesn't work for what you're trying to do,
Andrew: well, now what if there's a security vulnerability with version one?
Andrew: Now you've created these two branches that should be maintained by someone.
Andrew: And if that happens a second time and a third time, you just create this web
Andrew: of challenges. So, you know, it would be great if we could open source everything,
Andrew: you know, and let everyone see all that.
Andrew: And I'm sure there are paths to that. It just doesn't, it's not very easy.
Andrew: Like you said, it's very challenging. It's not as simple as like,
Andrew: well, just make it public.
Andrew: You know, you've got to do a lot of management to make that work.
Andrew: And, you know, as things ebb and flow, that can be really hard to keep up with,
Andrew: you know, responsibly and reliably.
Matthias: Yeah. And even if you didn't open source it, and even if you just used a crate
Matthias: across different teams, that is still a challenge. Do you have a monorepo?
Matthias: What does the structure look like right now?
Andrew: Yeah, so it is a monorepo. We have various repos for services and things like
Andrew: that, but the Rust core itself and all of the client apps built on it is a monorepo,
Andrew: which gives us some benefits from sort of a build and release avenue, right?
Andrew: Where everything, you don't have to bring in other, you know,
Andrew: repos and so forth to keep everything up to date. It's all in one place.
Andrew: But like you said, that means that, you know, you know, merge conflicts can
Andrew: be a problem and all these different things that are reliant on what you're working on.
Andrew: And that's, that's tricky, especially from, you know, a team sitting where we
Andrew: do kind of at the very bottom of the whole stack.
Andrew: You know, when we make a change, there tends to be a very significant ripple
Andrew: effect that, you know, and you get into challenges like, how do we manage that, right?
Andrew: Does our team, you know, if you have code that depends on that,
Andrew: do we just tell you, hey, we're going to break this and you need to be ready
Andrew: to work with us to do the new thing or do we fix it for you,
Andrew: you know, and so forth. And there can be a lot of communication that needs to occur as you grow.
Andrew: And, you know, you talk about like the mythical man month, you know, concept, right?
Andrew: Of like throwing two more people
Andrew: onto a project does not give you two more people's worth of time, right?
Andrew: You have extra communication. There's all these new channels that have been
Andrew: created between those people and the other people and so forth.
Andrew: And it will test your processes and
Andrew: your org structure and your code architecture, you know, at every turn.
Andrew: And so when you get to hundreds of developers working on sometimes completely
Andrew: different, you know, things, right? They're not even password management concepts.
Andrew: They are, you know, they may be related to secrets, right? Like the SSH agent that 1Password has.
Andrew: Well, that's related to some amount of secret, but it is not at all related,
Andrew: you know, to profiles or settings in the password manager or things like that.
Andrew: So you can get into challenging situations where you're trying to coordinate
Andrew: underlying frameworks across completely different feature teams and,
Andrew: Yeah, that's problematic at times, and it definitely can slow things down,
Andrew: but it also becomes a strength when you're learning from things.
Matthias: Let's talk scale, because people like numbers.
Matthias: Can you share some of the, let's say, I would say more interesting numbers about
Matthias: the code size and the number of people working with Rust and crates maintained and so on?
Andrew: Yeah, so we are, I think I just looked the other day and we were at like five
Andrew: or 600,000 lines of rust in the core across like 600-ish crates.
Andrew: I think we're just shy of 600 crates internal plus dependencies.
Andrew: So it's a pretty big code base, right? And like I said, that core makes up as
Andrew: much of the business logic as we can get it to.
Andrew: So I don't know the exact spread of all of it because, you know,
Andrew: when you're implementing a feature, you have to also implement it in the native clients, right?
Andrew: So some amount, you know, every feature is going to have some Swift and some
Andrew: Kotlin and TypeScript and all these different things.
Andrew: So I would say, you know, and then you've got server-side stuff,
Andrew: which not all of our server-side stuff is in Rust. There's a lot of Go on the server side.
Andrew: So I would say probably 50% of everything new that we write is in Rust,
Andrew: but it's definitely spread across all those other things.
Andrew: You know, writing an end-to-end feature will involve Go and Rust.
Andrew: And Swift and Kotlin and TypeScript and possibly go on the CLI side and so forth.
Andrew: So there's quite a bit that goes into that. But there's probably,
Andrew: I think we have a couple hundred developers now working across all the different
Andrew: products that 1Password and Collide and Passage and Trelica and so forth have.
Andrew: But we probably have, I don't know, I'd say 100 people that work pretty heavily
Andrew: in Rust or maybe 50% or more in Rust, something like that. But that's definitely a bit of a guess.
Matthias: Awesome which ffi bridge works the least well for you right now like what gives
Matthias: you the most headaches at the moment.
Andrew: It depends i think as far as the actual ffi goes the most challenging ones are
Andrew: probably well i guess android can be a little bit tricky sometimes,
Andrew: you know, especially just Android, the ecosystem's a little bit different.
Andrew: You know, we were built originally as a macOS app. And so I think there's still
Andrew: a little bit of that sort of, you know, Apple product DNA built in,
Andrew: which, you know, we've tried to spread to all these different platforms and
Andrew: make everything work equally well. But I'm sure there's still a little bit of that in there.
Andrew: And then the Wasm boundaries can be tough.
Andrew: I didn't know anything about Wasm until about a year ago, right?
Andrew: I don't think I'd ever really touched it at all. I'd heard of it and knew nothing about it.
Andrew: And then you find out little things like, I think there's like no built-in clock
Andrew: in Wasm or something like that.
Andrew: So you're like faking all these time-based things that you used to do.
Andrew: And so that can be a little bit tough when you have concepts baked into how
Andrew: the native clients communicate across the FFI that just don't work in Wasm.
Andrew: And we end up with a lot of abstractions.
Andrew: Like we have, you know, there's send and sync traits in Rust,
Andrew: but we have a maybe send and maybe sync trait that handles, you know,
Andrew: four people translating between, oh, is this going to be single threaded or
Andrew: multi-threaded and so forth?
Andrew: And so the WASM boundaries are probably the toughest, to be honest.
Matthias: Okay. Yeah. The maybe send and maybe sync sounds interesting.
Matthias: Wouldn't that be a question mark send and question mark sync?
Andrew: I think it could be. Yeah, I think what we have, so the way we've broken this
Andrew: out, right, is because we want, you know, code to be reusable across platforms, right?
Andrew: But it may not necessarily work quite the same. Let's say for pass keys,
Andrew: you know, or autofill, like, that happens completely differently on iOS than
Andrew: it does on Android than it does on, you know, Windows or wherever.
Andrew: And so there's all these different implementations.
Andrew: But you want to do the same process. It's just that all the underlying calls,
Andrew: you know, there may be syscalls or things like that, we reaching back out in
Andrew: the OS, they work different.
Andrew: Or, you know, some of them, you have to do it in the moment,
Andrew: some of them, you have to seed some operation at unlock, you know, or whatever.
Andrew: And so what we try to do is we have these, you know, we have FFIs by platform.
Andrew: And then we have like an app level, where you could have multiple platforms
Andrew: that use the same app level crate.
Andrew: But then what that does, that reaches out to services. And the service level
Andrew: is where all of your actual functionality comes in.
Andrew: For example, sync, there is a service that handles sync, and there's,
Andrew: you know, a different service that handles autofill and so forth.
Andrew: And then those reach out into underlying implementations of things.
Andrew: And so what we do is we try and make a trait that is like, okay,
Andrew: here is, you know, let's say, I don't know, pass keys or something like that.
Andrew: And then underneath that, it's like, all right, get the pass keys.
Andrew: But then below that, it's like, well, what do we need to get?
Andrew: Which data are we getting?
Andrew: When are we getting this? You know, how is it being provided?
Andrew: Do we need to get anything from, you know, the enclave or the operating system
Andrew: or whatever it is? and those get implemented by platform.
Andrew: And that gives us a little bit of flexibility into how we.
Andrew: Build up things. But what that means is, if I'm doing something that is generally,
Andrew: you know, we have built this and have used it, you know, let's say four or five
Andrew: years now, completely async, right?
Andrew: And we've been using, you know, we've spent up a ton of threads to do this thing
Andrew: or whatever, or it's got to be on a background task or something like that.
Andrew: Well, now maybe that doesn't work in wasm.
Andrew: And so I think maybe send and maybe sync have been our easy,
Andrew: traits to kind of layer in to be like, all right, put this on there to make sure,
Andrew: you know we can sort of keep these things
Andrew: moving forward and then that becomes a big challenge throughout
Andrew: everything right when you've got you know 500 000 lines of
Andrew: rust and now you're trying to build in
Andrew: wasm that's a lot different than we're starting from scratch and
Andrew: building in wasm support right we could have done things probably a
Andrew: lot differently but there are hundreds of features across
Andrew: you know a dozen products or whatever that are
Andrew: trying to use this stuff and going back and refactoring them maybe just doesn't
Andrew: doesn't add up right now and we don't have time for that and so there are probably
Andrew: other ways we could use those x or whatever they call them to to better better
Andrew: use i think these are probably our temporary temporary solutions but.
Matthias: I fully get it because at some point you also need to ship things and this seems
Matthias: like an escape hatch that kind of was a pragmatic approach to solving this problem.
Matthias: What a lot of people forget is that these are apps that eventually need releases
Matthias: to change or if they have breaking changes, then it's not that easy because
Matthias: they might be used in other parts of the application.
Matthias: So the other thing I wondered was,
Matthias: Is there a guideline for keeping the core synchronous Rust?
Matthias: How deep does the async part go? And at what point do you stop using async?
Matthias: And do you start having your own little, say, domain, which is sync?
Matthias: Is there such a differentiation? Or would you say that doesn't make any sense
Matthias: at your scale or for the project you do?
Andrew: You know, it might make sense in some places. I don't know that we really do
Andrew: it much, though. It's definitely pretty async down to the root of things.
Andrew: I mean, the way our requests from the client into the core work is,
Andrew: you know, you make this request over the FFI, which is sort of inherently async, right?
Andrew: Because that's not an API where you're waiting, you know, it's not baked into
Andrew: like the HTTP protocol where you're going to get a response and blah, blah, blah.
Andrew: So you sort of send this request into the core of the FFI. And then what the
Andrew: core does is that FFI holds on to a sender and it hands off that request to
Andrew: a thread that is watching the receiver.
Andrew: And that dispatches invocations, you know, these requests that do the work and
Andrew: handle the type process and blah, blah, blah, and then come back.
Andrew: And we, you know, the way that we handle that is the front end sends in a callback
Andrew: that once the request is done processing,
Andrew: that invocation dispatch loop executes the callback with the resulting data
Andrew: as an input to that callback.
Andrew: So that way, the client side handles sort of a registry of here are all the
Andrew: requests that I have made and sort of an identifier.
Andrew: So when that callback gets executed, I can put it into my local cache or store
Andrew: or plug it into this React component or whatever it might be.
Andrew: But the whole process is sort of inherently async.
Andrew: It is baked into everything from client side all the way down to the database and back.
Andrew: So I'm sure there are some things where it's like, hey, we need to do multiple
Andrew: things and they need to be synchronous in order, but they're a bit of like a
Andrew: bubble that's still within an async, you know, flow end to end.
Matthias: The callback mechanism that you describe sounds a lot like the Waker in Tokio,
Matthias: but just on a higher level, I would assume, like on a data level somehow.
Andrew: Exactly. Yeah, it is kind of like that where the we are.
Andrew: It is sort of, yeah, I guess it's like building in our own hand rolled waker
Andrew: in some way of like the the client technically sends off this request and it's done in theory.
Andrew: Right. Or technologically it is done. But it has set up this function,
Andrew: you know, that is somewhere in memory, and it has handed us off where this thing is.
Andrew: So then when we go call that code with this data, it is now kicking off a new
Andrew: thing. It just so happens that we have in code tied those two pieces together.
Andrew: Right? So it's sort of like the mechanism of a waker, except that we're not
Andrew: actually, you know, pulling it or anything like that.
Andrew: It is inherently async to when that thing gets called back.
Matthias: Did you have to use any unsafe code to pull it off?
Andrew: I'm sure in the FFI there's some, you know, just the nature of, you know, how FFI works.
Andrew: I'm sure the calls across the actual FFI are wrapped in unsafe.
Andrew: And then I think there's probably some, you know, we talked about zeroizing alloc.
Andrew: I'm sure there's some amount of wrapping there to ensure that bytes are flipped
Andrew: to zeros, you know, in things that were not being handled by the Rust allocator.
Andrew: But in general, the process is fairly safe, right?
Andrew: You make the call over the FFI, and because that is holding this object with
Andrew: the sender, once you send that all across, you know, that's fine.
Andrew: Everything there can be done safely. And then executing the actual callback
Andrew: probably technically has to be wrapped and unsafe, I would guess,
Andrew: because you are sort of triggering just a memory location.
Andrew: But a lot of that is based on, you know, guarantees that we can make in both sides of the equation.
Matthias: So you mentioned 600 crates. And when I heard that number, I was, well, awestruck.
Matthias: How do you not drown in dependency hell? How do you keep all of these crates up to date?
Andrew: Yeah, I think you're assuming we aren't drowning in that. No,
Andrew: it's, it's, it's not too bad. I mean, so part of it is automation, right?
Andrew: You know, making sure that we're running things to ensure like we have dependency
Andrew: monitoring for vulnerabilities and stuff like that to keep all that stuff up
Andrew: to date and make sure that we're not, you know, we're not passing pipelines
Andrew: that have vulnerable dependencies.
Andrew: And also, if we have a dependency that has become vulnerable,
Andrew: we're updating it quickly and so forth.
Andrew: But beyond that, it can be tricky, you know, to manage all those dependencies.
Andrew: A lot of it just comes with the architecture of like, let's make sure,
Andrew: you know, we have directories, right? So we have, let's say, a data directory.
Andrew: And this data directory, you know, okay, you are allowed to import anything
Andrew: else that's in the data directory or any, you know, third party dependency that's safe.
Andrew: But you can't import anything from services or apps or so forth,
Andrew: right? And we create a bit of a inherent hierarchy to these things.
Andrew: And so a lot of it is just making sure that we have clearly defined like app crates and.
Andrew: Cannot, or app grates can really import anything that they want, except the FFI.
Andrew: And app grates should only be imported by the FFI. No one else should ever import
Andrew: an app grate. So there's our first, you know, level. And then we go to services.
Andrew: It's like, all right, services can be imported by each other or app grates, but nobody else.
Andrew: And now, you know, data can do this, right? And you create a little bit of these
Andrew: rules, and then you end up with a ton of things that fall outside of,
Andrew: you know they may be crates that define types or
Andrew: a server client or all these different things and that's
Andrew: where you get into the difficult parts of like well if we have
Andrew: a server client who's allowed to use that you know
Andrew: can anyone call that from anywhere or should
Andrew: that ever talk to the data you know crates or
Andrew: should we keep those separate inherently what about types what
Andrew: if we have a type that is this way on the server and it is also that
Andrew: way in the database well do we want to duplicate that to let those
Andrew: like to let drift be okay so that way we're not you know
Andrew: modifying both all the time or one all the time so you
Andrew: definitely get into challenges that that come with having that many crates and
Andrew: there's probably more automation that wants to happen but there's definitely
Andrew: you know the short answer i guess is there is a lot of human process that comes
Andrew: into you know we have different teams that own different concepts they don't
Andrew: own necessarily specific bits of code.
Andrew: They may, but in general, they own concepts.
Andrew: And because you're owning that concept, we have automation, you know,
Andrew: that we'll say, okay, these bits of code definitely touch that concept.
Andrew: Or just a lot of people going, okay, this should probably be reviewed by such
Andrew: and such team or so forth.
Andrew: And just making sure that we keep everything as clean as possible.
Andrew: And that we are, when we do code review, we're looking at it to go,
Andrew: does this order of imports make sense?
Andrew: You know, and we'll have things with like, oh, they implemented this on some
Andrew: model, but it doesn't need to be there.
Andrew: It was only being consumed in one place. So let's just inline all of that.
Andrew: Let's get rid of that crate, you know, that whole model concept.
Andrew: Let's just inline it right where it was.
Andrew: And then we may find out in the future, oh, well, now somewhere else wants to
Andrew: use it. Okay, let's pull it back out and put it back into the model and we'll
Andrew: do that, you know, and that's okay.
Andrew: You know, we want to avoid the like, well, we might need it in the future because
Andrew: that's when crates really become a problem.
Andrew: You know, we put it in this new crate because we might need it somewhere else.
Andrew: Let's not do that let's just enlighten till we know we need it somewhere else
Andrew: and then we'll deal with pouring out and that's really the only way that you
Andrew: know you kind of bake in avoidance of crates just proliferating everywhere it.
Matthias: Sounds like you're treating crates like cattle i wonder how much the rust ecosystem
Matthias: helps with that and how much of that is still manual because there are tools
Matthias: like cargo deny for example or,
Matthias: I don't know. Do you have any tools that you like outside of,
Matthias: say, Clippy that you prefer?
Andrew: Yeah, a lot of Clippy here. Definitely Cargo Deny as well.
Andrew: Make sure that, again, that if it's an FFI crate, you should only be importing
Andrew: certain things, or if it's an app, you shouldn't be importing certain things or so forth.
Andrew: Again, it probably wants more automation than we have.
Andrew: A lot of it is just the code architecture, right? If we make sure that the data
Andrew: layer is self-contained, and then we make sure FFI to app to service is a self-contained
Andrew: flow, well, everything else then becomes domain-specific, right?
Andrew: We're not creating global things beyond that, wherever we can avoid it at least.
Andrew: So that way, if your team, let's say, works on autofill, you get to use the
Andrew: data layer in prescribed ways, and then you import all your stuff into a service
Andrew: which goes into an app which goes into the FFI.
Andrew: Anything in between there, that's up to you. You know, if you create a circular
Andrew: dependency or whatever, that's your problem.
Andrew: Or if you create a bad architecture between these things, well,
Andrew: you have to own that, right? You've got to maintain that.
Andrew: So you're going to find the problems with that and generally fix it yourself.
Andrew: And so a lot of it is just bad architecture of like, keep the top layer contained,
Andrew: keep the bottom layer contained.
Andrew: And then everything in the middle is feature specific or domain specific.
Andrew: So that way, those bounded domains are dealing with themselves.
Andrew: And there's a couple other things that have this, like we have a foundation
Andrew: directory, which deals with OS-specific calls.
Andrew: You know, anything that's Windows-specific has its own area and so forth.
Andrew: So there are a few other things we do like that, where it's like,
Andrew: okay, that is maintained by this platform advancement team.
Andrew: And if you need changes there, you work with them to make those changes.
Andrew: But you are just consuming those things. And any, you know, interweaving of
Andrew: crates that you do within your bounded domain needs to happen within the box you were provided.
Andrew: But it definitely does rely on a little bit of that automation,
Andrew: like you said, of like cargo deny and clippy and so forth, to ensure that we
Andrew: are cleaning that up where we can, but it's definitely a manual process a lot of times.
Matthias: I'm not sure if it's true, but I heard that 1Password uses Nix for their development environment.
Matthias: Can you say a few words about that?
Andrew: Yeah, so we definitely have a lot of Nix usage.
Andrew: So it powers our, on the core side at least, it powers a lot of our build processes and stuff like that.
Andrew: And then we use it to manage our tool chains on the core side as well.
Andrew: And that has been really, really nice because there is so much,
Andrew: you know, tool chain stuff that you need when you're working across all these platforms.
Andrew: You know, you might need all of these Linux-specific tools and you need these
Andrew: Mac-specific tools and all these things that make Android Studio work and there's
Andrew: all these different things that come together as well as, you know, internal tools, right?
Andrew: We have the 1Password CLI, which we use for things and the TypeShare tool,
Andrew: which is a CLI tool that is used for things And so whenever we make updates
Andrew: to TypeShare, that needs to become part of developers' toolchains.
Andrew: And so we use Nix to manage all of that.
Andrew: And that way, whenever there is an update, if I pull main and that Nix flake
Andrew: has changed on main, well, my toolchain just gets updated.
Andrew: I don't know the difference. So, you know, I have to wait a few seconds for
Andrew: it to pull everything in, build the tool chain. But from that point, I don't care.
Andrew: And there's the direnv tool, which basically sets up that directory to automatically
Andrew: use the Nix Flake as your tool chain for that directory. So I don't have to
Andrew: worry about version management.
Andrew: I don't have to worry about, you know, did I run the install script too long
Andrew: ago and somebody added something to it since then?
Andrew: That's not my problem. Nix just handles all of that.
Andrew: And it's been really amazing. I mean, there's a bit of knowledge,
Andrew: Nix-specific knowledge, flakes in the language and all these different things
Andrew: can require a little bit of.
Andrew: A little bit of playing with to really understand what's happening,
Andrew: but it's been really powerful and it's helped.
Andrew: We have a concept here, which I know some other companies have as well.
Andrew: We call them guilds of like cross team interest in a topic. So like there is a Rust guild, right?
Andrew: And we have all the developers on different teams. Like my team has,
Andrew: let's say four Rust developers and then some other developers.
Andrew: But then the autofill team has a couple of Rust developers and this team and so forth.
Andrew: And we don't maybe collaborate all that directly, but now in the Rust guild, we can just talk Rust.
Andrew: We don't worry about, you know, interactions with other things.
Andrew: We just talk about how do we want to write Rust?
Andrew: Here's an interesting article about Rust, so forth. And we have one of those for Nix as well.
Andrew: So, you know, the build and release people are in there and some of the people
Andrew: from my team who are all just kind of Nix fans are in there and so forth.
Andrew: And that helps us to make sure that, you know, if someone who maybe owns the
Andrew: tool chain is working on something, you know, like, hey, I couldn't figure out how to do this.
Andrew: Well, now there's a bunch of other people who just work on this in their spare
Andrew: time or, you know, have their home set up in Nix and so forth.
Andrew: Like, oh, I've done that here. And we can, we can sort of augment the knowledge
Andrew: on various teams and, and really help to skill people up.
Andrew: Like the Rust Guild, we run a Rust study group that has, I think we had like
Andrew: 160 people in it or something like that the last time I looked.
Andrew: And so it's people, you know, going through the book together and learning from
Andrew: some of the experts that work here in Rust and then working on projects together
Andrew: and all these different things and it helps us to really kind of like raise
Andrew: the floor of skills in all these different domains, including them.
Matthias: About the external dependencies, so any of the popular crates,
Matthias: are there things that you use from the ecosystem that you kind of wanted to mention?
Matthias: Could be anything, really.
Andrew: Yeah, I mean, so obviously, like I said, we use Tokio a lot,
Andrew: and we use the whole Tokio ecosystem, right?
Andrew: We're using Tokio itself for channels and all these different things.
Andrew: We use axum for various services and so forth.
Andrew: Tower, you know, the tower middleware is really, really helpful because it allows us to share things.
Andrew: You know, if we have one, again, feature that's totally independent from some
Andrew: other feature, but if they create a middleware that handles,
Andrew: oh, this handles authentication.
Andrew: Great. Well, now this one can use it too because we auth the same way.
Andrew: And so it allows a lot of sharing there.
Andrew: Tracing is really critical. Obviously, again, having so many features and so
Andrew: many layers using tracing gives us a lot of visibility into,
Andrew: you know, did I make something that just slowed everything down?
Andrew: Or where is my slowdown occurring, you know, and so forth. So that's great.
Andrew: Serde is obviously pretty critical as well, right? We have to do a lot of serializing
Andrew: and deserializing, you know, coming in from the server and so forth.
Andrew: So that's really helpful.
Andrew: And then some of our internal crates, right? Like I mentioned,
Andrew: like TypeShare, passkey-rs, and so forth.
Andrew: The only other ones that I know of that come up a lot are like RustQLite is
Andrew: important for SQLite interaction. That makes things a lot easier.
Andrew: And then testing things, right? Like MockAll is good.
Andrew: Cool Asserts we use. We also have some developers here who have public crates
Andrew: that we use that are nice.
Andrew: Like one of our developers, I think, works on Neon, which handles like Node.js
Andrew: bindings for the Electron apps.
Andrew: Nathan West, who's here, made nom-supreme, which is a NOM helper library,
Andrew: which is really cool. I really like NOM Supreme.
Andrew: And so forth. And so there's, there's a lot of really cool things.
Andrew: One of my teammates, Ivan, also has a project called Crane, which helps with Nick's projects.
Andrew: And I don't think we actually are using Crane here. But it's funny,
Andrew: because we'll come across open source projects, we'll talk to other teams.
Andrew: We were talking to the team that built the Zed editor, when we were at RustConf.
Andrew: And, you know, so we, we opened up some communication with them,
Andrew: trying to, you know, help kind of beta test some stuff that they were working on.
Andrew: And they were using Crane to build Zed. And so I was like, oh,
Andrew: that's funny. That's something that one of the people here made.
Andrew: So it's kind of neat seeing the ecosystem come together.
Matthias: Yeah, totally. That's always surprising.
Matthias: The code sharing that happens between different companies using Rust,
Matthias: that's always very surprising.
Matthias: And by the way, if someone wants to listen to the episode with Zed,
Matthias: it's been a couple episodes before.
Matthias: These guys are doing amazing work. we can't
Matthias: talk about all of the crates but one thing specifically
Matthias: that i wanted to ask you about was the tracing ecosystem of crates specifically
Matthias: if you use tracing in combination with the log crate or if you use tracing for
Matthias: tracing plus logging which is a very popular usage of that crate.
Andrew: Yeah so that is actually something i think that is coming about now i mean we've
Andrew: used tracing primarily for tracing, right, for sort of internal performance telemetry.
Andrew: A lot, and we have a lot going on there. But one of the other things now is
Andrew: especially, you know, in our cloud infrastructure, trying to make sure that,
Andrew: you know, we have crates in, or crates, we have services in various languages,
Andrew: especially now, you know, that all these other companies are coming into the fray.
Andrew: There's, you know, Ruby and, you know, .NET, I think, and all these other things.
Andrew: And so we want to make sure that our observability is not hindered by that.
Andrew: So we have some, you know, intermediate things that are handling,
Andrew: let's say logging output and making sure that it is normalized so that we can
Andrew: handle observing, you know, cross service, you know, the whole ecosystem, right?
Andrew: And so that's something that I think we're actually doing now is using tracing,
Andrew: tracing subscriber to make sure that our log output is, you know,
Andrew: meets the standard essentially so that any given Rust app can output their logs
Andrew: in the right format and make sure that they're consumable by these other services.
Matthias: Are there any things that you specifically like about tracing or you dislike
Matthias: about tracing right now?
Andrew: I don't know. I didn't work on the implementation of it too much,
Andrew: so I would say I don't have a lot that I dislike about it.
Andrew: It's really nice to be able to just jump on, to have some function and go,
Andrew: I'll just add the tracing macro to this, right?
Andrew: And now I'm getting data about this, you know, and I can instrument these things
Andrew: very quickly and easily and it adds in there and I'm seeing,
Andrew: you know, everything that I want.
Andrew: It's been a huge help to be able to go. You know, one of the things we were looking at was like,
Andrew: when we do these invocations, right, they are sort of inherently async,
Andrew: but you may have invocations that are performance heavy, right?
Andrew: Something that has to like, you know, let's say iterate through a giant list,
Andrew: and it just has to happen that way.
Andrew: So we have a blocking thread pool, you know, Tokio gives you,
Andrew: you can do a blocking task right on this blocking thread pool.
Andrew: But you also have async tasks and async tasks tend to be more,
Andrew: you know, cooperative in a lot of cases.
Andrew: And so we got down to the point where it was like, you know,
Andrew: we have hundreds of these invocations.
Andrew: Well, how do we know which ones should be blocking and should be async,
Andrew: right? If it's got an await, it needs to be async.
Andrew: But out of the hundred that maybe don't have an await, which ones are better
Andrew: served on which thread pool?
Andrew: And so we can use tracing and just go through and just literally flip it from
Andrew: blocking to async and, you know, do the little minor, you know,
Andrew: syntax conversions we need to do and just do AB,
Andrew: you know, one to the other and go, okay, here was this one here was this one
Andrew: oh it's faster that let's
Andrew: put it over there you know and it makes it really nice to be able to just kind
Andrew: of jump through all these things i did that i think it was i guess two years
Andrew: ago now and just ran through every invocation as it was and then flip them all
Andrew: to the other one and see which one's faster and had an mr with just everything
Andrew: here is what came up faster with tracing and here's all my results so tracing
Andrew: is i mean it's incredible that's.
Matthias: A really nice use case for tracing the other thing is Tokio console which is
Matthias: similar i don't know if you've heard about it, but it shows you
Matthias: the state of Tokio and the futures that can execute it and how long they take,
Matthias: Did you hear about that?
Andrew: Yeah, Tokio console is pretty awesome as well. We have, I think Ivan actually,
Andrew: the one that I mentioned who works on Crane, I think he also is one of the maintainers of Tokio.
Andrew: But Ivan has helped me learn a lot. He said, I had no Rust background at all
Andrew: when I got here. And Ivan is a Rust expert.
Andrew: We have a few others as well. Like I mentioned, Nathan is absolutely incredible
Andrew: just the things that they know when it comes to Rust.
Andrew: But the last three years, I feel like I've learned more from Ivan in the last
Andrew: three years than I learned in the prior seven or eight combined, everything that I did.
Andrew: But yeah, showing me, When you get through and you've got the client app running
Andrew: and it makes this request and we're dispatching this one task and then that
Andrew: may kick off other tasks and the database access is all asynchronous and so
Andrew: there's a queue of these requests and then there's events that get triggered
Andrew: off of that which may trigger other async tasks.
Andrew: And so all this stuff is going on and, you know, looking at all this stuff in
Andrew: the console is a huge help to get visibility, like what's really happening, right?
Andrew: You know, I see a log message just starting sync and then I see a log message just sync completed.
Andrew: But inside of that, there was so many tasks that spun up and some were slow
Andrew: and some were fast and so forth. And so, yeah, like you said,
Andrew: Tokio console is really great for understanding the futures that happened.
Andrew: You know, we use tracing, I think, more for like function calls, right?
Andrew: Like, in our case, at least, like, we called this function, it is instrumented,
Andrew: I see this function took 10 seconds.
Andrew: And inside of that, this one took, well, I guess 10 would be a long time.
Andrew: Let's say it took 100 milliseconds.
Andrew: And inside of that, here are three others that took 10 milliseconds each, right?
Andrew: And I get that visibility. But seeing the futures that kicked off and how long
Andrew: those took does really give you another level of visibility into what's really happening under them.
Matthias: If I understand correctly, you don't trace every future, but you kind of group
Matthias: them into logical units, things that make sense to be executed together,
Matthias: just to get some insights.
Andrew: Yeah, and that's another thing. I mean, that's really great about the way that
Andrew: the Tokio Tracing crate is set up is you have like the spans, right?
Andrew: And you can create spans and enter spans and all these different things.
Andrew: And so we use those pretty heavily. there are probably scenarios or code paths
Andrew: where individual futures are traced.
Andrew: But in general, you know, a lot of times it'll just be when we're going to some
Andrew: block of code, what, you know, how long is this taking, right?
Andrew: What do we, what is happening here? And so it's not always every single future.
Andrew: It is, it is specifically what we're trying to instrument, right?
Andrew: And so the, having the console as well is very helpful, but those spans make
Andrew: it nice to have logical grouping, like you said, of different concepts.
Andrew: And so I can see within the span, how long is the whole span taking and then
Andrew: how long are individual blocks of that span taking, whether it's multiple futures
Andrew: or one, I may or may not care, you know, how long the individual futures take, right?
Andrew: Let's say I have some future that does some unit of work and that unit of work has to happen.
Andrew: And there's no other way to do it. Maybe I don't necessarily care how long those
Andrew: futures take, but I do care how long this block takes.
Andrew: And that may help me, you know, determine, oh
Andrew: this is a vec and it doesn't need to be a vec maybe now we can
Andrew: order it and it you know with the data sets we have it makes more
Andrew: sense to spend the upfront time hashing this or making it a btree set or something
Andrew: like that and ordering it all so that you know most often what we're acting
Andrew: on is we're inserting somewhere okay this gives us that that speed up on insert
Andrew: and it's often worth you know the time to sort it or something so yeah the spans are really nice.
Matthias: Well, some people might listen to this and say, I will never get to this level of proficiency.
Matthias: And maybe they are curious what it feels like on a day to day to work together
Matthias: with maybe 100 other Rust developers on the same code base.
Matthias: What are some of the patterns that you see evolve over time?
Matthias: What is the complexity like?
Matthias: Is it a thing that people can teach themselves?
Matthias: Are there any guidelines? What are some things that you would say evolved since
Matthias: you started working with Rust that you would say you should definitely follow
Matthias: this and these are the best practices that we see at 1Password?
Andrew: Yeah, I mean, so I would say first and foremost, when I got here,
Andrew: when they interviewed me, they tried to show me some Rust code because I had
Andrew: never done it at that point.
Andrew: They showed me some russ code and it was essentially a like
Andrew: a number aware sort so making sure that you
Andrew: know one password you know shows up in the right
Andrew: order in this list or something like that right and i could not get through
Andrew: it at the time like i i wasn't i was generally following like it's trying to
Andrew: do this but it was using these chunks and i didn't really know what that was
Andrew: and then it did a little bit of memory stuff and i had never touched memory
Andrew: coming from Column TypeScript. So I had no clue what was going on.
Andrew: And luckily they hired me anyway. I don't know why, but they did.
Andrew: And so, you know, in the last three years is everything that I have learned
Andrew: when it comes to Rust, right? That's the only time I've ever seen it.
Andrew: And so it's definitely something you can pick up. I have not done it alone.
Andrew: Right. I have had multiple teachers who are absolutely incredible when it comes to Rust.
Andrew: And like I mentioned, we had that Rust study group, which is something that
Andrew: I took part in in the first iteration when my co-workers Tanner had set it up.
Andrew: And then it had been so massively helpful to me that I ended up taking it over
Andrew: and running the second iteration of it. And now, you know, we've grown this
Andrew: into this whole Rust guild.
Andrew: So all that to say, anytime you can learn from other people,
Andrew: it will be massively helpful, I think, especially in Rust, because you may be
Andrew: coming from languages where the concepts just don't exist, right?
Andrew: And you're trying to learn all these brand new things.
Andrew: So that's a huge help. You can learn it on your own.
Andrew: There are a lot of good resources out there now. One of the things that's been
Andrew: really incredible in the Rust community is there's a lot of the books that are
Andrew: amazing books in Rust that are just free, right?
Andrew: Like the Rust Atomics and Locks book from Mara is free.
Andrew: You can get it on the internet, you know, also buy it if you can.
Andrew: But there's so many of these things. The Rust book itself is really great.
Andrew: One of the hardest things to get from, you know, zero to productive,
Andrew: I found, actually, I spoke at RustConf a bit about this this past year.
Andrew: What we found teaching people is the issue is not learning the Rust syntax.
Andrew: It tends to be that Rust, good Rust, requires a bit of engineering knowledge.
Andrew: And that's not maybe engineering knowledge that the book is covered,
Andrew: right? Right. And you and I talked about this a little bit, right?
Andrew: Like, how do I know when I make a new crate?
Andrew: Well, the Rust book doesn't, I mean, it's not going to tell you that that's
Andrew: not really a Rust problem, right? That's a, your code base problem.
Andrew: It's an engineering problem.
Andrew: It tends to expose your, you know, lack of knowledge in other areas,
Andrew: which again, I also have no computer science background.
Andrew: I was a construction worker before and I worked at, you know,
Andrew: a drugstore before I became a developer. So I had no idea about binary trees
Andrew: and sorting. I still really don't to be honest but.
Andrew: Rust has a way of bringing those things out in you, which is hard at first,
Andrew: but it also helps you to shore them up because you see where your gaps are.
Andrew: And there is so much information out there that you can learn from,
Andrew: but it does take time and not everybody has time.
Andrew: So that can be a bit difficult. But anyway, that's a long way to say you can
Andrew: learn on your own, but learn from others wherever you can.
Andrew: Look at code, try and understand it. And then getting into patterns,
Andrew: a lot of it has been thinking about two things, I think, heavily.
Andrew: What, like, why are you abstracting something? Because abstractions are really big in Rust, right?
Andrew: You know, like I said, we use RusQlite, but we wrap every transaction so that
Andrew: we add some counter that we want to follow in debug mode.
Andrew: And then we have to wrap, you know, maybe the call to save to increment that counter, right?
Andrew: To insert a row to increment that counter. So you end up with abstractions on
Andrew: abstractions on abstractions.
Andrew: And if you don't know what you're looking at, those are even harder to follow.
Andrew: So making sure that your abstractions make sense. And if they have to be complicated
Andrew: for any reason, document them really well.
Andrew: That's, I mean, critical. Even if you're the one that's going to come back and
Andrew: read it, make sure you know why you made this abstraction.
Andrew: That's really important. And then the second thing I think is trying to make
Andrew: sure that you are not driving performance at the cost of simplicity, right?
Andrew: You know, if I look at this and it's like, well, technically the fastest way
Andrew: is, you know, creating this one shot channel that watches. And once that gets
Andrew: something, it kicks off a receiving loop on a receiver. And then it does this look up.
Andrew: If you can do that with a Vec.Find or whatever, just do it.
Andrew: Like, you know, unless you really, really need to eke out that performance,
Andrew: keep things simple where you can, you know, you'll get to places like 1Password
Andrew: where it's like, yeah, we have to do all this zeroizing allocation and we can't log dynamic strings.
Andrew: There's a whole loggable trait
Andrew: in this whole process it has to go through to make it, you know, work.
Andrew: But that's not, you know, you don't need to understand the internals of that every time, right?
Andrew: You know, just understand what you have to, keep things simple wherever you can.
Andrew: Even here, you know, we try to make sure that we're holding each other accountable
Andrew: for like, it doesn't need to be that hard.
Andrew: Keep it simple. Yes, it's technically faster this other way.
Andrew: We don't have realistic data sets that need that speed up.
Andrew: You know, when we get there, we'll do that. Yes, in the future,
Andrew: we might have those and we might see a slowdown. And guess what,
Andrew: when that happens, we'll fix it.
Andrew: And so forth. So I think a lot of it just comes back to Rust is complex.
Andrew: Keep it simple wherever you can. Learn everything that you come across,
Andrew: but, you know, take it one thing at a time. You will eventually get there.
Andrew: And that pattern is true all the way up into 1Password where things are complex.
Andrew: Keep it simple where you can and then learn things one at a time when they can't be simple.
Matthias: I can totally get behind that. Makes a lot of sense. It matches with what I see out there.
Andrew: Yeah
Matthias: We're
Matthias: getting close to the end and traditionally the final question is what's your
Matthias: statement to the rust community.
Andrew: Uh yeah so i mean i think one of the you know there there is some amount of
Andrew: like thank you to the rust community for keeping things open right like i said
Andrew: there's so many books out there and the rust book and the brown version of the
Andrew: rust book is is amazing with the quizzes and things.
Andrew: That's all great. I think we also need to continue to or increase how often
Andrew: we relate to people, right?
Andrew: Meet people where they're coming from, understanding, you know,
Andrew: what they do and don't know.
Andrew: Obviously, with someone coming from my background, that was really critical
Andrew: that people here did that, you know, and there's a lot of, you know,
Andrew: acronyms or computer science knowledge.
Andrew: And there's so many people in Rust that are so smart, it can be difficult,
Andrew: you know, sometimes coming in and going, you know, they're like,
Andrew: well, you should make a beat reset.
Andrew: Why? What? I don't understand why that matters, you know, and so forth.
Andrew: And, and just a lot of little things, right.
Andrew: Understanding how did you learn what you learned, you know? And it's like,
Andrew: oh, well, in my OS class, we did this.
Andrew: It's like, oh, I didn't have, you know, so there's, there's a lot of that,
Andrew: that I think we can continue to improve on just finding paths to not just go
Andrew: from zero to rust, which I think we've started on, or we have a really good
Andrew: path to with the Rust book or to go from Haskell to Rust or C++ to Rust or whatever, right?
Andrew: I think there is a missing middle gap of how do you go from TypeScript to Rust
Andrew: or this or that where it's like, I understand variables. I understand all that.
Andrew: So the whole beginning of the Rust book feels boring and I'm getting lost.
Andrew: And then we go straight into memory or, you know, async and really how async
Andrew: works. And there are these missing pieces sometimes.
Andrew: And so I think finding practical applications of Rust is really critical.
Andrew: One of the big things I push for a lot within 1Password in the Rust study group
Andrew: is how can we build one cohesive thing?
Andrew: So it's not learning all the Rust concepts one by one, and it's not getting
Andrew: straight into the 1Password code base, which is terrifying.
Andrew: It is how do I learn these concepts and
Andrew: apply them to one thing that I have now built and now I
Andrew: understand why you know
Andrew: a Tokio Tokio mutex and
Andrew: a standard mutex they're different and why it matters that they're different
Andrew: and you might need both in different cases right I've seen it in use and now
Andrew: I can understand you know thread boundaries or whatever it might be or weight
Andrew: boundaries I should say and stuff starts to to make sense to you because you've
Andrew: had a real use case for it,
Andrew: not just what it was when I learned it, right?
Andrew: There's things like rustlings, which is incredible for exercise-based learning.
Andrew: But at the end, I didn't see exercise six mesh with exercise 10.
Andrew: And now I understand that those are two maybe related things.
Andrew: And so a lot of people say, right, build stuff. That is the greatest way.
Andrew: But that's not realistic for everyone. I was working before I became a developer.
Andrew: I was working 50, 60 hours a week and then you know, coming home and trying
Andrew: to keep, you know, my apartment clean and all these things.
Andrew: And I did not have that much time left to just build things.
Andrew: It took me a year or more just to try and figure out, you know.
Andrew: React or whatever it was at the time.
Andrew: And if it were Rust, I was trying to learn, I have no idea how long it would have taken, you know.
Andrew: And so I think just finding ways to bridge those gaps and make cohesive learnings
Andrew: that make concepts actually make sense holistically.
Andrew: I think that's a really, really huge area that will help a lot of people close
Andrew: the gap of Rust, which will help Rust adoption.
Andrew: You know, I've seen that even here, where it's like, we may have something we
Andrew: build and go because ramping that team up on Rust is just not feasible in the timeframe we have.
Andrew: Well, if there's better resources to get from Go to Rust, that becomes a non-issue,
Andrew: you know, and I think we'll see Rust grow the more we are open to bringing people
Andrew: in from more intermediate spaces into Rust.
Matthias: What an amazing and insightful answer. I can really get behind it.
Matthias: I can fully relate to that.
Matthias: Andrew, thanks so much for being on the podcast today.
Andrew: Yeah, thank you very much for having me. I really, really enjoyed it.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen. For show notes,
Matthias: transcripts, and to learn more about how we can help your company make the most
Matthias: of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.