Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias Endler from corrode, and today we are talking to Folkert de
Matthias: Vries from Tweede Golf about how to modernize critical time-sinking infrastructure with Rust.
Matthias: Folkert, welcome to the show.
Folkert: Thanks.
Matthias: Can you just briefly talk about yourself and about your role at Tweede Golf?
Folkert: Sure. So I've been using Rust for about four years. In my spare time,
Folkert: I mostly work on the Roc programming language, where I'm one of the main contributors.
Folkert: And it's been a really cool place to try out a lot of low-level stuff.
Folkert: And then at work at Tweede Golf, I mostly work on systems programming projects,
Folkert: decently low-level, and in recent years, mostly on our big open-source projects.
Matthias: When you mentioned Roc, I know it's a Rust podcast, but maybe people might
Matthias: still be interested in what it is.
Matthias: Can you maybe say a few words about that and where Roc comes from?
Folkert: Yeah, it's a pure functional language. So it is in that tradition.
Folkert: It's mostly, well, it's sort of a combination of like Haskell and Elm are sort
Folkert: of the two main languages.
Folkert: And really what makes Roc different is that it compiles to native code.
Folkert: And so occasionally we also just compile straight to assembly.
Folkert: That's a fun skill to have.
Folkert: And we want to have a very high performance ceiling for this language.
Folkert: So functional programming is typically associated with, it might look kind of
Folkert: fancy, but it's kind of slow, but actually with recent sort of research,
Folkert: we're able to create, you know, like to eventually output very fast code.
Matthias: You mentioned Haskell, you mentioned Elm I wonder where Rust fits in there like
Matthias: you still seem to be in the in the Rust space you still seem to be working with
Matthias: Rust how does Rust tie into that picture.
Folkert: This is interesting so like when we when we wanted to make the raw compiler
Folkert: uh sort of the obvious choice would have been Haskell given that we were functional
Folkert: programmers we quite liked Haskell,
Folkert: but we had seen some existing Haskell projects and we just knew that,
Folkert: In terms of performance, it wasn't the best that it could really be.
Folkert: So we knew that in theory with Rust or with like a low-level language,
Folkert: we could do like, you know, SIMD and various other kinds of optimizations.
Folkert: We had no idea how to do any of them, right?
Folkert: Like we were functional programmers and had heard about this sort of optimization
Folkert: stuff, you know, in university or other places, but we hadn't actually experienced that.
Folkert: And so in practice, C++ would have been a non-starter for us.
Folkert: But with Rust, we felt right at home, given its type system that was very familiar.
Folkert: And then, you know, the rest of the language is sort of imperative.
Folkert: But, you know, you are exposed to that anyway.
Folkert: We know how to write a for loop, right?
Folkert: So that is sort of how we got started with Rust. And, you know,
Folkert: sometimes we look back at some of the old code, and it's quite bad in terms
Folkert: of, you know, idiomatic Rust.
Folkert: But over time, we got a lot better and we actually sort of descended down to
Folkert: the lowest level of the computer where we're now reasoning about like cache
Folkert: locality and using SIMD and minimizing the number of syscalls, that sort of stuff.
Matthias: When you explained that, I was wondering if you used LLVM as a backend.
Matthias: So for the uninitiated, LLVM is a compiler backend infrastructure that,
Matthias: for example, powers Rust as well. Well, is that the same for Roc-lang,
Matthias: or do you have your own compiler backend?
Matthias: I know for the people that are listening and are not too technical,
Matthias: apologies for the detour, but I'm just genuinely curious about it.
Folkert: Yeah, so we have different modes, and this is in fact something that Rust will
Folkert: also soon have that many languages are now looking at.
Folkert: So LLVM is really great for getting the best final output. put.
Folkert: So if you want a sort of super optimized program, LLVM is basically the state of the art.
Folkert: The problem is that most of the time when you compile your code,
Folkert: you don't really need that level of optimization.
Folkert: You really just want to check whether the change you made sort of fixes the
Folkert: test or, you know, like the program now runs correctly.
Folkert: And then LLVM, the sort of trade-off that it makes is sort of a bad one for
Folkert: you because it spends a lot of time optimizing a program.
Folkert: And even if you ask it to sort of not do any work, it's still very slow.
Folkert: And, you know, you will will notice this all the time with Rust,
Folkert: where compile times are very slow.
Folkert: And often, this is really not because of the borrow checker or anything.
Folkert: That is compiler code written by Rust developers, and they can make that fast.
Folkert: The problem is usually LLVM, or a significant part of it is LLVM.
Folkert: So a lot of languages are looking at alternative backends for development.
Folkert: We have custom assembly backends for x86, AARCH, and WebAssembly.
Folkert: They are pretty much on par now with the LLVM, but they are like blazingly fast.
Folkert: So Roc can actually compile and link and then run Hello World faster than Python
Folkert: can print it, which is about like, yeah.
Folkert: So like that is the sort of speed that your computer can actually achieve if you don't use LLVM.
Folkert: We can beat LLVM, you know, Like thousands of hours of labor have gone into
Folkert: sort of having that produce very fast programs.
Folkert: But it just makes the tradeoff of like, we're going to set everything up to optimize.
Folkert: And then if you decide not to do that, you've wasted a lot of time and resources, basically.
Matthias: Isn't it also true that if you are at a stage where you can compile a reasonably
Matthias: sophisticated language, maybe that is similar to Haskell or Elm to some extent,
Matthias: I'm talking about Roc in this case, if you have such a backend,
Matthias: couldn't you also use it for Rust, which is also in this ML sort of space of languages,
Matthias: similar to, it's pretty similar to Haskell.
Matthias: Could you use the backend for Rust at some point?
Folkert: It's, that would require quite a bit of additional work, I think.
Folkert: Like it's really specialized towards the sort of operations that Roc allows for.
Folkert: Rust has a lot more primitives, ultimately, that you would, like in particular,
Folkert: things around pointers, right? That you would need to somehow translate.
Folkert: It's totally feasible. It's like an extension that you could make.
Folkert: But Rust is currently looking at the Cranelift backend, which is really cool.
Folkert: There's a lot of sort of cool research that goes into how that one works.
Folkert: And then also the GCC backend, which I believe is mostly about supporting more
Folkert: embedded targets, but potentially also has performance improvements over the LLVM backend.
Matthias: Rust does not have a garbage collector. Does Roc have one?
Matthias: And if so, how does it impact the language design or the compiler design? design so.
Folkert: In Roc we use reference counting
Folkert: as our garbage collection strategy so we
Folkert: don't have a garbage collector in a classic sense where you
Folkert: run into like garbage collecting pauses and things like that effectively like
Folkert: everything is wrapped in an arc right and in rust terms and so cloning is cheap
Folkert: but you do need to like that that reference count needs to be updated all the
Folkert: time so there is a certain overhead associated with that also all of your allocations
Folkert: are slightly slightly bigger.
Folkert: You need space for that reference count to live.
Folkert: But generally speaking, that is a very,
Folkert: like sort of good trade-off to make. We get very reliable performance.
Folkert: You need something like this anyway.
Folkert: And then for a functional language in particular, we also get mutation.
Folkert: So normally in a functional language, you have this like referential transparency
Folkert: idea where you can actually update or mutably update any values.
Folkert: So that means a lot of cloning unless you're smart, but we can actually allow for in-place updates.
Folkert: Basically when the reference count is one you get a immutable reference basically
Folkert: because there is just a single owner of that value and you can you can make updates.
Matthias: Feels like we went a bit ahead of ourselves sure we kind of jumped right in
Matthias: and I kind of would like to tie it back into the topic of this podcast which
Matthias: is about NTP and other protocols which are useful for various things like timing.
Matthias: But I guess before we jump into that, maybe we can take a quick step back and
Matthias: say the overarching theme of what we talked about so far is,
Matthias: in my opinion, performance and understanding how computers work.
Matthias: Is that also what motivates you to work on these problems?
Folkert: Yeah, it's really, you know, it's like I now end up being able to work with
Folkert: a lot of things that I never really thought I would have worked on in university, for instance.
Folkert: So I was this very sort of Haskell-y programming language theory sort of person.
Folkert: And then Rust very naturally allowed me to sort of come into contact with a
Folkert: lot of these low-level concepts, right?
Folkert: Like, you can use Rust in a way that is quite similar to Python in a way.
Folkert: In practice, people will quibble with this.
Folkert: But I think it's sort of true. Like, you don't need to understand a lot about
Folkert: the machine to still write Rust code.
Folkert: And then in practice, it is actually quite performant.
Folkert: But very slowly, right, you click through the documentation,
Folkert: you read some stuff, you click on go to source a couple of times,
Folkert: and you will end up at these places where a lot of low-level details actually start to matter.
Folkert: And I think this is really nice about Rust that like go to source generally
Folkert: works even into the standard library. It's like occasionally you hit some macro
Folkert: and that is kind of tricky, but generally you will find something that makes
Folkert: sense and you can learn about it.
Folkert: This is very different in say Haskell or Python or Java or whatever,
Folkert: where the primitives are implemented in a different language and you have no idea how they work.
Matthias: I kind of share the same background. Maybe I wasn't as deeply involved in Haskell
Matthias: as you, but back in university, I did some Haskell.
Matthias: And for some mathematics departments, I wrote some algorithms and implemented
Matthias: some geometric functions.
Matthias: Functions and what i felt was when i
Matthias: was introduced to rust at some point i realized
Matthias: that it's real world Haskell or it's pragmatic Haskell to some extent and it
Matthias: felt like i could actually use all of the things that i learned from Haskell
Matthias: type theory and algebraic data
Matthias: types and so on in a real world project did it feel the same to you yeah.
Folkert: It's well like i think if you come at it with that background where the type
Folkert: system isn't actually that intimidating to you.
Folkert: And there's just some like syntax to learn, you know, like what is a trait? What is an impl block?
Folkert: That sort of stuff, right? Like you can pick that up very quickly.
Folkert: And so I think with that background,
Folkert: Rust isn't nearly as intimidating as when you are sort of coming from say like
Folkert: a Java or a Python background where Rust's approach to types and then lifetimes
Folkert: is is quite different from what you're used to.
Folkert: Like to me, it made total sense. Like, okay, sure, we have a problem.
Folkert: We're going to use the type system to sort of make that problem go away. Great.
Folkert: Like I know exactly how that sort of general form of solution works.
Folkert: And so that was very comfortable. But then like, as you say,
Folkert: like actually it is much easier to get practical stuff done with Rust.
Folkert: I also just think that it helps that Rust is a much more modern language.
Folkert: Which Haskell really shows sort of signs of aging at this point.
Folkert: It is over 30 years old.
Folkert: Around that sort of age, it just shows. It shows in the standard library.
Folkert: It shows in an accumulation of features that we don't use anymore and have to
Folkert: caution people to not use.
Folkert: And so Rust has a sort of major advantage there in that it was able to learn
Folkert: really from Haskell, from ML, from C++, from all the other languages.
Matthias: Early on, you mentioned that Rust made
Matthias: it really accessible for newcomers to learn
Matthias: about the underlying implementation and
Matthias: you mentioned that you can click on
Matthias: view source in the documentation i do think
Matthias: that this is an amazing feature are there other features like that that are
Matthias: just maybe a little more hidden that people might not know about that maybe
Matthias: also enable you to learn how rust works under the hood or other things that
Matthias: make the language more accessible to newcomers i.
Folkert: Think With regards to how things work, I think it's also something that is sort
Folkert: of valued in the Rust community that this sort of low-level detail is sort of appreciated.
Folkert: Like in a lot of other languages, people will say, oh, don't worry about that.
Folkert: The runtime will take care of it. Perhaps even the compiler will take care of it.
Folkert: But of course, in Rust, we know if you actually build the compiler or are quite
Folkert: close to it, you will know exactly what limitations it has.
Folkert: And also, you will know how fun it is to actually put a bunch of bytes in the
Folkert: right order and have that constitute a program, right?
Folkert: The sort of like hacking around with and really playing with computers,
Folkert: really commanding the machine, I think is something that you can certainly find in the Rust community.
Folkert: Community you know a couple of other communities as well at this point like
Folkert: it's something you don't see all that often but that excitement about really
Folkert: commanding your hardware is that is really cool.
Matthias: Yes and there's this other notion of explicit versus implicit where a lot of
Matthias: things in rust are not hidden they are visible in plain sight for example how
Matthias: vectors are implemented or or what allocates and what doesn't allocate.
Folkert: Yeah, I was particularly... Well, I think a cool example of this is the mutex
Folkert: synchronization primitive, which I'm sure I've seen that in universities somewhere.
Folkert: Some professor must have mentioned it at some point, like this is a mutex.
Folkert: But I don't think we ever really got to play with it.
Folkert: And I certainly didn't understand how it actually works. Like,
Folkert: how does this mutex know whether anyone else is currently also looking at the same piece of data?
Folkert: How does it get notified that the lock is now available and it can be taken by the current thread?
Folkert: And in Rust, you can actually just go to source and it's right there.
Folkert: It's a bunch of more primitive values and you actually can understand how it works.
Folkert: Like most other languages don't have this. Like even in C or C++,
Folkert: if you go to the source code of like the libc or the standard library,
Folkert: it is incredibly arcane.
Folkert: It looks very different from sort of standard C source code,
Folkert: in my view anyway, as not a very experienced C programmer.
Folkert: Like that stuff just looks really weird.
Folkert: And I mean, in Rust, you can just find something, find an implementation that makes sense.
Matthias: That is very true. I can still remember that back in the day,
Matthias: we needed to use mutexes in C, for example.
Matthias: And the one thing that still pops up or still top of mind is deadlocks.
Matthias: And I do think that you could run into a lot of these situations with Rust as well.
Matthias: I mean, it doesn't save you from deadlocks because essentially it's a logic
Matthias: error, I would say. but at least if you use high level primitives it becomes
Matthias: much much harder to use them wrong.
Folkert: Yeah i i also think you
Folkert: like in rust it'll be really clear
Folkert: when you need a mutex right whereas i think in c it might be much harder to
Folkert: know like okay i'm using this value over here but is some other thread also
Folkert: touching this right now like you you sort of have no idea whereas in rust a
Folkert: the type checker will tell you, right? Like, oh, you're using this.
Folkert: Or, well, first of all, you have an immutable reference, so you can't mutate that thing ever.
Folkert: If you have a mutable reference, then you know it isn't shared between multiple threads.
Folkert: And if you do want something that is both shared and mutable right now,
Folkert: you need to wrap it in a mutex.
Folkert: Then, of course, there's still logic errors you can make.
Matthias: That's one of my pet peeves with C, because even if it looks extremely explicit
Matthias: about various things, it is actually quite implicit about what it does under the hood.
Matthias: So a lot of things, a lot of details are conventions, or you need to be aware
Matthias: that things return errors and so on.
Matthias: In Rust, that is very rarely the case.
Matthias: So it's the right kind of explicit, if you like. You don't need to deal with
Matthias: memory management, which technically is also possible in C,
Matthias: and you can be very explicit about it, but it allows you to make decisions which
Matthias: can impact the program's design.
Folkert: Yeah, for sure. Sure. I think it works out really well, the sort of approach
Folkert: to, well, yeah, not exactly preventing that blocks,
Folkert: but certainly making them more or like less likely to happen.
Matthias: It feels like, imagine we have a conversation like this and there's someone
Matthias: from the business side, maybe a CTO or a decision maker listening to us.
Matthias: They might think these are problems that
Matthias: only nerds care about or people that
Matthias: are highly technical at least how do you bridge the gap between technology and
Matthias: business then because at some point you need to be able to sell that and maybe
Matthias: explain why rust or a technical solution is superior or gives you a business advantage yeah.
Folkert: I mean like we would need to look more at like i guess like concrete cases
Folkert: but it's true right like i need to eat and so like money needs to come in at
Folkert: some point from someplace.
Folkert: So I think you can ultimately distill this into, you know, value for users.
Folkert: A lot of this low level stuff is really about user experience.
Folkert: If you want to think about it that way, right? Like performance is ultimately user experience.
Folkert: And I think we don't really appreciate that enough, but there's a lot of research
Folkert: from like Google, right?
Folkert: About like if a page load is this many milliseconds longer than this percentage
Folkert: of people will just like lose interest, right? This is very well established.
Folkert: In practice, a lot of software is really slow and only people sort of excited
Folkert: about this sort of low level stuff can fix that.
Folkert: And when they do, it's really it's like magic kind of.
Folkert: There's been a couple of recent very interesting also Rust projects,
Folkert: actually, like I think there is a Python linter right now that is like thousands,
Folkert: like a thousand X faster than some of the other tooling in the Python space.
Folkert: Like that is magic, right? For a Python programmer, like how did that happen?
Folkert: But it happened because a couple of systems people got very excited and felt
Folkert: like, hey, we can actually do much
Folkert: better at this task than the existing implementations. Let's go do it.
Folkert: And like, yeah. And then imagine the sort of like value that that actually has,
Folkert: the sort of multiplier of this one person going really deep down into that rabbit
Folkert: hole, coming up with a more efficient implementation. limitation.
Folkert: But then all users of that tooling, and of course, that tooling,
Folkert: like its usage is spreading very quickly, because 1000x improvement is ridiculous.
Folkert: You know, the sort of the amount of time saved, the amount of extra productivity
Folkert: gains is really something that is almost hard to even capture, right?
Folkert: So like, that is ultimately the power of this low level stuff.
Folkert: Like, yes, Yes, it is very nerdy. Yes, it is very, it seems to be very distanced
Folkert: from sort of the real world sometimes, but it has real outputs.
Matthias: But at the same time, if you approach me and you said, oh yeah,
Matthias: we will have a 10x or 100x improvement in performance.
Matthias: And I ask you, how long will it take? And you say, yeah, it will take five or 10 years to get there.
Matthias: I'm not sure if that convinces me right there because it's a big time investment
Matthias: and it might be even hard to quantify.
Folkert: That that's true so you
Folkert: really need to pick your battles there there is
Folkert: just certain certain areas certain problems where this sort of approach works
Folkert: well for some tasks like it it really doesn't matter and you know like in our
Folkert: spare time we might still They'll play around with optimizing the last couple
Folkert: of milliseconds or whatever,
Folkert: but it doesn't have a lot of business value.
Folkert: You need to be strategic about this, for sure.
Matthias: Tweede Golf
Matthias: Folkert, at Tweede Golf, you do a lot of open source work, I assume.
Matthias: What does open source work look
Matthias: ? How do you pick a project? How do you get started?
Matthias: like at
Matthias: Can you name a few examples that you worked on?
Matthias: How does that general process look like?
Folkert: Yeah. We're a very small company, really, in a very unknown city in the eastern
Folkert: part of the Netherlands. But
Folkert: we do a lot of, I think, very interesting and sort of high quality work.
Folkert: So historically, mostly in the web space, but then also now doing embedded.
Folkert: We do a bunch of training and then we work on a couple of these like big open
Folkert: source Rust projects right now.
Folkert: So like, you know, we've always liked open source and might have contributed
Folkert: occasionally to, you know, something or other.
Folkert: But the big projects that we run now are sort of big funded open source projects
Folkert: such that we can actually dedicate a good amount of like labor to them.
Folkert: And so the current main ones are projects around sudo, ptp, and ntp,
Folkert: where sudo is a sort of super user do command on Linux.
Folkert: Obviously very sort of security critical that that implementation is sound.
Folkert: And in that case, it also sort of needs to mirror the behavior of the existing implementation.
Folkert: And then P2P and N2P stand for the Precision and Network Time Protocols.
Folkert: And these are sort of important building blocks of the Internet.
Folkert: And so that is sort of the main reason that those projects originally got started.
Folkert: So sudoand NTP are funded originally by the Internet Security Research Group
Folkert: Prossimo
Folkert: through their Prossimo initiative. These are the people behind Let's Encrypt.
Folkert: These are people that are very
Folkert: passionate about the Internet and sort of preparing it for the future.
Folkert: And this
Folkert: initiative is basically their project to fund rewrites or implementations
Folkert: of foundational building blocks of the Internet in memory-safe languages,
Folkert: languages, which in practice is a euphemism for Rust.
Folkert: But for funding purposes, we need to say memory-safe language.
Folkert: And if we're generous, we could include C-sharp or Swift in that definition.
Folkert: But that's currently not something that we do.
Matthias: Right. So is Let's Encrypt a corporate entity?
Matthias: Is that the business that funds all those projects? Or does Let's Encrypt also
Matthias: just belong to another organization?
Matthias: I only vaguely remember that they started in the Mozilla realm somewhere,
Matthias: but I don't know. I haven't followed up what they do now.
Folkert: Yeah, so the ISRG and Let's Encrypt, they're all not-for-profit businesses.
Folkert: Businesses, they run on effectively, or at least the projects that we do are basically,
Folkert: the money is ultimately from the big tech companies and the ISRG facilitates
Folkert: sort of putting that money to good use.
Folkert: So they do a lot of, I don't know, the convincing to someone at,
Folkert: you know, a major tech company like,
Folkert: hey, you have this budget that is, you know, left over or hey,
Folkert: you have this problem with this old piece of software that is not really maintained anymore.
Folkert: That might be a bit of security risk at this point.
Folkert: Why don't you fund us to create an implementation in Rust, a modern one,
Folkert: one that we can sort sort of guarantee will be maintained for the foreseeable future.
Folkert: And so they really play this facilitating role with regards to our projects.
Folkert: I think Let's Encrypt is something they run themselves. I'm not exactly sure
Folkert: what the funding model for that is, but it is still a not-for-profit sort of
Folkert: initiative that they set up.
Matthias: Let's focus on NTP for a moment. What is NTP? What does it stand for? When was it introduced?
Matthias: Give us some background on that.
Folkert: Right. So NTP stands for the Network Time Protocol, and it is a protocol for
Folkert: synchronizing time over the network.
Folkert: And in practice, the network generally means the Internet. So the network is
Folkert: an untrusted network that might span the globe.
Folkert: So this really has been sort of with us since the start of the internet.
Folkert: So it's like sort of around 40 years old at this point.
Folkert: Very old. It's gone through a number of iterations. So the initial versions,
Folkert: you know, have some flaws with the benefit of hindsight.
Folkert: So we're currently on version 4 of the NTP specification.
Folkert: And, you know, it runs on most devices, you know, around you that are connected
Folkert: to the internet. they will occasionally synchronize their time to the internet
Folkert: just to sort of be up to date and display the right time.
Folkert: So for most consumer devices, that is all there is to it. Occasionally,
Folkert: they sort of reach a server and ask like, hey, what is your time right now?
Folkert: And they will just adopt that time basically.
Folkert: But NTP is important for the internet because it is sort of crucial for a lot
Folkert: of the infrastructure that keeps the internet together.
Folkert: So in particular, like in a a data center, you actually want all your devices,
Folkert: all your machines to be sort of synchronized to one another with regards to,
Folkert: you know, when did a backup happen?
Folkert: You really want that timestamp to be accurate, right? Like if some error happens,
Folkert: for instance, you want to be able to interleave the logs of two machines and
Folkert: have that constitute a sort of an accurate global timeline.
Folkert: If the machines are not correctly synchronized, you could have a message that
Folkert: is received before it's even and send, and then good luck debugging that,
Folkert: right? Like that is awful.
Folkert: So just for sort of sanity, synchronization is very useful and very important
Folkert: in a sort of a data center or cloud sort of context.
Folkert: And then for security, you have this thing called TLS, right?
Folkert: The sort of foundation of HTTPS.
Folkert: And it relies on being able to expire certificates.
Folkert: These certificates just expire at a certain point in time.
Folkert: And if you were able to manipulate the clock of a machine somehow,
Folkert: you could trick it into using an expired certificate, and that basically breaks
Folkert: all of the whole security model of TLS.
Folkert: So NTP is important both sort of for infrastructure reasons,
Folkert: but also for security reasons.
Folkert: And that means that our implementation also needs to be very secure.
Folkert: You need that that implementation to be good right otherwise you'll run into issues i.
Matthias: Tried to remember when was the last time that
Matthias: i manually set the clock on my computer and it's probably been a decade by now
Matthias: is it also powered by ntp under the hood so i'm on mac os i don't know if it
Matthias: supports all the platforms or if different platforms use different protocols it's.
Folkert: It's generally the same protocol call, but most sort of consumer devices will
Folkert: use a thing called simple NTP, so SNTP.
Folkert: And that just does a simple call of like, hey, server at Apple or Microsoft
Folkert: or on Linux, there's this thing called the NTP pool.
Folkert: And that is a big pool of sort of servers that provide time for you.
Folkert: And it will just adopt whatever gets sent back. So for these sort of more serious
Folkert: applications in like a data data center,
Folkert: we actually want to have multiple sources of time because one might get compromised
Folkert: or there might be a power outage or whatever.
Folkert: And then you don't want to just desynchronize. And we also sort of more sort
Folkert: of accurately steer the clock at that point.
Folkert: So generally, what you can do is sort of jump forward or backwards.
Folkert: That is what a laptop will do. If I have a couple on a shelf here,
Folkert: and like if you boot them after three months, then the clock will actually be quite far behind.
Folkert: And then basically, it will just jump forward to the current time.
Folkert: But if you want really good accuracy, what we actually do is sort of speed up
Folkert: and slow down the clock slightly to sort of get its frequency to match the real frequency.
Folkert: And this is tricky because the frequency actually depends on like the temperature
Folkert: in the room, for instance.
Folkert: So there isn't one true frequency. It always depends on the current circumstances of that machine. sheet.
Matthias: It is true that when you haven't used your
Matthias: computer in a while or your smartphone that the
Matthias: clock might be off at some point i realize now that it is actually a problem
Matthias: that i also ran into and then it takes a moment for it to be up to date again
Matthias: i wonder if the hardware vendors sometimes do that on purpose even and integrate
Matthias: cheap hardware because they think that ntp or or SNTP will solve the problem.
Matthias: Yeah. I mean, technically it runs on billions of devices, right?
Matthias: So it's kind of a big market.
Folkert: In theory, of course, your computer could contain an atomic clock and you would
Folkert: never have this issue, right?
Folkert: But in practice, we don't because those things are expensive,
Folkert: they're heavy, and they are sort of, if you drop them, they generally don't work anymore.
Folkert: So the clocks in our actual devices are generally fine, but they will just drift.
Folkert: They will drift quite seriously, actually, if you have equipment that is accurate enough to measure that.
Folkert: So at our office, we actually have a GPS antenna that is a good source of sort
Folkert: of true time, the real accurate time in the absence of an atomic clock.
Folkert: We need to throw that thing out onto the street because our office turns out
Folkert: to be a Faraday cage at the GPS frequencies.
Folkert: So there's just this lone antenna in the street to give us an accurate time signal.
Folkert: And then we can actually measure, like on a Raspberry Pi, for instance,
Folkert: how much it drifts versus the true signal that we get.
Folkert: And it's very measurable very quickly. But of course, as a human,
Folkert: you would only really notice once it exceeds perhaps a couple of seconds,
Folkert: but certainly a minute, right? And that's...
Folkert: That takes a while before the desynchronization gets that bad.
Matthias: It's a very funny intersection between hardware and software because you could
Matthias: probably make hardware more accurate in order to improve your maybe accuracy in terms of time.
Matthias: But at the same time, you can throw software at it and solve the problem at the software level.
Matthias: But what i'm wondering about is
Matthias: if you wanted to improve the situation
Matthias: and you wanted to improve time synchronization with software looking at what
Matthias: ntp is and the implementation itself would you say that there could be improvements
Matthias: with regards to the protocol or would you say the main improvements come from
Matthias: fixing the software flaws that are in the existing implementations.
Folkert: So there are certainly improvements that can be made. This is in part what PTP was designed for.
Folkert: So PTP stands for Precision Time Protocol. It is more precision.
Folkert: And so generally, it uses some additional hardware features to get more accurate timestamps.
Folkert: But also, a lot of the sort of problem with the accuracy on the internet is
Folkert: the structure of the internet.
Folkert: So, in particular, you don't exactly know how far that signal needs to travel
Folkert: between you and the server that has the time.
Folkert: And that transmission delay and sort of variance in the transmission delay,
Folkert: because on the internet, sort of the path from A to B can be different from
Folkert: the return path from B back to A.
Folkert: It can go through different hardware, basically, different wires.
Folkert: The sort of variance that that introduces and the general noise,
Folkert: your time signal basically, determines how accurate your synchronization is.
Folkert: Now, it turns out that one of my colleagues is really into making better algorithms for that problem.
Folkert: So we believe that we have the best synchronization algorithm out there today.
Folkert: And so there's probably still further improvements that can be made algorithmically.
Folkert: But fundamentally you're sort of fighting against the
Folkert: noise in the network and so what ptp allows you to do is say okay i have 20
Folkert: meters of fiberglass cable right here right between these two devices and you
Folkert: can program that in and then it will account for that and so long as that doesn't
Folkert: change you can sort of reduce the the noise in that way.
Matthias: Initially i thought oh why can't we just move from NTP to PTP,
Matthias: but it feels like that's not feasible because you need all of that information about the environment.
Matthias: You need to code that into the protocol to be more accurate.
Matthias: But it feels like what you touched on before, which is this better algorithm
Matthias: could be the way forward.
Matthias: And is that part of newer version of NTP or would you need a completely different
Matthias: implementation, completely different protocol to make that happen?
Folkert: Yeah, just to clarify, PTP assumes the network is trusted.
Folkert: And so it can take a couple of shortcuts because of that.
Folkert: NTP needs to sort of guard itself against potentially untrusted input becoming my wing.
Folkert: And so what it does is if you listen to multiple servers, it will pick the most
Folkert: reliable ones, the ones that agree with one another.
Folkert: So it needs to expend some extra effort basically to guard against malicious input.
Folkert: So what's sort of interesting about the NTP specification is,
Folkert: one, there is a specification.
Folkert: And in practice, this is mostly what our implementation is based on.
Folkert: It's not really based on the existing implementations, but we really started
Folkert: from from the ground up based on that specification.
Folkert: The specification does specify a synchronization algorithm, but it is sort of
Folkert: a public secret that that synchronization algorithm isn't very good.
Folkert: It may have been at the time, but certainly now we just know more about sort
Folkert: of how to deal with the noise in that signal that we can get more accurate synchronization.
Folkert: So the best implementations use a custom algorithm today And so technically,
Folkert: they're not spec compliant, but practically, they behave as if they are.
Folkert: They just give a more accurate time signal.
Folkert: We are looking at contributing towards NTPv5, so the next version of the standard.
Folkert: And in particular, something that we are trying to get in there is to not specify
Folkert: the synchronization algorithm.
Folkert: And to sort of leave that up to particular implementations, basically,
Folkert: such that innovation in that space can happen without sort of being technically
Folkert: not compliant with the spec.
Matthias: But these are all things that you solve on the protocol level,
Matthias: and you don't necessarily have to rewrite everything in Rust for that.
Matthias: Maybe, can you briefly touch on some of the flaws of the C implementation that
Matthias: made you think that Rust was a better case for this?
Folkert: Yeah, so like, ultimately, this project started because the ISRG said,
Folkert: like, okay, this is a fundamental building block of the internet,
Folkert: it is not memory safe, there have been some incidents in the past.
Folkert: And even though you know, it's like, if an attacker can know what time you have,
Folkert: that isn't so bad, but it's sort of a jumping board for further attacks, basically.
Folkert: So I think there are good reasons both to have modern implementations of NTP
Folkert: and then in particular for picking Rust for the implementation language.
Folkert: So we sort of touched on this before. When you're sort of looking at a 4NC code
Folkert: base, it is very hard to understand what happens.
Folkert: And for instance, around synchronization, and that is what this sort of software
Folkert: needs to do, it sends a bunch of messages over the network. So it is inherently
Folkert: sort of a threaded application.
Folkert: In C, that is very hard to understand. And so the existing implementations are
Folkert: quite old, cross-platform C codebases, and they have a hard time attracting
Folkert: new contributors for, I think, very obvious reasons.
Folkert: And so that also means that the existing pool of maintainers is very small and
Folkert: aging rapidly, basically.
Folkert: Basically so a lot of these folks have been
Folkert: around since the invention of the protocol itself and so
Folkert: they're absolutely close to if not past their
Folkert: retirement age and they might decide that they don't want to maintain this
Folkert: sort of software anymore and then the rest of the internet would have a problem
Folkert: so i think there's a lot to gain from a modern implementation where you just
Folkert: write in a language that has good tooling has good documentation and in general
Folkert: sort of makes that implementation more accessible to more people,
Folkert: basically, in terms of understanding what the code does.
Folkert: Then you think Rust is sort of an obvious choice at that point.
Folkert: We can guarantee that there are no memory safety problems.
Folkert: We get good performance. We use
Folkert: and Async to sort of handle all of that for us.
Folkert: So we don't need to do our own threading in the application at all.
Folkert: And of course rust has amazing tooling in terms of testing fuzzing fuzzing is
Folkert: very important because we are accepting untrusted input from the internet we
Folkert: want to just make sure that you know we don't run into panics or infinite loops or stuff like that,
Folkert: and also rust is nice to work with and in practice we see that we actually get
Folkert: a bunch of contributions to the code base because it is a rust code base so
Folkert: these are not contributions to to the algorithmic side of NTP,
Folkert: but more for instance, supporting the muscle lip C,
Folkert: or FreeBSD or something like that, much more around the edges where you just
Folkert: need a little bit of extra work to make our implementation run on those platforms.
Folkert: And this is work that anyone, any Rust programmer can do without intimate knowledge
Folkert: with the internals of NTP.
Matthias: I wonder how a setup of such a huge project looks like.
Matthias: I certainly see that there's a funding case for this, but then you still need
Matthias: to set up a project structure and have a timeline.
Matthias: And you mentioned testing and you also mentioned fuzzing. I wonder how you set
Matthias: that up, how you structure the project,
Matthias: how you go from the idea to the implementation to the rollout and so on.
Matthias: And maybe briefly for people that are not in the know, can you just quickly
Matthias: explain what fuzzing is as well?
Folkert: Yeah, let's talk about fuzzing. So fuzzing is the idea that you throw random,
Folkert: in quotes, random input at some program.
Folkert: And verify that it behaves correctly.
Folkert: And here correctly doesn't necessarily mean that it gives the right output because
Folkert: your input might basically be invalid input.
Folkert: But because we effectively listen for messages from the internet,
Folkert: a, well, some sort of error or a malicious attacker can send any sequence of bytes our way.
Folkert: And we just need to make sure that our program remains operational even if we
Folkert: get some weird sequence of bytes as our input.
Folkert: So we want to make sure that we don't crash, but we handle these errors gracefully,
Folkert: and also that we don't run into other sort of weird error conditions where we
Folkert: might enter an infinite loop or just use a lot of resources that could be used
Folkert: to do some sort of denial of service attack.
Folkert: We use fuzzing for a lot of our parsing logic, for instance,
Folkert: and then also in several other cases, just to verify the correctness of certain
Folkert: data structures, for instance.
Matthias: Do you have any hard guarantees around the implementation itself that it doesn't
Matthias: produce any errors? Is that written down somewhere?
Matthias: And in general, how is the testing infrastructure and the general project infrastructure
Matthias: for this sort of endeavor? ever.
Folkert: So in Rust, it's really hard to guarantee that you have no panics in your code.
Folkert: There are also certain cases where we do want to panic, and this really makes things tricky.
Folkert: There are some very hacky ways to guarantee that a Rust program doesn't panic.
Folkert: You need to do some linker stuff.
Folkert: But we do actually want to panic in certain cases where we recognize that the
Folkert: signal that we get is so messy that we can't make a good decision about how to synchronize.
Folkert: And then we basically decide, Like we're just going to stop right here instead
Folkert: of going in a wrong direction.
Folkert: We're assuming that sort of having the clock go on as it currently is,
Folkert: is better than us sort of moving at very radically or potentially in a sort
Folkert: of incorrect direction.
Folkert: So that makes it really tricky to guarantee that we have no panics.
Folkert: And this is why the fuzzing is so important to just sort of make sure that for
Folkert: any input that you give our program, it will not hit a panic.
Folkert: Right and in Rust a lot of these panics are sort of hidden in like some sort of slice index or,
Folkert: An unwrap somewhere is very easy to write sometimes, but we try to be very careful, basically.
Folkert: This is unfortunately sort of a... Well, the fuzzing really helps,
Folkert: but ultimately we also just need to be careful.
Folkert: We need to be sort of dedicated to not write unwraps and other panics that can
Folkert: happen in a real program code.
Matthias: If i was a critic i
Matthias: would wonder if i couldn't do the same thing in
Matthias: a different code base for example with the existing c code base i could also
Matthias: run the fuzzer and i could say this is just blind rewriting rust enthusiasm
Matthias: around this project what do you think about this how would you answer to such criticism yeah.
Folkert: So like it's it's important to emphasize here that like this this is not a straight-up
Folkert: port of an existing C program.
Folkert: There are some other projects where that does happen and where it actually still
Folkert: makes sense to do that rewrite in Rust.
Folkert: But we are really just a new implementation, a from-the-ground-up new implementation
Folkert: because for this sort of project,
Folkert: we believed and still believe that we could do better in terms of architecture
Folkert: architecture or create a more idiomatic Rust program by just starting from scratch
Folkert: based on the specification.
Folkert: I think there are still guarantees that you can give about a Rust program that
Folkert: you cannot reasonably give about a C program.
Folkert: Of course, a lot of the fuzzing tooling does help.
Folkert: Problems still occur, right? CVEs, like critical security vulnerabilities,
Folkert: still happen in C codebases because it's very easy to miss something, actually.
Folkert: In fact, we had one sort of mishap in our codebase where some part of the code just didn't get fuzzed.
Folkert: And of course, there was some out-of-bounds access in there.
Folkert: And it would panic if you send it an incorrect input stream.
Folkert: So you still need to actually be very careful even with that tooling.
Folkert: I think the Rust compiler just gives us so much baseline security that we have
Folkert: less to worry about because out-of-bounds access will at least panic, right?
Folkert: It won't silently continue to run.
Folkert: It will actually sort of fail loudly.
Folkert: But then also, I think Rust has all of these other benefits of,
Folkert: it is a modern language, it has good documentation, it has a good beginner experience,
Folkert: experience we can onboard new people onto the project easily so
Folkert: besides like all of the people sort of working on the implementation right now
Folkert: are sort of you know n20s early 30s so we have we have a bunch of decades that
Folkert: we can remain active on this project and hopefully we can sort of keep that going but.
Matthias: Shouldn't you get an error if you do something strange with with an input for
Matthias: For example, I imagine that, let's say you want to convert it to UTF-8 or so,
Matthias: you would assume to get an error message that you can handle.
Matthias: Or why would you get a panic in such a case? Is it because of the parsing logic?
Folkert: Yeah, it's generally parsing is sort of where we have untrusted input, basically. Yeah.
Folkert: And there, you sometimes just write a straight up index, say,
Folkert: OK, I want the 11th element out of this slice.
Folkert: And then if you give an invalid input to that function that does this slice
Folkert: access, then you can run into issues there.
Folkert: This is, of course, rare because you do, or at least we test.
Folkert: We have a very extensive test suite, which means that in practice,
Folkert: the size of our code base is roughly comparable to the C code base,
Folkert: but they generally seem to have way less in terms of testing of the actual individual bits of logic.
Folkert: So of course, they will actually run the implementation and sort of verify that
Folkert: it runs correctly in the wild.
Folkert: But we have much more atomic, basically, tests of just the parsing logic or
Folkert: of just the sort of spawning logic or synchronization logic, et cetera.
Matthias: Were you able to use any existing crates for the parsing logic,
Matthias: or did you have to write that yourself?
Folkert: So that is interesting, and this is true for a bunch of these projects,
Folkert: because they are... Well, basically, we want to have very few dependencies for a couple of reasons.
Folkert: First of all, generally, the logic that we have is quite simple,
Folkert: and so we can just write out the actual logic.
Folkert: Like this is not complicated parsing it's really you get 48 bytes they all mean
Folkert: something you need to sort of pull that apart so we can write that without some
Folkert: sort of specialized parser crate,
Folkert: we originally had a bunch of additional dependencies like clap for the command
Folkert: line interface and there's a couple others in there that ultimately we've decided
Folkert: to remove move, both because of simplicity,
Folkert: reducing binary size, and also just supply chain attack risk ultimately.
Folkert: Also, build times get better when you do this sort of thing.
Folkert: So over time, we got quite hardcore about eliminating dependencies.
Folkert: We still have a bunch. I think it's like around 100 total if you walk the tree
Folkert: of dependencies, basically.
Folkert: This is mostly like tokio and rust-tls, I think, are the two big parts of that tree.
Folkert: But yeah no we we try to
Folkert: be actually quite hardcore about doing the
Folkert: low-level stuff ourselves because we we're
Folkert: just i don't know more more lean more agile we can actually just write that
Folkert: out and have full control over exactly what happens how errors are handled we
Folkert: don't want some extra layer in between that might obscure what really goes on
Folkert: or introduces inefficiencies.
Matthias: Tokio is kind of a huge crate, or it has at least many sub crates.
Matthias: What do you use it for exactly?
Folkert: Yeah, so the architecture of this project is that we have this sort of pure
Folkert: core of the program that implements all the logic. So this is parsing logic.
Folkert: This is sort of there is a state machine in there that decides what to do with
Folkert: the clock. This is very testable logic that, you know, in theory,
Folkert: this would run on sort of any device.
Folkert: Like it doesn't depend on any operating system capabilities.
Folkert: And then we have the sort of outside layer that does all of the input output.
Folkert: And because we want to maintain connections with multiple servers,
Folkert: there's a bunch of timers in there for when to ask them next for what their
Folkert: time is. We need to keep connections going without blocking the whole program, basically.
Folkert: If we synchronize with three different servers, we want to run those requests
Folkert: concurrently, but actually in parallel so that, well, that is just essential
Folkert: for how this program works.
Matthias: It means the core itself is synchronous and on the outer layer,
Matthias: you have an async wrapper around it.
Matthias: Okay, that makes sense.
Folkert: Yeah, and that means like the core is very testable and very portable.
Folkert: tokio
Folkert: And then the outside layer, because of tokio, is actually also quite portable.
Folkert: There is some stuff at the very bottom with regards to actually updating the
Folkert: clock and with regards to configuring sockets to sort of capture timestamps that we need.
Folkert: And that is the sort of operating system specific bit that needs custom work
Folkert: for Linux or Mac or FreeBSD.
Matthias: With regards to that, do you sometimes wish that the
Matthias: ecosystem,
Matthias: was a bit more granular or maybe the entire async Rust ecosystem was a bit more
Matthias: granular and you could mix and match things and there were clean interfaces
Matthias: between those dependencies?
Matthias: Whereas nowadays you have
Matthias: as a dependency and you pretty much buy into
Matthias: the entire ecosystem there.
Matthias: Would it help you in your situation or doesn't it really matter that much after all?
Matthias: And maybe the coherence of the entire ecosystem is also an advantage.
Folkert: Yeah, for sure. I think for us, it really helps that we don't have to think
Folkert: about threading a whole lot. It's just like
Folkert: will take care of it.
Folkert: We are careful with limiting through feature flags sort of what gets pulled in.
Folkert: It's still quite a lot, but I think all of those parts make sense and are a
Folkert: kind of complexity that we don't really want to bother with.
Folkert: is a sort of well-regarded dependency, so in terms of supply chain risk,
Folkert: this is just a risk that we're sort of willing to take.
Folkert: Also we don't upgrade immediately, so if something does happen,
Folkert: we will have some time to sort we can just not upgrade to newer versions. Right?
Folkert: So we're generally okay with that. I think
Folkert: just makes a lot of sense and
Folkert: it provides us with a lot of things that we just need and wouldn't want to do ourselves.
Folkert: There are some cases where actually
Folkert: is a limitation because we're doing
Folkert: such low-level and sort of niche things.
Folkert: But we actually got some stuff upstreamed into
Folkert: , which is really nice.
Folkert: So these days, it's always fun, right, if you can sort of throw some of your
Folkert: unsafe code into a dependency and have it not be your maintenance burden anymore.
Folkert: So that's worked out. And hey, right, everyone can use it now.
Folkert: So that's been really fun. And also a sort of a cool part of the structure of
Folkert: this project is that we have time to contribute back to the dependencies that we use.
Folkert: or for libc or whatever else
Folkert: And so when that makes sense, either for
Folkert: might come up in the future, we're very happy to do that.
Folkert: And in general, these projects are very receptive and sort of active.
Folkert: So that's been a good experience.
Folkert: That m
Matthias: eans you put
Matthias: through the test and you maybe check it at the boundary of what it's capable of.
Matthias: I imagine that it's one of the areas that were not really well tested or maybe
Matthias: you don't have that many use cases at such critical pieces of infrastructure.
Matthias: And probably you run into problems that they haven't even encountered yet or even thought about.
Folkert: Yeah, we were just doing such low-level things that the API support just wasn't quite there.
Folkert: And so again, this is kind of cool, right? That like in Rust,
Folkert: actually work though, right?
Folkert: we can go to source and just learn about like, okay, how does
Folkert: And how can I make this, how can I implement this feature?
Folkert: Because I didn't really know much about the internals of async before this project.
Folkert: And then now I know at least something about the
Folkert: internals.
Folkert: There's a lot to go through, but,
Folkert: I think that is really nice that you aren't limited by the current state of
Folkert: your dependencies, even the current state of things.
Folkert: You can actually go in there and sort of make it work yourself.
Folkert: And at first, we did this just internally in our code base. We can take some shortcuts.
Folkert: But of course, ultimately, the right solution, the more robust solution also,
Folkert: .
Folkert: is to integrate it into, in this case,
Matthias: Looks like you made great progress here. and it also feels like you're almost
Matthias: done with the implementation or the protocol, the implementation itself is quite sophisticated.
Matthias: If I wanted to play around with what you've built so far, how could I get started?
Matthias: And what's next for the project?
Folkert: Yeah, so basically the implementation of the protocol is complete.
Folkert: We are working on some improvements to the protocol.
Folkert: So this is the next version of
Folkert: NTP. I believe we are the only version that implements the current draft.
Folkert: It's behind a feature flag. Don't worry. It won't actually make it into release
Folkert: binaries, but we can play around with it.
Folkert: We're also looking at making it easier to use a thing called NTS,
Folkert: which basically uses TLS to establish connections.
Folkert: So you get a bit of extra security on your connection that, or at least you
Folkert: know that that you get your time from a reliable server.
Folkert: So we've had a 1.0 release last fall.
Folkert: And so you can go and download installers. We have nice installers for Linux anyway.
Folkert: And this installer will also do a bunch of setup because you need some stuff in certain places.
Folkert: On other platforms, you need to build from source and you then need to set that up.
Folkert: But that will also work on FreeBSD and macOS because these are so close that
Folkert: it was just so easy to support them that we might as well, basically.
Matthias: What's the name of the package?
Folkert: It's called ntpd-rs is the name of the repository. repository.
Folkert: I believe also the package in the sense that we are taking steps towards getting
Folkert: included in common package repositories, for instance, Debian and Fedora.
Folkert: That takes a while to get your project in there, but we're sort of on that path
Folkert: and collaborating with maintainers there to make that happen.
Folkert: It takes a while in general. And then also because we are a Rust project,
Folkert: there are just some extra setup things that need to happen and take a bit of time.
Folkert: I think in In general, that process is pretty good these days,
Folkert: but it's still kind of different from what they're used to.
Folkert: And so there is a little bit of extra work that needs to happen.
Matthias: Well i
Matthias: And at some point, will I be able to say apt-get install ntpd and it will pull the Rust version?
Folkert: t would be apt-install ntpd-rs, or ntpd-rs, because it can't overlap with the
Folkert: existing implementation.
Folkert: But that is the plan, yes. Yes, like ultimately, we would like it to be that easy to install.
Folkert: It would also be really cool if certain Linux distributions pick it as the default
Folkert: NTP synchronization mechanism.
Folkert: Because even in the sort of simple case for just a consumer device,
Folkert: we can actually be more accurate with little extra resources used.
Folkert: So hopefully that'll happen at some point.
Matthias: And then will I be safer? Will it be more secure?
Matthias: Are you doing certain things to make the setup as secure as possible for me?
Matthias: Do I get a lot of benefits from the Rust implementation all of a sudden?
Folkert: Mostly you just get that warm, fuzzy feeling of running Rust software.
Folkert: I think on a personal machine, it's just nice to be running modern software.
Folkert: I don't think the impact is that big. The impact is never that big for,
Folkert: I think, your consumer individual device.
Folkert: The real benefit is when you sort of deploy this at scale.
Folkert: And so this is actually a sort of an interesting aspect of the project.
Folkert: It's like, you know, we've made this new fancy implementation.
Folkert: We're very happy with it. As far as we know, we have the best accuracy.
Folkert: You know, we have good performance. The binary isn't like excessively large
Folkert: or anything. thing. So we're doing good along all of these sort of technical dimensions.
Folkert: And so like you hope that such a thing would sell itself.
Folkert: And this turns out to like sort of not be the case. Certainly rewriting in Rust alone is not enough.
Folkert: People just will not upgrade if their existing setup is sort of fine.
Folkert: They just won't touch it.
Folkert: Right. So that is why we already invest heavily in making sure we have good
Folkert: documentation, that we have good observability, that, you know,
Folkert: Our performance is at least on par, but hopefully better, than existing solutions.
Folkert: We need to put in a lot of extra work to actually get existing users to move.
Folkert: In particular, because NTP is something that you forget about,
Folkert: I assume most people don't know about it really, even though it is this sort
Folkert: of vital underpinning of the internet, it's very unknown.
Folkert: And once you have a good setup that doesn't cause you issues, why would you switch?
Folkert: And so we really need to put in a lot of work where at this point,
Folkert: I would say, at least for the outside world, that it's implemented in Rust should
Folkert: be an implementation detail.
Folkert: What you get is actually a very mature and modern implementation of this crucial dependency.
Folkert: And yeah, that's sort of the approach we're going with.
Folkert: Like even without the Rust part, This implementation should be able to convince
Folkert: users that generally don't care about what language their software is written in.
Matthias: Now I have this small part written in Rust. Can I grow from within and rewrite other parts?
Matthias: Does it make sense to spread out?
Matthias: And are there any other protocols that you see might be a great fit for Rust in the same realm?
Folkert: Yeah, so this is sort of interesting, I think, in terms of how do these projects happen.
Folkert: So I mentioned that NTP was originally an ISRG project.
Folkert: So they, I think, funded it for about a year, but the project's been going for
Folkert: two years because we got a big grant from the Sovereign Tech Fund to continue
Folkert: development, basically.
Folkert: And so even though the money isn't sort of abundant necessarily,
Folkert: I think it is less of a blocker for this sort of project at this point.
Folkert: Main blockers are, you need to actually do the work, right? Like NTP is niche.
Folkert: So like how many people really exist in the world that both know Rust and then
Folkert: are excited about this domain and want to improve it.
Folkert: So you need to have a lot of domain knowledge and then also the Rust knowledge
Folkert: for, you know, the technical knowledge to sort of make a modern implementation.
Folkert: Funding still requires work, but it is available. It is there.
Folkert: Then the major issue at this point for us anyway is just how do we create that
Folkert: adoption? How do we facilitate that?
Folkert: Because, I mean, that is ultimately sort of crucial to making this project really successful, right?
Folkert: We think that sort of along technical dimensions, it is very successful.
Folkert: But in terms of real-world impact, you actually also need a lot of real-world use.
Matthias: Yes, and I can imagine that if you do that again, you repeat the same process
Matthias: for a different protocol,
Matthias: you can reuse a lot of what you built with regards to packaging and testing
Matthias: and the entire infrastructure that you have now is a bit of a template for future
Matthias: projects. Do you agree? For.
Folkert: Sure. Yeah. There's also a
Folkert: lot of, so there's a lot of infrastructure that you can reuse. Absolutely.
Folkert: So packaging is one, I think also sort of performance benchmarks are hopefully
Folkert: another one that we can sort of standardize on soon, but a lot of the knowledge
Folkert: is actually much more intangible.
Folkert: I would like, how do you get something accepted into the Debian package repository?
Folkert: For instance, how do you reduce dependencies and how sort of,
Folkert: dedicated should you be in that regard. So there's a lot of experience that
Folkert: you get from running a project like this. And actually, we've run a couple.
Folkert: We will run more in the future where you also see it's much easier to get up to speed at this point.
Folkert: And certainly also something that you learn is how to acquire the funding.
Folkert: Once you've done that once, it's relatively easy to replicate.
Folkert: You may not always end up getting the demands that you want,
Folkert: but at least the process is now pretty clear.
Matthias: What is the process?
Folkert: Well, so for the ISRG is they have
Folkert: their own agenda of our sort of list of projects that they want to do.
Folkert: If they can find the funding for it, they will sort of pick an implementation partner.
Folkert: So then it really matters that you sort of show that you have the technical
Folkert: competence for that particular subject.
Folkert: So for a lot of the timing-related projects, we have that experience now.
Folkert: For other domains, you know, you have to show that you do or that you can acquire
Folkert: that knowledge sort of along the way, right?
Folkert: For the Sovereign Tech Fund, and like there are initiatives in a lot of European
Folkert: countries for something similar, you just have to write a good proposal.
Folkert: And really what that comes down
Folkert: to is sort of, making the sort of public benefit of it sort of obvious.
Folkert: So for some projects that is easier than for others, but generally you can make
Folkert: that story work for most of these foundational projects.
Matthias: Can you imagine any critical software right now that we use on a day-to-day
Matthias: that is severely underfunded,
Matthias: where you say there's definitely a case for this, we need to have more eyeballs,
Matthias: we need to have better funding?
Folkert: So I think sort of paradoxically, a lot of this software isn't actually that obvious.
Folkert: I worked on the Dutch system for emergency response messages,
Folkert: basically, the sort of a system that will wake up someone because there is some
Folkert: sort of accident and they need to get there.
Folkert: So luckily, this is now modernized, right? But for a couple of years,
Folkert: the state of that software was pretty bad.
Folkert: Like it was just some big binary blob that the source code wasn't available.
Folkert: It ran fine, as far as I know, there were never any incidents,
Folkert: but like that is not a good situation to have for something that crucial.
Folkert: And you wouldn't really know about this because because that is not public knowledge, right?
Folkert: But there's a lot of software like that in a lot of critical domains,
Folkert: like energy, shipping, any form of transport.
Folkert: A lot of these very vital sectors have a lot of this very dated software on very dated hardware.
Folkert: That is a major issue, a major risk.
Folkert: With regards to the internet, I know there's currently work on,
Folkert: well, TLS is still continuing with Russell's, of course. There is work on DNS.
Folkert: I think that covers most of the major protocols for sort of the public internet, at least.
Folkert: And there is now some branching out. So you can sort of see sudo was also an ISRG project.
Folkert: Their connection to the internet is already much less obvious,
Folkert: but sudo is in the chain of running internet infrastructure.
Folkert: And so ISRG still thought that that would be sort of in scope and sort of a
Folkert: useful project to do. There is now sort of a focus more on media decoders.
Folkert: So you may remember this recent problem on Apple devices, I think, with the font decoding.
Folkert: There was a pretty serious bug.
Folkert: This happens every so often. So either the image decoding or font decoding or
Folkert: something like that will have a critical security vulnerability. Right.
Folkert: And so that is the next sort of domain where, at least sort of internet related,
Folkert: where the memory safety will have, should have a huge benefit.
Folkert: But there, the performance is much more crucial, right?
Folkert: Like you can't trade safety for performance there.
Folkert: You need to maintain the performance and then also somehow guarantee the security.
Folkert: So that is quite a tricky sort of engineering project where you want to limit
Folkert: the amount of unsafe Rust code, but also you need some amount of unsafe Rust
Folkert: code just to make sure that the performance is there.
Matthias: It's funny to think how a lot of our public infrastructure runs on very deprecated legacy software.
Folkert: Yeah, don't think about that.
Matthias: There was this story on Hacker News the other day about some German train provider
Matthias: looking for a Windows 3.11 admin.
Matthias: I saw that.
Matthias: Maybe in this context, it might even make sense.
Matthias: I don't know all the details, but for sure, there's a lot of legacy out there
Matthias: and a lot of deprecated things.
Matthias: And people don't know how it works. Maybe they don't even have the source code, as you mentioned.
Matthias: So it is true. true so there might be future projects that are worth funding.
Folkert: Yeah there's this this uh there's this
Folkert: fun story about the dutch tax collector's office that basically
Folkert: can't collect tax anymore because they don't have any cobalt programmers this
Folkert: is a major issue where all of that stuff still runs on these very old mainframes
Folkert: and they just have a really hard time modernizing that infrastructure where
Folkert: it now impacts the actual performance of sort of their their core business as it were.
Folkert: This problem is everywhere.
Matthias: I wonder if that's a bug or a feature, but that's the topic of another.
Matthias: No, n
Folkert: o, no. Like everyone agrees it is a bug because in effect, they can't make
Folkert: any changes to tax policy because they just can't get the computer systems to go along with that.
Folkert: So yeah, it's a big problem. It's the most obvious one too, right?
Folkert: Like there's a lot of stuff that is, of course, hidden under the surface,
Folkert: but that would benefit hugely from having a more modern software and hardware, really.
Matthias: Time flies. We're already getting towards the end.
Matthias: There's a tradition around here to ask a last final question.
Matthias: That is, what would be your message to the Rust community?
Folkert: Yeah, so I have this, I don't know if it's a thing, but I am very passionate
Folkert: about compile time performance. performance.
Folkert: And I think we should be more serious at, first of all, admitting that this is a problem.
Folkert: Of course, the performance is better than C++, but doing better than C++ is
Folkert: not hard and shouldn't be our benchmark.
Folkert: Realistically, for all the projects that I do, compile times slow us down.
Folkert: They are a major source of frustration and just lost,
Folkert: productivity. And again, like any sort of even small improvement here has sort
Folkert: of majorly sort of outsized benefits to all Rust programmers in the world,
Folkert: which I believe is like over a million or over 2 million now.
Matthias: 3.7 million as far as I'm aware.
Matthias: So y
Folkert: ou know, even one second saved, right, per day translates into huge benefits.
Folkert: I think we can do much better than that. There are very cool existing initiatives
Folkert: in that direction, like the parallel frontend, etc.
Folkert: But I think having more of that focus is important.
Folkert: Because of course, we all have our favorite unstable feature,
Folkert: right, that we would like to see stabilized.
Folkert: But the performance of the compiler just hits you every time you hit compile
Folkert: or test or check or whatever.
Folkert: And like, that is no joke, like we need to really, I think, do better there collectively.
Folkert: And then separately, I would like to say something about the RustNL conference
Folkert: that we will be organizing the 7th and 8th of May this year in Delft,
Folkert: which is reasonably close to Amsterdam.
Folkert: For everyone from outside of the Netherlands, it's very, very close.
Folkert: And so that was a lot of fun last year. We had a lot of, well,
Folkert: both national and sort of international speakers. speakers and it was just a
Folkert: great time and we hope to replicate that this year.
Matthias: If you're in the area come by i will most likely be there and i hope to see
Matthias: a lot of people it was so much fun to talk to you today i learned a lot about
Matthias: protocols so thanks a lot for all the input it was amazing,
Matthias: and i hope to see you in person at some point in time awesome thanks.
Matthias: Rust in Production is a podcast by corrode and hosted by me, Matthias Endler.
Matthias: For show notes, transcripts, and to learn more about how I can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.