Matthias: This is Rust in Production, a podcast about companies who use Rust to shape
Matthias: the future of infrastructure.
Matthias: My name is Matthias endler from corrode, and today we talk to Vegard Sandein
Matthias: from KSAT about talking to satellites with Rust.
Matthias: Vegard, can you introduce yourself and KSAT, the company you work for?
Vegard: Thanks for having me. My name is Vegard Sandengen. I have a master's in computer
Vegard: science, I have worked most of my professional career in the space domain,
Vegard: even though it's usually on the ground, and I've been working at KSAT now for the last four years.
Matthias: And recently, you became a father, so there's one more rustacean in this world. Congratulations.
Vegard: Thank you.
Matthias: So, can you say a few words about KSAT? I know that the slogan is,
Matthias: connect space and Earth, and I really like that, but what is it about?
Vegard: KSAT is the abbreviation of the company, which is... Kongsberg satellite sources.
Vegard: So we're getting data from space to Earth, and then we're using that data.
Vegard: So ground network operations and Earth observation networks.
Vegard: I work in the ground network, which
Vegard: is our distributed network of antennas situated all around the world.
Vegard: And we enable satellite owners to talk with their satellites and get their data.
Matthias: A lot of people only know about satellite technology from television or from popular science.
Matthias: And the knowledge they have is probably rooted in the 60s and 70s.
Matthias: But a lot has happened since then.
Matthias: What has happened since the 60s?
Vegard: Yeah, satellite industry was traditionally operated by satellite companies,
Vegard: and they're using their software to just deliver
Vegard: on whatever their satellite had and a satellite
Vegard: itself started like in the 60s with the russians launching
Vegard: Sputnik and it was very expensive i mean launching a
Vegard: satellite took a government agency so all
Vegard: the way until basically Space X and a lot of the other newcomers in the satellite
Vegard: business or in the launch business came along it was extremely expensive to
Vegard: launch satellites so it was mostly just agencies and government entities that
Vegard: it could afford to put satellites into orbit.
Vegard: And some of those satellites are geostationary satellites in delivering your
Vegard: satellite communications for TV or for your sat phone, if you had that.
Vegard: And from the old days, it was mostly communication-based, but NASA and ESA also
Vegard: launched scientific instruments to monitor the earth or to monitor the sun or
Vegard: to send probes into outer space to do some other readings.
Vegard: And the way satellites communicate is almost,
Vegard: exclusively through radio frequency communication and different wavelengths
Vegard: on the radio frequency spectrum.
Vegard: And the type of wavelength you use determines what's the quality of your transmission.
Vegard: And Earth observation satellites, very close to the Earth, they orbit the Earth
Vegard: maybe 14, and 15 times a day, they can produce a lot of data.
Vegard: And as the instruments have gotten better, the resolution of whatever measurements
Vegard: they're doing is getting higher.
Vegard: The amount of data is getting higher. And the majority of the way to actually
Vegard: get data done is to have contact with the ground station.
Vegard: And you have a limited visibility over a ground station.
Vegard: So you only get like 10 to 15 minutes of visibility maximum that's peak and
Vegard: you have to push down gigabytes of data so the amount of data we're talking
Vegard: about is ever increasing.
Matthias: Yeah thanks for the overview but one thing i always wondered as some somewhat
Matthias: of a bystander is what is the standardization of the communication protocols. Do
Matthias: we keep using the same protocols since the 60s or does every satellite have
Matthias: its own protocol or is it something in between?
Vegard: It's everything and nothing, unfortunately. So there is a standardization body
Vegard: called CCSDS that a lot of the government agencies contribute into from the
Vegard: early days of the 80s-ish, if I remember correctly.
Vegard: So a lot of the hardware-related radio frequency protocols and how to handle
Vegard: data on the physical link has a lot of different standards.
Vegard: And in order to push data over the air, you also need some error correction
Vegard: and you need to be able to sequence your data just like TCP/IP,
Vegard: there's an equivalent standard in the space industry.
Vegard: Coming into the new space era there's a lot of
Vegard: new contenders on the market that are software
Vegard: companies that are using spacecrafts and
Vegard: not spacecraft companies using software they're
Vegard: also not really following some
Vegard: of these standards from the agency era so you get a lot of compatibility issues
Vegard: where you're basically having to custom fit okay how do we talk to this spacecraft
Vegard: because This is a new software company that has just looked at the standard and said,
Vegard: ah, we don't really need this. We'll do it our way. And it works for them.
Vegard: But at some level, you have a minimum viable product that you can share on a radio frequency level.
Vegard: And most people are compatible with that. But after that, all bets are off.
Matthias: Sounds like that approach would generate a ton of legacy code in a very short time.
Vegard: Yeah.
Matthias: Now, let's talk about the size of operations at KSAT.
Vegard: KSAT started off 25 years ago as a company, and we started off with one antenna
Vegard: and one customer, and that's about it.
Vegard: And as KSAT grew its company and this market shift into new space with all these
Vegard: new software actors really exploded the number of satellites launched into space,
Vegard: And KSAT followed suit and built up both their antenna park on how many antennas
Vegard: you have and how many employees we have to deal with this and how many engineers.
Vegard: And at this point in time, we're roughly at ballpark between 100 to 300 active antennas.
Vegard: It is one of the biggest providers of commercial
Vegard: ground station services.
Matthias: I think the official website mentions 23 sites worldwide, which sounds crazy to me.
Matthias: What is a site specifically and what goes in there to maintain that?
Vegard: A antenna site for us is mostly, it's a place where we need a lot of power and
Vegard: we need fiber optic cable, hopefully.
Vegard: We don't have that at every site, but what qualifies as a good site is that
Vegard: it's far enough apart from any other site we have, and that it covers a lot
Vegard: of ground we don't actually get from other sites in the vicinity.
Vegard: And the placement of the sites are usually depending a bit on what orbit the satellites go in.
Vegard: So the satellites usually have two orbits that are relevant.
Vegard: It's like polar orbit, where they go from pole to pole, and then it's the other
Vegard: one where they just follow the equator.
Vegard: And if you only have ground station at the equator and you have a polar orbiting
Vegard: satellite, you only get the visibility twice a day.
Vegard: But if you have a ground station near the poles, you get 10,
Vegard: 12, 14 contacts a day. So it really depends.
Vegard: Each contact has a duration of anything between 5 to 15 minutes, really.
Vegard: And that can generate anything from a few gigabytes to 100 gigabytes per contact.
Matthias: Data processing can come later, but the data exchange happens during that time frame.
Vegard: The data exchange between the satellite and the ground station, yes.
Vegard: And because of the volume of data increasing so much, our main concern going
Vegard: forward is not really building enough antennas, it's actually just building
Vegard: enough infrastructure to handle all this data.
Vegard: Because there's so much data and you need to push it around and you need to
Vegard: provide it to the customer in a reliable fashion.
Vegard: And it can be quite unreliable networking between a remote site in Canada or
Vegard: in New Zealand and you have the customer on the west coast of the US.
Vegard: That's a challenge really going forward.
Matthias: Okay, so to summarize, the setup is a bit like this.
Matthias: You have a ton of satellites circling the Earth on a regular basis.
Matthias: They go around the Earth 10 to 15 times a day or so, roughly like that.
Matthias: And then on the ground, you have antennas on ground stations.
Matthias: And then these antennas, they connect with the satellites, do the data exchange,
Matthias: and then you need to send the data over, say, Fiber to a central place.
Vegard: Usually delivered straight to the customer but due to our volume of data we
Vegard: also have to temporarily store it on the site itself but do not lose data in the process so yeah.
Matthias: Two things come to mind first it needs
Matthias: to be extremely reliable because if you lose the data that is big outage and
Matthias: probably a loss to the customer as well and the second part is how often can
Matthias: you make changes to that code how often can you modify the code that also needs to be reliable,
Matthias: i'm guessing you probably even have limitations as to how often you can access
Matthias: these ground stations and make changes.
Vegard: Yes that is correct that that
Vegard: has to be really reliable but you're actually right on
Vegard: point with how do we update the code because it's not
Vegard: that we're using the antenna 100 of the time but the ecosystem around the antenna
Vegard: with our software running in different hardware close to the antenna it is not
Vegard: easy to access i don't have access to it for instance but so i just have to
Vegard: push code and hope that someone else deploys it worst case it can take,
Vegard: weeks before something is deployed worldwide that
Vegard: is a process we're obviously trying to optimize and get better at
Vegard: but it's it is a pain point because it's also in
Vegard: inaccessible sites and the most inaccessible sites we have is probably in our
Vegard: antarctica and troll station that also doesn't have fiber optic cable so anything
Vegard: that you put down there we also have to beam up to a geostationary sites satellites
Vegard: so we can beam it down to earth again a place where we have fiber.
Matthias: I guess the huge advantage here is that for code that is written in Rust,
Matthias: you could just deploy a static binary and people would just be able to run it on the deploy target.
Vegard: It's generally that easy. I mean, everything we do nowadays is dockerized.
Vegard: On all our ground stations, we're running some variant of Kubernetes and just
Vegard: running it on OpenStack or Kubernetes directly.
Matthias: One could think that since you operate in the ground station and you probably
Matthias: have access to a rack or so, you're not resource-constrained.
Matthias: But one thing that people might forget is that you don't do constant updates
Matthias: to the hardware over there.
Vegard: We're definitely resource-constrained on a lot of our sites.
Vegard: Not all of them, but a lot of them. It can take us eight months to get a new
Vegard: computer just ordered from our vendor, and then we have to ship it to anywhere
Vegard: in the world, and you have to get people there on-site to install it.
Vegard: So we are resource constrained in the fact that we don't want to over-provision
Vegard: every data center around the world near to all our antennas on our ground station sites.
Vegard: Because, first of all, we don't necessarily have the resources to do that,
Vegard: and we don't have the ability to do it at some point.
Vegard: So it's nice to use something that doesn't hog all the resources.
Matthias: Wouldn't it then be super easy
Matthias: to fall into a trap of being extremely conservative about tech decisions?
Matthias: People might associate space technology with a lot of very old conservative
Matthias: technology, and maybe for a good reason, because it's tried and tested.
Vegard: I think the satellite industry or space industry is definitely very conservative.
Vegard: It takes a lot of effort to qualify something to run in space. I know,
Vegard: RustConf, last year there was one of the sponsors was K2 Space.
Vegard: They're actually a space company that with a lot of recruits from former AWS
Vegard: and SpaceX, they wanted to do everything in Rust. They wanted to build the satellite.
Vegard: They wanted to build the firmware. They wanted to build all the ground resources.
Vegard: 100% of Rust. They had a lightning talk at RustConf. It's probably out on YouTube.
Vegard: So there are definitely contenders out there that want to not be so conservative.
Vegard: But from the old space they are very conservative but i wouldn't say that's
Vegard: necessarily true on the ground the ground is a bit more like we can touch this
Vegard: we can fix it it's not the same in space earlier.
Matthias: You said there was a shift in the industry so
Matthias: we moved from space companies using software to
Matthias: software companies doing space things mostly two
Matthias: companies come to mind right away one would be spacex and the other one would
Matthias: be blue origin but i'm assuming that's just a tiny little slice of the picture
Matthias: and maybe there are other software companies that i might have heard of that
Matthias: pushed into the into the space.
Vegard: Into the space yes you also have a few other providers that
Vegard: are up there trying to and successfully doing
Vegard: so like rocket lab but from but these
Vegard: are launch providers they're facilitating the software
Vegard: companies to launch something into space but otherwise
Vegard: aws is actually going right
Vegard: at it and they're going after the data primarily they want the data because
Vegard: that's aws's business model is data and there's a lot of data in space a couple
Vegard: of years back or three or four years back they launched a ground station service
Vegard: which is i I wouldn't say a direct competitor to us, but they are definitely a competitor.
Vegard: And we have made a strategic partnership with AWS to be,
Vegard: a ground network of network providers. So people can come to us and they can
Vegard: use the resources in AWS, their antennas, their setup, but they can do it through us.
Vegard: But the business model is a bit different because AWS, as I said, they're a data company.
Vegard: They really just care about getting their data into the AWS data center.
Vegard: So you can do whatever you want with it there.
Vegard: So the space part is just a means to an end, really.
Matthias: So we move from space exploration to data exploration.
Matthias: What has changed on the language side? How did the story go at KSAT?
Vegard: Initially, everything was engineers writing Perl scripts and just making it work.
Vegard: And that has scaled very well, but it's still written in Perl,
Vegard: and it's not the newest version.
Vegard: And at some point, we needed to have a bit more control of whatever is running
Vegard: on our antennas. And that was developed in Java in the mid-2000s with an Oracle
Vegard: database. And that has scaled well.
Vegard: We're very thankful for the legacy that was provided to us so that we can even
Vegard: be here today to do something else at a bigger scale.
Vegard: Because that would not be possible without the humble beginnings.
Matthias: The 2000s were definitely the time of Java. It has some really nice traits,
Matthias: and I think it resonated well with the challenges of its time.
Matthias: But then what happened in the 2010s at KSAT?
Vegard: Yeah, so at some point, we're kind of scaled up with a bit more developers and
Vegard: with a bit more modern scripting and kind of Python took over.
Vegard: We have multiple Python applications still in production today from that era.
Vegard: But yeah, we started to see that due to how that Java application,
Vegard: and not necessarily Java in itself, but just the database and all the Perl integrations
Vegard: that unfortunately had direct database access,
Vegard: meant that we had a distributed network all over the world with scripts being
Vegard: able to access the raw contents of our database.
Vegard: And that was not very scalable. We launched
Vegard: an initiative to move away from this world into
Vegard: a more modern world where we can have more
Vegard: control over the life cycle of the data that
Vegard: we put in the database. 20-25 years ago everything was on ftp xml drop boxes
Vegard: you can call that an api as well but we've decided that we can try to offload
Vegard: responsibility into like segregated new Postgres database where access to the
Vegard: data is tightly controlled through an HTTP API.
Vegard: Yeah, so we're employing a sort of a strangle pattern on that and just trying
Vegard: to just grope in any responsibilities and kind of rewrite and repurpose it and
Vegard: have successfully launched a.
Vegard: Competing solution in-house now where half of the antennas are on the old system
Vegard: the old api was written in pearl and it was strangled on the hftp layer into a coffin application,
Vegard: and then we nipped at it and moved different responsibilities and endpoints around and now,
Vegard: i would say from a responsibility point of view where like it's 40 60 in rust
Vegard: right now but a lot of the boring parts are in the kotlin application but we're
Vegard: actively working to to migrate the remaining kotlin portions as well over trust earlier.
Matthias: You mentioned the strangler pattern how does it work.
Vegard: So strangler pattern is very
Vegard: convenient when you have a code base
Vegard: or a interface layer where you
Vegard: can very well design know what's
Vegard: going in and what's going out and you know that everything below
Vegard: this is just complete mess and you don't
Vegard: understand anything but you understand the interfaces or
Vegard: the boundaries and you can replace the implementation
Vegard: under each boundary with very
Vegard: great control and see the differences in implementation and behavior and you
Vegard: make it entirely seamless to all consumers that you have actually done anything
Vegard: which is very nice but it It requires that you have some sort of abstractions
Vegard: that actually make this feasible.
Vegard: And from an HTTP API layer, very easy, because the contract is in how the API
Vegard: responds or what parameter it takes.
Vegard: And you can replace that in any language.
Vegard: It's not really that hard. It's just a lot of verification that you've actually
Vegard: replicated all the behavior.
Matthias: Now, let's focus on the API for a second.
Matthias: You mentioned that it's a split between Kotlin and Rust at the moment.
Matthias: Where do you draw the line?
Vegard: I don't think there's a natural split now, other than whatever developer or
Vegard: team took that responsibility and what they were comfortable with.
Vegard: So we've had a very open policy at KSAT on what languages we would use to solve
Vegard: whatever problem. And it has definitely been a pushback to introduce Rust in
Vegard: some capacity by some team members in different teams.
Vegard: And I'm not necessarily sure all of their concerns are, I would call,
Vegard: valid, but there are definitely concerns.
Vegard: And some of the pushback I've heard is usually it's not mature enough or the ecosystem is not there.
Vegard: And I feel that is a sentiment that is often held about Rust that I'm not necessarily
Vegard: sure is true anymore, because I feel the ecosystem is very much present.
Vegard: I can do everything I want in the ecosystem in Rust today.
Vegard: And the other part is maybe just a lack of knowledge of how do you use such
Vegard: complex terms, because it comes from a system background.
Vegard: And a lot of regarding boroughs and lifetimes and stuff like that,
Vegard: it can seem a bit intimidating.
Vegard: To someone that's usually just very happy in their Java or .NET environment
Vegard: where that is not necessarily a concern for 99% of what they're doing.
Vegard: There are also positive receptions of Rust.
Vegard: And I have personally been able to, I don't know, convert a couple of teams to use Rust.
Vegard: So yeah, we're approximately three or four teams now using Rust in production
Vegard: at KSAT with maybe four-ish people in each team that's actively writing Rust.
Matthias: How does that usually go for you when you approach a team and they are curious
Matthias: about Rust, but they are not entirely convinced yet?
Vegard: The conversation often goes in the direction of this is what is very good about
Vegard: Rust, and that's what I start with.
Vegard: And you have to make some concessions. And the concessions are obviously just, is it a good team fit?
Vegard: Because I don't think Rust is hard to use once you've gotten over that initial,
Vegard: whoa, what happened here? It's a shock.
Vegard: But a lot of teams have their experiences in their toolboxes in other languages
Vegard: and know how to solve them.
Vegard: And if you don't really have a champion on that team itself,
Vegard: I don't think it's possible to really introduce Rust into a team because the
Vegard: team has to embrace it themselves.
Vegard: That it's a no-go if the team is not championed from within, really.
Vegard: So my job is more just like I try to do some good mentoring and try to have
Vegard: some common guidelines and try to curate some crates and make some internal
Vegard: crates that help the process along internally with the tooling and the way we do things.
Vegard: But ultimately, you require that team champion as well to be on your team.
Matthias: What's your success rate here? Have you lost some of these battles?
Vegard: Not on a team level, but maybe on an individual level, yes.
Vegard: But the general vibe is that it's going more and more into us for a lot of our distributed systems.
Vegard: And just because it's so nice to use once you actually get to know it.
Vegard: So it's just that hurdle of inviting people in that haven't used it before.
Matthias: I'm almost too afraid to ask it, but has Go ever come up in that conversation?
Vegard: Go has come up multiple times, and we have production code in Go as well.
Vegard: I'm a bit annoyed at that sentiment as well, because Go is maybe annoyed is
Vegard: not the right word, and I'm a bit intrigued by the why don't we just do it in Go?
Vegard: Because Go was released in March 2012.
Vegard: It's three years older than Rust. at this point is 10 and 13.
Vegard: It's not that big and much of a difference.
Matthias: In terms of age, but in terms of functionality and in terms of developer ergonomics, maybe?
Vegard: Yeah, but Go had a very simple language to begin with. So it was very easy to get going with Go.
Vegard: But I also think that there is an ecosystem in Go,
Vegard: but the ecosystem is harder to engage with than it is the Rust ecosystem because
Vegard: the tooling and with cargo on the kin as just miles above any tooling you have in Go.
Vegard: So that makes it, for me, also a no-brainer just because, like,
Vegard: disregarding just the language itself and the features and ergonomics of the
Vegard: language, just the tooling and the ecosystem with using the language is what
Vegard: makes Rust the number one contender on the market.
Matthias: Go is very much a day-one language. and starting a project and getting to your
Matthias: first production version is usually very ergonomic, very quick, very elegant.
Matthias: The problems start to arise on day two. Not exactly day two,
Matthias: but when you have a larger code base, you feel the limitations of the language of the ecosystem.
Matthias: It's trying to constrain you somehow. almost feels like it's strangling you.
Vegard: And you're not strangling it.
Matthias: I probably would have made the same decision in your position, of course.
Matthias: Obviously, I'm biased, but you have to maintain this software for a very long time.
Vegard: Yeah, so definitely, from my experience point of view, just being able to model
Vegard: your code in a way that just, it just feels,
Vegard: you just know where the boundaries of what you've made in the main And it's
Vegard: very easy to move that along and refactor it.
Vegard: So back in eons ago, I was a C and C++ developer, and I did a bit of that and a bit of that.
Vegard: And just trying to refactor a C++ code base and having confidence that you've
Vegard: actually done it correctly, I have never had that.
Vegard: But Rust, if it compiles it works, it basically is that.
Vegard: And that sentiment is overused i think
Vegard: but it still feels very true at some point because the
Vegard: compiler is so powerful but whenever it compiles i'm confident and i also have
Vegard: a few tests here and there and where the tests run as well which they do 99
Vegard: percent of the time after i've done a major refactor i'm confident i will push no problem funny.
Matthias: That you say you have a few tests here and there does that mean you lean into
Matthias: Rust's strong type system a lot as well.
Matthias: And maybe you don't have to write that many tests that you would have to write
Matthias: in other languages, more dynamic languages like Python.
Vegard: Oh, definitely. Our tests is, I think there's a concept called like a diamond-shaped
Vegard: testing or something where you basically, you have very few unit tests,
Vegard: you have very few system tests, but you have a lot of integration tests.
Vegard: And those integration tests are placed on the boundaries of the network layer, so HTTP.
Vegard: And I have all my
Vegard: tests are basically just HTTP related API tests
Vegard: because I don't really care how the structs
Vegard: or functionality within the Rust code base behaves because what's important
Vegard: is just what is the contract or the HTTP boundaries so we have a few tests down
Vegard: to the database over the HTTP layer but from unit test point of view almost nothing Thanks.
Matthias: But in order for that to work, you would have to lean very heavily into the
Matthias: Rust mechanics, into the type system, and you would have to rely on it.
Matthias: Are there patterns that you commonly use to fully embrace that part of Rust?
Vegard: Yeah, so I use quite a lot of new types.
Vegard: For instance, a UID, I will new type it into a variant that represents this resource.
Vegard: Meaning that the API layer is very communicative of what it's actually expecting,
Vegard: or not the API layer, but the code base itself that serves.
Vegard: So it's very easy to modularize different components that work in some form
Vegard: of hierarchy because the types are so strong that you can convey so much with
Vegard: both the primitive types themselves, but also some types in form of enums.
Vegard: The one thing I miss every time I go to any other language is just the enum.
Vegard: I think this I could model very well in an enum, and I don't have this capability. And it saddens me.
Matthias: Do you have an example for an enum that, for example, comes to mind where modeling
Matthias: some certain business logic was very ergonomic?
Vegard: So I'm a big fan of the one-off pattern.
Vegard: It is represented in, for instance, OpenAPI definitions.
Vegard: There is like a one-off you can represent there. doing code
Vegard: gen for one-offs in open api to any other
Vegard: language is horrible but code
Vegard: gen 2 was very easy to use and being
Vegard: able to represent the fact that this resource has different
Vegard: properties depending on which kind it is is very powerful because even though
Vegard: at some level you're talking about this resource it has one resource id but
Vegard: it can manifest itself as different forms of different versions or represent
Vegard: different physical attributes on the network.
Vegard: And on some abstractions, you don't really care about those properties, but on others you do.
Vegard: It's very nice to be able to represent just the exact properties that are present
Vegard: and not load of optionals that are present only is this is true and this is true.
Vegard: And you have to carry that logic throughout the code, that makes it harder to refactor as well.
Vegard: If you know that this can only be set if this other value is set,
Vegard: and that's invariance in your code that you kind of encode with the enums instead.
Matthias: I'm not too familiar with it, but I know that in a schema you can say this is
Matthias: one of these variants, one of these kinds, and I guess it maps really well to enums.
Matthias: If you go further one step, you're probably also using the serde ecosystem of things
Matthias: and say this is my input type and so I convert it from the schema.
Vegard: So we're leaning heavily into serde. It's an excellent library.
Matthias: Any other crates that you personally like for that sort of work?
Vegard: You usually have to do some customizations on top of serde with serde with or
Vegard: stuff like that to actually do the proper transformations.
Vegard: I've been also experimenting now with Utopia to generate OpenAPI specifications.
Matthias: It's called UTO IPA. It's a very common misspelling, unfortunately.
Matthias: I made it a dozen times until someone pointed it out.
Vegard: Yeah, I will probably continue to misspell it.
Matthias: The reason why it's called UTOIPA, by the way, is IPA is API backwards.
Matthias: And it's also a good beer. That's from the READMEs.
Vegard: Of course. Sorry. Yeah.
Vegard: One slight issue I have with serde is that it's very versatile,
Vegard: well, but it doesn't really give you that great of a structured way of accessing errors.
Vegard: And that boggles me a bit because I really want to give good structured feedback in our API surfaces.
Vegard: And I don't want to fork Siri to just fix that because then I'm incompatible with everything.
Vegard: I'm not entirely sure how to solve that on an ecosystem level.
Vegard: But right now, I've just wrapped the outputs and parsed the strings to extract
Vegard: the vital information that I want.
Vegard: But I would definitely like to see a bit more structured error responses on
Vegard: what went wrong in the serialization process.
Matthias: I personally see serde more of a contract.
Matthias: You have the value type, you have to serialize, you have these traits,
Matthias: that's your building block.
Matthias: So what keeps you from building structured error messages from these smaller building blocks?
Vegard: Because the serde error type doesn't give you, like the serde error type, it is,
Vegard: well, I think it's possible, but it is, we're using the JSON, serde-json,
Vegard: because it's what we communicate over, and the serde-json error type eradicates
Vegard: any references to which field, for instance, was the error at.
Vegard: So you will have to parse the stringified message to extract it was at this
Vegard: field to get it out or you have to fork serde-json and fix it there.
Vegard: I could probably do that as well but I've seen it in multiple JSON parsing libraries
Vegard: as well that the level of programmatic access to the variants are not that great.
Vegard: But other than that, the serde ecosystem is amazing. You can do a lot of stuff with it.
Vegard: Just have to be a bit more forgiving on how you output the errors to the end
Vegard: user because that's kind of what matters here i mean for me as a programmer
Vegard: i don't really care but it's not the consumer of the api that cares from.
Matthias: What i can tell from our conversation so far stability is the main focus.
Vegard: From listening to a lot of your other guests
Vegard: on this podcast doing a lot of cool shit and they're
Vegard: it's it's very fun to listen to but and i get the feeling that our first usage
Vegard: is boring we're just using the top level just web frameworks and sqlx and axum
Vegard: and serde and just putting it all together and just making it work. I have a good example of that because,
Vegard: a couple of months back we needed to do some changes in a few of our services
Vegard: running and I went into the repository for that service to actually fix it,
Vegard: and I saw the last commit was one and a half year ago, and it's just been running. One and a half years.
Vegard: I haven't touched it, and I have never had that experience in my professional career.
Vegard: That service was the main authentication authorization service that authenticated
Vegard: and managed every API key and principle, so it was used on every request.
Vegard: It's really chugging along, so it's amazing. I've had only good experiences on that front end.
Matthias: Did you also have any bad experiences with Rust?
Vegard: You can call it a bad experience, but I would camouflage it as a good experience.
Vegard: So we've been running on-prem coaster for many years, and that on-prem coaster
Vegard: hasn't really gotten that much love and attention.
Vegard: So it's just chugging it on with the resources it had six years ago when it was installed.
Vegard: We also do a lot of calculations regarding satellite trajectories and visibilities
Vegard: to our run stations and stuff like that.
Vegard: So one of the things i wanted to calculate was just
Vegard: okay when is a satellite visible over
Vegard: our ground stations and we support quite
Vegard: a lot of satellites and we have a lot of ground stations so there's a
Vegard: lot of maths to figure out when are you where
Vegard: and when can i talk to you and i naively
Vegard: just put everything in a loop and then
Vegard: i slammed rayon on it and i pushed
Vegard: it to production and a couple of days later one of
Vegard: my devops team came and just like our production cluster is like running at
Vegard: 80% cpu it's struggling a bit also and it's majority from the service i just
Vegard: updated and yeah the computations work fine but it had a wider impact on our
Vegard: other production services,
Matthias: So it's too performant
Vegard: Too good i had to dial that back.
Matthias: Okay i can see how that might also be a benefit or how i could see it as a win
Matthias: but are there any other issues with the wider rust ecosystem that come to mind.
Vegard: Yeah i mean we're a big user of async because we're using axum and everything
Vegard: is just on a tokio runtime and it just works very well just doing basic features
Vegard: to handle HTTP requests and doing features to send database queries and get responses.
Vegard: And that just works very well. But when you're trying to combine that with a
Vegard: feature in the HTTP layer to also provide some computations, we ran into some issues.
Vegard: So a few months back, someone used our API in a way that we hadn't anticipated.
Vegard: And there was too much traffic on something that blocked. and just everything,
Vegard: everything just stagnates and response time speaks and it affects everything.
Vegard: And just trying to hunt down where we actually block or do computation for so
Vegard: long that you're starving the tokio runtime, that was very challenging.
Matthias: What I see a lot is teams using their development laptop to start a larger tokio
Matthias: application with say 16 or 32 cores.
Matthias: And then when they deploy the same service to production it ends up running
Matthias: on a two core node and obviously that's a completely different environment.
Matthias: Was it one of these cases where,
Matthias: the production system was very resource constrained and when you tested it in development it was not.
Vegard: The problem manifested itself when the traffic increased
Vegard: enough to actually trigger it so we
Vegard: didn't really trigger it we could reproduce it locally at some point when we
Vegard: actually knew what traffic to induce but so we had some inklings when stuff
Vegard: went wrong but it was quite a goose chase down this set of futures and where
Vegard: do you actually, how do you measure what blocks?
Vegard: And trying to use tooling like tokio console, it's a great project,
Vegard: but it's just not insightful enough at that level yet.
Vegard: So I would say the tooling is probably not right for the abstractions we need
Vegard: to be able to efficiently bisect where is the issue and how do I solve it?
Vegard: Solving it is very easy in tokio. You just spawn it on the blocking runtime
Vegard: and it's fine but it's definitely something to be aware of so it's a pitfall
Vegard: for newer developers and it got us as well.
Matthias: The typical pattern is that you see a spike
Matthias: on the cpu and there's really
Matthias: not much traffic coming in anymore it blocks on the the api layer but in reality
Matthias: your cpu is super busy with some computation but then it still doesn't tell
Matthias: you where that computation happens You just need to dig deeper and understand
Matthias: the business logic of it all.
Vegard: So that's somewhere also where the distributed tracing you have in an application
Vegard: and how you have insight into that comes well into mind i also like the tracing
Vegard: ecosystem very good love it but figuring out how you actually use tracing and
Vegard: like a distributed sense,
Vegard: it's a learning curve where you have to basically puzzle about pieces together
Vegard: yourself to figure at how do you actually get the correct level of tracing in
Vegard: the applications and across applications.
Vegard: That's also probably an area where there would be a good fit for some higher
Vegard: level abstraction crates for server application that just needs to have good
Vegard: defaults on everything.
Matthias: Do you use tracing across language boundaries or just within the Rust context?
Vegard: We use the W3C trace context standard to send trace parent headers to correlate
Vegard: tracing information across applications, but that works fine.
Vegard: We set up our own tracing infrastructure using tracing to create with a custom
Vegard: subscriber to Azure App Insights.
Vegard: App Insights is a good service, but it's also quite expensive. but
Vegard: just knowing where to wire up what you need
Vegard: to call when and how where in
Vegard: tracing and how do you model that into whatever
Vegard: subscriber you have so using for
Vegard: instance OpenTelemetry versus Honeycomb versus App
Vegard: Insights they all have a different behavior on how you open spans and when you
Vegard: close them and how you annotate them and when you actually send the event it's
Vegard: a learning curve to just employing correct tracing in your application is not
Vegard: something that's extremely easy to understand.
Vegard: So you usually spend a few months on it.
Matthias: From our conversation so far, it feels like a lot of services run on Azure or,
Matthias: cloud in more general terms, but how does that relate to whatever you do on the ground stations?
Vegard: The API layers we've developed over the years, it's been primarily,
Vegard: as you say, in a cloud setting, but due to the widespread nature of our antennas
Vegard: and where they are, we're also resource constrained, as we touched on earlier,
Vegard: on the resources we have on each antenna.
Vegard: And our challenges are often related to having running code there that can run
Vegard: forever and not have any downtime, really.
Vegard: Many years ago, we deployed at least one service on each antenna stack throughout,
Vegard: which is written in Rust.
Vegard: And it's just responsible for ping-ponging back whatever is in the cloud.
Vegard: What should I do on this antenna? So it's what we call our scheduler.
Vegard: We schedule anything and synchronize what we have there. And that has been running
Vegard: flawlessly on 120 antennas or something for three years now.
Vegard: I think I've had two bugs on it, and it's been purely logic bugs.
Vegard: The problem with bugs in that is that when there's a bug there,
Vegard: it affects everything. Because, yeah, nothing is happening on the antenna if the scheduler is down.
Vegard: Other than that, we also have data distribution and just pushing metrics from
Vegard: our basement equipment to the antennas. And everyone wants to consume those,
Vegard: your customers, system engineers,
Vegard: big part of our infrastructure is also just having the correct tooling on each
Vegard: antenna to be able to send out this infrastructure and we're using rust for that as well.
Matthias: It's incredible how far you are in your rust journey already i had no idea really. About
Matthias: the scheduler so what inputs does it take and what outputs does it generate.
Vegard: So it's running a in-house
Vegard: protocol to synchronize whatever schedule is available
Vegard: in the cloud and the cloud database is
Vegard: the source of truth and the schedulers
Vegard: on each antenna site is just figuring out what to
Vegard: synchronize from the cloud so i can operate autonomously in
Vegard: case of network failure without network connectivity
Vegard: we can still operate and take your contacts and yeah
Vegard: so it just synchronizes whatever it does
Vegard: there over a custom HTTP protocol and
Vegard: as the contact is about to begin it kicks off a event to a another service which
Vegard: we call the controller it's the controller of the entire contact just controls
Vegard: all the baseband equipment and whatever firewalls and whatnot needs to be opened
Vegard: and controlled. It's a just-in-time scheduler.
Matthias: And the reason why it doesn't pull everything is resource constraints again.
Vegard: Yeah and it also doesn't need to have the full state of the entire database
Vegard: because from clouds it only needs to know what do i need to do it would be very
Vegard: inefficient to synchronize the remote states from the cloud to every antenna
Vegard: that would not be feasible.
Matthias: But
Matthias: the calculation for knowing what it needs is that CPU bound or is the focus
Matthias: again on reliability here.
Vegard: Solely on reliability. So for scheduling or synchronizing,
Vegard: it's configurable for a scheduler, but usually it's deployed to one or three
Vegard: days ahead, so it can run a while until if we're,
Vegard: losing network connectivity and we can still salvage a lot of data even if we
Vegard: don't have connectivity to the cloud.
Matthias: Very impressive but that means the entire chain from the satellite all the way
Matthias: to the customer is at least in
Matthias: parts written in rust nowadays. What is your message to the Rust community?
Vegard: I think my primary message to the Rust community is just polish up async.
Vegard: Get it to be the best experience it can ever be.
Vegard: There are some pitfalls now, even though Rust 2024 edition stabilized the async closures.
Vegard: Very happy about that. But there are still some questions around observability
Vegard: of what is happening within an async context and how do you navigate that?
Vegard: And just, yeah, and getting to the bottom of issues related to,
Vegard: as I said, the blocking issues we have and just cancellation safety and drop
Vegard: safety and async drop and all these paper cuts that just are not completely answered.
Vegard: That would be my message to really polish up that. That would make selling Rust to others much easier.
Matthias: Yes, I could get behind this. Vegard, thanks so much for taking the time and for being a guest today.
Vegard: It's my pleasure. Thank you for having me.
Matthias: Rust in Production is a podcast by corrode. It is hosted by me,
Matthias: Matthias Endler, and produced by Simon Brüggen.
Matthias: For show notes, transcripts, and to learn more about how we can help your company
Matthias: make the most of Rust, visit corrode.dev.
Matthias: Thanks for listening to Rust in Production.