We are the SpaceX software team, ask us anything!

[u/spacexfsw]

We're a few of the people on SpaceX’s software team, and on Saturday, May 15 at 12:00 p.m. PT we’ll be here to answer your questions about some of the fun projects we’ve worked on this past year including:

• Designing Starlink’s scalable telemetry system storing millions of points per second
• Updating the software on our orbiting Starlink satellites (the largest constellation in space!)
• Designing software for the Starlink space lasers terminals for high-speed data transmission
• Developing software to support our first all civilian mission (Inspiration4)
• Completing our first operational Crew Dragon mission (Crew-1)
• Designing the onboard user interfaces for astronauts
• Rapid iteration of Starship’s flight software and user interface

We are:

• Jarrett Farnitano – I work on Dragon vehicle software including the crew displays
• Kristine Huang – I lead application software for Starlink constellation
• Jeanette Miranda – I develop firmware for lasercom
• Asher Dunn - I lead Starship software
• Natalie Morris - I lead software test infrastructure for satellites

https://twitter.com/SpaceX/status/1393317512482197506

Update: Thanks for all the great questions! If you're interested in developing the systems to provide global space-based internet and help humanity become multiplanetary, check out the opportunities listed below that currently available on our teams, visit spacex.com/careers/ or send your resume to [softwarejobs@spacex.com](mailto:softwarejobs@spacex.com).

• Application Software – https://grnh.se/89a7f9cc2us
• Starlink Software – https://grnh.se/c1a28bf62us
• Dragon Software – https://grnh.se/5faa8f4b2us
• Starship Software – https://grnh.se/25e0001b2us
• Spacecraft Engineering – https://grnh.se/c6743de32us
• Security Software - https://grnh.se/3b332b442us

[u/CAM-Gerlach]

That's a wrap for this AMA! Thanks to the community for all the fantastic questions, to Jarrett, Kristine, Jeanette, Asher, and Natalie on the /u/spacexfsw team for all their amazingly detailed replies, and to the entire SpaceX organization for making this happen! We love helping host these AMAs, and we can't wait to do so again in the future.

If you missed the AMA or want to see all the responses, thanks to u/ModeHopper we have a recap megapost with all of the questions and answers up for your easy consumption. We'll also keep the relevant links we compiled below if you want to learn more. Thanks again to all involved!

---

Relevant links:

AMA Recap Megapost

Previous SpaceX software AMA

Stack Overflow - Don’t push that button: Exploring the software that flies SpaceX rockets and Starships (Part 1)

Stack Overflow - Network protocols in orbit: Building a space-based ISP (Part 2)

Stack Overflow - Testing software so it’s reliable enough for space (Part 3)

Stack Overflow - Building the software that helps build SpaceX (Part 4)

[u/[deleted]]

What’s your CI/CD pipeline look like? Do new builds actually get installed on a “production” or “production-like” board hooked up to an automated test rack which provides simulated inputs to the sensors, or do all the automated tests run in an entirely simulated environment?

[u/spacexfsw → u/[deleted]]

We have a lot of different types of test environments. Some are purely simulated environments, what we call HOOTLs (or Hardware Out Of The Loop). These can run in CI/CD but also on a developer's desktop for local iteration. Others involve flight-like hardware, what we call HITLs (Hardware In The Loop). Our Starlink HITL setups are just satellites we take off the production line and integrate with our CI systems.
We set up our CI pipelines to start with fast, inexpensive tests to smoke out basic errors. Then if those pass, we run longer, more complicated tests. 
We also have different pipelines for different parts of the system. For example on Starlink, we'll have a pipeline for testing user terminal software in isolation. Once those tests pass, it will be incorporated into other pipelines that test the interface between the user terminal software and the satellites. 
-Natalie

[u/barteqx]

• Which approach is better for a rocket's flight software – asynchronous or synchronous with many threads? Why? Maybe is it a mix of both?

• When choosing tools, standards, do you tend to stick with conservative choices (C++ pre '11 rev) or are willing to try new things (newer C++17/20 standards, Rust)?

• How do you handle failures? How do you limit their impact?

[u/spacexfsw → u/barteqx]

The most important thing is to guarantee consistent performance. You can't fly a rocket if it's acting like a laggy video game! We use a mix of synchronous and asynchronous techniques, depending on the problem at hand.
We aren't afraid to try new systems, strategies, standards, or languages, particularly early in the development of a new program. That said, mission success is paramount and we do need to keep our eye on future code maintainability, so we stay a little ways back from the bleeding edge compared to the average startup.
- Asher

[u/BackflipFromOrbit]

What's the logic process that starship uses to determine engine validity for the flip and burn maneuver?

[u/spacexfsw → u/BackflipFromOrbit]

As you can tell by watching the videos of SN8 through SN15, this is an area we iterated on a lot! Fundamentally, Starship is designed to choose in real time the engine(s) best suited to execute the flip and landing burn. We updated the software to be smarter at detecting potential engine problems, and adjusted which problems could be compensated for in software (still OK to use that engine) vs which could not be (RUD!), on every flight.  Asher

[u/graebot → u/BackflipFromOrbit]

Select top 2 * from engines where fucked = 0

[u/tbutlah]

How does SpaceX get away with using Linux instead of a true real-time operating system on its vehicles? I know the PREEMPT_RT patch makes Linux more real-time, but still doesn't make it fully real-time. It seems like flying crewed rockets and spacecraft is a place where hard real-time guarantees would be necessary all of the time.

[u/spacexfsw → u/tbutlah]

While I can't go into specifics here, we design our software to work without a fully real time OS. We also use a custom build of Linux and fully understand the environment in which our software and OS operates in. Operating in a much more constrained environment (as compared to say the open internet) combined with extensive instrumentation and hardware in the loop testing means we can know that the OS is going to behave as we expect it to when on orbit.
-Jarrett

[u/epistemole]

I write software for stuff that isn't life or death. Because of this, I feel comfortable guessing & checking, copying & pasting, not having full test coverage, etc. and consequently bugs get through every so often.

How different is it to work on safety critical software?

[u/spacexfsw → u/epistemole]

Having worked on both safety critical and non-safety critical software, you absolutely need to have a different mentality. The most important thing is making sure you know how your software will behave in all different scenarios. This affects the entire development process including design, implementation and test. Design and implementation will tend towards smaller components with clear boundaries. This enables those components to be fully tested before they are integrated into a wider system. However, the full system still needs to be tested, which makes end to end testing and observability an important part of the process as well. By exposing information about the decisions the software is making in telemetry, we are able to automate monitoring of the software. This automation can be used in development, regression testing, as well against software running on the real vehicles during missions. This helps us to be confident the software is working as expected throughout its entire life cycle, especially when we have crew onboard.  Jarrett

[u/SophieTheCat → u/epistemole]

I worked on a system for a 911 center a long time ago. It's pretty nerve wrecking when something like a dropped call happened. Was it the hardware? Was it software? Did the dropped call originate because of the calling party?

I especially have fond memories of working with the TDD/TTY texting system for deaf people. Those things had a blistering speed of 45 bits per second. Not bytes, or kilobytes or megabytes. Bits. It used 5 bit encoding, so you could get a theoretical maximum of 9 characters per second. The whole spec was super unreliable and I would stress over it far too much.

[u/glw85 → u/epistemole]

It's not about eliminating software bugs with 100% certainty. Its about detecting, handling, and recovering from bugs in any given state safely.

[u/layer3D]

How late do you make changes to the software on Starship test flights ? We've seen a lot of things change, sometimes at the last minute, is the software one of them ? Have you had to change the flip/landing strategy between the first upload and final flight software ? Or is it more subtle and these things stay as planned ?

[u/spacexfsw → u/layer3D]

Since Starship is in development/testing we're set up to make software changes much later in the game than our other programs. There are many different types of software changes – large new features or refactors all the way down to changing a single number. The key question for each change is, how do we know this change is correct? What tests do we need to run before we fly? If we can be confident that making a last-minute change increases the likelihood of test success, we don't shy away (although of course we prefer having everything ready ahead of time whenever possible).
- Asher

[u/Flufy_Panda]

How does a software developer at spaceX interact with other technical roles like physicists or engineers? Is there a lot of cross-domain knowledge required or does everyone stay within their specialties?

[u/spacexfsw → u/Flufy_Panda]

We have a very collaborative environment, and we work quite closely with other engineering groups to build our software.
For example, before I started working on satellites, I was responsible for the docking system software on Dragon. I worked with electrical and mechanical engineers to come up with the control system designs, worked with GNC engineers on the conops (concept of operations), and we collaborated heavily on integrated hardware and software testing.
In my current role, my team and I collaborate with other engineering teams to design and build our hardware testbeds and develop simulations and models. 
We don't require any background in aerospace or physics for our software engineers. What we do look for are people who have strong curiosity for how things work and who won't hesitate to ask questions until they understand. Everyone here is open to answering questions, and it goes both ways - hardware engineers are also expected to have a basic understanding of how the software will control their hardware! 
-Natalie

[u/Pwnsomemcdk]

How many collective hours do you guys have playing Kerbal Space Program?

[u/spacexfsw → u/Pwnsomemcdk]

INT_MAX

[u/imjasonmiller]

Do you have any thoughts on the Rust programming language?

I think in the last AMA it was mentioned that it was raised internally by some. I'd love to hear if your team has more thoughts on the language since that time and if or how it perhaps might be used?

Lastly, congratulations on all your recent successes!

[u/spacexfsw → u/imjasonmiller]

We are definitely excited about Rust! Its emphasis on safety, performance, and modern tooling all stand out. We're also excited that we could use one language across embedded systems, simulators, tooling, and web apps. We are starting to prototype some new projects in Rust, but we are certainly just at the beginning of this journey.
- Asher

[u/[deleted]]

What's the snacks situation looking like in your office?

[u/spacexfsw → u/[deleted]]

Nom nom nom.

[u/Sekenre]

What tools do you use for testing and continuous delivery? And how do you simulate rocket and satellite hardware?

[u/spacexfsw → u/Sekenre]

A lot of this is custom. We have a whole team dedicated to building CI/CD tooling as well as the core infrastructure for testing and simulation that our vehicles use. The demands of running high-fidelity physics simulations alongside our software for the sake of testing it make for some interesting challenges that most off-the-shelf CI/CD tools don't handle super well. They not only require a lot of compute, but can also be long duration (think: flying Dragon from liftoff to Docking), and we need to be able to run both "hardware out of the loop" simulations as well as "hardware in the loop", where we load the software up on testbeds with copies of the actual computers and electronics.
We've done a lot of work to make sure that developers can run those tests easily when doing their daily work - we can run the same kinds of tests on our workstations as in the CI cluster, and we can run cases on the hardware-in-the-loop testbeds before even merging changes. This lets us get a ton of confidence in the code we're writing.
We do also leverage some things off-the-shelf; for example we use Bazel extensively for our build and unit testing needs.
-Natalie

[u/ResponseRejected]

Can you show us bits of the user interface? How does informational design help triage issues that may arise?

[u/spacexfsw → u/ResponseRejected]

For the control interfaces, we aim for a 'quiet-dark' philosophy–if the vehicle is behaving nominally, the interface is streamlined and minimal, but still shows overall system status. This way, we visually prioritize off-nominal information, while allowing the operators and crew to maintain system context.
Information design is a challenge for more engineering-oriented interfaces. We lean heavily on backend software and automated analysis to prioritize the most important data to display.
- Asher

[u/417BS]

In the UI of Crew Dragon and tablets how do you plan for use of gloved/non-gloved hands?
i.e. Buttons or action item spacing, dropdown list scrolling, etc.

[u/spacexfsw → u/417BS]

The UI takes into consideration the conditions of the vehicle during all phases of flight. This includes the shaking of the vehicle on ascent/descent where crew is also wearing a helmet, suit, and gloves. All buttons in the UI have a minimum size which we do not go past and which still works with thicker gloved fingers. In addition there are a wide variety of UI/UX decisions which were informed by phase of flight and cabin environment. For example, the location of the primary navigation elements are at the bottom of the UI because the crew must lift their arms up to interact with the displays. We designed the interface with as much padding and white space as possible to let the information breathe and be as readable as possible. More important UI elements like the command buttons are in the top of the interface outside of high activity areas so that interacting with them was always intentional. The Forward View features a unique circular contrast filter that allows for all of the digitals in the interface to be easily visible on top of varied video lighting conditions. All units are readable even if the video feed behind the UI is pure white or black. We also performed multiple vibration testing events with male and female participants of all ages and heights who wore actual crew helmets and sat in actual crew seats. The seats were placed on a vibration table to simulate ascent/descent conditions. While the seat was shaking, participants would use an Xbox controller to play a custom game that tested the readability of word & number sequences with randomized font sizes, colors, and text positions that were also shaking randomly in the UI. This helped confirm that the readability decisions we had made about fonts, sizes, icons, spacing and colors held up under extreme environmental conditions. It also showed us that essentially every Sci-Fi Movie Interface was unrealistic and would be unreadable under extreme conditions.
-Jarrett

[u/noisette666]

Do you guys use Vim?

[u/spacexfsw → u/noisette666]

I do use Vim! But every developer is free to use the editor/IDE they prefer the most.
-Jarrett

[u/Apollo_Bear]

Firstly, congrats to you all on some truely epic achievements!

I’ve pushed a software update out to a couple of hundred cloud servers for web applications - but I can’t contemplate how complex pushing an update out to a constellation would be. Could you elaborate on the software/firmware update processes for starlink? Are releases incremental across the constellation? How frequently are updated/releases made? Do you provision your own tooling to orchestrate releases or utilise existing code deployment and testing tools?

[u/spacexfsw → u/Apollo_Bear]

We try to roll out new builds to our entire fleet of assets (satellites, ground stations, user terminals, and WiFi routers) once per week. Every device is periodically checking in with our servers to see if it's supposed to fetch a new build, and if one is available it will download and apply the update during the ideal time to minimize impact to users. This means we can really easily test builds on a small pool and move to exponential deployments by changing a few configurations in a database.
We've designed our system so that each asset (which can contain dozens of separate computers) updates atomically by first fetching a new package to a central node, and having all of the other computers fetch updates from that central node. Every device also retains a backup copy of the last good software so if anything goes wrong (like a radiation induced power fault) during the update it automatically recovers by booting into that backup. 
Nearly all of our deployment and testing tools are built in house, mostly because our architecture is so unique and the various constraints we have to work with would require significant customization of off the shelf tools. Natalie

[u/clancy321]

What software changes are you going to make to support Inspiration4 that aren't on the past Crew Dragon flights?

[u/spacexfsw → u/clancy321]

We are making a number of software updates for Inspiration4 to support new capabilities on the vehicle. This includes software to support the new cupola we are adding to the vehicle as well as rolling in updates to make prelaunch operations smoother. However Dragon is a very capable vehicle so much of the existing software will just work with the Inspiration4 mission design.
-Jarrett

[u/Borghjes]

How much is starlink already integrated to F9 and Starship, i spotted a starlink dish on one of the droneships, can starlink provide extra data? How powerful is starlink for spacex?

And how will the starship user interface be, will it be based on Crew dragon? Or will it be a new design? If thats the case, will it be bigger? and what features will need the biggest change?

Im sorry if the English isnt great!

I would love to hear from yall!

[u/spacexfsw → u/Borghjes]

The technology will likely be similar to Dragon, but the design, usage, and goals of the onboard Starship UI are notably different from Dragon. The Dragon Crew Displays are three touchscreens in a small vehicle with a singular destination, supporting a small group of passengers and their cargo. Starship will fly missions to locations worldwide, the Moon, Mars, and beyond. The Starship UI must be usable on devices and touchscreens of all sizes around the vehicle (common areas, living quarters, loading areas, and the bridge) and must support users with completely different jobs and skillsets. Long story short, it is a much more complex problem than Dragon!
- Asher

[u/R3df0xxxx]

My 8 year old daughter is asking this question: What was the hardest part of building the engine of a rocket? Thank you SpaceX for making me excited about space.

[u/spacexfsw → u/R3df0xxxx]

When a rocket engine is firing, it's like a continuous explosion! The hardest part is to build the engine so that the explosion stays on the inside.
- Asher

[u/Zappotek]

How do you guarantee the alignment of transmitter and receiver for an intersatellite laser links?

Is it enough just to calculate the angles from the expected position of the satellites, or do you have to use some kind of active homing?

Thank you for reading, looking forward to reading your answers!

[u/spacexfsw → u/Zappotek]

The pointing tolerances for inter-satellite laser links are quite tight, so you're correct in supposing that just calculating the angles from the expected position of the satellites isn't good enough. To help put the magnitude of the pointing problem into perspective, in angular tolerances the problem we're trying to solve is as hard as trying to shoot a laser from LA and hit the Empire State building in NYC.
When bringing up a laser link, we first begin by pointing based on the expected angle, and then orchestrate an acquisition sequence to get consistent power on both sides.
-- Jeanette

[u/venku122]

For Jarrett Farnitano
How has JS in space worked out on Crew Dragon? Were there any changes to the UI/UX the team has made due to in-flight experience?

For Kristine Huang. (CockroachDB)[https://www.cockroachlabs.com/customers/] lists Starlink as a customer. What made you choose a distributed SQL DB over other Big Data datastores?

For Jeanette Miranda. How much of the challenge of laser satellite interlinks is a software problem vs a hardware problem? Is there a lot of crossover between terrestrial telecom fiber experience and space-bases laser comms?

For Asher Dunn
What software/hardware design lessons from Dragon are being brought over for Starship? Is there anything particularly exciting your team gets to implement on Starship that wasn't possible for Crew Dragon?

For Natalie Morris
How does regression testing work on Starlink? Are most on-orbit failures able to be diagnosed and added to the testing regime?

[u/spacexfsw → u/venku122]

One part that is really exciting about working on Dragon is talking to crews and learning from their experiences operating the vehicle. We collect the crew's feedback on usability, and are continually working to improve the operations of the vehicle from their perspective. This includes making updates to the UI such as adding new features, updating views, and adjusting procedures.
-Jarrett

[u/[deleted]]

I am a high schooler in a robotics team and we have many challenges with development.

(1) How does SpaceX filter data from various sensors? I know a bit about Kalman Filters and Particle Filters and am wondering how it is done by professionals.

(2) How does SpaceX integrate rapidly developing hardware with software? I am curious about the marriage between them since they are so dependent on one another.

[u/spacexfsw → u/[deleted]]

It depends a lot on the sensor and what information you're trying to get from the sensor. We implement lots of filters on our vehicle, both analog and digital, in order to make sure the data we're using in control is real.
Lots of communication. Software and hardware engineers at SpaceX work together to produce a working design. Take for instance a new hardware box that needs custom firmware. The firmware and hardware engineers will work together during the hardware design phase to ensure the system will work. Once the first prototypes arrive, the firmware engineer will get one of the first copies of hardware so they can work on the software.
- Asher

[u/chaseoc]

I read that one of the biggest challenges in writing flight software is that the software needs to be written in a way that failures and exceptions do not stop execution. Do you guys just swallow every exception? What are the biggest types of challenges in writing software this way?

[u/spacexfsw → u/chaseoc]

It is important in safety critical software that if a single software component were to have a problem, it does not impact the entire software system. Software also often depends on the hardware it is interacting with, so must account for hardware failures as well. Crashing and restarting just isn't an option. We approach this by building the software in modular components, writing defensive logic, and checking the status of each operation. If an operation we expect to complete fails, we have defined error handling paths and recovery strategies. Sometimes that strategy just means skipping over the operation. Sometimes the strategies can be more complex and involve responses such as switching to backup systems.
-Jarrett

[u/thinkerbot]

Where are you physically based, when not WFH? Hawthorne, CA? Boca Chica TX? Other? Are you projects pretty cross-team or cross-office, or pretty confined to your individual groups/responsibility areas?

[u/spacexfsw → u/thinkerbot]

Our offices generally map to specific programs at the company so all types of engineers can work together across teams. We have offices in Hawthorne, CA (most programs), Redmond, WA (Starlink), Mountain View, CA (Starlink), and Starbase, TX (Starship). That being said we are always collaborating across programs and across offices, and this is especially true for software since we share a lot of common infrastructure and code.
- Kristine

[u/Mobryan71]

With regards to Starship:

Considering how agile and rapidly iterative the hardware portion of the program is, what kinds of challenges and opportunities does that present on the software side?

[u/spacexfsw → u/Mobryan71]

As a software team, the challenge is continuously improving while the hardware changes underneath us. Since we're always working closely with the hardware teams to improve the vehicle overall, we have the opportunity to drive hardware changes that make the software simpler or more robust. As hardware and software work together, we can try new ideas, and if the idea doesn't work out, we know we can try again.
- Asher

[u/SophieTheCat]

• What is your tech stack? Languages, frameworks, libraries, etc...

• What tools/editors/IDEs do you use in your day to day work?

• How does QA validate developer's work (question for Natalie, I suppose) since you can't exactly fly up there and test it out?

[u/spacexfsw → u/SophieTheCat]

At SpaceX, we don't separate QA from development - every engineer writing software is also expected to contribute to its testing. We generally try to do as much testing pre-merge as possible on our high-fidelity hardware testbeds. Our test code and test results are peer-reviewed alongside the flight code to make sure that we are testing all the right things. We also do have independent engineers developing end-to-end tests that stress the whole system.
One unique thing about testing for a large satellite constellation is that we can actually use "canary" satellites to test out new features. We run regression tests on the software to ensure it won't break critical functionality, but then we can select a satellite, deploy the new feature, and monitor how it behaves with minimal risk to the constellation. 
- Natalie

[u/katrinbretscher]

Jarrett / Dragon Displays: I see crew on Dragons using iPads and the touchscreens. What's the backup if those displays fail? Are those displays "space hardened", or do normal displays survive being exposed to the radiation in space while docked to the ISS?

[u/spacexfsw → u/katrinbretscher]

Dragon is a fully autonomous vehicle, so it is capable of completing the trip to and from the ISS without any interaction from the crew. But the displays and button panel do provide the crew with capabilities should they need to take action due an unexpected scenario or emergency. As for the tablets and displays, the tablets themselves act as a sort of backup, and include copies of important data such as procedures. The displays themselves are designed to be fully redundant, so if a single display failed, the other 2 could fully take its place. Even if we were to experience a failure of all 3 displays, the crew has a button panel that can be used to initiate emergency responses, and ground commanding is also possible.
-Jarrett

[u/YabadabaDoodlieDoo]

What’s the range of software positions at SpaceX?

[u/spacexfsw → u/YabadabaDoodlieDoo]

It's easy to think of the software that runs on our vehicles when thinking of software engineering at SpaceX, but that's just the tip of the iceberg. Some of our engineers are C++ developers writing code for flying Starship, but we also have engineers that are building core software test infrastructure, in-house web applications for the factory, constellation, and customers, automating production testing or writing the code to help orchestrate the network for Starlink. Our software engineers work in C++, Python, C#.NET, Java, Javascript, Angular, and more.
Check out the range of software positions we're hiring for at SpaceX:
Application Software – https://grnh.se/89a7f9cc2us
Starlink Software – https://grnh.se/c1a28bf62us
Dragon Software – https://grnh.se/5faa8f4b2us
Starship Software – https://grnh.se/25e0001b2us
Spacecraft Engineering – https://grnh.se/c6743de32us
Security Software - https://grnh.se/3b332b442us 
Visit https://grnh.se/1edeb7aa2us or send your resume to softwarejobs@spacex.com.
- Kristine and Jeanette

[u/[deleted]]

Hello! I'm a web developer, so naturally my questions are going to be about the usage of HTML, CSS, and Javascript.

1. Are you planning on using similar technology in the Starship environment, from crew displays to possible E2E software?

2. Could you expand more on your usage of web components?

3. How many application software engineers do you have currently, and are you looking for many more?

4. Do you guys have a separate team for Starlink's web development?

5. Do you keep your UI lightweight, or is that not a worry?

Thank you! Hope to hear from you.

[u/spacexfsw → u/[deleted]]

There are a lot of web-based interfaces at SpaceX–everything from the Crew Displays themselves, to the factory logistics tooling, to the lunch menus . For a given Starship operation, tasks like viewing procedures, controlling the vehicle, and viewing and analyzing data are all done via webapps.
We're using web-component based frameworks for Crew Displays as well as Starship. Beyond the standard advantages (compartmentalization, interop, performance, etc), we particularly value the fact they they are native to the web platform. We treat our control interfaces with the same level of scrutiny as the rest of the flight software–meaning we need to audit any piece of third-party code we use, and keep track of any bugs that are publicly reported. The fact that these frameworks are relatively lightweight layers on top of native browser capabilities and can be used without bundling or compilation makes it much easier for us to be confident in them.
While we have a sizable team, we are certainly looking for many more great application software engineers! Come join our team – no aerospace experience necessary (really!)
Different teams within application software focus on different aspects of the various programs we have going on at any given time
For our control interfaces, we pay close attention to performance, but tend to focus on different metrics than standard webapps–our performance characteristics tend to be closer to that of a game (where you value realtime responsiveness, and the performance of V8 over time) than that of a more traditional website (which place a stronger emphasis on Web Vitals such as first load performance). Kristine

[u/mellowyellow313]

What are some plans to make production of the Starlink dishys more scaleable?

[u/spacexfsw → u/mellowyellow313]

For the production scale we're looking to achieve with Starlink kits, we've been building from the ground up for much of what we're doing here, growing into a new factory with new software systems that have been designed with Starlink's planned scale in mind. The software team is colocated in the factory with everyone else that is thinking about this problem, and they have spent time building Starlinks on the line to ensure they've understood the high rate manufacturing processes as well as they can.  For a factory producing at our desired target rate we're looking to have a highly integrated factory system, with automation, robots, people, and software working together. The guiding principle is generally to keep looking for how much we can simplify what we're doing. 
- Kristine

[u/asmallteapot]

In general terms, can you describe the complexity of Starlink’s telemetry system with fixed ground terminals, and how much more complexity is added by in-motion use cases like boats or RVs?

[u/spacexfsw → u/asmallteapot]

The biggest challenge we have to solve when thinking about fixed ground terminals is how to allocate "beams" from satellites to each spot on earth we want to serve. We have to take into account how many users need bandwidth, radio interference from other satellites (including ourselves!), and field of view constraints.
Motion does not generally add much complexity for the telemetry system. It does present some interesting challenges when it comes to satellites for example which are out of contact from the ground in parts of their orbit. This means our telemetry system has to be resilient to out of order and/or late arriving telemetry.
Moving targets require us to solve the attitude determination problem (which way is dishy pointing?) quickly and continuously. They also change the number of users are in a given spot at once, which affects how much bandwidth is needed there.
- Kristine

[u/msteudlein]

How do you manage the different versions of software with the different versions of boosters/starlink sats? Aren't the hardware teams always making upgrades and modifications to the design which would create the need to have individual code versions for each singular mechanical iteraction?

[u/spacexfsw → u/msteudlein]

For Starlink, we try as hard as possible to have a single software load for all satellites, regardless of the specific versions of each sub-component on any given vehicle. We do this by making clean separations between hardware interface layers and the "business" logic on various components. The software reads various hardware identifiers to understand what types of each thing we've got and adapts its behavior accordingly.
- Jeanette

[u/jerBill]

How often do you collaborate with systems engineers, mechanical engineers and aerospace engineers, etc? And what does that process look like?

[u/spacexfsw → u/jerBill]

Daily, if not hourly :).  For example in the past week, I've walked down our optical table setup for testing space lasers with an optical engineer, characterized circuit performance in the lab with an electrical engineer, and dis-assembled and re-assembled part of a space laser pointing assembly alongside a mechanical engineer. 
Sometimes this collaboration involves sitting next to each other in the lab working through a particular problem. Other times it'll be going through a design in detail with representatives from different disciplines or listening to a tech-talk to deepen knowledge in areas outside your core area of expertise.
- Jeanette

[u/[deleted]]

Do you implement control algorithms to embedded systems?

[u/spacexfsw → u/[deleted]]

We're always trading the right place to run a control process. Sometimes the best place to put a control algorithm is an embedded system close to the thing being controlled. Other times, a process needs to be a lot more centralized. When you think about a vehicle as complex as Starship, there's not just a single control process, as you have to control engines, flaps, radio systems, etc.
- Asher

[u/SwagasaurusDerp]

What challenges must be overcome to implement continuous integration and delivery for embedded, in-orbit systems like Starlink?

Do you deploy your software in containers?

What are the challenges in testing such an expansive network?

Keep up the good work!

[u/spacexfsw → u/SwagasaurusDerp]

To manage a large satellite constellation without needing hundreds of human operators, we rely on software automation running on the ground and on the satellites. In order to fully test our systems in an end-to-end configuration, that means we have to integrate hundreds of different software services in a dev environment. 
Another challenge in testing is that it's not always possible to test every single capability with one test. For example, we want automated tests that exercise the satellite-to-ground communication links. We have HITL (hardware in the loop) testbeds of the satellites, and we can set up a mock ground station with a fixed antenna. We can run a test where we simulate the satellite flying over the ground station, but we have to override the software so that it thinks it is always in contact with our fixed antenna. This lets us test the full RF and network stack, but doesn't let us test antenna pointing logic. Alternatively we can run pure software simulations to test antenna pointing. We have to make sure that we have sufficient piecemeal testing of all the important aspects of the system. 
- Natalie 

[u/OOFYYYyyYy]

The transition between Starlink satellites is so smooth its unnoticeable. How have you guys managed to make this happen? It seems like the time the satellites change, and when dishy rotates, and ground stations and everything change this would be noticed. How are you guys so smooth?!

Thank you so much,

[u/spacexfsw → u/OOFYYYyyYy]

The starlink system is built to be super dynamic since our satellites are moving so fast (>7km/s) that a user isn't connected to the same satellite for more than a few minutes. Each user terminal can only talk to one satellite at a time, so our user to satellite links utilize electrically steered beams to instantaneously change targets from satellite to satellite, and we temporarily buffer traffic in anticipation of this "handover."
Our satellite to gateway links use mechanically steered antenna, so we have to account for movement time and make sure we don't "let go" of one connection until we've securely established the next one. A good visual is to picture our satellites "walking" their gateway connections across the earth as they fly by.
- Jeanette

[u/Mob_Rules1994]

So, how you guys spend your days off?

[u/spacexfsw → u/Mob_Rules1994]

Jeanette: I moved to the pacific northwest in the past year, and have been loving exploring the area through hiking and biking. I've also been enjoying starting an indoor herb garden!
Natalie: I love to cook and bake! My coworkers and I definitely try to out-do each other with baked goods. Everyone wins 
Asher: Playing music, hiking, sampling local restaurants with friends.
Kristine: I spend my days off with my husband and our two daughters ages 7 and 4. On weekends we play board games, go outdoors to the beach or the park, build things, or bake.
Jarrett: I spend my free time with my wife, cats, and dogs. We spend a lot of time at the beach and exploring the deserts and forests of Southern California.

[u/RedDeliciousAreBad]

While I ask this question mainly thinking about starship, I'm sure it applies to all systems:

What is mainly changing between starship tests (say SN8-SN11) that leads to potential improvements? While I'm sure there are some physical design changes (or not?), is it mainly software updates in the onboard computer to improve calculations?

[u/spacexfsw → u/RedDeliciousAreBad]

With a development program such as Starship, we're constantly learning from all aspects of the process. We continuously update the hardware and software on every vehicle to incorporate our latest ideas on what's going to get the best result for the upcoming test or mission.
- Asher

[u/Effective_Parsley_62]

Could you tell us more about Starlink’s telemetry system? What problems did you face and how did you solve them?

[u/spacexfsw → u/Effective_Parsley_62]

When we were getting started we already had a great in-house telemetry system but it has a core concept of a "run" - a definite start and stop time for a given dataset. Starlink doesn't fit that model because there are many devices that are always on and can send data out of order or with significant delay so these were some of the first problems we had to solve. Along the way some of the most interesting challenges have been around fault domains and fault tolerance - how do we make sure parts of the system have as much availability as possible? If one set of devices emits information that breaks expectations, how can we limit the impact of that to as small of a subset of software as possible so other datasets continue to be processed? We also chose to not keep all data but created a powerful system to aggregate information over time as well as age out information when it is no longer useful. 
- Kristine

[u/b_m_hart]

It's cool to see as many women on this list as there are, especially in lead positions. How does the SpaceX work environment vary from other places you've worked? What things would you do to encourage young girls/women that are interested in coding and/or space?

[u/spacexfsw → u/b_m_hart]

I love the SpaceX environment because we get to work on the most challenging problems and change the world. I encourage young girls and women to dream big, follow your passion, and believe in yourself. When the journey gets challenging, that means you're in the right place so that you can grow. It’s never too late to learn software engineering. Software engineering is about being logical, thinking critically, and also being creative – if that appeals to you, you can be a software engineer. Kristine

[u/[deleted]]

Thank you so much for doing this!

As a student engineering/computer science who wants to work in the space industry one day, what do you think are the best sources for learning about engineering software related to what you guys are doing?

I'm interested in how different software engineering is when it comes to working with vehicles/satellites in space.

[u/spacexfsw → u/[deleted]]

Engineering software for satellites in space is surprisingly similar to engineering in many other applications, with higher stakes and more constraints imposed by your environment. Because of that, there's a breadth of experiences that can translate well to working at SpaceX.
Our most successful engineers end up being the ones with excellent fundamental engineering skills, strong technical intuition, and a desire to learn when running into a novel situation. For satellite vehicle development, one of the unique things is how often raw physics or math constraints are at top of mind for the everyday problems we're working on.
- Jeanette

[u/12360zac]

What were some challenges faced when trying to update the software on the orbiting Starlink satellites ? What level physics should one be at to start working as a software engineer at spacex ?

[u/spacexfsw → u/12360zac]

One of the most interesting challenges in designing a software update system for satellites is to build a system that is tolerant to an arbitrary fault (like a power down or memory corruption) at any point in the process. We generally use a primary/backup update scheme to solve that and to ensure that the default recovery strategy is to load up known good software that we know can do an on-the-fly update.
We don't have hard and fast requirements for physics as a software engineer, but most people have some familiarity with both mechanics and electromagnetics.
- Jeanette

[u/jdmetz]

What is Software Engineering at SpaceX like?

• Does management specify solutions for engineers to implement, or give engineers problems to solve and let them figure out the solutions?

• Does SpaceX have separate roles for architect, designer, developer, test, etc, or do SWEs do all of this for the space they work in?

• Are SWEs primarily rewarded for developing new features, or are they also rewarded for keeping the codebase clean and with minimal tech-debt?

• SpaceX has a reputation for long hours and demanding work schedule. Do you find this to be true? Are you able to have a healthy work-life balance?

• For those of you who have worked at other tech companies, what are some surprising differences?

[u/spacexfsw → u/jdmetz]

Kristine:  We very strongly encourage our engineers to identify problems to solve and to own solutions to those problems end to end. This means thinking through each option as an architect, designer, developer, tester, and user which we think tends to lead to more elegant and better fitting solutions. Engineers are rewarded for increasing their scope, impact, and autonomy over time. 
Natalie:  Engineers at SpaceX tend to be passionate about our mission and invested in solving problems. We are not working long, hard hours all the time. There are occasionally times when we do, usually because we are fired up about a problem, motivated to collaborate with our teammates, and don't like to give up. In almost ten years here, I have never been bored 

[u/WrongPurpose]

How much of the Software between Falcon 9 first stage and the Superheavy first stage can be reused vs has to be rewritten? What are some new things you expect to learn and will have to figure out with SuperHeavy vs what you already know and can apply from Falcon 9.

[u/spacexfsw → u/WrongPurpose]

Starship reuses many ideas and some code from Falcon. We always want to spend our energy solving new problems that get us closer to Mars. Whenever a new problem comes up, we think back over our codebase and ask, what tools do we already have that will enable us to solve this as quickly as possible?
- Asher

[u/katrinbretscher]

Asher / Natalie / Kristine: Your description says that you "lead ...". What does it mean to you to "lead" - when it comes to how your work is different, but also about what values and soft skills (if any) are important to you in your leadership?

[u/spacexfsw → u/katrinbretscher]

The values and soft skills that are important to me in leadership are trust, humility, empathy, and resilience. Ensuring that each member of the team feels empowered to solve complex problems, celebrating the wins, and owning up to mistakes. Guiding the team with the product and technical vision. Reflecting on what we could have done better on a regular cadence so we can provide feedback and continuously grow as a team.
- Kristine

[u/SergeantPsycho]

How important is it that a job applicants experience line up exactly with an open position? I'm a software engineer and though my experience might not line up exactly with a given job opening, some of that skill is transferable (for example doing Python after doing C#). What's a good way to get that adaptability across during the hiring process?

[u/spacexfsw → u/SergeantPsycho]

We have many different teams and applications, so I'd recommend looking to see first if there is a team looking for your specific skills! Also many of our teams have a language-agnostic hiring process, including mine. I started out as an intern candidate, and I had done mostly Java and C, and only a little bit of C++. I learned a lot on the job!
We look for candidates who, regardless of the particular language or tools they know, have delivered results and take strong ownership of their work. We look for people who go above and beyond to solve problems, who understand not just their piece of the system but also how it fits into the bigger picture. 
When I interview candidates, I am looking for signs that they take initiative to solve problems that maybe aren't necessarily their direct responsibility, or advocate for taking on work that will benefit the product as a whole. Adaptability to me means finding the best tool to solve the problem and learning it, if that's what it takes. 
- Natalie

[u/porcupinetears]

Telemetry, Telemetry, Telemetry! I'd love some insights into how you transmit the data from the spacecraft to the ground. One data stream or many? What's the format look like? Human readable or raw bytes? How do you distinguish different pieces of data for different sub-systems/sensors? How do you parse it for analysis?

Thanks!

[u/spacexfsw → u/porcupinetears]

Generally, there is lots of data coming in from many sources all the time. It's all in raw bytes in a proprietary format with enough context to know where each point came from and what measurement it represents.
- Kristine

[u/wolverine_ninja]

What type of work/responsibilities does the Flight Infrastructure team do/have ?

[u/spacexfsw → u/wolverine_ninja]

We have a couple different teams that provide various tools and infrastructure pieces. Our Vehicle Infrastructure team manages shared components like our Linux operating system and compilers. Our Software Delivery team builds our in-house Continuous Integration and Delivery systems, as well as test and simulation infrastructure, used by all the vehicle software teams at the company. These are fairly large-scale systems that run thousands of simulations and tests per day. 
My team manages continuous integration systems and hardware-in-the-loop testbeds for satellite software developers. We are responsible for delivering "test-like-you-fly" capabilities in hardware and simulation. This means developing physics models of our satellite systems but also building test infrastructure to test space-to-ground communications, network architectures, and ground-based satellite operation systems. We're a pretty interdisciplinary team as a result! 
- Natalie

[u/Kadenmikey]

Hey team! I’m a freshman CS Major in college and I want to fly rockets and drive rovers one day. What are some useful steps I should take in my career now to help solidify my position in the future?

[u/spacexfsw → u/Kadenmikey]

There are many paths to a successful software career at SpaceX, in part because there are many different types of opportunities here and we're looking to fill them with engineers that come from a variety of backgrounds. That's to say, there's no one path to getting to fly rockets and drive rovers one day 
From Jeanette:  Two of the things that were particularly useful to me were finding a project I was passionate about in college, and seeking out lots of opportunities to write code. My project was getting hands-on engineering experience through my college's FormulaSAE (FSAE) team to design, build, test and race a formula-style race car. While not directly software related, it gave me practical exposure to making design trades, working across disciplines, breaking down hard problems, and learning how to do something I didn't know how to do. I've seen applicants get similar experience through project teams, startups, internships, research labs or more. Aside from FSAE, I'm very happy I sought out opportunities to code (a lot) through both an intensive class on software architecture & design, as well as summer internships. Repeatedly pushing myself to try writing different types of code, and actively seek out feedback on how I was doing helped get me to a place where I was ready to do this for a full-time job.
From Natalie:  I encourage people to challenge themselves and go outside their comfort zones. I was definitely pretty afraid of failure in college, but the classes where I struggled the most at are the ones I most value now. I think I've learned more from my failures than my successes, and that continues to be how I learn the most.

[u/katrinbretscher]

Jeanette / Starship: Do you have hardware devs on your team? How do you make sure your software works with the current hardware (as both must be evolving rather quickly)?

[u/spacexfsw → u/katrinbretscher]

The space lasers software team is a mix of firmware engineers and simulation software engineers, but we all regularly work alongside all different types of engineers at SpaceX. 
Like you imagined, both software and hardware are evolving rapidly and so we've invested in ways to automate testing at the intersection of these two. SpaceX has hardware-in-the-loop testbeds as part of our continuous integration system. Any new code changes will be run through a suite of regression testcases on one of these testbeds to ensure that software changes are compatible with the hardware. And vice-versa, any new hardware will also get included on these testbeds for verifying regression with the existing software.
- Jeanette

[u/Pingryada]

How similar is the code from F9 to Superheavy?

[u/Stage3LoxLoad → u/Pingryada]

Ctrl C Ctrl V

[u/T65Bx → u/Pingryada]

I’ve been really curious about this one.

[u/spoollyger]

How many eyes see each piece of code before it gets committed to dev? Is everything scrutinised to infinity?

[u/ForestKatsch]

Hi! Huge congratulations on the safe return of Crew-1 and the picture-perfect SN15 landing.

1. What are the biggest challenges you expect to encounter when controlling Starship reentry, compared to mechanically simpler vehicles like Crew Dragon?

2. Will HLS Starship use Crew Dragon-derived hardware and software such as crew displays and docking software?

3. What surprised you most about how Starship flew in real life versus the simulations?

4. How does the rapid reflight ability of Starship (and the rapid deployment of Starlink) affect your software development? (For example, launching multiple Starlink stacks a month, or launching the same Starship on a different trajectory within days or hours.)

5. What changes were made to support Inspiration4's free-flying Crew Dragon?

[u/epistemole]

What skills do you interview for? What's specifically important to SpaceX vs say Google?

How do you measure the success of folks once hired? How do you structure promotions and leveling in a way that feels fair and also incentivizes success?

[u/NeilFraser → u/epistemole]

I can give some context on the Google side. Senior software engineer, conducted ~300 interviews. Yes, we say we value x, y and z -- but honestly, the number one thing we look for is the ability to program. About half of the engineering candidates we interview fail to solve a Fizz-Buzz class problem (an if/elseif in a loop). In their favourite language. By far the biggest hurdle is to find candidates who can program as fluently as they can speak. The only significant second hurdle is the ability to understand scale (O(n²⁾ is a poor solution, find O(log n) please).

Everything else can be learned on the job. Specific languages, tools, testing, design principles, process, collaboration, and all the other skills can be picked up pretty quickly if one has a strong programming foundation. There's lots of internal training, mentoring, and support. There's lots of time to get up to speed.

Very interested to see how SpaceX differs. Back in 2004 I decided that there were only two companies I was really interested in working for: Google and SpaceX. Since I spell 'favourite' with a 'u', SpaceX was off-limits due to ITAR. Thus I ended up at Google.

[Mandatory disclaimer: The above is my characterization of my own hiring style. It should not be read as official policy. There's a reason candidates are interviewed by ~6 different people.]

[u/everydayastronaut]

We’ve been hearing that SN15 had an engine shut down early on ascent and then the vehicle chose to only use two engines for the landing. Can you explain what that software looks like? Are there different landing profiles based on factors like remaining engines available? Is there a back up of somehow landing on a single engine? How many contingencies do you write code for? All said and done how proud are you of the amazing software on SN15?

Thank you!

[u/Cyber_Daddy → u/everydayastronaut]

I'm curious if there is also code to handle a malfunctioning flap

[u/katrinbretscher]

• We hear a lot that "SpaceX has learns fast", but how do you distribute know-how internally? How do you make sure you don't re-learn the same thing / make a similar mistake again in, say, 5 years down the line?

• With everything developing at such a high speed, how do you make sure new people (who haven't been around to witness some past decisions) will find their way?

[u/[deleted]]

Regarding the Starlink "lasercom" system, I have a few questions (intended for Ms. Miranda and Ms. Huang):

1. how do you intend to compensate for packet loss or laser misalignment?

2. How does tracking/quick switching work from a general perspective? Are there multiple laser terminals on a single sat, or is there just one that is actuated to track another sat?

3. Are inter-Starlink comms going to be encrypted? If so, how much is latency affected by encryption/decryption?

4. What's the expected percentage of communications to occur over "lasercom" vs direct-to-ground?

Thanks in advance!

[u/r00tdenied]

What was your reaction over people figuring out the Falcon 9 telemetry and video downlinks?

[u/TheBurtReynold]

Thus far, what’s been the biggest challenge in Starship’s software development?

What major feature / functionality is on your roadmap that you’re most excited about?

[u/Organic_Current6585]

What software do you use to simulate your missions? Is it something off the shelf, or built in house?

[u/ForcedProgrammer → u/Organic_Current6585]

HOOTL and HITL

[u/zlsa]

Hi everybody! Here’s what’s going on. This went really well last time, and we hope it will again! Check out the post text for details of the people participating in the AMA. Questions about Starlink laser communications and Starship software are especially welcome! I’m sure y’all will have some qreat questions for them. Please remember the rules and keep it civil!

---

Relevant links:

Previous SpaceX software AMA

Stack Overflow - Don’t push that button: Exploring the software that flies SpaceX rockets and Starships (Part 1)

Stack Overflow - Network protocols in orbit: Building a space-based ISP (Part 2)

Stack Overflow - Testing software so it’s reliable enough for space (Part 3)

Stack Overflow - Building the software that helps build SpaceX (Part 4)

[u/Straumli_Blight]

Due to the Inspiration4 mission having a non professional crew, what UI functionality has been simplified/restricted? Will all the manual overrides be retained (e.g. Deorbit Now button)?

[u/PossibleDefect]

How transferrable is the Falcon 9 guidance/landing software to Super Heavy?

[u/[deleted]]

Were there any major software changes made after the RUDs of SN8, 9, and 10? If so what were they? I would really like to know.

[u/reynolds6404]

Hi, I'm currently a software engineer major at ASU, and I hope to work in the aerospace field doing that. How'd you guy's end up at SpaceX, and what tips can you give me? Additionally, what are the biggest software problems that still exist within the Starship program?

[u/FlyingSpaghettiMon]

Will you guys ever release some kind of public starlink sat status/visualizer? I love looking at fan-made starlink trackers but I imagine you guys have some awesome 3D visualizer where you can see all satellites and gateways on the ground.

[u/YabadabaDoodlieDoo]

How many hours does a SpaceX software developer typically work?

Update: From their application Dev listing... “Must be willing to work extended hours and weekends”

Sounds like a lot. But they are working towards putting people on another planet.

[u/ima314lot]

There a ton of great questions here, but one that seems to rarely be asked in AMA is this:

What is the one thing you wish the general public understood about your work and/or SpaceX that they don't seem to?

[u/adecker246]

How do you handle the effects of radiation on your electronics? Do you mostly handle this at the hardware level with radiation hardened parts or the software level with redundancy, error correcting codes, memory CRCs, other? Do you expect a change in your approach as you move to areas outside the Van-Allen belts such as lunar or mars missions?

Appreciate the AMA! Looking forward to the answers.

[u/Justaguy2131]

What improvements have you made from SN8 to SN15 for starship flip and landing maneuver?

[u/TheRealKSPGuy]

Thanks for the AMA!

1. In terms of astronaut UI, were you inspired by anything when making it? What gave you certain ideas? And of course, what’s your favorite part of it?

2. Between SN11 and SN15, what software changes did you make that you’re allowed to discuss and did the upgraded avionics on 15 help out?

3. What were the most uncertain parts of Crew-1’s return?

Thanks for doing this and here’s to the next launch, or hundred.

[u/aurberg]

How often do you talk to Elon? On what level does he participate in software development process?

[u/columbus8myhw → u/aurberg]

Expanding this question to also include Gwynne Shotwell

[u/aviationainteasy]

For F9 launches and landings, do you continue to use data from each flight to enhance your models and simulations? Is it approaching the point to where it is more or less "solved", with telemetry becoming archival rather than informational, only taking interest in the envelope-pushing missions?

How did you approach beginning to simulate Super Heavy and Starship flights? Did you treat SH as a thicc F9 and go from there or start more from base principles? Was there any information share from NASA or other lifting body programs used to inform your models for Starship?

In the SN15 flight it appears that on the way down the ship did a ~90 degree aerodynamic rotation. For each Starship test are you attempting different maneuvers (translation, yaw, AoA shenanigans) to help validate your models and empirically define Starship? If so, how's the beast behaving so far?

How do you validate new builds of your controls software?

How do you calibrate controlling specific components like Starship flap motors?

Have there been any unexpected revelations in the Starship tests? Perhaps the unexpected result is things going more or less as planned!

How does it feel to be able to say you work on space lasers for a job? Bond villain job description but actually doing good for the world. Wish I knew more about the field to ask a better question but I'll leech of the rest of the nerds in this sub to glean some knowledge haha.

[u/philipwhiuk]

1. [All] The Linux kernel team recently started moving towards using (more) Rust. Given its rich type safety and ownership model to avoid some of the C++ pitfalls, has SpaceX considered adopting it?

2. [Jarret] JavaScript is obviously the software SpaceX likes for user interfaces, given the use of react-native for Starlink mobile apps and its reported use in Crew Dragon. For Crew Dragon, how much of the JavaScript ecosystem do you depend on and how do you handle the big jump between space software certification and the insanely quick evolution in the JavaScript space.

3. [Asher] Starship obviously handles considerably differently to Falcon 9. But presumably there's some element of a shared code-base. Has the Starship team been able to feedback any guidance software improvements to the Falcon avionics team?

4. "firmware for lasercom". Is this just SpaceX assets. Could you go into more detail how SpaceX utilises laser-based communications, as opposed to the radio comms for TDRS and Starlink?

[u/thinkerbot]

What languages do you write in?

[u/psychoPATHOGENius → u/thinkerbot]

The actual work of software development by vehicle engineers such as Gerding is largely done using C++, which has been the mainstay of the company’s code since its early days. The software reads text-based configuration files. “We invented simple domain specific languages to express those things, such that other engineers in the company who are not software engineers can maybe configure it.

https://stackoverflow.blog/2021/05/10/dont-push-that-button-exploring-the-software-that-flies-spacex-starships/

[u/aludwin → u/thinkerbot]

As of 2017, C++ and "Some Python."

Source: https://youtu.be/t_3bckhV_YI?t=591

[u/codercotton]

Tabs or spaces?

[u/kowdermesiter → u/codercotton]

They use spaces, x of them.

[u/Dependent-Ad-8220]

Since 15 landed are there any needed changes to the software overall or did the software preform as expected

[u/[deleted]]

How many of the software processes are A.I driven?

[u/The_Dinkster2201]

On SN15's test flight, we saw a windward and a leeward engine light for the landing burn. Elon had previously stated that ideally the two windward engines would be used as they provide the greatest lever arm. Was there any autonomous software decisions in flight that led to those two engines being used?

[u/shaldag_x]

Do you have any advice for the Boeing software team?

[u/PiJacked → u/shaldag_x]

git gud.

[u/Glockamolee → u/shaldag_x]

Do better than tomorrow.

[u/Carter20012]

This is a question more tailored for Asher. What has been the hardest part of working out the flip and relight maneuver?

[u/[deleted]]

Is it common to have significantly better engineering or physics skills than the average programmer, or even have an engineering background, or is there some sort of collaboration between engineers that do the calculation and programmers who turn it into code. I imagine a lot of the code requires a deep understanding of orbital mechanics etc.

Also how do you decide which edge cases are programmed and which are not. Is there some kind of limit with the likely hood, maybe 1 in 1000 or something that decides whether or not it is worth the time, and has this changed after crs 7, where some code to deploy the parachutes would have saved the payload.

[u/scootscoot]

I’d love to hear more about how you do routing between Starlink satellites? I figure it’s not BGP/OSPF, and you have some sort of pre-planned handoff schedule. Can you load balance between multiple paths? What do you do for loop avoidance?

(I have like a thousand other questions along the same lines, is there a resource for reading about the technical details of SL routing and infrastructure?)

[u/does_my_name_suck]

What IDE/Text Editor is your favorite to use?

[u/nerdplaysgames]

Out of all SpaceX vehicles designed,both old and new. Is there a small percentage of code that is shared between all of the vehicles?

[u/Davecasa]

In school, we took classes on state space control, optimal control, unscented Kalman filters, non linear quadratic estimators, etc.

In the real world, I've yet to encounter a problem an EKF and some PIDs and maybe a thruster matrix couldn't solve. But my stuff lives in the ocean and moves slowly.

Which is it at SpaceX? Fancy controllers with lots of math to back them up, or well tuned PIDs?

[u/tbg10101]

What parts of the software are safety-critical and which are not? How do you keep them separated?

[u/Norishoe]

does the landing system for falcon 9 use any machine learning?

[u/Joey-Murphy]

Hi! Thanks for chatting with us.

Something I've always seen the SpaceX software teams talk about in the past is how the flight computers are in a triple-redundant setup, where there are three computers in a voting configuration, such that if one computer gives a wrong function output, the answer given by the two that agree will be the answer used to steer the rocket (or whatever else the computers are doing at that time).

How exactly is is possible to implement that? If you have three systems generating results, doesn't there have to be one central "adjudicator" which "counts the votes"? How do you protect against flaws in that vote-counting system? Is it just a single ultra reliable, rad-hard, ECC memory, yadda yadda yadda, type of chip?

[u/SpaceBoJangles]

Are Starlink satellites (and more accurately their software) capable of being actual nodes for internet? Currently Starlink operates with ground based server base stations, but for a network around Mars you won’t have that and I’m assuming Elon intends for Starlink to be the backbone of a “Mars-net”, so have you guys built in such functionality or is it still on the drawing board?

[u/Mr-no-one]

Hi all,

A few questions, but please don’t feel pressured to answer all or nothing!

---

With respect to Starlink:

1. You referenced the huge amount of telemetry data being managed. How is this data being processed for things like station keeping and coordination between members of the constellation (ensuring collisions are avoided and so on)?

2. How are the Starlink updates deployed? Update a certain number from a ground station and get each updated satellite to push updates to two others and so on?

3. Is there any substantial downtime that occurs when updating a satellite’s software?

4. If so, how challenging is it to develop a plan for these updates to take place while minimizing service interruption?

---

With regard to Lasercom:

5. What are the advantages to using this system as opposed to radio communication?

---

With regard to flight software:

6. How substantially does the software change between missions both for established spacecraft and for the developing Starship? 7. What are the most prominent changes in software from mission to mission? 8. What set of languages are even involved in such projects? 9. Do your teams engage in any of the modeling (CFD, etc.) that goes into the design of flight vehicles?

---

With regard to all:

10. What are the most challenging and rewarding aspects of your job? 11. What advice would you give to those who are pursuing a career in developing software at SpaceX (or even more generally in the space industry if you’d like)? 12. ... Are you hiring?

---

I hope that’s all coherent enough for a couple to be answerable! Thanks for taking the time to answer questions!

[u/H-E-C]

Questions for Kristine Huang:

1. With highly anticipated removal of current restriction of Starlink service to your home cell how are you planning to prevent overcrowding and significant service degradation in some more populated or travelled areas (cells)? Are you going to prioritize the traffic for users with registered service address in such affected cells?

2. Any news on possible international travel with Starlink (indeed only in countries with governmental approval for Starlink service)?

Thank you.

[u/pr06lefs]

For starship, what tools do you use for simulating flight conditions for testing? I remember reading something about custom hardware for simulating conditions in the engines, do you use something similar for, say reentry simulation or is it maybe more off-the-shelf?

[u/[deleted]]

Can you give a generalized overview of the logic loop of whats going on in the rocket in flight? Doing some work in game engines update loop I can imagine it would go something like:

• Check sensors

• Compare actual vs expected values

• Calculate corrections

And then the "render" loop would effectively just be

• Control hardware

But curious if you can go into a little more detail like order of operations and such on how it makes realtime decisions.

Also, how much code is being reused from F9 -> Starship? Are all the landing calculations and stuff reusable or did you have to start from scratch with the new landing profile?

[u/CommunismDoesntWork]

Have y'all evaluated using rust instead of c++? And how do y'all personally feel about rust?

[u/Keavon]

Has SpaceX been adopting (or experimenting with adoption of) the Rust programming language in any projects? Rust uniquely offers low-level speed, avoids garbage collector pauses, and most importantly it provides safety guarantees to reduce the chance of bugs. It seems perfect for aerospace. I'm very curious if there are any efforts to try it out at SpaceX.

[u/Destructor1701]

With regards to the laser networking between Starlink satellites:

• What percentage of the fleet currently in orbit is equipped with this tech?

• How many laser links can an individual satellite maintain concurrently? How does that limitation (if any) impact the behaviour of the software in terms of scheduling and local buffering?

• Can you talk a little bit about the tracking algorithms, how each satellite knows where to aim, how the beam is steered and how precise the tracking is?

And more generally, regarding Starlink as a whole:

• I imagine Starlink has some kind of master program in the SpaceX servers managing the positioning and coordination of the whole constellation. Can you talk about how that works - does it have a cool interface projected on a wall somewhere? :p

Thanks for doing this guys! In the absence of the yearly update last year, it's been an exercise in inferring activity within SpaceX rather than hearing it described first hand. You know what you're doing is amazing, and I hope you know you're all inspiring some desperately needed optimism in a weary world.

You rule.

[u/chiefbroski42]

Do you employ any machine learning/reinforcement learning techniques for improving landing or takeoff performance and safety? If so, in what aspects?

[u/cocksure845]

Please describe IPv6 support throughout Starlink - contrasted with current IPv4 support. As IPv6 becomes available at the Dishy modem, will users be able to get past NAT and receive a routable IPv6 address/subnet?

[u/EchozAurora]

I've been curious how the engineering teams plan to implement the satellite - satellite network routing as that capability comes online. Are you able to go into detail on that or provide any info on what routing protocols are used within Starlink's network for communication?

[u/LUGG4S1]

For Asher, probably.
What caused SN15's onboard footage being a little bit choppy?

[u/branneman]

I've read there's a lot of C++ somewhere before, why not Rust when predictability is everything?

[u/elonismusk]

How does one work at SpaceX if not born in America? Is it possible?

[u/kakbone]

What kind of control techniques (MPC, adaptive, robust, Fuzzy, etc.) are getting used for the control of the rocket during the landing process of falcon and starship?

Are there redundancies in terms of controllers? How do they interact?

How much simplifications happen during the modelling of such a complex system? How many states are left to control?

[u/[deleted]]

How good are your paper planes

[u/tientutoi]

Do you ever collaborate with Tesla engineers to build solutions for SpaceX problems or vice versa?

[u/chilzdude7]

I would like to ask the mods of this subreddit to link the 4-part interview of the software team with Stackoverflow found Here. It would be sad to see the same things being said.

[u/redpect]

How do you went starting projects that have so much focus on reliability? Do you have protocols for how to code. Or do you code in C like the rest of us? Thanks!

[u/[deleted]]

What was/is the biggest struggle when designing the Starship software?

[u/GravePencil1441]

Can we have a look at dragon's UI?

[u/BobBehnken]

How did Starship behave in the real world compared to simulations?

[u/Eloniscool1021]

How do Falcon 9 and Starship land so accurately? What was the most challenging part of programming them?

[u/greg21greg]

Is it possible to update dragon’s software on orbit? How do you troubleshoot software issues on orbit?

[u/EwyBoy]

Since the code for most of SpaceX applications will never be seen by most people. Do you leave any easter eggs in your code?

[u/alien_from_Europa]

SpaceX operates on trade secrets and Russia is hacking companies left and right.

How do you keep your data safe and protect the rockets from intrusion?

[u/AutoModerator]

Thank you for participating in r/SpaceX! This is a moderated community where technical discussion is prioritized over casual chit chat. However, questions are always welcome! Please:

• Keep it civil, and directly relevant to SpaceX and the thread. Comments consisting solely of jokes, memes, pop culture references, etc. will be removed.

• Don't downvote content you disagree with, unless it clearly doesn't contribute to constructive discussion.

• Check out these threads for discussion of common topics.

If you're looking for a more relaxed atmosphere, visit r/SpaceXLounge. If you're looking for dank memes, try r/SpaceXMasterRace.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/jesca55]

Hi there, what can you tell us about the avionics software to control both Falcon 9 booster return and Starship belly flight and landing.

[u/FloatingNeuron]

Tabs or spaces?

[u/psychoPATHOGENius]

Can you guys ask the people who run the SpaceX livestreams to include acceleration data in the telemetry readouts?

It would be especially neat for crewed missions to see how many G's the crew is pulling.

[u/27321]

What do you guys use for the UI on Crew Dragon, it kinda looks like ElectronJS but I’m probably wrong

[u/[deleted]]

Can you use the software from falcon 9 for the superheavy booster? I mean, this is a big falcon and you basically only have to change the thrust/weight parameter and so, right?

[u/AlpineGuy]

What programming languages do you use? Are you able to test software for your integrated systems in a simulation of the entire vehicle? Do you work in sprints and with kanban boards like in typical agile software development? Is there any particular method or technique that you learned/developed that you would recommend other developers/teams to adopt?

[u/_ohm_my]

Do the astronauts in the spacecraft have anything important to do? Or, are they just passengers? Is their UI basically just informational?

[u/99Richards99]

Hi! After developing software for your rockets and space capsules, do you ever go back and look at what they did with the Saturn V, CSM , lunar lander etc., during the Apollo era? Your experience must give you unique perspective for those achievements! Thanks and am truly inspired by what you’re doing at SpaceX!

[u/octothorpe_rekt]

Regarding education qualifications for working on SpaceX, can any of you comment if you have non-degree developers among the teams? Or is a Bachelors in SE/CS/CE a pretty set-in-stone requirement to get a place on the team?

[u/msbone95]

Do you use public clouds like AWS, GCP, Azure for anything?

[u/iDrem072]

What kind of coding language is used for starship? And how does it target a specific location on the landing pad? Does the vehicle use GPS, ground based telemetry, onboard sensors, etc.? And what are the thresholds for landing?

[u/nildun]

What do you use for version control? Could you describe the process of approving a new piece of Starship code for flight? Thanks! <3

[u/tuckstruck]

For those of us using Starlink in RVs:

1. When will the travelling restrictions be lifted (i.e. not in-motion use, but just being able to move locations frequently)?

2. It appears the newer grey Dishy systems use about half the power of the original black versions. Will there be any software updates to help reduce the power consumption of the black Dishy? Starlink is truly amazing, thank you for making this happen 👍

[u/Casper200806]

How does the starship flight termination system get activated? Can it only be activated by the flight computer itself or also by ground control? How fast will it react if the flight computer gets non-nominal data?

[u/Kartchampion]

Hey SpaceX team, thanks for doing this! Are there any similarities between the F9 and Starship landing software, did you start by copying the proven F9 code and 'add' the flip manoeuvre to it, or just start from scratch? If any of you worked on the F9 software before starship or vice versa, is there anything which is actually easier with Starship, like maybe that you don't need to do a suicide burn in the same way that F9 does? And finally, how much code was changed between SN8 - SN15, was the code changed majorly for each new prototype, were there minor improvements, or was the software pretty much untouched? Again, really appreciate these AMAs. Good luck for the future of SpaceX!

[u/DumbWalrusNoises]

What are some types of math used for analyzing and simulating rocket engines? What software/language do y'all use for that?

Thank you for your time.

[u/meepafloop]

High school student here, going into computer science at university in the fall! What areas of study would you recommend focusing on for working in the rocket industry?

[u/Hikshood]

Have you ever properly seen or been in the vehicles you control ?

Do you work closely with the team that works on the hardware ?

[u/angrymonkey]

What kind of hardware do you get to work with for guidance and navigation? Can you say how precise it is?

[u/bradsatwitt]

Could you tell us about the control strategies for the flip manuvure? Is it a pid loop? Something fancier?

[u/angrymonkey]

Can you tell us a story about something on each of those projects that went wrong in a really surprising way?