chiplynch.com

So this weekend has been one of the highlights of my career, in a lot of ways. I hadn’t contributed to SpaceX for the Falcon Heavy last year, but now I’ve been here just over 12 months, and while each flight has been exciting and was a learning experience, this one really brought together everything I’ve worked on in very tight focus, where I could see the impacts of software my teams have worked on in specific ways.

I’ve worked on two major projects here — our ERP and our Telemetry system. SpaceX’s ERP system is custom built, which is a bit of a badge of honor, and a challenging delight to work with compared to years watching companies and government entities mismanagement of billion dollar ERP rollouts. Our telemetry system is similarly in-house, and has been growing over the years in a similar fashion. I’ve worked with NASA and the Air Force and never seen anything remotely like our telemetry system, from the rocket’s sensor design to our ground software, to the openness that allows basically every simulation and test to leverage the same technology stack to make sure test are in as close to “real” as imaginable. Common goal in the industry, but it’s nice to see up close.

JOB: Borg Software Developer – work w/ all levels of the stack to build mission-critical @SpaceX code I http://t.co/Wr7ZEAjZ2Z #techjobs

— SpaceX Jobs (@SpaceXJobs) February 20, 2014

So that’s what I’ve worked ON. That and the two dozen side projects that pop up all the time… SpaceX’s software teams are extremely lean, so we host lots of other responsibilities. I’ve worked on a supplier portal, customer facing sites, I got to push the first code that introduced the telemetry system for “Starships” which was just cool; I’ve done dashboarding and metrics, data science APIs, a dozen different infrastructure, languages, and DevOPS projects. The PACE of SpaceX is so furious that I’ve certainly done more in a year here than I have anywhere else. Heck I pushed production code in my first DAY.

But last night — even from when the rocket went vertical on the stand — through the near-midnight ignition, and for the few hours that followed it as we watched the last minute checks and tests, watched the launch, watched the little plush doll — er… Zero-G Indicator — start to float… last night I felt the most connected to the entire launch, because I got to see the software work end to end and see how I’ve impacted it. Hugely humbling, and immensely cool.

The technicians working on the rocket up to the last minute were following instructions from the ERP system. We could follow along as they performed tasks using the UI the team coded, watch the database we modeled get updated directly, and actually watch video as people used the software we’d been pushing code changes to for much of the past year. If someone turned a wrench, pressed a button, or pulled a part from the rockets in the hangar – waiting for our next mission – we knew what was going on. Our role was monitoring at this point; from our end the work was done and if we had messed something up it was too late to make changes. Well… we may have had plans in place, but they weren’t anywhere near plan b (or even c, d, and e). But we had processes running and things in place; the systems were doing their job tracking every little thing.

And when the holy mouse click happened, this time for the crew demo 1 mission, all the hardware was now under control of the ground and flight software. The feeling of being fully interconnected really hit me as most of the company watched telemetry streaming on a dozen screens in mission control, and our group of 20 or so flight software engineers huddled around a TV and a dozen laptops and workstations.

The flight went off as planned, but it was still more exciting to watch. For the first time we had cameras on our final payload. Stage 1 is amazing to watch, but stage 2 has always had the best views of the earth and the sky but you really have to be a satellite to see everything. Starman got some of the best views last time ’round, but this time we got to watch a camera from the nose cone of the dragon capsule go from pitch black to showing the sunrise over the earth. It took a second at first to figure out what was happening, but a long thin arc took over the screen and eventually became a sunrise (which I can’t show you because it’s not public, but here, enjoy Starman again).

The rocket is basically fully automated, but we see a ton of data. As we watched the temperatures for the liquid oxygen and rocket propellant tanks, I said we should throw up the temperature in the cabin… it was, after all, a human factors test. We clicked through telemetry, found someone who knew the name of the temperature channel, and threw them on the screen. I knew how all of that worked behind the scenes, because we had made changes to the source code for the last few months. When someone got excited that their piece of flight software worked… that the pressure sensor, venting mechanism, temperature or attitude control system they worked on was working, it’s because they saw it in the telemetry system we were managing.

I’ve never seen that level of excitement from people directly using software I’ve helped write; not like this. It’s intoxicating. Smiles and cheers were infectious despite the late night hours. As the night wore on we’d think about leaving, but it was always just a few more minutes until the next event. The capsule turned to point a certain way in 10 minutes, or was going to fire in 45 minutes to set the course for the next 20 hours.

Heading back in again today to watch the actual docking with the ISS. It’s been 24 hours since launch; and the excitement is still very high. Definitely glad I came out here.

UPDATE: Yeah, we finally got some amazing pictures. These aren’t renders — this is real! And we’re watching live!

Proximity docking sequence underway. Dragon shadow visible on on @Space_Station solar arrays. pic.twitter.com/ELsxgLIvf0

— Elon Musk (@elonmusk) March 3, 2019

The @SpaceX Crew Dragon is attached to the @Space_Station! It's a first for a commercially built & operated spacecraft designed for crew! #LaunchAmerica pic.twitter.com/ddck58TVnP

— NASA Commercial Crew (@Commercial_Crew) March 3, 2019

SpaceX team in Hawthorne control, Dragon docked to Station above pic.twitter.com/JUWkOrWjsH

— Elon Musk (@elonmusk) March 3, 2019

UPDATE /morez: The exhilaration just continues. It’s currently 5:27 AM in Hawthorne, CA where I just recently left SpaceX to come home. We successfully docked with the space station and the hatch was recently opened and astronauts IN SPACE went inside. This probably happened on our cargo trips, but this is the first time it’s happened on a privately built capsule designed for people.

The Zero-G indicator (a plush Earth doll) just got whacked and is floating around the cabin staring at the camera. What a happy looking earth! #AstronautToys #NASA #spacex pic.twitter.com/XMFa0pDpYG

— Chip Lynch 👾🚀 (@chipmonkey75) March 3, 2019

Watched the launch for a few hours inside the avionics lab this time, with some of our compatriot Flight Software coders and QA team. Got to say hi to Doug Hurley briefly, and had Elon and Bob Behnken floating around nearby. Again, same experience as last night but moreso… getting to see telemetry and people using our software, but being even closer to the mission control aspect with teams that were more responsible for coding the individual steps the capsule took as it approached the ISS; doing test approaches and retreats until every test passed and we successfully docked. This was an amazingly smooth launch from T-0 on, and it’s just been a delight to work with everyone on it. This team rocks.