Author Topic: Scrappy or crappy? Critique by former SpaceX lead engineer  (Read 39463 times)

Offline eeergo

Wonderful opinion piece I haven't found linked elsewhere about the dichotomy between "scrappy" (understood as a positive term describing "doing more with less") and "crappy" sans the leading 's', and the fine line that separates them.

I am posting it in the General Section since it deals with not only Starship (main focus) or Falcon (counter examples) but with recovery and a little bit of everything contained in the SpaceX subforum.

The author is a former SpaceX lead engineer responsible for the successful debut of F9 v1.1 in Falcon 9's 6th flight in 2013, as well as leading the design of the ASDS barges. In other words, one of the people that directly enabled F9 to be the engineering marvel and workhorse it is today, and SpaceX to become the hallmark of space innovation. So, hardly a "hater", "concern troll" or otherwise party with vested interests against SpaceX's way of doing things, as he disclaims in the opening, and instead a very seasoned insider from the heroic SpaceX times that forged the legend upon which everyone supporting Starship rely to justify their apparent shortcomings.

Indeed one of the first examples of "crappy" vs "scrappy" in his text is a (new?) disclosure of how OCISLY's surface was extended, and then repurposed.

https://thenext30trips.com/p/scrappy-special-edition

Some excerpts from the long piece that I found relevant:

Quote
DISCLAIMER: I want to make clear that I am not picking on SpaceX here. If you read it that way, take a deep breath, check yourself, and put down all the water you’ve been carrying for billionaires.

Quote
A few weeks before the [OCISLY] barge was slated to go out on its first mission, our little Recovery team got a panicked phone call from headquarters saying they were worried that the barge wasn’t big enough for the rocket to land on. [...] With no time (and no additional money) we designed something that would technically work, but which in reality looked and functioned like shit. A paltry extra ten feet of reach was added on each side, spanning about 50 feet along the barge. As the missions went by, they rusted and dented up until we finally just cut them off and turned them into blast shielding.

Quote
Like a lot of programs trying to get to orbit, the next steps are picking up the pieces, reviewing the data, and figuring out how to try again [with Starship]. Elon is saying they will be ready to go in 1 - 2 months, which is simply not going to happen.

Quote
They told us we were dumb kids [during F9 development times], that we didn’t know what we were doing, and that we were going to fuck it up. They were mostly right, at various times, if I’m honest. It took a minute, but we eventually figured things out. Something changed, though, around the time we started landing Falcon 9 first stages. Suddenly, we weren’t the underdogs anymore; we were the leaders. That change felt odd, and I remember it happening in real time. [...] As the number of articles, books, videos, and personalities trying to make a living talking about SpaceX online multiplied, I noticed that a weird, decidedly male, scam-adjacent faction started to form.

Quote
A great example of a test that was widely and rightly derided was one by Pythom Space. [...] To be blunt, I see a lot more similarity between this test and the Super Heavy launch than I’d like to. Both were ill-considered, dangerous, destructive, and would’ve benefitted from some real soul searching about why the test was done as well as how it should be done safely. Both of these tests had the air of a circus and a “lol fuck it send it” mentality.

Quote
It’s important that we strive to conduct safe, well-considered tests knowing that things can go wrong, while minimizing the impact. I fear that by celebrating this test and spinning it as necessary progress, we may do more long-term harm than good to the space program and our approach to innovation.

Quote
the further someone is from the action, the less you should trust their excitement about a mission’s relative success. So just because someone sprayed people down with champagne after this Starship flight, as was widely reported, that doesn’t mean it was much of a success.

Quote
Keep in mind, that Starship has nothing inside of it except the tanks, valves, and wires needed to make it fly. There is no payload bay. There are no seats. There is no life support system. We don’t know if the re-entry tiles will work. Honestly, if this was any company other than SpaceX I would declare them toast. [...]  while in my opinion this test was firmly on the crappy side of the “s/crappy” divide, that doesn’t doom them to remain there. [...] Until then, though, it would behoove the rest of us to judge everyone’s declarations of success equally, fairly, and with a critical eye.

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
LOL send it

EDIT: really the interview sounds less like a critique and more like a fair evaluation of the process of making something which starts out impossible and ends up utterly routine. It’s a pretty natural journey to start out crappy, make mistakes that are pretty obviously crappy, but which form the basis of tacit knowledge going forward. Failure is an important part of finding the optimum solution when constrained by resources.

Having worked in a place where you’re not allowed to do that… and it sucks.

(Also, he’s coming from someone who is now an outsider to the company who’s overall—and understandably—sick of E’s s*** and I’m pretty sure that colors his perspective here…)
« Last Edit: 04/28/2023 01:17 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Online ZachS09

  • Space Savant
  • Senior Member
  • *****
  • Posts: 8393
  • Roanoke, TX
  • Liked: 2339
  • Likes Given: 2053
I feel that this critique made a bunch of valid points. For example, I should've paid more attention to the launch control room view after they showed the Hawthorne crowd cheering.

I also agree with the concern of the OLM next to Pad 39A following the Starship IFT. That pad is a historic relic, not just a simple launch pad.
Liftoff for St. Jude's! Go Dragon, Go Falcon, Godspeed Inspiration4!

Offline eeergo

really the interview sounds less like a critique

Not an interview, it's an opinion piece by the author.

Quote
It’s a pretty natural journey to start out crappy, make mistakes that are pretty obviously crappy

Not his conclusion, and not true in general: you don't need to start luxurious to avoid "crappy" mistakes, when those are obvious or easily avoidable.
-DaviD-

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
He’s also now in a competing company that specifically does GSE/launch pad tech.

Early 20s: intern, doesn’t know what they’re doing. Crappy.

Late 20s: scrappy.

Early 30s: scrappy, becoming truly competent.
Mid to late 30s: competent, somewhat seasoned, no longer scrappy, judge mental of those who are not yet competent, impatient with the crappy interns, still steps in when they see Them starting to make a mistake because they know the pain of a mistake.

50s and later: competent, highly seasoned, sees the intern making the mistake, knows the pain that will cause, recognizes that as a necessary part of the whole process. “Let them cook.” This is where folks like Gerstenmeier and Wayne Hale are.
« Last Edit: 04/28/2023 01:46 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline eeergo

You know, or not.
-DaviD-

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
I also think the hate on Pythom was a little much.
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Online matthewkantar

  • Senior Member
  • *****
  • Posts: 2055
  • Liked: 2494
  • Likes Given: 2190
The author lost me when he compared the recent launch attempt to Pythom’s efforts.

Offline shark0302

  • Member
  • Posts: 20
  • Liked: 11
  • Likes Given: 0
It felt like he was trying to be objective but also had a cognitive bias towards the team working on the starship program.  What i got out of the entire post. The team should have known all the things that people who were part of the original falcon program who have since left already knew.  This, to me shows that not enough senior people who were part of the old mistakes are over seeing the starship program. Which means simple things got  relearned the hard way. On the flip side of that they also learned something I'm sure we just don't know what yet.
« Last Edit: 04/28/2023 05:58 pm by zubenelgenubi »

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
Starship may be the last chance for SpaceX engineers to make big mistakes that look really dumb. None of this stuff would be allowed on the Falcon 9 side of things.

Starship is the last hurrah for crappy or even scrappy at SpaceX. Much of the organization has already evolved to a far more polished, process-driven, incremental approach to operations. Starship will get there, too. It has to, eventually, to be a successful workhorse that supplants Falcon 9.

But the idea they should run development the same way they need to run operations… is the path to a Blue Origin-like pace. It would’ve been impossible to develop for less than $100 billion.
« Last Edit: 04/28/2023 03:33 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline mandrewa

  • Full Member
  • ****
  • Posts: 549
  • Liked: 449
  • Likes Given: 8160
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #10 on: 04/28/2023 04:12 pm »
Hindsight is wonderful.  But it's difficult to fairly evaluate a situation if you don't know the whole context and you have the benefit of knowing what is going to happen next.

For all we know there was really only one mistake.  Or in other words the design of what I'm calling the floor of the launch pad.  Every other failure could have been a consequence of that one failure.  Now I doubt that.  I'll bet there were issues that were separable from what happened with the pad.

And that's part of the value of this test launch.  Now they have data that can influence and change everything up to stage separation and maybe beyond.

I like Ben Kellie's scrappy versus crappy distinction.  And as he repeatedly stated, there's a fine line between the two.

Now I'm sure Ben Kellie knows this, despite not saying it, or maybe he doesn't know it, but in any case he doesn't mention the huge difference in cost and space used between say Pad 39B, which is being used for SLS/Orion, and the Boca Chica launch site.

That NASA launch pad has cost how much to build?  Is it $5 billion?  Or is it actually more than that?  And how many acres does that launch pad occupy?

A big part of this is about money.  Elon Musk is trying to lower the cost of moving mass to space by orders of magnitude. He doesn't have an infinite amount of money.  So given that context and the lack of room at Boca Chica, well of course, he's trying to do it cheaper.

Well he went too cheap!

The exciting thing is that it may be, as Ben Kellie says, "The fix will require complete repouring of the foundation around the launch mount, installation of some kind of cooled-plate to prevent this from happening again, as well as likely refurbishment of the damaged ground tanks.9"

And if that's true, then we will end up with a launch pad that can be built for a much lower cost than say Pad 39A.  I hope it's true.

Offline Greg Hullender

  • Full Member
  • ****
  • Posts: 600
  • Seattle
    • Rocket Stack Rank
  • Liked: 440
  • Likes Given: 333
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #11 on: 04/28/2023 04:15 pm »
Interesting read. Thanks for posting it!

Something he doesn't talk about is the fact that B7 was already obsolete. So on the one hand, they didn't have a lot to lose by flying it (except for losing the launch pad!), but on the other hand, I'm not sure how much they stood to learn from it either. Is it possible that the whole exercise was pointless? I hope not, but I'm not sure.

The one thing they clearly did learn from this was that you can't cut corners with the launch pad. Should they have really known that already? Maybe, but there's lots of stuff they "should have known" (e.g. you can't return a first stage propulsively, and even if you did, you couldn't reuse it cost-effectively) that turned out to be false. Would it have saved a lot of time and money if el cheapo launch pads worked? If so, was the experiment worth the gamble, or could they have learned this a lot more cheaply? Those are good questions, but I'm not sure anyone outside of SpaceX can really answer them with any authority.

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #12 on: 04/28/2023 04:44 pm »
SpaceX also would’ve had to wait 3 more months to get the better solution installed. So if repair and finish building that solution takes no more than 3 months, then SpaceX is actually STILL ahead and they couldn’t have done better even with hindsight (although I do think extra layers of refractory cement probably would’ve been worth doing with the benefit of hindsight).
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Offline ugordan

  • Senior Member
  • *****
  • Posts: 8507
    • My mainly Cassini image gallery
  • Liked: 3519
  • Likes Given: 757
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #13 on: 04/28/2023 05:03 pm »
Hindsight is wonderful.  But it's difficult to fairly evaluate a situation if you don't know the whole context and you have the benefit of knowing what is going to happen next.

For all we know there was really only one mistake.  Or in other words the design of what I'm calling the floor of the launch pad.  Every other failure could have been a consequence of that one failure.  Now I doubt that.  I'll bet there were issues that were separable from what happened with the pad.

And that's part of the value of this test launch.  Now they have data that can influence and change everything up to stage separation and maybe beyond.

Way back in the day (maybe 2013?) when the first F9-R triple-engine test launch happened in McGregor and blew up, SpaceX said something to the effect of: we were only monitoring 1 single string of multiple available engine sensors and the sensor went bonkers so we lost the vehicle. But, fear not, regular F9 launches don't short-circuit this multiple sensor logic, so we're fine for normal F9 launches.

A number of seasoned posters back then said that it was a dumb move to ignore the redundant engine sensors (or something to that effect) (and I can't really argue with that).

I wonder if this former engineer was part of that program as well, because the comments seem relevant to the SH launch - i.e. we should have known better. But, for some reason, the F9 recovery program gets a pass, while this SH/SS launch is seen as too cowboy-like and "just YOLO it"?

Reason I post this is, 10 years later, we all see what eventually became of F9 recovery, even if there were "dumb" mistakes made. It's not the first time SpaceX made a wrong decision and had to back-track, and it's very likely not the last.

They knew that using Fondag was a calculated risk, it appeared to perform OK during the 50% throttle static test. They were wrong. They will pay the price for the wrong assessment in both money and time, but, again, it's not the first time they've likely made a bad call - albeit this one is very publicly visible.

Does that make SH/SS test program inherently more risky than F9 recovery program? I dunno. History will tell.

Offline RoadWithoutEnd

  • Full Member
  • **
  • Posts: 283
  • Liked: 340
  • Likes Given: 442
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #14 on: 04/28/2023 07:34 pm »
SpaceX also would’ve had to wait 3 more months to get the better solution installed. So if repair and finish building that solution takes no more than 3 months, then SpaceX is actually STILL ahead and they couldn’t have done better even with hindsight (although I do think extra layers of refractory cement probably would’ve been worth doing with the benefit of hindsight).

Plus, whatever solution was in work wouldn't have had the benefit of all this empirical data, so could have still been inadequate while costing more time and money to fix than what they do now.

Basically, they just have to stop Stage 0 from digging itself a Stage -1. 

The jury isn't in on all the ways the rocket was compromised by its own ignition, but initial appearances are that it's a certified beast that got most of the way to MECO even after having taking a shotgun blast of concrete magma to the gut.  So, depending on how well the OLM and OLT faired, things like reflected shockwaves and heat that normally have to inform GSE development might be a retired risk, and only the pad itself is left to deal with. 

Fix that one thing and the rest might be straightforward.
Walk the road without end, and all tomorrows unfold like music.

Offline JayWee

  • Full Member
  • ****
  • Posts: 986
  • Liked: 981
  • Likes Given: 1795
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #15 on: 04/28/2023 08:49 pm »
SpaceX also would’ve had to wait 3 more months to get the better solution installed. So if repair and finish building that solution takes no more than 3 months, then SpaceX is actually STILL ahead and they couldn’t have done better even with hindsight (although I do think extra layers of refractory cement probably would’ve been worth doing with the benefit of hindsight).

Plus, whatever solution was in work wouldn't have had the benefit of all this empirical data, so could have still been inadequate while costing more time and money to fix than what they do now.

Basically, they just have to stop Stage 0 from digging itself a Stage -1. 

The jury isn't in on all the ways the rocket was compromised by its own ignition, but initial appearances are that it's a certified beast that got most of the way to MECO even after having taking a shotgun blast of concrete magma to the gut.  So, depending on how well the OLM and OLT faired, things like reflected shockwaves and heat that normally have to inform GSE development might be a retired risk, and only the pad itself is left to deal with. 

Fix that one thing and the rest might be straightforward.
Model verification is always good. I am rather interested in the counter-question (with the image of the enormous flame diverter of Soyuz in mind):
Given the size of the rocket, has the pad worked better than assumed by people always expecting a flame diverter?
« Last Edit: 04/28/2023 08:50 pm by JayWee »

Offline JayWee

  • Full Member
  • ****
  • Posts: 986
  • Liked: 981
  • Likes Given: 1795
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #16 on: 04/28/2023 09:04 pm »
From the article:
Quote from: Ben Kellie
→ So, why the hell didn’t that same testing happen with the Super Heavy stage?

I have no idea. The last static fire they conducted before launch didn’t even fire all the engines! Only 31 of the 33 engines were tested during that static fire. In fact, they never tested the full set of engines all together. That’s not testing like you fly. That’s not running a comprehensive test program.
This is the point I disagree the most.  DUH. They tested exactly like you fly - by, umm,  flying!.

Somehow I am reminded of this thing:
Quote from: http://heroicrelics.org/info/all-up/all-up-flight-testing.html
George Mueller, shortly after being named NASA Deputy Associate Administrator for Manned Space Flight, introduced the concept of "all-up testing" to the Saturn/Apollo program. Rather than traditional method of testing rockets, which called for a slow, methodical program, testing one stage before adding another live stage, "all-up" called for each rocket stage and each spacecraft module to be live and representative of the form which would be used in the actual lunar mission.

« Last Edit: 04/28/2023 09:05 pm by JayWee »

Online Robotbeat

  • Senior Member
  • *****
  • Posts: 39252
  • Minnesota
  • Liked: 25205
  • Likes Given: 12104
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #17 on: 04/28/2023 09:15 pm »
The one critique that I would make of Starship is that SpaceX did really well with Falcon 9 in part because they went absolutely nuts with ground testing. They tested the absolute s*** out of the engines, the rocket, the stage separation mechanism, etc. the only thing they didn’t do was test the upper stage in vacuum, which would’ve costed like $400 million or something (so they were like Just Send It, cheaper to test in flight for $40 million). I think if SpaceX had done more work on Starship booster GSE earlier on, they could’ve probably made progress sooner.

However, that’s traveling like a year or two back in time. If we talk about a couple weeks back in time, I’m not sure their decisionmaking was actually bad. They would’ve had to wait 3 months for the flame diverter to be installed. That’s STILL potentially longer than it’ll take to get a booster back on the pad now.
« Last Edit: 04/28/2023 09:18 pm by Robotbeat »
Chris  Whoever loves correction loves knowledge, but he who hates reproof is stupid.

To the maximum extent practicable, the Federal Government shall plan missions to accommodate the space transportation services capabilities of United States commercial providers. US law http://goo.gl/YZYNt0

Online Coastal Ron

  • Senior Member
  • *****
  • Posts: 8827
  • I live... along the coast
  • Liked: 10170
  • Likes Given: 11898
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #18 on: 04/28/2023 09:44 pm »
Starship may be the last chance for SpaceX engineers to make big mistakes that look really dumb. None of this stuff would be allowed on the Falcon 9 side of things.

We have to remember that the SpaceX of 2010 needed to produce revenue with each launch, so they didn't launch rockets unless they were getting revenue.

The SpaceX of today (2023) can afford to do incremental rocket testing, and in fact has to with the Starship because it is so big and complex.

Quote
Starship is the last hurrah for crappy or even scrappy at SpaceX.

I don't know why that would be. Didn't SpaceX just send up a bunch of Starlink satellites that didn't work as well as planned? I think if they need test data in a short period of time and a reasonable cost, I think they will still risk destruction of hardware.

Quote
Much of the organization has already evolved to a far more polished, process-driven, incremental approach to operations. Starship will get there, too. It has to, eventually, to be a successful workhorse that supplants Falcon 9.

How you run a business with a mature product is different than how you run a business doing product development. Sustaining innovation is different than disruptive innovation, for many good reasons. But that doesn't mean SpaceX has to forget how to do disruptive innovation, however it does mean they have to work hard to keep that ability, since once companies start cost cutting they inevitably go after the cost centers that have less connections to revenue - which is usually new product development.

Quote
But the idea they should run development the same way they need to run operations… is the path to a Blue Origin-like pace. It would’ve been impossible to develop for less than $100 billion.

I have no idea what is happening at Blue Origin, but yes, let's hope SpaceX NEVER becomes like them. And Blue Origin had so much potential going for them...  :(
If we don't continuously lower the cost to access space, how are we ever going to afford to expand humanity out into space?

Offline alugobi

  • Full Member
  • ****
  • Posts: 1480
  • Liked: 1543
  • Likes Given: 0
Re: Scrappy or crappy? Critique by former SpaceX lead engineer
« Reply #19 on: 04/28/2023 09:52 pm »
They would’ve had to wait 3 months for the flame diverter to be installed. That’s STILL potentially longer than it’ll take to get a booster back on the pad now.
Key word 'potentially'.  That means a booster ready to static fire by July. 

Still skeptical of that.  We still don't know from them all that they have to fix yet.  Standing by...

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0