My Unsolicited Take on Starship Flight Testing

[Jim]

 SpaceX doesn't have the budget like NASA did in the 60's.  This has led to some choices like flight testing emphasis over ground testing.  Ground testing requires expensive infrastructure.  SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design. With modern avionics, SpaceX can get more data from more test points on a vehicle in flight than NASA did during ground tests.   NASA had 4 non-flight test articles built for each stage for Saturn V (static fire, structural loads, facilities, and ground dynamics) testing. So isn't it until the fifth (or 7th to include Apollo 4 & 6) Starship launch before a comparison to the Saturn V can be made?

[ulm_atms]

Yes.....all of it.   

[vaporcobra]

Do you mean comparing Starship to Saturn V is unwise in the sense that "Starship isn't that impressive because Saturn V reached orbit on its first launch" ignores the different ways they were developed?

[abaddon]

I don't really see the need to ask questions, the statement is very clear and concise.

[RedLineTrain]

With the size of Starship (ultimately at least ~3x the thrust of Saturn V), it seems like extensive ground testing is nigh on impossible.  Even NASA's Apollo-era budgets, willingness to use governmental powers, and lesser focus on environmental impact might not cover it.

Because of this, I hesitate to compare the Saturn V and Starship.  They're just in different vehicle classes (Moon-class and Mars-class, respectively) and are each a product of their times.  And then Saturn V doesn't have a long enough flight history to know for sure about its ultimate reliability.

[Metalskin]

I've seen people scorn and praise SpaceX's approach and I've never known how to assess when people raise previous programmes.

Thank you Jim, your take really helps me a lot. =Obviously I'm just an anonymous armchair rocket fan, but what you state makes sense.

The only thing I don't know how to factor in is how computer modeling and simulation has impacted the ability to test designs. I imagine that is a huge advantage for SpaceX, but not sure how that would impact testing methodology.

[Brigantine]

SN8 - SN11 and SN15 had no RVacs and no heat shield, so lets consider them as mere component testing. Discount S24 as a 2nd stage test since stage separation never happened, so S25 was the 1st complete stage test of stage 2.

The more I get into the details, the more I agree with OP.
It's bizarre but it seems stage 2 ID#s 1-19, 21-24, 26, 27 all don't even count as equivalent to Saturn V ground test articles.

[Jim]

Do you mean comparing Starship to Saturn V is unwise in the sense that "Starship isn't that impressive because Saturn V reached orbit on its first launch" ignores the different ways they were developed?

Exactly

[kevin-rf]

Somewhere someone added up all the Raptor flight time and in two test flights it has exceeded all Saturn V F1 flight time....

[TrueBlueWitt]

Somewhere someone added up all the Raptor flight time and in two test flights it has exceeded all Saturn V F1 flight time....

Don't forget, Saturn V got very lucky it Survived POGO and probably some other issues on early flights. Even with all the testing.
And it wasn't trying to be re-usable. 

https://www.nasa.gov/history/50-years-ago-solving-the-pogo-effect/

If this was Saturn V, no one would have cared at all if the booster blew up AFTER stage sep. Correct?

[meekGee]

One point of comparison is that (to my knowledge) the several Saturn V development articles represented a design that wasn't evolving as much as SS is.

Meaning: NASA won't typically build a multi stage rocket without really knowing how staging will work.  Or try launching it from a stage 0 that's "probably going to break" just to see if maybe they can get away with it.

The fact that BFR/ITS/SS evolved so much (material of choice, diameter, EDL method) is not unprecedented, because that was mostly on paper.   But the amount that the 9 m stainless SS is evolving (e.g. heat shield method, staging method) - that pretty much is, and warrants the "failures" during the development flights.

[edzieba]

One point of comparison is that (to my knowledge) the several Saturn V development articles represented a design that wasn't evolving as much as SS is.

Meaning: NASA won't typically build a multi stage rocket without really knowing how staging will work.  Or try launching it from a stage 0 that's "probably going to break" just to see if maybe they can get away with it.

The Saturn programme swapped out engines and stages as it progressed. Saturn-I and Saturn-V were entirely different vehicles, despite the progression from one to the other being clear and direct.
As for radical mid-programme changes; at the time Saturn-V was being drawn up, EOR was still the proposed CONOPS. The entire Apollo system architecture changed during vehicle development.

[kevin-rf]

Don't forget as Saturn was evolving into what it became they had concepts for first stage reuse and even went so far as to dunk an engine into the Gulf of Mexico after test firing it.

[meekGee]

One point of comparison is that (to my knowledge) the several Saturn V development articles represented a design that wasn't evolving as much as SS is.

Meaning: NASA won't typically build a multi stage rocket without really knowing how staging will work.  Or try launching it from a stage 0 that's "probably going to break" just to see if maybe they can get away with it.

The Saturn programme swapped out engines and stages as it progressed. Saturn-I and Saturn-V were entirely different vehicles, despite the progression from one to the other being clear and direct.
As for radical mid-programme changes; at the time Saturn-V was being drawn up, EOR was still the proposed CONOPS. The entire Apollo system architecture changed during vehicle development.

Well the assertion being countered was that "Saturn V" flew the first time, so I ignored Saturn 1.

Also for the same reason didn't talk about Starship refiling and the extended mission profile, just basic rocket functionality (which in this case includes EDL)

[Asteroza]

Would Saturn 1 flights count, in the same way Hoppy and early starship hops could count?

[envy887]

SpaceX did build a lot of ground test boosters and ships, and tested all of ground handling, flight dynamics, static firing, and structures with them.

That said, the difference in resources driving different ground test thoroughness and fidelity is an excellent point. NASA spent almost 50 billion on Saturn V (inflation adjusted) before its first successful flight.

[Barley]

SpaceX did build a lot of ground test boosters and ships, and tested all of ground handling, flight dynamics, static firing, and structures with them.

That said, the difference in resources driving different ground test thoroughness and fidelity is an excellent point. NASA spent almost 50 billion on Saturn V (inflation adjusted) before its first successful flight.

Jim did mention modern avionics and telemetry, but I think people are ignoring how much this affects the ability to do inflight testing, and hence how you do testing.

Restricted telemetry requires more ground testing because you can't get as much data from a test flight.  Compare the Saturn V camera pods with sticking on another GoPro, on board image processing to get critical frames and adding that to the 100Mbps of telemetry.  Remember Apollo was closer in time to a Sopwith Camel than a StarShip.

[Yggdrasill]

I completely agree they are products of different times.

At it's peak, the Apollo program had 400,000 people working on it. SpaceX has 12,000 employees, and most of them aren't working on Starship. Though they do have suppliers that increase the number as well. So, maybe something like 10,000? 2.5% of Apollo.

With modern technology, fewer people can do more, but you still have fewer eyes on everything. Some of the mistakes SpaceX has made would probably have been caught in the Apollo program.

[spacenut]

SpaceX cannot do a full thrust test on the ground of all 33 booster engines.  It would break away from the hold down clamps.  Saturn V did not try to save the booster either after staging.  Sure they can test each engine separately, but not all at one time. 

SpaceX is building this rocket on the cheap compared to NASA.  As said 400,000 vs 12,000 employees. 

[jpo234]

SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design.

NASA never planned to mass produce the Saturn V. SpaceX on the other hand wants to build hundreds or even thousands of Starships and boosters. Being hardware rich comes naturally from this goal.

[DanClemmensen]

SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design.

NASA never planned to mass produce the Saturn V. SpaceX on the other hand wants to build hundreds or even thousands of Starships and boosters. Being hardware rich comes naturally from this goal.

Indeed. Among other things, They are prototyping their efficient high-volume production process, and that would require them to build prototypes even if they did not intend to fly them. May as well go ahead and fly them if they have any chance of clearing the pad.

[edkyle99]

SpaceX doesn't have the budget like NASA did in the 60's.  This has led to some choices like flight testing emphasis over ground testing.  Ground testing requires expensive infrastructure.  SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design. With modern avionics, SpaceX can get more data from more test points on a vehicle in flight than NASA did during ground tests.   NASA had 4 non-flight test articles built for each stage for Saturn V (static fire, structural loads, facilities, and ground dynamics) testing. So isn't it until the fifth (or 7th to include Apollo 4 & 6) Starship launch before a comparison to the Saturn V can be made?

It is going to take more than five, I think.  Probably a lot more.  Ground testing allows so much more rigorous examination of problems than flight testing.  Consider, for example, that NASA's first "T-Bird" S-1C stage conducted twenty-two test firings.  Each Super Heavy gets to lift off just one time at this point in development.  S-2-T did nine test firings before it exploded on the A2 test stand.  NAA also test fired a battleship S-2 many more times and used it for literally hundreds of propulsion system tests, including consideration of thermal effects that led to much improved insulation on the engines and feed lines, etc.  (I suspect that Super Heavy and Starship both have a long way to go on TPS development.)

 - Ed Kyle 

[alugobi]

So?  Hardware, at SX, is cheap.  Use it, tweak it, use it some more, tweak it.  That's how they got to Falcon 9, the envy of the launch market.

[edkyle99]

So?  Hardware, at SX, is cheap.  Use it, tweak it, use it some more, tweak it.  That's how they got to Falcon 9, the envy of the launch market.

No, that is not how they developed Falcon 9.  They static test-fired Falcon 9 stages extensively at McGregor, especially the early stages.  Full duration testing.  The first "Run Tank" stage was tested there many times for more than a year during 2007-08.  They broke some stuff there on the test stands and learned important lessons.  They still acceptance test fire stages there.  With Super Heavy/Starship it is going to be launch and explode until and if they get it right. 

 - Ed Kyle

[Jim]

SpaceX doesn't have the budget like NASA did in the 60's.  This has led to some choices like flight testing emphasis over ground testing.  Ground testing requires expensive infrastructure.  SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design. With modern avionics, SpaceX can get more data from more test points on a vehicle in flight than NASA did during ground tests.   NASA had 4 non-flight test articles built for each stage for Saturn V (static fire, structural loads, facilities, and ground dynamics) testing. So isn't it until the fifth (or 7th to include Apollo 4 & 6) Starship launch before a comparison to the Saturn V can be made?

It is going to take more than five, I think.  Probably a lot more.  Ground testing allows so much more rigorous examination of problems than flight testing.  Consider, for example, that NASA's first "T-Bird" S-1C stage conducted twenty-two test firings.  Each Super Heavy gets to lift off just one time at this point in development.  S-2-T did nine test firings before it exploded on the A2 test stand.  NAA also test fired a battleship S-2 many more times and used it for literally hundreds of propulsion system tests, including consideration of thermal effects that led to much improved insulation on the engines and feed lines, etc.  (I suspect that Super Heavy and Starship both have a long way to go on TPS development.)

 - Ed Kyle 

They aren't going to be able to fly the whole mission within one or two launches?   Then do orbital launches and then start recovery tests.

[envy887]

So?  Hardware, at SX, is cheap.  Use it, tweak it, use it some more, tweak it.  That's how they got to Falcon 9, the envy of the launch market.

No, that is not how they developed Falcon 9.  They static test-fired Falcon 9 stages extensively at McGregor, especially the early stages.  Full duration testing.  The first "Run Tank" stage was tested there many times for more than a year during 2007-08.  They broke some stuff there on the test stands and learned important lessons.  They still acceptance test fire stages there.  With Super Heavy/Starship it is going to be launch and explode until and if they get it right. 

 - Ed Kyle

It is how they developed Falcon 9 recovery and reuse. The first 20 or so F9 boosters launched and exploded before they started recovering successfully.

The Super Heavy booster doesn't seem to have a problem with launch. And Starship was only seconds away from SECO. The 3rd flight will probably reach orbit, and from that point onward they can use it for payloads while they figure out reuse.

[catdlr]

So?  Hardware, at SX, is cheap.  Use it, tweak it, use it some more, tweak it.  That's how they got to Falcon 9, the envy of the launch market.

No, that is not how they developed Falcon 9.  They static test-fired Falcon 9 stages extensively at McGregor, especially the early stages.  Full duration testing.  The first "Run Tank" stage was tested there many times for more than a year during 2007-08.  They broke some stuff there on the test stands and learned important lessons.  They still acceptance test fire stages there.  With Super Heavy/Starship it is going to be launch and explode until and if they get it right. 

 - Ed Kyle

It is how they developed Falcon 9 recovery and reuse. The first 20 or so F9 boosters launched and exploded before they started recovering successfully.

The Super Heavy booster doesn't seem to have a problem with launch. And Starship was only seconds away from SECO. The 3rd flight will probably reach orbit, and from that point onward they can use it for payloads while they figure out reuse.

You are so right Ed,
The public and especially the media have forgotten that (NSF members excluded).  People should stop worrying/kidding SpaceX about failed launches or explosions and start seeing the investment into testing, technology, and making a better product.

Here are some F9 stats (from running an AI query)
Query:  How many Falcon 9 launches were there until the first booster landed successfully?

----AI Response ---------

As of October 4, 2023, SpaceX has successfully landed 248 out of 261 Falcon 9 first-stage boosters, with 139 out of 144 (96.5%) for the Falcon 9 Block 5 version. This means there have been 13 failed landings of the Falcon 9 first-stage booster.

The first successful landing of a Falcon 9 first-stage booster occurred on December 22, 2015, on the sixth flight of a Falcon 9 and the maiden launch of the v1.1 rocket version. Since then, SpaceX has successfully landed the first stage of a Falcon 9 rocket on multiple occasions, including on land, at sea, and on autonomous spaceport droneships.

Here is a breakdown of the number of Falcon 9 launches and the number of successful first-stage booster landings:

Launch Version   Total Launches   Successful Landings
Falcon 9 v1.0          5                    0
Falcon 9 v1.1         18                    4
Falcon 9 v1.2         25                    14
Falcon 9 Block 5         248                    230

Overall, SpaceX has a very high success rate for landing the first-stage booster of its Falcon 9 rockets. This is a significant achievement, as it allows SpaceX to reuse the rocket's first stage for future launches, saving money and reducing the environmental impact of spaceflight.

---  end AI query --

[edzieba]

SpaceX choose to be hardware rich and spend money and time gaining experience and building the vehicle and refining its design.

NASA never planned to mass produce the Saturn V. SpaceX on the other hand wants to build hundreds or even thousands of Starships and boosters. Being hardware rich comes naturally from this goal.

NASA did initially plan to produce Saturns en masse (and Novas), for initial plans of at least 6 launches per year. Hence why the VAB is sized for 4 stacks in simultaneous operation, and LC-39 initially sized at 4 pads with layout planned up to 8 pads. Plans changed during development as the Apollo mission and architecture developed.

[Emmettvonbrown]

Saturn C-1 and C-2 were to be build like sausages and at least partially reused - thanks to Ryan giant paragliders with as much wing area as a B-70 Valkyrie.
The Army Project Horizon, and early Apollo plans, were EOR with lots and lots of C-1s and C-2s. Plus the Saturn tankage was derived from missiles already in large scale production (never quite remember which one, Redstone or Jupiter).

[edkyle99]

They aren't going to be able to fly the whole mission within one or two launches?   Then do orbital launches and then start recovery tests.

I would be happy to see that result, but surprised.  These things rarely happen in a straight line from failure toward full success.  I always think of Thor, which was rushed to the pad with no ground testing.  The first four blew up, then they had a success.  Terrific, right?  Then three of the next five failed.  Then four more failed.  Then they built a static test stand.

 - Ed kyle

[woods170]

They aren't going to be able to fly the whole mission within one or two launches?   Then do orbital launches and then start recovery tests.

I would be happy to see that result, but surprised.  These things rarely happen in a straight line from failure toward full success.  I always think of Thor, which was rushed to the pad with no ground testing.  The first four blew up, then they had a success.  Terrific, right?  Then three of the next five failed.  Then four more failed.  Then they built a static test stand.

 - Ed kyle

Lot's of launch vehicles have been quite successful without integrated ground testing of their booster stages. STS and SLS for example, where only the individual components were ground tested separately, but never all-up together until the first launch. Same for Ariane 5, Ariane 4 and Ariane 3. Ditto for Delta IV Heavy Falcon Heavy, every single version of Atlas with add-on SRBs, etc, etc.

IMO comparing development of Thor, when building rockets was in its infancy, to the development method of a very seasoned company with 21 years of development experience, is not the smartest thing to do (putting it mildly).

[mandrewa]

So?  Hardware, at SX, is cheap.  Use it, tweak it, use it some more, tweak it.  That's how they got to Falcon 9, the envy of the launch market.

No, that is not how they developed Falcon 9.  They static test-fired Falcon 9 stages extensively at McGregor, especially the early stages.  Full duration testing.  The first "Run Tank" stage was tested there many times for more than a year during 2007-08.  They broke some stuff there on the test stands and learned important lessons.  They still acceptance test fire stages there.  With Super Heavy/Starship it is going to be launch and explode until and if they get it right. 

 - Ed Kyle

You make a good point, but does SpaceX have a real choice?

One of the things that bothers me about the FAA's recent launch approval (for the second development flight) was that the US Fish and Wildlife Service added in a restriction of thirty uses of the flame deflector system per year.

How does that square with a large static fire testing campaign?  To do what SpaceX did for the Falcon 9, SpaceX would have to build another launch mount in a location without these restrictions and quite likely build another Super Heavy factory to go with it.

[edzieba]

the US Fish and Wildlife Service added in a restriction of thirty uses of the flame deflector system per year.

The USFWS did no such thing.
The USFWS asked SpaceX how many discharges they expected the deluge system to make per year. SpaceX estimated that with 10 launches and 3 discharges per launch (two static fires and one launch) that would be 30 per year. The USFWS then estimated impacts based on that 30 discharges figure.

Note that the launches per year figure came from SpaceX, just like the 5 launches per year figure from the 2022 PEA. And just as that estimate went from 5 to 10 per year under the most recent WR, it can continue to rise as needed just as it already has with every single other pad SpaceX operates (all of which have flight rates today many multiples of those assumed in their initial EISes).

[mandrewa]

Well, thanks for that information, edzieba.  It isn't so much the part about the number coming from SpaceX, which I would have actually guessed.  But it's that they were asked to set a limit.

Where would we be if IFT-2 had had three engines shut down prematurely at launch?  Where would be if SpaceX was just discovering now that they probably had to do a lot of static fires?

In any case, I think the most important part of this is that the limit can be raised as successes accumulate and the need for more launches is demonstrated.  I hope this continues.

[DanClemmensen]

Well, thanks for that information, edzieba.  It isn't so much the part about the number coming from SpaceX, which I would have actually guessed.  But it's that they were asked to set a limit.

Where would we be if IFT-2 had had three engines shut down prematurely at launch?  Where would be if SpaceX was just discovering now that they probably had to do a lot of static fires?

In any case, I think the most important part of this is that the limit can be raised as successes accumulate and the need for more launches is demonstrated.  I hope this continues.

Presumably SpaceX can ask for FWS to re-evaluate after SpaceX has more actual experience with the deluge system. As of now, they don't really have a lot of data on actual impact on the environment, and they had not choice except to use worst-case numbers. After looking at the data, they can also do additional mitigation if it's useful. Someone posted a document(?) here showing that any company can ask FWS for an opinion on impact at any time, in order to help a company that is trying to reduce environmental impact.  This sort of proactive stuff would allow for a quick and smooth WR for an FAA license in the future if it is needed.

[edzieba]

But it's that they were asked to set a limit.

They did not 'set a limit'. They evaluated the potential impacts based on the flight rate estimate SpaceX provided. To fly more often, the exact same FONSI reissuance as has occurred at SpaceX's other pads as and when needed to support increased flight rates would occur.

[Negan]

All this talk about testing and no mention of booster hops! It was such a no brainer back in the day.  

[mandrewa]

But it's that they were asked to set a limit.

They did not 'set a limit'. They evaluated the potential impacts based on the flight rate estimate SpaceX provided. To fly more often, the exact same FONSI reissuance as has occurred at SpaceX's other pads as and when needed to support increased flight rates would occur.

You say that like it is nothing.  How many days could a FONSI reissuance take, assuming it is granted?

A limit has been set.  To change it they have to do another FONSI reissuance.

[chopsticks]

But it's that they were asked to set a limit.

They did not 'set a limit'. They evaluated the potential impacts based on the flight rate estimate SpaceX provided. To fly more often, the exact same FONSI reissuance as has occurred at SpaceX's other pads as and when needed to support increased flight rates would occur.

You say that like it is nothing.  How many days could a FONSI reissuance take, assuming it is granted?

A limit has been set.  To change it they have to do another FONSI reissuance.

Another WR wouldn't be enough?