Author Topic: FAILURE : Astra R3.3 - LV0010 - TROPICS Flight 1 - CCSFS SLC-46 - 12 June 2022  (Read 38997 times)

Offline FutureSpaceTourist

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 48463
  • UK
    • Plan 28
  • Liked: 82196
  • Likes Given: 37085
https://astra.com/news/lv0010-launch-investigation-update/

Quote
LV0010 LAUNCH INVESTIGATION UPDATE
SEPTEMBER 28, 2022

The team has made significant progress in the investigation into the LV0010 early shutdown of the upper stage. Our investigation process consists of four core steps:

Flight Data Review
Timeline Reconstruction
Fault Tree Analyses
Implementing Corrective and Preventative Actions

We have completed steps #1 and #2, and are nearing completion of step #3. We’ve determined that the upper stage shut down early due to a higher-than-normal fuel consumption rate. Through the review of flight data, reconstruction of flight timelines, and the construction of an extensive fault tree, we have narrowed the root cause to an issue with the upper stage engine. We have also completed many rounds of ground testing, including multiple tests that yielded results consistent with the failure condition in flight.

The team is conducting additional experiments to verify the root cause before wrapping up the investigation with the FAA. We are focused on conducting an exhaustive investigation and ensuring that we extract all lessons learned.

Once the investigation is finished, we look forward to sharing our lessons learned in a future blog post.

Offline FutureSpaceTourist

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 48463
  • UK
    • Plan 28
  • Liked: 82196
  • Likes Given: 37085
https://twitter.com/astra/status/1631052117262888961

Quote
Astra has concluded the TROPICS-1 mishap investigation and received our formal closure letter from the @FAANews. Read more about the conclusion here:

https://astra.com/news/conclusion-tropics-1-mishap-investigation/

Quote
CONCLUSION OF TROPICS-1 MISHAP INVESTIGATION
MARCH 1, 2023

By Andrew Griggs, Head of Mission Assurance, and Adam London, Founder and Chief Technical Officer

After over six months of rigorous testing and analysis, Astra has received a formal closure letter from the FAA concluding the TROPICS-1 mishap investigation. At this time, we’d like to share our conclusions from the investigation and lessons learned.

The TROPICS-1 mission launched on June 12, 2022 on Astra’s Rocket 3.3, serial number LV0010. The rocket completed a nominal first stage flight, stage separation, and upper stage ignition. Shortly after the ignition of the upper stage engine, the upper stage’s fuel consumption rate increased and remained anomalously high for the remainder of the flight. About 250 seconds after upper stage ignition, the stage exhausted its fuel supply with approximately 20% of the liquid oxygen still remaining onboard. As a result, the upper stage was only able to obtain about 80% of the required orbital velocity. The stage was unable to deliver its payloads to orbit, and subsequently re-entered the atmosphere, ending the mission.

Our analysis showed that the anomalous fuel consumption during the upper stage flight was due to a combustion chamber wall burn-through that occurred 18 seconds into upper stage flight. Flight data showed that the burn-through was precipitated by a substantial blockage of the fuel injector. The mechanics of combustion and regenerative cooling are complex and this failure did not have an immediately apparent root cause, so extensive testing and analysis was required to recreate the failure mode and to understand both how the injector blockage was created and how it led to a burn-through.

About regenerative cooling

Let’s begin with a quick primer on regenerative cooling. Most liquid rocket engines require cooling to prevent the very hot combustion gases from melting the chamber wall and causing the engine to fail. Rocket 3.3’s upper stage engine used regenerative cooling to accomplish this. In a regeneratively-cooled rocket engine, cooling is achieved by routing the fuel through many cooling channels embedded within the combustion chamber wall. This allows heat from the wall to be absorbed into the flowing fuel, keeping the wall at a low enough temperature to prevent failure.

On TROPICS-1, a failure of the regenerative cooling system led to a chamber burn-through.

What caused the burn-through?

Astra concluded that the primary factor leading to the combustion chamber burn-through was a partial blockage of the injector. When the fuel injector is partially blocked, the rate of fuel passing through the cooling channels decreases. This reduces the amount of heat the fuel can absorb and makes the combustion chamber wall hotter. If the wall gets hot enough, the temperature of the wall can exceed the local boiling point of the fuel, causing some of the fuel to boil along the wall inside the channels. Sometimes this condition can “self-heal”, because a small amount of boiling can actually enhance the cooling ability of the fuel, bringing the wall temperature back down. However, if too much of the fuel boils, its cooling capability is significantly and adversely impacted, and the wall temperature can go up and up until the wall fails and “burns through,” dumping a portion of the fuel flow directly into the combustion chamber – essentially wasting it. This is what we determined had occurred during the TROPICS-1 launch.

In addition to the partial blockage of the injector, Astra determined that a secondary factor for the burn-through was thermal barrier coating erosion. Portions of the upper stage engine’s combustion chamber have a thermal barrier coating on the inside to insulate the chamber wall and reduce the heat that the fuel is required to absorb as a coolant. If some thermal barrier coating is missing, even a very small amount, that portion of the wall can get much hotter and increase the likelihood of a local burn-through. During the investigation, Astra found that there was a small amount of missing thermal barrier coating on the LV0010 upper stage engine. This missing coating was in a location that was considered acceptable by engineers at the time, but further analysis showed that we had underestimated the need for coating in this region under flight conditions (more on the ground vs. flight differences in the next section).

What caused the injector blockage?
While it was relatively straightforward to determine that a blockage of the injector had occurred during upper stage flight, it took much longer to conclusively determine what had caused the blockage. We used a combination of analysis and testing to systematically investigate each potential blockage source. Three credible sources for the blockage were the focus of this investigation:

Foreign object debris (“FOD”), such as particles of metal
Gaseous helium
Gaseous fuel

After a review of flight results and testing to recreate the failure, we were able to conclude that the injector blockage was caused by a gas. This ruled out solid foreign object debris such as metal particles.

Next we examined helium, which is used for the upper stage’s pneumatic and pressurization systems, and theoretically could have leaked into the fuel lines. We performed a barrage of tests on our pneumatic systems, attempting to cause helium to leak under flight-like conditions of vacuum, vibration, and low temperatures. We were unable to substantiate any meaningful leaks, nor did data from the LV0010 upper stage indicate any leaks before or during flight. The only other credible source of helium in the upper stage is in the pressurization system, found at the start of upper stage flight in a small “ullage” bubble at the top of the fuel tank. Analysis showed it is highly unlikely that this helium could migrate to the bottom of the tank and be ingested into the engine at the beginning of the burn. Even if it had migrated, it’s even more unlikely that this helium could have remained in the bottom of the tank and sustained the injector blockage for the amount of time seen in flight (helium, since it’s much lighter than fuel, tends to migrate toward the top of the tank as soon as the engine lights and the stage begins accelerating).

So, we were left with gaseous fuel as the main suspect. During ground acceptance testing, the fuel in the upper stage engine gets warm but we had never observed boiling or near-boiling within the cooling channels. However, the exhaust jet of the upper stage engine on the ground is “separated” from the inside of the engine nozzle by the pressure of the atmosphere around it, and therefore transfers less heat into the fuel. In flight, the engine is surrounded by vacuum and the exhaust jet expands to become “fully attached” to the inside of the nozzle. Therefore, the fuel passing through the engine during flight is heated to a higher temperature than during ground testing.

We conducted numerous experimental engine tests with fuel pre-heated to various temperatures to simulate the effect of full attachment. This allowed us to create a more sophisticated thermal model to predict the temperature of the fuel inside the engine with better accuracy than before. This analysis showed that, in flight, the fuel at the injector would have thin margins with respect to its boiling point. The most significant contributors to this thin margin were unique to the Rocket 3 architecture: a pressure-fed upper stage engine that operates at relatively low pressures, as well as the selection of a kerosene-like fuel with a higher vapor pressure than traditional rocket-grade kerosene (e.g. RP-1) to simplify testing and operations.

Given this thin margin, small factors — like the warm sunny weather in Cape Canaveral on the day of launch, which meant the fuel was slightly warmer than in prior flights – helped to tip the fuel over into a boiling regime on the TROPICS-1 mission. Our analysis has concluded that the boiling fuel caused the partial injector blockage in flight and, together with the eroded thermal barrier coating mentioned previously, caused a thermal “run-away” event that worsened the injector blockage and eventually led to the burn-through.

Lessons learned

Almost immediately following the TROPICS-1 mishap, we made the strategic decision to focus the majority of Astra’s resources on developing Astra’s next-generation launch vehicle: Rocket 4. We focused the TROPICS-1 investigation on learning all that we could to inform the design and operation of this new, larger, and more reliable rocket. To that end, Rocket 4 incorporates key architectural choices (most notably, a different upper stage engine design and a different fuel) that completely eliminate the causes of this mishap. We have also introduced controls designed to eliminate a number of other potential failure modes (like FOD and helium ingestion) that the investigation concluded did not occur on the TROPICS-1 flight, but that Rocket 3.3 could have been susceptible to. For example, we are upgrading our helium diffuser design to prevent frothing in the propellant tanks and ingestion of helium into the engine.

In parallel with the technical cause investigation, Astra also conducted an internal investigation related to improving our processes, systems, and culture to increase the reliability of our fourth generation rocket. Astra has come a long way from the company that we were when we designed Rocket 3 back in 2018 and 2019; our team is now larger and more experienced, we have made significant investments in quality control and failure analysis, and we have learned vital lessons from both launch successes and failures. Still, we know that we have room to improve further, so we are implementing dozens of company-wide initiatives designed to ensure the reliability of Rocket 4. These improvements include an overhauled design review process, a more robust test-like-you-fly qualification process, and a refreshed set of Astra core values. I’m confident we now have the right team and systems in place to make Rocket 4 a success.

This was easily the most complex investigation that Astra has ever conducted. We were committed to rigorously and thoroughly investigating the cause of this mishap, and produced an incredible amount of analysis and test results to understand the failure and support our conclusions. Although we came to the preliminary conclusion of an injector blockage and a chamber burn-through quite quickly, we spent several additional months to ensure that we learned everything we could from this launch failure and left no stone unturned. I am thankful for the team’s hard work and we are grateful for the support and partnership of NASA and the FAA throughout this process. We are putting these lessons into action as we prepare for the first launch of Rocket 4.

Captions:

Quote
Schematic of regeneratively cooled rocket engine combustion chamber

Quote
Comparison of (ground) separated rocket engine flow vs (vacuum) fully-attached rocket engine flow

Quote
Upper stage engine testing shows streaks of molten metal from a hot wall burn-through event
« Last Edit: 03/01/2023 09:06 pm by FutureSpaceTourist »

Online ZachS09

  • Space Savant
  • Senior Member
  • *****
  • Posts: 8431
  • Roanoke, TX
  • Liked: 2367
  • Likes Given: 2066
Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Liftoff for St. Jude's! Go Dragon, Go Falcon, Godspeed Inspiration4!

Offline Exastro

  • Full Member
  • *
  • Posts: 171
  • USA
  • Liked: 142
  • Likes Given: 101
Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Sounds more like a design error than a QC issue: they thought their thermal margin was higher than it turned out to be, because their testing (at sea-level pressure) didn't result in as much heating of the chamber wall as happens in flight conditions.  That gave them false confidence that their thermal barrier was thick enough.  In fact, all their flights were operating at the edge of failure, and it took only a small perturbation, like a barrier thinner than usual and a launch site warmer than Kodiak, to push them over that edge.

I think it's admirable that they took the time to publicly describe the flaw and the process they went through to find and confirm it.  They think they've learned the right lessons from this experience, and based on their description I think they're probably right.

Best wishes for a successful return to flight, with a bigger, better rocket.


« Last Edit: 03/02/2023 03:07 am by Exastro »

Offline FutureSpaceTourist

  • Global Moderator
  • Senior Member
  • *****
  • Posts: 48463
  • UK
    • Plan 28
  • Liked: 82196
  • Likes Given: 37085
NASA’s (recently) former head of science:

https://twitter.com/dr_thomasz/status/1631176541265711105

Quote
Really appreciate reading this review.

When @nasa launched #TROPICS-1 with @Astra, more risk was taken, which is what is necessary to get new capabilities online. As a minimum, the outcome of such experiments has to be learning. Mission success can still be achieved w 4 sats.

Offline TrevorMonty

Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.

Offline Gliderflyer

Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.

The reason it boiled was because they never did proper testing with fully attached flow in the nozzle. The amount of heat you pick up with regen is highly variable; you can make estimates, but the only way to actually know how got things get is to test it. They should have found this out in dev testing. Also pretty sure they use the same kerosene as XCOR, and we never had any issues with it.
I tried it at home

Online ZachS09

  • Space Savant
  • Senior Member
  • *****
  • Posts: 8431
  • Roanoke, TX
  • Liked: 2367
  • Likes Given: 2066
Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.

I saw what the primary cause of failure was. It’s just that when I saw the line of Astra underestimating the amount of thermal barrier coating, I misinterpreted the situation as bad quality control because I didn’t understand the difference between QC and design error.
« Last Edit: 03/02/2023 12:40 pm by ZachS09 »
Liftoff for St. Jude's! Go Dragon, Go Falcon, Godspeed Inspiration4!

Offline edzieba

  • Virtual Realist
  • Senior Member
  • *****
  • Posts: 6171
  • United Kingdom
  • Liked: 9430
  • Likes Given: 39
Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.

The reason it boiled was because they never did proper testing with fully attached flow in the nozzle. The amount of heat you pick up with regen is highly variable; you can make estimates, but the only way to actually know how got things get is to test it. They should have found this out in dev testing. Also pretty sure they use the same kerosene as XCOR, and we never had any issues with it.
Astra also never had any problems with that fuel. Until they did.
This sounds like a swiss-cheese failure mode - lots of little things that individually were within margin or served to non-critically reduce margin (the thinned TBC, the flow attachment, the fuel used, the environmental temperature at launch) that all together resulted in failure.

Offline ParabolicSnark

  • Full Member
  • *
  • Posts: 148
  • CA
  • Liked: 189
  • Likes Given: 124
The reason it boiled was because they never did proper testing with fully attached flow in the nozzle. The amount of heat you pick up with regen is highly variable; you can make estimates, but the only way to actually know how got things get is to test it. They should have found this out in dev testing. Also pretty sure they use the same kerosene as XCOR, and we never had any issues with it.

Testing with fully attached flow in the nozzle for a vacuum-optimized engine is easier said than done. You need either a vacuum chamber large enough to operate a rocket engine in or a huge diffuser you can use to locally drop the pressure near the exhaust. Both are quite expensive to design and/or operate. NASA has such vacuum chambers, but contracting to use one for your engine program can cost you in the ballpark of $1MM per test. For these smaller launch vehicles, that typically trades unfavorably and they make the risk-call to fly without the test and collect data from test (or operational) flights.

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1