LV0010 LAUNCH INVESTIGATION UPDATESEPTEMBER 28, 2022The team has made significant progress in the investigation into the LV0010 early shutdown of the upper stage. Our investigation process consists of four core steps:Flight Data ReviewTimeline ReconstructionFault Tree AnalysesImplementing Corrective and Preventative ActionsWe have completed steps #1 and #2, and are nearing completion of step #3. We’ve determined that the upper stage shut down early due to a higher-than-normal fuel consumption rate. Through the review of flight data, reconstruction of flight timelines, and the construction of an extensive fault tree, we have narrowed the root cause to an issue with the upper stage engine. We have also completed many rounds of ground testing, including multiple tests that yielded results consistent with the failure condition in flight.The team is conducting additional experiments to verify the root cause before wrapping up the investigation with the FAA. We are focused on conducting an exhaustive investigation and ensuring that we extract all lessons learned.Once the investigation is finished, we look forward to sharing our lessons learned in a future blog post.
Astra has concluded the TROPICS-1 mishap investigation and received our formal closure letter from the @FAANews. Read more about the conclusion here:
CONCLUSION OF TROPICS-1 MISHAP INVESTIGATIONMARCH 1, 2023By Andrew Griggs, Head of Mission Assurance, and Adam London, Founder and Chief Technical OfficerAfter over six months of rigorous testing and analysis, Astra has received a formal closure letter from the FAA concluding the TROPICS-1 mishap investigation. At this time, we’d like to share our conclusions from the investigation and lessons learned.The TROPICS-1 mission launched on June 12, 2022 on Astra’s Rocket 3.3, serial number LV0010. The rocket completed a nominal first stage flight, stage separation, and upper stage ignition. Shortly after the ignition of the upper stage engine, the upper stage’s fuel consumption rate increased and remained anomalously high for the remainder of the flight. About 250 seconds after upper stage ignition, the stage exhausted its fuel supply with approximately 20% of the liquid oxygen still remaining onboard. As a result, the upper stage was only able to obtain about 80% of the required orbital velocity. The stage was unable to deliver its payloads to orbit, and subsequently re-entered the atmosphere, ending the mission.Our analysis showed that the anomalous fuel consumption during the upper stage flight was due to a combustion chamber wall burn-through that occurred 18 seconds into upper stage flight. Flight data showed that the burn-through was precipitated by a substantial blockage of the fuel injector. The mechanics of combustion and regenerative cooling are complex and this failure did not have an immediately apparent root cause, so extensive testing and analysis was required to recreate the failure mode and to understand both how the injector blockage was created and how it led to a burn-through.About regenerative coolingLet’s begin with a quick primer on regenerative cooling. Most liquid rocket engines require cooling to prevent the very hot combustion gases from melting the chamber wall and causing the engine to fail. Rocket 3.3’s upper stage engine used regenerative cooling to accomplish this. In a regeneratively-cooled rocket engine, cooling is achieved by routing the fuel through many cooling channels embedded within the combustion chamber wall. This allows heat from the wall to be absorbed into the flowing fuel, keeping the wall at a low enough temperature to prevent failure.On TROPICS-1, a failure of the regenerative cooling system led to a chamber burn-through.What caused the burn-through?Astra concluded that the primary factor leading to the combustion chamber burn-through was a partial blockage of the injector. When the fuel injector is partially blocked, the rate of fuel passing through the cooling channels decreases. This reduces the amount of heat the fuel can absorb and makes the combustion chamber wall hotter. If the wall gets hot enough, the temperature of the wall can exceed the local boiling point of the fuel, causing some of the fuel to boil along the wall inside the channels. Sometimes this condition can “self-heal”, because a small amount of boiling can actually enhance the cooling ability of the fuel, bringing the wall temperature back down. However, if too much of the fuel boils, its cooling capability is significantly and adversely impacted, and the wall temperature can go up and up until the wall fails and “burns through,” dumping a portion of the fuel flow directly into the combustion chamber – essentially wasting it. This is what we determined had occurred during the TROPICS-1 launch. In addition to the partial blockage of the injector, Astra determined that a secondary factor for the burn-through was thermal barrier coating erosion. Portions of the upper stage engine’s combustion chamber have a thermal barrier coating on the inside to insulate the chamber wall and reduce the heat that the fuel is required to absorb as a coolant. If some thermal barrier coating is missing, even a very small amount, that portion of the wall can get much hotter and increase the likelihood of a local burn-through. During the investigation, Astra found that there was a small amount of missing thermal barrier coating on the LV0010 upper stage engine. This missing coating was in a location that was considered acceptable by engineers at the time, but further analysis showed that we had underestimated the need for coating in this region under flight conditions (more on the ground vs. flight differences in the next section).What caused the injector blockage?While it was relatively straightforward to determine that a blockage of the injector had occurred during upper stage flight, it took much longer to conclusively determine what had caused the blockage. We used a combination of analysis and testing to systematically investigate each potential blockage source. Three credible sources for the blockage were the focus of this investigation:Foreign object debris (“FOD”), such as particles of metalGaseous heliumGaseous fuelAfter a review of flight results and testing to recreate the failure, we were able to conclude that the injector blockage was caused by a gas. This ruled out solid foreign object debris such as metal particles.Next we examined helium, which is used for the upper stage’s pneumatic and pressurization systems, and theoretically could have leaked into the fuel lines. We performed a barrage of tests on our pneumatic systems, attempting to cause helium to leak under flight-like conditions of vacuum, vibration, and low temperatures. We were unable to substantiate any meaningful leaks, nor did data from the LV0010 upper stage indicate any leaks before or during flight. The only other credible source of helium in the upper stage is in the pressurization system, found at the start of upper stage flight in a small “ullage” bubble at the top of the fuel tank. Analysis showed it is highly unlikely that this helium could migrate to the bottom of the tank and be ingested into the engine at the beginning of the burn. Even if it had migrated, it’s even more unlikely that this helium could have remained in the bottom of the tank and sustained the injector blockage for the amount of time seen in flight (helium, since it’s much lighter than fuel, tends to migrate toward the top of the tank as soon as the engine lights and the stage begins accelerating).So, we were left with gaseous fuel as the main suspect. During ground acceptance testing, the fuel in the upper stage engine gets warm but we had never observed boiling or near-boiling within the cooling channels. However, the exhaust jet of the upper stage engine on the ground is “separated” from the inside of the engine nozzle by the pressure of the atmosphere around it, and therefore transfers less heat into the fuel. In flight, the engine is surrounded by vacuum and the exhaust jet expands to become “fully attached” to the inside of the nozzle. Therefore, the fuel passing through the engine during flight is heated to a higher temperature than during ground testing.We conducted numerous experimental engine tests with fuel pre-heated to various temperatures to simulate the effect of full attachment. This allowed us to create a more sophisticated thermal model to predict the temperature of the fuel inside the engine with better accuracy than before. This analysis showed that, in flight, the fuel at the injector would have thin margins with respect to its boiling point. The most significant contributors to this thin margin were unique to the Rocket 3 architecture: a pressure-fed upper stage engine that operates at relatively low pressures, as well as the selection of a kerosene-like fuel with a higher vapor pressure than traditional rocket-grade kerosene (e.g. RP-1) to simplify testing and operations.Given this thin margin, small factors — like the warm sunny weather in Cape Canaveral on the day of launch, which meant the fuel was slightly warmer than in prior flights – helped to tip the fuel over into a boiling regime on the TROPICS-1 mission. Our analysis has concluded that the boiling fuel caused the partial injector blockage in flight and, together with the eroded thermal barrier coating mentioned previously, caused a thermal “run-away” event that worsened the injector blockage and eventually led to the burn-through.Lessons learnedAlmost immediately following the TROPICS-1 mishap, we made the strategic decision to focus the majority of Astra’s resources on developing Astra’s next-generation launch vehicle: Rocket 4. We focused the TROPICS-1 investigation on learning all that we could to inform the design and operation of this new, larger, and more reliable rocket. To that end, Rocket 4 incorporates key architectural choices (most notably, a different upper stage engine design and a different fuel) that completely eliminate the causes of this mishap. We have also introduced controls designed to eliminate a number of other potential failure modes (like FOD and helium ingestion) that the investigation concluded did not occur on the TROPICS-1 flight, but that Rocket 3.3 could have been susceptible to. For example, we are upgrading our helium diffuser design to prevent frothing in the propellant tanks and ingestion of helium into the engine.In parallel with the technical cause investigation, Astra also conducted an internal investigation related to improving our processes, systems, and culture to increase the reliability of our fourth generation rocket. Astra has come a long way from the company that we were when we designed Rocket 3 back in 2018 and 2019; our team is now larger and more experienced, we have made significant investments in quality control and failure analysis, and we have learned vital lessons from both launch successes and failures. Still, we know that we have room to improve further, so we are implementing dozens of company-wide initiatives designed to ensure the reliability of Rocket 4. These improvements include an overhauled design review process, a more robust test-like-you-fly qualification process, and a refreshed set of Astra core values. I’m confident we now have the right team and systems in place to make Rocket 4 a success.This was easily the most complex investigation that Astra has ever conducted. We were committed to rigorously and thoroughly investigating the cause of this mishap, and produced an incredible amount of analysis and test results to understand the failure and support our conclusions. Although we came to the preliminary conclusion of an injector blockage and a chamber burn-through quite quickly, we spent several additional months to ensure that we learned everything we could from this launch failure and left no stone unturned. I am thankful for the team’s hard work and we are grateful for the support and partnership of NASA and the FAA throughout this process. We are putting these lessons into action as we prepare for the first launch of Rocket 4.
Schematic of regeneratively cooled rocket engine combustion chamber
Comparison of (ground) separated rocket engine flow vs (vacuum) fully-attached rocket engine flow
Upper stage engine testing shows streaks of molten metal from a hot wall burn-through event
Underestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.
Really appreciate reading this review. When @nasa launched #TROPICS-1 with @Astra, more risk was taken, which is what is necessary to get new capabilities online. As a minimum, the outcome of such experiments has to be learning. Mission success can still be achieved w 4 sats.
Quote from: ZachS09 on 03/01/2023 11:06 pmUnderestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.
Quote from: TrevorMonty on 03/02/2023 07:48 amQuote from: ZachS09 on 03/01/2023 11:06 pmUnderestimating the amount of thermal barrier coating? Sounds like an instance of bad quality control to me.Primary cause of failure was injector blockage, this reduced fuel cooling bell resulting in burn through at spot with reduced thermal barrier coating. If injector hadn't blocked then may never have burnt through. Could also blame fuel ie kerosene instead of RP1.The reason it boiled was because they never did proper testing with fully attached flow in the nozzle. The amount of heat you pick up with regen is highly variable; you can make estimates, but the only way to actually know how got things get is to test it. They should have found this out in dev testing. Also pretty sure they use the same kerosene as XCOR, and we never had any issues with it.
The reason it boiled was because they never did proper testing with fully attached flow in the nozzle. The amount of heat you pick up with regen is highly variable; you can make estimates, but the only way to actually know how got things get is to test it. They should have found this out in dev testing. Also pretty sure they use the same kerosene as XCOR, and we never had any issues with it.
