Schedule Pressure Increases as Contractors Delay Key Events
On page 20 there is an example saying Crew Dragon's TPS has been identified as a high risk hazard. Wonder why that is?
As a result, the program’s quality assurance team witnessed SpaceX engineers installing spherical tiles on their crew capsule’s heat shield to verify the required process was followed.
NASA has also made changes to the contracts that have increased their value. While the contracts are fixed-price, their values can increase if NASA adds to the scope of the work or otherwise changes requirements. As of October 2016, NASA had increased the value of contract line item 001 for Boeing by $47 million for hardware and software requirement changes, and contract line item 001 for SpaceX by $91 million for a hardware requirement change and the addition of cargo during an ISS test flight.In addition to its budget, NASA submits quarterly reports to the appropriations committees on the Commercial Crew Program’s progress that include cost and schedule updates from each contractor and information on contractor specific-risks, including procurement-sensitive material that is not publically releasable.SpaceX officials told us they completed integrated testing and qualification of several major spacecraft components, including its environmental control and life support system—which controls air quality and temperature, among other functions, to ensure crew survivability—and its spacesuit in November 2016.As of October 2016, the program’s schedule risk analysis indicated that both contractors’ certification dates would likely slip into early 2019.In 2015, the United States modified its contract with Roscosmos to provide crew transportation to the ISS for six astronauts through 2018 with rescue and return through late spring 2019. The contract extension was valued at $491 million or approximately $82 million per seat. NASA’s contract with Roscosmos permits it to delay the use of the final seat by up to 6 months to late spring 2019, with a return flight approximately 6 months later.The ISS program office stated that NASA could construct another “year in space” mission, similar to the mission undertaken by NASA astronaut Scott Kelly. According to the ISS program office, NASA and Roscosmos are discussing this option; however, there is no agreement in place. Under this scenario, an astronaut could begin a mission near the end of calendar year 2018 using one of the final Soyuz seats and would return to Earth via one of the Commercial Crew Program contractors’ crew transportation systems approximately one year later in 2019.In March 2016, Boeing modified its previously approved parachute test plan by replacing six drop tests, which simulate select forces—for example, mass—on the parachute system for one full-scale test event, which simulates all aspects of a parachute system. Through discussions with the program, Boeing has increased the number of full-scale test events to five, with an option for two additional tests if deemed necessary.Finally, both the program and a NASA advisory group consider SpaceX’s plan to fuel the launch vehicle after the astronauts are on board the spacecraft to be a potential safety risk. SpaceX’s perspective is that this operation may be a lower risk to the crew; NASA and SpaceX’s risk evaluation is ongoing.SpaceX requested, and the program approved, proposals to split its critical design review into three reviews because portions of its design had not been ready at previous reviews. ... SpaceX’s final planned design review was held in August 2016; however, the program reported that a number of outstanding areas, primarily related to ground systems, still needed to be reviewed. SpaceX officials told us these areas were reviewed in November 2016. Further, according to SpaceX, these separate reviews were in order to perform review of designs that were completed earlier than anticipated, to allow SpaceX and NASA teams to focus in greater detail on certain systems, and to accommodate design updates driven in part by changes to NASA requirements.Under each contract, the current loss of crew requirement is 1 in 270, meaning that the contractors’ systems must carry no more than a 1 in 270 probability of incurring loss of crew. Near the end of the Space Shuttle program, the probability of loss of crew was approximately 1 in 90. Program officials told us that Commercial Crew is the first NASA program that the agency will evaluate against a probabilistic loss of crew requirement. They said that if the contractors cannot meet the agency loss of crew requirement at 1 in 270, NASA could still certify their systems by employing operational alternatives. This would entail a potentially increased level of risk or uncertainty related to the level of risk for the crew.The Commercial Crew Program gained visibility into the contractors’ parachute qualification testing plans through technical forums. As a result, the program has proposed six additional tests for each contractor to increase the reliability of their parachute systems.The Commercial Crew Program has developed productive working relationships with both contractors, but the level of visibility that the program has required thus far has also taken more time than the program or contractors anticipated. The program’s standing review board has stated that the contract is structured to allow NASA unprecedented levels of visibility, but that it was intended to be used primarily for high-risk areas. However, the standing review board found, and both contractors told us, that the program has requested high levels of visibility on most items and there are signs that the contractors’ patience is waning. Both contractors expressed concerns that the program requests more interaction and data than they originally anticipated at the time of the contract award. For example, Boeing and SpaceX officials told us that the program often requests additional in-person engagement with their engineers, such as repeat presentations to multiple boards on the same technical issue, and has also asked for the same data in multiple formats or from multiple stakeholders.A verification closure notice is required for each of the 332 ISS requirements, which are applicable to anyone who docks with the station, and the 280 Commercial Crew Program requirements.In order to ensure that the United States has continued access to the ISS if the Commercial Crew Program’s contractors experience additional schedule delays, we recommend the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results.
Sustaining Program’s Level of Visibility Might Be Difficult as Schedule Pressure BuildsThe Commercial Crew Program has developed productive working relationships with both contractors, but the level of visibility that the program has required thus far has also taken more time than the program or contractors anticipated. The program’s standing review board has stated that the contract is structured to allow NASA unprecedented levels of visibility, but that it was intended to be used primarily for high-risk areas. However, the standing review board found, and both contractors told us, that the program has requested high levels of visibility on most items and there are signs that the contractors’ patience is waning. Both contractors expressed concerns that the program requests more interaction and data than they originally anticipated at the time of the contract award. For example, Boeing and SpaceX officials told us that the program often requests additional in-person engagement with their engineers, such as repeat presentations to multiple boards on the same technical issue, and has also asked for the same data in multiple formats or from multiple stakeholders. Program officials told us that they are constantly working to find a balance between obtaining the visibility they need to be able to eventually certify the crew transportation systems for human spaceflight while giving the contractors room to independently work through issues for their systems.
Program Office Workload Is an Emerging Schedule RiskProgram officials told us that one of their greatest upcoming challenges will be to keep pace with the contractors’ schedules so that the program does not delay certification. Specifically, they told us they are concerned about an upcoming “bow wave” of work because the program must complete two oversight activities—phased safety reviews and verification closure notices—concurrently in order to support the contractors’ ISS design certification reviews, uncrewed and crewed flight test missions, and final certification.
The Commercial Crew Program’s review and approval of the contractors’ hazard reports have taken longer than planned. The program originally planned to complete phase two in early 2016 but currently does not expect to complete this phase until June 2017. The Commercial Crew Program has a goal of reviewing hazard reports within 8 weeks of receiving them, but a recent report by the NASA Office of Inspector General found that the reviews are taking longer than anticipated and a backlog has developed. In response to the Inspector General’s report, NASA officials noted that, while the timeliness of the hazard review process is important, what is more important is having thorough, detailed hazard reports in order to understand safety risks and ensuring the safety of each system. As of October 2016, the Commercial Crew Program had approved 117 of the anticipated 195 hazard reports and planned to approve approximately half of the remaining reports for both contractors by the end of 2016. Program officials told us that the hazard reports that are still open are related to items that they would not expect to be closed because they involve some of the more complicated design work that the contractors have not yet finalized. Program officials also pointed out other ways that the contractors have contributed to phase two delays, including receiving incomplete hazard reports that required several iterations before the program could begin its formal review.
Gaining this level of visibility has taken more time than the program or contractors anticipated, but the early upfront investment in time should ultimately make the certification process go smoother. In addition, while the Commercial Crew Program should be mindful about placing undue burdens on its contractors, it ultimately has the responsibility for ensuring the safety of U.S. astronauts, and its contracts with Boeing and SpaceX give it deference to determine the level of insight required to do so.
Recommendation 1: In order to ensure that the United States has continued access to the ISS if the Commercial Crew Program’s contractors experience additional schedule delays, we recommend the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results.
Management’s Response: NASA concurs with the recommendation. NASA will develop a contingency plan for maintaining a presence on the ISS beyond 2018 if the Commercial Crew Program’s partners experience additional schedule delays. NASA anticipates the contiengency plan to be ready in approximately six weeks. Estimated Completion Date: March 13, 2017
Quote from: Folgers25 on 02/16/2017 08:45 pmOn page 20 there is an example saying Crew Dragon's TPS has been identified as a high risk hazard. Wonder why that is?It's high-risk because failure of TPS would have a very high chance of causing a crew fatality. In context, they're using it as an example to show how the risk is being mitigated. Hence the next line in the report:QuoteAs a result, the program’s quality assurance team witnessed SpaceX engineers installing spherical tiles on their crew capsule’s heat shield to verify the required process was followed.
The most disappointing part of the report in my view is that NASA can't keep its hands-off. A major goal of the commercial cargo and crew programs was to re-orientate NASA's relationship with the launch providers.
You're conflating risk with criticality. Criticality explores the expected outcome with the assumption that a given component has failed. Risk includes the likelihood of the component failing.
Quote from: rockets4life97 on 02/17/2017 03:34 pmThe most disappointing part of the report in my view is that NASA can't keep its hands-off. A major goal of the commercial cargo and crew programs was to re-orientate NASA's relationship with the launch providers.No, NASA already has re-oriented its relationship with the launch providers. The goal of the crew program was to use similar processes.
Quote from: SWGlassPit on 02/17/2017 03:25 pmYou're conflating risk with criticality. Criticality explores the expected outcome with the assumption that a given component has failed. Risk includes the likelihood of the component failing.I won't argue with your definition, but I was simply using the language from the report.
Commercial cargo: re-orientate relationship with launch providers (accomplished)
Quote from: rockets4life97 on 02/17/2017 03:47 pmCommercial cargo: re-orientate relationship with launch providers (accomplished)No, existing launch services for science spacecraft has re-orientated relationship with launch providers since the 90's. This was needs to be applied to crew.Commercial cargo is an outlier. It is for low risk logistics.
The most disappointing part of the report in my view is that NASA can't keep its hands-off. A major goal of the commercial cargo and crew programs was to re-orientate NASA's relationship with the launch providers. I hope NASA takes the GAO concerns seriously and considers how they can move towards a lighter touch, while still getting to a high-standard for certification.
This pivot back to a traditional insight/oversight system for the final stage of the Crew Program was 100% intentional.