SpaceX COTS Demo 2/3 Updates (THREAD 2)

[dcporter]

However, after the May 17 docking of Soyuz TMA-04M, there is a long period, possibly two months, with no other traffic.  Much as I hate to say it, that span looks to me like a good target for COTS-2/3 if they don't go in early April.

The tail end of your two-month span is my birthday. A fella can dream.

[mmeijeri]

Obviously Spacex underestimated how much testing they would have to do but I wonder if their are actually major issues that have been popping up or if they are just being very thorough. 

Most of that should have been automated, and at least some of it must have been.

[zerm]

Back in early January I contacted an old friend of mine at the Cape to tell him I'd gotten my credentials to cover the SpaceX COTS2/3 launch and I'd drop in and see him.

He replied "Great! See ya' in 6 months."

I hope for sooner.

[Antares]

I'd bet it's more of a collection of minor issues that really just manifest themselves as stuff taking longer than expected.

[docmordrid]

That and an abundance of caution given the stakes and Russia's annus horribilis. Everyone in the funding chain for CCDev is watching.

[oldAtlas_Eguy]

NOTE: The posts that follow are speculations about what happened and what needs to happen regarding the software based on experience as retired government software SRB chariman.

The Progress failure still is causing a cascading effect on other ISS activities. Without the hardware and software ISS upgrades which were originally scheduled to be completed in Oct-Nov timeframe the all up integrated operations test/simulation could not be done which is the test that showed SpaceX had some more work to do to resolve problems. You don’t know you have any problems until you test. That would have caused the same length of slip 2-3 months as it has now but would have been a slip from Dec /Jan launch to a Feb-Mar instead of the slip from Feb to May-Jun.

In another item SpaceX software test simulation lab needs to be upgraded to address deficiencies in their testing of the Dragon software, which is part of the current delay. Without the changes to the simulation lab (which is probably an all virtual sim model) the software cannot be tested to see if the problems have been fixed. The sim model which is the rest of what the real world reactions to software output and input occur is driven by observations of real world and the modeling of it. Currently little to no real world data exists for the ISS interaction portion of the Dragon operations. So the model is best guesses and the all up test showed that some of those guesses were wrong. SpaceX is probably now doing an extensive review to determine if other assumptions incorporated into the simulation could also be wrong. A sim model that is wrong will cause software to be developed that in the real world will not work!

[mmeijeri]

In another item SpaceX software test simulation lab needs to be upgraded to address deficiencies in their testing of the Dragon software, which is part of the current delay.

This deserves more attention than it's getting. This sounds like a process deficiency.

So the model is best guesses and the all up test showed that some of those guesses were wrong.

Well, all models are no more than approximations and the only real "all-up" test is a full test flight. But I'm wondering how those problems were detected. If NASA detected problems through better simulation, then why are NASA's simulations better than SpaceX's? It's not a big deal if one or two problems slip through, but if it causes major delays it suggests something is wrong with the process.

[happyflower]

In another item SpaceX software test simulation lab needs to be upgraded to address deficiencies in their testing of the Dragon software, which is part of the current delay. Without the changes to the simulation lab (which is probably an all virtual sim model) the software cannot be tested to see if the problems have been fixed. The sim model which is the rest of what the real world reactions to software output and input occur is driven by observations of real world and the modeling of it. Currently little to no real world data exists for the ISS interaction portion of the Dragon operations. So the model is best guesses and the all up test showed that some of those guesses were wrong. SpaceX is probably now doing an extensive review to determine if other assumptions incorporated into the simulation could also be wrong. A sim model that is wrong will cause software to be developed that in the real world will not work!

Just asking, but wouldnt the HTV / ISS operations procedures be a good model for the Dragon berthing? Seems like good real world data to me.

[oldAtlas_Eguy]

In another item SpaceX software test simulation lab needs to be upgraded to address deficiencies in their testing of the Dragon software, which is part of the current delay.

This deserves more attention than it's getting. This sounds like a process deficiency.

Yes.

So the model is best guesses and the all up test showed that some of those guesses were wrong.

Well, all models are no more than approximations and the only real "all-up" test is a full test flight. But I'm wondering how those problems were detected. If NASA detected problems through better simulation, then why are NASA's simulations better than SpaceX's? It's not a big deal if one or two problems slip through, but if it causes major delays it suggests something is wrong with the process.

I believe NASA doing the simulation is a misconception in that NASA was a participant of a SpaceX test which included the ISS astronauts and actual ISS on orbit systems. What was in error as far as how the systems would perform was SpaceX’s assumptions about NASA systems and how they would interact with SpaceX systems and software. This was the first of such tests with a heavy NASA participation. This was the first time such a test could have been done since it needed the new systems (hardware software ISS upgrades) that didn’t previously exist on the ISS although they did exist on the ground in the ISS simulator. The problems should have been caught with tests with the ISS simulator the reason they were not is a question that should be answered.

Your own tests with your own hardware/software will always be less of a stress test than if it is tested with other real systems that it must interact with.

[krytek]

Please correct me if I'm wrong.
They feed it data which they "think" is close to the data it will receive on a regular flight, and see how the computer reacts to that.
In essence, either what they "think" is off, and therefore they can't test the software properly or both software and simulated conditions are off (more likely). Always wondered how they test stuff like that.

I'm guessing the simulation lab upgrade is probably software?

[mmeijeri]

The problems should have been caught with tests with the ISS simulator the reason they were not is a question that should be answered.

Exactly.

Your own tests with your own hardware/software will always be less of a stress test than if it is tested with other real systems that it must interact with.

Of course. As they say: end to end is further than you think. And whenever you discover a limitation in one layer of tests that was not discovered in earlier layers, you improve the earlier layers. The fact that a lot of flaws seem to have slipped through (judging by the delay) is what worries me. If the tests depend on new hardware / software on the ISS, then that might explain the delay. However, the question then becomes why earlier versions of the hardware / software weren't deployed earlier. Another question is to what extent these problems could have been caught in COTS 2 if COTS 2/3 had not been combined.

[woods170]

No doubt this whole development will make the end-to-end testing with Cygnus a whole lot more interesting. Big learning curve here with Dragon. The sensible thing to do is to apply any 'lessons-learned' to Cygnus to avoid a repeat.

[simpl simon]

No doubt this whole development will make the end-to-end testing with Cygnus a whole lot more interesting. Big learning curve here with Dragon. The sensible thing to do is to apply any 'lessons-learned' to Cygnus to avoid a repeat.

Maybe there will be a learning curve on the NASA side, but wouldn't SpaceX claim "proprietary" as far as its experiences are concerned?

[oldAtlas_Eguy]

No doubt this whole development will make the end-to-end testing with Cygnus a whole lot more interesting. Big learning curve here with Dragon. The sensible thing to do is to apply any 'lessons-learned' to Cygnus to avoid a repeat.

Maybe there will be a learning curve on the NASA side, but wouldn't SpaceX claim "proprietary" as far as its experiences are concerned?

The only thing NASA can really do for Cygnas is to move the scheduled full integrated ops sim farther to the left in the schedule so that problems can be resolved without affecting the launch date. That is the lessons learned that a too success oriented schedule means that there will be significant slips. The integrated ops sims were extreemly optimistic that there would not be problems.

[Antares]

Atlas E Guy: what's the source of your information?  You are posting things that it looks like only SpaceX or NASA people performing SpaceX insight would know or possibly industry gossip from SpaceX or NASA people you know.  Unless you are SpaceX, you don't have the data rights to post this information publicly - and even if you are, I doubt your hypothetical employer would want you doing so.

Commercial contracts are different than Shuttle, and I know it sucks for everyone who worships at the altar of Assange but we don't get to know everything.  We should respect the contracts - and the professionalism of what would we want for ourselves if the tables were turned.

[Chris Bergin]

Thread trim returned intact. Thanks for everyone's patience.

[Robotbeat]

Thread trim returned intact. Thanks for everyone's patience.

I really appreciate that you are being forthright about what you're doing, though.

[oldAtlas_Eguy]

Atlas E Guy: what's the source of your information?  You are posting things that it looks like only SpaceX or NASA people performing SpaceX insight would know or possibly industry gossip from SpaceX or NASA people you know.  Unless you are SpaceX, you don't have the data rights to post this information publicly - and even if you are, I doubt your hypothetical employer would want you doing so.

Commercial contracts are different than Shuttle, and I know it sucks for everyone who worships at the altar of Assange but we don't get to know everything.  We should respect the contracts - and the professionalism of what would we want for ourselves if the tables were turned.

Posts on how software is developed and tested is based on extensive knowledge both as a government manager of software projects/programs (Atlas E/F, 1980’s SDI Kill Vehicle Test Program and Shuttle avionics and ground support systems) for space LV’s and as a contractor software developer/manager for other projects/programs with the AF and commercial customers. I have additional experience on other commercial software development projects for commercial customers using software industry best practices for how to develop and test complex software systems. BTW the commercial systems where much more complex and orders of magnitude much more difficult to test than the space systems I had worked on 10 years previous requiring specialized software (much much simpler than the software being developed though) that was used to perform the tests on the software under development (a virtual sim lab) in order to be able to do repeatable tests that could determine if a fix for a problem indeed worked, which is a fairly standard practice currently for complex commercial software development. You could call these small test programs a virtual software test jig.

[Lars_J]

My twitter feed detected a new SpaceX post:

Behind the scenes photo from the making of the Dragon panorama http://pic.twitter.com/7TzYPs0G

This must have been taken many months ago, if this was the COTS2/3 Dragon.

[watermod]

In the photo is that a white light 3D scanner plugged into the Dell Laptop?