Proton-M Failure Reaction and Discussion Thread - July 2, 2013

[owais.usmani]

http://www.russianspaceweb.com/proton_glonass49.html#0805

Kremlin provides final details on Proton accident

On Aug. 5, 2013, Russian government released a transcript of an investigative commission meeting chaired by Dmitry Rogozin. The head of investigation, Aleksandr Lopatin, reported that a Proton rocket for its 47th GLONASS mission had been manufactured in accordance with a federal contract between Ministry of Defense and GKNPTs Khrunichev signed on March 20, 2010. During its production, a total of 19 permissions were issued for changes in the standard design and manufacturing documentation of the launch vehicle.

Lopatin reiterated that all pre-launch activities had been proceeding normally until around 0.4 seconds before liftoff, when the emergency flight algorithm was activated. Around 6.8 seconds after the liftoff signal, the telemetry showed a sharp increase in the movement of steering mechanisms in engines No. 1, 3, 4 and 6, with their actuators reaching maximum angles. At T+7.7 seconds in flight, gimbal angles along the yaw axis reached their maximum possible angle of 7.5 degrees. Practically from the beginning of the flight, an unstable process of deviation from the correct yaw axis was observed. At T+12.7 seconds in flight, a signal indicating the exceeding of the maximum allowable angle was issued, as the stabilization system (of the rocket) was no longer able to control the yaw. As a result, at T+12.733, the "launch vehicle failure" command had been generated, Lopatin reported.

He then reiterated that the telemetry analysis had confirmed that the anomaly in the movement of the vehicle along the yaw axis had been caused by an abnormal operation of angular velocity sensors, DUS, in the PV-301 instrument unit. A total of six PV-301 units are mounted in two groups on a platform in the aft section of the second stage of the Proton rocket. Three units are responsible for the pitch and three for yaw axis of the flight trajectory.

Lopatin also confirmed that two out of three incorrectly installed yaw DUS sensors had been identified after their recovery from the crash site, thanks to remnants of red and yellow paint. The third such instrument was not positively identified because its paint cover had been completely burned.

The commission recommended NPO Technomash design bureau to compile a list of operations in the rocket industry that would require photo and video documentation and add respective changes to the existing manufacturing standards, known as OST. GKNPTs Khrunichev and NPTs AP design bureau were assigned to modify the PV-301 instrument and its holding platform to exclude the possibility of its incorrect installation. The commission also required to introduce photo and video documentation of the installation process and introduce additional checks of cable connections leading to the instruments.

[Mighty-T]

OK, but how does the activation of the emergency flight algorithm at T0 -0.4s relate to the falsely installed angular velocity sensors? Or is that an unrelated anomaly?

[Nicolas PILLET]

OK, but how does the activation of the emergency flight algorithm at T0 -0.4s relate to the falsely installed angular velocity sensors? Or is that an unrelated anomaly?

Unrelated anomaly.

[xm11]

http://www.ilslaunch.com/newsroom/news-releases/ils-frob-concludes-glonass-proton-launch-anomaly-investigation


ILS FROB Concludes GLONASS Proton Launch Anomaly Investigation
The ILS Failure Review Oversight Board (FROB) concluded its review on August 13, concurring with the conclusions of the Russian State Inter-agency Commission on the root cause and the associated corrective action plan of the July 2, 2013 Proton M/Block DM mission failure with three GLONASS navigational satellites for the Russian Federal Government.

The members of the FROB agreed with the findings of the Russian investigation that the root cause of the failure was due to the improper installation of the three yaw angular rate sensors located on the Proton launch vehicle, which caused the vehicle to deviate from its flight path shortly after lift-off.

“We very much appreciate the time, effort and participation of our customers, the insurance underwriters and technical experts in the FROB process. They worked tirelessly with us to ensure that the review was conducted thoroughly. As we work towards the return to flight of the Proton vehicle, we thank all of our customers for their continued support,” said ILS Vice President of Programs and Operations, John Palmé.

The ILS Proton return to flight mission will be the Astra 2E satellite for SES on September 15, 2013. The scheduling of the remainder of the ILS Proton near term manifest for 2013 is currently being determined

[xm11]

http://www.ilslaunch.com/newsroom/news-releases/ils-frob-concludes-glonass-proton-launch-anomaly-investigation

The ILS Failure Review Oversight Board (FROB) concluded its review on August 13, concurring with the conclusions of the Russian State Inter-agency Commission on the root cause and the associated corrective action plan of the July 2, 2013 Proton M/Block DM mission failure with three GLONASS navigational satellites for the Russian Federal Government.

The members of the FROB agreed with the findings of the Russian investigation that the root cause of the failure was due to the improper installation of the three yaw angular rate sensors located on the Proton launch vehicle, which caused the vehicle to deviate from its flight path shortly after lift-off.

“We very much appreciate the time, effort and participation of our customers, the insurance underwriters and technical experts in the FROB process. They worked tirelessly with us to ensure that the review was conducted thoroughly. As we work towards the return to flight of the Proton vehicle, we thank all of our customers for their continued support,” said ILS Vice President of Programs and Operations, John Palmé.

The ILS Proton return to flight mission will be the Astra 2E satellite for SES on September 15, 2013. The scheduling of the remainder of the ILS Proton near term manifest for 2013 is currently being determined

[owais.usmani]

And there we go, the witch hunt starts 

http://ria.ru/science/20130823/958134582.html

http://ria.ru/science/20130823/958143958.html

"For the improper performance of duties in the manufacture and preparation for the launch rocket" Proton-M "released from their positions: deputy director of the quality of the Khrunichev Center Alexander Kobzar, final assembly foreman Valery Greeks, Head of Technical Control Mikhail Lebedev," - said Rogozin , who oversees the government CMO.

[spectre9]

3 heads a rolling.

That's not what I was hoping for really.

They need to retrain people to do the job properly.

Bring in new people they might make new mistakes.

[akula2]

Crash attributed to improperly installed motion sensors? 

How did the Rocket got the clearance in the first place? If my memory is correct, it was the second GLONASS payload (3 satellites each) loss. I'm almost sure. I don't know what was the loss in terms of time, let alone money. At least this launch should've drained $150 million. Unacceptable. 

Last year, Sukhoi Superjet 100 jet crash was quite fishy too. How can a a senior and reputed Test Pilot turn off brand new flights Terrain Warning System which was loaded with 50 people on a demo flight? I don't know what to say (being a hobby pilot).

Just a few weeks ago, a Kilo Class Submarine blown up killing more than 18 Navy Crew. Huge money, importantly valuable crew lost. Reason? Not yet determined! Perhaps Battery related (Hydrogen pile up) or Liquid Fuel leakage from Missiles - both instances are well known from the previous Sub disasters.

I'd take lot more 'measures' because things are not adding up! 

[Comga]

Here is an updated summary from Mr. Zaks site:
Agency confirms key facts about the accident

On July 18, Roskosmos confirmed all key facts about the accident reported here previously. 
(snip)
At the same time, the agency confirmed that the launch vehicle had lost stabilization along its course axis, as a result of the improper installation of the DUS angular velocity sensors 180 degrees away from their correct direction.

How sensitive are these "angular velocity sensors"?

I think that there are some, like laser gyros, that can sense the rotation of the Earth.  If powered up inverted they would give a clearly negative signal. 

I see lots of flight hardware installation.  Bolts get torqued to many tons of preload.  It is easy to imagine denting a cover by pushing it onto pins using a torque wrench without ever knowing that something was wrong.  One would think that the functional testing afterwards would make the error apparent.

If your rocket tells you that the Sun will rise in the west, it's a bad sign.

[Kabloona]

Here is an updated summary from Mr. Zaks site:
Agency confirms key facts about the accident

On July 18, Roskosmos confirmed all key facts about the accident reported here previously. 
(snip)
At the same time, the agency confirmed that the launch vehicle had lost stabilization along its course axis, as a result of the improper installation of the DUS angular velocity sensors 180 degrees away from their correct direction.

How sensitive are these "angular velocity sensors"?

I think that there are some, like laser gyros, that can sense the rotation of the Earth.  If powered up inverted they would give a clearly negative signal. 

I see lots of flight hardware installation.  Bolts get torqued to many tons of preload.  It is easy to imagine denting a cover by pushing it onto pins using a torque wrench without ever knowing that something was wrong.  One would think that the functional testing afterwards would make the error apparent.

If your rocket tells you that the Sun will rise in the west, it's a bad sign.

The sensors were checked out electrically after installation and gave the expected output signal for a zero-rate condition. Apparently they're not designed to sense rotation of the earth. 

[Remes]

Some well known components in Action:

http://npcap.ru/en/main-directions-of-activity/inertial-control-systems/creation-of-inertial-control-systems/development-of-inertial-measuring-devices-and-systems/

(even if it is not exactly the arrangement which caused the problems, but the first video is nice).

[woods170]

3 heads a rolling.

That's not what I was hoping for really.

They need to retrain people to do the job properly.

Bring in new people they might make new mistakes.

That's not how (former) Soviet Russian culture works.

[Mader Levap]

It is more like "beating will continue until morale improves". Rampant corruption (significantly bigger even than in USA) of course does not help.

[akula2]

I don't think corruption alone is the issue. They are still in rebuilding mode - no liberty of ten of billions each year, shortage of man power, management, soviet-era infrastructure etc. But still they are managing to put up a great job. I also suspect 'fishy' element because such things are very much possible in this 'digital' era.

No, corruption there is definitely not bigger than in US (if one follows Oversight & Government Reform Committee).

Anyway, even with billions and best management/testing, such things can happen anywhere. I just read about an aborted Japanese launch (Epsilon-1 computer problems).   

[Stan Black]

How come Proton QC issues always emerge on Russian federal missions, and never on ILS missions?

Coincidence. But that will undoubtly be questioned by the more conspiracy-oriented forum members here. 

Leave conspiracy aside, but we know that Khrunichev charges ~100mill $ to ILS customers. Do we know how much the Russian govt pays Khrunichev for federal missions? Is it slightly less or is it "significantly" less? (I'm assuming Stan Black or anik would know the figure for latest contracts)

 I can no longer get the rosspending.ru website to show historical information. It did have the costs of the Uragan batches.

 For this launch I think I have only found the cost of the fairing plus a second piece without individual charges.

 Here is another breakdown to give an idea

Ekspress-AM4

Proton-M
1 123 000 000

Fairing
136 365 000

Delivery of Proton-M and fairing to launch site
23 378 000
 - based on cost for three, breakdown not available

Briz-M
348 900 000

Delivery of Briz-M to launch site
13 325 000
 - based on cost for two, breakdown not available

Launch campaign
570 000 000

So if I haven't messed it up, it was ~68 million $ for Ekspress-AM4.

You get what you pay for, if I may add. 

For the Ekspress satellite launches I have come across additional payments to Khrunichev to support the launch. For AM4 it was 90 600 000 Российский рубль
«Создание космических комплексов системы фиксированной связи, подвижной президентской связи и телерадиовещания с космическими аппаратами на геостационарной орбите в части адаптации КРК «Протон-М» и КРБ «Бриз-М» для обеспечения запуска КА «Экспресс-АМ4»
http://www.zakupki.gov.ru/pgz/public/action/orders/info/common_info/show?notificationId=743605

[Mader Levap]

No, corruption there is definitely not bigger than in US (if one follows Oversight & Government Reform Committee).

Despite lobbyism cancer that rots USA from within, I find it hard to believe USA already have so high corruption level that is comparable to private empire of el presidento Putin.

[HVM]

https://twitter.com/JohnPalme/status/1675608661039644672

Update and first hand description for this decade old thread from International Launch Services ILS COO & Vice President John Palmé

The Yaw rate sensors were a blind install with off center pin holes that were supposed to make sure that the unit was installed in correct orientation. Unfortunately the pins on the mounting bracket were only press fit so when the new tech installed them blind by reaching through an access hole, he slid them onto the studs, and then torqued the nuts down on the studs. The act of torquing the studs drove the press fit pins back in the bracket and allowed the unit to sit flush on the bracket. The debris of the units showed the mark on the baseplate that the alignment stud made as it was driven back into the bracket. Root cause They were building so many Protons, they used a secondary horizontal stand at the factory for building the stage. The primary stand only allowed 1 orientation of +Y up so the install documentation did not ask for orientation verification. The secondary stand allowed either +Y or -Y up, and this LV was in -Y up. The tech installed the unit per the documentation, but the stage was inverted. The quality person looked through an access hatch after install and verified the orientation arrow on the unit was pointing up per the documentation.

The install is a blind install from the backside of the bracket. The verification was through an access hatch where you could visibly see the unit, but not reach it.