Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 958668 times)

Offline arnezami

  • Full Member
  • **
  • Posts: 284
  • Liked: 262
  • Likes Given: 346
Aha!

I think I found a way to detect the beginning of macroblocks! :)

They seem to have a fixed 11-bit macroblock_escape code (0000000100b) followed by an 1-11 bit macroblock_address_increment! (very useful for detecting where the macroblock should be). If this is actually true then finding the start of macroblocks might actually become quite easy. And its even followed by a macroblock_type of 1-9 bits. That's a lot of bits to be non-random.

See attachments.

Regards,

arnezami

[edit] I'm sorry this is of course MPEG-2. Duh. This probably does not apply to MPEG-4.
« Last Edit: 05/05/2014 07:45 PM by arnezami »

Offline laszlo

  • Full Member
  • *
  • Posts: 183
  • Liked: 198
  • Likes Given: 39
can someone please explain to me why spacex still hasn't released the footage they got from their airplane? Why are they being so secretive about it? I mean seeing is believing so why not just release the footage and prove to everyone in the world they really accomplished such an unprecedented feat?

I mean they did actually record it from their airplane right?

The left landing gear didn't deploy and they didn't want the embarrassment of showing it skidding off the runway. Oh, wait. Wrong landing. My bad.

Offline Syrinx

  • Member
  • Posts: 31
  • San Carlos, CA, USA
  • Liked: 46
  • Likes Given: 28
2) Is it sensible to use this way to have ffmpeg run under windows? http://www.wikihow.com/Install-FFmpeg-on-Windows

First, you need to install the custom ffmpeg version that's been made available in this thread on github.  You probably already knew that, but I wanted to make it explicit.

I didn't try the install mechanism presented at that link.  I installed mine on top of Cygwin.  It look about an hour and I had no problems, but of course you also need to install Cygwin if you don't have it.

Offline grythumn

  • Full Member
  • **
  • Posts: 214
  • Liked: 126
  • Likes Given: 147
2) Is it sensible to use this way to have ffmpeg run under windows? http://www.wikihow.com/Install-FFmpeg-on-Windows

This might work without a full cygwin install (at least cygcheck thinks I got all the DLLs). But I recommend it anyway, or at least some sort of improved shell.

EDIT: This is a command line utility. Please only use if you know how to use such.

-Bob
« Last Edit: 05/05/2014 06:57 PM by grythumn »

Offline vanderhilst

  • Member
  • Posts: 2
  • Amsterdam
  • Liked: 0
  • Likes Given: 2
Yay! Thank you both, Syrinx and Bob!

The ffmpeg as downloadable in the post by Bob above, combined with the method in the wikihow link I posted a few posts earlier, actually allow me to get a correct response to a command line command "ffmpeg -version" under 64-bit Windows 7.

I will proceed to try and run the other commands explained before to recreate the results posted by others.
If successful, I will try to see whether automation will do anything useful. I will start with I-frame 11, if any.

edit: for clarity: I did NOT install Cygwin.  :)
« Last Edit: 05/05/2014 07:13 PM by vanderhilst »

Offline cosmonautdjp

  • Member
  • Posts: 64
  • Liked: 24
  • Likes Given: 51
I asked Doug Ellison, the JPL guy who did the MSL descent video, if he had any tips.

@danieljphilo:@doug_ellison They're trying to improve the SpaceX first stage landing video at http://nasaspaceflight.com  Any tips? http://forum.nasaspaceflight.com/index.php?topic=34597.135

@doug_ellison:@danieljphilo Nope. Beyond my skills.

Offline Jarnis

  • Full Member
  • ****
  • Posts: 1133
  • Liked: 581
  • Likes Given: 159
I asked Doug Ellison, the JPL guy who did the MSL descent video, if he had any tips.

@danieljphilo:@doug_ellison They're trying to improve the SpaceX first stage landing video at http://nasaspaceflight.com  Any tips? http://forum.nasaspaceflight.com/index.php?topic=34597.135

@doug_ellison:@danieljphilo Nope. Beyond my skills.

Not surprised. MSL video was about doing high quality interpolation from fairly good keyframes.

Here the keyframes are in tatters and you really need to do painstaking manual reconstruction...

Offline Chris Bergin

I asked Doug Ellison, the JPL guy who did the MSL descent video, if he had any tips.

@danieljphilo:@doug_ellison They're trying to improve the SpaceX first stage landing video at http://nasaspaceflight.com  Any tips? http://forum.nasaspaceflight.com/index.php?topic=34597.135

@doug_ellison:@danieljphilo Nope. Beyond my skills.

Top man. Met him in real life and really happy he ended up at JPL.

Keep up the good work here. It's all over my head, but clearly a hardcore process. Very impressive!

Remember, any big breakthrough, send me a message and I'll fast track you to someone at SpaceX. There's probably a free ticket to ride on Dragon in it for a win! ;)

Offline Syrinx

  • Member
  • Posts: 31
  • San Carlos, CA, USA
  • Liked: 46
  • Likes Given: 28
We're not placing macroblocks at the correct locations as is replacing corrupted macroblocks with blocks from _elsewhere_ in the image. Every macroblock in the image is a reference+changes to either the block directly above it or the block directly to the left (this choice is made when the video stream is created). If you replace a macroblock with another block that has "roughly" the same "+changes" as the actual block then the created block will look roughly like the original and blocks that reference it will look roughly how they are supposed to.

In some cases where things are too messed up we look for a macroblock that has changes that we know should be at a specific location (the legs) and forcibly put it there. That doesn't fix any previous blocks, but as you mention it often makes other blocks that follow it snap into place. Ideally after you put that block in place you should work backwards (right to left, bottom to top) toward the other bad blocks trying to replace those as well while maintaining the same "block address" of the block you fixed.

Everything else is correct.

Is it really just a matter of finding a suitable stand-in for corrupt macroblocks?  Maybe the process is not so bad then!

But I was looking at iframe 11.  Macroblock 0:0 is valid (not corrupt) as reported by the debug info, but somebody replaced it anyway (with 5:0 I think) and it greatly improved the image.

(Maybe I am mistaken.  Maybe iframe 11 0:0 is corrupt.  I would check but I'm not on my home computer right now.)

OK, I double-checked and macroblock 0:0 is indeed valid:

[mpeg4 @ 0x80062980] MB pos/size: 0 00:00:10020 128
[mpeg4 @ 0x80062980] MB pos/size: -1 01:00:10148 102
[mpeg4 @ 0x80062980] Error at MB: 1
[mpeg4 @ 0x80062980] MB pos/size: 0 02:00:10250 109
[mpeg4 @ 0x80062980] I cbpy damaged at 3 0
[mpeg4 @ 0x80062980] MB pos/size: -1 03:00:10359 4
[mpeg4 @ 0x80062980] Error at MB: 3
[mpeg4 @ 0x80062980] I cbpc damaged at 4 0
[mpeg4 @ 0x80062980] MB pos/size: -1 04:00:10363 6
[mpeg4 @ 0x80062980] Error at MB: 4
[mpeg4 @ 0x80062980] MB pos/size: 0 05:00:10369 102
[mpeg4 @ 0x80062980] MB pos/size: 0 06:00:10471 64
[mpeg4 @ 0x80062980] MB pos/size: 0 07:00:10535 92
[mpeg4 @ 0x80062980] MB pos/size: 0 08:00:10627 52
[mpeg4 @ 0x80062980] MB pos/size: 0 09:00:10679 64


So should I "simply" find suitable stand-ins for corrupt macroblocks?  Should I replace (some) valid macroblocks?  Both?

Offline arnezami

  • Full Member
  • **
  • Posts: 284
  • Liked: 262
  • Likes Given: 346
OK, I double-checked and macroblock 0:0 is indeed valid:

[mpeg4 @ 0x80062980] MB pos/size: 0 00:00:10020 128
[mpeg4 @ 0x80062980] MB pos/size: -1 01:00:10148 102
[mpeg4 @ 0x80062980] Error at MB: 1
[mpeg4 @ 0x80062980] MB pos/size: 0 02:00:10250 109
[mpeg4 @ 0x80062980] I cbpy damaged at 3 0
[mpeg4 @ 0x80062980] MB pos/size: -1 03:00:10359 4
[mpeg4 @ 0x80062980] Error at MB: 3
[mpeg4 @ 0x80062980] I cbpc damaged at 4 0
[mpeg4 @ 0x80062980] MB pos/size: -1 04:00:10363 6
[mpeg4 @ 0x80062980] Error at MB: 4
[mpeg4 @ 0x80062980] MB pos/size: 0 05:00:10369 102
[mpeg4 @ 0x80062980] MB pos/size: 0 06:00:10471 64
[mpeg4 @ 0x80062980] MB pos/size: 0 07:00:10535 92
[mpeg4 @ 0x80062980] MB pos/size: 0 08:00:10627 52
[mpeg4 @ 0x80062980] MB pos/size: 0 09:00:10679 64


So should I "simply" find suitable stand-ins for corrupt macroblocks?  Should I replace (some) valid macroblocks?  Both?

First. I'm not sure you have already but if not please read/look at this fairly thourough explanation of how it all works:

http://forum.nasaspaceflight.com/index.php?topic=34597.msg1193685#msg1193685

Second. I did intentionally replace 0,0 with a different block. But this is just for experimentation.

The reason why 0,0 has advantages is that there are no blocks above or left of it. How mpeg-4 works is that a block "inherits" the traits of the blocks left and above it. So if you're at 0,0 there is no corruption from other blocks before that.

You have to understand that in mpeg-4 everything is chained somehow: vertical, horizontal and time-wise. But its all chained in one direction: left to right, top to bottom, previous to next. This really saves in the number of bytes it takes to store/transmit a video but it also makes it very hard to reconstruct it when something is only slightly off, because a lot of other things depend on it being right: errors propagate rapidly. This is the difficulty we are trying to overcome.

But when you start a block at 0,0 you eliminate most of all this domino-effect troubles. You lose the information what you would have gotten from a top and left block when you started at your normal x and y, but its just to see which blocks are good or bad. Which is much easier to determine from 0,0. So I just used 0,0 to prove this point. Don't worry about it. The first block from 0,0 might be correct and completely ok. But I didn't care for that in this particular experiment: I wanted a starting block with no left and upper blocks! And there is only one block in the game that has that: 0,0.
« Last Edit: 05/05/2014 08:44 PM by arnezami »

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2014
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 654
  • Likes Given: 241
We're not placing macroblocks at the correct locations as is replacing corrupted macroblocks with blocks from _elsewhere_ in the image. Every macroblock in the image is a reference+changes to either the block directly above it or the block directly to the left (this choice is made when the video stream is created). If you replace a macroblock with another block that has "roughly" the same "+changes" as the actual block then the created block will look roughly like the original and blocks that reference it will look roughly how they are supposed to.

In some cases where things are too messed up we look for a macroblock that has changes that we know should be at a specific location (the legs) and forcibly put it there. That doesn't fix any previous blocks, but as you mention it often makes other blocks that follow it snap into place. Ideally after you put that block in place you should work backwards (right to left, bottom to top) toward the other bad blocks trying to replace those as well while maintaining the same "block address" of the block you fixed.

Everything else is correct.

Is it really just a matter of finding a suitable stand-in for corrupt macroblocks?  Maybe the process is not so bad then!

But I was looking at iframe 11.  Macroblock 0:0 is valid (not corrupt) as reported by the debug info, but somebody replaced it anyway (with 5:0 I think) and it greatly improved the image.

(Maybe I am mistaken.  Maybe iframe 11 0:0 is corrupt.  I would check but I'm not on my home computer right now.)

"Suitable" is still rather difficult though. I personally don't have a good grasp of "suitable" other than trial and error.

I haven't looked at the image in question, but often a block is corrupted such that its still has valid data. Corruption is not binary. That means a 1 bit change could just slightly tweak the color for example and would still be completely valid. I've been looking into making some more modifications to allow for overriding from the command line the contents of a block so we can try to fix the block rather than replace it.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline didacticus

  • Member
  • Posts: 14
  • Canada
  • Liked: 9
  • Likes Given: 62
Very interesting reading all the discussion here! Just wanted to say I appreciate the effort you are all putting in to retrieve this video. Good luck!

Offline LucR

  • Member
  • Posts: 54
  • Liked: 15
  • Likes Given: 42
I think I found a way to detect the beginning of macroblocks! :)
They seem to have a fixed 11-bit macroblock_escape code (0000000100b) followed by an 1-11 bit macroblock_address_increment!

Sorry, the macroblock_escape won't help for decoding I-frames, because:
Quote from: MPEG-2_Video_IS.doc section 6.3.17 Macroblock
macroblock_escape -- The macroblock_escape is a fixed bit-string 0000 0001 000 which is used when the difference between macroblock_address and previous_macroblock_address is greater than 33. It causes the value of macroblock_address_increment to be 33 greater than the value that will be decoded by subsequent macroblock_escape and the macroblock_address_increment codewords.
In an I-frame all macroblocks are typically coded (i.e. not skipped, "There shall be no skipped macroblocks in I-pictures except [...]" at the end of section 6.3.7) so there will be no occurrences of macroblock_escape.

However, in I-frames the immediately following macroblock_address_increment code should encode a value of 1 which are coded as the bitstream bits 0000 0101 01 (from Table B-1 --- Variable length codes for macroblock_address_increment).
This is still 10 bits of constant data, hope that helps some.

If there is no spatial scalability, Table B-2 applies for the immediately following macroblock_type code, which means that it is either 1 (Intra) or 01 (Intra, Quant); another 1 or 2 constant bits.

I haven't looked at the actual bitstreams (yet); are slices start codes not used? This would make it easier to restart part-way through a frame.

By the way, for background: I have some amount of MPEG experience but it's not at the macroblock level. However, I do know how to read the bitstream syntax specs.

Edit: Added note about macroblock_type.
« Last Edit: 05/05/2014 09:43 PM by LucR »

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2014
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 654
  • Likes Given: 241
In figuring out good methods, here's how I'm doing it.

I use the CLI line: ./ffmpeg -mmb `cat mmb.txt` -debug mb_pos_size -err_detect ignore_err -s:0 704:480 -i iframes/iframe_8.mpg4-img -f image2 img-001.png && open img-001.png

I use ` ` to pipe the text document into the CLI so I can rapidly change it and the open command (Mac OS X specific) to open the image.

Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2014
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 654
  • Likes Given: 241
I think I found a way to detect the beginning of macroblocks! :)
They seem to have a fixed 11-bit macroblock_escape code (0000000100b) followed by an 1-11 bit macroblock_address_increment!

Sorry, the macroblock_escape won't help for decoding I-frames, because:
Quote from: MPEG-2_Video_IS.doc section 6.3.17 Macroblock
macroblock_escape -- The macroblock_escape is a fixed bit-string 0000 0001 000 which is used when the difference between macroblock_address and previous_macroblock_address is greater than 33. It causes the value of macroblock_address_increment to be 33 greater than the value that will be decoded by subsequent macroblock_escape and the macroblock_address_increment codewords.
In an I-frame all macroblocks are typically coded (i.e. not skipped, "There shall be no skipped macroblocks in I-pictures except [...]" at the end of section 6.3.7) so there will be no occurrences of macroblock_escape.

However, in I-frames the immediately following macroblock_address_increment code should encode a value of 1 which are coded as the bitstream bits 0000 0101 01 (from Table B-1 --- Variable length codes for macroblock_address_increment).
This is still 10 bits of constant data, hope that helps.

I haven't looked at the actual bitstreams (yet); are slices start codes not used? This would make it easier to restart part-way through a frame.

By the way, for background: I have some amount of MPEG experience but it's not at the macroblock level. However, I do know how to read the bitstream syntax specs.

Thanks for the tips. I don't know MPEG as well as you so if you could investigate and find that out and see if that works that'd be awesome!
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Shanuson

  • Full Member
  • **
  • Posts: 292
  • Liked: 197
  • Likes Given: 596
I am also really impressed by the work done here.

Attached you find a version of the ISO/IEC 14496 which defines the MPEG 4-2 standard.
(Edit: all these sections and tables and stuff mentioned in LucR's posting are found in there)
If I read it correctly there are no fixed end-codes of a block or marcoblock, but I only jump through the document.
Llke a good specification document it is really hard to read, at least for me and my knowledge of not too many such documents ;)
Maybe it helps to understand more of the problem.

Good Luck with finding the right bits :D

Shanuson
« Last Edit: 05/05/2014 09:51 PM by Shanuson »

Offline SwissCheese

  • Full Member
  • *
  • Posts: 165
  • Liked: 250
  • Likes Given: 94
mlindner was asking for a non-corrupted image, so this one comes from the reentry burn of Cassiope launch


Offline LucR

  • Member
  • Posts: 54
  • Liked: 15
  • Likes Given: 42
If I read it correctly there are no fixed end-codes of a block or marcoblock, but I only jump through the document.
AFAIK this is correct, the "End of block" code varies, it can be either 10 (Table B-14) or 0110 (Table B-15), as described in 7.2.2 Other coefficients. In either case the bitstring is probably too short to provide for meaningful recovery.

Offline Adaptation

  • Full Member
  • *
  • Posts: 160
  • Liked: 39
  • Likes Given: 38
I updated my Transport stream tool.  It's probably not going to help much at this point but it does have a hex editor and powerful search feature with a bit level wildcard filter, ability to return results with a specified tolerated number of single bit errors and the results are easy to visualize and locate. 

To do:
Disable controls until a ts file is opened.
Make the search capable of finding results that are not byte aligned. 
Add a replace feature with a wildcard filter.
Enable bit level insertion and deletion ability. (would bit shift everything after that point in the file)


I think its a good companion tool for this transport stream editor. 
http://www.videohelp.com/tools/TS-Packet-Editor

-Update, version 3 is out-
http://forum.nasaspaceflight.com/index.php?topic=34597.msg1197014#msg1197014
« Last Edit: 05/12/2014 06:14 AM by Adaptation »

Offline TrueBlueWitt

  • Space Nut
  • Senior Member
  • *****
  • Posts: 2005
  • Mars in my lifetime!
  • DeWitt, MI
  • Liked: 78
  • Likes Given: 72
I feel like we're missing an important piece of this puzzle.

Was the video data stream was on it's own channel, or there was other data being transmitted on the same channel with some other controls in place to keep the data segregated?

If it was on it's own channel, having an analogue(time domain) copy of the original transmission, perhaps even with time code synced to the video file provided, there might be a lot more that could be done to extract the information.  Having the the time domain data stream it would also be much easier to tell where and when data was dropping out.
« Last Edit: 05/05/2014 10:52 PM by TrueBlueWitt »

Tags: