Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 1201590 times)

Offline Syrinx

  • Member
  • Posts: 48
  • San Carlos, CA, USA
  • Liked: 134
  • Likes Given: 53
I think I understand what's going on now.

There are 44 * 30 macroblocks in the picture.  ffmpeg is dumping the x-y locations of the macroblocks, and the byte offset and byte size of each macroblock.

E.g.,

[mpeg4 @ 0x60007a520] MB pos/size: 0 30:00:12347 104
[mpeg4 @ 0x60007a520] MB pos/size: 0 31:00:12451 53
[mpeg4 @ 0x60007a520] MB pos/size: 0 32:00:12504 91
[mpeg4 @ 0x60007a520] MB pos/size: 0 33:00:12595 116
[mpeg4 @ 0x60007a520] MB pos/size: 0 34:00:12711 77
[mpeg4 @ 0x60007a520] MB pos/size: 0 35:00:12788 133
[mpeg4 @ 0x60007a520] MB pos/size: 0 36:00:12921 73
[mpeg4 @ 0x60007a520] MB pos/size: 0 37:00:12994 95
[mpeg4 @ 0x60007a520] MB pos/size: 0 38:00:13089 110
[mpeg4 @ 0x60007a520] ac-tex damaged at 39 0
[mpeg4 @ 0x60007a520] MB pos/size: -1 39:00:13199 7


The fields are:

[mpeg4 @ 0x60007a520] MB pos/size: return-code x:y:offset size]

You guys are manually looking at the picture, picking out a familiar macroblock and placing it at the correct x-y location using the -mmb override.  When a macroblock is placed at the correct x-y location, it might also make other macroblocks snap in to place.  Then you repeat for another macroblock.

A return code of -1 means ffmpeg couldn't decode the macroblock.  0 means success, but it doesn't necessarily mean the macroblock is in the correct x-y location.

Is that all correct?

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2908
  • Space Capitalist
  • Silicon Valley, CA
  • Liked: 2204
  • Likes Given: 818
I need some "truth" data. Do we have any camera views from any other video that shows what the lower portion of the image (the rocket itself) looks like at all? I'm trying to figure out something remarkable about it so I can tell one block from another.

I need to order a bunch of puzzle pieces. Each has numbers on the bottom that need to be in order, they are perfectly square, varying flat white and there are either too few pieces or too many pieces and I need to either make some new ones or throw some away.

There are images from the rocketcam in this thread

http://forum.nasaspaceflight.com/index.php?topic=34502.105

However, I think these are actually taken from 2nd stage (higher up in the stack, slightly different angle) and a camera on the 1st stage was used only for this landing footage, so the image is not entirely same. You'd still get some idea as to what the rocket body looks like.

I do agree that it would probably help if SpaceX would release at least a couple of frames from the 1st stage camera from higher up. Reportedly they have good video of the braking burns but chose not to release it, perhaps to keep their secret sauce secret.

Na, I need an image from the lower camera. Preferably during retroburn or something, but any time works.
LEO is the ocean, not an island (let alone a continent). We create cruise liners to ride the oceans, not artificial islands in the middle of them. We need a physical place, which has physical resources, to make our future out there.

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2908
  • Space Capitalist
  • Silicon Valley, CA
  • Liked: 2204
  • Likes Given: 818
I think I understand what's going on now.

There are 44 * 30 macroblocks in the picture.  ffmpeg is dumping the x-y locations of the macroblocks, and the byte offset and byte size of each macroblock.

E.g.,

[mpeg4 @ 0x60007a520] MB pos/size: 0 30:00:12347 104
[mpeg4 @ 0x60007a520] MB pos/size: 0 31:00:12451 53
[mpeg4 @ 0x60007a520] MB pos/size: 0 32:00:12504 91
[mpeg4 @ 0x60007a520] MB pos/size: 0 33:00:12595 116
[mpeg4 @ 0x60007a520] MB pos/size: 0 34:00:12711 77
[mpeg4 @ 0x60007a520] MB pos/size: 0 35:00:12788 133
[mpeg4 @ 0x60007a520] MB pos/size: 0 36:00:12921 73
[mpeg4 @ 0x60007a520] MB pos/size: 0 37:00:12994 95
[mpeg4 @ 0x60007a520] MB pos/size: 0 38:00:13089 110
[mpeg4 @ 0x60007a520] ac-tex damaged at 39 0
[mpeg4 @ 0x60007a520] MB pos/size: -1 39:00:13199 7


The fields are:

[mpeg4 @ 0x60007a520] MB pos/size: return-code x:y:offset size]

You guys are manually looking at the picture, picking out a familiar macroblock and placing it at the correct x-y location using the -mmb override.  When a macroblock is placed at the correct x-y location, it might also make other macroblocks snap in to place.  Then you repeat for another macroblock.

A return code of -1 means ffmpeg couldn't decode the macroblock.  0 means success, but it doesn't necessarily mean the macroblock is in the correct x-y location.

Is that all correct?

We're not placing macroblocks at the correct locations as is replacing corrupted macroblocks with blocks from _elsewhere_ in the image. Every macroblock in the image is a reference+changes to either the block directly above it or the block directly to the left (this choice is made when the video stream is created). If you replace a macroblock with another block that has "roughly" the same "+changes" as the actual block then the created block will look roughly like the original and blocks that reference it will look roughly how they are supposed to.

In some cases where things are too messed up we look for a macroblock that has changes that we know should be at a specific location (the legs) and forcibly put it there. That doesn't fix any previous blocks, but as you mention it often makes other blocks that follow it snap into place. Ideally after you put that block in place you should work backwards (right to left, bottom to top) toward the other bad blocks trying to replace those as well while maintaining the same "block address" of the block you fixed.

Everything else is correct.
LEO is the ocean, not an island (let alone a continent). We create cruise liners to ride the oceans, not artificial islands in the middle of them. We need a physical place, which has physical resources, to make our future out there.

Offline mlindner

  • Software Engineer
  • Senior Member
  • *****
  • Posts: 2908
  • Space Capitalist
  • Silicon Valley, CA
  • Liked: 2204
  • Likes Given: 818
Since there are only 12 iframes, does that mean there are only 12 images in the entire video?

There are 12 iframes which are complete images that reference no previous images. Every other frame is a reference to one of those frames. To be specific, any frames in between two iframes reference the previous iframe, other frames that also reference that iframe or pieces of the frame reference other pieces of the same frame. Welcome to the world of high-compression video codecs. :P Errors propagate like hell.
« Last Edit: 05/05/2014 10:04 am by mlindner »
LEO is the ocean, not an island (let alone a continent). We create cruise liners to ride the oceans, not artificial islands in the middle of them. We need a physical place, which has physical resources, to make our future out there.

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
Na, I need an image from the lower camera. Preferably during retroburn or something, but any time works.

This picture of iframe2 contains data at the (nearly) bottom:

 http://forum.nasaspaceflight.com/index.php?topic=34597.msg1193527#msg1193527

Note that the clock is the lowest block. So this is what you want I think. It's just that everything in this picture is moved up because I "fast-forwarded" at some point (essentially skipped like 4 rows). So especially the left is what you want. And the stage is probable symetrical. So could be useful.

Offline SwissCheese

  • Full Member
  • *
  • Posts: 166
  • Liked: 256
  • Likes Given: 107
Na, I need an image from the lower camera. Preferably during retroburn or something, but any time works.

Maybe you can check the Cassiope launch highlights movie (at 2:40) where they show the reentry burn: http://forum.nasaspaceflight.com/index.php?topic=32946.msg1109241#msg1109241

Offline fatdeeman

  • Full Member
  • *
  • Posts: 134
  • Liked: 112
  • Likes Given: 29
Absolutely in awe of what's being achieved here. I had heard before that videos with errors still contained a lot of usable information but I had no idea it could be retrieved to this extent. I had visions of having to reconstruct entire frames by moving blocks around like a puzzle and guessing if they were in the right place.

Absolutely fantastic work being done here!

One more thing, Elon mentioned fan sourcing helped restore video footage before. Does any one know of any examples?

Online Asmegin

  • Member
  • Posts: 37
  • Canada
  • Liked: 26
  • Likes Given: 3
Absolutely in awe of what's being achieved here. I had heard before that videos with errors still contained a lot of usable information but I had no idea it could be retrieved to this extent. I had visions of having to reconstruct entire frames by moving blocks around like a puzzle and guessing if they were in the right place.

Absolutely fantastic work being done here!

One more thing, Elon mentioned fan sourcing helped restore video footage before. Does any one know of any examples?

+1 to being amazed. I wish I could help, but I literally have no idea what you guys are talking about :D

As for the video, it's possible he was referencing the Mars Curiosity Landing video which was amazing after someone took 4 weeks to fix it:

Offline fatdeeman

  • Full Member
  • *
  • Posts: 134
  • Liked: 112
  • Likes Given: 29
Absolutely in awe of what's being achieved here. I had heard before that videos with errors still contained a lot of usable information but I had no idea it could be retrieved to this extent. I had visions of having to reconstruct entire frames by moving blocks around like a puzzle and guessing if they were in the right place.

Absolutely fantastic work being done here!

One more thing, Elon mentioned fan sourcing helped restore video footage before. Does any one know of any examples?

+1 to being amazed. I wish I could help, but I literally have no idea what you guys are talking about :D

As for the video, it's possible he was referencing the Mars Curiosity Landing video which was amazing after someone took 4 weeks to fix it:

Ah yes I was watching this the other day funnily enough but it didn't click! The amount of detail in this one compared to the version NASA released is incredible!

I couldn't do what these guys are doing either! A lot of the things being said make sense to me but as far as doing it myself I wouldn't have a clue! I'd be happy if there was anything slightly less perplexing like cheap labour!

Offline llanitedave

  • Senior Member
  • *****
  • Posts: 2286
  • Nevada Desert
  • Liked: 1545
  • Likes Given: 2052
Wow, all that and sound too!  Beautiful!
"I've just abducted an alien -- now what?"

Online Jeff Lerner

  • Member
  • Full Member
  • ****
  • Posts: 617
  • Toronto, Canada
  • Liked: 270
  • Likes Given: 240
I also want to say how amazed I am at the effort and progress being made with this video.

Having said that, I thought I'd toss in my $0.02 as a retired IT project manager of 38 years. I often  found that the  developers working for me would get so caught up developing their code that they has no time to see the bigger picture...that's where I came in as PM..

So,having said that, I'm wondering if anyone is aware of any other efforts on any other space related sites, photo or video sites or for that matter, even at SpaceX, to answer Elon's call to arms to fix up this video...I'm just thinking that it would be sad to have all this great effort wasted because some where else, someone else has already done this work.....

...and thats maybe where Chris can help out as he has some contacts at SpaceX and can inquire as to whether they are aware of any other progress being made...if little else, Chris could at least make SpaceX know that people on this site are working on a solution and might require some info from SpaceX.....

Offline Syrinx

  • Member
  • Posts: 48
  • San Carlos, CA, USA
  • Liked: 134
  • Likes Given: 53
We're not placing macroblocks at the correct locations as is replacing corrupted macroblocks with blocks from _elsewhere_ in the image. Every macroblock in the image is a reference+changes to either the block directly above it or the block directly to the left (this choice is made when the video stream is created). If you replace a macroblock with another block that has "roughly" the same "+changes" as the actual block then the created block will look roughly like the original and blocks that reference it will look roughly how they are supposed to.

In some cases where things are too messed up we look for a macroblock that has changes that we know should be at a specific location (the legs) and forcibly put it there. That doesn't fix any previous blocks, but as you mention it often makes other blocks that follow it snap into place. Ideally after you put that block in place you should work backwards (right to left, bottom to top) toward the other bad blocks trying to replace those as well while maintaining the same "block address" of the block you fixed.

Everything else is correct.

Is it really just a matter of finding a suitable stand-in for corrupt macroblocks?  Maybe the process is not so bad then!

But I was looking at iframe 11.  Macroblock 0:0 is valid (not corrupt) as reported by the debug info, but somebody replaced it anyway (with 5:0 I think) and it greatly improved the image.

(Maybe I am mistaken.  Maybe iframe 11 0:0 is corrupt.  I would check but I'm not on my home computer right now.)

Offline swervin

  • Full Member
  • *
  • Posts: 168
  • Viper Driver
  • GA
  • Liked: 44
  • Likes Given: 10
Perhaps the folks who assisted in the cleaned up Curiosity video could be contacted for 'lessons learned' on that video? ... Or even assist on this one if they were interested.

I thought it would be cool if the 'code' was broken on how to clean this up, if that was made available, a la SETI @ home, to use the computing power of a bunch of computers to clean it up faster. Then again, maybe not much computing power is req'd.

This coming from someone who has no idea how to do this ... Haha

Cheers,
Splinter

Offline grythumn

  • Full Member
  • **
  • Posts: 243
  • Liked: 167
  • Likes Given: 246
I also want to say how amazed I am at the effort and progress being made with this video.

Having said that, I thought I'd toss in my $0.02 as a retired IT project manager of 38 years. I often  found that the  developers working for me would get so caught up developing their code that they has no time to see the bigger picture...that's where I came in as PM..

So,having said that, I'm wondering if anyone is aware of any other efforts on any other space related sites, photo or video sites or for that matter, even at SpaceX, to answer Elon's call to arms to fix up this video...I'm just thinking that it would be sad to have all this great effort wasted because some where else, someone else has already done this work.....

Aside from the youtube video, only attempt I've seen is:

temp bana/dtZOQ

That's a photoshop, not actual data.

-Bob
EDIT: Apparently I can't link to it. Try an imaging site that begins with an i and ends with an r. Popular on facebook and such.
« Last Edit: 05/05/2014 06:32 pm by grythumn »

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6160
  • California
  • Liked: 677
  • Likes Given: 195
Perhaps the folks who assisted in the cleaned up Curiosity video could be contacted for 'lessons learned' on that video? ... Or even assist on this one if they were interested.

The Curiosity video is a completely different scenario. No cleanup of re-interpretation of missing data was needed - and if it was, not anywhere near this level.

The real achievement of the Curiosity video was the smooth frame interpolation from still frames to create the illusion of a full frame rate video.
« Last Edit: 05/05/2014 05:54 pm by Lars_J »

Offline moralec

I'm truly amazed on the outstanding work that is being done in recovering the I-frames. Kudos to arnezami for developing the approach, and for everyone else that has been involved. Even if only those 12 images are recovered, I'm sure they are going to be really valuable for Space X.

Just a question: Is anybody collecting/monitoring the progress on this task? As several of you are working on parallel, I believe a monitoring tool would be useful in order to lower the risk of having several individuals doing the same thing. 

To start, may I suggest on keeping a table like the one originally describing the I-Frames (http://forum.nasaspaceflight.com/index.php?topic=34597.msg1193316#msg1193316) in order for everyone to know who is working on each one?


I'll be glad to take that responsibility if you, the guys actually working on the images,  find it useful...
« Last Edit: 05/05/2014 05:59 pm by moralec »

Offline neoforce

  • Full Member
  • ****
  • Posts: 426
  • Liked: 371
  • Likes Given: 20
Perhaps the folks who assisted in the cleaned up Curiosity video could be contacted for 'lessons learned' on that video? ... Or even assist on this one if they were interested.

The Curiosity video is a completely different scenario. No cleanup of re-interpretation of missing data was needed - and if it was, not anywhere near this level.

The real achievement of the Curiosity video was the smooth frame interpolation from still frames to create the illusion of a full frame rate video.

Some additional details on the differences... 

In the Falcon 9 video, we have compression that lost blocks due to transmission fall out.  In the Curiosity video, it was by design 4 frames per second.  All of the data for each of those 4 frames per second were recovered. So what he did was "create" an additional 26 frames for each second of footage to get it to 30 frames per second.

It is a very different problem.  His was enhancing, adding full frames between existing frames of complete information.  In this thread, here they are trying to repair data within each frame of the video.

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
<snip>

arnezami I've been talking to michaelni a lot today in learning how to do this process and I think your concept of "end markers" is off. Every block has a pre-defined size which is determined by its format. When a block is mis-interpreted then the size automatically becomes wrong and the block "eats" extra bits from the next block or misses bits from its block shoving them into the next block. Occasionally these errors cancel each other out (because there is a small finite number of block types and sizes) and the streaming is "resynced" to the proper block ordering so you get a string of correct blocks. You often don't see this resync because after its resynced the blocks are still referencing all the previous corrupted blocks. This is why after you fix one block suddenly lots more blocks become readable IMO.

If michaelni could comment though that would probably clear more things up.

Edit: The key difference that matters from your explanation is that changing the start position of a block can _increase_ or _decrease_ how far away the stop position is for that block because of how the block is interpreted.
Hi,

I think I found most the answer to this question.

In the MPEG 4-2 specification ( I googled for "Video Bitstream Syntax and Semantics" and found a doc file called "MPEG-2_Video_IS.doc") its says in section 6.2.6 that blocks end with "End of block"-bits (10b I believe). Later on in section 8.2 it says that macroblocks will end with the "End of block"-symbol of its last block. We have 4 blocks in a macroblock I believe. See the attachments.

So I think my intuition of the macroblocks ending with stop-bits ("end markers") is pretty spot on, albeit with a little nuance that the stop-bits are in the blocks not in the macroblock. The document also gives a lot of info on bits that may be recognizable somehow. Not sure yet how we can use this for semi-automation.

Regards,

arnezami

[edit] I'm sorry this is of course MPEG-2. Duh. This probably does not apply to MPEG-4.
« Last Edit: 05/05/2014 07:45 pm by arnezami »

Offline vanderhilst

  • Member
  • Posts: 2
  • Amsterdam
  • Liked: 0
  • Likes Given: 2
Hi all, I'm really enjoying to learn from the inspiring progress that is made by all you guys,
as well as by the efforts of SpaceX to create this video in the first place!!

As an experienced programmer in Mathematica, I wasn't able to chip in up to this point, but it seems to me that you have come to a point where automated calls to ffmpeg may be beneficial to the process of recovering the I-frames. I'm willing to jump in, and have two questions:

1) Did anybody listen to Jeff Lerner's comments, reply #132?

2) Is it sensible to use this way to have ffmpeg run under windows? http://www.wikihow.com/Install-FFmpeg-on-Windows

Thanks and enjoy the revealing!
« Last Edit: 05/05/2014 06:41 pm by vanderhilst »

I was toying with the idea of creating a web app so that people could do this visually and would make it easier to distribute the workload. I think it would be a good idea but I also think that by the time it's done people will have managed to get most of the way there. I don't know if something like that would have any re-use value save for something like this happening again.

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1