Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 1201518 times)

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 82
I think I have figured it out! :)

Way to go!!  ;D That pframe looks absolutely delicious!

And look at all this nasty  interlacing - talk about different luminance values in the same block! Sheesh!  :o

ouch...and I got ticked off by this one row in 52... :o Hopefully it doesn't show in the video :)
I love how the legs are "cut out" in 170 :D

Offline JamesH

  • Full Member
  • ****
  • Posts: 525
  • United Kingdom
  • Liked: 284
  • Likes Given: 7
Please forgive me if this has already been covered somewhere in the previous 50 pages....trying to get my understanding correct.

As I understand it I-frames are complete standalone frames - they need no other data to represent aa full frame. P frames are difference frames (as are B but I don't think they are used here). A p-frame contains all the information required to go from the previous P or I frame to the current frame. The result of applying a P-frame to the previous frame should be the current frame. A p-frame is a bunch of 64x64 macroblocks, plus offset information (where the 64x64 base data came from). These contain the difference information from wherever the MB came from and where is it now. So the current frame is basically the previous frame (fully rendered) with the current MB set applied over the top. The result should be a perfect representation of the frame, minus any high frequency compression differences (a bit like JPEG)

So, if you have a broken/completely wrecked) I-frame you can generate a new one by working out what the previous frame looks like, and using that. Simply inject a new i-frame in to the bitstream. Or, you could, if you have a partial I-frame, generate one from a combination of what you do have that looks OK, plus any data from the prewious frame.

And of course in H264 every frame can be an I-frame, so you could process through, creating a new bitstream from the old one, replacing all P-frames with I-frames.

Is that what people are attempting?


Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
So, if you have a broken/completely wrecked) I-frame you can generate a new one by working out what the previous frame looks like, and using that. Simply inject a new i-frame in to the bitstream. Or, you could, if you have a partial I-frame, generate one from a combination of what you do have that looks OK, plus any data from the prewious frame.

Pretty good understanding so far. However, the trouble is that every last one of the iframes in the stream that was just barely received from the first stage were more or less wrecked, so there wasn't a previous frame to use.

If you check out the progress site, you can see how bad things were at the beginning:



The focus over the past few weeks has been to tease out reasonable-looking iframes from the jumble of bits, and since we're in reasonably good shape there, the focus has started to shift to teasing out reasonable-looking pframes to go with them, as you've observed in the last several posts.

Here's what a seven-year veteran of video repair, Benoît Joossen, said after SpaceX had engaged him to scrape out the two partial frames that Elon posted on Twitter:

Quote
Why the video stream can’t be recovered
...
I’ve been repairing videos full time for the last 7 years, so I immediately knew that the bitstream was corrupt and that at best we could hope to partially fix some frames. Maybe a 50% chance of recovering one or two frames.

Based on previous “edge-of-impossible” repairs, I know that the error rate must be close to zero. A 20kb frame, for example, contains 160,000 bits. If more than 2 or 3 of them are wrong, it becomes impossible to repair it.

This was absolutely true when he wrote it, but the amazing fact is that over the past 50 pages and few weeks worth of work, the participants here have greatly advanced the state of the art of video recovery, through the repair of the transport stream headers, the presentation timestamps, correction of individual macroblocks, and so on. And of course, the indispensible aid from Michael Niedermayer himself, the curator of the "ffmpeg" tool since 2004, was a key part of that advance.
« Last Edit: 05/19/2014 03:43 pm by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
Please forgive me if this has already been covered somewhere in the previous 50 pages....trying to get my understanding correct.

As I understand it I-frames are complete standalone frames - they need no other data to represent aa full frame. P frames are difference frames (as are B but I don't think they are used here). A p-frame contains all the information required to go from the previous P or I frame to the current frame. The result of applying a P-frame to the previous frame should be the current frame. A p-frame is a bunch of 64x64 macroblocks, plus offset information (where the 64x64 base data came from). These contain the difference information from wherever the MB came from and where is it now. So the current frame is basically the previous frame (fully rendered) with the current MB set applied over the top. The result should be a perfect representation of the frame, minus any high frequency compression differences (a bit like JPEG)

So, if you have a broken/completely wrecked) I-frame you can generate a new one by working out what the previous frame looks like, and using that. Simply inject a new i-frame in to the bitstream. Or, you could, if you have a partial I-frame, generate one from a combination of what you do have that looks OK, plus any data from the prewious frame.

And of course in H264 every frame can be an I-frame, so you could process through, creating a new bitstream from the old one, replacing all P-frames with I-frames.

Is that what people are attempting?
Hi JamesH,

Thanks for the idea. I believe it has not been suggested, but I don't think it will work in practice. It might work in situation where there is a I-frame broken due to some kind of localized media defect. But with random noise like we have its unlikely to be feasable.

The reason for this is that it is very unlikely that the cumulative errors from 19 P-frames are going to less than the errors in the I-frame following it. In other words: all errors propagate so the last P-frame before an I-frame is going to be the worst of all frames.

Besides that: we are not even close to decoding the 19th P-frame. More like first or second.

That is unless we get lucky and there are some intra-macroblocks in this last P-frame, then it might work a little.

But I like your thinking. Maybe you can come up with other ideas. It's a tough problem though when you go into the details. ;)

Regards,

arnezami

« Last Edit: 05/19/2014 03:42 pm by arnezami »

Offline JamesH

  • Full Member
  • ****
  • Posts: 525
  • United Kingdom
  • Liked: 284
  • Likes Given: 7
If I get a chance, I'll have a chat with our H264 codec team (I work for a company that amongst other things designs and build H264 encoder/decoder chips). It's not my field, though I do have to deal with it sometimes, and sadly the guy who sat opposite me until he left to cycle round NZ a few months ago was quite an expert.


Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
If I get a chance, I'll have a chat with our H264 codec team (I work for a company that amongst other things designs and build H264 encoder/decoder chips). It's not my field, though I do have to deal with it sometimes, and sadly the guy who sat opposite me until he left to cycle round NZ a few months ago was quite an expert.
Just point them to this thread (or the wiki). Any help is welcome. :)

Offline JamesH

  • Full Member
  • ****
  • Posts: 525
  • United Kingdom
  • Liked: 284
  • Likes Given: 7
Actually, thinking more about repairing I-frames, although you may not be able to reconstruct an entire I-frame from the previous P-frame set due to bad cumulative error, the P-frame set MAY be able to fill in gaps in the I-frame if valid data does exist. Might recover a few blocks here and there.

What is the I-frame rate in the bitstream btw? (Intra-refresh?)  If I were SpaceX I'd increase that at the expense of higher bandwidth - gives much better frame recovery in the event of transmission issues.



Offline SwissCheese

  • Full Member
  • *
  • Posts: 166
  • Liked: 256
  • Likes Given: 107

I will also try to figure this out myself. Since we trying something new here (P-frames in combination with others) we can expect some unexpected results. We'll just have to figure out what is going on.

Regards,

arnezami
Hi guys,

I think I have figured it out! :)

We were doing it wrong. By extracting the P-frames individually and then making png's of the mpg4-img files we were interpreting the P-frames as if they were I-frames! As you can imagine the mmb for such an instance would not work when the same frame was interpreted as a P-frame.

Attached are two interpretations of frame 170. One is simply the extraction of the png using the frame170.mpg4-img file. The other is an extraction of the png using fixed_edit8_part_169.ts with "FRAME0:0:0:-1". This mmb-command completely erases frame 169. Because of that you can see 170 in the next frame interpreted as a P-frame. Looks much better!

So when making mmb's for P-frames set all frames before it to "FRAMExx:0:0:-1". Then you will see the frame as interpreted as a P-frame.

Regards,

arnezami

PS. This is still an hypotheses, but I think this is actually correct or close to it.

That looks good! But when I try doing the same, I don't get this, I get +/- empty frames until the 6th P-frame... Could you generate the *.mpg4-img files for everyone? or at least a test sample between 169 and 189?

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 125
Actually, thinking more about repairing I-frames, although you may not be able to reconstruct an entire I-frame from the previous P-frame set due to bad cumulative error, the P-frame set MAY be able to fill in gaps in the I-frame if valid data does exist. Might recover a few blocks here and there.

What is the I-frame rate in the bitstream btw? (Intra-refresh?)  If I were SpaceX I'd increase that at the expense of higher bandwidth - gives much better frame recovery in the event of transmission issues.

There are plenty of opportunities to pull blocks forward to make up for missing data. At full speed it noticeably degrades video quality, so the less we have to do this the better.

We get an i-frame once every 1.3 seconds or so. I-frame plus 19 p-frames at 15 fps.

There are a number of things SpaceX could do to improve video transmission on future flights. Most importantly adding some sort of data integrity check and turning off interlacing as mvpel mentioned a few pages back. I think they are probably using an off-the-shelf video system, so any improvements would have to come from the vendor.

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
That looks good! But when I try doing the same, I don't get this, I get +/- empty frames until the 6th P-frame... Could you generate the *.mpg4-img files for everyone? or at least a test sample between 169 and 189?
Hi,

I posted all .mpg4-img files here in a zip called fixed_edit8_all_frames_mpg4-img.zip

What happens if you download fixed_edit8_part_169.zip. Unzip it. Then edit mmb_starting_from_169.txt to contain "FRAME0=0:0:-1=FRAME1=0:0:-1=FRAME2=0:0:-1". And run this command:

Quote
./ffmpeg -r 44999/3003 -mmb `cat mmb_starting_from_169.txt` -debug mb_pos_size -err_detect ignore_err  -i fixed_edit8_part_169.ts -f image2 fixed_edit8_part_169_%03d.png

You should get three gray frames followed by the 3rd P-frame. And you should be able to do this for every n-th frame. I will test this myself later when I have some more time but I expect this to work.

Regards,

arnezami

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
Hi,

I just tested it. It works fine for me.

See attachments  8)

Regards,

arnezami

PS. I used these mmmb's (multiple-mmb's)

For frame 170:
Quote
FRAME0:0:0:-1

For frame 171:
Quote
FRAME0:0:0:-1=FRAME1:0:0:-1

For frame 172:
Quote
FRAME0:0:0:-1=FRAME1:0:0:-1=FRAME2:0:0:-1

For frame 173:
Quote
FRAME0:0:0:-1=FRAME1:0:0:-1=FRAME2:0:0:-1=FRAME3:0:0:-1

And you can rinse and repeat this of course. Of course ignore the gray frames you've just disabled. They obviously won't contain data. The first non-gray picture is the one you're after. Which is non-disturbed by the gray frames before it.
« Last Edit: 05/19/2014 07:59 pm by arnezami »

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
That's really great stuff, Arnezami!

You can really see how each pframe relates back to the previous pframe, like in 171-172-173 where you see the expanding front of the water being blasted outward with the saturated white froth blocks remaining unchanged. It's an excellent way to visualize how the MPEG encoding actually works.
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline SwissCheese

  • Full Member
  • *
  • Posts: 166
  • Liked: 256
  • Likes Given: 107
That looks good! But when I try doing the same, I don't get this, I get +/- empty frames until the 6th P-frame... Could you generate the *.mpg4-img files for everyone? or at least a test sample between 169 and 189?
Hi,

I posted all .mpg4-img files here in a zip called fixed_edit8_all_frames_mpg4-img.zip

What happens if you download fixed_edit8_part_169.zip. Unzip it. Then edit mmb_starting_from_169.txt to contain "FRAME0=0:0:-1=FRAME1=0:0:-1=FRAME2=0:0:-1". And run this command:

Quote
./ffmpeg -r 44999/3003 -mmb `cat mmb_starting_from_169.txt` -debug mb_pos_size -err_detect ignore_err  -i fixed_edit8_part_169.ts -f image2 fixed_edit8_part_169_%03d.png

You should get three gray frames followed by the 3rd P-frame. And you should be able to do this for every n-th frame. I will test this myself later when I have some more time but I expect this to work.

Regards,

arnezami

I know, but I cannot manage to make the multiple mmb work correctly, still the same issue... Maybe I should try to install everything on a unix/linux machine?

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
I know, but I cannot manage to make the multiple mmb work correctly, still the same issue... Maybe I should try to install everything on a unix/linux machine?
Are you using the mmb's your earlier created for the mpg4-img files? Because they are indeed incompatible with multiple frame decoding. In other words: they are useless.

You have to create new mmb's by using the multiple mmb format and with the fixed_edit8_part_169.ts file. Or is that also not working for you?

I will try it myself soon.

Regards,

arnezami

[edit] To make it absolutely clear: you should not use the mpg4-img files for P-frames!
« Last Edit: 05/19/2014 06:21 pm by arnezami »

Offline moralec


The following is important for all of us that are working on the raw video stream file. I will attach 15 parts of raw_edit8.ts each containing a I-Frame and the following P-frames too a follow on post.

<snip>

Maybe someone can add a wiki page where people can write down which part they are fixing.

Cheers
Shanuson


I will take care of this tomorrow. I'm currently debating over different options... the current structure is not ideal. I hope I'll be able to come with a better design for the wiki, in order to keep things more organized. Any suggestions will be greatly appreciated.

Hello everyone. I have now finished updating the wiki. The main page now contains:

- A brief introduction
- Links to the most important technical posts from this thread (it was nice to read the entire thread again)
- Some external resources, particularly useful for those that are just starting to contribute,
- Links to the official tools we are using (FFmpeg utility, Online editors)
- A Progress Reports and Media section, for those that are mostly interested in the results.
- A Work in Progress with links to the pages where we are pasting the latest reconstructed iframes AND a new page for the work that is being done on the pframes
- Links to the places where the discussion is being held (Here and in IRC)
- Additional External Links

Enjoy

Offline Swatch

  • Full Member
  • **
  • Posts: 275
  • Official Aerospace Engineer as of June 13th, 2009
  • Cincinnati
    • ProjectApollo/NASSP: Virtual Systems and Flight Simulation of the Apollo Program
  • Liked: 52
  • Likes Given: 19
And as a general reminder/note to newcomers...

Wiki:  http://spacexlanding.wikispaces.com/
Ex-Rocket Scientist in Training, now Rocket Scientist!
M-F trying to make the world of the future a smaller place through expanding horizons...

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
This is fixed.
Great! Works well now :)

I've been thinking about the P-frames for the online editor. I think what would be useful and fairly easy to add is a third dropdown. What you get is first the option of the ts file (raw_edit8 for example). Then you select the "base I-frame" in the second dropdown. And then you can choose the actual frame (which would default to the first frame (=I-frame)) in the third dropdown.

In the case of edit8 the third drowdown not only contains the I-frame but also the P-frames within the chosen base I-frame. And you keep using the framexxx.mpg4-img files as your input to ffmpeg. That way more ppl can start working on the P-frames. In the case of try1 and coerced you only get one option in the third dropdown, which is always chosen.

Of course this will only show the P-frame individually (not fed by I-frame data). And we should also make sure we can run it though the multi-mmb call to ffmpeg, but thats for later I guess.

Regards,

arnezami
Hi IainCole,

Since I last posted this we have come to a better understanding of how to decode the P-frames. Most importantly it looks like we should not use the framexxx.mpg4-img files for P-frames!

Instead what can be done is the following:

* Let somebody choose a P-frame (as described in the above quote with a third dropdown)
* Instead of using ffmpeg with the mpg4.img file you should use this command:
Quote
./ffmpeg -r 44999/3003 -mmb <the mmb commands> -debug mb_pos_size -err_detect ignore_err  -i fixed_edit8_part_169.ts -f image2 fixed_edit8_part_169_%03d.png
Note that this command is only valid for the P-frames 170 - 188. So the 169 should be replaced by the I-frame number this P-frame "belongs to" (the P-frame is in a part that starts with an I-frame, so you need to tell ffmpeg to use that .ts). The mmb-commands are the mmb commands the user submit. But these commands are prefixed with a series of null-commands like this (in the case this is the fourth P-frame counted from the I-frame which would be 173):
Quote
FRAME0:0:0:-1=FRAME1:0:0:-1=FRAME2:0:0:-1=FRAME3:0:0:-1=FRAME4=
* Since this creates around 20 files it is wise to remove all the pngs expect the one from the fourth P-frame Keep in mind ffmpeg starts counting with 1 for the files while it starts counting with 0 for the mmbs. So it would be file number 5 in this case.
* [edit] As for the log: you probably don't want the entire log in the browser. Only the part that concerns the frame in question. You would somehow have to detect when it starts with that frame.

That way we should be able to create valid mmb's for P-frames online :).

Regards,

arnezami

« Last Edit: 05/19/2014 06:48 pm by arnezami »

OK, I'll try and think of a decent way to make this work

How am I supposed to interpret the log? before I was just looking as MB positions and parsing that to give you info n the editor. How does this work for p frames?

Offline arnezami

  • Full Member
  • **
  • Posts: 285
  • Liked: 267
  • Likes Given: 378
How am I supposed to interpret the log? before I was just looking as MB positions and parsing that to give you info n the editor. How does this work for p frames?

You could count the number of occurrences of this:

Quote
[force CFR for input from stream 0:0 @ 0x1b354a0] N:0 PTS:0 T:0.000000 POS:0 INTERLACED:0 -> PTS:0 T:0.000000
[AVIOContext @ 0x1b410e0] Statistics: 0 seeks, 1 writeouts

This is repeated for each frame. This is the next:

Quote
[force CFR for input from stream 0:0 @ 0x1b354a0] N:1 PTS:6006 T:400.809307 POS:21432 INTERLACED:0 -> PTS:1 T:0.066735
[AVIOContext @ 0x1ba7cc0] Statistics: 0 seeks, 1 writeouts

Also, the log was extended with the frame number:

Quote
[mpeg4 @ 0x1b49d00] MB pos/size: 0 002:03:01:3521 128 dc: 140 159 28 32 - 128 129, MB_type: 513, MV: 0 0

But I dont think thats a good way of filtering all log lines since error lines will not contain the frame number. So I would go for "[AVIOContext @ 0x1ba7cc0] Statistics: 0 seeks, 1 writeouts" and count the number of times you say that. Keep in mind that 0x1ba7cc0 could be different. Also the number of seeks and writeouts.

« Last Edit: 05/19/2014 07:09 pm by arnezami »

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
1