Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 853753 times)

Offline mvpel

  • Full Member
  • ****
  • Posts: 1116
  • New Hampshire
  • Liked: 1280
  • Likes Given: 1676
I think we can also assume that the transport error and the scrambling control bits should never be set in any header, right? That may help constrain the problem a bit. Turning off the transport priority bit in everything may also be helpful, since we're dealing with a file rather than an over-the-air stream and wouldn't expect an overworked decoder to need to choose which packet to deal with first.
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline cscott

  • Senior Member
  • *****
  • Posts: 2638
  • Liked: 1843
  • Likes Given: 658
About the Iframes, i was thinking that manual moving the macroblocks around will be difficult if you don't have a lot of reference (i.e. if there aren't many features on the block).

As it turns out, it appears that the dirty splash on liftoff has been a great help, as there are lots of small dirty smudges in each frame which provide reference features.

Offline Jakusb

  • Full Member
  • ****
  • Posts: 586
  • NL
  • Liked: 307
  • Likes Given: 99
About the Iframes, i was thinking that manual moving the macroblocks around will be difficult if you don't have a lot of reference (i.e. if there aren't many features on the block).

As it turns out, it appears that the dirty splash on liftoff has been a great help, as there are lots of small dirty smudges in each frame which provide reference features.

Yes, see my earlier post with an image annotated with 14 dirt spots that clear are on the lens.


A version with the dirt spots numbered for better reference in conversation

Edit: Added dirt group 14

« Last Edit: 05/12/2014 06:28 PM by Jakusb »

Offline Shanuson

  • Full Member
  • **
  • Posts: 258
  • Liked: 171
  • Likes Given: 412
About the Iframes, i was thinking that manual moving the macroblocks around will be difficult if you don't have a lot of reference (i.e. if there aren't many features on the block).

As it turns out, it appears that the dirty splash on liftoff has been a great help, as there are lots of small dirty smudges in each frame which provide reference features.

Yes, see my earlier post with an image annotated with 14 dirt spots that clear are on the lens.


A version with the dirt spots numbered for better reference in conversation

Edit: Added dirt group 14



Dont forget the one that is overlapping with part of the right leg. (wiki IF 5 shows a complete right leg)

Online mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1887
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 465
  • Likes Given: 188
Hi guys,

I have a bit of a mystery going on and it's holding me back from doing great stuff. ;) Hopefully somebody else has an idea what is going on.

This is essentially about how luminance/chrominance (errors) propagate. From what I understood so far each (macro)block propagates to its right (macro)block and to its bottom (macro)block. Luminance does it on a block (8x8pixel) basis while chrominance does it on a macroblock basis. So if you set macroblock 0,0 to "more red" the entire picture becomes that much "more red". Right?

Now here is my mystery example.

Take iframe 12 (try1) and enter 0:0:-1,13:24:72730 as mmb. It shows the bottom part of the picture and it looks quite nice.

Now enter 0:0:-1,12:24:72687 and suddenly you see a luminosity error appearing far from the block we just changed. How is this possible??

I suspect this has to do with the fact that the lower-right block of 24:26 is getting different input from above or left, but I don't understand why it could be so different from before. Maybe some kind of dc-clipping? (but there is no error log of that I think) Is this block reading the bits differently because it has different input? Because I always thought that the reading/meaning of the bits was not determined by the input.

I'm confused by this.

Regards,

arnezami

So this confused me for a while too. First off a block does not propagate to the left AND right. It is better to say that a given block propagates from EITHER the left OR the top block. Further, this is not defined in the file specifically. It is defined by comparing the block above, the block to the left, AND the block to the upper left. Based on the DC (brightness/color) of the blocks in these directions it PICKs which block to propagate from. Because of this when you adjust a value you can cause a given block to swap which block it inherits from.

This is even worse because this prediction direction even determines the scan order through the DCT table. So causing a block to swap which block it inherits from can cause the DCT scan order to become screwed up resulting in the pixels in the block to be transposed (mirrored and flipped vertically).
« Last Edit: 05/12/2014 06:44 PM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Lars_J

  • Senior Member
  • *****
  • Posts: 6161
  • California
  • Liked: 665
  • Likes Given: 195
A version with the dirt spots numbered for better reference in conversation

Edit: Added dirt group 14

Also, don't forget the large smudge under dirt #3 (just below to the left), which obscures/blurs part of the leg in every frame. (that has a leg)
« Last Edit: 05/12/2014 06:43 PM by Lars_J »

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 341
Hi guys,

I just noticed that there are no B-frames in the file. Only I-frames and P-frames.

Quote
./ffprobe try1.ts -show_frames > frames.txt

Not sure yet what the implications are yet. Hopefully it will get easier that way.

Regards,

arnezami

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 341
So this confused me for a while too. First off a block does not propagate to the left AND right. It is better to say that a given block propagates from EITHER the left OR the top block. Further, this is not defined in the file specifically. It is defined by comparing the block above, the block to the left, AND the block to the upper left. Based on the DC (brightness/color) of the blocks in these directions it PICKs which block to propagate from. Because of this when you adjust a value you can cause a given block to swap which block it inherits from.
Aha! Interesting!

Thank you for that. That way it does indeed make more sense.

Online mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1887
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 465
  • Likes Given: 188
I've seen this too. Either the offending block is messed up in some way that propagates the error, or MPEG behaves badly when values are clipped. The problem seems to be worse on frames where the top row is missing. I attached an image that shows the path from the bad block to where it gets noticeably bad.
Yeah. I'm starting to realize that it is even harder than I thought it would be.

I wanted to create a "clean enviroment" for checking bad block propagation by starting at the end and working backward therefore avoiding the issue of having propagation from above and left.

But this won't work if block propagation below and to the right is different when adding more blocks to the left and above.

Hmmm. A rethink is required here.

Thanks for that diff!

Regards,

arnezami


Quick thought from a quiet computer Geek in the audience.... Can Space X provide you with a good 30 sec chunk of video from the same setup so you can see what good video is supposed to look like?  That would get you ground truth as to formatting etc.  It would also give you the ability to selectively damage the bit stream to better understand propagation related errors.

Yes we asked about that. But nothing has come yet. Supposedly something is in the works.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline Shanuson

  • Full Member
  • **
  • Posts: 258
  • Liked: 171
  • Likes Given: 412
Parsing raw.ts using info from http://en.wikipedia.org/wiki/MPEG_transport_stream

The file consists of 188-byte ts-packets.

1st byte is always 47.

2 next bytes are 000<packetid13bits>, and apparently
they are always 03e8 for non-null ts-packets and 1fff for null ts-packets.

there are more than 2 PID values in the ts packets/ file


To find out how many different TS-PIDs exists, I counted all different "0x47xxxx" appearances in raw_edit1.ts:
This are all which occured 20 times or more:
20 471be8
20 4733e8
21 470fe8
25 4703e9
63 474703
74 474020
81 474000
269 4743e8
4968 471fff
15016 4703e8


I think the first 4 are permutations of the last ones by bitflips.
4743e8 and 4703e8 have the same PID,
When one checks where 474703 is found, one sees that the next byte is E8. So the first 47 is the last byte of a previous packet, and therefore 474703 is not part of a TS-header.
So we end up with only 4 PIDs (13 bit long):
0x0000, 0x0020, 0x03E8,0x1FFF
Each PID has its own continuity counter, which should help deciding which packet is which.
 
With some different flagbits we end with the following TS-headers:
0x4740001W
0x4740201X
0x4703E81Y, 0x4703E83Y and 0x4743E83Y
0x471FFF1Z
with the 4 different counters W,X,Y,Z
« Last Edit: 05/12/2014 07:08 PM by Shanuson »

Offline moralec

Offtopic: An updated version of the sequence from video I-Frames that I posed some pages ago, now with the legs clearly moving!



Offline Jakusb

  • Full Member
  • ****
  • Posts: 586
  • NL
  • Liked: 307
  • Likes Given: 99
Offtopic: An updated version of the sequence from video I-Frames that I posed some pages ago, now with the legs clearly moving!
Nice! Clearly great progress already.
If only we could somehow get the legs in Iframe 4 resolved...
I assume they should be just opened and slightly away from the stage. Or not

Ps: very much on topic if you ask me. ;)
« Last Edit: 05/12/2014 08:37 PM by Jakusb »

Online mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1887
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 465
  • Likes Given: 188
Ok, I am done with my work on the headers

attached you find a raw_edit1.ts where only the 15 I-Frame headers are changed (plus a few '47' bytes maybe), Also the one 188 byte block is reshifted by 4 bits to align it.

The last XX XX was part of vbv_occupancy (1 marker bit and the frist 7 of the latter half of vbv_occupancy)
According to michaelni, the actually value is not needed to extrace I-frames.

Cheers
Shanuson

I tried it, some iframes appear to have extra data and are clearer, others are worse. iframe8 is missing, and I'm not sure if there are new iframes or not. Several of the frames have apparently significantly less errors though.
« Last Edit: 05/12/2014 09:14 PM by mlindner »
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline IRobot

  • Full Member
  • ****
  • Posts: 1181
  • Portugal & Germany
  • Liked: 214
  • Likes Given: 203
As soon as the work finishes on recovering the information, I guess some post processing can be done, namely noise filtering/smothering and some morphing to create intermediate frames!

Offline Jakusb

  • Full Member
  • ****
  • Posts: 586
  • NL
  • Liked: 307
  • Likes Given: 99

Ok, I am done with my work on the headers

attached you find a raw_edit1.ts where only the 15 I-Frame headers are changed (plus a few '47' bytes maybe), Also the one 188 byte block is reshifted by 4 bits to align it.

The last XX XX was part of vbv_occupancy (1 marker bit and the frist 7 of the latter half of vbv_occupancy)
According to michaelni, the actually value is not needed to extrace I-frames.

Cheers
Shanuson

I tried it, some iframes appear to have extra data and are clearer, others are worse. iframe8 is missing, and I'm not sure if there are new iframes or not. Several of the frames have apparently significantly less errors though.

Any idea yet how to proceed?
Is it useful to add them to the online editor v2?
I am getting pretty curious to see this potential new or improved data in actual images...

Offline AncientU

  • Senior Member
  • *****
  • Posts: 5228
  • Liked: 3140
  • Likes Given: 4452
About the Iframes, i was thinking that manual moving the macroblocks around will be difficult if you don't have a lot of reference (i.e. if there aren't many features on the block).

As it turns out, it appears that the dirty splash on liftoff has been a great help, as there are lots of small dirty smudges in each frame which provide reference features.

Yes, see my earlier post with an image annotated with 14 dirt spots that clear are on the lens.


A version with the dirt spots numbered for better reference in conversation

Edit: Added dirt group 14



Dont forget the one that is overlapping with part of the right leg. (wiki IF 5 shows a complete right leg)
Also, don't forget the rocket body and it's dirty water 'paint job' which should remain constant except for lighting effects.
"If we shared everything [we are working on] people would think we are insane!"
-- SpaceX friend of mlindner

Offline Shanuson

  • Full Member
  • **
  • Posts: 258
  • Liked: 171
  • Likes Given: 412

Ok, I am done with my work on the headers

attached you find a raw_edit1.ts where only the 15 I-Frame headers are changed (plus a few '47' bytes maybe), Also the one 188 byte block is reshifted by 4 bits to align it.

The last XX XX was part of vbv_occupancy (1 marker bit and the frist 7 of the latter half of vbv_occupancy)
According to michaelni, the actually value is not needed to extrace I-frames.

Cheers
Shanuson

I tried it, some iframes appear to have extra data and are clearer, others are worse. iframe8 is missing, and I'm not sure if there are new iframes or not. Several of the frames have apparently significantly less errors though.

Any idea yet how to proceed?
Is it useful to add them to the online editor v2?
I am getting pretty curious to see this potential new or improved data in actual images...

First there is a small python script which helps to get the iframes.
After ffmpeg -vstats -i file.ts-vcodec copy -an -f image2 frame%d.mpg4-img produced all frames
use the following python script

import os
import sys

ld=os.listdir('./')

for ff in ld:
    if "frame" in ff:
        with open(ff, "rb") as f:
            b1 = f.read(4).encode("hex")
            i=0       
            #print b1,ff
            if b1=='000001b0':
                print ff
                os.system('cp '+ff+' i'+ff)


It will make copys of those frames that are I-frames.

attached you find a zipfile containing all 15. But some still need some work, 1 is only 78 byte large, will another is of 30Kb.
But this is progress. Maybe to early to add them to the wiki, i dont know.

Edit: maybe should give a maping from the new ones to the old ones:
New -> Old:
iframe1.mpg4-img -> IF0
iframe14.mpg4-img ->
iframe32.mpg4-img ->  IF1
iframe52.mpg4-img -> IF2
iframe72.mpg4-img -> IF4
iframe92.mpg4-img -> IF5
iframe112.mpg4-img  ->  IF6
iframe131.mpg4-img  ->
iframe150.mpg4-img  ->  IF7
iframe169.mpg4-img  ->  IF8 
iframe190.mpg4-img  -> IF9
iframe210.mpg4-img  -> IF10
iframe230.mpg4-img  -> IF11
iframe249.mpg4-img  -> IF12
iframe269.mpg4-img  -> IF13
 

Edit2:Fixed the 78byte IF, it is now 20472 Bytes large
« Last Edit: 05/12/2014 09:58 PM by Shanuson »

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 116
As soon as the work finishes on recovering the information, I guess some post processing can be done, namely noise filtering/smothering and some morphing to create intermediate frames!

Already working on it. Want to help?

Online mlindner

  • Software Engineer
  • Full Member
  • ****
  • Posts: 1887
  • Space Capitalist
  • Silicon Valley, CA -- previously in Ann Arbor, MI
  • Liked: 465
  • Likes Given: 188
As soon as the work finishes on recovering the information, I guess some post processing can be done, namely noise filtering/smothering and some morphing to create intermediate frames!

Already working on it. Want to help?

A lot of work still needs to be done on iframes before that can be done.
Internal combustion engine in space. It's just a Bad Idea.TM - Robotbeat

Offline saliva_sweet

  • Full Member
  • ****
  • Posts: 508
  • Liked: 380
  • Likes Given: 1252
To find out how many different TS-PIDs exists, I counted all different "0x47xxxx" appearances in

Back when I was messing with the raw.ts I looked at the PMTs (because I thought corrupted PMTs may mess up decoding I made a consensus PAT and PMT and attached those to the start of the file and removed all others, don't think it made much difference). I remember there were 3 PIDs defined. 3e8 - video, 3e9 and 3f0. I have no idea what the last two are, but I think they're unneeded. Those packets may still be present though, but I don't remember actually seeing one. Even if the header said 3e9 the counter suggested it was actually still 3e8.

This is even worse because this prediction direction even determines the scan order through the DCT table. So causing a block to swap which block it inherits from can cause the DCT scan order to become screwed up resulting in the pixels in the block to be transposed (mirrored and flipped vertically).

That's just disgusting.

If only we could somehow get the legs in Iframe 4 resolved...
I assume they should be just opened and slightly away from the stage. Or not

I've spent the last two evenings on this and it's been awful. I'm inching closer, but the data is in dreadful condition. I've seen no leg at all. They appear out of thin air in iframe 5 for all I can tell. I'm calling it a night for today. My best effort so far is pretty abysmal, but still shows a bit more than the previous efforts (I hope it's more or less correct)

mmb: 5:1:4200,0:3:10968,8:3:14762,28:3:16550,24:4:20286,40:4:-1,9:6:21857,13:11:39488,37:11:41549,
10:12:42693,29:12:43895,21:13:50986,10:14:-1,13:14:-1,21:15:-1:60:60::::,22:15:55249,2:16:-1,
8:16:58952,13:16:59649,27:16:63538,18:17:69075,27:17:-1,13:18:74459

Tags: