Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 948284 times)

Offline mvpel

  • Full Member
  • ****
  • Posts: 1124
  • New Hampshire
  • Liked: 1295
  • Likes Given: 1686
lum/chroma correction is a nightmare though, particularly row 10 (dirt spot)...the luminance error doesn't seem to be a propagation problem, i.e. it occurs in several blocks (which is odd...)

What if it's a rainbow effect from the cloud we're passing through, or reflected light from the engine flare? Peering at the reconstructed Youtube video you can get a little sense of that in some of the intervening pframes.

Edit:

See, for example, pale orange MB's in seconds 2 and 3, and nice orange flame around second 9 at the left side.
« Last Edit: 05/18/2014 07:18 PM by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline SwissCheese

  • Full Member
  • *
  • Posts: 164
  • Liked: 249
  • Likes Given: 90
Added some data to I-frame 248, also difficult with dc values, it can for sure be improved a lot:

0:0:-1,
37:0:3175,
27:1:-1,
29:1:5452,
35:2:-1,
42:4:15183,
5:9:-1,
12:9:29306,

edit: argh made a mistake at the top and all dc correction are awful now... so giving the mmb without corrections... see the wiki for the full mmb used in the image
« Last Edit: 05/18/2014 10:44 PM by SwissCheese »

Offline SwissCheese

  • Full Member
  • *
  • Posts: 164
  • Liked: 249
  • Likes Given: 90
A gif animation (reconstruction with matlab as described in a previous post) and all my mmb for the P-frames.
« Last Edit: 05/18/2014 07:45 PM by SwissCheese »

Offline moralec

A gif animation (reconstruction with matlab as described in a previous post) and all my mmb for the P-frames.

That looks super!

Offline Xspace_engineerX

  • Member
  • Posts: 51
  • Liked: 31
  • Likes Given: 19
A gif animation (reconstruction with matlab as described in a previous post) and all my mmb for the P-frames.

Wow. The quality in that is much, much better than I thought could be achieved. You guys are incredible.

Offline moralec

PROGRESS REPORT: WHAT WE KNOW SO FAR

For almost three weeks, we have been working on a video that was (for the most part) indecipherable. After many hours of work from a bunch of highly committed experts, now we have enough images to actually interpret what exactly happened one month ago (April 18), with Space X's CRS-3 first stage booster. And let me tell you: the recovered images tell an amazing story!

For this analysis, I considered all the i-frames that are on the wiki and a few p-frames that have been posted here. To put things on a time line, I used the same clock that appears on the video itself.

Phase 1: Controlled Descent (Frames 1-52)
The first 4 seconds of the video (19:35:01-19:35:04), reveal the controlled descent of the booster. This part of the video is composed of 4 i-frames and around 48 p-frames. All i-frames from this phase where successfully extracted, but only two have been repaired so far. To this day, the p-frames also remain untouched. As most of the work has been oriented into other (more interesting) sections of the video, the effort to recover images from this phase has been limited.

However, with the small number of images we have obtained so far, we can confirm that the booster went in a controlled descent over the ocean, falling vertically in the intended position for landing.  As we do not have any reference points, at this time it is impossible to asses the degree of rotation or the speed of the rocket. However, we can confirm that the booster was in one piece and looking good (expect for some splashes of dirty water, that actually have played an important role in the reconstruction of the frames)

Phase 2: Leg Deployment (Frames 72-150)
The next seven seconds of the video (19:35:5-19:35:12), with 5 iframes and about 80 p-frames, show the entire process of leg deployment. All five iframes have been fixed, and p-frames are slowly starting to emerge. From these, we can confirm that the leg deployment was successful. The latest version of frame 72 show no legs yet but leg deployment is clearly visible on frame 92. We can assume that the process started somewhere in between. 

From the images we have so far, it seems that there was a slight difference between how right and left legs deployed. It seems either the left one (from the perspective of the camara) deployed some milliseconds earlier or the legs actually deployed at different speeds. Legs appear to reach a 90 degree angle with respect of rocket around frame 131 (in this frame they appear to be longer for this reason). That frame has also the best view of the legs, and with that we can confirm that they were in prime condition (there was some speculation here at some point of a possible hole in the right leg: that was just an image artifact that has been used by some here to actually get the macro blocks in the right place). Deployment continues and legs get to be close or at landing position at iframe 150. The precise end of leg deployment is somewhere in the p-frames that precede or follow iframe 150.

Phase 3: Touch down (Frames 169-229)
The next five seconds of the video (19:35:12-19:35:17), show a successful landing. We currently have 4 very clear iframes and several fixed p-frames from this phase. The rocket continued to fall vertically, this time with the legs (at least the two visible ones) completely deployed. The supersonic exhaust stream from the rocket becomes visible on iframe 189, as it hits the water and dissipates horizontally. Considering the tests done with the grasshopper and the 9FR, we can assume that the engine has been running all this time, but the flame was not visible before as it was directly under the rocket (I'm guessing only the center engine is running, while the ones outside are only used during lunch).  From the most recent p-frames (the sequence was just posted by SwissCheese) in this stage we can confirm that there is almost no rotation in the rocket body (we have enough detail on the ocean to see how things move from one frame to the next). Flame becomes smaller as the rocket gets even closer to the water (Frame 209). Actual touchdown is somewhere in the p-frames between 209-229. On iframe 229, legs seem to be submerged in water following a sucesful touchdown. 8)

Phase 4: Tip Off (Frames 229+)
The last two seconds (19:35:18-19:35:20) show what happens to the booster after it hits the water.  We currently have 2 reconstructed images of this phase. Legs appear to partially resurface after the landing. Instants later, the booster starts to tip off. From the original video it seems that it fells on its right side (we will be able to confirm this, once we analyze the pframes from this section). Camera finally dies, only two p-frames after iframe 268.

Many times, when things develop slowly, changes tend to pass unnoticed. So let's take a moment to step back, and actually reflect on what has been achieved.  Many of us came to this forum for the first time, looking for more details on this landing. And well... we have been able to obtain them!

Thanks to everybody that has contributed in some way or another! But in particular to the guys that have been working directly with the frames, and developing the different systems that we all have been using! You are all amazing!

Lets continue with the great work! there are still many frames to be fixed, and many more details to be uncovered!
« Last Edit: 05/19/2014 08:40 PM by moralec »

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345
Do you get "nice images"? For me it corrects the first image but does nothing to the next ones...

I checked the code to understand a bit better, and put a printf there:

[snip]

it gives me for example this:
frame number ffmpeg: 3
frame number mmb: 3
token_frame = 6:20:-1,7:20:84967

it reads correctly all mmb sequences, and works as expected for the frame 0, but after that the frame numbers jump between frames 1,2 and 3 quite randomly. Is it the normal behavior? At this point I'm really lost... Maybe it's something related to windows?

Having this in the web app would for sure help a lot, we would have a reference program and only one set of data ;)

I also have mmbs for several P-frames, but can only use them on stand-alone *.mpg4-img frames
I will take a look into it, but my feeling is starting to be that P-frames act different individually than they do in combination with other frames. Even without mmb-changes. For example frame 170 looks very different individually compared to what it seems to change to frame 169 if its combined with it. Not sure yet.

Here is one test you can do. Add this green line of code:

Quote
static int get_bitpos_from_mmb_part (MpegEncContext *s, GetBitContext *gb, GetBitContext *gb_blank, int mb_x, int mb_y, const char *mmb_part, int *do_reverse, int *nr_of_hits_to_ignore) {

[snip]

        if (mb_x == mmb_x && mb_y == mmb_y && mb_y >= 0 && mb_x >= 0) {
            int i;
            if (mmb_pos < -2 || mmb_pos > gb->size_in_bits - 1) {
                av_log(NULL, AV_LOG_ERROR, "mmb bit pos invalid\n");
                return INT_MIN;
            }

            av_log(NULL, AV_LOG_ERROR, "new bitpos for %d:%d: from %d to %d\n", mmb_x, mmb_y, bitpos, mmb_pos);
            bitpos = mmb_pos;
That way you can see for which blocks it is actually changing its position. It will be colored red since its logged as error.

I will also try to figure this out myself. Since we trying something new here (P-frames in combination with others) we can expect some unexpected results. We'll just have to figure out what is going on.

Regards,

arnezami
« Last Edit: 05/18/2014 08:30 PM by arnezami »

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345
A gif animation (reconstruction with matlab as described in a previous post) and all my mmb for the P-frames.
Wooow! Super!! :)

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
lum/chroma correction is a nightmare though, particularly row 10 (dirt spot)...the luminance error doesn't seem to be a propagation problem, i.e. it occurs in several blocks (which is odd...)

What if it's a rainbow effect from the cloud we're passing through, or reflected light from the engine flare? Peering at the reconstructed Youtube video you can get a little sense of that in some of the intervening pframes.

Edit: -video-

See, for example, pale orange MB's in seconds 2 and 3, and nice orange flame around second 9 at the left side.

interesting point! I was also thinking that since the encoding is interlaced (right?), a very sudden change in brightness could actually result in different luminance values within the same macroblock. Apparently (according to the spec sheet posted earlier 6.1.3.6), the brightness values are not recorded simultaneously for all 4 subblocks...

Offline billh

  • Full Member
  • ***
  • Posts: 358
  • Houston
  • Liked: 244
  • Likes Given: 141
It's been said many times but, really, you guys are amazing!!

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Frame 92:

23:14:49203 gives you a piece of the circle between the legs. Only the left blocks are correct though... Doesn't help the right side of the right leg either :p

Also there is something at 54419. I thought it was the edge of the rocket base but it doesn't fit anywhere. Might be a wave...

I think it *is* part of the rocket base! Put it at 24:15, perfect match. You can also remove what's at 34:15 to check, it works :) Nice find!
« Last Edit: 05/18/2014 09:37 PM by Untribium »

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345
PROGRESS REPORT

For one month now, we have been working on a video that was (for the most part) indecipherable. After many hours of work from a bunch of highly committed experts, now we have enough images to actually interpret what exactly happened with Space X's CRS-3 first stage booster, that evening of April 18. And let me tell you: these images tell an amazing story!

For this analysis, I considered all the recovered frames that are on the wiki and a few p-frames that have been posted here. To put things on a time line, I used the same clock that appears on the video itself.

Phase 1: Controlled Descent (Frames 1-52)
The first 4 seconds of the video (19:35:01-19:35:04), reveal the Controlled Descent of the booster. This part of the video isof 4 iframes and around 48 p-frames. All iframes from this phase where successfully extracted, but only two have been repaired. The p-frames also remain untouched The effort to recover images from this phase has been limited, as most of the work has been oriented into other sections of the video.

However, with the small number of images we have obtained so far, we can confirm that the booster went in a controlled descent over the ocean, falling vertically in the intended position for landing.  As we do not have any reference points, at this time it is impossible to asses the degree of rotation or the speed of the rocket. However, we can confirm that the booster was in one piece and looking good (expect for some splashes of dirty water, that actually have played an important role in the reconstruction of the frames)

Phase 2: Leg Deployment (Frames 72-150)
The next seven seconds of the video (19:35:5-19:35:12), with 5 iframes and about 80 p-frames, show the entire process of leg deployment. All five iframes have been fixed, and p-frames are slowly starting to emerge. From these, we can confirm that the leg deployment was successful. The latest version of frame 72 show no legs yet but leg deployment is clearly visible on frame 92. We can assume that the process started somewhere in between. 

From the images we have so far, it seems that there was a slight difference between how right and left legs deployed. It seems either the left one (from the perspective of the camara) deployed some milliseconds earlier or the legs actually deployed at different speeds. Legs appear to reach a 90 degree angle with respect of rocket around frame 131 (in this frame they appear to be longer for this reason). That frame has also the best view of the legs, and with that we can confirm that they were in prime condition (there was some speculation here at some point of a possible hole in the right leg: that was just an image artifact that has been used by some here to actually get the macro blocks in the right place). Deployment continues and legs get to be close or at landing position at iframe 150. The precise end of leg deployment is somewhere in the p-frames that precede or follow iframe 150.

Phase 3: Touch down (Frames 169-229)
The next five seconds of the video (19:35:12-19:35:17), show a successful landing. We currently have 4 very clear iframes and several fixed p-frames from this phase. The rocket continued to fall vertically, this time with the legs (at least the two visible ones) completely deployed. Flame from the rocket becomes visible on iframe 189, as it hits the water and dissipates horizontally. Considering the tests done with the grasshopper and the 9FR, we can assume that the engine has been running all this time, but the flame was not visible before as it was directly under the rocket (I'm guessing only the center engine is running, while the ones outside are only used during lunch).  From the most recent p-frames (the sequence was just posted by SwissCheese) in this stage we can confirm that there is almost no rotation in the rocket body (we have enough detail on the ocean to see how things move from one frame to the next). Flame becomes smaller as the rocket gets even closer to the water (Frame 209). Actual touchdown is somewhere in the p-frames between 209-229. On iframe 229, legs seem to be submerged in water following a sucesful touchdown. 8)

Phase 4: Tip Off (Frames 229+)
The last two seconds (19:35:18-19:35:20) show what happened to the booster after it hit water.  We currently have 2 reconstructed images of this phase. Legs appear to have partially resurfaced after landing, and afterwards the booster started to tip off. From the original video it seems that it fell on its right side. Camera finally died, only two p-frames after iframe 268.

Many times, when things develop slowly, changes tend to pass unnoticed. So let's take a moment to step back, and actually reflect on what has been achieved.  Many of us came to this forum for the first time, looking for more details on this landing. And well... know we have been able to obtain them!

Thanks to everybody that has contributed in some way or another! But in particular to the guys that have been working directly with the frames, and developing the different systems that we have been using! You are all amazing!

Lets continue with the great work! there are still many frames to be fixed, and many more details to be uncovered!
Love this post.  :)

Offline Shanuson

  • Full Member
  • **
  • Posts: 282
  • Liked: 197
  • Likes Given: 498

The following is important for all of us that are working on the raw video stream file. I will attach 15 parts of raw_edit8.ts each containing a I-Frame and the following P-frames too a follow on post.

<snip>

Ok, now had time to make those parts: There is overlap between those parts. except for the last and the first it is always one 188 byte TS Packet before and one Packet after a block of I+P frames. This can give you some reference information on CC or so. Maybe you have to look also into raw.ts or raw_edit8.ts for further info if that block is messed up.
rawsplit_part_0 contains everything before the first I-Frame. Don't know if that is usefull at all.
rawsplit_part_15 contains only 2 P-frames AFAIK.
part_1 to part_14 should contain 1 I-Frame and 19 P-Frames each.
My idea is that people can pick on of the 15 and start fixing the TS-headers and more importantly the  P-Frame headers. I wrote how those should look like in the post quoted. First try to find the position of the P-Frame headers, they should be more or less equally distributed through the file. See if the counter bytes are messed up, calculate what they are, compare what they should be and change the header bytes if necessary.
If you have questions you can write me or ask here. I will check when I have more time.
When all parts are fixed we can put them all pack together and have us a new improved ts file from which we can hopefully extract all Frames.

Maybe someone can add a wiki page where people can write down which part they are fixing.

Cheers
Shanuson

But even with the work done already the latest video looks really impressive considering with what we started :D
« Last Edit: 05/18/2014 11:51 PM by Shanuson »

Offline Sohl

  • Full Member
  • **
  • Posts: 241
  • Liked: 77
  • Likes Given: 214
Yes, very nice summary post, moralec!

Offline mvpel

  • Full Member
  • ****
  • Posts: 1124
  • New Hampshire
  • Liked: 1295
  • Likes Given: 1686
My idea is that people can pick on of the 15 and start fixing the TS-headers and more importantly the  P-Frame headers. I wrote how those should look like in the post quoted. First try to find the position of the P-Frame headers, they should be more or less equally distributed through the file. See if the counter bytes are messed up, calculate what they are, compare what they should be and change the header bytes if necessary.

The transport stream headers are mostly correct after the work to suss out the correct program IDs and sequence numbers, right? Are there any details we need to know about that? Also, if you have any other links regarding bitmaps for the packetized elementary stream headers and what have you, that'd be helpful. Thanks for all your work!
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline Untribium

  • Member
  • Posts: 32
  • Switzerland
  • Liked: 32
  • Likes Given: 78
Improved frame #92 and cleaned up the mmb...this is tough...
X:54419:00,X:53655:00,0:0:550,14:0:-2:0:0:0:0:12:-6,2:5:14749,39:7:-1,18:9:27558,25:12:39009,29:12:-1,
17:13:42121,18:13:42526,10:14:-1,18:14:47626,26:14:-2:0:0:0:0:8:-6,30:14:50837,1:15:-1,5:15:51900,9:15:-1,
21:15:53655,27:16:61466,42:19:-1,43:19:83153,43:20:-1,1:21:90585,11:22:-1,17:23:105394,22:24:-1,
23:24:111581,0:27:-1,25:27:125621,28:28:-1,3:29:140154

Offline moralec


The following is important for all of us that are working on the raw video stream file. I will attach 15 parts of raw_edit8.ts each containing a I-Frame and the following P-frames too a follow on post.

<snip>

Maybe someone can add a wiki page where people can write down which part they are fixing.

Cheers
Shanuson


I will take care of this tomorrow. I'm currently debating over different options... the current structure is not ideal. I hope I'll be able to come with a better design for the wiki, in order to keep things more organized. Any suggestions will be greatly appreciated.

Offline wronkiew

  • Full Member
  • *
  • Posts: 186
  • 34.502327, -116.971697
  • Liked: 105
  • Likes Given: 121
I updated frame 131 (and can finally make .png as output):

Frame 131 from edit8:

X:614:84,X:638:1,X:822:a1,X:1166:4,X:1246:8,X:1430:4,12:0:-1,
7:2:5895,4:7:-1,11:7:20490,26:14:-1,18:15:56376,13:16:-1,
14:16:62505,20:16:-1,21:16:64068,27:16:-2,28:16:-1,32:16:65707,
25:20:-1,14:21:92080,24:25:-1,14:26:117098,41:26:-1,
14:27:120887,10:29:-1

Edit: Linked SwissCheese's previous version.
« Last Edit: 05/19/2014 04:36 PM by wronkiew »

Offline arnezami

  • Full Member
  • **
  • Posts: 282
  • Liked: 262
  • Likes Given: 345

I will also try to figure this out myself. Since we trying something new here (P-frames in combination with others) we can expect some unexpected results. We'll just have to figure out what is going on.

Regards,

arnezami
Hi guys,

I think I have figured it out! :)

We were doing it wrong. By extracting the P-frames individually and then making png's of the mpg4-img files we were interpreting the P-frames as if they were I-frames! As you can imagine the mmb for such an instance would not work when the same frame was interpreted as a P-frame.

Attached are two interpretations of frame 170. One is simply the extraction of the png using the frame170.mpg4-img file. The other is an extraction of the png using fixed_edit8_part_169.ts with "FRAME0:0:0:-1". This mmb-command completely erases frame 169. Because of that you can see 170 in the next frame interpreted as a P-frame. Looks much better!

So when making mmb's for P-frames set all frames before it to "FRAMExx:0:0:-1". Then you will see the frame as interpreted as a P-frame.

Regards,

arnezami

PS. This is still an hypotheses, but I think this is actually correct or close to it.
« Last Edit: 05/19/2014 08:50 AM by arnezami »

Offline Shanuson

  • Full Member
  • **
  • Posts: 282
  • Liked: 197
  • Likes Given: 498
My idea is that people can pick on of the 15 and start fixing the TS-headers and more importantly the  P-Frame headers. I wrote how those should look like in the post quoted. First try to find the position of the P-Frame headers, they should be more or less equally distributed through the file. See if the counter bytes are messed up, calculate what they are, compare what they should be and change the header bytes if necessary.

The transport stream headers are mostly correct after the work to suss out the correct program IDs and sequence numbers, right? Are there any details we need to know about that? Also, if you have any other links regarding bitmaps for the packetized elementary stream headers and what have you, that'd be helpful. Thanks for all your work!


All 15 I-Frame headers are ok. with all the right values for the variable parts.
For the 4 Byte TS-Header, there are good parts and bad parts in the ts-file. But at least they are all aligned now i think. so every 188th byte starts a new TS-packet like it should be. That does not mean there is a "47"-byte at every 188th position. Sometimes there are huge gaps spanning 10-20 ts-packets with no correct ts-header.

See the picture below. It shows the first few ts-packets of rawsplit_part_8.ts. You can see where Ts-Packets should be, marked by the black line. Blue circles show the correct CC bytes, in Ts-Headers in green are not correct. there are some bytes wrong, due to biterrors. Red shows the I-Frame Header. Cyan marks the two variable parts of the header, which are increasing.

Here they are in dez:
0x329c2B -> 3251243
0x8d228d -> 9249421

The following P-header is: 47 43 E8 3E 07 10 00 31 A7 E6 7E 00 00 00 01 E0 00 00 81 80 07 21 01 8D 51 79 FF FF 00 00 01 B6 51
coloured are both variable parts: there value in dez is:
31A7E6 -> 3254246 -> 3003 more than before, this is correct, because 20*3003=60060, the difference between I-Frames

The second is more complicated I just found out: There is a no equal distance between all PTS values of the I-Frames, but there is a pattern: So i would suggest to decode all PTS values and try to find a pattern. Apply it to all those where the pattern brakes.

       PTS Hex   PTS Dez   Difference to the last value
IF1    59797D     5863805   
IF2    6123ED     6366189   502384
IF3    67CE5D     6803037   436848
IF4    6F78CD     7305421   502384
IF5    77233D     7807805   502384
IF6    7DCDAD     8244653   436848
IF7    85781D     8747037   502384
IF8    8D228D     9249421   502384
IF9    93CCFD     9686269   436848
IF10   9B776D    10188653   502384
IF11   A321DD    10691037   502384
IF12   A9CC4D    11127885   436848
IF13   B176BD    11630269   502384
IF14   B9212D    12132653   502384
IF15   BFCB9D    12569501   436848


This was wrong, PTS has to be decode since it is 33 Bits spread over 5 bytes. See here

I hope this helps some.
« Last Edit: 05/22/2014 09:55 AM by Shanuson »

Tags: