Author Topic: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread  (Read 1201821 times)

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1020 on: 05/26/2014 10:24 am »
I cleaned the transport stream headers on part 131, attached below.

My notes during the cleaning:

Packet 163 at offset 30644 is a VOP start frame, but it needs a bit of attention. Unlike the other start packets, it has an 11-byte adaptation field and the 000001e0 starts 4 bytes later in the ts packet at offset 0x7708. I adjusted the AF length to allow the 0x000001e0 to line up as needed, but I'm not sure why this one is different and what the extra four bytes are for.

The four bytes in question, 0x181803c0, register as padding based on the adaptation field's bitmap, so theoretically should be 0xff, but I'm not sure if it might be something else that's supposed to fall in those four bytes. But it does appear that a reasonable-looking pframe came out.


Thanks for your work:
First this is part_8.

There is no 4 bit offset. but some nice bitflips that make it look like there is:
               from raw: 8F 9F D0 74 0E 20 00 63 67 52 F8 64 18 18 03 C0 00 01 01 E0 00 02 13 2A E1 DD FB 43 8C 00 06 D9 49
and shifted 4 byte left: 0E 20 00 63 67 52 F8 64 18 18 03 C0 00 01 01 E0 00 02 13 2A E1 DD FB 43 8C 00 06 D9 49 19 B0 11 0A
    and 4 bytes cut out: 8F 9F D0 74 0E 20 00 63 67 52 F8 C0 00 01 01 E0 00 02 13 2A E1 DD FB 43 8C 00 06 D9 49 19 B0 11 0A
      what it should be: 47 43 E8 3x 07 10 00 31 b3 a1 7E 00 00 00 01 E0 00 00 81 80 07 21 01 8D 80 65 FF FF 00 00 01 B6 5x

Comparing the first 3 to the last it shows that there is no 4 byte shift. Also the next packet is aligned with no 4byte shift.

Quote

At offset 0x12d18, this appears to be a null packet, but the continuity counter is off, the count goes from 0x1 to 0x2 across this gap in raw.ts as well:    null packet at 0x24a7ec and at 0x251324  However in raw.ts, this packet at 0x24ea04 reads 0x2e3ffe2f - much closer to 0x471fff1x than a 0x3e8 packet. Setting it to 0x471fff1f at 0x12d18 in fixed_edit8_part_131.ts - looks like it turned out okay.
That is ok.

Quote
Offset 0x21004 - a clear null packet with an incorrect CC - 0xd to 0x0 and later 0xe - perhaps an alignment packet inserted by Shanuson?
Sometimes there are single null packets inserted with a CC = 0 that break the CC for null packets. I tried to explain this before.
Quote
Offset 0x401ec - the adapation field length 0x77 (119) may be wrong here, but can't tell due to corruption. The last ff is at 0x4021A, which is 42 length, 0x2a, but we then have 0x7F (1 bit) 0xfc (1 bit) then a few more ff's. Leaving it at 0x77 since it doesn't exceed the packet length. The data at this AF length starts with 0x95591f6b.

The pframes look  promising, I've attached one of the better ones for your enjoyment - it's flames lapping out to the left side, and you can see the outline of the legs as the lighting changed from the previous frame.

This still needs work in the MPEG headers, but we should have a solid transport stream.

Edit: (Depressingly, the pframes that emerge from ffmpeg checksum as identical to the pframes out of the original ts file, but I guess you have to actually do the cleaning in order to find that out.)

The 77 looks ok for me. The corruption starts later in that packet.

I had to clean up a few TS-headers where the CC was not correct. Fixed the time_increment in the VOP headers.

Edit: fixed 4 minor things in the ts file, thanks mvpel for pointing them out.
« Last Edit: 05/26/2014 04:24 pm by Shanuson »

Offline mhenderson

  • Member
  • Posts: 71
  • USA
  • Liked: 101
  • Likes Given: 18
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1021 on: 05/26/2014 02:56 pm »
Narrative inserts for the final video.

These tell the background story and are meant to be interleaved with before / after video. They are in .html format, so anyone who cares to integrate with video can play in your browser and screen capture to record.

Wouldn't it be great to get somebody awesome to record this as a voiceover?  (My candidates include Tom Hanks, James Earl Jones, Leonard Nimoy, Bill Shatner, George Takei, Morgan Freeman, ... and Elon Musk) I would drop the scrolling graphics in a blink for a celebrity voice. Does anybody have an "in" with one of them?

Downlad and drop the five files in any folder, change the .txt extensions to .html, open the .html files with your browser.

(No javascript, just basic html.)

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1022 on: 05/26/2014 03:12 pm »
Thanks for pointing it out again and more clearly than I did before. It could be that the last few MB's of a frame are part of that last packet. So identifying this is important for the last row of a file.

I found an instance of this that I missed in the file you attached above:

0x6D6C : 47 03 E8 19 3E 00 FF ...

...should be...

0x6D6C : 47 03 E8 39 3E 00 FF ...

And there's another at 0x256FC:

0x257FC : 47 03 E8 1A 1E 16 ...

Looks like it should be:

0x257FC : 47 03 E8 3A A9 FF ...

... as it appears that the stuffing is corrupted for a little while but then clears up by the time it gets to 86 60 E0 C8 which appears to be the payload.

Also, should the next packet after this one, at 0x258B8, be 0x4740201D instead of 0x4740001D?

At offset 0x19EEC, the 0x4783E81F should be 0x4703E81F.

At offset 0x 2DCE8, there's a 0x4723E81F which should be 0x4703E81F.

The final packet of a frame at 0x351AC appears to not need to be aligned to the TS packet so it doesn't have an adaptation field to push the remaining payload over - it happens to end at the end of the packet.

Everything else looks good. Do you want to apply those changes to your file and modify the post above so we save some disk space?

Edit: Updates applied to the attachment in Shanuson's post, above.
« Last Edit: 05/26/2014 05:10 pm by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1023 on: 05/26/2014 04:04 pm »
For voice over, how about James Oliver Cromwell, who played Zefram Cochrane in Star Trek: First Contact?
http://en.m.wikipedia.org/wiki/James_Cromwell

The video's soundtrack would need "Magic Carpet Ride" in that case. :D
« Last Edit: 05/26/2014 07:15 pm by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline mhenderson

  • Member
  • Posts: 71
  • USA
  • Liked: 101
  • Likes Given: 18
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1024 on: 05/26/2014 04:42 pm »
Cromwell! That'll do, pig. That'll do. ;)

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1025 on: 05/26/2014 05:21 pm »
Here is rawsplit_part_2_fixed.ts again this time with all TS-data packets headers and stuff fixed.

I will now check all those parts that are submitted by others,

Done is part: 1,2,7,8,15

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1026 on: 05/26/2014 06:55 pm »
mvpel asked about this and probably a few others are also interested:

here are my 2 python scripts i use:

getheaderbitdistance.py:

This tries to find all P-Frame header positions in a file by comparing the data to a mask and counting bit discrepancies.
You have to un-comment the right I-Frame, so it can calculate the right PTS for each P-Frame:

The input is the TS file name

Output will be something like this:

read byte : 0
read byte : 50000
read byte : 100000
read byte : 150000
read byte : 200000
read byte : 250000
['47', '43', 'E8', '3x', '07', '10', '00', 'xx', 'xx', 'xx', '7E', '00', '00', '00', '01', 'E0', '00', '00', '81', '80', '07', '21', '01', 'xx', 'xx', 'xx', 'FF', 'FF', '00', '00', '01', 'B6', '5x']
trying to find header: 1
          -> ['47 43 E8 3x 07 10 00 34 67 ba 7E 00 00 00 01 E0 00 00 81 80 07 21 01 A3 50 C9 FF FF 00 00 01 B6 5x ']
0x415c 0 -> ['47 43 e8 34 07 10 00 34 67 ba 7e 00 00 00 01 e0 00 00 81 80 07 21 01 a3 50 c9 ff ff 00 00 01 b6 51 '] 0
0x21238 13 -> ['47 43 e8 32 07 10 00 34 c5 92 7e 00 00 00 01 e0 00 00 81 80 07 21 01 a5 c8 29 ff ff 00 00 01 b6 57 '] 48048

followed by a repetition of the last 4 lines for all 19 P-Frames.
Of this last 4 lines the first gives the P-Frame number, the next how it should look like, and finally the 2 closest data packets.
There the output is: position, bit distance, data , PTS distance to the right PTS in decimal.
In the end the right P-Frame header should have bit distance and PTS distance = 0, like it is the case here.

and the secound script is
tsheaderCCcheck.py
It also takes the file name as an argument.

it gives 4 blocks of information:
First the readin part like above
Next is about the time_increment_bits(TTB) part of the VOP-header.
It list for both I-Frame and P-Frame the position of the Header start; the first 5 byte after the startcode 0x000001b6; TTB in dec of the current Frame, the TTB in dec and then hex of what the next frame should be, TTB in dec and hex of what the last frame should be.

The last 2 parts are print outs of first the headers, and finally the tails of a Frame, at least that should be it. sometimes normal data packets are hidden in here if some ts-headers were not fixed correctly. So one has to look if there are only 20/21 headers and only tail packets with AF fields printed out.

Cheers
Shanuson

Offline saliva_sweet

  • Full Member
  • ****
  • Posts: 614
  • Liked: 476
  • Likes Given: 1826
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1027 on: 05/26/2014 09:40 pm »
Some slight improvements to iframe 72. My relationship with this frame is a case of stockholm syndrome by now.

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1028 on: 05/26/2014 10:09 pm »
I need some help.

I was working on part_13, the file from princess and mvpel.

There is one P-frame header that I can't find. (between 0x2f468 and 0x37374 in their file)

Attached you find the part of raw.ts that should include the missing P-frame. I starts with the previous P-frame (the one at  0x2f468 in the part file) and ends where the next P-frame begins (0x37374). Both PTS and the counter shows that there has to be an other P-frame between those two.
That part also includes a 56 byte shift, which i fixed by including 56 FFs and declaring one ts-packet a null packet. (it was actually 132 bytes too many for the CC of all PID 0x3e8 packets). The attached file does not include this fix.

The missing P-frame header is:
47 43 E8 3x 07 10 00 36 d5 71 7E 00 00 00 01 E0 00 00 81 80 07 21 01 B7 07 A5 FF FF 00 00 01 B6 5x

There is a bunch of FF's in the middle of the file (pos 0x4150) which i thought are part of the null packet. ( I added the 56 FFs here, so they are not part of a data-packet.) but maybe not.

Maybe someone of you can point out where that header should be.

Cheers
Shanuson

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1029 on: 05/26/2014 10:21 pm »
Also some good news:

Here are part 11 and part 14 based on the work of deruch and princess.

Cheers
Shanuson

Offline SwissCheese

  • Full Member
  • *
  • Posts: 166
  • Liked: 256
  • Likes Given: 107
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1030 on: 05/27/2014 12:36 am »
We are slowly adding frames to the movie, so I put an updated version. Big thanks in particular to the anonymous people who cleaned part 248!

97 frames (some of them only very partially recovered) out of ~270 were used.

Offline S.Paulissen

  • Full Member
  • ****
  • Posts: 442
  • Boston
  • Liked: 334
  • Likes Given: 511
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1031 on: 05/27/2014 01:08 am »
Wow, Swiss. That video is looking pretty awesome.  Those legs coming out is getting silky smooth, as is the engine fire/splashdown.
"An expert is a person who has found out by his own painful experience all the mistakes that one can make in a very narrow field." -Niels Bohr
Poster previously known as Exclavion going by his real name now.

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1032 on: 05/27/2014 01:11 am »
The missing P-frame header is:
47 43 E8 3x 07 10 00 36 d5 71 7E 00 00 00 01 E0 00 00 81 80 07 21 01 B7 07 A5 FF FF 00 00 01 B6 5x

The packet at the end of the range you mention, at 0x37374, has a PTS of 79.67960, while the earlier one has a contiguous PTS of 79.612866.

I'm looking over what I uploaded using Wireshark, filtering on mp2t.af.pcr_flag == 1. It's recognizing the following PTS values:

0 78.81206 PCR base = 2132727600
 1 78.87880
 2 78.94553
 3 79.01226
 4 79.0790
 5 79.14573
 6 79.21246
 7 79.27920
 8 79.34593
 9 79.41266
10 79.47940
11 79.54613
12 79.61286
13 79.67960
14
15 79.81306
16 79.87980
17 79.94653
18 80.01326


The only packet between 13 and 15 which has the PCR bit turned on looks like this:

0x37664 : 47 03 e8 37 80 b2 f1 13 0c 83 2b 2d 50 b5 e9 12 34 34 78

This is obviously wrong, since it only appears four TS packets after the previous PTS of 79.6796 and I'm pretty sure a pframe can't be that small, aside from the fact that it has ES priority and transport-private bits turned on which is incorrect. So that's not the one. And there doesn't seem to be enough room in that space to fit another pframe. Are the PTS numbers correct across this file and into the next and prior ones? Maybe something tripped over this packet, or the missing one, and got confused.

Anyway, based on the rough number of TS packets per pframe, I'm thinking that our most likely candidate is the TS packet at immediately following the null packet, at offset 0x33680. I know that null packets sometimes appear in the midst of pframes, but this one is far enough down that it seems like it might be padding between frames. And here's the actual bytes as compared to the expected pframe header:

47 03 E8 10 03 B2 05 5E 38 49 FB C9 2F 0E BC D0 73 CF E1 F8 92 0D 44 BE 5C 8B F8 F7 F1 F0 F4 DC 02
47 43 E8 3x 07 10 00 36 d5 71 7E 00 00 00 01 E0 00 00 81 80 07 21 01 B7 07 A5 FF FF 00 00 01 B6 5x


It's kind of a thin reed, I know, but perhaps mashing the header in there is worth a try to see if a pframe pops out. The PTS value of the next frame would need to be adjusted, as I discussed above, so that this one would get 79.67960 and 13 (above) would get 79.74633.

Edit: It also occurs to me that the bytes at the beginning of the null frame are far enough off from FF that they might be the end bytes of the previous pframe, and so the null packet there should be converted into a 0x4703e83x header and the bytes pushed to the end of the packet with an appropriate amount of stuffing.
« Last Edit: 05/27/2014 04:47 pm by mvpel »
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1033 on: 05/27/2014 01:15 am »
97 frames (some of them only very partially recovered) out of ~270 were used.

Wow, fantastic!

Since SpaceX actually designed and built the rocket, I'm thinking they should get top billing, right?  ;D
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline deruch

  • Senior Member
  • *****
  • Posts: 2422
  • California
  • Liked: 2006
  • Likes Given: 5634
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1034 on: 05/27/2014 02:12 am »

The missing P-frame header is:
47 43 E8 3x 07 10 00 36 d5 71 7E 00 00 00 01 E0 00 00 81 80 07 21 01 B7 07 A5 FF FF 00 00 01 B6 5x


I've looked, but can't seem to find it either.  Though I think PTS should be B6 07 A5.
Shouldn't reality posts be in "Advanced concepts"?  --Nomadd

Offline dorkmo

  • Full Member
  • ****
  • Posts: 710
  • Liked: 338
  • Likes Given: 848
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1035 on: 05/27/2014 02:42 am »
97 frames (some of them only very partially recovered) out of ~270 were used.

Wow, fantastic!

Since SpaceX actually designed and built the rocket, I'm thinking they should get top billing, right?  ;D

shoutout to wikispaces.com too. i bet theyre seeing quite a load of traffic.

ps is anyone backing up all the data?
« Last Edit: 05/27/2014 02:46 am by dorkmo »

Offline Shanuson

  • Full Member
  • ***
  • Posts: 395
  • Liked: 327
  • Likes Given: 2542
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1036 on: 05/27/2014 08:25 am »

The missing P-frame header is:
47 43 E8 3x 07 10 00 36 d5 71 7E 00 00 00 01 E0 00 00 81 80 07 21 01 B7 07 A5 FF FF 00 00 01 B6 5x


I've looked, but can't seem to find it either.  Though I think PTS should be B6 07 A5.

The last bit of the B7 byte has to be a 1 (it is a marker bit) so it is a 7.

I think i will take mvpels suggestion for the location.

Offline princess

  • Member
  • Posts: 65
  • Liked: 106
  • Likes Given: 25
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1037 on: 05/27/2014 11:46 am »
Here is my cleanup of part 10 of the split TS file. If Shanuson, mvpel and the others can have a look and let me know if I've missed anything, that would be great.

Offline mvpel

  • Full Member
  • ****
  • Posts: 1125
  • New Hampshire
  • Liked: 1303
  • Likes Given: 1685
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1038 on: 05/27/2014 12:13 pm »
Here is my cleanup of part 10 of the split TS file. If Shanuson, mvpel and the others can have a look and let me know if I've missed anything, that would be great.

Pretty good so far!

It looks like you may be leaving some of the bits in the header on - at offset 0x5F78  and 0x15C18, for example, the header is 0x4723e81x, which indicates that the transport priority bit is on. There's bits in the TS header which we know should always be turned off: the transport error indicator, transport priority, and scrambling control. Insuring that these are always off not only avoids problems if a decoder takes the TEI or scrambling seriously and skips the packet as a result, but it also makes it easier to visually scan the file in an editor to see that the headers are consistent, and makes pattern matching more straightforward.

This page shows the exact layout of the header: https://en.wikipedia.org/wiki/MPEG_transport_stream
"Ugly programs are like ugly suspension bridges: they're much more liable to collapse than pretty ones, because the way humans (especially engineer-humans) perceive beauty is intimately related to our ability to process and understand complexity. A language that makes it hard to write elegant code makes it hard to write good code." - Eric S. Raymond

Offline SwissCheese

  • Full Member
  • *
  • Posts: 166
  • Liked: 256
  • Likes Given: 107
Re: SpaceX Falcon 9 v1.1 CRS-3 Splashdown Video Repair Task Thread
« Reply #1039 on: 05/27/2014 12:54 pm »
Here is rawsplit_part_2_fixed.ts again this time with all TS-data packets headers and stuff fixed.

I will now check all those parts that are submitted by others,

Done is part: 1,2,7,8,15

Part 7 now has a (nearly empty) p-frame at the end (thus 1 i-frame and 20 p-frames in the file!), otherwise if we discard it, it looks good, part 8 also has now the right number of p-frames and could directly be put in the online editor.

Could you check a last time the end of part 7?

Thank you for everything!

Tags:
 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0