Is it valid to cut / paste data from adjacent frames to allow the codec to continue? IMHO, absolutely yes. That is a necessary part of the restoration. Let the brave little booster sing.
Quote from: Quialiss on 06/09/2014 10:41 pmIIRC, SwissCheese and I worked on those frames.. from our notes on the wiki, 269 and 270 were completely unrecoverable in fix8 version, and some data became recoverable in the raw_final_fixed.ts, though they're still missing large chunks of data. But they don't appear to pause(outside of what's caused by missing a third of a frame) and the timestamps all look correctly recovered.The TS timestamps for 270 were altered to make the frame look correct, but I don't know what happened with the MPEG4 stream.Basically at that point in the data the stream was all fine with no errors, and then suddenly from around offset 0x003dbb04 to offset 0x003dd284 there was a huge burst of corruption that knocked out 32 TS packets. If you look at it with a TS analyser it's very bad, just totally mangled.Anyway, the boundary between 269 and 270 occurred in the middle of this. Usually the corruption is light enough that you can kind of eyeball the closest packet, or break out the Hamming-distance analysis, but in this case it's useless. Whoever did the byte duplicate in raw_final_fixed.ts was basically just pasting the start of 271 into this and resetting the timestamp to be correct. It's the TS equivalent of saying "this macroblock is shot to pieces, use one that looks similar".So yes, this will allow extra data to be recovered - the good macroblocks in 270 after the corruption burst finished. But be aware that the end of 269 and the start of 270 are mangled beyond recovery.
IIRC, SwissCheese and I worked on those frames.. from our notes on the wiki, 269 and 270 were completely unrecoverable in fix8 version, and some data became recoverable in the raw_final_fixed.ts, though they're still missing large chunks of data. But they don't appear to pause(outside of what's caused by missing a third of a frame) and the timestamps all look correctly recovered.
But it should be frames 254 and 255 where this happened.
Quote from: Shanuson on 06/10/2014 07:55 amBut it should be frames 254 and 255 where this happened.Ah yes, good point! When I was working out which frames it occurred in, I forgot that raw_final_fixed.ts hasn't got the first 15 P-frames before the I-frame. The corruption is in 269/270 in rocket.ts and 254/255 in raw_final_fixed.ts.Thank you for the correction Shanuson!
I had to correct a little bit i frame 121 to align it to the p-frames. The problem is that the colors are wrong now and I'm not good at correcting them... Could someone recover nicer colors?The new mmb is in attachment, I haven't posted it to the wiki yet.
Quote from: princess on 06/10/2014 08:22 amQuote from: Shanuson on 06/10/2014 07:55 amBut it should be frames 254 and 255 where this happened.Ah yes, good point! When I was working out which frames it occurred in, I forgot that raw_final_fixed.ts hasn't got the first 15 P-frames before the I-frame. The corruption is in 269/270 in rocket.ts and 254/255 in raw_final_fixed.ts.Thank you for the correction Shanuson!Yep, frames 254 and 255 are the same, contain both the 2 p-frames, and are quite a big mess
Quote from: SwissCheese on 06/10/2014 10:46 amI had to correct a little bit i frame 121 to align it to the p-frames. The problem is that the colors are wrong now and I'm not good at correcting them... Could someone recover nicer colors?The new mmb is in attachment, I haven't posted it to the wiki yet.If dcs need to be redone may I suggest including the changes I posted here:http://forum.nasaspaceflight.com/index.php?topic=34597.msg1209819#msg1209819It's a couple of xors to 0:0 macroblock (ajmartins triple flip) that I think restores it to the original, it might give a better foundation to the overall color of the frame and make it fit better into the sequence. but it looks like it breaks the dcs much more than Swisscheeses change so might not be worth the effort.
I've been playing with the error analysis a bit and I have a few mildly interesting new ways to visualise the bitflips.
I've been playing with the error analysis a bit and I have a few mildly interesting new ways to visualise the bitflips.The first is a file that is the XOR of raw_aligned.ts and rocket.ts - a 1 bit represents a bitflip, and a zero bit represents an uncorrupted bit. The game is to try to find if there's any predictability in the 1s. The file is attached to this message.The second is that I've made a series of images representing the bitflips. Each pixel represents a bit, and each image line represents 56 bytes of the data file. A black pixel means there's no error, a red pixel means a transmission error set a bit to 1 when it should be 0, and a green pixel means a transmission error cleared a bit to 0 when it should be 1. I've attached the first few images directly here so you can see what it looks like, then I've also included a ZIP file with all of the images in.Let's see if we can determine any patterns...
Most interesting. Do I understand correctly that the green areas are padding that should be xFFs and the diagonal lines of flips are TS packet headers (that you have data for)?
Most of these regions look to be in absolutely abysmal condition (except maybe the last picture). Surely this can't be representative of the whole file. Or is it?
Edit: Also, would it be possible to make separate pics for every frame?
Well, there's the one pattern that screams out, the periodic short runs of bitflips in otherwise okay data. Strong peak at ~184 bits between these corrections, with 5 resonant peaks due to occasional gaps in the pattern.
Quote from: saliva_sweet on 06/10/2014 05:23 pmEdit: Also, would it be possible to make separate pics for every frame?If you feel that would help I can do it, but the frame starts don't always align with 56-byte offsets. I could do pictures that are 188 bytes wide if you like?
Quote from: Quialiss on 06/10/2014 06:44 pmWell, there's the one pattern that screams out, the periodic short runs of bitflips in otherwise okay data. Strong peak at ~184 bits between these corrections, with 5 resonant peaks due to occasional gaps in the pattern. My guess is that you're using a byte offset and not a bit offset, and these are the TS packets at 188 bytes long. Most of the TS file is MPEG4 data packets - we can repair the headers of these packets but not the data, so you see a burst of errors representing the TS header bytes separated by 188 bytes. As we start applying MMBs to the data, you'll see pixels appearing in between these bursts.If that truly is BIT error lengths and not bytes, then we have something to work on...
Also -- could I ask for a graph or other visualization of correlations between bit flips? I'm thinking of a simple bar graph (or just a table) that says what's the probability of a flip at n+1, n+2, n+3, n+4, etc given the existence of a flip at n. That would be a nice quantitative sanity-check on the "triple flip" pattern (which should leap out) as well as maybe see if the triple flip patterns are indeed consistent across all frames, or maybe are useful only in frames with "low corruption".