No, I knew that, what I meant to ask was whether blocks from one macroblock can end up inside another in a way that would result in macroblock contents being offset by half of macroblock. It probably isn't. Which would mean that the odd object is not the tip of the leg.
So it seems like it should be practical to write a tool that can walk through the raw.ts and use the rules of the packet header formatting and sequence numbers to reconstruct a more coherent transport stream. Even just getting all the errors out of the null packets by replacing everything with FF might make it easier to discern good patterns.Edit: For example, in raw.ts there are bytes missing or added between 0x37ACC and 0x371EC - that's the first place where the columns shift in a 47-column hex display.
I found out why iframe 0 didn't extract - the transport stream header for the first packet of the iframe had its "transport error indicator" bit flipped, and so that packet containing the very first piece of the iframe was ignored, resulting in the entire iframe being ignored. If you take try1.ts and change offset 0x3E3D1 from 0x43 to 0x03, clearing the TEI bit, then iframe 0 should pop out. It may have some issues since the sequence numbers are rather jumbled right there - it looks like 0x3E48F should be 0x5C rather than 0x5F, and that might contain the adaptation field info for the iframe. The sequence doesn't seem to get back on track until you get to 0x3E83B.
Quote from: mvpel on 05/11/2014 02:49 pmI found out why iframe 0 didn't extract - the transport stream header for the first packet of the iframe had its "transport error indicator" bit flipped, and so that packet containing the very first piece of the iframe was ignored, resulting in the entire iframe being ignored. If you take try1.ts and change offset 0x3E3D1 from 0x43 to 0x03, clearing the TEI bit, then iframe 0 should pop out. It may have some issues since the sequence numbers are rather jumbled right there - it looks like 0x3E48F should be 0x5C rather than 0x5F, and that might contain the adaptation field info for the iframe. The sequence doesn't seem to get back on track until you get to 0x3E83B.No its more than that. The headers are wrong so it doesn't decode and by "wrong" i mean complete garbage beyond a certain point. I don't think the TEI bit is related.
Quote from: mlindner on 05/11/2014 03:10 pmQuote from: mvpel on 05/11/2014 02:49 pmI found out why iframe 0 didn't extract - the transport stream header for the first packet of the iframe had its "transport error indicator" bit flipped, and so that packet containing the very first piece of the iframe was ignored, resulting in the entire iframe being ignored. If you take try1.ts and change offset 0x3E3D1 from 0x43 to 0x03, clearing the TEI bit, then iframe 0 should pop out. It may have some issues since the sequence numbers are rather jumbled right there - it looks like 0x3E48F should be 0x5C rather than 0x5F, and that might contain the adaptation field info for the iframe. The sequence doesn't seem to get back on track until you get to 0x3E83B.No its more than that. The headers are wrong so it doesn't decode and by "wrong" i mean complete garbage beyond a certain point. I don't think the TEI bit is related.Not complete garbage, but very close - Based on the null-packet error estimation there is something like a 15% *average* error rate during iframe 0. This is substantially worse than the rest of the data, which hangs between 0.1% and 1.5%.
Quote from: theshadow27 on 05/11/2014 03:26 pmQuote from: mlindner on 05/11/2014 03:10 pmQuote from: mvpel on 05/11/2014 02:49 pmI found out why iframe 0 didn't extract - the transport stream header for the first packet of the iframe had its "transport error indicator" bit flipped, and so that packet containing the very first piece of the iframe was ignored, resulting in the entire iframe being ignored. If you take try1.ts and change offset 0x3E3D1 from 0x43 to 0x03, clearing the TEI bit, then iframe 0 should pop out. It may have some issues since the sequence numbers are rather jumbled right there - it looks like 0x3E48F should be 0x5C rather than 0x5F, and that might contain the adaptation field info for the iframe. The sequence doesn't seem to get back on track until you get to 0x3E83B.No its more than that. The headers are wrong so it doesn't decode and by "wrong" i mean complete garbage beyond a certain point. I don't think the TEI bit is related.Not complete garbage, but very close - Based on the null-packet error estimation there is something like a 15% *average* error rate during iframe 0. This is substantially worse than the rest of the data, which hangs between 0.1% and 1.5%.It was very oddly placed, at the EXACT location that the user data header was to start, it was replaced with a different "error" header with random noise in it. All the header data before that location was perfect without a single bit flip. I could not find anything that looked remotely like the standard data (thats the same for every iframe) in it. It almost looked like it was intentional and the encoder decided to not use this iframe.
Edit: BTW with some quick math, assuming raw.ts is ~30s, those big spikes of 5-8 packets (30-45ms) are almost certainly lightning EMI. Must have been some storm...
Oh... that's strange... I always need to seek by exactly 132 bytes to getts-packet headers to realign... can't be a coincidence!
Overall, there seems to be six sections of correctly-sized ts-packets:1st_ofs - offset_of_last_recognizable_ts_packet_header0000000 - 0037acc00381ec - 00d6e2000d7250 - 0215c580215cdc - 0356ae00357378 - 03dbe8c03dd16c - 04460f004461aa (eof, last packet is two bytes short)Between each section is a block of few hundred bytes.Its size is always N*188 + 132. I can "repair" the file by padding each section so that its size is a multiple of 188,and then repair ts-packet headers: it's easy te determine whether the packet isa null packet or payload packet, and then fix headers to be either4703e81x or 471fff1x.Will this be useful in any way?
The script to cut up raw.ts into sections with 188-byte ts-packets.The last two commands prove that concatenation of sections is the same as raw.ts (both have 042433dfc39c70a51203fae581a5461f md5sum).#!/bin/shmk_sect(){ local start=$1 local end=$2 dd bs=1 skip=$(($start)) count=$(($end - $start)) <raw.ts >$3.ts cat section?.ts >z.$$ hexdump -s $(($start)) -v -e '"%07.7_ax " 188/1 "%02x" "\n"' <z.$$ >$3.hex rm z.$$}# 1st_ofs - offset_of_last_recognizable_ts_packet_header# 0000000 - 0037acc# 00381ec - 00d6e20# 00d7250 - 0215c58# 0215cdc - 0356ae0# 0357378 - 03dbe8c# 03dd16c - 04460f0# 04461aa (eof, last packet is two bytes short)## It is not obvious where exactly realignment happens.# What can be seen is where the last packet with "old" alignment is,# and where is the first recognizable packet with "new" alignment is.#rm section?.tsmk_sect 0x0000000 0x00381ec section1mk_sect 0x00381ec 0x00d7250 section2mk_sect 0x00d7250 0x0215cdc section3mk_sect 0x0215cdc 0x0357378 section4mk_sect 0x0357378 0x03dd16c section5mk_sect 0x03dd16c 0xfffffff section6cat section?.ts | md5sumcat raw.ts | md5sum
I can "repair" the file by padding each section so that its size is a multiple of 188, and then repair ts-packet headers: it's easy te determine whether the packet is a null packet or payload packet, and then fix headers to be either 4703e81x or 471fff1x.Will this be useful in any way?
An improved version of the annotated iframe8 with many more common dirt spots I see in several other iframes.They seem to be very useful for further aligning MB's in other iframes