Author Topic: Digitized NASA data tapes  (Read 33085 times)

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Digitized NASA data tapes
« on: 08/27/2018 07:06 pm »
Disclaimer: We are not funded and/or endorsed by NASA.
Greetings,

What have we done:

For the last few months I and my team have been working hard to acquire and digitize NASA computer tapes from the 60's to 70's. By now we have been able to pick the fruits of our labor as we have digitized about 6 reels out of 10. We expect to be able to digitize an other one after our drive got upgraded. The tapes so far have been acquired from Ebay.

Here is a photo of one of our tapes from Apollo 16, in its fitting tape drive:


This tape came from General Electric, and is one of our four Switch Action Table tapes. The other Switch Action table tapes came from Skylab. We also got Pioneer 11 tapes, likely with image data. We are still working on decoding the binary data into modern images so you can open it in a modern computer. Any help with this would be great.

The raw binary data is saved in the SIMH .tap format, which is wildly used by museums and computer simulations to mount magtapes. If we know what kind of data it is, we will also convert it to ASCII. Like the Switch Action Table tapes and Pione-QK799H tapes which contained EBCDIC text that we converted to ASCII in a separated file.

What are we working on:
We would love to identify what the Switch Action table tapes were used for at NASA. The tapes were clearly meant for something Saturn-V related as the text file mentions many parts of it. If you know anybody with more info, please ask for their help. They are from early to mid 70's. See tape photos for more info on the labels.

We don't know the image format used in the Pioneer 11 tapes (Pione-) and would love to process the raw data into a modern image file so everybody can see them. The Pione tapes came from NASA and the state uni of Arizona. They are from the mid to late 70's and mention the people "TOMASKO" shows up in the tape header, who turns out to be
Martin G. Tomasko of ASU. And Zellner.

In the tape you'll see LWSA "Long wavelength small aperture" and LWR -
"Long Wavelength Redundant"

A sample from QK7992H:
"530 Vesta, 12 MIN,SM.AP.,LO-DISP
ZELLNER/MAY 21 1978 3
CCAM 2 IMAGE 1530 PROC VERSN 211
PARAM SET NO 8 4 C
NO LAMP-EXP TIME=719 SECS"

Where can you download the tapes?

You can find the data with photos and tape details here: https://archive.org/details/SpaceData

How can you help:
Help us identify what our tapes were used for, help us decode them into a modern format. If you know the location of tapes or any other medium for that matter, contact me to talk about data and info recovery. I can digitize a vast amount of mediums so do not be afraid to ask me!

Contact: [email protected]
Support me on: Patreon

Offline david-moon

  • Member
  • Member
  • Posts: 15
  • Liked: 5
  • Likes Given: 0
Re: Digitized NASA data tapes
« Reply #1 on: 08/27/2018 07:15 pm »
You ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.

Offline Phil Stooke

  • Full Member
  • ****
  • Posts: 1354
  • Canada
  • Liked: 1424
  • Likes Given: 1
Re: Digitized NASA data tapes
« Reply #2 on: 08/27/2018 07:25 pm »
Imaging specialist Ted Stryk has done a lot with older image data and has been hoping to find Pioneer data for a long time.  I have contacted him about this, so he may be in touch.


Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #3 on: 08/28/2018 01:26 am »
Cool, digital archaeology. I had a little fun looking at your Pione-QK7992H .tap file. While I was intrigued in the past by work on the Pioneer Anomaly (now resolved), it struck me that the interpretation of your tapes could be approached similarly.

The raw binary data is saved in the SIMH .tap format, which is wildly used by museums and computer simulations to mount magtapes. If we know what kind of data it is, we will also convert it to ASCII. Like the Switch Action Table tapes and Pione-QK799H tapes which contained EBCDIC text that we converted to ASCII in a separated file.

I wrote a tiny program to parse the data based on qk7992h.log for SIMH Magtape file format to try to gain more insights in its contents.

We don't know the image format used in the Pioneer 11 tapes (Pione-) and would love to process the raw data into a modern image file so everybody can see them. The Pione tapes came from NASA and the state uni of Arizona. They are from the mid to late 70's and mention the people "TOMASKO" shows up in the tape header, who turns out to be
Martin G. Tomasko of ASU. And Zellner.

While I do not know the exact file format of the data in there, we can make some educated guesses and ask for more help from more informed experts on this forum. Dr. Tomasko was indeed mentioned with the Imaging Photopolarimeter (IPP) of Pioneer spacecraft and Section 7 and Appendix 1 at SP-349/396 PIONEER ODYSSEY describe many of its inner workings.

Quote
In the imaging mode, the data are converted to 64 levels of intensities (6 bits) and stored in a 6144-bit buffer onboard the spacecraft.
The instrument overwrites this buffer as it starts each "vertical" scan with each rotation of the spacecraft. The memory read-in time is
approximately one-half second, and the spacecraft rotation period is approximately 12.5 seconds, which means that there are approximately
12 seconds available for reading out the data from the memory. To read out these 6144 bits in the 12 seconds that are available requires
a data rate of 512 bits per second. The IPP instrument receives 50 percent service rate on the spacecraft's telemetry downlink.
Thus a 1024 bits per second telemetry downlink to Earth is the minimum data rate at which all the IPP data taken can be returned to Earth.

Also, some example data from the IPP (with interpretation) can be found at
Pioneer 10/11 Imaging Photopolarimeter Data

Now, looking at the hexdump of qk7992h.tap, I see e.g.

0101340    00  00  00  00  00  00  00  00  01  00  00  00  00  00  00  00
0101360    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
*
0101560    00  01  01  01  02  02  02  02  03  02  02  03  02  03  04  03
0101600    02  02  04  02  04  04  04  05  06  08  08  08  08  09  08  0b
0101620    09  0a  0b  09  0a  0a  0e  0d  0e  10  0c  0f  0f  0e  13  12
0101640    13  11  14  12  14  14  15  15  15  15  16  16  14  14  15  17
0101660    16  18  1b  18  18  18  18  18  19  16  18  18  18  1b  16  18
0101700    1a  15  19  19  18  1a  16  17  1a  17  16  18  16  18  1a  1a
0101720    17  14  15  14  18  15  17  15  10  15  13  13  11  11  10  0f
0101740    10  10  0d  0f  0e  0e  0e  0c  0b  0d  0a  09  0a  0a  09  0a
0101760    07  07  07  08  07  06  06  05  05  04  04  03  03  02  02  01
0102000    01  01  00  00  00  00  00  00  00  00  00  00  00  00  00  00
0102020    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

While the interpretation of this enthusiast could be completely wrong :) this could loosely be interpreted as scanlines of gray pixels. It seems that with a bit more study of the old formats, a conversion to PNG or similar looks feasible.

Help on the exact interpretation of the IPP data from the original engineers would be immensely useful though, and exciting!

HTH

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #4 on: 08/28/2018 09:29 am »
Cool, digital archaeology. I had a little fun looking at your Pione-QK7992H .tap file. While I was intrigued in the past by work on the Pioneer Anomaly (now resolved), it struck me that the interpretation of your tapes could be approached similarly.

The raw binary data is saved in the SIMH .tap format, which is wildly used by museums and computer simulations to mount magtapes. If we know what kind of data it is, we will also convert it to ASCII. Like the Switch Action Table tapes and Pione-QK799H tapes which contained EBCDIC text that we converted to ASCII in a separated file.

I wrote a tiny program to parse the data based on qk7992h.log for SIMH Magtape file format to try to gain more insights in its contents.

We don't know the image format used in the Pioneer 11 tapes (Pione-) and would love to process the raw data into a modern image file so everybody can see them. The Pione tapes came from NASA and the state uni of Arizona. They are from the mid to late 70's and mention the people "TOMASKO" shows up in the tape header, who turns out to be
Martin G. Tomasko of ASU. And Zellner.

While I do not know the exact file format of the data in there, we can make some educated guesses and ask for more help from more informed experts on this forum. Dr. Tomasko was indeed mentioned with the Imaging Photopolarimeter (IPP) of Pioneer spacecraft and Section 7 and Appendix 1 at SP-349/396 PIONEER ODYSSEY describe many of its inner workings.

Quote
In the imaging mode, the data are converted to 64 levels of intensities (6 bits) and stored in a 6144-bit buffer onboard the spacecraft.
The instrument overwrites this buffer as it starts each "vertical" scan with each rotation of the spacecraft. The memory read-in time is
approximately one-half second, and the spacecraft rotation period is approximately 12.5 seconds, which means that there are approximately
12 seconds available for reading out the data from the memory. To read out these 6144 bits in the 12 seconds that are available requires
a data rate of 512 bits per second. The IPP instrument receives 50 percent service rate on the spacecraft's telemetry downlink.
Thus a 1024 bits per second telemetry downlink to Earth is the minimum data rate at which all the IPP data taken can be returned to Earth.

Also, some example data from the IPP (with interpretation) can be found at
Pioneer 10/11 Imaging Photopolarimeter Data

Now, looking at the hexdump of qk7992h.tap, I see e.g.

0101340    00  00  00  00  00  00  00  00  01  00  00  00  00  00  00  00
0101360    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00
*
0101560    00  01  01  01  02  02  02  02  03  02  02  03  02  03  04  03
0101600    02  02  04  02  04  04  04  05  06  08  08  08  08  09  08  0b
0101620    09  0a  0b  09  0a  0a  0e  0d  0e  10  0c  0f  0f  0e  13  12
0101640    13  11  14  12  14  14  15  15  15  15  16  16  14  14  15  17
0101660    16  18  1b  18  18  18  18  18  19  16  18  18  18  1b  16  18
0101700    1a  15  19  19  18  1a  16  17  1a  17  16  18  16  18  1a  1a
0101720    17  14  15  14  18  15  17  15  10  15  13  13  11  11  10  0f
0101740    10  10  0d  0f  0e  0e  0e  0c  0b  0d  0a  09  0a  0a  09  0a
0101760    07  07  07  08  07  06  06  05  05  04  04  03  03  02  02  01
0102000    01  01  00  00  00  00  00  00  00  00  00  00  00  00  00  00
0102020    00  00  00  00  00  00  00  00  00  00  00  00  00  00  00  00

While the interpretation of this enthusiast could be completely wrong :) this could loosely be interpreted as scanlines of gray pixels. It seems that with a bit more study of the old formats, a conversion to PNG or similar looks feasible.

Help on the exact interpretation of the IPP data from the original engineers would be immensely useful though, and exciting!

HTH

Hello HTH,

Thanks for your detective work, your sample looks pretty good. Keep in mind  that QK7992H starts with ASCII metadata and has  it before each New file.

I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great. May I include  the program in the archive? I will credit you.

Thanks again  and kind regards,
Niels
Support me on: Patreon

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #5 on: 08/28/2018 04:21 pm »
I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great.

Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #6 on: 08/28/2018 09:17 pm »
I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great.

Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!

Followup on yesterday's post and reaction. I managed to write a parser that dumps all tape blocks in ASCII form. This can be processed easily with awk/perl/python into something else. For the original
Pione-QK7992H .tap file, here are the result files, and a first image interpretation :)

First, I reformatted the .tap into qk7992h.raw.hex.plus.edcdic.asc. That still contains EBCDIC .

The 5344 blocks of data look like
   blockNumber 0 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc
   blockNumber 1 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc

Note that you guys processed 5358 records. I believe the difference is my parser error correcting code after block 131, which took most of the time to do build. I am sure there is an official way to find the next valid block but I followed a more ad-hoc way. It seems a minor difference in the result file qk7992h.raw.hex.plus.edcdic.asc. Also note that I chose to make an ASCII file instead of of .bin which would require yet-another-parser-program.

Second, we have  qk7992h.all.ascii.ascqk7992h.all.ascii.asc where the first line is

   blockNumber 0 length 360 error  0 ::                         0001000107680768   1 2 2020 1530            1  C      *      *      *   *   *   720*      *   *  * * * * * *     *  2  CLWR1530   VESTA,12MIN,SM.AP.,LO-DISP  ZELLNER/MAY 21,1978           3  CCAM 2 IMAGE 1530        PROC VERSN 211 PARAM SET NO 8               4  C       NO LAMP-EXP TIME=719 SECS                                    5  C

Image aka pixel data is easily identifiable here by searching for "length 768" in the block identifier, e.g.,
   blockNumber 21 length 768 error  0 :: 00 00 00 00 00 00 00 00 00 00 ...
which has pixel values in the range [0..64]. The block identifier is, e.g.,
   blockNumber <tape blocknumber> length <in bytes> error  <was there a tape error or not> :: <delimiter>
You'll note that a sequence of 768 blocks of length 768 makes an image.

Third, throwing out all image data to get just the meta data and descriptions gives qk7992h.readable.txt which has various image descriptions, names etc. Interpretation by an expert would be useful :)

Finally, I had a first go at extracting the first image of 768 lines 768 pixel values. I believe these are Red/Blue pairs. I made 2 quick PNG images which show a disc. Nice to see such an image after all the years since 1978. First PNG image and second PNG image. I am sure one of you can make better images out of the data as I do not know how to simulate the green channel. Hence, I aimed for a simple, gray scale image.

I have not yet uploaded the C++ parser. A bit hackish but it did the job. Let me clean it up a bit. Based on the data files above, if someone has requests on what else the parser should do better then just let me know.

What do you think?

Offline Comga

  • Senior Member
  • *****
  • Posts: 6466
  • Liked: 4572
  • Likes Given: 5136
Re: Digitized NASA data tapes
« Reply #7 on: 08/28/2018 10:20 pm »
You ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.
Right!
Look up the Lunar Orbiter Image Recovery Project.
A quick plug into Google will pull up many links to this now completed project to digitize and process the data from old Lunar Orbiter data tapes, just like what you are doing.
Best of luck!
(Cowing is at NASAWatch.com.)
What kind of wastrels would dump a perfectly good booster in the ocean after just one use?

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #8 on: 08/29/2018 11:24 am »
I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great.

Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!

Followup on yesterday's post and reaction. I managed to write a parser that dumps all tape blocks in ASCII form. This can be processed easily with awk/perl/python into something else. For the original
Pione-QK7992H .tap file, here are the result files, and a first image interpretation :)

First, I reformatted the .tap into qk7992h.raw.hex.plus.edcdic.asc. That still contains EBCDIC .

The 5344 blocks of data look like
   blockNumber 0 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc
   blockNumber 1 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc

Note that you guys processed 5358 records. I believe the difference is my parser error correcting code after block 131, which took most of the time to do build. I am sure there is an official way to find the next valid block but I followed a more ad-hoc way. It seems a minor difference in the result file qk7992h.raw.hex.plus.edcdic.asc. Also note that I chose to make an ASCII file instead of of .bin which would require yet-another-parser-program.

Second, we have  qk7992h.all.ascii.ascqk7992h.all.ascii.asc where the first line is

   blockNumber 0 length 360 error  0 ::                         0001000107680768   1 2 2020 1530            1  C      *      *      *   *   *   720*      *   *  * * * * * *     *  2  CLWR1530   VESTA,12MIN,SM.AP.,LO-DISP  ZELLNER/MAY 21,1978           3  CCAM 2 IMAGE 1530        PROC VERSN 211 PARAM SET NO 8               4  C       NO LAMP-EXP TIME=719 SECS                                    5  C

Image aka pixel data is easily identifiable here by searching for "length 768" in the block identifier, e.g.,
   blockNumber 21 length 768 error  0 :: 00 00 00 00 00 00 00 00 00 00 ...
which has pixel values in the range [0..64]. The block identifier is, e.g.,
   blockNumber <tape blocknumber> length <in bytes> error  <was there a tape error or not> :: <delimiter>
You'll note that a sequence of 768 blocks of length 768 makes an image.

Third, throwing out all image data to get just the meta data and descriptions gives qk7992h.readable.txt which has various image descriptions, names etc. Interpretation by an expert would be useful :)

Finally, I had a first go at extracting the first image of 768 lines 768 pixel values. I believe these are Red/Blue pairs. I made 2 quick PNG images which show a disc. Nice to see such an image after all the years since 1978. First PNG image and second PNG image. I am sure one of you can make better images out of the data as I do not know how to simulate the green channel. Hence, I aimed for a simple, gray scale image.

I have not yet uploaded the C++ parser. A bit hackish but it did the job. Let me clean it up a bit. Based on the data files above, if someone has requests on what else the parser should do better then just let me know.

What do you think?
Hello Leovinus,

I think it looks awesome!

The metadata is much more readable now, they used some kind of window formatting around sets of text. Perhaps on their computers/printers this would look like a line? Something like PETSCII and IBM extended ASCII, which was used to make pseudo-graphics in MS-DOS programs.

Its great how we can look at each complete string now, when you swipe through it you can see the 00 (white?) pixels, and the pixels with a grayscale value, really interesting. It does look a bit like your processed images.

Would be be possible to have a function where it doesn't add any block information before the data? so we just have the tape data.

I am incredible happy that we got our first pictures. Are you positive they are correct? Grayscale or not, they look awesome. I am not image specialist, so I would not be able to help with using the colour data. Perhaps you can seperate the two colour channels in their own file, merge those two files together and add a third colour file as a mask? adjusting it until it looks right in combination with the two other colour files. I am sure there are programs that let you merge 3 different pictures and allows you to attach a colour channel to them, i think I have seen this in astronomy processing programs. Maybe Photoshop can do it too.

I notice the files are 255x255 while the metadata talks about 768x768, is this because of the missing colour channels?

Great work, a lot of people will be happy with this.

Kind regards,
Niels

Support me on: Patreon

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #9 on: 08/29/2018 03:37 pm »
You can clearly see in Leov's ASCII file how each image has metadata.

I find it interesting that some images are clear, like at blockNumber 2694, but sometimes they look nothing like each other.

blockNumber 3485 for example, looks nothing like 2694 and is looks like total garbage in the ASCII conversion. Any idea as to why this is?
« Last Edit: 08/29/2018 03:37 pm by apollo16uvc »
Support me on: Patreon

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #10 on: 08/29/2018 04:52 pm »
You can clearly see in Leov's ASCII file how each image has metadata.

I find it interesting that some images are clear, like at blockNumber 2694, but sometimes they look nothing like each other.

blockNumber 3485 for example, looks nothing like 2694 and is looks like total garbage in the ASCII conversion. Any idea as to why this is?

TL;DR not garbage, just a tool artifact on my side.

I presume you mean blockNumber 3486 length 818? Not 3485 which is the last line of a textual description?
In any case, the reason the data at block  3486 in qk7992h.all.ascii.ascqk7992h.all.ascii.asc looks like garbage is that I used a heuristic that said : if line size =768 or 2*768 then we assume those are pixels and print the values, otherwise convert to ASCII. The line of length 818 actually could be a 'picture' or measurement with 2 channels of 414 pixels. One channel is steady at level 1/2/3/4 and the other channel various a lot. Not sure whether that is a picture though. Look at blockNumber 3486, even and odd entries, in qk7992h.raw.hex.plus.edcdic.asc for comparison.

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #11 on: 08/29/2018 05:12 pm »
I have been trying to convert the data in .tap to a binary .bin file so people can look at it without having to know the SIMH format. If you could convert the binary data from the pione tapes SIMH files to a .bin file that would be great.
     
Sure, I'll have a go to extend the simple parser to convert this data to a more portable format. Will update when I have something new!

Followup on yesterday's post and reaction. I managed to write a parser that dumps all tape blocks in ASCII form. This can be processed easily with awk/perl/python into something else. For the original
Pione-QK7992H .tap file, here are the result files, and a first image interpretation :)

First, I reformatted the .tap into qk7992h.raw.hex.plus.edcdic.asc. That still contains EBCDIC .

The 5344 blocks of data look like
   blockNumber 0 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc
   blockNumber 1 length 360 error  0 :: 40 40 40 40 40 40 40 40 40 40 40 40 40 40 etc

Note that you guys processed 5358 records. I believe the difference is my parser error correcting code after block 131, which took most of the time to do build. I am sure there is an official way to find the next valid block but I followed a more ad-hoc way. It seems a minor difference in the result file qk7992h.raw.hex.plus.edcdic.asc. Also note that I chose to make an ASCII file instead of of .bin which would require yet-another-parser-program.

Second, we have  qk7992h.all.ascii.ascqk7992h.all.ascii.asc where the first line is

   blockNumber 0 length 360 error  0 ::                         0001000107680768   1 2 2020 1530            1  C      *      *      *   *   *   720*      *   *  * * * * * *     *  2  CLWR1530   VESTA,12MIN,SM.AP.,LO-DISP  ZELLNER/MAY 21,1978           3  CCAM 2 IMAGE 1530        PROC VERSN 211 PARAM SET NO 8               4  C       NO LAMP-EXP TIME=719 SECS                                    5  C

Image aka pixel data is easily identifiable here by searching for "length 768" in the block identifier, e.g.,
   blockNumber 21 length 768 error  0 :: 00 00 00 00 00 00 00 00 00 00 ...
which has pixel values in the range [0..64]. The block identifier is, e.g.,
   blockNumber <tape blocknumber> length <in bytes> error  <was there a tape error or not> :: <delimiter>
You'll note that a sequence of 768 blocks of length 768 makes an image.

Third, throwing out all image data to get just the meta data and descriptions gives qk7992h.readable.txt which has various image descriptions, names etc. Interpretation by an expert would be useful :)

Finally, I had a first go at extracting the first image of 768 lines 768 pixel values. I believe these are Red/Blue pairs. I made 2 quick PNG images which show a disc. Nice to see such an image after all the years since 1978. First PNG image and second PNG image. I am sure one of you can make better images out of the data as I do not know how to simulate the green channel. Hence, I aimed for a simple, gray scale image.

I have not yet uploaded the C++ parser. A bit hackish but it did the job. Let me clean it up a bit. Based on the data files above, if someone has requests on what else the parser should do better then just let me know.

What do you think?
The metadata is much more readable now, they used some kind of window formatting around sets of text. Perhaps on their computers/printers this would look like a line? Something like PETSCII and IBM extended ASCII, which was used to make pseudo-graphics in MS-DOS programs.

You are correct. There are programming languages from the time, like FORTRAN, with 72 columns, related to the old typewriters I believe with a fixed number of columns. When you take the data from qk7992h.readable.txt  and format in 72 columns, you get a much more readable file qk7992h.readable.72columns.txt which looks like this (with a proper FORTRAN 'C' in column 1 for comment :) when formatted
 
Quote
Would be be possible to have a function where it doesn't add any block information before the data? so we just have the tape data.

Just do this on the command line in your terminal and you are left with the hex data only.
awk '{ split($0,a,"::"); print a[2] }'  qk7992h.all.ascii.asc  | sed 's/^ //'  > newoutput

Quote
I am incredible happy that we got our first pictures. Are you positive they are correct? Grayscale or not, they look awesome.

The positive evidence is that the comments in the tape data explicitly say "RAW IMAGE", we have 768 lines of size 768, and a conversion to PNG gives a square image.
The negative evidence is that the comments say that the date of this tape is  16:35Z MAY 31,'78   
According to the Wiki timelines for Pioneer 11, it was between Jupiter and Saturn at the time.
Therefore, if this is an image then what are we looking at? The transmitter disk? Any idea's?

Quote
I notice the files are 255x255 while the metadata talks about 768x768, is this because of the missing colour channels?

You are correct that it should be 768 x 768 but I took the top-left and bottom-right 255x 255 to work around a limitation in one of my tools. Hence the PNG is 255x255, a slice of a visible circular disk.
« Last Edit: 08/29/2018 07:02 pm by leovinus »

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #12 on: 08/29/2018 05:17 pm »
You ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.
Right!
Look up the Lunar Orbiter Image Recovery Project.
A quick plug into Google will pull up many links to this now completed project to digitize and process the data from old Lunar Orbiter data tapes, just like what you are doing.
Best of luck!
(Cowing is at NASAWatch.com.)

Thank you! The link you gave has a typo though. It should be www.moonviews.com it seems.

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #13 on: 08/29/2018 05:40 pm »
Thanks for the update Leo, awesome work.

I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations.

I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.

« Last Edit: 08/29/2018 05:41 pm by apollo16uvc »
Support me on: Patreon

Offline Comga

  • Senior Member
  • *****
  • Posts: 6466
  • Liked: 4572
  • Likes Given: 5136
Re: Digitized NASA data tapes
« Reply #14 on: 08/29/2018 05:46 pm »
You ought to get in touch with Keith Cowing, who did quite a bit of this data archaeology already.
Right!
Look up the Lunar Orbiter Image Recovery Project.
A quick plug into Google will pull up many links to this now completed project to digitize and process the data from old Lunar Orbiter data tapes, just like what you are doing.
Best of luck!
(Cowing is at NASAWatch.com.)

Thank you! The link you gave has a typo though. It should be www.moonviews.com it seems.
You are welcome, correct, and due an apology for my sloppy phone typing.
Searching  “Lunar Orbiter Image Recovery Project” on Google pulls that up with many others including now-official websites from NASA that compare the recovered images to those from the recent Lunar Reconnaissance Orbiter LRO
What kind of wastrels would dump a perfectly good booster in the ocean after just one use?

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #15 on: 08/29/2018 06:58 pm »
Thanks for the update Leo, awesome work.

I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations.

I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.

How about this? Fixed my tool to allow the full PNG image size. This is again the first image, now in full 768x768, gray scale. Both based on the block numbers 21 to 788, and one contrast normalized. Would this make sense as an image of zodiacal light?
first.image.768x768.grayscale.contrast.eq.png
first.image.768x768.grayscale.png

second.image.768x768.grayscale.contrast.eq.png
second.image.768x768.grayscale.png

third.image.768x768.grayscale.contrast.eq.png
third.image.768x768.grayscale.png

EDIT: added two more 768x768 from the same tape. These are all the 768x768 from the tape. There are a few sections 768x1536 which might be color coded. Will have a look
« Last Edit: 08/29/2018 07:18 pm by leovinus »

Offline hoku

  • Full Member
  • ****
  • Posts: 766
  • Liked: 662
  • Likes Given: 329
Re: Digitized NASA data tapes
« Reply #16 on: 08/29/2018 07:47 pm »
...
While I do not know the exact file format of the data in there, we can make some educated guesses and ask for more help from more informed experts on this forum. Dr. Tomasko was indeed mentioned with the Imaging Photopolarimeter (IPP) of Pioneer spacecraft and Section 7 and Appendix 1 at SP-349/396 PIONEER ODYSSEY describe many of its inner workings.
...
Very interesting, and very nice to follow the fast progress in de-coding the data!

The date suggests that these might be calibration data. According to the IPP description at  NSSDC:

"(...) Long-term absolute calibration of the instrument was accomplished by means of a sunlight diffusor/attenuator element located in the spacecraft antenna structure. Primary radiometric calibration was obtained throughout the mission by periodically commanding the telescope to view this diffuse backlighted (sunlight) source.(...)"

Some details on the tape data structure used for IPP data storage can also be found at https://nssdc.gsfc.nasa.gov/nmc/datasetDisplay.do?id=PSSB-00015

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #17 on: 08/29/2018 08:11 pm »
Thanks for the update Leo, awesome work.

I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations.

I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.

How about this? Fixed my tool to allow the full PNG image size. This is again the first image, now in full 768x768, gray scale. Both based on the block numbers 21 to 788, and one contrast normalized. Would this make sense as an image of zodiacal light?
first.image.768x768.grayscale.contrast.eq.png
first.image.768x768.grayscale.png

second.image.768x768.grayscale.contrast.eq.png
second.image.768x768.grayscale.png

third.image.768x768.grayscale.contrast.eq.png
third.image.768x768.grayscale.png

EDIT: added two more 768x768 from the same tape. These are all the 768x768 from the tape. There are a few sections 768x1536 which might be color coded. Will have a look

Hi Leo,
I have been feeling ecstatic yesterday and today, but today really beats it. My long-term goal was to be able to digitize the tapes and present their contents in a modern format. You have done that! incredible work, I can't thank you enough.

I keep several forums updated on the status of these projects, may I add your files? I will credit you of course.

I have send the files to Ted Stryk to hear what he has to say about them.

Yes I suspect a wide 768x image has two colour channels separated into two square parts. Should not be hard to merge into a colour image? will just have to add a false third colour layer manually, perhaps by processing the average out of the two colour layers we can determine the third color intensity value? It would be best to provide both grayscale layers so people can mess with it themselves.

This project has worked because everybody always provided transparent and raw data for the next person to take up and work with. It is key that everybody is transparent about what they have done to a file so we all understand each other. Everything has been volunteering work.

Thanks, it has been a real high-speed roller-coaster. And I got several more tapes on the way.

« Last Edit: 08/29/2018 08:12 pm by apollo16uvc »
Support me on: Patreon

Offline leovinus

  • Full Member
  • ****
  • Posts: 1095
  • Porto, Portugal
  • Liked: 867
  • Likes Given: 1727
Re: Digitized NASA data tapes
« Reply #18 on: 08/29/2018 08:45 pm »
Thanks for the update Leo, awesome work.

I talked with Ted Stryk before, and he said QK7992H contains IPP data from zodiacal light observations.

I am sure we can just take 4 individual parts out of the tool and mix them together. Once we have the complete image I will send it to ted Stryk to see what he has to say about it. or maybe it will be easier to modify the tool to support higher resolution output.

How about this? Fixed my tool to allow the full PNG image size. This is again the first image, now in full 768x768, gray scale. Both based on the block numbers 21 to 788, and one contrast normalized. Would this make sense as an image of zodiacal light?
first.image.768x768.grayscale.contrast.eq.png
first.image.768x768.grayscale.png

second.image.768x768.grayscale.contrast.eq.png
second.image.768x768.grayscale.png

third.image.768x768.grayscale.contrast.eq.png
third.image.768x768.grayscale.png

EDIT: added two more 768x768 from the same tape. These are all the 768x768 from the tape. There are a few sections 768x1536 which might be color coded. Will have a look

Hi Leo,
I have been feeling ecstatic yesterday and today, but today really beats it. My long-term goal was to be able to digitize the tapes and present their contents in a modern format. You have done that! incredible work, I can't thank you enough.

You are welcome and I share the feeling! It is such a fun project to look at the old spacefaring tapes and find out what's on them. Right at the intersection of computer science, astronomy, and space engineering. Thank you for publishing the tapes! PS: Actually, first name is Hans. Sorry for the confusion.

Quote
I keep several forums updated on the status of these projects, may I add your files? I will credit you of course.

From my side, that is fine. I believe we all profit from improved insights in our spacefaring history and we can all be grateful to NASA, the engineers, scientists and politicians, who enabled these spaceprobes to enrich our knowledge. The work on this data is a small contribution to preserve our history and it happens to be fun as well. Thank you all.

I have added a link to the C++ parsing code for the .tap as well. It was made on the quick but got us started. parseTape.aug29.cpp and the checksum parseTape.aug29.md5

Quote
Yes I suspect a wide 768x image has two colour channels separated into two square parts. Should not be hard to merge into a colour image? will just have to add a false third colour layer manually, perhaps by processing the average out of the two colour layers we can determine the third color intensity value? It would be best to provide both grayscale layers so people can mess with it themselves.

Indeed, a quick try is first.image.768x1536.red.blue.png as well as first.image.768x1536.red.blue.contrast.eq.png. On first glance, they look very similar to the earlier images.

Quote
This project has worked because everybody always provided transparent and raw data for the next person to take up and work with. It is key that everybody is transparent about what they have done to a file so we all understand each other. Everything has been volunteering work.

Thanks, it has been a real high-speed roller-coaster. And I got several more tapes on the way.



Thanks for the .gif. It looks very interesting. Now we'd love to know what it represents, and what you can find on other tapes. (PS: I had a quick look at the other two Pioneer tapes you published but to me, it does not look like ASCII data nor images. Not sure whether they are too far degraded or what data it is. Hope we can find out.)
« Last Edit: 08/29/2018 08:47 pm by leovinus »

Offline apollo16uvc

  • Full Member
  • **
  • Posts: 262
  • Astrophotography on the moon rules
  • Next to LEM, at Descartes Highlands, on the Moon!!!
    • Dutchsteammachine
  • Liked: 291
  • Likes Given: 2
Re: Digitized NASA data tapes
« Reply #19 on: 09/01/2018 09:29 pm »
Hello then, Hans.

I download the C++ code, it is impressive. But I have to admit I don't know code beyond some basic...BASIC. I am sure it will continue to be useful to convert metadata found on tapes. How do you find the record length for each block?

JMAP03 Reads just garbage, if you read the log file it will say every read attempt has a parity error. We decided to include it anyways, maybe somebody will figure something out. Our tape drive should be upgraded soon so we will give several tapes an other go.

As for Pione-4138N, we don't find any recognizable character sets or data either. All we know is from the label, and that can be deceiving.  The label says Pioneer 11 EDR UA/IPP from OCT 1975

If there was a read/parity error it should be in the log file, I don't know if your own error detection scheme will detect more. As for data being corrupt or not, if a tape has too much reading errors we check out the tracks and several times have found them to be very faint, likely wiped. This fate has likely befallen JAMP03.

Any read/parity errors should be in the log file. if you suspect a piece of data to be corrupt that is the place to check first.

Yes the GIF is nice. I send 3 photos to Ted Stryk, maybe he can tell us more about them.
I did some reading on how you are suppose to make a colour image, interesting. Can you send me the RED and BLUE channel individually so I can mess around with it? You are suppose to add your own GREEN channel.

I'm curious on the technical details of how you convert the files. Have you seen this kind of image format before?

Ted Stryk suspected QK7992H contains zodiacal light observations. He said this before we had any pictures, he has not yet responded to them.

This is data-archaeology at its greatest.

Niels
« Last Edit: 09/01/2018 10:03 pm by apollo16uvc »
Support me on: Patreon

 

Advertisement NovaTech
Advertisement Northrop Grumman
Advertisement
Advertisement Margaritaville Beach Resort South Padre Island
Advertisement Brady Kenniston
Advertisement NextSpaceflight
Advertisement Nathan Barker Photography
0