Author Topic: Declassified KH-9 HEXAGON imagery is difficult to access  (Read 11257 times)

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
http://blogs.fas.org/secrecy/2014/03/imagery-access/


Newly Declassified Intelligence Satellite Imagery is Hard to Access


The declassification of historical intelligence satellite imagery has been a boon to scientists, environmentalists and other researchers since it began with President Clinton’s executive order 12951. So, for example, “The declassification of imagery from CORONA and subsequent intelligence satellite programs has inspired a revolution in landscape archaeology in the Near East,” wrote archaeologist Jason Ur.

But last year’s declassification of imagery from the KH-9 HEXAGON intelligence satellite will be slower to generate any such revolutionary impact because the newly declassified images are so hard to access and to use.

The KH-9 imagery was successfully transferred from the National Geospatial-Intelligence Agency to the National Archives. But in order to protect the perishable film it must be maintained in cold storage, and so it was all sent to a National Archives facility in Lenexa, Missouri Kansas. Researchers must make their best guess as to what images they are seeking, and then order the originals to be transferred from cold storage. It’s a slow and cumbersome process.

The larger policy issue is that the archival burden on the National Archives and Records Administration is growing faster than the available resources. The task of curating the nation’s documentary heritage appears to be escalating out of control. Meanwhile, the Archives is literally running out of space. Last month, Archivist of the United States David S. Ferriero announced the closure of three NARA facilities “as part of ongoing budget adjustments.”
*    *    *

Recently, one concerned researcher shared his frustrations about the current procedures for obtaining declassified satellite imagery. Secrecy News forwarded his comments to the National Archives and Records Administration, and a NARA official provided an annotated response, reproduced below.

SNIPPED THE REST, GO ON TO READ THE QUESTIONS AND RESPONSES.


Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #1 on: 04/13/2014 04:39 PM »
Note: I'm not the researcher who wrote those questions. I went out to NARA about a month ago but they couldn't help me at all. I got the sense that they don't understand the collection, and they are also rather frustrated about it (for various reasons). It does seem rather crazy that all this material was turned over to NARA and yet it will be extremely difficult for people to access it. NARA people tend to be very helpful, and it probably annoys them a lot that they cannot do anything more to make this stuff public.

A few years ago there was an announcement put out by a government agency (NGA or NARA?) asking for contract proposals to scan all the film and make it digitally available. I cannot find a link to that announcement--I think it was also posted to NSF so somebody here may be able to locate it after my search efforts failed--but it was clear that the original intention was to scan the imagery digitally. Obviously they never did that.

The difficulty of accessing the imagery is surprising. I would have assumed that the intelligence community had a way to easily reach back into old imagery to answer questions like “How long has that building been at that location?” But perhaps they never did that for their film records.

Offline AnalogMan

  • Member
  • Senior Member
  • *****
  • Posts: 3007
  • Cambridge, UK
  • Liked: 774
  • Likes Given: 23
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #2 on: 04/13/2014 05:18 PM »
A few years ago there was an announcement put out by a government agency (NGA or NARA?) asking for contract proposals to scan all the film and make it digitally available. I cannot find a link to that announcement--I think it was also posted to NSF so somebody here may be able to locate it after my search efforts failed--but it was clear that the original intention was to scan the imagery digitally. Obviously they never did that.

Might this RFI be the one?

Historical Imagery Digitization Data Project
Solicitation Number: HM1582-11-ACT-SN0001
Agency: Other Defense Agencies
Office: National Geospatial-Intelligence Agency
Posted Date: February 14, 2011

https://www.fbo.gov/index?id=d84b37e005a0c76d147b0a8ee97f07fd

Edit: Maybe this url will work instead
https://www.fbo.gov/index?s=opportunity&mode=form&id=d84b37e005a0c76d147b0a8ee97f07fd&tab=core&_cview=0
« Last Edit: 04/13/2014 06:37 PM by AnalogMan »

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #3 on: 04/13/2014 06:13 PM »
A few years ago there was an announcement put out by a government agency (NGA or NARA?) asking for contract proposals to scan all the film and make it digitally available. I cannot find a link to that announcement--I think it was also posted to NSF so somebody here may be able to locate it after my search efforts failed--but it was clear that the original intention was to scan the imagery digitally. Obviously they never did that.

Might this RFI be the one?

Historical Imagery Digitization Data Project
Solicitation Number: HM1582-11-ACT-SN0001
Agency: Other Defense Agencies
Office: National Geospatial-Intelligence Agency
Posted Date: February 14, 2011

https://www.fbo.gov/index?id=d84b37e005a0c76d147b0a8ee97f07fd

The name is right. The url doesn't work. But I searched using the title and got it:


Solicitation Number:
HM1582-11-ACT-SN0001
Notice Type:
Special Notice
Synopsis:
Added: Feb 14, 2011 10:41 am

Historical Imagery Digitization Data Project (HID2P)
Request for Information

Introduction
The National Geospatial-Intelligence Agency (NGA) is seeking to identify potential sources and conduct market research to help determine the most effective and efficient options to digitize large volumes of high resolution photographic imagery to specific criteria.

This RFI is issued for the purpose of determining interest, feasibility, and capability only and does not constitute a Request for Proposal or indication the Government will contract for any of the items and/or services contained in this notice. In accordance with FAR 52.215-3, the Government will not pay for any information/items submitted in response to this synopsis. Data will not be returned. This RFI does not commit the Government to award a contract. This RFI is for market research only and there is currently no solicitation for this effort. Any solicitation resulting from this announcement will be published separately. Any final plan will be done consistent with E.O 12951 and E.O 13526 dealing with historical intelligence imagery acquired by a space-based national intelligence reconnaissance system.

Background
The NGA is anticipating the potential declassification of significant amounts of film based imagery. A large volume of film based imagery may become declassified in 2011 and would be the basis of an initial effort to digitize historical film based imagery. The NGA is interested in exploring options to have the declassified film digitized and potentially stored with various distribution possibilities.

What we wish to accomplish: 1) seeking innovative solutions within government, industry and academia that are affordable, and practical, 2) provide access to digitized imagery at a level of quality that NGA deems useful and meaningful, while 3) leading to future enrichment of GEOINT observations in an unclassified environment.

Solution approach(es): There may be variations to this approach, but essentially industry is given access to original negative or 2nd generation dupe positive or 3rd generation dupe negatives, with supporting coverage and ephemeris data, for a period of 1 to 2 years to:
• Scan imagery at a minimum 10 micron or better spatial resolution (4-5 micron), with a dynamic range of at least 10 bits per pixel, either on government premises or at their place of work. Imagery is stored on 500 foot spools, with many frames up to several feet in length. The nominal thickness of the film is 2.5 microns for original negative, and in some cases may be as thin as 1 micron thickness. Dupe negative (2nd generation) or dupe positive (3rd generation imagery) is also available and has a nominal thickness of 2.5 mils, but may have image processing edge sharpness enhancements, other film handling artifacts, and for certain scenes may have lower resolution compared to the original negative imagery.
• Due to the classification of the current film imagery, and until such time as the imagery samples are declassified, details of the film format are not included in this RFI. The responders are encouraged to identify film handling limitations in their RFI responses.
• Provide or make available a digital copy of the scanned imagery to NGA, and a description of the type of storage media that may be provided.
• Retain rights to distribute declassified imagery and recoup investment, for a specified period of time (negotiable), seek greatest flexibility to allow industry to form partnerships to digitize and/or store initial digital data (up to 10 Petabytes) for online, nearline or offline access.
• Government retains rights to limit distribution of this imagery over certain areas of the world
• Security requirements: The work can be performed in an unclassified but access controlled environment. We are interested in understanding if the facilities can operate at the TOP SECRET/SCI level for potential follow on activity.
• Risk and cost is mostly on the contractor. The government performs in an advisory capacity, and oversight for security and support on logistical matters of common concern for the safe handling and use of the imagery.
• The RFI response should not be too specific to limit innovation or efficiencies that may be devised by the responder but should contain the necessary details to understand the underlying business model. We encourage a consortium of industry and/or academic partners to develop the most capable and flexible hardware and image processing technology.
o Maturity and availability of technology: The response should describe to the extent practical by the responder, which is not proprietary in nature or fact, a short description of engineering for example with hardware, optics, transport, electronics/computer peripherals, and/or detector components that can achieve a level of quality. If there are limitations on handling film widths up to 9 inches, please identify the limiting factors to the output of digitized imagery at a quality and quantity that we may seek,or that which can be achieved with engineering or fabrication of components, software and so forth. The government is not interested in a detailed quantitative description of engineering or other efforts, but only seeks to understand the nature of technical and schedule risks for a certain level of performance that can be addressed by respondants's innovative approaches and the degree of risk and cost that they may be able to handle or share in due consideration of benefits that they may seek to attain in their business case.

Information Requested
Interested companies are requested to submit the information outlined below.
• Experience in digitizing film with high resolution on a large scale of holdings. This should include previous examples with details such as the equipment used, volume (scale) of film digitized, resolution of digitization, any unique film attributes (size, length, thickness), timeline from start to finish to complete the project, and any post data processing requirements completed on the project.
• Business Case. Respondents should include sufficient details to understand the underlying business case to include assumptions, and any cost sharing arrangement.
• Overall notional schedule. The notional schedule should include assumptions used to develop the schedule. The schedule shall use a hypothetical start date of 1 October 2011 and continue through the end of life of the program. The duration should be based on digitizing up to approximately 4 million linear feet of film up to approximately 7 inches in width.
o Potential pilot effort. The Government is considering a pilot effort depending on the number and quality of the submissions. This pilot effort would include actual digitizing of up to 6 separate scenes to demonstrate respondent's capabilities. Two of the scenes may be color or false color film. Respondents should include details on their ability to support a pilot effort with a completion by December 2011 in their RFI response.
• Film handling requirements. The film will be made available once the final steps are completed to declassify the imagery. All respondents should identify how the film needs to be made available to them as well as how they will store, safeguard and handle the film (in compliance NARA requirements) and associated electronic data, during the film digitizing process in a facility that can be cleared to the Top Secret/SCI level. The film will need to be returned after the digitizing is completed.

« Last Edit: 04/13/2014 06:15 PM by Blackstar »

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #4 on: 04/13/2014 06:17 PM »
The February 2011 announcement was commented about here:

http://blogs.fas.org/secrecy/2011/02/imagery_declass/

Maybe never happened because of declining budgets.

Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #5 on: 04/13/2014 06:27 PM »
Interesting piece. Wonder what will happen in future decades when images will no longer be film but digital which comes with its own set of issues.
« Last Edit: 04/13/2014 06:27 PM by Star One »

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #6 on: 04/13/2014 07:25 PM »
Interesting piece. Wonder what will happen in future decades when images will no longer be film but digital which comes with its own set of issues.

Early KH-11 imagery was transmitted down and then burned to film. I believe that they called them film "chips" and they were something like six inches wide by twelve long. Certainly everything was recorded on magnetic tape, but I don't know if they made a complete duplicate set of the chips. For a long time the photo interpreters used the chips more than they used graphic workstations. You can imagine the difficulty with file size and storage and image manipulation even into the late 1990s.

So do the early film chips still exist? Or did they trash all that stuff and now they have tons of magnetic tape that they can no longer read?

Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #7 on: 04/13/2014 07:39 PM »

Interesting piece. Wonder what will happen in future decades when images will no longer be film but digital which comes with its own set of issues.

Early KH-11 imagery was transmitted down and then burned to film. I believe that they called them film "chips" and they were something like six inches wide by twelve long. Certainly everything was recorded on magnetic tape, but I don't know if they made a complete duplicate set of the chips. For a long time the photo interpreters used the chips more than they used graphic workstations. You can imagine the difficulty with file size and storage and image manipulation even into the late 1990s.

So do the early film chips still exist? Or did they trash all that stuff and now they have tons of magnetic tape that they can no longer read?

That's interesting to hear.

It often seems like there is nothing that goes obsolete so quickly as digital storage. What you've said reminds me of what happened with the Doomsday Book project in the eighties over here in the UK which was stored on laserdisc, and the fears later on that this information may be lost as there was less & less computers that could read the discs as time went on. Even now there are still issues with retrieving the data.

http://en.wikipedia.org/wiki/BBC_Domesday_Project


Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #8 on: 04/13/2014 08:29 PM »
It often seems like there is nothing that goes obsolete so quickly as digital storage.

And in this case we're not even talking about recent stuff. After all, the first KH-11 was operational in late 1976, so we are talking about digital media that is already 38 years old. How many generations has it gone through?

Offline Melt Run

  • Full Member
  • *
  • Posts: 143
  • Liked: 1
  • Likes Given: 2
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #9 on: 04/16/2014 01:26 PM »
Quote from: Star One on 13 Apr, 2014 19:39
It often seems like there is nothing that goes obsolete so quickly as digital storage.

And in this case we're not even talking about recent stuff. After all, the first KH-11 was operational in late 1976, so we are talking about digital media that is already 38 years old. How many generations has it gone through?"

At least five that I can think of. Much of the 14 inch tape is unreadable even if you have the mainframe which is unlikely.


Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #10 on: 04/16/2014 02:10 PM »

Quote from: Star One on 13 Apr, 2014 19:39
It often seems like there is nothing that goes obsolete so quickly as digital storage.

And in this case we're not even talking about recent stuff. After all, the first KH-11 was operational in late 1976, so we are talking about digital media that is already 38 years old. How many generations has it gone through?"

At least five that I can think of. Much of the 14 inch tape is unreadable even if you have the mainframe which is unlikely.

Anyone remember floppy discs of various sizes? These must have been used for storage in this case as they were ubiquitous at one point. Unfortunately they are also prone to becoming hard or impossible to retrieve stuff from after sufficient time.

Offline kevin-rf

  • Elite Veteran
  • Senior Member
  • *****
  • Posts: 8652
  • Overlooking the path Mary's little Lamb took..
  • Liked: 1123
  • Likes Given: 243
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #11 on: 04/16/2014 02:47 PM »
You mean eight inch floppies?

I once had a co-worker who had a 70's vintage hard drive platter about the diameter of a small coffee table.

Bonus points:
« Last Edit: 04/16/2014 02:48 PM by kevin-rf »
If you're happy and you know it,
It's your med's!

Offline Hog

  • Full Member
  • ****
  • Posts: 1253
  • Woodstock
  • Liked: 329
  • Likes Given: 995
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #12 on: 04/16/2014 03:56 PM »
And the 5 1/4" floppies.
Paul

Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #13 on: 04/16/2014 04:02 PM »
Where the small ones 3.5", as I remember using them but cannot remember the specific size.

Offline Hog

  • Full Member
  • ****
  • Posts: 1253
  • Woodstock
  • Liked: 329
  • Likes Given: 995
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #14 on: 04/16/2014 04:28 PM »
Yup, the smaller hard cased floppies were 3.5".  I have a stack of them right beside me, I use them occasionally.
Paul

Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #15 on: 04/16/2014 06:42 PM »

Yup, the smaller hard cased floppies were 3.5".  I have a stack of them right beside me, I use them occasionally.

Do they still work OK?

I tried one a while ago and it just didn't want to work.

On a serious point I really do fear that future archivists may encounter 'black holes' in the data from such satellites due to issues with the storage.

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #16 on: 04/17/2014 12:49 PM »
On a serious point I really do fear that future archivists may encounter 'black holes' in the data from such satellites due to issues with the storage.

They are of course aware of this stuff. That doesn't necessarily mean that they do anything about it.

I believe (and I'm basing this only on circumstantial info I've gathered, but I could ask people) that NASA's Planetary Data System includes efforts to make sure that old data is stored and kept so that it can be accessed. I think that their bigger concern is that new spacecraft are now starting to return so much data that it is going to swamp the old stuff. But they are aware of legacy issues and can reach back several decades.

Offline Stan Black

  • Senior Member
  • *****
  • Posts: 2903
  • Liked: 136
  • Likes Given: 104
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #17 on: 04/17/2014 01:11 PM »
I once had a co-worker who had a 70's vintage hard drive platter about the diameter of a small coffee table.

I knew a man who actually did convert one into a garden table…

Offline Star One

  • Senior Member
  • *****
  • Posts: 9631
  • UK
  • Liked: 1817
  • Likes Given: 183
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #18 on: 04/17/2014 01:26 PM »

On a serious point I really do fear that future archivists may encounter 'black holes' in the data from such satellites due to issues with the storage.

They are of course aware of this stuff. That doesn't necessarily mean that they do anything about it.

I believe (and I'm basing this only on circumstantial info I've gathered, but I could ask people) that NASA's Planetary Data System includes efforts to make sure that old data is stored and kept so that it can be accessed. I think that their bigger concern is that new spacecraft are now starting to return so much data that it is going to swamp the old stuff. But they are aware of legacy issues and can reach back several decades.

But would there be the time, money & inclination to do this with very old data from the KH-11. Unless it has some worth still in an ongoing capacity to the intelligence agencies I cannot see them really being that bothered by the upkeep of 30 or 40 year old data.

Offline Hog

  • Full Member
  • ****
  • Posts: 1253
  • Woodstock
  • Liked: 329
  • Likes Given: 995
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #19 on: 04/17/2014 06:01 PM »

Yup, the smaller hard cased floppies were 3.5".  I have a stack of them right beside me, I use them occasionally.

Do they still work OK?

I tried one a while ago and it just didn't want to work.

On a serious point I really do fear that future archivists may encounter 'black holes' in the data from such satellites due to issues with the storage.
Yes the 3.5" still work OK, I use them for transferring small files related vehicle engine calibrations as I am using an obsolete IBM Thinkpad to flah and download the vehciles ECM(Engine Control Module).

Last time I used a 5 1/4" floppy was to download "California Games" onto a Commodore 64 in the 80's.

I hope that this Recon. data can be safely stored in some manner.
Paul

Offline Targeteer

  • Senior Member
  • *****
  • Posts: 3871
  • near hangar 18
  • Liked: 982
  • Likes Given: 478
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #20 on: 04/17/2014 06:13 PM »

On a serious point I really do fear that future archivists may encounter 'black holes' in the data from such satellites due to issues with the storage.

They are of course aware of this stuff. That doesn't necessarily mean that they do anything about it.

I believe (and I'm basing this only on circumstantial info I've gathered, but I could ask people) that NASA's Planetary Data System includes efforts to make sure that old data is stored and kept so that it can be accessed. I think that their bigger concern is that new spacecraft are now starting to return so much data that it is going to swamp the old stuff. But they are aware of legacy issues and can reach back several decades.

But would there be the time, money & inclination to do this with very old data from the KH-11. Unless it has some worth still in an ongoing capacity to the intelligence agencies I cannot see them really being that bothered by the upkeep of 30 or 40 year old data.

You be surprised how vital historical reference imagery is when you're looking for details on say, underground bunkers :)
Best quote heard during an inspection, "I was unaware that I was the only one who was aware."

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #21 on: 04/17/2014 07:05 PM »
You be surprised how vital historical reference imagery is when you're looking for details on say, underground bunkers :)

And that's a really good point. Suppose, for instance, that at some point in the future the U.S. finds itself needing to blow up an underground bunker in some country. It might be useful to have the imagery that shows the construction of that facility, which may have taken place back in 1979.

But as to substantial effort to make sure that most imagery is saved in a format that allows it to be accessed in the future, I have my doubts that is taking place.

Offline simonbp

Yup, the smaller hard cased floppies were 3.5".  I have a stack of them right beside me, I use them occasionally.

Amongst other sizes: http://en.wikipedia.org/wiki/Floppy_disk_variants

"Superfloppies" with the same form factor as 3.5-inch 1.44 MB floppy, but 100+ MB capacity, were briefly popular in the late 90s as an alternative to Zip drives before writable CDs became popular.

And then there was Jaz, the intended replacement for Zip drives which never took off. God help you if you need to read one of those nowadays.

Offline kevin-rf

  • Elite Veteran
  • Senior Member
  • *****
  • Posts: 8652
  • Overlooking the path Mary's little Lamb took..
  • Liked: 1123
  • Likes Given: 243
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #23 on: 04/17/2014 10:05 PM »
God help you if you needed to read a jaz disk even back then. They would corrupt faster than Clinton could corrupt interns.

Blackstar, for hidden bunkers I would go back further to before the site was modified. That way you could see what was moved. That hill looks new, let's see if we can pop it...

 I think it would be a fair bet that they are combing through the historical data on all the abandoned military bases along the Ukrainian border that Russia just re-activated.
If you're happy and you know it,
It's your med's!

Offline Blackstar

  • Veteran
  • Senior Member
  • *****
  • Posts: 11378
  • Liked: 2892
  • Likes Given: 1
Re: Declassified KH-9 HEXAGON imagery is difficult to access
« Reply #24 on: 04/18/2014 12:21 PM »
Blackstar, for hidden bunkers I would go back further to before the site was modified. That way you could see what was moved. That hill looks new, let's see if we can pop it...

Yes.

I have heard from a retired intelligence analyst who had only tangential involvement in looking for bunkers that apparently they're pretty difficult intelligence targets. I think he told me that they had one former Soviet person in the 1990s who told them about a rather large underground facility that the CIA had no idea existed. So that's certainly a case where you would want to go back over every image ever taken in that area.

Tags: