Withholding Hubble Space Telescope DataBad:
In general, withholding Hubble Space Telescope data is a bad thing.
Note: This is a long article; the longest Bad Astronomy to date. This particular issue is one near and dear to me, so please bear with me!
How it works:
The idea that withholding Hubble Space Telescope (HST) data is bad is not something I have heard about in the media, but does seem to have some internet staying power. I have seen three general attitudes about it:
1. People simply being curious as to why HST data are withheld from public view for a year,
2. People that think a NASA conspiracy is doing this, and
3. People that feel that as taxpayers they have a right to immediate access to the data.
Why withhold the data at all?
Hubble is an extremely sensitive instrument, and is capable of taking data that cannot be taken from the ground. For those reasons alone, HST is a very popular instrument for astronomers. There is only so much time HST can spend collecting data in a year, and so the available time on the 'scope is quite limited. Every year, the people at the Space Telescope Science Institute (STScI) announce on behalf of NASA that they are accepting proposals for research projects using HST. The response is always overwhelming; they receive proposals for five times as much time as HST has! So 80% of the proposals must be eliminated. This makes for pretty tough competition among astronomers to get time on HST. Since the 'scope is so expensive to operate, only the very best proposals will get accepted to run. This ensures that we get the best use of our money from HST.
Not surprisingly then, preparing a proposal to get time on HST is a long, arduous process. You must show rigorously why these observations you propose cannot be done from the ground, and why they will help the astronomy community better understand the objects you are observing. You must itemize each expenditure and defend it. Note that in general a lot of the funding goes towards supporting a graduate student, who will help reduce and analyze the data (the idea being that the student will go on to become a productive scientist as well). There is a lot of math and science that goes into a proposal, and the whole process can take weeks or months to prepare! That is time the scientist is not doing other research; they are making a tradeoff between doing work on data now and doing work with HST data later.
But suppose that you have created a proposal and it has been successfully run on HST. They send you the data; it may be several gigabytes of images or spectra, with as much as ten times as much data in the form of calibration data (telescope sensitivities, camera performances, etc.). It is a daunting amount of information. It can take months just to get it organized and ready to analyze.
Now suppose again that the folks at STScI simply take your data and immediately dump it on the Web for anyone to retrieve. There may be someone else out there better prepared than you to process the data. They may scoop you on your own data! You may think that perhaps that other team deserves to analyze the data if they are better prepared for it, but that is short range thinking; if there were no guarantee that you get exclusive rights to the data for a year, there is no incentive to spend all that time getting a proposal ready. In my personal opinion, there is always someone out there better prepared than you are to look at data, but how else will you ever learn how to do the best you can? If only one group ever gets the data no one benefits at all.
This is why the data are held back from the astronomy public for a year. After all the time preparing to get the data, it is only fair to the team that made the proposal to give them enough time to work on it. A year is a good compromise between giving them all the time they need and getting the data out there quickly for others to see. A year is actually a pretty short period of time: I have some observations that are now 7 years old that I have not completely analyzed yet! However, if someone else wants that data, they are free for the taking; since the data are older than one year, all anyone needs to do is ask STScI for it. That gives me incentive to get it done, and it also gives me incentive to do it right: someone else out there may be looking at the same data I am, and so I have to make doubly sure that my analyses are correct. Also remember that scientists ask for and receive grants from STScI to work on analyzing the data. If someone else scoops them, that money is wasted. So it also makes good monetary sense to let the proposing team keep the data rights for a year.
[Note added September 5, 2001: Incidentally, Bad Reader Tuomas Koivula from Finland emailed me with the analogy that HST data is like having a patent; you have rights to the material, but only temporarily. Eventually, that time is up and the patent becomes public. Mind you, people can get patents for ideas on which they worked while employed by the government as well, so the analogy is a pretty good one.]
Early Release of Data/Images
An aside here: when you see the images released from STScI on the news, or in a newspaper, or even on the net, you are not seeing the actual data but the data converted to an image readable by computers (like JPG or GIF images). These formats cannot convey the detailed information in the actual data, and so cannot be used scientifically; you need the actual data numbers to produce any kind of real science. As someone who has tried to produce a nice picture from data, trust me here; it's hard! It takes a lot of time and effort to get the color balance right, to highlight the features you want to see, and then to put it in a format that everyone can read. And hardcopies still don't look as good. Unfortunately, STScI does not have the money or staff to produce nice pictures of all the data HST takes. It is up to the scientists themselves to produce those images, and so many images never make it to the public eye. Even so, there are many many spectacular images that are made public. I have been told that HST cost far less than a single B2 bomber, so even making a hardcopy of one HST image gives you more return on your tax dollar than you put in.
What about a conspiracy?
Do people have a right to the data?
Think of it another way: your tax dollars also pay for top-secret military projects, but these are kept from you. Tax returns are also paid for by tax dollars. Do you have the right to see your neighbor's returns? Now, these examples are more along the lines of actual secrets, while HST has no secrets. But the argument is simply that your tax dollars do not give you the right to see everything they pay for. A better argument here would be to say that in the end, HST is for everyone, and so you have the right to the data. And you do have access to the data, but only after the one year proprietary period is over, for the reasons I outlined above. Remember, no one "owns" the data; the proposers simply have one year to process them.
Why are astronomers so special anyway?
So, can you as a member of the public get your hands on the data? Not easily! The data are stored and maintained in an archive by STScI, which spends a lot of time and money keeping it up and supporting it. The format of the data, as stated above, is not easily interpretable by someone without a lot of special software, and of course it really does take years of training to know what to do with the data. So access is not simple to come by unless you can show a definite need for the actual data. Of course, it isn't for any other type of data either; given an x-ray, for example, how would you examine it? Well, you might simply want it as a nifty picture. That's fine, and STScI has many of HST's images online which you can download. Another good site is the HST Nuggets page which has lots of images as well.
My philosophy on all this
I think of myself as a professional and an amateur. I get paid to work with astronomy and I am damn lucky to be. But I also do it for free, in my backyard. I have a small 10" Newtonian that I take out on occasion and look at the Moon, or try to find a few nebulae or galaxies in my heavily light-polluted sky. The need to do this is part of my soul. The need to help others understand it all is also a fundamental part of me, or else I wouldn't keep these web pages updated during my spare time. I don't get paid for this; I just want others to feel what I feel when I look at the images and ponder their implications. I think that most astronomers would agree with me as well.
DISCLAIMER:I gratefully acknowledge the help and advice of the Office of Public Outreach at STScI. However, I am fully responsible for what has been said here; I do not speak for NASA, AURA or STScI in any way.