Astro2010 report and "virtual astronomy" in the next decade

The Astro2010 report is out, and the results are ... mixed. Most people interested in data-intensive astronomy, VO, etc., are delighted with the strong blessing of the LSST (as expected with a high probability). But what does the report imply for the field of data-driven, computationally-enabled astronomy, VO, astroinformatics, etc.?

Ten years ago, the previous decadal survey brought the concept of the VO in the forefront, and the enterprise took off. The vision was there: science transformed and enabled by the new technology of bits. Many other fields of e-Science look at astronomy with an envy and appreciation, we are supposedly the poster child of the e-Science/Cyber-Infrastructure/X-informatics movement. We went through 7 Moore's law cycles since then, with a 100-fold increase in data volumes. The importance of the information and data technologies ought to be even more obvious today. So, what is the vision for the next decade?

The new report offers predictable and obvious blessings of data archiving, and VO is mentioned as a provider of standards for such activities. It pretty much says that the NSF and NASA should continue with their existing policies on this front. That is good. But that is all. It is all "just archiving" - hardly any intellectual substance worth bothering with. You can practically envision the committee yawning and moving on to the more exciting stuff. The word "computation" is taken to mean "HPC in service of ever more ambitious numerical simulations". Hardware is what matters. Never mind that the facilities like LSST, WFIRST, etc., are critically dependent on a different kind of information and computation technologies and methodologies to fulfill their scientific promise and potential (not just to archive their data).

What about the new paradigm for doing science? What about the knowledge discovery and science that simply cannot be extrapolated form the old, pointed-observations, small-samples mode? What about the rapidly evolving methodology of science in the 21st century? It is all about the bits and knowledge, not so much about the atoms assembled in the form of expensive facilities. We are still in the midst of what may be the most profound transformation of science and scholarship ever, driven by the exponentially growing information and computation technology. Unfortunately, there is not much of an indication in the present report of a true understanding of this transformation of science, and not a trace of the vision it inspires. As a field, we may have stepped back. How did we get here?

The decadal survey represents fairly the views, priorities, and the zeitgeist of the astronomy community as a whole. The committee had some tough decisions to make, and they did convey, accurately, in my opinion, what the astronomy community as a whole thinks about VO and such things. VO is (at best) regarded as "just archives", and this opinion is not entirely wrong, as the things stand. So I can understand where the Astro2010 committee is coming from, even though I may wish that they had shown at least as much vision and understanding in this arena as their predecessors did 10 years ago. Sure, some relevant testimony to the committee was provided, and a couple of pertinent and thoughtful white papers were submitted - lost among the hundreds of others, many of which can be fairly described as so much self-serving propaganda, so it would be easy to miss the good stuff. But the cold, cruel fact is that the astronomy community's appreciation, enthusiasm, and support for the VO idea evidently has not grown over the past decade; rather, it seems to have declined, and it may well continue in the same direction.

The problem, I think, is that we, the "greater VO" community, have failed miserably to convey the importance and the potential of these technologies and methodologies to the broader astronomy community as a whole, in a convincing and compelling fashion. Certainly, a lot of hard, unglamorous, and necessary work has been done. However, we have not delivered on the promises of a new path to scientific discovery, envisioned 10+ years ago. I have argued elsewhere why I think this is, and I accept my own share of the blame. But how do we find a constructive way forward?

This matters. The funding agencies are supposed to take their hints from the decadal report, everyone follows the money, and students and postdocs are wise to enter the well-supported fields and projects. This is not just a U.S. problem - these dominoes tend to fall in order, and far away.

Given this situation, what is the long-term future of the VO? Will it become a fully owned subsidiary of the LSST Corp.? Maybe that is not such a bad thing, considering the management and the political skill the LSST leadership has shown, and the growing collection of e-Science talent the LSST project has assembled. We have already seen a number of excellent young people who were engaged in the formative stages of the VO move on to the scientifically greener pastures elsewhere. Maybe VAO will devolve to be a standard-setting body and a debating society, much like IVOA, say, with the exciting development and applications of discovery tools happening elsewhere.

Or maybe we need to think about the broader evolution of our field, and the methodologies it has to develop in order to function effectively in the era of an exponential data abundance, in a project-independent way. After all, this is all about a new, universal scientific methodology, not about the particular needs of any one given project. Imagine a situation where there are no independent detector, instrument, or telescope engineering and development efforts, other than through a few blessed mega-projects. Is this really an optimal path for our field? Or is it simply the only available path?

In any given situation one must operate within the framework determined by the political realities. The astronomy community produced the Astro2010 report, and it should now stand behind it, such as it is, even though there are several sub-fields which feel shortchanged by it. That is the playing field.

So, where does the VO/astroinformatics community really want to be 10 years from now, and how do we get there?

What do you think?

The inadequacy of universities

Picking up on some themes we touched upon in our education sessions, here are some good, provocative links to check:

http://www.edge.org/3rd_culture/tapscott09/tapscott09_index.html

http://www.nytimes.com/2009/04/27/opinion/27taylor.html?pagewanted=all

http://www.youtube.com/watch?v=dGCJ46vyR9o

The 10 problems

One of our goals is to define a list of 10 problems/challenges which we can address successfully over the next year. This is following Jim Gray's idea of the 20 queries: some specific requirements which can focus our work. This blow is a good forum to start suggesting the ideas. So, let me start with some suggestions:

1. Provide a user-friendly data mining service, which will be easy to learn and apply by a typical astronomer, and provide at least some new functionality which is not commonly available. Obviously, this is something that can grow in time.

2. Create a virtual exchange forum for educational materials and experiences for teaching the methods of computational, data-driven science on a serious undergraduate or graduate level.

3. Provide a user-friendly visualization package/service/toolkit for exploration of highly-dimensional data. It could be more than one approach.

4. Establish an effective virtual forum where people interested in astroinformatics can meet and exchange and debate ideas, both in real time and in an off-line fashion. It could be a combination of different tools and approaches.

4a. Have an entirely virtual conference/workshop in this arena.

5. Establish an electronic, non-commercial, peer-reviewed journal for large data sets/archives, algorithms, and other astroinformatics tools and methods.

Let's hear more ideas, and the comments and improvements on these!

Welcome to the AstroInformatics2010 blog

This blog is intended as an open discussion forum for the AstroInformatics2010 conference, http://www.astroinformatics2010.org

Vigorous, but polite and professional discussion is encouraged!