2011 Workshop:DBs in CEDAR

From CedarWiki
Revision as of 12:08, 1 July 2011 by Emery (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Using, Administering, and Submitting Data to the VSTO and Madrigal Databases within the CEDAR Community

Location, Date/Time and Duration

2 hours


Barbara Emery
Bill Rideout

Workshop Categories

Altitudes: IT - Latitudes: global - Inst/Model: radar - Other: data from all MLT/IT instruments plus models

Format of the Workshop

short presentations plus hands-on and discussion about future plans

Estimated attendance


Requested Specific Days


Special technology requests

internet access and possible need for extra laptops for hands-on by participants


The VSTO and Madrigal database systems within the CEDAR community are trying to merge so that data in either system are available to the other system, with both access modalities retained. Both systems are also thinking about newer formats so this is a good time to bring these plans to the community to get their feedback.


The Madrigal and CEDAR databases have been around a long time, but have undergone a number of recent developments. Madrigal was originally developed at MIT Haystack Observatory (http://madrigal.haystack.mit.edu/madrigal/) to allow access to upper atmospheric realtime or archived data, but for the next release Jicamarca Radio Observatory is a co-developer. Madrigal is a local database that automatically shares metadata, allowing each site to act like a virtual observatory. The CEDAR DB (http://cedarweb.hao.ucar.edu) was developed at NCAR for incoherent scatter radar data as well as optical and other data, and stores all data locally, allowing central archiving. Both the CEDAR DB and Madrigal presently share the CEDAR database format based on 16 bit integers. Madrigal was installed at NCAR in 2009 (http://madrigal.hao.ucar.edu), with the capability of automatically downloading and archiving data from all other Madrigal sites, making it unnecessary to submit data separately to Madrigal and the CEDAR DB. Madrigal delivers data as ASCII flat files or in any variant of the CEDAR database format via a web interface or via application program interfaces (API's) in python, Matlab, or IDL. The CEDAR data are delivered via the on-line Virtual Solar Terrestrial Observatory (VSTO) system. The data can be retrieved in ASCII files, in netCDFv3 files, and in binary form using the openDAP protocol.

This workshop will feature discussions of future directions in community database design and database formats. Madrigal is exploring HDFv5 while VSTO is exploring netCDFv4 (.nc) with the goal of ingesting and serving NCAR model output (TIE-GCM, TIME-GCM, WACCM) in their native netCDF format. Both formats should be able to be supported especially since netCDFv4 uses HDFv5 as a storage layer. Our goal is to merge the Madrigal and CEDAR systems while allowing access via Madrigal or VSTO. We also plan to showcase examples of how new instruments' data are put into either system, and accessed via either system, or across systems. Attendees are encouraged to discuss ways in which the utility of these systems can be enhanced.


Workshop Summary

This is where the final summary workshop report will be.

Presentation Resources

Upload presentation and link to it here. Links to other resources.

Upload Files Here

  • Add links to your presentations here, including agendas, that are uploaded above. Please add bullets to separate talks. See further information on how to upload a file and link to it.