eSDO 1211: Quicklook and Visualization Plan

This document can be viewed as a PDF.
Deliverable eSDO-1211
E. Auden, T. Toutain
23 August 2005

Introduction

The SDO mission will produce 2 TB of raw data every day. Although low and high level science products will only comprise a tenth of that volume, 200 GB per day represents a vast amount of information for a person to process. The eSDO quicklook and visualization plans have two goals: firstly, to help scientists search for and identify the SDO data they require, and secondly, to use this targeted search strategy to reduce the number of large data requests transferred across the network. Quicklook and visualization products will primarily take the form of thumbnail images, movies, and catalogues.

Quicklook Products

Images

As the science products are pulled from the US data centre to the UK, SDO processing software will extract images from all low level AIA and HMI science products archived in the UK data centre. Images will be converted to 200 by 200 pixel thumbnail GIFs of approximately 100 kB each and then archived for the duration of the SDO mission. Relevant metadata such as observation time, wavelength or magnetic data, and other science product information will be written to a searchable thumbnail catalogue. Thumbnail images will be produced for the following science products:

  • HMI line-of-sight magnetograms
  • HMI dopplergrams
  • HMI 20 minute averaged filtergrams (intensitygrams)
  • AIA low resolution, full disk images in 8 wavelengths
  • AIA high resolution, tracked active region images in 8 wavelengths (for ~1 - 10 active regions, depending on solar activity)

Movies

Movies shall be generated from thumbnail images on-the-fly by user requests. Users will specify the start time, end time and science products with which they wish to generate a movie; the relevant thumbnail images will be retrieved from the UK data centre and encoded as an MPEG file using the Berkeley MPEG encoder1, currently in use with AstroGrid solar tools. Movies will be cached along with start / stop time and science product metadata, but they will not be permanently archived.

Catalogues

Two catalogues will be generated to aid data searches: a thumbnail catalogue and a small event / coronal mass ejection (CME) dimming region catalogue. The first catalogue will contain information about each thumbnail image generated and archived in the UK data centre. Users will be able to search for thumbnail images based on observation time, science product type, wavelength (for AIA images) and NOAA active region number (for AIA tracked active images). The second catalogue will contain observation time, coordinate information, AIA science product name and other data for small solar events and CME dimming regions identified by the small event detection2 and CME dimming region recognition3 tools processing AIA data on a UK machine.

User Access

User access to thumbnail images, generated movies and catalogues will all be available through the AstroGrid portal. The thumbnail and small event / CME dimming region catalogues will be searchable through an instance of the AstroGrid DSA4 software, while thumbnail images will be retrieved from the archive using an AstroGrid CEA5 tool and placed in the user's home computer or AstroGrid

MySpace
6 area.

In addition, a JSP page will be made available through the eSDO portal that will use the AstroGrid DSA and CEA webservices to display thumbnail images, movies, and catalogue results directly to a webpage. This page will provide simple forms that will allow users to first delineate start time, stop time, and science product type and then search the two quicklook catalogues, view thumbnails, or generate movies.

Technical Requirements

Software

Four types of software are required for the quicklook and visualization strategy described:

  1. Software to extract images from SDO science products and convert them into thumbnail GIFs
  2. The Berkeley MPEG encoder for movie generation
  3. A database such as MySQL to store tabular information for the thumbnail and small event / CME dimming region catalogues
  4. AstroGrid DSA and CEA modules

Hardware

The UK data centre will require a staging machine to pull data from the US data centre and load it into the UK archive (please see DataCentre1321 for more details). To avoid additional data transfers, this machine can also host the image extraction / thumbnail conversion software. A machine will also be required to host the MPEG encoder, database, and AstroGrid software. Because these three pieces of software will not require large data transfers, they can be hosted on either the staging machine or at a second location.

References

  1. Devadhar S., Krumbein C., Liu K., Smoot S. and Eugene H. The Berkeley MPEG Encoder. Berkeley Multimedia Research Company. http://bmrc.berkeley.edu/frame/research/mpeg/mpeg_encode.html. Last viewed 23/08/05.
  2. Please see PhaseASmallEventDetection for more information.
  3. Please see PhaseACMEDimmingRegionRecognition for more information.
  4. Astrogrid DSA, Data Set Access (formerly Publishers Astrogrid Library). http://www.astrogrid.org/maven/docs/1.0.1/pal/index.html. Last viewed 23/08/05.
  5. Astrogrid CEA, Common Execution Architecture. http://www.astrogrid.org/maven/docs/1.0.1/applications/index.html. Last viewed 23/08/05.
  6. Astrogrid MySpace (software component known as FileManager), http://www.astrogrid.org/maven/docs/1.0.1/filemanager/index.html. Last viewed 23/08/05.

-- ElizabethAuden - 23 Aug 2005

Edit | Attach | Watch | Print version | History: r6 | r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions...
Topic revision: r2 - 2005-08-23 - ElizabethAuden
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback