Difference: QuicklookVisualization1211 (r5 vs. r4)

eSDO 1211: Quicklook and Visualization Plan

This document can be viewed as a PDF.
Deliverable eSDO-1211
E. Auden, T. Toutain
23 August 2005

Introduction

The SDO mission will produce 2 TB of raw data every day. Although low and high level science products will only comprise a tenth of that volume, 200 GB per day represents a vast amount of information for a person to process. The eSDO quicklook and visualization plans have two goals: firstly, to help scientists search for and identify the SDO data they require, and secondly, to use this targeted search strategy to reduce the number of large data requests transferred across the network. Quicklook and visualization products will primarily take the form of thumbnail images, movies, and catalogues.

Visualization Tool

The JSOC team has identified a streaming tool that can be developed for visualization of AIA and HMI datasets. A user will open a browser-based GUI, choose either an HMI line-of-sight magnetogram or one of the 10 AIA channels, and select a start time and start cadence. The GUI will stream the AIA or HMI data product from the nearest local disc cache as a full solar disc, low resolution 512 by 512 image to the user's screen. The user will then be able to pan and zoom in space, time, and data product while studying evolving behaviour of solar features.

Description

Zoom and Pan in Space

The eSDO project will work with the JSOC team to develop a powerful SDO visualization tool, which shall be referred to as the SDO streaming tool in this document. Users may be familiar with web-based mapping tools such as Google Maps or MultiMap that allow the user to open a street map or satellite map in a web browser, zoom in and out, and pan in eight directions. The SDO streaming tool will provide a similar facility for HMI line-of-sight magnetograms, HMI continuum maps, and 10 channels of AIA images. However, this visualization tool will not only allow researchers to zoom and pan in space, but they will also be able to switch between science products, “pan” in time by rewinding or fast forwarding to other products in a data series, and even “zoom” in time by increasing or decreasing the cadence at which data products are displayed.

Zoom and Pan Science Use Case in Time

Switch Data Products

  1. A solar researcher wishes to identify solar events that have occurred in the past 24 hours in order to download the relevant datasets from an SDO data centre.
  2. The user opens a web browser and navigates to the SDO streaming tool.
  3. The user defines a start time, stop time, cadence, and SDO data product in the web browser GUI.
  4. The GUI displays the full disk / low resolution SDO data product that most closely matches the user’s defined start time.
    1. Zoom in space with static time: the user can click the image and zoom in through four levels of resolution. All images are displayed in a 512 by 512 window:
      • 1st level: full disk, low resolution image
      • 2nd level: 512 by 512 extract from data product at 1024 by 1024 resolution
      • 3rd level: 512 by 512 extract from data product at 2048 by 2048 resolution
      • 4th level: 512 by 512 extract from data product at 4096 by 4096 resolution
    2. Pan in space with static time: at any level of resolution, the user can pan left, right, up, down, and in four diagonal directions.
    3. Pan in time with static space: By pressing a “play” button, the user can view SDO data products in the same or nearby series to the original data product. The images displayed to the user will correspond to the chosen instrument cadence; for instance, a user could choose to view 1 line-of-sight magnetogram from every hour, every 10 minutes, or every 45 seconds of HMI observation. The user can then rewind and fast forward to data products available between the defined start and end times.
    4. Zoom in time with static space: If the user has chosen a cadence that is lower than the instrument cadence for a given data products (for example, 45 seconds for a line-of-sight magnetogram or 18 seconds for a specific AIA channel image), then the user can “zoom in” to the instrument cadence or “zoom out” to a lower cadence such as one data product per hour or per day. Data products corresponding to the chosen cadence will be played in the GUI like a movie.
    5. Zoom and pan in time and space: The user will ultimately be able to explore different spatial resolutions and locations of SDO data products with at many cadences.
    6. Switch between data products: At any time, the user can switch between HMI line-of-sight magnetograms, HMI continuum maps, and any of the 10 channels of AIA images.

Technical Case

Technical development for the SDO streaming tool will involve three elements of work. First, a tool to stream HMI and AIA data products in multiple resolutions must be developed; this reduces the amount of data streamed across the network by only sending the piece of image data requested by the user, plus a buffer on all sides to allow quick spatial panning. Second, the tool must be able to stream data from different SDO data centres. The tool should select the nearest SDO data centre to the user’s location that can provide a cached version of the HMI or AIA data product. If the user pans spatially or zooms temporally to a product not held in the user’s local SDO data centre cache, the tool should stream the data from another cache with minimal interruption to the user. Finally, the tool requires a web browser GUI that will allow the users to define input parameters, stream and display the images, and provide facilities to pan and zoom.

Quicklook Products

Images

As the science products are pulled from the US data centre to the UK, SDO processing software will extract images from all low level AIA and HMI science products archived in the UK data centre. Images will be converted to 200 by 200 pixel thumbnail GIFs of approximately 100 kB each and then archived for the duration of the SDO mission. Relevant metadata such as observation time, wavelength or magnetic data, and other science product information will be written to a searchable thumbnail catalogue. Thumbnail images will be produced for the following science products:

  • HMI line-of-sight magnetograms
  • HMI dopplergrams
  • HMI 20 minute averaged filtergrams (intensitygrams)
  • AIA low resolution, full disk images in 8 wavelengths
  • AIA high resolution, tracked active region images in 8 wavelengths (for ~1 - 10 active regions, depending on solar activity)

Movies

Movies shall be generated from thumbnail images on-the-fly by user requests. Users will specify the start time, end time and science products with which they wish to generate a movie; the relevant thumbnail images will be retrieved from the UK data centre and encoded as an MPEG file using the Berkeley MPEG encoder1, currently in use with AstroGrid solar tools. Movies will be cached along with start / stop time and science product metadata, but they will not be permanently archived.

Catalogues

Three catalogues will be generated to aid data searches: a thumbnail catalogue, a small event / coronal mass ejection (CME) dimming region catalogue, and a mode parameters catalogue. The thumbnail catalogue will contain information about each thumbnail image generated and archived in the UK data centre. Users will be able to search for thumbnail images based on observation time, science product type, wavelength (for AIA images) and NOAA active region number (for AIA tracked active images). The small event / CME dimming region catalogue will contain observation time, coordinate information, AIA science product name and other data for small solar events and CME dimming regions identified by the small event detection2 and CME dimming region recognition3 tools processing AIA data on a UK machine. Finally, the mode parameters catalogue will provide a table of frequency, lindewidth, amplitude and powers for a range of low l degrees.

User Access

User access to thumbnail images, generated movies and catalogues will all be available through the AstroGrid portal. The thumbnail and small event / CME dimming region catalogues will be searchable through an instance of the AstroGrid DSA4 software, while thumbnail images will be retrieved from the archive using an AstroGrid CEA5 tool and placed in the user's home computer or AstroGrid

MySpace

6 area.

In addition, a JSP page will be made available through the eSDO portal that will use the AstroGrid DSA and CEA webservices to display thumbnail images, movies, and catalogue results directly to a webpage. This page will provide simple forms that will allow users to first delineate start time, stop time, and science product type and then search the two quicklook catalogues, view thumbnails, or generate movies.

Technical Requirements

Software

Four types of software are required for the quicklook and visualization strategy described:

  1. Software to extract images from SDO science products and convert them into thumbnail GIFs
  2. The Berkeley MPEG encoder for movie generation
  3. A database such as MySQL to store tabular information for the thumbnail and small event / CME dimming region catalogues
  4. AstroGrid DSA and CEA modules

Hardware

The UK data centre will require a staging machine to pull data from the US data centre and load it into the UK archive (please see DataCentre1321 for more details). To avoid additional data transfers, this machine can also host the image extraction / thumbnail conversion software. A machine will also be required to host the MPEG encoder, database, and AstroGrid software. Because these three pieces of software will not require large data transfers, they can be hosted on either the staging machine or at a second location.

References

  1. Devadhar S., Krumbein C., Liu K., Smoot S. and Eugene H. The Berkeley MPEG Encoder. Berkeley Multimedia Research Company. http://bmrc.berkeley.edu/frame/research/mpeg/mpeg_encode.html. Last viewed 23/08/05.
  2. Please see PhaseASmallEventDetection for more information.
  3. Please see PhaseACMEDimmingRegionRecognition for more information.
  4. Astrogrid DSA, Data Set Access (formerly Publishers Astrogrid Library). http://www.astrogrid.org/maven/docs/1.0.1/pal/index.html. Last viewed 23/08/05.
  5. Astrogrid CEA, Common Execution Architecture. http://www.astrogrid.org/maven/docs/1.0.1/applications/index.html. Last viewed 23/08/05.
  6. Astrogrid MySpace (software component known as FileManager), http://www.astrogrid.org/maven/docs/1.0.1/filemanager/index.html. Last viewed 23/08/05.

-- ElizabethAuden - 23 Aug 2005

View topic | View difference side by side | History: r6 < r5 < r4 < r3 | More topic actions
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback