eSDO 1211: Quicklook and Visualization Plan
This document can be viewed as a PDF.
E. Auden, T. Toutain
23 August 2005
The SDO mission will produce 2 TB of raw data every day. Although low and high level science products will only comprise a tenth of that volume, 200 GB per day represents a vast amount of information for a person to process. The eSDO quicklook and visualization plans have two goals: firstly, to help scientists search for and identify the SDO data they require, and secondly, to use this targeted search strategy to reduce the number of large data requests transferred across the network. Quicklook and visualization products will primarily take the form of thumbnail images, movies, and catalogues.
The eSDO project will work with the JSOC team to develop a powerful SDO visualization tool, which shall be referred to as the SDO streaming tool in this document. Users may be familiar with web-based mapping tools such as Google Maps
that allow the user to open a street map or satellite map in a web browser, zoom in and out, and pan in eight directions. The SDO streaming tool will provide a similar facility for HMI line-of-sight magnetograms, HMI continuum maps, and 10 channels of AIA images. However, this visualization tool will not only allow researchers to zoom and pan in space, but they will also be able to switch between science products, “pan” in time by rewinding or fast forwarding to other products in a data series, and even “zoom” in time by increasing or decreasing the cadence at which data products are displayed.
Science Use Case
- A solar researcher wishes to identify solar events that have occurred in the past 24 hours in order to download the relevant datasets from an SDO data centre.
- The user opens a web browser and navigates to the SDO streaming tool.
- The user defines a start time, stop time, cadence, and SDO data product in the web browser GUI.
- The GUI displays the full disk / low resolution SDO data product that most closely matches the user’s defined start time.
- Zoom in space with static time: the user can click the image and zoom in through four levels of resolution. All images are displayed in a 512 by 512 window:
- 1st level: full disk, low resolution image
- 2nd level: 512 by 512 extract from data product at 1024 by 1024 resolution
- 3rd level: 512 by 512 extract from data product at 2048 by 2048 resolution
- 4th level: 512 by 512 extract from data product at 4096 by 4096 resolution
- Pan in space with static time: at any level of resolution, the user can pan left, right, up, down, and in four diagonal directions.
- Pan in time with static space: By pressing a “play” button, the user can view SDO data products in the same or nearby series to the original data product. The images displayed to the user will correspond to the chosen instrument cadence; for instance, a user could choose to view 1 line-of-sight magnetogram from every hour, every 10 minutes, or every 45 seconds of HMI observation. The user can then rewind and fast forward to data products available between the defined start and end times.
- Zoom in time with static space: If the user has chosen a cadence that is lower than the instrument cadence for a given data products (for example, 45 seconds for a line-of-sight magnetogram or 18 seconds for a specific AIA channel image), then the user can “zoom in” to the instrument cadence or “zoom out” to a lower cadence such as one data product per hour or per day. Data products corresponding to the chosen cadence will be played in the GUI like a movie.
- Zoom and pan in time and space: The user will ultimately be able to explore different spatial resolutions and locations of SDO data products with at many cadences.
- Switch between data products: At any time, the user can switch between HMI line-of-sight magnetograms, HMI continuum maps, and any of the 10 channels of AIA images.
Technical development for the SDO streaming tool will involve three elements of work. First, a tool to stream HMI and AIA data products in multiple resolutions must be developed; this reduces the amount of data streamed across the network by only sending the piece of image data requested by the user, plus a buffer on all sides to allow quick spatial panning. Second, the tool must be able to stream data from different SDO data centres. The tool should select the nearest SDO data centre to the user’s location that can provide a cached version of the HMI or AIA data product. If the user pans spatially or zooms temporally to a product not held in the user’s local SDO data centre cache, the tool should stream the data from another cache with minimal interruption to the user. Finally, the tool requires a web browser GUI that will allow the users to define input parameters, stream and display the images, and provide facilities to pan and zoom.
The eSDO team will develop a prototype with Rasmus Larsen at Stanford University during the early part of Phase B.
- Web browser streaming tool is invoked with start time, end time, cadence, and SDO data product.
- Nearest SDO data product corresponding to start time is streamed from nearest data centre cache as multi-resolution data - only full disk, low resolution image is sent to user.
- User zooms in space - next level of resolution (plus buffer containing surrounding spatial edge) is streamed.
- User pans in space - next set of spatial edges is streamed and held in buffer; if product doesn't exist in local cahce, product is streamed from US cache
- User zooms in time - next data product in series is streamed; if product doesn't exist in local cache, product is streamed from US cache
- User pans in time - next or previous data product series is streamed; if product doesn't exist in local cache, product is streamed from US cache
SDO processing software will extract images from low level AIA and HMI science products archived in the US data centre. Images will be converted to 200 by 200 pixel thumbnail GIFs of approximately 100 kB each. They may be held in either the US or UK data centre (to be determined) for the duration of the SDO mission. Relevant metadata such as observation time, wavelength or magnetic data, and other science product information will be written to a searchable thumbnail catalogue. Thumbnail images will be produced for the following science products:
- HMI line-of-sight magnetograms
- HMI dopplergrams
- HMI 20 minute averaged filtergrams (intensitygrams)
- AIA low resolution, full disk images in 8 wavelengths
- AIA high resolution, tracked active region images in 8 wavelengths (for ~1 - 10 active regions, depending on solar activity)
Movies shall be generated from thumbnail images on-the-fly by user requests. Users will specify the start time, end time and science products with which they wish to generate a movie; the relevant thumbnail images will be retrieved from the UK data centre and encoded as an MPEG file using the Berkeley MPEG encoder1
, currently in use with AstroGrid
solar tools. Movies will be cached along with start / stop time and science product metadata, but they will not be permanently archived.
Three catalogues will be generated to aid data searches: a thumbnail catalogue, a small event / coronal mass ejection (CME) dimming region catalogue, and a mode parameters catalogue. The thumbnail catalogue will contain information about each thumbnail image generated and archived in the UK data centre. Users will be able to search for thumbnail images based on observation time, science product type, wavelength (for AIA images) and NOAA active region number (for AIA tracked active images). The small event / CME dimming region catalogue will contain observation time, coordinate information, AIA science product name and other data for small solar events and CME dimming regions identified by the small event detection2
and CME dimming region recognition3
tools processing AIA data on a UK machine. Finally, the mode parameters catalogue will provide a table of frequency, lindewidth, amplitude and powers for a range of low l
User access to thumbnail images, generated movies and catalogues will all be available through the AstroGrid
portal. The thumbnail, small event / CME dimming region, and mode parameters catalogues will be searchable through an instance of the AstroGrid
software, while thumbnail images will be retrieved from the archive using an AstroGrid
tool and placed in the user's home computer or AstroGrid
In addition, a JSP page will be made available through the eSDO portal
that will use the AstroGrid
DSA and CEA webservices to display thumbnail images, movies, and catalogue results directly to a webpage. This page will provide simple forms that will allow users to first delineate start time, stop time, and science product type and then search the two quicklook catalogues, view thumbnails, or generate movies.
Five types of software are required for the quicklook and visualization strategy described:
- Software to extract images from SDO science products and convert them into thumbnail GIFs
- The Berkeley MPEG encoder for movie generation
- A database such as MySQL to store tabular information for the thumbnail and small event / CME dimming region catalogues
- AstroGrid DSA and CEA modules
- SDO streaming tool and associated web browser GUI
The UK data centre will require a staging machine to pull data from the US data centre and load it into the UK archive (please see DataCentre1321
for more details). To avoid additional data transfers, this machine can also host the image extraction / thumbnail conversion software; this facility may be hosted in the US or UK. The SDO streaming tool may have multiple instances hosted in the US, the UK, and elsewhere in Europe. A machine will also be required to host the MPEG encoder, database, and AstroGrid
software. Because these three pieces of software will not require large data transfers, they can be hosted on either the staging machine or at a second location.
- Devadhar S., Krumbein C., Liu K., Smoot S. and Eugene H. The Berkeley MPEG Encoder. Berkeley Multimedia Research Company. http://bmrc.berkeley.edu/frame/research/mpeg/mpeg_encode.html. Last viewed 23/08/05.
- Please see PhaseASmallEventDetection for more information.
- Please see PhaseACMEDimmingRegionRecognition for more information.
- Astrogrid DSA, Data Set Access (formerly Publishers Astrogrid Library). http://www.astrogrid.org/maven/docs/1.0.1/pal/index.html. Last viewed 23/08/05.
- Astrogrid CEA, Common Execution Architecture. http://www.astrogrid.org/maven/docs/1.0.1/applications/index.html. Last viewed 23/08/05.
- Astrogrid MySpace (software component known as FileManager), http://www.astrogrid.org/maven/docs/1.0.1/filemanager/index.html. Last viewed 23/08/05.
- 23 Aug 2005