create new tag
view all tags

eSDO Phase A Summary Report

This document is also available as a pdf.
eSDO Summary Report
Elizabeth Auden
30 September 2005


PPARC has funded the eSDO project to make data and algorithms from the Solar Dynamic Observatory (SDO) available to the UK solar community using the virtual observatory. The projected is funded for three years beginning on 1 October 2004 and terminating 30 September 2007. Elizabeth Auden is the eSDO technical lead / project manager, and the four developers are Vito Graffagnino, Mike Smith, Thierry Toutain, and Sergei Zharkov. The project is advised by seven scientists: Len Culhane, Bill Chaplin, Yvonne Elsworth, Andrzej Fludra, Richard Harrison, Michael Thompson, and Lidia van Driel-Gesztelyi.

The Atmospheric Imaging Assembly (AIA) and Helioseismic and Magnetic Imager (HMI) instruments on board SDO will produce ~ 2 TB of raw data per day, and the high-level pipeline data products will be approximately one tenth of that volume. Four UK Solar Physics groups have Co-Investigator involvement in the SDO AIA and HMI investigations: MSSL, Rutherford Appleton Laboratory, and the Universities of Birmingham and Sheffield. The huge SDO data volume requires special measures to ensure effective data handling:

  • Local and global helioseismology specialist algorithms and feature recognition procedures for AIA and HMI images
  • Development of visualization tecnniques and summary data to allow high speed searches of the SDO databases
  • Implementation of AstroGrid software and close coordination with the Virtual Solar Observatory at the US SDO data centre hosted at Stanford University
  • Deployment of a UK SDO data centre to provide access to science data products, catalogues and thumbnail images

The eSDO Phase A ends on 30 September 2005; the deliverables generated in this initial year of research describe how solar algorithms, data centres, and data visualization will be achieved in two years of development during Phase B. The six deliverables achieved at the end of Phase A are outlined below. At the end of this summary report is a synopsis of workpackages for corresponding Phase B deliverables, followed by hyperlinks to the Phase A formal deliverables. The remainder of this report presents a broad overview of the conclusions reached during the Phase A period of the eSDO grant on how best to achieve the deliverables of Phase B.

  • eSDO 1111: List of solar algorithms that eSDO institutions will develop and deploy as grid services. Completed 1 April 2005
  • eSDO 1121: Proposed solutions to each of the 11 algorithms detailed in eSDO 1111. Completed 30 September 2005.
  • eSDO 1131: Plan for integrating algorithms as grid services with AstroGrid, the JSOC pipeline, and SolarSoft.
  • eSDO 1211: Plans for quicklook products and visualization techniques including catalogues, thumbnail, gallery and movie generation, and the SDO streaming tool. Completed 30 September 2005
  • eSDO 1311: Design plans for implementing the UK data centre and integrating resource requests with the US data centre. Completed 1 April 2005
  • eSDO 1321: Design plans for integrating the UK data centre and, where possible, the US data centre with AstroGrid. Completed 30 September 2005


Solar Algorithms

The four institutions of the eSDO project will design and implement 11 solar algorithms covering three disciplines: image and feature recognition, global helioseismology, and local helioseismology. A full list of these algorithms will institutional responsibilities can be viewed as deliverable eSDO 1111: Solar Algorithm List. Individual technical writeups for each algorithm constitute the second deliverable, eSDO 1121: Solar Algorithm Proposed Solutions.

MSSL and RAL will develop image and feature recognition algorithms for use with AIA and HMI data. These algorithms include coronal loop recognition, non-linear magnetic field extrapolation, helicity computation, small event detection, differential emission measure (DEM) computation, and coronal mass ejection (CME) dimming region recognition. The universities of Birmingham and Sheffield will concentrate on global and local helioseismology respectively; the Birmingham group will implement mode frequency analysis and mode asymmetry analysis algorithms while the Sheffield group will develop subsurface flow analysis, perturbation map generation and computation of local helioseismology inversion algorithms.

Algorithm Distribution

Solar algorithms developed by eSDO will be made available to UK users in three ways. First, each algorithm will be deployed as an AstroGrid CEA web service hosted in the UK. AstroGrid CEA web services can be accessed through the AstroGrid portal, workbench and workflow systems. Second, all suitable algorithms will be wrapped in IDL for SolarSoft distribution through the MSSL gateway. Initial trials for wrapping C modules as IDL procedures has been documented at WrappingCInIDL. Third and finally, algorithms will be deployed in the Joint Science Operations Committee (JSOC) pipeline systems at Stanford University and Lockheed Martin. The JSOC team will designate some pipeline modules to run automatically, and other modules will be invoked by user requests. UK SDO co-investigators and their teams will be able to access the JSOC pipeline directly through accounts at Stanford University. The eSDO project will investigate installation of a CEA application that can allow authorized registered AstroGrid users to execute JSOC pipeline commands. For more detail, please view deliverable eSDO 1131: Algorithm Integration.

Data Visualization

The huge volume of SDO data collected every day makes it imperative to search the data archive efficiently. Visualization techniques such as streaming tools, catalogues, thumbnail extraction and movie generation aid scientists in archive navigation. More details about quicklooks and visualiztion are available in deliverable eSDO 1211: Quicklook and Visualization Plan.

Quicklook Products

Users will have access to three types of quicklook products: image thumbnails, catalogues, and movies. Software developed in conjunction with Rick Bogart at Stanford University will extract thumbnail images from AIA and HMI FITS files, add labels and instrument metadata to the images, and save the images as GIFs. These images can then be stored in the US or UK. A thumbnail catalogue will store basic scientific metadata about each extracted image such as observation time, wavelength, and product type: full disk AIA image, tracked active region AIA image, HMI line-of-sight magnetograms, HMI 20 minute averaged filtergrams, or HMI dopplergrams. Users will interact with a web browser GUI to specify start time, end time, cadence and data products that will generate image galleries or MPEG movies on-the-fly. The images or movie will be displayed in the web browser.


In addition to the thumbnail catalogue, two science catalogues will be generated from eSDO algorithms. One catalogue will store stastical information about small solar events and CME dimming regions. This catalogue will be annotated continuously as the two relevant algorithms automatically process AIA data in the UK. A second catalogue will provide monthly helioseismology information generated from the mode parameters analysis algorithm; this catalogue will be generated in the US using eSDO software. Both catalogues will be searchable the AstroGrid system through instances of DSA.

Visualization Techniques

The SDO streaming tool, originally conceptualized by Phil Scherrer, will allow users to view HMI and AIA science products in a web browser and then interact with a GUI to pan and zoom in both space and time. A user will open a web browser, navigate to the SDO streaming tool, and then specify a start time, stop time, cadence, and data product. Three types of SDO products will be available: HMI line-of-sight magnetograms, HMI continuum maps, and AIA images from 10 channels. The user will be able to zoom in spatially from a full disk, low resolution image to a full resolution display of a solar area. Zoomed images will be able to be panned in eight directions. Similarly, once a user has selected a cadence (for instance, 1 image per hour), data products matching that cadence will be displayed on the screen; users will be able to "zoom" in time by increasing or decreasing this cadence, while rewind, fast forward and pause facilities will allow users to "pan" in time.

Development of this tool will be based on wavelet compression algorithms developed by Rasmus Larsen at Stanford University. This type of compression allows data to be streamed effectively since only pixels pertaining to the user's requsted resolution are sent over the network. Pixels around the image boundary are also held in a buffer so that the user can spatially pan smoothly. The streaming tool will be refined by intelligently streaming data from the disc cache closest to the user.

Data Centres

The primary SDO data centre will be based in the US, and users can access the data directly through the Virtual Solar Observatory (VSO). A second data centre will be established in the UK. Rather than providing a full data mirror, this data centre will cache recent data products and popular data requests. This local data cache will provide UK scientists with more rapid access to SDO data and allow SDO data searches to be included in AstroGrid workflows.

UK Data Centre

The UK data centre will provide searchability and fast access to SDO data for users within the UK. The UK data centre will require three software components to enable this functionality. A MySQL database will hold searchable metadata for SDO science products, and an instance of the AstroGrid DataSet Access (DSA) module will make the database searchable through AstroGrid workflows. An instance of an AstroGrid Common Execution Architecture (CEA) application will transfer data from the UK data centre to the user's virtual storage area or local storage. More details of the UK data centre implementation can be found in deliverable eSDO 1311: Data Centre Implemenation Plan.

The eSDO development and science advisory teams investigated several data centre storage models with input from the JSOC and UK solar community. A "light footprint" has been chosen for development; this data centre model will use a ~30 TB disc cache to store a 60 day rolling cache of the most recent AIA science products along with a "true cache" of HMI and older AIA science products requested by UK users through the AstroGrid system. When a UK user requests an HMI or AIA science product through the AstroGrid system, the request will be sent to the UK data centre. If the data is not available there, the request will be redirected through an AstroGrid / VSO interface, and the relevant data products will be returned to the UK in export format (FITS, JPEG, VOTable, etc). The data will be cached in the UK data centre when a copy is transferred to the user. A fuller explanation of this data centre model along with descriptions of other models investigated is provided in deliverable eSDO 1321: Data Centre Integration Plan.

US Data Centre

The US data centre will be maintained by the JSOC team at Stanford University. VSO will be front end for user requests to the JSOC data centre, so eSDO development of an AstroGrid / VSO interface will permit AstroGrid users in any country to include VSO searches of SDO data in AstroGrid workflows. Colleagues at Stanford are investigating three AstroGrid components to aid grid integration of SDO data and tools: DSA for data searching, CEA for data transfers and access to pipeline commands, and the JES workflow engine to drive pipeline execution flows. eSDO developers will advise and support on the installation and configuration of these modules; during Phase A, a reference implementation of all AstroGrid modules was completed on the eSDO server at MSSL. The AstroGrid installation was documented along with a science user guide as a resources for the UK solar community and JSOC scientists. Please see links to these documents in the appendix.

Network latency tests will be undertaken between Stanford University, MSSL, UCL and RAL early in Phase B. The results of these tests will aid development of the AstroGrid / VSO bridge, configuring the CEA application to transfer AIA and HMI data from the JSOC data centre to the UK data centre, and how data will be streamed in the SDO visualization tool described in the next section. Details of eSDO involvement with the US data centre can be read in deliverable eSDO 1321: Data Centre Integration.

Community Support

The needs of the UK solar community drive the eSDO project. Development and grid accessibility of data centres, solar algorithms and visualization techniques will be refined as feedback from the community shapes the requirements of these three areas of work. During Phase A, community feedback has been solicited through an article and questionnaire in UK Solar News along with interviews with solar research groups, meetings with the JSOC team and SDO co-investigators, and collaboration with the AstroGrid development team. In addition, eSDO development plans have been disseminated to the UK solar community, the US solar community and the UK grid community through conferences and workshops. A full list of eSDO attendences can be viewed online at http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/ConferencesIn2004 and http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/ConferencesIn2005.

At the beginning of Phase B, the full Phase A report will be distributed to key members of the UK solar community to elicit final feedback of the eSDO research phase. This report, along with all eSDO documents, are permanently available online at http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO. As Phase B progresses, interaction with the community will be maintained in a number of ways. First, data centre development will continue in conjunction with the JSOC team, the ATLAS storage facility, and the AstroGrid project. Second, the eSDO team will attend an SDO algorithm workshop in February 2006 with JSOC developers; a number of parallel sessions will allow algorithm developers from the US, UK and elsewhere in Europe to collaborate on complex problems such as DEM measure and magnetic field extrapolation. The high performance computing centre at UCL will assist in enabling grid accessibility to algorithms that require parallel processing, and rigorous science testing of algorithms with test data from Solar-B and other instruments will occur in late 2006. Finally, eSDO developers will work closely with Stanford University scientists in the development of the SDO streaming tool.

Future Plans

The eSDO Phase B development stage will last from 1 October 2005 to 30 September 2007, and a post-launch support proposal will be completed by the eSDO consortium in early 2007. The following workpackages have been identified:

  • 2100 Solar Algorithms
    • 2110 Algorithm Coding
      • 2111 Completed code and grid interfaces for algorithm applications 30/06/06
      • 2112 Test script for scientific testing of algorithms 01/09/06
      • 2113 Completed tests with scientists' comments 22/12/06
      • 2114 Completed, refined code satisfying scientific testing results 30/03/07
    • 2120 Algorithm Grid Integration
      • 2121 Completed integration of solar algorithms with AstroGrid 31/05/07
      • 2122 Completed integration of solar algorithms with JSOC 31/07/07
      • 2123 Completed integration of solar algorithms with SolarSoft 30/09/07
  • 2200 Quicklook and Visualization
    • 2210 Quicklook Products
      • 2211 Completed application to extract labelled thumbnail images from FITS files 31/03/06
      • 2212 Completed web browser application to generate image galleries on the fly 30/06/06
      • 2213 Completed web browser application to generate movies on the fly 31/07/06
    • 2220 Catalogues
      • 2221 Completed database and DSA instance for thumbnail catalogue 31/08/06
      • 2222 Completed database and DSA instance for small event / CME dimming region catalogue 30/09/07
      • 2223 Completed database and DSA instance for helioseismology mode parameters catalogue 30/09/07
    • 2230 SDO Streaming Tool
      • 2231 Rapid prototype for streaming tool 30/06/06
      • 2232 Evaluation of rapid prototype functionality 31/07/06
      • 2233 Completed functionality to stream data from multiple caches 22/12/06
      • 2234 Completed streaming tool 30/06/07
  • 2300 Data Centres
    • 2310 UK Data Centre
      • 2311 Completed implementation of data centre on eSDO development server 22/12/06
      • 2312 Completed integration of SDO data centre with AstroGrid 30/09/07
    • 2320 US Data Centre
      • 2321 Completed network latency tests 31/03/06
      • 2322 Completed AstroGrid / VSO interface 30 /09/06
      • 2323 Completed support for AstroGrid modules installed at JSOC data centre 30/09/07

Web References

Formal Deliverables

  1. eSDO 1111: Solar Algorithm List, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/SolarAlgorithms1111
  2. eSDO 1121: Solar Algorithm Proposed Solutions, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/SolarAlgorithms1121
  3. eSDO 1131: Algorithm Integration Plan, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/AlgorithmIntegration1131
  4. eSDO 1211: Quicklook and Visualization Plan, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/QuicklookVisualization1211
  5. eSDO 1311: Data Centre Implementation Plan, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/DataCentreImplementation1311
  6. eSDO 1321: Data Centre Integration plan, http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/DataCentreIntegration1321

Appendices and Other Documents

  1. Phase A Report (Full) - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/PhaseAReport
  2. Phase A Summary Report - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/PhaseASummaryReport
  3. Phase B workpackages - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/PhaseBWorkpackages
  4. AstroGrid Userís Tutorial - - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/AstroGridTutorials
  5. AstroGrid Installation / Configuration Tutorial - - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/AstrogridInstallationV11
  6. Wrapping C modules in IDL - http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/CallingCfromIDL

-- ElizabethAuden - 30 Sep 2005

Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | More topic actions
Topic revision: r6 - 2005-10-14 - ElizabethAuden
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback