create new tag
view all tags

eSDO Consortium Meeting #2


Date and time: Wednesday, 4 May 2005, 10:00 - 16:00
Location: Room 115, Physics West, University of Birmingham

10:00 Welcome, Coffee EA
10:15 Review of Phase A - staff, meetings / conferences, tech EA
11:00 Data Centre: US and UK EA, LC
11:45 Demonstration of data centre and portal MS
12:15 Lunch all
12:45 Algorithm progress reports MS, SM, TT, SZ
13:45 Algorithm integration - pipeline, language, Solarsoft distribution discussion
14:30 Quicklooks discussion
15:00 Remainder of Phase A - deliverables, meetings / conferences EA
15:30 AOB all
16:00 finish -

Phase A Review


Institution Science Advisors Developers
Birmingham Yvonne Elsworth, Bill Chaplin Thierry Toutain
MSSL Len Culhane, Lidia van Driel-Gesztelyi Elizabeth Auden, Mike Smith
RAL Richard Harrison, Andrzej Fludra Simon Martin
Sheffield Mike Thompson Sergei Zharkov

Meetings / Conferences

Recent meetings / conferences: Since the last consortium meeting in November 2004, eSDO has been represented at 3 meetings and conferences:

  • Talk at UK Solar Missions forum, RAL, 8 & 9 February 2005, Elizabeth Auden
  • Participation in JSOC peer review, Stanford, 17 & 18 March 2005, Elizabeth Auden
  • Talk at UKSP, Birmingham, 4 - 8 April 2005, Lidia van Driel-Gesztelyi, plus attendance and meeting chairs by several eSDO scientists and developers

Upcoming meetings / conferences:

  • Visit from Karel Schrijver and Neal Hurlburt, 22 June 2005, London
  • UK e-Science All Hands meeting, Nottingham, 19 - 22 September 2005: applied for talk in Data Integration And e-Science workshop
  • AIA team meeting, Stanford, formerly 7 - 9 September 2005 (now postponed)
  • Phase A review with PPARC?, end of September / beginning of October
  • Any others?


The msslxx server now has the following utilities and data stores installed. If you need an account on this machine, please fill out the MSSL account request form in your meeting packet and mail to Elizabeth.

  • IDL with Solarsoft libraries for Trace, SOHO, Yohkoh, Goes, Rhessi, and initial Solar-B
  • AstroGrid components: registry, portal, cea, jes, filestore, filemanager, community. For more information and access URLs for each component, please see http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/AstrogridInstallation.
  • VO Tools: AstroGrid solar movie maker
  • Sample data: SOHO-MDI (30/12/01 - 02/02/02), Trace hourly files (30/12/01 - 02/02/02). Multi-wavelength SOHO-EIT data being added.

Data Centre

US Data Centre

The JSOC data centre will hold AIA and HMI data products at Stanford.

  • Image data and metadata / keywords will be held seperately: image data in tape storage, keywords in Oracle DB
  • Level 0 (raw) and level 1 (low level) data will be archived in tape storage and accessible through a fast tape robot - not much delay in data retreival.
  • Most level 2 and level 3 (high level) data products will be generated on the fly through database calls; most high level data products will not be archived.
  • Co-I teams will have access to JSOC machines, DBs, and internal libraries; it is undecided what level of access that grid users will have. Several solutions are being explored, such as cloning the JSOC user Oracle DB for grid access or using digital certificates to control which registered grid users can submit large data requests.

RAID disk storage
  • 400 TB initially
  • 100 TB annual increment
  • SATA drives (500 GB today)

Tape Archive
  • Two PB-sized tape libraries initially
  • PB per library annual increment
  • SAIT (500 GB, 30 MB/s today) or LTO (400 GB, 80 MB/s today)

JSOC architecture

UK Data Centre

These cost figures for temporary and permanent storage at the RAL Atlas petabyte store were obtained from David Corney and Andrew Sansum at RAL. It is understood that predicting equipment costs more than 5 years into the future provides estimates rather than hard figures, and David and Andrew's help is much appreciated.

Disc cache

The disc cache will hold a temporary cache of high and low level data. Following a solar event, a request will be triggered to transfer data products from the Stanford data centre to the UK data centre at RAL. While this data is held in the temporary cache, user requests from the UK will be directed to the RAL data centre; once the temporary storage time has expired, UK user requests will be redirected to the Stanford archive.

It is assumed that a single solar event will generate a request for 24 hours of raw (2TB) and high level (1TB) of data. The cost figures below assume that it is possible for at least one solar event to occur every 24 hours, so keeping data in a temporary cache for 10 days would require a 30 TB cache while keeping data for 90 days would require a 270 TB cache.

30 TB
Year New Data (TB) Cumulative Data (TB) Storage Media Media Costs (excl VAT) (K) Staff Costs(K) Total Charge (K)
2008 30 30 Disc 1 20 12.6 32.6
2009 30 30 Disc 1 0 13 13
2010 30 30 Disc 1 0 13.4 13.4
2011 30 30 Disc 1 0 13.8 13.8
2012 30 30 Disc 2 20 14.2 34.2
2013 30 30 Disc 2 0 14.6 14.6
2014 30 30 Disc 2 0 15 15
Total Cost, 30TB, 2008-2014: 136,600

270 TB
Year New Data (TB) Cumulative Data (TB) Storage Media Media Costs (excl VAT) (K) Staff Costs(K) Total Charge (K)
2008 30 30 Disc 1 175 63 238
2009 30 30 Disc 1 0 64.9 64.9
2010 30 30 Disc 1 0 66.8 66.8
2011 30 30 Disc 1 0 68.8 68.8
2012 30 30 Disc 2 175 70.9 246
2013 30 30 Disc 2 0 73 73
2014 30 30 Disc 2 0 75.2 75.2
Total Cost, 270TB, 2008-2014: 832,700

Tape storage

The tape storage will hold a permanent archive of specified low and high level data products for the lifetime of the SDO mission (late 2008 - mid-2014). The following cost figures assume that one 32 MB HMI filtergram and eight 32 MB AIA filtergrams are sampled every 1000 seconds and stored uncompressed, resulting in ~ 11.5 TB per year accumulating over 7 years.

Year New Data (TB) Cumulative Data (TB) Tape Media Media costs (excl. VAT) (K) Service Costs(K) Total Charge (K) Cost source
2008 11.5 12 9940B 3.5 5.8 9.2 estimated
2009 11.5 23 Titanium 1 2.8 11.5 14.3 estimated
2010 11.5 35 Titanium 1 1.4 17.3 18.6 estimated
2011 11.5 46 Titanium 1 1.4 23.0 24.4 estimated
2012 11.5 58 Titanium 2 6.9 28.8 35.7 estimated
2013 11.5 69 Titanium 2 3.5 35.5 39.0 extrapolated
2014 11.5 81 Titanium 2 3.5 42.3 45.8 extrapolated
Total Costs, 80.5TB cumulative tape store, 2008-2014: 177,800


  1. Which low and high level products should be stored in the temporary disc cache?
  2. How long should products be stored in the temporary disc cache?
  3. Should the temporary disc cache request new data from the Stanford archive every day, or only after events?
  4. If data requests are made only after events, which events should trigger a data request? How will this request be automatically triggered?
  5. If a "grid user" does not have full request rights to the JSOC database or database clone, can we develop a JSOC user account to generate the necessary low and high level data requests from the UK?
  6. Which low and high level data products should be stored in the permanent tape archive?
  7. Is 1000 seconds a good sampling rate? Would a different sampling rate be useful for HMI vs AIA data?
  8. Would a permanent store of sampled data be useful to the UK solar community?
  9. What visualization software would enhance user access to the data held in the permanent store?
  10. Can we obtain a small volume of tape storage at the RAL Atlas store to test data access mechanisms before eSDO ends in 2007?


AstroGrid portal and workflow

Leicester AstroGrid portal: http://cadairidris.star.le.ac.uk:8080/astrogrid-portal
Guest username: guest01
Guest password: guest01
Guest community: uk.ac.le.star

eSDO AstroGrid portal: http://msslxx.mssl.ucl.ac.uk:8080/astrogrid-portal


Algorithm Progress Reports

Birmingham, Thierry Toutain

  • Mode Frequency Analysis
  • Mode Asymmetry Analysis

MSSL, Mike Smith

  • Loop Recognition
  • Magnetic Field Extrapolation
  • Helicity Computation

RAL, Simon Martin

  • CME Dimming Region Recognition
  • DEM Computation
  • Small Event Detection

Sheffield, Sergei Zharkov

  • Subsurface Flow Analysis
  • Perturbation Map Generation
  • Local Helioseismology Inversion Workbench

Algorithm Integration and Development

JSOC Pipeline

The AIA and HMI teams are interested in the 11 primary eSDO algorithms; both teams have informally requested inclusion of these algorithms in the JSOC pipeline. Rick Bogart has stated that SDO pipeline algorithms should be C modules.

Contributions from HMI team Contributions from Co-I teams
Pipeline execution environment Software for intermediate and high level analysis modules
Software & hardware resources (Development environment, libraries, tools) Output data series definition
* Help with defining data series
* Help with porting code to JSOC API
* Algorithmic collaboration
* Verification
* Explain algorithms and implementation
* Help with verification
* Collaborate on improvements if required
  Algorithm Documentation
  Test data / verification Results

HMI pipeline products


JSOC pipeline environment will have C modules / libraries in place. JSOC will use the gcc compiler (available on linux and with cygwin for windows). Several unit testing packages are available for C such as C Unit Test System and Cutest.

Comparison of schedules:

List of standard science products September 2005 November 2005
List of analysis codes April 2005 April 2006
Prototypes of analysis codes June 2006 March 2006 (heritage modules?)
Data Product Modules March 2007 Jan 2008

IDL Solarsoft
If desired, the C modules can be wrapped as IDL routines and distributed through the MSSL Solarsoft gateway.


  1. Any objections to development towards inclusion in the JSOC pipeline?
  2. What further information about the pipeline should be obtained for inclusion in the algorithm integration with grids deliverable?
  3. Is distribution of the eSDO algorithms through Solarsoft useful to the UK solar community?


High level data products

HMI internal rotation maps
  internal sound speed maps
  full disk velocity / sound speed maps
  Carrington synoptics
  high-res velocity / sound speed maps
  deep focus velocity / sound speed maps
  far side activity index
  line of sight magnetic field maps
  vector magnetic field maps
  coronal magnetic field extrapolations
  coronal / solar wind models
  brightness images
AIA irradiance curves
  low resolution summary movies
  full resolution event / feature extraction movies
  coronal field-line models
  reconstructed temperature maps


  1. Which quicklook products can be developed from this list of high level data products?
  2. Which quicklooks will be generated by JSOC teams and which by Co-I teams? (Elizabeth and Theirry will discuss with Rick Bogart, Neal Hurlburt, Rasmus Larsen offline)
  3. Are there other quicklook products that would be useful that are not derived from the above products?
  4. Who in the UK solar community would use these quicklooks, either to aid data searches or to generate a long term synopsis of solar behaviour?
  5. Should quicklooks be archived in the permanent tape store?
  6. Should effort be concentrated on high level data products instead?
  7. Would a catalogue of events / features be useful? Is it in the remit of eSDO? Can we contribute events to the EGSO event / feature catalogues, or should we leave this to science users?

Remainder of Phase A


  • 1121 Proposed solutions for all algorithms, Proposed date change: 30/09/05
  • 1211 Compiled list of SDO data summary formats, 30/09/05
  • 1311 Plan for integration of algorithm applications with grids, Proposed date change: 30/09/05
  • 1321 Plan for integration of SDO data centre with grids, 30/09/05


Towards the end of June, MSSL's loop recognition algorithm will be written up as an example of how the 1121 algorithm documents should be written, including the sections listed below. (All developers with input from science advisors)
  • Inputs
  • Outputs
  • Interface (access through JSOC database call, Solarsoft, etc)
  • Science use case
  • Technical use case
  • Quicklook products
  • Support information (literature references, existing routines, outlines of new routines, list of code libraries, etc)


The list of SDO data summary formats will cover any quicklook / browse data products that will be generated for the JSOC pipeline, the permanent UK tape store, or any generated catalogues that the UK users may find useful - these products will be discussed offline in greater detail with Thierry Toutain and Yvonne Elsworth, and input will be sought from all eSDO scientists and developers. (Thierry / Yvonne / Elizabeth liasing with Rick Bogart)


The plan for integration of algorithm applications with the grid will cover how eSDO will integrate with the JSOC pipeline, how much of pipeline infrastructure should be duplicated for development purposes in the UK, and how "grid user" access to high level data products will be made. (Elizabeth / Sergei liasing with Rasmus Larsen and Rick Bogart at Stanford along with Neal Hurlburt at Lockheed Martin)


The plan for US and UK data centre integration with the grid will follow up the initial data centre implementation plan (1221) with a development / cost schedule. This will also include a report on the use of visualization software for the permanent data store in the UK. (Elizabeth / Mike / Simon / Len liasing with Rick Bogart and Karen Tian at Stanford)

Committee decisions

  • Should we approach PPARC to schedule the Phase A review at the end of September 2005 or wait for them to approach us?
  • Date for next consortium meeting between mid-November and mid-December?

-- ElizabethAuden - 03 May 2005

Topic attachments
I Attachment History Action Size Date Who CommentSorted ascending
GIFgif HMI_pipeline.gif r1 manage 23.3 K 2005-05-03 - 15:26 ElizabethAuden HMI pipeline products
GIFgif jsoc_architecture.gif r1 manage 26.4 K 2005-05-03 - 15:51 ElizabethAuden JSOC architecture
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | More topic actions
Topic revision: r2 - 2005-05-03 - ElizabethAuden
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback