create new tag
view all tags

Data Centre - Phase B

UK SDO Data Centre Plans
Elizabeth Auden
18 December 2006

US Data Centre

(Key points from AIAConcepts.pdf, v, 2004:)

Data Products

  • Estimated AIA and HMI full resolution image file size: 24 MB (4096 * 406 pixels * 12 bits / pixel)
  • Joint AIA and HMI data: 441 TB / year (p. A68) note: is this telemetry or calibrated data?
  • Products released to public within 5 minutes of JSOC receipt:
    • quick-look (DN/s, despiked) image products
    • sample movies
    • automatically generated event lists (flares, filament eruption, CME)
    • irradiance curves for each wavelength channel
  • Products released to public within 24 hours of JSOC receipt:
    • image data (flat-fielded, despiked, deconvolved, calibrated)
    • observer-annotated event lists
    • low-resolution summary movies of entire FOV (512 by 512 pixels)
    • full-resolution cutout movies of all identified active regions (5 by 5 arcmin)
    • Summary movies of other identified events
    • "The Sun Today" summary page with links to all generated movies
    • a few DEMs per day
    • a few global potential-field models per day

Online Data Policy

  • All raw telemetry, level-0 data and calibration software backed up on tape (p. A68)
  • All full resolution AIA level-1 data stored "near line" in compressed format (p. A68)
  • Most recent 90 days full res, AIA level-1 plus sub-sampled / clipped subsets of older data kept online (p. A68)

Use of UK Data Centre

All users in the US, UK and elsewhere will be able to browse SDO quicklook data and science data through the JSOC website planned at Stanford. SDO searches will also be available through the Virtual Solar Observatory. These search techniques will access the US SDO archive directly.

The UK data centre will expedite SDO data retrieval for users accessing data through

  1. AstroGrid DSA workflows
  2. AstroGrid HelioScope
  3. eSDO visualization tools: the Streaming Tool, the Image Gallery and the Movie Maker

A ~10 TB hard disk cache will act as a proxy server for SDO data retrieved through these three mechanisms. Popular datasets will be cached in the UK and integrated with the UK DRMS (data resource management system) instance. Initial dataset transfer between the US JSOC archive and the UK ATLAS archive will use the gridFTP protocol to take advantage of its speed. Users accessing the UK archive through one of the above methods will retrieve data with the http protocol.

The UK SDO data centre will make SDO data access more efficient in three ways. First, it will reduce network pressure on the US archive during high traffic periods (e.g., after events such as flares, filament eruptions and CMEs), thereby easing bottlenecks that reduce network speeds. Second, since UK users retrieving data with AstroGrid workflows or DSA since data will see a decrease in download time since their data will be transferred only across the SUPERJANET backbone. Third, users of the eSDO Streaming Tool will see smoother transitions between frames since images will be cached on the same machine hosting the Streaming Tool software.


Level 0 RHESSI data from 00:00 to 23:59 on 26 March 2003 has been downloaded to the eSDO server from a US archive at Goddard Space Flight Center in the US and the University of Glasgow in the UK. Members of the UK solar physics community studying solar flares with RHESSI data are also likely to use SDO (particularly AIA) data once available.

Network tests undertaken in April 2006 between Stanford, RAL, UCL and MSSL have demonstrated that institutions directly on the SUPERJANET backbone (such as RAL and UCL) can transfer data between themselves roughly 10 times faster than UK SUPERJANET backbone institutions can transfer data with Stanford University on the US west coast. See NetworkLatencyTests.

Archive Location # Files in 24 Hrs Total Size Avg File Size Transfer Protocol Download Time to MSSL
RHESSI GSFC US 19 1.43 GB 77.2 MB ftp 0d 02:10:49
RHESSI Glasgow UK 19 1.43 GB 77.2 MB http (wget) 0d 00:12:12
SDO[1] JSOC US 10560 247 GB 24 MB http 15d 16:34:39 [3]
SDO[2] eSDO at RAL UK 10560 247 GB 24 MB http 1d 11:12:56 [3]

[1] Site of future US SDO primary archive.

[2]Proposed site of future UK SDO secondary archive.

[3] Download time extrapolated from relative size of 24 hours' of RHESSI vs SDO data and download time for RHESSI data, e.g.

Download timeSDO = Download timeRHESSI (Total file sizeSDO / Total file sizeRHESSI)

Tech stuff

  • US transfer:
4.99user 27.97system 2:10:49elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+10277minor)pagefaults 0swaps
  • UK transfer:
3.81user 28.26system 12:12.17elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+8387minor)pagefaults 0swaps

-- ElizabethAuden - 18 Dec 2006

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | More topic actions
Topic revision: r2 - 2006-12-18 - ElizabethAuden
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback