Difference: AlgorithmIntegration1131 (r3 vs. r2)

eSDO 1311: Algorithm Integration with the Grid

This document can be viewed as a PDF.
Deliverable eSDO-1311
E. Auden
23 August 2005


AstroGrid CEA


The AstroGrid Common Execution Architecture (CEA) module allows scientists to deploy commandline applications as grid-accessible web services. AstroGrid users can then call these applications as part of AstroGrid workflows, thus automating the process of searching, retrieving, and processing data. The algorithms developed by CEA will be hosted on a UK machine as commandline applications, and instances of CEA will allow each algorithm to be registered with AstroGrid.

Integration Work

The CEA application module will be deployed on the eSDO server at MSSL. Each algorithm will be registered with AstroGrid as a CEA pplication. Eventually, the algorithms and CEA instances will be hosted on live servers in the UK. It may be prudent to restrict access to computationally expensive algorithms; this can be done by checking a user's AstroGrid login details for permission to execute the application in the process described for JSOC modules below.

JSOC Modules


The JSOC pipeline is a set of libraries and executables that will be used to process HMI and AIA data. Many routines will be called automatically whenever the pipeline runs; others will be called by users "on-the-fly" to generate specific high level data products. The pipeline may be distributed between Stanford University and Lockheed Martin. The majority of modules will be written in C. A more detailed description of the pipeline may be viewed at http://hmi.stanford.edu/doc/SOC_GDS_Plan/JSOC_GDS_Plan_Overview_CDR.pdf.

Integration Work

Algorithm modules developed by the eSDO project for AIA and HMI will be placed in the JSOC pipeline. Depending on user interest and other available code, the JSOC team will earmark suitable algorithms to be executed automatically during pipeline runs; others will be executed following user requests. Documentation for each algorithm shall be provided along with the relevant C modules.

The JSOC team and eSDO developers will investigate making these algorithms available to grid users via CEA applications. There are two differences in the approach taken to JSOC pipeline CEA applications versus the UK-hosted CEA applications detailed above: security and data location. Because the pipeline will run on computers reserved for the use of the JSOC team, co-investigators and other designated users, aunauthorized grid users should not be able to execute pipeline commands "just to see what will happen". Therefore, the CEA application that executes pipeline commands will first check the grid user's AstroGrid login details to see if the user is a recognized collaborator. UK solar researchers may apply to the JSOC team to have their names added to the list of collaborators. Then, to minimize large data transfers, the CEA application will obtain its input data from the JSOC data archive instead of receiving input from MySpace or a remote URL.

In addition to algorithm code and documentation, there are three elements of work. First, CEA must be deployed at the JSOC pipeline center, and an application should be built that can execute JSOC pipeline commands. Second, the CEA application should be enabled to check AstroGrid login details to ensure that only recognized collaborators can invoke the pipeline commands. Third, the CEA input should be modified to access the necessary datasets from the JSOC system instead of MySpace or remote URLs.


JSOC Module


SolarSoft is a distributed set of libraries, routines and ancillary data that allow scientists to process solar data in IDL. For suitable algorithms, the C modules developed by the eSDO project will be wrapped in IDL for distribution through the MSSL SolarSoft gateway. Each eSDO algorithm placed into SolarSoft will be accessible as a commandline routine rather than a GUI. These algorithms will assume that a user has access to the relevant SDO data in a local system. Certain algorithms, such as magnetic field extrapolation or the mode asymmetry analysis, may not be suitable for distribution through SolarSoft due to the high intensity computing that will be required.

During the Phase A research period, some experimentation in wrapping C modules in IDL was tried. Documentation of a SolarSoft installation may be viewed at http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/SolarSoftInstallation. In addition, an example of calling a Mandelbrot set C module as an IDL procedure is available at http://www.mssl.ucl.ac.uk/twiki/bin/view/SDO/CallingCfromIDL.

Integration Work

SolarSoft distribution will be the final development step for suitable algorithms. Each algorithm will be evaluated for SolarSoft based on computational requirements. For those deemed suitable, the C module will be wrapped in IDL following the instructions and example at CallingCfromIDL. Next, the IDL procedure will be tested with local data using the SolarSoft installation on the eSDO server at MSSL. Then standardized SolarSoft comments and documentation will be added to the procedure. The procedure and associated C code will be uploaded to the MSSL SolarSoft gateway.

-- ElizabethAuden - 23 Aug 2005

View topic | View difference side by side | History: r4 < r3 < r2 < r1 | More topic actions
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback