create new tag
view all tags

Solar-B Data Access

The Solar-B access prototype uses AstroGrid software to make files stored in the RAL ATLAS datastore available to users in the UK solar community. This prototype assumes that users will access data through the AstroGrid portal or workbench. The data itself is stored in the ATLAS tape archive, but the metadata database and searching tools are hosted at MSSL.

Update: After a week of OS maintenance on msslxx, the prototype is once again running. Test it out using the directions below and log into the eSDO portal. -- ElizabethAuden, 24 January 2006


This prototype was build to test the following technologies:

  • Configure an AstroGrid DSA module to access an RDBMS catalogue holding metadata for files stored at a remote location.
  • Enable third party data transfers; direct user requests via the catalogue to the remote storage facility and return data directly to user (the data should not be sent via the facility hosting the catalogue).
  • Once user submits a catalogue query, data should be returned without further interaction required from user.
  • Test facilities - a sample MySQL RDBMS and AstroGrid DSA are hosted at MSSL; sample files are held remotely at the ATLAS tape storage facility. Users can submit catalogue requests from any location using a web browser.

Diagram of prototype information flow

User Guide

Build a query

You will build an ADQL query to search the Solar-B metadata stored in a database at MSSL. This metadata includes keywords from the Solar-B fits file headers along with URLs pointing to Solar-B fits files.

  1. Log in to the eSDO portal at http://msslxx.mssl.ucl.ac.uk:8080/astrogrid-portal with your username, password and community.
  2. Click the "Queries" button in the top bar of your welcome page.
  3. Find the Solar-B database table by clicking the "Select a Table" button.
  4. Enter part of a Resource name ('DSA2'), a Resource Publisher ('esdo'), a Resource title ('eSDO DSA2 for use with Solar-B') or description ('solarb') and click the "search for catalogues" button.
  5. Find the resource entitled 'SDO DSA2 for use with Solar-B'
    • If you want all matching columns from the database table, click 'Select' next to Table name: solarb.
    • If you would like to choose specific columns, click 'column' to view the list of possible column names before clicking 'Select'.
  6. The Data Query Builder text box now says "FROM solarb AS T1". You must add a "select" component to the query, and you may also add optional "where" statements. (Click the 5 different examples to view a selection of ADQL queries.) For now, add "select *" to the ADQL query to retrieve all solarb data. The ADQL query should read "SELECT * FROM solarb AS T1".
  7. Click the "Save to MySpace" button. A MySpace Microbrowser will pop up - choose a folder (or make a new folder if you wish) and save the query by entering a name (like 'solarbquery.xml') into the filename textbox. Click 'Save'.

Build a workflow

The workflow executes a sequence of commands that first queries the Solar-B database, then returns the Solar-B metadata as a VOTable formatted XML file (including URLs pointing to the Solar-B files), and finally downloads the Solar-B files from URLs to your MySpace area.

  1. Log in to the eSDO portal if you haven't already.
  2. Click the 'Workflows' button in the top toolbar.
  3. Add a Name and Description to the workflow template. Click "update workflow details".
  4. First step - add the Solar-B query you just built.
    • Edit -> Insert step -> here
    • Scroll to bottom of page to the step dialogue. Click 'Select task' and choose 'Browse registry'. Search for the eSDO-DSA2 CEA Application (the tool that submits your solar-B query) by entering 'DSA2' into either the Task name, Task title, or Description textfields. Click "search for tasks". When you see a resource with title 'eSDO DSA2 database querier' and authorityID 'esdo.mssl.ucl.ac.uk', click 'Select'.
    • Back at the step dialogue, add a Step name of your choice (such as "Query Solar-B files"), and type "source" in the Var. name textfield.Click "update step details".
    • Scroll to the top of the page and click once on the "Step" in your diagram to bring up the Step parameters dialogue.
      1. First input: click "Browse" and search the microbrowser for the query file you saved above. Click "Select". Make sure righthand checkbox is checked.
      2. Second input: leave as "VOTABLE" and make sure checkbox remains unchecked.
      3. First output: type "Result" and make sure checkbox remains unchecked. Click "Update parameter values".
  5. Second step - write a groovy script that will extract URLs from the Solar-B query result VOTable file and then download the Solar-B files fro mthe URLs into MySpace.
    • Click once on "Step" in the diagram. Then click Edit -> Insert logic -> script -> after
    • Scroll to bottom of page to the script dialogue.
    • Enter a description (like "Extract URLs and get Solar-B files". Paste in the script below into the Body Text box. When you are finished, click "Update script details". Note: in the line starting "outstream", the output filename assumes that you have a folder hierarchy "Data/solarb" under your home Myspace directory. Either add these two folders to your MySpace area OR changet the filename in the "outstream" line of the script.

if (source.size() > 0){ 
  // Get the VOTable XML contents from the output of the previous step
  votable = source.Result;
  // Replace ampersands with & - this is a workaround for an external SAX parser
  votable = votable.replaceAll("&","&");
  // Transform the VOTable to an array, then extract values in the URL column
  table=astrogrid.starTableBuilder.makeStarTableFromString(votable, 'votable')
  urlcol = (0 ... table.columnCount).find{table.getColumnInfo(it).name == 'URL'}
  // Now iterate over the array of URLs
  table.iterator().each {
    // Print out a list of URLs as a sanity check
    print "URL: "
    print url

    // Extract just the filename from the URL text
    // Include some debug information
    jes.info("Reading "+url)

    // Now read the remote URL containing data and write the contents to a MySpace file 
    ev = astrogrid.ioHelper.getExternalValue(url)

Save your workflow: File -> Save -> enter filename -> click "Save".

Submit workflow

  1. tt>File -> Submit
  2. Result: " Workflow successfully submitted to JES; you can view its status in the job monitor page."
  3. Click the "job monitor" link to see a list of jobs you've recently submitted - this workflow should be the top job. Status can be initializing, running, error, or completed.
  4. If you see error, click the Job ID of your workflow - this will show you the status of each step, and you can click the "Workflow transcript" button for more details.
  5. If the workflow completed successfully, click the "MySpace" button on the top tool bar and navigate to the folder where your new Solar-B data should be.

Development Guide

This architecture assumes that Solar-B files held in that ATLAS datastore can be accessed as URLs using a servlet installed on a machine at RAL. The servlet uses the ATLAS tape command to stage files from ATLAS to a jukebox. After launching the AstroGrid portal or workbench, a user first builds an ADQL query that will interrogate a database of Solar-B metadata, such as keywords from FITS headers and the URL of each file. The query is incorporated into a workflow that will schedule the database query followed by the retrieval of matching files. Metadata from files matching the ADQL query are returned to the user in VOTable XML format. A script parses the URLs from the VOTable file and then uses each URL to copy the Solar-B file to the user's MySpace storage area.


Access ATLAS Files through a Servlet (RAL)

  1. Upload Solar-B files to ATLAS tape store or disc.
  2. Install Tomcat on a machine external to ATLAS. Drop CommandRun.war into $TOMCAT/webapps - this war file provides a set of java classes that will execute a command listed in the tape.properties file and make the results available as a URL.
  3. Edit the tape.properties file to run the appropriate commands, for instance
execute.command=tape -m<filenumber> -r -f /path/to/jukebox/directory/<filename> <owner> <tapeid>

Test: Use the wget command to test that Solar-B files are accessible as URLs (don't forget to put the URL inside quotes):

% wget "http://your.server.com:8080/CommandRun/Extract?tapeid=yourTapeId&filenumber=1&filename=yourFileName&owner=yourOwner"

Enter Solar-B Metadata in MySQL (MSSL)

  1. Create a database and table for Solar-B metadata on a MySQL instance running at MSSL.The table should contain column names matching the Solar-B fits header keywords, plus a column for the file's URL.
  2. Extract the FITS header keywords from each Solar-B file as it is uploaded into ATLAS. Add a row to the MySQL Solar-B table containing the keywords. Construct a URL for the file based on the URL pattern above.

Test: start MySQL and submit an SQL query for Solar-B metadata. Ensure that the metadata and URL for each row match the relevant fits file.

Install / Configure AstroGrid DSA (MSSL)

  1. Install and start up Tomcat on a machine at MSSL.
  2. Download the AstroGrid DSA (data set access) .WAR file from http://www.astrogrid.org/maven/org.astrogrid/wars/ (note: this tutorial used the SkyCatServer DSA v. 1.1.7). Rename the file to your-dsa.war and copy to $TOMCAT/webapps.
  3. Check the DSA installation at http://your.server.com:8080/your-dsa/
  4. Follow the configuration documentation that is installed with the DSA component, or refer to http://www.mssl.ucl.ac.uk/twiki/bin/view/AstrogridInstallation.
  5. Follow instructions to create the following files:
    • $TOMCAT/conf/your-dsa.properties
    • $TOMCAT/webapps/your-dsa/WEB-INF/classes/your_dsa_metadoc.xml
    • $TOMCAT/common/classes/your-dsa-metadata.xml
  6. Open the Tomcat admin pages and add the following environment variable to the your-dsa context:
    • name: org.astrogrid.config.filename
    • type: java.lang.String
    • value: $TOMCAT/conf/your-dsa.properties
    • override: leave checked
    • description: Your DSA properties file
  7. Click "Save", "Commit Changes", and then go to the Tomcat Manager app. Reload the your-dsa context.
  8. Follow instructions in the your-dsa documentation to register the DSA with an AstroGrid registry. You must register these three components:
    • Your-DSA Tabular Database
    • Your-DSA CEA Service
    • Your-DSA CEA Application

Test: Log in to the AstroGrid workbench or portal and construct a workflow like the one described in the User's Guide above, but search for tables and DSA tools pertaining to your-dsa rather than DSA2.

-- ElizabethAuden - 13 Jan 2006

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatwar CommandRun.war r1 manage 3.6 K 2006-01-13 - 18:09 ElizabethAuden CommandRun.war for RAL servlets
JPEGjpg SolarBaccessURLs.jpg r1 manage 50.1 K 2006-01-13 - 18:05 ElizabethAuden  
JPEGjpg prototype.jpg r1 manage 36.7 K 2006-01-24 - 15:41 ElizabethAuden Diagram of prototype information flow
Unknown file formatwf solarbURLs.wf r1 manage 3.0 K 2006-01-13 - 18:12 ElizabethAuden Solar-B workflow: query and download Solar-B files
XMLxml solarbURLs.xml r1 manage 0.9 K 2006-01-13 - 18:11 ElizabethAuden Solar-B ADQL query
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions
Topic revision: r5 - 2006-01-24 - ElizabethAuden
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback