Tags:
create new tag
view all tags

Astrogrid Boot Camp

View this page as a pdf

Astrogrid Software

Download: http://www.astrogrid.org/maven/, http://www.astrogrid.org/maven/org.astrogrid/wars/ Download directly into $Tomcat/webapps.

Astrogrid Components

  • CEA
  • Community
  • File Manager
  • File Store
  • JES
  • mySpace
  • PAL
  • Portal
  • Registry

PAL

PAL (Publisher's Astrogrid Library) allows users to query for data (soon to be renamed DSA.

Download: http://www.astrogrid.org/maven/org.astrogrid/wars - choose recent astrogrid-portal war file, such as astrogrid-portal-0.9-b002p.war

Note: Since March '05, PAL component .war files have adopted the following naming convention: "astrogrid-pal-skycatserver-n.n-bnnnpl.war", where n represents the version numbers.

Installation

  1. Place war file in $Tomcat/webapps and restart Tomcat.
  2. Documentation is bundled with war file: go to http://your_domain:8080/your_pal_name/. The left column contains links to installation, release notes, and configuration. Example: http://msslxx.mssl.ucl.ac.uk:8080/pal_esdo/.
  3. Copy $Tomcat/webapps/warname/WEB-INF/classes/default.properties to $Tomcat/common/classes/astrogrid.properties
  4. Use the Tomcat Administration page to edit these properties in JNDI. Go to Tomcat Admin and click Tomcat Server -> Service -> Host -> Context (astrogrid-pal) -> Resources -> DataSources, click on the the context file, and fill in or change any properties in the text boxes to the right of the screen. Click "Commit Changes." This creates an XML file in $Tomcat/conf/Catalina/locahost with your updated properties. Astrogrid and Tomcat will always look for these properties in JNDI - any further changes made in a text editor to $Tomcat/webapps/warname/WEB-INF/classes/default.properties and $Tomcat/common/classes/astrogrid.properties will be ignored. (See note below.) * datacenter.querier.plugin * Important 4 lines:
      • datatacenter.plugin.jdbc.drivers
      • datacenter.plugin.jdbc.url
      • datacenter.plugin.jdbc.user
      • datacenter.plugin.jdbc.password
    • If your database Postgres, look for "datacenter.queryier.plugin.sql.translator" - uncomment
    • If your database offers circle or crossmatching (mainly astrophysical databases), uncomment
  5. Note: if you undeploy the PAL component or install a new version of Tomcat, the PAL properties will no longer be in JNDI. Open $Tomcat/common/classes/astrogrid.properties in the Tomcat Administration page again and click "commit changes" to add the properties to JNDI. For this reason, it is good practice to manually add any properties changes you make in JNDI to the $Tomcat/common/classes/astrogrid.properties file.
  6. Note 2: if your PAL is named anything other than "pal.war", be sure to reflect this change in the astrogrid.properties file.
  7. PAL Fingerprint: http://yourdomain:8080/your-pal-name/admin/fingerprint.jsp - This is a useful page to check JNDI names, properties, etc.
  8. Note 3: The astrogrid.properties variable "datacenter.max.return" defines the maximum number of records that will be returned by PAL in response to a query, regardless of the query source. The default setting is 2000.

Workflow steps:

  1. User sends ADQL query to PAL (query received by PAL web service). PAL has internal web service used for internal tests - results can be sent to local directory, URL, or email address. The public PAL CEA web service (inputs: ADQL, outputs: lookup identifier [registry identifier, ie ivo://org.filemanager]) sends query results to the lookup identifier. Lookup identifiers are specified in the portal workflow. They can be a directly specified filemanager (such as ivo://uk.ac.mssl/filemanager) OR a community-based name that will lookup a registered user's mySpace area (such as ivo://MSSL/Elizabeth since user Elizabeth is known to the MSSL community). Advanced - specify community user and filemanager - ivo://MSSL/Elizabeth#ivo://uk.ac.mssl/filemanager#ivo://uk.ac.roe/filestore
  2. PAL CEA web service looks up which plugin is used to query data, ie, JDBC, FitsQuerier, EGSOQuerier - this is specified in the pal.config file.
  3. ADQL is converted to SQL or XQuery, and query is submitted to the database. Results are returned as resultset and converted to VOTable. VOTable is returned to filemanager specified by lookup identifier

Query for fits files and receive List of URLs back

  • 2 perl programs: findfits.pl and PrepareXML.pl inside ~griduser//downloads - find fits files and return list of URLs. (Can download from $Tomcat/webapps/servlets-examples/downloads.) findfits.pl searches for filenames and returns URLs for each file found (whether the machine is publicly accessible or not). PrepareXML.pl changes URLs (replaces non-publicly accessible machine name with a publicly accessible machine name) and calls a servlet to redirect files from non-public location (machine with no http access) to public location (machine with http access). Files are not actually transferred from one machine to another - they are simply redirected.

Get fits files back

fileStore

FileStore holds files.

CEA

Installation / Configuration docs: distributed with cea-commandline war file - http://msslxy.mssl.ucl.ac.uk:8080/astrogrid-cea-commandline-MSSL/provider/ApplicationConfiguration.html

  • cea-PAL: data queries
  • cea-commandline: any commandline script
  • cea-http: can call any http service like cgi, perl, etc

mySpace

Installation

Installation Notes -- ElizabethAuden 29 Mar 2005

JES

JES, or the Job Execution Server, controls the execution of jobs, ie queries, data acquisition, tools, etc. (Question: does PAL tell JES when the query is finished, or does FileManager tell JES when the query is finished?) Further Astrogrid documentation is available at http://www.astrogrid.org/maven/docs/HEAD/jes/astrogrid-workflow-userguide.pdf

JES uses the Groovy language for scripting - please see http://groovy.codehaus.org/ for further information, examples, and tutorials.

Portal Workflow

Technical use case: User -> portal -> JES -> PAL

  1. Query SEC (usually by time)
  2. Query XML DB (wavelength, time, date_obs, other fits header keywords)
  3. Get a VOTable file containing list of URLs pointing to fits files
  4. Script that strips URLs out of VOTable (Groovy script in portal)
  5. Pass URLs to CallMakeMPEGFitsImages.sh along with output name outputFile.mpg

Astrogrid Examples

Technical Examples

Building a Query

Step by step instructions from MikeSmith. (Screenshots coming soon. -- ElizabethAuden, 23 Mar 2005)

  1. Login to portal
  2. Select blue "Queries" button.
  3. Type query into scratchpad area, e.g. SELECT * FROM trace.
  4. Select "Save to MySpace" button.
  5. Save query under chosen area/name.

Query is now built and can be accessed through mySpace.

Building a Workflow

Step by step instructions from MikeSmith. (Screenshots coming soon. -- ElizabethAuden, 23 Mar 2005)

  1. Login to portal (if you haven't already done so).
  2. Select blue "Workflows" button.
  3. If there's a workflow currently loaded, move cursor to "File" on the toolbar and select "New". If not, i.e "new workflow" displayed in the "Name:" box, go to Step 5.
  4. Click "OK" on the "Any unsaved workflow information will be removed..." pop-up.
  5. Move cursor to "Edit" on the toolbar, then select "Insert step" -> "here".
  6. Verify "Step" appended to "Sequence".
  7. Select "Step" (highlighted with yellow box) and verify Step/Task panel displayed at the bottom of the webpage.
  8. Click-on the --Select task-- dropdown menu, move cursor to chosen task (highlighted in blue) and click using left mouse button. Note: The dropdown menu displays a small fraction of the available registered tasks. Any task not shown in the list, such as the one used in the "pal_esdo" workflow to connect to the msslxx MySQL database ("astrogrid.mssl/pal_esdo/ceaApplication") will need to be manually entered.
  9. Enter meaningful "Step name:" in the left of the Step/Task panel and click-on the "update step details" button.Wait while webpage is refreshed. Note: May be necessary to repeat step 13 as the "Step name" is sometimes lost at the first attempt.
  10. Click-on workflow "Step".
  11. Verify a step parameters box is displayed with "VOTABLE" and "Query" as inputs and "Result" as single output.
  12. Click-on "Browse" button on the "Query" input line and verify the MySpace page is displayed.
  13. Find and click-on desired query, verify that query is highlighted and that its name appears in the "Name:" box.
  14. Click-on "OK".
  15. Verify that the MySpace page is removed and the query input parameter now displays the MySpace pathname of the selected query.
  16. Click-on "Browse" button on the "Result" output line and verify the MySpace page is displayed.
  17. Find and click-on existing output file or type in the name of a new file into the "Name:".
  18. Click on "OK".
  19. Verify that the MySpace page is removed and the result output parameter now displays the pathname of the selected query.
  20. Select "Update parameter values" button. Wait while webpage is refreshed.
  21. Add meaningful name and description to the task "Name:" and "Description:" boxes respectively.
  22. Move cursor to "File" on the toolbar, then select "Save".
  23. Verify that the MySpace page is displayed.
  24. Select appropriate sub-directory for the workflow, type in suitable name and press "OK".
  25. Verify that the MySpace page is removed.

Workflow is now saved in mySpace and is ready to be loaded and submitted as a job.

Accessing mySpace

(Screenshots coming soon. -- ElizabethAuden, 23 Mar 2005)

  1. Login to portal if you have not already done so.
  2. Click the blue
    "MySpace"
    button.
  3. To upload a file or URL:
    • Click on folder icons in the mySpace browser until you reach the desired folder.
    • Enter filename (next to "Upload File"), click "Browse" and select a file, or enter URL (next to "Upload URL")
    • Click "Upload".
  4. To view a query, votable result, or workflow:
    • Click on the appropriate folder icons in the mySpace browser until you see the name of your file.
    • Click the filename once to highlight in yellow.
    • Under "File" at the top of the mySpace broswer, select "Properties and Downloads". A dialogue box will open with details of your file.
    • Inside the dialogue box, click the link shown next to "Path" - your file should open in the same window.
  5. To rename a file:
    • Under "File" at the top of the mySpace broswer, select "Rename".
    • Click on the appropriate folder icons in the mySpace browser until you see the name of your file.
    • Click the filename once to highlight in yellow. This will populate the "Location" and "Name" boxes at the top of the mySpace browser. Press "OK" - the browser should reload, showing a copy with the new name of your file.
  6. To delete a file:
    • Click on the appropriate folder icons in the mySpace browser until you see the name of your file.
    • Click the filename once to highlight in yellow.
    • Choose "Delete" from the "File" menu. The browser should reload showing your mySpace area without the file.

Solar Examples

Search for MDI data

Get MDI data

SEC query

Solar Movie Maker tool

Solar Overlay tool

Generic Movie Maker

Known AstroGrid quirks

Registry

There are occasions when it is desirable to be able remove a registered component, such as when the component has been created accidentally, or added for experimental purposes but is no longer required. There is no facility for removing a resource from the Registry at present, although this feature is to be added in the near future.

If a component is no longer required, then the Resource status should be set to "inactive" to prevent it from being accidentally built into a workflow (although it will still appear in the Registry list) and the "title" modified to show that the component has been deprecated.

Portal Workflow

The dropdown Task menu at the foot of the Workflow page displays only a small fraction of the total number of registered components. If the task name is known but not displayed it can be entered alongside the "Task name:" label or selected from the browser.

Portal MySpace

Output files created by cancelled workflow jobs, i.e. jobs stopped before completion (successful or otherwise) cannot be deleted, moved or renamed.

Logging AstroGrid problems

Any AstroGrid problems encountered can be checked and reported at the AstroGrid Bugzilla website: http://www.astrogrid.org/bugzilla/

Additional Support Software

Java

Download: http://java.sun.com/downloads/index.html Installation: Download a JDK such as JSEE 1.4.2. Set environment variable JAVA_HOME to the location of the directory where the JDK software is installed (for example, c:\j2re1.4.2_03 or /usr/local/j2re1.4.2_03).

Tomcat

Tomcat is a free, open-source servlet container that works with the Apache webserver. Tomcat is also an EJB container, but Astrogrid software only uses the servlet container aspects. Other free servlet containers include Jetty and Resin.

Download: http://jakarta.apache.org/site/downloads/downloads_tomcat-5.cgi

Install: Download zip or gz file and extract.

Note: Make sure that JAVA_HOME environment variable is set. Set environment variable CATALINA_HOME (also recommend TOMCAT_HOME) to directory where Tomcat is installed (for example, c:\jakarta-tomcat-5.0.28 or /usr/local/jakarta-tomcat-5.0.28).

Note: Set environment variable CATALINA_OPTS to specify the size of TOMCAT's working memory: this will affect the size of files that can be spooled from the AG server to MySpace. For example, CATALINA_OPTS="-Xmx400M" will allocate 400 MB of memory to TOMCAT. Don't forget to export CATALINA_OPTS!

  • http://msslxy.mssl.ucl.ac.uk:8080
  • webapps (currently used)
    • astrogrid-cea-commandline-MSSL (runs movie maker)
    • astrogrid-pal-MSSL (query XML database)
  • later webapps:
    • astrogrid-registry-MSSL
    • astrogrid-filestore-MSSL
    • astrogrid-community-MSSL
    • astrogrid-filemanager-MSSL
    • astrogrid-palSEC-MSSL (on msslxz)
  • msslxy:/home/griduser/workarea/jakarta-tomcat-5.0.19
    • bin - startup.sh, shutdown.sh
    • conf - contains config files. Note: Changes made to the config files in this directory do not require Tomcat to be restarted. This config file is picked up before changes inside the war directory /WEB-INF/web.xml. Protocol: make changes to $Tomcat/conf/Catalina/localhost/warname.xml or in Tomcat Administration - $Tomcat/webapps/warname/WEB-INF/web.xml is ignored (even after Tomcat restart). Also contains server.xml (contains port number configuration). Warname.xml files contain configuration properties.
    • webapps - contains .war files (Tomcat automatically unzips war files into directory structures). Once unzipped, war file can be renamed
      • inside war directory (ie, astro.war becomes /astro/), there is a WEB-INF/webapps.xml file - this will make a config file in the Tomcat "conf" directory, ie $Tomcat/conf/Catalina/localhost/astro.xml
      • in webapps/warname/WEB-INF/classes, look for astrogrid.properties file. This should be the last place that properties are searched for.
    • logs - directory containing Tomcat logs - catalina.out is particularly helpful as it holds a log of all Tomcat messages and errors.
    • Notes: After deploy, if warname.xml is not in $Tomcat/conf/Catalina/localhost/, go to Tomcat administration and click "commit changes" button.

  • Tomcat manager
  • Tomcat administration
    • Tomcat Server -> Service -> Host-> Context -> environment entries: click on any context name and change any type, value, or description in a context. Also, click under Context Actions to create and delete new properties. Note: in this page, value lengths are limited to 70 characters. For values containing more than 70 characters, edit warname.xml in $Tomcat/conf/Catalina/localhost.
  • Tomcat Web Application - lists all webapps - use this page to stop, reload, and undeploy services

Maven

Download: http://maven.apache.org/start/download.html Installation: Download and install. Place "build.properties" file in home directory (on linux machine). Inside build.properties, put maven.remote.repo="http://www.astrogrid.org/maven, http://www.ibiblio.org/maven"

CVS

Code repository containing Astrogrid source code - can be used to download code directly as an alternative to downloading .war files into $Tomcat/webapps.

Download - should be installed on linux / unix machines Installation - :pserver register for /astrogrid/community/pal, then run maven astrogrid-install-artifact. cvs checkout Header (do this once), cvs update-/astrogrid (do this N times to get updates)

eXist data base

EXist is an XML database (as opposed to a relational database like MySQL, Oracle, etc). XML files can be stored in eXist and queried with the XQuery syntax. XQuery tutorial:

Download:http://exist.sourceForge.net/ Install: Can install as a resource external to Tomcat - download zipfile and unzip to location external to Tomcat. Run on a hidden port, such as 9080 - this hides access to outside world. Zip file contains /thirdparties/etc/jetty.xml - can change port numbers.

ElizabethAuden, 22 Mar 2005

Edit | Attach | Watch | Print version | History: r12 < r11 < r10 < r9 < r8 | Backlinks | Raw View | More topic actions
Topic revision: r12 - 2008-03-06 - ElizabethAuden
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2017 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback