Difference: DataCentreIntegration1321 (r5 vs. r4)

eSDO 1321: Data Centre Integration Plan

This document can be viewed as a PDF.
Deliverable eSDO-1321
E. Auden
23 August 2005

UK Data Centre

The UK data centre will host some data products from the Helioseismic and Magnetic Imager (HMI) and the Atmospheric Imaging Assembly (AIA). Helioseismologists will be interested in HMI data collected for long periods of time. By contrast, AIA data will cater to solar physicists analysing events such as flares and coronal mass ejections; this audience will primarily be interested in recent data. Low and high level data products from both instruments will be permanently available through the US data centre, so the UK data centre holdings will focus on providing fast access to data of the most interest to solar scientists across the UK.

Archived System Data

HMI

Three architecture models have been investigated for the UK data centre: one light footprint and two heavy footprints. In the light footprint model, we assume that the role of the UK data centre will be to provide fast access to cahced SDO export data products. The two heavier models describe an active global role in the JSOC's Data Resource Management System (DRMS) and Storage Unit Management System (SUMS). At the end of the eSDO project's Phase A, other global DRMS and SUMS instances are being considered in the US and possibly in Germany.

The following HMI data products will be archived for the nominal duration of the SDO mission, 2008 - 2014: full resolution line-of-sight magnetograms (1 / 45 seconds), full resolution dopplergrams (1 / 45 seconds), and full resolution filtergrams averaged over a 20 minute period (1 / 20 minutes). Uncompressed storage for these data products is currently estimated at ~33 TB per year, culminating in 198 TB total storage by 2014.

Architecture Model 1: Light Footprint

Data ProductInstrumentEstimated SizeEstimated CadenceStorage

The "light footprint" approach will provide a 30 TB online disc cache that will store HMI and AIA data products in export format (FITS, JPEG, VOTable, etc) only. Export files will be retrieved from a major SDO data centre in the US or Europe. This 30 TB will be divided between a 'true cache' for popular HMI and AIA datasets and a rolling 60 day systematic cache of AIA products. No tape archive will be required. The light footprint is currently favoured by the eSDO science advisory and JSOC teams.

AIA

AIA products will be held in a rolling 60 day cache; this will provide solar physicists with data from two most recent solar revolutions. Cached low level data will include low resolution full-disk images (8 per 10 seconds) along with high resolution images of tracked active regions (8 per 10 seconds). Several high level products generated at a much lower cadence will also be cached: thermal maps, DEM measures, irradiance estimates, and magnetic field extrapolations (1 to 10 per day). The storage requirement for a rolling 60 day cache is estimated at 11 TB.

Quicklook and Visualization

To aid visualisation and searching of the vast SDO archives, “quicklook” products such as thumbnail images will be generated for all low level HMI and AIA data held in the UK archive. The UK solar community has expressed interest in generating movies of this data; rather than storing pre-canned movies, a web service will generate movies of user-delimited start and end times from archived HMI or AIA thumbnail images. All thumbnails will be archived for the duration of the SDO mission, requiring an addition estimated 10 TB of space per year.

Synopsis of Storage Requirements

YearInstrumentStorage FacilityCumulative Size
08/2008 - 08/2009 HMI ADS tape archive (permanent) 33 TB
08/2009 - 08/2010 HMI ADS tape archive (permanent) 66 TB
08/2010 - 08/2011 HMI ADS tape archive (permanent) 99 TB
08/2011 - 08/2012 HMI ADS tape archive (permanent) 132 TB
08/2012 - 08/2013 HMI ADS tape archive (permanent) 165 TB
08/2013 - 08/2014 HMI ADS tape archive (permanent) 198 TB
08/2008 - 08/2009 AIA ADS disc cache (60 days) 11 TB
08/2009 - 08/2010 AIA ADS disc cache (60 days) 11 TB
08/2010 - 08/2011 AIA ADS disc cache (60 days) 11 TB
08/2011 - 08/2012 AIA ADS disc cache (60 days) 11 TB
08/2012 - 08/2013 AIA ADS disc cache (60 days) 11 TB
08/2013 - 08/2014 AIA ADS disc cache (60 days) 11 TB
08/2008 - 08/2009 Thumbnails ADS tape archive (permanent) 14 TB
08/2009 - 08/2010 Thumbnails ADS tape archive (permanent) 28 TB
08/2010 - 08/2011 Thumbnails ADS tape archive (permanent) 42 TB
08/2011 - 08/2012 Thumbnails ADS tape archive (permanent) 66 TB
08/2012 - 08/2013 Thumbnails ADS tape archive (permanent) 80 TB
08/2013 - 08/2014 Thumbnails ADS tape archive (permanent) 94 TB

Total storage requirements: 11 TB rolling cache, 47 TB / year permanent storage over 6 years (282 TB at the end of year 6)

  • UK Data Centre: light footprint:
    UK Data Centre: light footprint

Architecture Model 2: Heavy Footprint

The "heavy footprint" approach would provide a 300 TB disc cache that is a significant percentage of the US SDO disc cache size. This 300 TB cache would be interfaced to a UK instance of the DRMS and SUMS; entire storage units would be cached. Along with the DRMS and SUMS, the UK data centre would require software to extract export formats from storage units. This system would play active global role in storage unit management with other SDO data centres. This 300 TB will be divided between a 'true cache' for popular HMI and AIA datasets and a rolling 60 day systematic cache of AIA products. No tape archive will be required.

  • UK Data Centre: heavy footprint:
    UK Data Centre: heavy footprint

Architecture Model 3: Heavy Footprint with Tape Archive

This final "heavy footprint with tape archive" model is considered to be a fallback position if there is no major European SDO data centre and one is considered to be required. Similar to the "heavy" footprint described above, this approach would provide a 300 TB disc cache that is a significant percentage of the US SDO disc cache size. This 300 TB cache would be interfaced to a UK instance of the DRMS and SUMS; entire storage units would be cached. Along with the DRMS and SUMS, the UK data centre would require software to extract export formats from storage units. This system would play active global role in storage unit management with other SDO data centres. This 300 TB will be divided between a 'true cache' for popular HMI and AIA datasets and a rolling 60 day systematic cache of AIA products. In addition, HMI export format data products would be written to the ATLAS tape store to provide a permanent European helioseismology archive.

  • UK Data Centre: heavy footprint with tape archive:
    UK Data Centre: heavy footprint with tape archive

AIA and HMI Datasets

A number of level 1 and level 2 HMI science products will be available to users, including magnetograms, dopplergrams and continuum maps. Assuming that the "light" footprint data centre architectures is followed, HMI products will be held in export format on a disc cache following user requests. By contrast, if the "heavy" model is followed, then following user requests HMI products will be imported to the UK as JSOC storage units of ~20 GB. These storage units will be held in the large disc cache, and instances of DRMS and SUMS will be updated accordingly. Export formats of data products will be extracted from the storage units and returned to the user. Finally, if the "heavy with tape storage" architectural model is used, then HMI data will be systematically pulled from the JSOC archive and written to the ATLAS tape store inside JSOC storage units. Uncompressed storage for these data products is currently estimated at ~25 TB per year, culminating in 150 TB total storage by 2014.

AIA products will be held in a rolling 60 day cache; this will provide solar physicists with data from two most recent solar revolutions. Cached low level data will include low resolution full-disk images (8 per 10 seconds) along with high resolution images of tracked active regions (8 per 10 seconds). Several high level products generated at a much lower cadence will also be cached: thermal maps, DEM measures, irradiance estimates, and magnetic field extrapolations (1 to 10 per day). The storage requirement for a rolling 60 day cache is estimated at 11 TB.

InstrumentData ProductEstimated SizeEstimated CadenceStorage
HMI Line-of-sight magnetogram (full disk, full res) 15 MB 1 / 45 s cached on user request
HMI Vector magnetograms (tracked active region, full res) 3 MB 5 / 10 minutes cached on user request
HMI Averaged continuum maps (full disk, full res) 15 MB 1 / hour cached on user request
HMI Dopplergrams (full disk, full res) 20 MB 1 / 45 s cached on user request
AIA Images from 10 channels (full disk, full res) 15 MB(?) 8 / 10 s rolling 7 day cache
AIA Images from 10 channels (full disk, low res) ~1 MB(?) 8 / 10 s rolling 60 day cache
AIA Images from 10 channels (regions of interest, full res) ~1 MB(?) 40 / 10 s rolling 60 day cache
AIA Movies of regions of interest ~10 MB(?) 1 / day? rolling 60 day cache
AIA Other level 2 and 3 products (DEM, irradiance maps, etc) ~ 10MB? 10 - 20 / day? rolling 60 day cache

Integration Work

AstroGrid Deployment

The major tasks for integrating the UK eSDO centre with AstroGrid will be the deployment of the DataSet Access (DSA) and Commmon Execution Architecture (CEA) AstroGrid modules on a remote machine that can access data in the ATLAS storage facility. This development will be undertaken by MSSL early in Phase B in conjunction with work done to access Solar-B data also held at ATLAS. A relational database (MySQL) containing AIA and HMI data product metadata will reside on a remote machine. The DSA module will interface with this database, allowing a user to identify which data products are required. A request for the identified data products is sent to a CEA application on the same machine. The CEA application will issue the ATLAS commands necessary for data to be transferred from the ATLAS facility to the user's remote AstroGrid storage area, or "MySpace".

A number of test datasets will be placed in a disc cache at the ATLAS facility. Next, a MySQL database will be configured on the eSDO server at MSSL with sample metadata relating to the test datasets. Instances of DSA and CEA will be also be deployed on the eSDO server; the DSA will interface with the MySQL database and the CEA application will interface with ATLAS. Requested test datasets will be returned to an instance of the AstroGrid filestore and filemanager on the eSDO server.

Interface with JSOC Data Centre

Assuming that the "light footprint" architecture model is followed, export formatted SDO data products will need to be transferred from the JSOC data centre to the UK data center. In this model, when a UK user makes an SDO data request through the AstroGrid system, the request will first be sent to the UK data centre. If the required data is not present, the request will be redirected to the JSOC data centre. The required datasets will be exported back to the UK. The dataset will be cached in the UK system before a copy is passed to the user's MySpace area. In addition to user requests, the data centre will poll the JSOC system for new AIA data approximately twice an hour, and this data will be held in the UK cache for 60 days.

Development work will require a mechanism to poll the JSOC data centre new AIA data as well as a CEA application to pass user requests that cannot be fulfilled by the UK data centre to the JSOC system. This CEA application should cache the returned datasets in ATLAS, update the metadata accessible to the DSA deployed on the eSDO server, and pass the data on to the user's MySpace area.

US Data Centre

System

Detailed plans of the JSOC data centre and pipeline can be viewed at http://hmi.stanford.edu/doc/SOC_GDS_Plan/JSOC_GDS_Plan_Overview_CDR.pdf.

Archived Data

In addition to the HMI and AIA products listed above, a full description of archived and cached SDO products can be viewed at http://hmi.stanford.edu/doc/SOC_GDS_Plan/HMI_pipeline_JSOC_dataproducts.pdf.

Integration Work

AstroGrid Deployment

The Virtual Solar Observatory (VSO) will be the user front end to the JSOC data centre in the US. However, AstroGrid users may wish to incorporate a VSO search into an AstroGrid workflow that submits data to eSDO tools. Therefore, an AstroGrid to VSO interface will be developed using the CEA module. In addition, the JSOC data centre team is reviewing three AstroGrid components for use with their backend system. First, Karen Tian at Stanford University is investigating the DSA and CEA modules to enable data searching and data retrieval through the grid. Second, Rick Bogart has expressed interest in the AstroGrid workflow engine for use in driving JSOC pipeline execution.

The eSDO project will advise the JSOC team and aid development with these three AstroGrid components. As part of the Phase A research effort, Mike Smith has installed and configured the major AstroGrid components at MSSL: DSA, CEA, the workflow engine (JES), the filemanager / filestore, the registry and the portal. Documentation of this deployment is available to the solar community at http://www.mssl.ucl.ac.uk/twiki/bin/view/AstrogridInstallationV11, and it is also included as an appendix in the eSDO Phase A Report.

Network Latency Tests

Introduction

Network tests have been undertaken to establish baselines for network latency involved in data transfer under various protocols. This information is required for the specification of both the optimal sizes for the eSDO data cache elements and the overall architecture for an interoperating system involving the JSOC capabilities for efficient data management at the two (or more) sites. Each node represents a different use case for future SDO data requirements:

  1. Primary archive (Stanford)
  2. Secondary archive (RAL)
  3. Algorithm host machine (UCL)
  4. User machine (MSSL)

Test Design

Institutions

  • MSSL, msslxx.mssl.ucl.ac.uk, linux (fedora core 2)
  • UCL, zuserver1.star.ucl.ac.uk, linux (red hat)
  • RAL, tornado.badc.rl.ac.uk, dec alpha
  • Stanford, solarport.stanford.edu, linux (red hat)

Data

  • TRACE, T171_2004Jul15-16-X.mov, 1 Apple QuickTime movie file 1.21 GB
  • Yohkoh, 1522 SXT uncompressed FITS files totalling 1.2 GB
  • SOHO-CDS, 5003 compressed FITS files totalling 1.1 GB

Protocols

Three protocols were tested using ssh with PPK encryption for automated login. GridFTP will be tested between MSSL and Stanford at a later date.

  • scp
  • sftp
  • rsync

Scripts

The network test scripts used passphraseless public / private key encryption to automate file transfers with rsync, sftp and scp over ssh. Because of the security risks with passphraseless keys, MSSL and UCL firewall holes were restricted to specific named machines for the duration of the network tests. The sftp transfers were run in batch mode to allow multiple non-interactive file transfers At MSSL, Stanford and UCL the bash time utility was used in conjuction with the append and out facilities to log the user, system and elapsed time along with CPU usage for each rsync and scp files transfer. A single time was provided for all sftp transfers executed in a single batch.

Non-Interactive PPK

First, a set of DSA-2 public and private keys were generated on msslxx for each of the 4 test machines. Each of the machines stored its private key in the ~$user/.ssh directory, and the public key for each machine was distributed to all 4 machines' ~$user/.ssh directories.

  1. %ssh-keygen -b 1024 -t dsa
  2. %enter filename: $user_$institution_id_dsa
  3. %enter passphrase: hit enter for no passphrase
  4. %re-enter passphrase: hit enter again

Repeating this process on msslxx created a pair of public and private keys of the format $user_$institution_id_dsa and $user_$institution_id_dsa.pub for each of the 4 institutions involved. Once created, keys were transferred from msslxx to the appropriate machines.

Automated, Non-Interactive Scripts

RSYNC push

For MSSL, Stanford, and UCL machines:

echo "RSYNC PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_rsync.txt ; date >> ~$user/log/esdo_$institution_timelog_rsync.txt
for file in `cat ~$user/$instrumentfiles.txt`
do
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_rsync.txt rsync -e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa" $user/DataStore/$INSTRUMENT/$file $remoteuser@$remotemachine:/home/$remoteuser/TestData/$institution/.
done
ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa $remoteuser@$remotemachine << EOD1
cd ~$remoteuser/TestData/$institution/
rm -rf *.*
EOD1

For RAL machine:

echo "RSYNC PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_rsync.txt ; date >> ~$user/log/esdo_$institution_timelog_rsync.txt
echo "RSYNC PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_rsync.txt ; date >> ~$user/log/esdo_$institution_transferlog_rsync.txt
foreach file (`cat ~$user/$instrumentfiles.txt`)
( time rsync -e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa" $user/DataStore/$INSTRUMENT/$file $remoteuser@$remotemachine:$remoteuser/TestData/$institution/. >> ~$user/log/esdo_$institution_transferlog_rsync.txt ) >>& ~$user/log/esdo_$institution_timelog_rsync.txt
end
ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa $remoteuser@$remotemachine << EOD1
cd ~$remoteuser/TestData/$institution/
rm -rf *.*
EOD1

RSYNC pull

For MSSL, Stanford, and UCL machines:

echo "RSYNC PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_rsync.txt ; date >> ~$user/log/esdo_$institution_timelog_rsync.txt
for file in `cat ~$user/$instrumentfiles.txt`
do
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_rsync.txt rsync
-e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa"
$remoteuser@$remotemachine:/home/$remoteuser/TestData/$institution/$file
$user/TestData/$remoteinstitution/
done
cd ~$user/TestData/$remoteinstitution/
rm -rf *.*

For RAL machine:

echo "RSYNC PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_rsync.txt ; date >> ~$user/log/esdo_$institution_timelog_rsync.txt 
echo "RSYNC PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_transferlog_rsync.txt ; date >> ~$user/log/esdo_$institution_transferlog_rsync.txt
foreach file (`cat ~$user/$instrumentfiles.txt`)
( time rsync -e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa" $remoteuser@$remotemachine:$remoteuser/DataStore/$instrument/$file $user/TestData/$institution/. >> ~$user/log/esdo_$institution_transferlog_rsync.txt ) >>& ~$user/log/esdo_$institution_timelog_rsync.txt
end
cd ~$user/TestData/$remoteinstitution/
rm -rf *.*

SCP: push

For MSSL, Stanford, and UCL machines:

echo "SCP PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_scp.txt ; date >> ~$user/log/esdo_$institution_timelog_scp.txt
for file in `cat ~$user/$instrumentfiles.txt`
do
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_scp.txt scp -i ~$user/.ssh/esdo_$institutionhost_id_dsa ~$user/DataStore/$INSTRUMENT/$file $remoteuser@$remotemachine:@$user/TestData/$institution
done
ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa $remoteuser@$remotemachine << EOD
cd ~$remoteuser/TestData/$remoteinstitution/
rm -rf *.*
EOD

For RAL machine:

echo "SCP PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_scp.txt ; date >> ~$user/log/esdo_$institution_timelog_scp.txt
echo "SCP PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_transferlog_scp.txt ; date >> ~$user/log/esdo_$institution_transferlog_scp.txt
foreach file (`cat ~$user/$instrumentfiles.txt`)
( time scp -e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa" $user/DataStore/$instrument/$file $remoteuser@$remotemachine:$remoteuser/TestData/$institution/. >> ~$user/log/esdo_$institution_transferlog_scp.txt ) >>& ~$user/log/esdo_$institution_timelog_scp.txt
end
ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa $remoteuser@$remotemachine << EOD1
cd ~$remoteuser/TestData/$institution/
rm -rf *.*
EOD1

SCP: pull

For MSSL, Stanford, and UCL machines:

echo "SCP PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_scp.txt ; date >> ~$user/log/esdo_$institution_timelog_scp.txt
for file in `cat ~$user/$instrumentfiles.txt`
do
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_scp.txt scp -i ~$user/.ssh/esdo_$institutionhost_id_dsa ~$user/DataStore/$INSTRUMENT/$file $remoteuser@$remotemachine:@$user/TestData/$institution/$file ~$user/TestData/$remoteinstitution/ 
done
cd ~$user/TestData/$remoteinstitution/
rm -rf *.*

For RAL machine:

echo "SCP PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_scp.txt ; date >> ~$user/log/esdo_$institution_timelog_scp.txt
echo "SCP PULL: $institution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_transferlog_scp.txt ; date >> ~$user/log/esdo_$institution_transferlog_scp.txt
foreach file (`cat ~$user/$instrumentfiles.txt`)
( time scp -e "ssh -i ~$user/.ssh/esdo_$institutionhost_id_dsa" $remoteuser@$remotemachine:$remoteuser/DataStore/$instrument/$file $user/TestData/$remoteinstitution/ >> ~$user/log/esdo_$institution_transferlog_scp.txt ) >>& ~$user/log/esdo_$institution_timelog_scp.txt
end
cd ~$user/TestData/$remoteinstitution/
rm -rf *.*

SFTP: push

For MSSL, Stanford, and UCL machines:

echo "SFTP PUSH: $institution TO $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_sftp.txt ; date >> ~$user/log/esdo_$institution_timelog_sftp.txt
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_sftp.txt sftp -o 'IdentityFile ~$user/.ssh/esdo_$institutionhost_id_dsa' -b ~$user/sftp_push_$instrument.batch $remoteuser@$remotemachine:$user/TestData/$institution
ssh -i ~$remoteuser/.ssh/esdo_$remoteinstitutionhost_id_dsa $remoteuser@$remotemachine << EOD1
cd $remoteuser/TestData/$institution/
rm *.*
EOD1

For RAL machine:

echo "SFTP PUSH: $institution TO $remoteinstitution: $instrument, #" >> ~$user/log/esdo_$institution_timelog_sftp.txt ; date >> ~$user/log/esdo_$institution_melog_sftp.txt
echo "SFTP PUSH: $institution TO $remoteinstitution: $instrument, #" >> ~user/log/esdo_$institution_transferlog_sftp.txt ; date >> ~user/log/esdo_$institution_transferlog_sftp.txt
( time sftp -o 'IdentityFile ~user/.ssh/esdo_$institutionhost_id_dsa' -b ~$user/sftp_push_$instrument.batch $remoteuser@$remotemachine:$remoteuser/TestData $institution >> ~$user/log/esdo_$institution_transferlog_sftp.txt ) >>& ~$user/log/esdo_$institutionimelog_sftp.txt
ssh -i ~$user/.ssh/esdo_$institutehost_id_dsa $remoteuser@$remotemachine << EOD1
cd $remoteuser/TestData/$institution/
rm *.*
EOD1

SFTP push batch files:

Text file listing get command plus one file per line (no path information). For example:

get s100r00.fits.Z

SFTP: pull

For MSSL, Stanford, and UCL machines:

cd ~$user/TestData/$remoteinstitution/
echo "SFTP PULL: $insitution FROM $remoteinstitution: $INSTRUMENT, #" >> ~$user/log/esdo_$institution_timelog_sftp.txt ; date >> ~$user/log/esdo_$institution_timelog_sftp.txt
/usr/bin/time -a -o ~$user/log/esdo_$institution_timelog_sftp.txt sftp -o 'IdentityFile ~$user/.ssh/esdo_$institutionhost_id_dsa' -b ~$user/sftp_pull_$instrument.batch $remoteuser@$remotemachine:DataStore/$INSTRUMENT
rm ~$user/TestData/$remoteinstitution/*.*

For RAL machine:

echo "SFTP PULL: $institution FROM $remoteinstitution: $instrument, #" >> ~$user/log/esdo_$institution_timelog_sftp.txt ; date >> ~$user/log/esdo_$institution_melog_sftp.txt
echo "SFTP PULL: $institution FROM $remoteinstitution: $instrument, #" >> ~user/log/esdo_$institution_transferlog_sftp.txt ; date >> ~user/log/esdo_$institution_transferlog_sftp.txt
( time sftp -o 'IdentityFile ~user/.ssh/esdo_$institutionhost_id_dsa' -b ~$user/sftp_push_$instrument.batch $remoteuser@$remotemachine:$remoteuser/DataStore/$instrument >> ~user/log/esdo_$institution_transferlog_sftp.txt ) >>& ~$user/log/esdo_$institutionimelog_sftp.txt
cd $user/TestData/$remoteinstitution/
rm *.*

SFTP pull batch files:

Text file listing put command plus one file per line (full local path information). For example:

put /home/eca/DataStore/CDS/s100r00.fits.Z

Results

Test Matrix

PROTOCOLPUSH/PULLINSTITUTIONREMOTE INSTITUTIONINSTRUMENTKB/SECELAPSED SECSIZE NUMBER OF TRIALS
RSYNC PULL MSSL RAL CDS        
RSYNC PULL MSSL RAL TRACE 4336.470601 289.21 1211107245 2
RSYNC PULL MSSL RAL Yohkoh        
RSYNC PULL MSSL RAL CDS        
RSYNC PULL MSSL STANFORD TRACE 401.1485084 2948.34 1211107245 2
RSYNC PULL MSSL RAL Yohkoh        
RSYNC PULL MSSL RAL CDS        
RSYNC PULL MSSL UCL TRACE 3194.875417 370.51 1211107245 2
RSYNC PULL MSSL RAL Yohkoh        
RSYNC PULL RAL MSSL CDS 610.577447 2059.333333 1149635736 3
RSYNC PULL RAL MSSL TRACE 2223.200026 549.25 1211107245 4
RSYNC PULL RAL MSSL Yohkoh 1060.68955 1311.666667 1271676928 3
RSYNC PULL RAL STANFORD CDS 72.37772289 15514 1149635736 3
RSYNC PULL RAL STANFORD TRACE 194.2172718 6912.5 1211107245 4
RSYNC PULL RAL STANFORD Yohkoh 203.7822694 10757 1271676928 4
RSYNC PULL RAL UCL CDS 651.5660324 1882.916667 1149635742 5
RSYNC PULL RAL UCL TRACE 3777.983018 306 1211107245 4
RSYNC PULL RAL UCL Yohkoh 4859.713795 297.875 1271676928 5
RSYNC PULL STANFORD MSSL CDS 35.83666908 31328 1149635757 1
RSYNC PULL STANFORD MSSL TRACE 392.0841767 3016.5 1211107245 1
RSYNC PULL STANFORD MSSL Yohkoh        
RSYNC PULL STANFORD RAL CDS        
RSYNC PULL STANFORD RAL TRACE 187.9424629 6293 1211107245 1
RSYNC PULL STANFORD RAL Yohkoh        
RSYNC PULL STANFORD UCL CDS 52.50145758 21384 1149635757 1
RSYNC PULL STANFORD UCL TRACE        
RSYNC PULL STANFORD UCL Yohkoh 130.6821004 9503 1271676928 1
RSYNC PULL UCL MSSL CDS 523.4064667 2144.97 1149635757 1
RSYNC PULL UCL MSSL TRACE        
RSYNC PULL UCL MSSL Yohkoh        
RSYNC PULL UCL RAL CDS 335.2317614 3349 1149635757 1
RSYNC PULL UCL RAL TRACE        
RSYNC PULL UCL RAL Yohkoh        
RSYNC PULL UCL STANFORD CDS        
RSYNC PULL UCL STANFORD TRACE        
RSYNC PULL UCL STANFORD Yohkoh        
RSYNC PUSH RAL MSSL CDS 482.6726017 2468.166667 1072275345 6
RSYNC PUSH RAL MSSL TRACE 5892.994284 299.5 1211107245 4
RSYNC PUSH RAL MSSL Yohkoh 3874.416717 584.75 1271676928 4
RSYNC PUSH RAL STANFORD CDS 75.44878282 15909 1149635736 3
RSYNC PUSH RAL STANFORD TRACE 201.9613884 5856.333333 1211107245 3
RSYNC PUSH RAL STANFORD Yohkoh 139.8397937 9456.75 1271676928 4
RSYNC PUSH RAL UCL CDS 1219.858809 950.3333333 1149635736 3
RSYNC PUSH RAL UCL TRACE 5322.647664 223.5 1211107245 4
RSYNC PUSH RAL UCL Yohkoh 3394.940724 563.6666667 1271676928 3
RSYNC PUSH STANFORD MSSL CDS 48.52608075 23141.33333 1149635757 3
RSYNC PUSH STANFORD MSSL TRACE 498.286099 2373.58 1211107245 1
RSYNC PUSH STANFORD MSSL Yohkoh        
RSYNC PUSH STANFORD UCL CDS 41.36666061 27140 1149635757 1
RSYNC PUSH STANFORD UCL TRACE 374.7855231 3155.73 1211107245 1
RSYNC PUSH STANFORD UCL Yohkoh        
RSYNC PUSH STANFORD RAL CDS 45.06655027 24912 1149635757 2
RSYNC PUSH STANFORD RAL TRACE        
RSYNC PUSH STANFORD RAL Yohkoh        
RSYNC PUSH UCL MSSL CDS 483.1439128 2328.5 1149635757 2
RSYNC PUSH UCL MSSL TRACE 1703.424817 694.32 1211107245 1
RSYNC PUSH UCL MSSL Yohkoh 1359.346746 913.58 1271676928 1
RYSNC PUSH UCL RAL CDS 334.5325295 3356 1149635757 1
RYSNC PUSH UCL RAL TRACE 5331.178359 221.85 1211107245 1
RYSNC PUSH UCL RAL Yohkoh        
RSYNC PUSH UCL STANFORD CDS        
RSYNC PUSH UCL STANFORD TRACE 392.0036057 3017.12 1211107245 1
RSYNC PUSH UCL STANFORD Yohkoh        
RSYNC PUSH MSSL RAL CDS 258.9123048 4399.565 1149635757 2
RSYNC PUSH MSSL RAL TRACE 1864.27901 659.47 1211107245 5
RSYNC PUSH MSSL RAL Yohkoh        
RSYNC PUSH MSSL STANFORD CDS        
RSYNC PUSH MSSL STANFORD TRACE 366.5222661 3443.943333 1211107245 6
RSYNC PUSH MSSL STANFORD Yohkoh        
RSYNC PUSH MSSL UCL CDS        
RSYNC PUSH MSSL UCL TRACE 3208.935466 371.088 1211107245 5
RSYNC PUSH MSSL UCL Yohkoh        
SCP PULL MSSL RAL CDS        
SCP PULL MSSL RAL TRACE        
SCP PULL MSSL RAL Yohkoh        
SCP PULL MSSL STANFORD CDS        
SCP PULL MSSL STANFORD TRACE        
SCP PULL MSSL STANFORD Yohkoh        
SCP PULL MSSL UCL CDS        
SCP PULL MSSL UCL TRACE        
SCP PULL MSSL UCL Yohkoh        
SCP PULL RAL MSSL CDS 550.9280575 2826.6 1149435740 5
SCP PULL RAL MSSL TRACE        
SCP PULL RAL MSSL Yohkoh 2009.796939 1108 1271676928 4
SCP PULL RAL STANFORD CDS 84.18699449 14008.33333 1149635736 3
SCP PULL RAL STANFORD TRACE        
SCP PULL RAL STANFORD Yohkoh 266.6911728 6820.25 1271676928 4
SCP PULL RAL UCL CDS 668.0376442 1794.8 1149635736 5
SCP PULL RAL UCL TRACE        
SCP PULL RAL UCL Yohkoh 2873.355229 462.5 1271676928 4
SCP PULL STANFORD MSSL cds        
SCP PULL STANFORD MSSL TRACE 384.8102238 3073.52 1211107245 1
SCP PULL STANFORD MSSL Yohkoh        
SCP PULL STANFORD RAL cds        
SCP PULL STANFORD RAL TRACE 191.8961779 6186.5 1211107245 2
SCP PULL STANFORD RAL Yohkoh        
SCP PULL STANFORD UCL cds        
SCP PULL STANFORD UCL TRACE 291.0931916 4471.87 1211107245 3
SCP PULL STANFORD UCL Yohkoh        
SCP PULL UCL MSSL CDS 70.3512571 15010 1081315706 1
SCP PULL UCL MSSL TRACE        
SCP PULL UCL MSSL Yohkoh        
SCP PULL UCL RAL cds        
SCP PULL UCL RAL TRACE        
SCP PULL UCL RAL Yohkoh        
SCP PULL UCL STANFORD cds        
SCP PULL UCL STANFORD TRACE        
SCP PULL UCL STANFORD Yohkoh        
SCP PUSH MSSL RAL CDS 1514.116472 4771.123333 1149635757 3
SCP PUSH MSSL RAL TRACE 1936.348918 610.8 1211107245 1
SCP PUSH MSSL RAL Yohkoh        
SCP PUSH MSSL STANFORD CDS 43.50101506 25808.39 1149635750 1
SCP PUSH MSSL STANFORD TRACE 3016.4 3016.4 1211107245 2
SCP PUSH MSSL STANFORD Yohkoh        
SCP PUSH MSSL UCL CDS 231.5858438 24163.805 1152429578 1
SCP PUSH MSSL UCL TRACE        
SCP PUSH MSSL UCL Yohkoh        
SCP PUSH RAL MSSL CDS 462.8397723 2993.6 1149635744 5
SCP PUSH RAL MSSL TRACE        
SCP PUSH RAL MSSL Yohkoh 1879.278741 698.25 1271676928 4
SCP PUSH RAL STANFORD CDS 83.86395588 14217.6 1149635744 5
SCP PUSH RAL STANFORD TRACE        
SCP PUSH RAL STANFORD Yohkoh 102.090398 12388 1271676928 4
SCP PUSH RAL UCL CDS 580.3874277 2054.333333 1149635736 3
SCP PUSH RAL UCL TRACE        
SCP PUSH RAL UCL Yohkoh 3678.889365 354.75 1271676928 4
SCP PUSH STANFORD MSSL CDS 47.34337475 23931.5 1154648019 2
SCP PUSH STANFORD MSSL TRACE 271.5666778 4447 1211107245 3
SCP PUSH STANFORD MSSL Yohkoh        
SCP PUSH STANFORD RAL CDS 45.3153481 24810.5 1149635747 2
SCP PUSH STANFORD RAL TRACE 177.5547844 6747 1211107245 3
SCP PUSH STANFORD RAL Yohkoh        
SCP PUSH STANFORD UCL CDS        
SCP PUSH STANFORD UCL TRACE 278.2749051 4827.15 1211107245 3
SCP PUSH STANFORD UCL Yohkoh        
SCP PUSH UCL MSSL CDS        
SCP PUSH UCL MSSL TRACE        
SCP PUSH UCL MSSL Yohkoh        
SCP PUSH UCL RAL CDS 319.5000338 3513.9 1149635757 2
SCP PUSH UCL RAL TRACE        
SCP PUSH UCL RAL Yohkoh        
SCP PUSH UCL STANFORD CDS 54.34879542 20657.15 1149635754 1
SCP PUSH UCL STANFORD TRACE        
SCP PUSH UCL STANFORD Yohkoh        
SFTP PULL MSSL RAL CDS        
SFTP PULL MSSL RAL TRACE        
SFTP PULL MSSL RAL Yohkoh        
SFTP PULL MSSL STANFORD CDS        
SFTP PULL MSSL STANFORD TRACE        
SFTP PULL MSSL STANFORD Yohkoh        
SFTP PULL MSSL UCL CDS        
SFTP PULL MSSL UCL TRACE        
SFTP PULL MSSL UCL Yohkoh        
SFTP PULL RAL MSSL CDS 1145.359972 984.3333333 1149635736 6
SFTP PULL RAL MSSL TRACE 2431.7182 490.2 1211107245 5
SFTP PULL RAL MSSL Yohkoh 1677.521889 759.6666667 1271676928 3
SFTP PULL RAL STANFORD CDS 79.77291835 14108.16667 1149635736 6
SFTP PULL RAL STANFORD TRACE 192.5336876 6186.714286 1211107245 7
SFTP PULL RAL STANFORD Yohkoh 138.6082475 8985.6 1271676928 5
SFTP PULL RAL UCL CDS 1744.604418 644.75 1149635736 4
SFTP PULL RAL UCL TRACE 3737.637272 318.3333333 1211107245 3
SFTP PULL RAL UCL Yohkoh 2416.093385 514 1271676928 1
SFTP PULL STANFORD MSSL CDS        
SFTP PULL STANFORD MSSL TRACE        
SFTP PULL STANFORD MSSL Yohkoh        
SFTP PULL STANFORD RAL CDS        
SFTP PULL STANFORD RAL TRACE        
SFTP PULL STANFORD RAL Yohkoh        
SFTP PULL STANFORD UCL CDS        
SFTP PULL STANFORD UCL TRACE        
SFTP PULL STANFORD UCL Yohkoh        
SFTP PULL UCL MSSL CDS        
SFTP PULL UCL MSSL TRACE 2078.524338 569.02 1211107245 1
SFTP PULL UCL MSSL Yohkoh        
SFTP PULL UCL RAL CDS        
SFTP PULL UCL RAL TRACE 5783.198469 204.51 1211107245 1
SFTP PULL UCL RAL Yohkoh        
SFTP PULL UCL STANFORD CDS        
SFTP PULL UCL STANFORD TRACE        
SFTP PULL UCL STANFORD Yohkoh        
SFTP PUSH MSSL RAL CDS        
SFTP PUSH MSSL RAL TRACE 1674.25054 725.5314286 1211107245 7
SFTP PUSH MSSL RAL Yohkoh        
SFTP PUSH MSSL STANFORD CDS        
SFTP PUSH MSSL STANFORD TRACE 232.9840935 5078.25 1211107245 4
SFTP PUSH MSSL STANFORD Yohkoh        
SFTP PUSH MSSL UCL CDS        
SFTP PUSH MSSL UCL TRACE 2885.151698 411.8 1211107245 2
SFTP PUSH MSSL UCL Yohkoh        
SFTP PUSH RAL MSSL CDS 3782.20674 801.3333333 1149635736 6
SFTP PUSH RAL MSSL TRACE 2642.108355 456.4 1211107245 5
SFTP PUSH RAL MSSL Yohkoh 1773.040648 702.4 1271676928 5
SFTP PUSH RAL STANFORD CDS 1692.597933 3855.25 1149635736 8
SFTP PUSH RAL STANFORD TRACE 559.6286682 5137.571429 1211107245 7
SFTP PUSH RAL STANFORD Yohkoh 1070.325022 3999.357143 1271676928 14
SFTP PUSH RAL UCL CDS 1613.334598 696.6666667 1149635757 3
SFTP PUSH RAL UCL TRACE 4927.278178 261 1211107245 4
SFTP PUSH RAL UCL Yohkoh 2444.667814 508 1271676928 2
SFTP PUSH STANFORD MSSL CDS        
SFTP PUSH STANFORD MSSL TRACE 228.3963692 5178.5 1211107245 4
SFTP PUSH STANFORD MSSL Yohkoh        
SFTP PUSH STANFORD RAL CDS 141.3325448 8065.5 1149635757 6
SFTP PUSH STANFORD RAL TRACE 197.0703629 6004 1211107245 4
SFTP PUSH STANFORD RAL Yohkoh 152.6766658 8134 1271676928 1
SFTP PUSH STANFORD UCL CDS        
SFTP PUSH STANFORD UCL TRACE 215.378449 5499.5 1211107245 3
SFTP PUSH STANFORD UCL Yohkoh        
SFTP PUSH UCL MSSL CDS        
SFTP PUSH UCL MSSL TRACE 1835.682408 652.87 1211107245 2
SFTP PUSH UCL MSSL Yohkoh        
SFTP PUSH UCL RAL CDS        
SFTP PUSH UCL RAL TRACE 5412.010797 220.5325 1211107245 4
SFTP PUSH UCL RAL Yohkoh        
SFTP PUSH UCL STANFORD CDS        
SFTP PUSH UCL STANFORD TRACE 239.9500845 4935 1211107245 4
SFTP PUSH UCL STANFORD Yohkoh        

-- ElizabethAuden - 23 Aug 2005

I Attachment History Action Size Date Who Comment
GIFgifeSDO_DC_heavy.gif r1 manage 13.7 K 2005-09-29 - 17:51ElizabethAuden UK Data Centre: heavy footprint
GIFgifeSDO_DC_heavytape.gif r1 manage 14.6 K 2005-09-29 - 17:51ElizabethAuden UK Data Centre: heavy footprint with tape archive
GIFgifeSDO_DC_light.gif r1 manage 13.0 K 2005-09-29 - 17:51ElizabethAuden UK Data Centre: light footprint

View topic | View difference side by side | History: r7 < r6 < r5 < r4 | More topic actions
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback