Tags:
create new tag
view all tags

Odyssey

Overview

Odyssey has been developed here at MSSL as a generic Test Framework. Currently its main use is as the test framework for the Gaia project. CNES (in France) have specified a particular Facade architecture for use by CU6, CU4, and CU8. This Facade architecture is the main input to the CNES SAGA system. It is also the input to the Odyssey framework. The reason for Odyssey's existence is that the SAGA framework will not be locally accessible, so another similar Framework needed to be developed. The current Testbed framework, located in SVN tagged code, is, in practice, always in a state of flux. Also everything is inextricably mixed in with the Test classes, thus making makes integration tests harder. Odyssey, using a description in an XML file, should make integration tests easier, because it separates out how the interfaces, and other logic, are called. Odyssey is here to help develop your tests at both module and integration level. For the most part it does what SAGA can do when dealing with chaining of Facades and their associated inputs and outputs. There are, however, some differences between SAGA and Odyssey - see the FAQ.

Odyssey works by reading in a particular XML file that contains chain(s). Inside a single chain will be one or many 'Tests' (these you can consider synonymous with a Facade). It is important to note that these tests are, by default, read like a sequence and the tests are called in that sequence. So, for example, using the alphabet as test designators, you can assume: A->B->C....->Z. However, with the complex nature of Gaia, you cannot expect tests to be called one after another. Hence many features are built in so you can 'goto' other tests, thus allowing you to skip forwards or backwards. See the 'Feature' list of Odyssey below. See also 'How to Download', 'How to Install' and 'XML Description' below. Some links go to other wiki pages when necessary.

See the 'I want to run the Pipelines now' if you just want to jump right in and start running the CU6 pipelines. MSSL will create the initial starting point to begin running the pipeline. The client 'you' can download the needed data and begin running pipelines and analysing the results.

Features

  • Reads in Data Models from Ascii or JDBC connection
  • Special 'Logic' constructs ('probably' like Technical Modules in Saga). These are explained in more detail later.
    • Logic allows goto's to skip forward or backwards between tests. It can also tell Odyssey to do the skipping immediately before or after the current Facade.
    • Logic allows you to 'END' the Chain, because certain Sequences may have multiple 'END's'.
    • Logic can be applied to BEGIN, FIRST, MID, POSTPRECONTROL, POSTCALL, POSTPOSTCONTROL, LAST, END parts of a Test(Facade); see 'Order of Calls' and Logic xml page below.
    • Logic can be used for just about anything you like, pure java with access to the data. Such as plotting or at times possible fudges are needed.
  • Outputs of one Test can be supplied to any other test in the chain.
  • Different Inputs to a Test or Facade can be done by detection of previous tests.
  • Writes of Data Models to ASCII files or JDBC. (Also can write outputs to GBIN format)
  • A personal Storage(Hashtable) is available to store particular objects to help in your testing. Examples of use are described later.
  • Non-Data Model (DM) objects are stored in the 'Storage' Hashtable and logged after each test.

Requirements

  • a gaiatools jar/library (version should not matter)
  • a $DPCCCOMMON jar/lib 'dpcc-??.jar' version greater than 6.1,.
  • Your DM jar/lib(s). There is no preference on version or whether they are sourced from commonCU6/MDB/DPC. Any DM interfaces that use 'GaiaRoot' (which they all do) are acceptable.
  • Property files if you use them in your algorithms, also a Property file that defines your connections and FileStore such as JDBC.
  • Finally the actual Module jar/lib(s). However, this may be automatic depending on how you wish to run Odyssey.
  • If you use ASCII input files, then you have to make sure you have 'input' and 'output' directories under your main FileStore property location for CU6 modules - this tends to be the 'data' directory.
  • Including XML files have now been enabled. Unfortunately GaiaTools and TMTools (if you use as dependencies) are packaging an old xml parser that is not needed. Investigating why they have done this. Be sure in your ivy.xml to do a (for GaiaTools and TMTools):

I want to run the Pipelines now

Below is extra information about Odyssey and setting it up to run on your own code base. If you simply want to get into running the CU6 chains such as PreProcessing, Extraction, Calibration, STA, and MTA then look at Gaia Odyssey Pipeline. This wiki page will set you up to run the pipelines and begin analysing results. I would still advise having a brief read through on the information below about Odyssey in particular the area about Logic classes you would find useful to do special enhancements on the pipeline and the order of how calls are made to the facades.

Download

Download of Odyssey : You will find the Odyssey library below as an attachment.

How to Run

Odyssey, much like other libraries(jars) such as GaiaTools or dpcc, can be integrated directly into your Module. Alternatively, because you will be doing Integration tests on multiple modules, you could create another project/module such as, for example, ExtractionFramework. You can then more easily incorporate other module(s) jar(s)/lib(s) into this new project.

Here are the steps of how to run Odyssey.

  • Download and place the Odyssey.jar into your lib or classpath much like you do other libraries.
  • Be sure to place all your needed libraries on your classpath as stated in the requirements.
  • Place the property files needed in a conf directory.
  • If you use Ascii files, be sure to have the needed input files for tests in the 'input' directory. Note again that Odyssey will be able to use outputs of other tests as inputs to another.
  • Now you need to create the xml file that describes all your test(s). This is described in more detail on another wiki page - see below. This xml file should be placed in a sub-directory of your basedir. Lots of samples used: xml/odyssey.xml (but you can use a different directory than 'xml', or a different file name).
  • Now you can call Odyssey via the command line as an ant task, or a Java file such as a Junit test or TestBed. Next line shows you the 'Java' way to call it which can be done in one line (though see the Ant below if you don't even want to deal with writing a small java file or using your testbed):
    • new mssl.ucl.odyssey.Odyssey().beginJourney(file); //The file is a String reference such as "xml/odyssey.xml"
    • new mssl.ucl.odyssey.Odyssey().beginJourney(file,true,propertyFile,skipTests,skipLogic);
      • true - is to clean tables in the database normally true.
      • propertyFile - property file to load. Normally in the conf directory and Normally is your database connection property file.
      • skipTests - comma separated list of Tests you wish to skip and not bother running. Must match 'id' of a i.e. Run FullExtraction but skip PBG and Deblending.
      • skipLogic - comma separated list of Logic classes. Currently must be the full classname of the Logic class.
  • That is all. You may place multiple chain(s) in one xml file, or call Odyssey on multiple xml files using the same one line java as above.
  • One step above that might be missing is the 'Logic'. This is described below. It is an optional piece of Java that you may write to do special analysis or decision making.

In the attachment area you will find a build.xml that can be used or modified for your own pipeline runs.

Example of running FullExtraction and connecting to a properties file to a different dataset. (This will probably become the norm):
ant -Dfilename=xml/FullExtractionChain.xml -DpropertyFile=conf/gaiae2e_corrected.properties file

Order of Calls

So how is your Facade very similar to SAGA the order is described here, Note Odyssey is now in a batch mode for writing results by default, you must use one of the 'dbWrite' elements i.e ADDONLY as part the 'oport' to turn it off. See oport section near the bottom ofOdyssey XML page.:
  • Logic (BEGIN) if any this is optional
  • Loading of Param Data
  • Call Facade.init()
  • Loop (if needed starts here for facades that take single DM but loaded multiple DM's)
    • Logic (FIRST) if any this is optional
    • Loading of Input Data
    • Logic (MID) if any this is optional
    • Call Facade.preControl()
    • Logic (POSTPRECONTROL) if any this is optional
    • Call Facade.call()
    • Logic (POSTCALL) if any this is optional
    • Call Facade.postControl()
    • Logic (POSTPOSTCONTROL) if any this is optional
    • Write DM type outputs, if 'not' in batch mode.
    • Log Non-DM Storage Hashtable.
    • Logic (LAST) if any this is optional
  • Loop (end)
  • Write DM type outputs, if in batch mode
  • Logic (END) if any this is optional

Storage Table

You will see in both the XML and Logic lots of text about 'Storage'. Storage is nothing more than a hashtable. As it is a generic Hashtable, it can contain any kind of object. It has two main purposes.
  • NON-DM objects that are outputs of your Facade are stored in this Storage object and written to logs later. These objects can also be accessed by you via the XML file or Logic classes.
  • Personal Use: you may wish to store an object, string, integer to be used later in Tests, or just printed out to the logs. A good example is to store a small loop integer that you may use later in the XML or Logic classes. Or what about a time value or set of transitid's? This could allow you, with the help of some Logic classes, to increment the variable and then, via a loop back through your tests, you could do a specific SQL query based on this Storage.
  • You will see from xml files a handy putInStorage tag that will initially put values into the Storage table. NOTE these are string values placed into the hashtable. So if you refer to them in a Logic you might have to typecase (valueOf(string)) to an integer, long, etc...

The XML

The XML file that Odyssey uses is the most important input, because it describes all your Tests/Facades, inputs, outputs, how you want to change the inputs and outputs, what tables or ascii files to use. Essentially everything is described in this XML. Probably after the first piece of XML you write you will find it easier and easier to produce the XML about your Tests. To go over every piece of XML along with sample attachments to help understand the xml, a dedicated wiki page is here:

Odyssey XML. Note the wiki page is quite verbose describing all elements but see attachments - you will notice it is not that difficult.

Logic and Gotos

Logic is an 'optional' piece of Java that must have a particular method defined. It extends an Abstract class defined in the Odyssey library. This Logic lets you have access to your Facade, as well as a Storage Hashtable, that you can use to your advantage. All this will help you do looping, or possibly construct certain SQL queries based on time, and much more. It also allows you to 'END' the Chain in case you inspect the data and decide an error has happened and it is not worth continuing, or possibly this was a successful end point. It also has goto methods (in the Abstract class) that can be called that allow you to skip to some other test in the chain. You can define as many Logic classes as needed for a Test and you may elect to do special things such as Plotting data or writing out other info. Again I thought this needs special attention so gave it it's own wiki page here:

Odyssey Logic

FAQ

Common questions you might have. FAQ

Tips

  • Use the DummyFacade: 'mssl.ucl.odyssey.commonfacade.gaia.DummyFacade' You will find this DummyFacade which does nothing quite useful. The primary purpose is to use Logic classes normally 'MID' to interrogate inputs (or outputs).
  • PASS_THRU: A new 'type' can be done on your 'port' which simply takes your input and set it both as inputs and output. Found this most useful when making a gbin file.
  • See the new DumpToGbin.xml file which shows you how to write a GBIN and uses both the above points. Attached in: Odyssey XML Or svn Framework/xml
  • LoopLogic: As the datasets keep getting bigger and bigger we can't hold everything in memory. In comes a 'TransitLoop' Logic class. See greater detail here: Odyssey LoopLogic

Releases

Jan 17th 2014

This version has a few fixes since it March 2013. Namely Odyssey catches a 'Throwable' instead of 'Exception' in the main area, this is so we can log issues that are not Exceptions. Then a few fixes on for writing gbins.

March 13th 2013

The speed efficiency gained on the new Odyssey released on February 12th created a bug on Array and Collection outputs that were not being saved. This bug is now fixed. It was also discovered on the new 14.0.0 release of GaiaTools that a particular setProperty method changed causing the method signatures to be incorrect and causing an issue. The Odyssey.jar is for 14.0.0 releases and later, and a Pre14.0.0.jar release is below for older releases. It should be noted the Pre14.0.0 does not have a batch mode enabled. Finally a TESTONLY and TESTSKIPPED property can be read to target particular tests that need running instead of every test in a chain more details on this new feature can be found on reading about the new full pipeline Gaia Odyssey Pipeline.

February 12th 2013

The February 12th 2013 release is still backwards compatible with all Odyssey releases and xml files. It has one major significant change, by default Odyssey will now run in a batch mode for writing data models. For Example: Lets Assume a Facade has 1 input and 1 output. Odyssey though has 15,000 inputs to go through. Odyssey would loop 15,000 times calling the Facade and writing 15,000 individual times to the database. This approach though more memory efficient, was very slow because of the constant database connections. Most facades seem to be written in this particular way. Odyssey is now in a batch mode and will commit all 15,000 results after all the looping. This technique takes up much more memory to save results then committing at the end, but has a huge speed increase. If needed you can still turn off batching by using a 'dbWrite' with ADDONLY in the oport, See the 'oport' sections near the bottom in the Odyssey XML page.

March 2nd 2011

The March 2nd 2011 release is still backwards compatible with all Odyssey releases and xml files, but it made several available options to help the user construct xml files. And has significant updates. One of the major problems has been big XML files that get hard to read and several XML files being created that are 'almost' exactly the same just a property file is different or sometimes just one 'Test' is being skipped. This release allows has several of these problems fixed.
  • See the Sample 'How to Run' ant tasks above, a New one has been created to allow extra arguments. You will notice 3 new arguments that can be ran. The 2 main ones are:
    • skipTests - This allows you to skipTests as if they were never in the chain. Odyssey simply does not add the Tests to its 'pipeline' of tests to be ran. It is comma separated and matches the 'id' tag of the Tests.
    • propertyFile - Allows you to load a properties file via the command line. This is normally a property file to another dataset or database.
  • XInclude - XML includes are now available to do this see the FullExtractionChain.xml attached in Odyssey XML. It requires two thins in your xml file:
    • At the top in your first element have a xmlns:xi attribute i.e.:
    • then when your ready to include the file do:
    • Be sure in your ivy.xml if you use GaiaTools or TMTools to add the () so an old xml parser does not get added to your lib directory.
  • skipLogic - is also available, but rare to use. It involves skipping Logic classes defined in your XML. Such as Debug type logic classes you would not want to run. It is comma separated and must be the long classname defined as your logic class.
    • An added ability to a logic class is to pass properties, If a property 'ALWAYSRUN' is set then this logic class is still ran even if defined in your skipLogic argument ie:
       <logic type="MID" className="mssl.ucl.odyssey.commonlogic.gaia.DBLoopLogic" prop="ALWAYSRUN" />
      This was added so the user can skip the Logic class in Tests where it is not important, but still use it for other Tests.

Finally it is getting more common to define global type entries into the 'storage' area at the top of an xml file. Originally you were only allowed 1 <putInStorage .. element to define all your key,value pairs. Making a very very long 'putInStorage' element, this has now been changed to allow many 'putInStorage' elements in the same Test. Again see FullExtractionChain.xml sample.

The above fixes should allow you to more easily organise your XML documents and easily change to other datasets/databases without creating a new XML file every time.

Special Notes

1.) When querying (using DBRANGE) in the database or updates and removes, then the cu1/dal needs to build a 'where' clause on the tables. By default it tends to use a column 'id', but often you want to use a different column. So here is a typical property to be set:
This is used for querying in a particular range.
gaia.cu1.tools.query.{DMInterface Name}.whererange={col} between ? and ?
gaia.cu1.tools.query.TransitDescription.whererange=solutionId between ? and ?
This is used mostly for the Update or Delete of a DM object (this might not be needed but is available) see 'dbWrite' element in the Odyssey XML. Note needed if your just adding or deleting/cleaning the whole table and then adding.
gaia.cu1.tools.query.{DMInterface Name}.where={col}=?
gaia.cu1.tools.query.TransitDescription.where=solutionId=?

Recent Changes

  • Mar 13, 2013 -- The batch mode for speed efficiency introduced a bug on Array outputs that is now fixed. A pre14.0.0 release was released because of GaiaToos. See Release information above for more details.
  • Feb 12, 2013 -- Defaults to a batch mode for writing results, to write results at the end of the looping. See Release Information above for more detail.
  • Mar 2, 2011 -- XML include is now allowed. Added ability to skip Tests, skip Logics, and Pass in a properties file all in the command line.
  • Aug 19, 2010 -- TransitLoop common Logic added. Logic declarations in the xml can now take a string property. DummyFacade added to be used for tests that really only need to do Logic and no real facade.
  • Mar 25, 2010 -- GBin files can now be written for output DM objects. See XML file for description.
  • Mar 25, 2010 -- Storage info is no longer printed out after each test. See the Logic page for a 'common' Logic that can print out the storage.
  • Mar 25, 2010 -- Outputs that are Null was throwing an exception when it shouldn't because it blindly tried to write the output. This has now been fixed.
  • Nov 18, 2009 -- putInStorage tag happens only once now, so the user if they desire can change it Later and on a goto will not get it reset to the original.
  • Nov 18, 2009 -- cleanTables now has an optional 'incallbacks' attribute that can be set to 'ONCE' to only clean it one time and to NOT clean it again if a goto or callback to that test happens.
  • Nov 18, 2009 -- Logic classes were not resetting a particular flag, causing the possibility of another Logic class in the same test to never be executed. Now fixed.
  • Nov 6, 2009 -- Primitive type inputs were not working. It is fixed now and note in the xml file for primitives className attribute is needed.
  • Nov 2, 2009 -- Added a BEGIN and END logic capability that are executed outside any looping of the Facade. See new 'Order of Calls' above.
  • Nov 2, 2009 -- Added a new ability to store a primitive value into the personal storage hash table via the xml file.
  • Nov 2, 2009 -- init() and param inputs are outside any looping. Again see 'Order of Calls' above to better understand.
  • Nov 2, 2009 -- Bug saw that the input of index '0' on a looping of Facades was again done as the last input. Hence first and last were the same, this is now fixed.
  • Oct 19, 2009 -- Found a bug in Array outputs that are Data Model(DM) objects just simply forgot to typecast, hence DMArray on output did not work. Now fixed.
  • Sept 28,2009 -- Loading DM Arrays as Inputs (Facade takes Single DM) would always re-query the database when looping over the array. This is unreliable and not efficient. So fixed the problem by placing it in a cache, for the looping.
  • Sept 7, 2009 -- First Real release on the public wiki of Odyssey.

Contact Info and Support

Best is to subscribe to a odyssey community email. Email odyssey@majordomo.mssl.ucl.ac.uk and then 'I' (Kevin) will add you. Post your questions to this email list once you are subscribed.

  • Odyssey.jar: Odyssey Library with all the necessary classes.

  • Originally a zipped up set of examples were here but now it is best to see SVN ExtractionFramework, MTA_Integration, Framework (under 630)

  • logging.properties: Example logging properties to remove many of those annoying gaiatools dal warnings. Just add to your 'conf' directory and it should get picked up automatically.

  • ivy.xml: Sample Ivy file used in Extraction Framework chain

  • build.xml: Sample Build.xml file used in Extraction Framework chain.

  • Odyssey.jar: Update of Odyssey for later than 14.0.0 release. Details of changes described in wiki above.
Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatjar Odyssey.jar r21 r20 r19 r18 r17 manage 34.8 K 2014-04-25 - 10:08 KevinBenson Update of Odyssey for later than 14.0.0 release. Details of changes described in wiki above.
Unknown file formatjar Odyssey_Pre14.0.0.jar r1 manage 40.9 K 2013-03-13 - 09:26 KevinBenson Odyssey library for releases before 14.0.0 GaiaToosl/Dpcccommon
XMLxml build.xml r1 manage 11.3 K 2011-03-03 - 09:26 KevinBenson Sample Build.xml file used in Extraction Framework chain.
XMLxml ivy.xml r1 manage 1.4 K 2011-03-03 - 09:25 KevinBenson Sample Ivy file used in Extraction Framework chain
Unknown file formatproperties logging.properties r1 manage 0.7 K 2011-03-03 - 09:24 KevinBenson Example logging properties to remove many of those annoying gaiatools dal warnings.
Compressed Zip archivezip odyssey_fullexamples.zip r3 r2 r1 manage 0.6 K 2009-11-23 - 15:02 KevinBenson No longer used, outdated see SVN modules. e.g. ExtractionFramework, MTA_Integration
Edit | Attach | Watch | Print version | History: r39 < r38 < r37 < r36 < r35 | Backlinks | Raw View | More topic actions
Topic revision: r39 - 2014-04-25 - KevinBenson
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback