create new tag
view all tags

Reaper Improvements

These are things that have not been designed into the system originally (were not in the DPM at CDR) but which have been noted during development as ways to improve REAPER.

General Doppler Calculation

This algorithm currently uses 2 consecutive records to estimate the velocity vector. This is imprecise compared to getting the information direct from the orbit. We could either use the orbit file at L2 and extract position and velocity (and this would also mean we could extract the 1Hz values directly rather than interpolating the 20Hz), or these fields could be put into the L1 product after the existing call.

Slope correction and elevation

We currently slope-correct the OCOG result. Would it be better to also slope-correct the ICE2 result?


There are some concerns about the time taken to execute the L2 chain. On a Transtec desktop (msslc2) with a 3.0GHz Intel Core 2 CPU, total execution time averaged 89 seconds, as of early Aug 2011.

Total Execution Time

Time taken to execute L2 chain on msslc2, using test test_run_s1_o1, taking the average of three executions:

  • 30 Aug 2011: 69 seconds
  • Early Aug 2011: 89 seconds

Profiling Results

Some profiling was performed to see which activities/algorithms consumed the most time. Testing was performed on a Transtec desktop (msslc2) with a 3.0GHz Intel Core 2 CPU. Times are in seconds:

  • MLE retracker (j_L2CMLE4_proc): 25
  • Logging at DEBUG level: 12 (logging at PROGRESS level reduced this to 3 seconds)
  • L2_name2id lookups (see below): 8
  • Writing to netCDF: 4
  • Profiling: 3
  • ComputePercentageInDistanceS3: 3
  • j_L2UOCOG_proc: 2
  • Each other algorithm, or activity type, consumed approx 1.5 seconds or less.

The Google profiler and GNU profiler disagree significantly in their results. Google profiler is unable to identify where the code is for 33% of the time. With GNU this rises to 67%. The difficulty may be with tracing the parent functions by traversing the stack, or I/O time, interrupts, but is not known. But they roughly agree on which of the (identifiable) tasks are consuming the most time.

Running the same 32-bit code, the Viglen Gaia server (msslus) was about 5% slower. Amongst many hardware differences the Viglen has a 64-bit CPU with 2.4 GHz clock speed.

Experimental Temporary Optimisations

These have been temporarily tried but not adopted in Reaper:

  • netCDF writes caching was tested by adding a caching layer (see attached netCDF_writer_cache.zip) on top of the netCDF API. In other projects such as Sentinel-3 and GlobICE, writing variable values in arrays has improved speed by a factor of 10 or 100, but in Reaper this was slower than writing values individually. The use of caching added about 15 seconds to total execution time. Several factors may explain why Reaper does not benefit from netCDF caching:
    • Caching seems to bring greater benefit when all dimensions are fixed length (not NC_UNLIMITED when calling nc_def_dim). In a test with a fixed length Record dimension, writing the netCDF variables took 107 seconds compared to 4 with an 'unlimited' Record dimension. Caching the writes reduced this from 107 to 7 seconds.
    • Execution time without caching increases exponentially with the number of records in a netCDF file. Reaper's L2 products have a relatively modest 6000 records, approximately, compared with 100,000 for Sentinel-3 and 1.7 million for GlobICE.
    • The presence of a 3-dimensional variable in the netCDF file appears to slow down writes of [cached] arrays to netCDF variables (using nc_put_vara), even if the 3-D variable is never read or written. This issue may need further investigation if caching is considered in future projects.
  • Reducing the maximum fitting iterations on the MLE retracker from 25 to 10 saved 6 seconds. This may affect the quality of the results, so would need further investigation.

Adopted Optimisations

Performance improvements which have been adopted in Reaper:

  • L2_name2id string lookups were replaced by a much faster lookup table method. See ReaperTesting for details.

-- DavidBrockley - 12 Aug 2011

Topic attachments
I Attachment History Action SizeSorted ascending Date Who Comment
Compressed Zip archivezip netCDF_writer_cache.zip r1 manage 3.9 K 2011-08-30 - 14:31 ChrisDolding Experimental cache for writes of variables to netCDF files.
Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions
Topic revision: r4 - 2011-08-30 - ChrisDolding
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback