Astronomers' perspective
by Andy Adamson
The OracDR data reduction system is intended to be a pipeline reducer
for incoming data. In the longer term, it will be integrated with the
ORAC observatory control system (and thus with the ORAC Observing tool
and procedures developed therein). It will also be the online
reduction system for all new instruments. There are a number of
differences between the OracDR method of reduction and the other
systems currently in use at UKIRT, and observers should not expect
OracDR to behave or be used in the same way as those systems.
- Firstly, OracDR aims to reduce data to a point where its quality
can be assessed; it will not generally produce publication-quality
results (though in certain circumstances it may do).
- Secondly, although OracDR also works offline, it is expected that
most observers will use their own preferred data reduction package to
work interactively work on their data. The rest of this document
summarizes and accounts for the operational differences between the
pipeline and existing packages.
It pipes, therefore it is
This is crucial. Everything else about the package is clear once this is
grasped. This is not a reduction package a la CGS4DR; it is a reduction
black box which knows the incoming data types (by their headers) and
transparently applies a reduction recipe to them. There is nothing
preventing you from running three simultaneous instances of the pipeline,
for example to (i) reduce the incoming data in real time, (ii) re-reduce a
previous group of files using a different reduction recipe and (iii)
reduce and file a single previous observation as a dark. You do this by
running three versions of oracdr, using the command-line switches to alter
their behaviour (recipe, start and end observation numbers to process,
graphics options, etc.). Each instance of the pipeline will go through the
required files (existing ones or files just arriving on disk as specified
on the command line) and reduce them. Once its remit of reduction is
complete, it will exit.
Control
The behaviour of OracDR is entirely controlled by the command line options
entered at startup. From that point on, the system either takes its
reduction recipe instructions from the file headers (this is the default)
or uses a hardwired recipe given on the command line itself. The recipe
is, in fact, the only allowed parameter on the command line - all the rest are
options. Once you start an instance of oracdr up, there is no further
control over it; this is a considerable change from the situation with
CGS4DR, for example, where the same package remains up once you start it,
and changes are made within the package. Corollary: if the pipeline fails
to find a required calibration frame, for example, the only logical thing
for it to do is exit cleanly, telling you why it did so. There is no
control from within the pipeline. Note that once the full ORAC system is
available, there will be plenty of pre-checking that your calibration
frames will indeed exist and be appropriate. Until that point, the
behaviour of the system is logical if you understand the underlying
philosophy and do not expect to be able to control the pipeline
in real time.
The command usage is as follows:
oracdr [-options] RECIPE_NAME
note again that the only parameter to the command is the recipe name.
These are all given on the web page linked to from the ufti pages. The
recipes are applied to the individual frames by an algorithm engine, which
is currently the STARLINK system (KAPPA, CCDPACK).
Options
The rest of the behaviour of the command is controlled by the option
switches. These options may be abbreviated to a unique substring. It's via
command line switches that you (for example) control the range of file
numbers to be reduced, force the system to use a particular calibration
file when reducing (e.g. to try a different flat exposure). This list
needs to be read thoroughly by anyone wanting to use the system.
-help this help text
-version print the version number
-verbose print engine messages (else just have oracdr messages)
-debug log debug messages to ORACDR.DEBUG
-nodisplay do not launch display
-from number of first observation
-to number of last observation
-list comma separated list of object *numbers*
Colons indicate a range (1,2,3:5,10 means 1,2,3,4,5,10)
-ut UT date of observations (defaults to current yyyymmdd)
-calib comma seperated key=value pairs (eg. -cal dark=file1 or
-cal dark=file1,bias=file2) of calibration *files*
-loop Type of data detection loop: list, inf, or wait
-loop list is equivalent to not using a loop at all
(only data specified is reduced)
-loop wait waits for data to appear before timing out
(data is reduced and we wait for new data)
-loop inf does not wait for data
(data is reduced and new data is assumed to be available)
-loop flag waits for completion files to appear before
procesing (data is reduced and we wait for new data
via the flag)
-batch Run in batch mode. Precalculate groups before proceesing
data. 'wait' loop mode should not be used with this option.
-skip Allow the data detection loop to skip missing observations.
Default is to stop the loop when a expected file
can not be found.
-noeng Do not start algorithm engines. Note: this will cause
the vast majority of recipes to fail.
-log Control the location of the session logging. Default
is to log messages to the screen and file (log=sf)
when DISPLAY environment variable is not set. If
the DISPLAY env variable is set default is to use log=fx.
Options are:
s - log to screen (Xterm=STDOUT)
f - log to file. The logfile is called .oracdr_12922.log
where 12922 is the current process ID - it is written
to ORAC_DATA_OUT (it is a hidden file)
x - log to an X widget. Has the advantage that warnings
and errors do not scroll off the screen
The options can be combined (ie log=sf will write log
messages to screen and to a file).
-warn Turn on warning messages (perl -w). This should be used
for debugging only. If -verbose is also true then full
diagnostics are turned on (see 'perldoc diagnostics')