Overview of this package

This package’s purpose is to allow easy scripting using Starlink routines from Python. It sets up the necessary environmental variables inside the package (so you do not have to source Starlink before running Python), and ensures that Starlink is running in a scriptable mode where it will not prompt the user for input, and will write to a custom ADAM directory.

This package also attempts to provide a more pythonic interface to the commands, to provide easy access to the return values from Starlink commands (without using parget), and to not require shell escapes to be passed to the commands.

This package contains:

• starlink.wrapper a module for calling Starlink commands (including ORAC-DR)
• starlink.utilities a module with a small set of utility commands.

This package also includes convenience modules to make it easier to script the most popular (for EAO/JCMT users) Starlink packages from Python. These include integrated automatically generated help and call signatures for the commands:

• starlink.convert CONVERT – utilties for converting files to and from NDF
• starlink.cupid CUPID – Clump finding in N-dimensions
• starlink.fluxes Fluxes – JCMT Posiiton and Flux density calibration
• starlink.kappa KAPPA – Various commands for data processing, manipulation and visualisation
• starlink.polpack POLPACK – applications for reducing/handling polarimetry data
• starlink.smurf SMURF – Submm User Reduction Facility
• starlink.picard Picard – Pipeline for analysis of JCMT data

We also include the following packages:

• starlink.atools ATOOLS – AST applications
• starlink.ccdpack CCDPACK – programs for reducing CCD-like data
• starlink.figaro Figaro – General purpose DR

Each module is available as starlink.<modulename>. The commands are then available as starlink.<modulename>.<commandname>, and the help as help(starlink.<modulename>.<commandname>. Each command will return a Python object giving all of given and returned values from the Starlink command.

You can also see a convenient short listing of the available comamnds in the Starlink module and what they do by using the starlink.utilties.starhelp() command, like so:

>>> from starlink.utilities import starhelp
>>> from starlink import kappa
>>> starhelp(kappa)

These packages were autogenerated from a Starlink build, and you should ideally use versions generated from the same version of Starlink as you are running. If you are using a different version of the starlink-pywrapper package than your Starlink, then you may be missing some commands or some argument or keyword options to the commands present. However, most commands will probably work correctly. You can see which Starlink version the modules were generated from by using:

>>> help(starlink.kappa)

The top of this will mention the Starlink version the package was generated from.

Installation

You should normally install this package via pip with:

pip install starlink-pywrapper

This will also install the necessary python dependencies. The package depends on the starlink-pyhds package, which itself requires a Numpy installation (version>=1.13).

If you have the Astroy python package installed then you will also be able to use the starlink.utilities.get_ndf_fitshdr() command to return an Astropy FITS Header object of an NDF files FITS extension. This dependency will not be installed automatically by pip.

The source code lives at https://github.com/Starlink/starlink-pywrapper .

You can install from source by cloning this repo and running:

python setup.py install

You must also have a working Starlink installation, which can be downloaded from https://starlink.eao.hawaii.edu.

Running Starlink commands from Python

Setting up Starlink

First, you have to let this module know where your Starlink software suite is installed, either by telling it directly or by setting up an environmental variable in your terminal before starting Python.

You can set the location directly inside a Python session/script as:

>>> from starlink import wrapper
>>> wrapper.change_starpath('/path/to/my/starlink/installation')

or if you don’t want to import the wrapper separately, you can access it from an imported package, e.g. for KAPPA you could do:

>>> from starlink import kappa
>>> kappa.wrapper.change_starpath('/path/to/my/starlink/installation')

The starpath you set in one package will then be set for all the starlink.<packagename> packages you load. You can change the starpath again at any time in a script, and afterwards all appropriate variables will have been changed to run from the new Starlink.

Alternatively, before you start Python you can set a STARLINK_DIR environmental variable to the location of your Starlink installation. For example, in a BASH shell you could run export STARLINK_DIR=~/star-2018A. This package will then detect and use that STARLINK_DIR location by default when you load it into Python.

To see which Starlink is currently being used examine the variable wrapper.starpath, or if you are using a module you can do e.g.:

>>> print(kappa.wrapper.starpath)

Running commands

You will need to import each Starlink package that you want to use. For example, to run the stats command from KAPPA on a file myndf.sdf you would do:

>>> from starlink import kappa
>>> statsvals = kappa.stats('myndf.sdf')

Each command will return a Python namedtuple object with all of the output found in $ADAM_USER/commandname.sdf.

>>> print(statsvals)
    mean    0.028790334098242488
   sigma    0.47779065479062127
  numpix    20480
  maxpos    [13, 1, 10]
  minwcs    0.02653818, 1, 2016-05-13 08:55:33
maxcoord    [0.026553806951968184, 1.0, 0.37904242114746]
 numgood    20480
kurtosis    110.59406086721943
  maxwcs    0.02655381, 1, 2016-05-13 09:05:49
 maximum    9.728691101074219
   total    589.6260423320055
 minimum    -8.658288955688477
skewness    2.668368146271298
     ndf    /home/sgraves/a20160513_00039_01_0001.sdf
   order    False
    comp    DATA
  minpos    [20, 1, 5]
  numbad    0
mincoord    [0.026538183309314123, 1.0, 0.371909411893284]

You can access these values by name. E.g., to print the returned mean value from the KAPPA stats command run above, you would do:

>>> print(statsvals.mean)

Or to see what fields are available, you can do:

>>> print(statsvals._fields)
('mean', 'sigma', 'numpix', 'maxpos', 'minwcs', 'maxcoord', 'numgood', 'kurtosis', 'maxwcs', 'maximum', 'total', 'minimum', 'skewness', 'ndf', 'order', 'comp', 'minpos', 'numbad', 'mincoord')

Inside an ipython terminal session or jupyter notebook you can use tab completion to interactively see the list of available fields.

Getting help.

This package includes docstrings for each command, summarising the command and its arguments and keywords. This can be seen in the normal Python way, e.g.

>>> help(kappa.ndftrace)

At the bottom of the command it should also give you the web URL to see the full documentation in the Starlink User Note (SUN) for that package.

To see the available commands in a package, there is a utiliity starhelp that will show you the name and the short one-line description for each command. You can use it on a Starlink module like so:

>>> from starlink.utilities import starhelp
>>> starhelp(kappa)
add         :     Adds two NDF data structures.
align2d     :     Aligns a pair of two-dimensional NDFs by minimising the residuals
aperadd     :     Integrates pixel values within an aperture of an NDF.
ardgen      :     Creates a text file describing selected regions of an image.
ardmask     :     Uses an ARD file to set some pixels of an NDF to be bad.
ardplot     :     Plot regions described in an ARD file.
axconv      :     Expands spaced axes in an NDF into the primitive form.
<truncated>

starhelp can also be called with a command name as the argument: it will then show you the full (and often extremely long) documentation for that command within your Python session. This full documentation is simply an RST version of the documentation from the Starlink SUN or from the Starlink inline help. The SUN version available online (from the URL shown at the end of the normal help(commandname)) will have nicer formatting however.

>>> starhelp(kappa.stats)

Directly running a command (without using the specific package).

You can directly run a Starlink command using starlink.wrapper.starcomm() method. This method is used by the individual Starlink modules to run their commands. For example, to run the KAPPA command ndftrace you could do:

>>> from starlink import wrapper
>>> results = wrapper.starcomm('$KAPPA_DIR/ndftrace', 'ndftrace', 'myndf.sdf', fullwcs=True)

This can be helpful if there is something wrong with the automatically-generated argument and keyword options in the specific commands, or if you are trying to run a command that is not included.

Logging and seeing the full output.

These modules use the standard Python logging module. To print the normal output of a starlink command to your screen, as well as the details of the command line being run, you will need to set the logging module to DEBUG, i.e run:

>>> import logging
>>> logger = logging.getLogger(__name__)
>>> logger.setLevel(logging.DEBUG)

or

>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)

You can also return all the information that is normally written to screen by setting the extra keyword argument returnstdout=True. This will cause your command to return a two part tuple of (<normal-output>, <string-of-output-from-screen>).

>>> statsvals, onscreenstats = kappa.stats('myndf.ndf', returnstdout=True)
>>> print(onscreenstats)
   Pixel statistics for the NDF structure /home/sgraves/a20160513_00039_01_0001
      Title                     : <undefined>
      NDF array analysed        : DATA
         Pixel sum              : 589.626
         Pixel mean             : 0.0287903
         Standard deviation     : 0.477791
         Skewness               : 2.66837
         Kurtosis               : 110.594
         Minimum pixel value    : -8.65829
            At pixel            : (20, 1, 5)
            Co-ordinate         : (0.02653818, 1, 2016-05-13 08:55:33)
         Maximum pixel value    : 9.72869
            At pixel            : (13, 1, 10)
            Co-ordinate         : (0.02655381, 1, 2016-05-13 09:05:49)
         Total number of pixels : 20480
         Number of pixels used  : 20480 (100.0%)

The returnstdout keyword is not documented in the help for the individual commands, but it should work in all of them apart from fluxes.get_flux.

Scripting ORAC-DR and PICARD

ORAC-DR is the data reduction Pipline that is shipped with Starlink – see http://starlink.eao.hawaii.edu/docs/sun230.htx/sun230.html for details. PICARD is a variant of ORAC-DR used for running data analysis pipelines on already reduced data (see http://www.starlink.ac.uk/docs/sun265.htx/sun265.html). In this package, ORAC-DR and Picard are called in a different way from the other Starlink packages, because they have a different syntax for calling and because they are in fact Perl scripts that call Starlink commands internally.

ORAC-DR

The wrapper package contains a routine for calling the ORAC-DR pipeline reductions. Although it may seem a little silly to have a python wrapper calling a Perl pipeline calling the Starlink commands, several fdifferent implementations have been seen in the wild. As there was a clear desire for this behaviour, we have provided an interface to it

The basic command is in starlink.wrapper.oracdr(), and can be called like so:

>>> from starlink import wrapper
>>> output = wrapper.oracdr('SCUBA2_850', rawfiles='inputfiles.lis')

The first argument is the instrument name, which for JCMT Users will normally be either ACSIS (for spectral observations), SCUBA2_850 or SCUBA2_450 (for the two SCUBA-2 wavelengths).

The rawfiles argument inputfiles.lis is a file containing the full path names to your input files, and inputfiles.lis is located in your current working directory. You could alternately provide a Python list, where each item is the full path to the datafiles being included. If you want to use relative paths to the input datafiles instead, note that these will be used assuming they are relative the actual directory ORAC-DR is run in, which will be a temporary working directory made by this wrappers.

If you wish to override the default recipe in the input files, you can provide the recipe= keyword argument with the name of the ORAC-DR recipe you wish to use instead.

You can also provide a recipe parameter file to customise your ORAC-DR run just as you can in the standalone ORAC-DR. This is done by providing a recpars= keyword argument, where the value is a string either representing the filename of the recipe parameter file or the recipe parameters directly (see the -recpars inh ttp://starlink.eao.hawaii.edu/docs/sun230.htx/sun230se6.html#Q1-7-83 for details).

>>> output = oracdr('ACSIS', rawfiles=listoffiles, recipe='REDUCE_SCIENCE_NARROWLINE', recpar='recpar.txt')

or

>>> output = oracdr('ACSIS', rawfiles=filelist, recpars='PIXEL_SCALE=3.0,SPREAD_METHOD=GAUSS')

This package also includes support for many other options for ORAC-DR – see starlink.wrapper.oracdr() for the full help.

The returned object from ORAC-DR is very different from the other commands in this package. It returns a namedtuple object that hopefully contains the names of all the output files you are interested in. The values provided are:

• runlog name of output logfile.
• outputdir path of output directory
• datafiles list of output data files.
• imagefiles list of output image files.
• logfiles list of log.* files
• status int, return code from suprocess.Popen
• pid (int): pid of perl parent process.

If the return status was not 0, the command will raise an error. However, you should check the logs carefully as some ORAC-DR errors may not cause you problems.

In order to prevent repeated calls to ORAC-DR from writing each other over, this command will create a working directory (by default in the current working directory) in which it runs ORAC-DR. It will not clean up this working directory, and instead it is up to the user to copy out whichever files they wish to save and delete the directory themselves.

Some examples of using starlink.wrapper.oracdr are provided in ?????

PICARD

PICARD is the data analysis pipeline, the equivalent of ORAC-DR but for already reduced files. There is a similar interface to that for ORAC-DR provided in starlink.wrapper.picard(). However, various convenience functions have also been provided for each of the PICARD recipes. These are located in starlink.picard():

>>> from starlink import picard
>>> starhelp(picard)
calc_scuba2_avpspec      : Run PICARD'S CALC_SCUBA2_AVPSPEC recipe.
calc_scuba2_fcf          : Run PICARD'S CALC_SCUBA2_FCF recipe.
calc_scuba2_nefd         : Run PICARD'S CALC_SCUBA2_NEFD recipe.
calibrate_scuba2_data    : Run PICARD'S CALIBRATE_SCUBA2_DATA recipe.
coadd_jsa_tiles          : Run PICARD'S COADD_JSA_TILES recipe.
create_moments_map       : Run PICARD'S CREATE_MOMENTS_MAP recipe.
create_png               : Run PICARD'S CREATE_PNG recipe.
crop_scuba2_images       : Run PICARD'S CROP_SCUBA2_IMAGES recipe.
estimate_image_alignment : Run PICARD'S ESTIMATE_IMAGE_ALIGNMENT recipe.
jsa_catalogue            : Run PICARD'S JSA_CATALOGUE recipe.
mosaic_jcmt_images       : Run PICARD'S MOSAIC_JCMT_IMAGES recipe.
picard_demonstrator      : Run PICARD'S PICARD_DEMONSTRATOR recipe.
scuba2_check_cal         : Run PICARD'S SCUBA2_CHECK_CAL recipe.
scuba2_check_rms         : Run PICARD'S SCUBA2_CHECK_RMS recipe.
scuba2_display_pca       : Run PICARD'S SCUBA2_DISPLAY_PCA recipe.
scuba2_jackknife         : Run PICARD'S SCUBA2_JACKKNIFE recipe.
scuba2_jackknife_psf     : Run PICARD'S SCUBA2_JACKKNIFE_PSF recipe.
scuba2_map_pspec         : Run PICARD'S SCUBA2_MAP_PSPEC recipe.
scuba2_mapstats          : Run PICARD'S SCUBA2_MAPSTATS recipe.
scuba2_matched_filter    : Run PICARD'S SCUBA2_MATCHED_FILTER recipe.
scuba2_photom            : Run PICARD'S SCUBA2_PHOTOM recipe.
scuba2_register_images   : Run PICARD'S SCUBA2_REGISTER_IMAGES recipe.
scuba2_remove_background : Run PICARD'S SCUBA2_REMOVE_BACKGROUND recipe.
scuba2_sassy             : Run PICARD'S SCUBA2_SASSY recipe.
stack_jcmt_frames        : Run PICARD'S STACK_JCMT_FRAMES recipe.
uncalibrate_scuba2_data  : Run PICARD'S UNCALIBRATE_SCUBA2_DATA recipe.
untrim_jsa_tiles         : Run PICARD'S UNTRIM_JSA_TILES recipe.

Currently, these methods do not yet have any integrated help; instead please see SUN XXX at XXX for the details.

All these methods do is set the recipe name for the Picard call, otherwise the interface is the same as that of the wrapper.picard method. Therefore the following two calls are equivalent:

>>> output = wrapper.picard('SCUBA2_MAPSTATS', 'myfiles.lis')
>>> output = picard.scuba2_mapstats('myfiles.lis')

The returned object has the same attributes as that returned by starlink.wrapper.oracdr():

• runlog name of output logfile.
• outputdir path of output directory
• datafiles list of output data files.
• imagefiles list of output image files.
• logfiles list of log.* files
• status int, return code from suprocess.Popen
• pid (int): pid of perl parent process.

How this package runs Starlink

This package sets up all of the necessary Starlink environmental variables inside Python, and then uses subprocess.Popen to run the Starlink commands. It is not necessary to source the Starlink setup scripts before using this package (e.g. by running source $STARLINK_DIR/etc/profile or similar) but you do have to tell the package where $STARLINK_DIR is, either by setting the environmental variable before starting Python, or by calling the starlink.wrapper.change_starpath command with the appropriate location.

It uses the starlink.hds module (which must be installed to use this package) to access the output data written into $ADAM_USER/commandname.sdf and return it to the user.

The wrapper modules around the specific Starlink packages are (mostly) automatically generated from a Starlink build by parsing the .hlp and .ifl files produced by Starlink. This script is present in the git repository for this project (https://github.com/Starlink/starlink-pywrapper, see the helperfunctions directory), but is not distributed when you install the software. Currently the FLUXES package is the only one with a manually created Python wrapper.

This package should be regenerated for each Starlink release; normally most commands will stay unchanged on a new release, but there are normally a few additional commands (or deletion of obsolete commands), and the call signature for some commands may also change, normally only to add additional arguments or keywords.

ADAM directory

This package uses a local, temporary $ADAM_USER directory created in the current working directory and deleted on exit, so it is safe to have multiple scripts running concurrently on the same machine.

If your scripts exit with an error or are killed by the user with Ctrl-C, this temporary directory can sometimes be left. It is safe to delete this yourself.

Commands that should be avoided

Some of the Starlink commands that are wrapped in this package should not be used, as they are either no longer necessary or will not work correctly. The following commands should normally be avoided.

parget

KAPPA’s parget command is used by Starlink to read values in from the $ADAMDIR/<commandname>.sf file created by Starlink commands. This does not need to be used with this wrapper, as all values are automatically returned to the user in a namedtuple object.

fitslist

KAPPA’s fitslist command only prints the FITS header values of an NDF to screen, therefore the returned object is not useful for scripting. It is possible to use the returnstdout=True keyword to get the values as a string, but instead we would recommend the following options:

• If you only need a single FITS header value you can use starlink.kappa.fitsval() to read it.
• If you need the whole header, then if you have Astropy python package installed on your computer you can use the :meth:`starlink.utilities.get_ndf_fitsheader method to return a astropy.io.fits.header.Header object. If you are not familiar with these objects, you can treat this much like a Python dictionary object.

Interactive usage

Many commands in Starlink have an interactive mode. This inclues both the basic mode where Starlink will prompt the user for needed values that weren’t passed on the command line, and more complex modes such as selecting positions from a displayed images. These interactive modes are not supported in this interface, as it was primarily designed for scripting.

Known Issues

1. When calling Starlink commands that are really Python scripts, such as starlink.smurf.jsasplit(), the module will not raise a proper error . Please ensure you can see the DEBUG info to identify problems. (This can be fixed if the scripts raise an exit code on error).