Command-line scripts¶

Each of the command-line scripts described below is also callable from python. The details of how you call it from python are described below.

Both these ways of invoking the script (from the command line or from python) is are wrappers around the python functions that do the real work. References to those functions, which tend to provide more control over what you can do at the expense of taking more effort to understand, are provided below where appropriate.

Running the complete standard workflow: `run_standard_header_process.py`¶

Usage summary¶

usage: run_standard_header_process.py [-h]
                                      (--dest-root DEST_ROOT | --overwrite-source)
                                      [--scripts-only] [-r {a,t,p}]
                                      [--no-blind] [--ignore-fits-ra-dec]
                                      [-o OBJECT_LIST] [--quiet-console]
                                      [--silent-console] [--debug]
                                      [--quiet-log]
                                      source_root

Positional Arguments¶

source_root: All directories below this one that contain images will be processed

Named Arguments¶

--dest-root

If set, image directories below source-root will be copied into this directory tree. Only directories that contain image files will be copied; any intermediary directories required to contain directories that contain images will also be created.

--overwrite-source

This flag must be used to overwrite images in the course directory.

Default: False

--scripts-only

This script will write a single shell script with the name provided in this option. No images will be modified or directories created, but the script can be run to do those things.

Default: False

-r, --run-only

Possible choices: a, t, p

Select which scripts you want to run. This can be any combination of [p]atch, [a]strometry and [t]riage.

--no-blind

Disable astrometry for images without pointing information

Default: False

--ignore-fits-ra-dec

Ignore ptg information in FITS headers

Default: False

-o, --object-list

Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed

--quiet-console

Log only errors (or worse) to console while running scripts

Default: False

--silent-console

Turn off all logging output to console

Default: False

--debug

Turn on very detailed logging output

Default: False

--quiet-log

Log only warnings (or worse) to FILES AND CONSOLE while running scripts

Default: False

Header patching: `run_patch.py`¶

For a detailed description of which header keywords are modified see Keywords purged before further processing.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary¶

usage: run_patch.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                    [--quiet-console] [--silent-console] [-o OBJECT_LIST]
                    [--overscan-only]
                    dir [dir ...]

Positional Arguments¶

dir: Directory to process

Named Arguments¶

-v, --verbose

provide more information during processing

Default: False

-d, --destination-dir

Directory in which output from this script will be stored

--debug

Turn on very detailed logging output

Default: False

-n, --no-log-destination

Do not write log files to destination directory

Default: False

--quiet-console

Log only errors (or worse) to console while running scripts

Default: False

--silent-console

Turn off all logging output to console

Default: False

-o, --object-list

Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed

Default: “https://raw.github.com/mwcraig/feder-object-list/master/feder_object_list.csv”

--overscan-only

Only add appropriate overscan keywords

Default: False

For each directory provided on the command line the headers all of the FITS files in that directory are modified to add information like LST, apparent object position, and more. See the full documentation for a list of the specific keywords that are modified.

This is basically a wrapper around the function patch_headers.patch_headers() with the options set so that:

”Bad” keywords written by MaxImDL 5 are purged.

IMAGETYP keyword is changed from default MaxIM DL style to IRAF style (e.g. “Bias Frame” to “BIAS”)

Additional useful times like LST, JD are added to the header.

Apparent position (Alt/Az, hour angle) are added to the header.

Information about overscan is added to the header.

Files are overwritten.

For more control over what is patched and where the patched files are saved see the documentation for patch_headers at patch_headers.patch_headers().

run_patch also adds the name of the object being observed when appropriate (i.e. only for light files) and possible. It needs to be given a list of objects; looking up the coordinates for those objects requires an Internet connection. See

For a detailed description of the object list file see Object file format.

for a detailed description of the function that actually adds the object name see patch_headers.add_object_info().

If no object list is specified or present in the directory being processed the OBJECT keyword is simply not added to the FITS header.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Invoking this script from the command line:

run_patch.py /my/folder/of/images

To work on the same folder from within python, do this:

from msumastro.scripts import run_patch
run_patch.main(['/my/folder/of/images'])

To use the same object list for several different directories do this:

run_patch.py --object-list path/to/list.txt dir1 dir2 dir3

where path/to/list.txt is the path to your object list and dir1, dir2, etc. are the directories you want to process.

From within python this would be:

from msumastro.scripts import run_patch
run_patch.main(['--object-list', 'path/to/list.txt',
               'dir1', 'dir2', 'dir3'])

DESCRIPTION¶

For each directory provided on the command line the headers all of the FITS files in that directory are modified to add information like LST, apparent object position, and more. See the full documentation for a list of the specific keywords that are modified.

Header patching¶

This is basically a wrapper around the function patch_headers.patch_headers() with the options set so that:

“Bad” keywords written by MaxImDL 5 are purged.

IMAGETYP keyword is changed from default MaxIM DL style to IRAF style (e.g. “Bias Frame” to “BIAS”)

Additional useful times like LST, JD are added to the header.

Apparent position (Alt/Az, hour angle) are added to the header.

Information about overscan is added to the header.

Files are overwritten.

For more control over what is patched and where the patched files are saved see the documentation for patch_headers at patch_headers.patch_headers().

Adding OBJECT keyword¶

run_patch also adds the name of the object being observed when appropriate (i.e. only for light files) and possible. It needs to be given a list of objects; looking up the coordinates for those objects requires an Internet connection. See

For a detailed description of the object list file see Object file format.

for a detailed description of the function that actually adds the object name see patch_headers.add_object_info().

If no object list is specified or present in the directory being processed the OBJECT keyword is simply not added to the FITS header.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES¶

Invoking this script from the command line:

run_patch.py /my/folder/of/images

To work on the same folder from within python, do this:

from msumastro.scripts import run_patch
run_patch.main(['/my/folder/of/images'])

To use the same object list for several different directories do this:

run_patch.py --object-list path/to/list.txt dir1 dir2 dir3

where path/to/list.txt is the path to your object list and dir1, dir2, etc. are the directories you want to process.

From within python this would be:

from msumastro.scripts import run_patch
run_patch.main(['--object-list', 'path/to/list.txt',
               'dir1', 'dir2', 'dir3'])

Adding astrometry: `run_astromtery.py`¶

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary¶

usage: run_astrometry.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                         [--quiet-console] [--silent-console] [-b] [-c]
                         [-o ODDS_RATIO]
                         [--astrometry-config ASTROMETRY_CONFIG]
                         [--camera CAMERA] [--avoid-pyfits]
                         [--ignore-fits-ra-dec] [--no-verify] [--force]
                         dir [dir ...]

Positional Arguments¶

dir: Directory to process

Named Arguments¶

-v, --verbose

provide more information during processing

Default: False

-d, --destination-dir

Directory in which output from this script will be stored

--debug

Turn on very detailed logging output

Default: False

-n, --no-log-destination

Do not write log files to destination directory

Default: False

--quiet-console

Log only errors (or worse) to console while running scripts

Default: False

--silent-console

Turn off all logging output to console

Default: False

-b, --blind

Turn ON blind astrometry; disabled by default because it is so slow.

Default: False

-c, --custom-sextractor

Use Feder-specific SExtractor settings

Default: False

-o, --odds-ratio

Change the odds-ratio for accepting a match from the default of 1e9.

--astrometry-config

File to use for configuring astrometry engine, including, e.g., the location of index files.

--camera

Name of camera; used to set pixel scale in solve. If omitted, uses settings for Apogee Alta U9.

--avoid-pyfits

Add options to avoid calls to pyfits.

Default: False

--ignore-fits-ra-dec

Ignore any RA/Dec information in the FITS header.

Default: False

--no-verify

Use to force astrometry.net to ignore any WCS present in the file.

Default: False

--force

Run astrometry.net even if WCS is already present

Default: False

For each directory provided on the command line add astrometry to the light files (those with IMAGETYP='LIGHT' in the FITS header).

By default, astrometry is added only for those files with pointing information in the header (specifically, RA and Dec) because blind astrometry is fairly slow. It may be faster to insert RA/Dec into those files before doing astrometry.

The functions called by this script set the WCS reference pixel to the center of the image, which turns out to make aligning images a little easier.

For more control over the parameters see add_astrometry() and for even more control, call_astrometry().

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Invoking this script from the command line:
run_astrometry.py /my/folder/of/images
To work on the same folder from within python, do this:
from msumastro.scripts import run_astrometry
run_astrometry.main(['/my/folder/of/images'])

DESCRIPTION¶

For each directory provided on the command line add astrometry to the light files (those with IMAGETYP='LIGHT' in the FITS header).

By default, astrometry is added only for those files with pointing information in the header (specifically, RA and Dec) because blind astrometry is fairly slow. It may be faster to insert RA/Dec into those files before doing astrometry.

The functions called by this script set the WCS reference pixel to the center of the image, which turns out to make aligning images a little easier.

For more control over the parameters see add_astrometry() and for even more control, call_astrometry().

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES¶

Invoking this script from the command line:
run_astrometry.py /my/folder/of/images
To work on the same folder from within python, do this:
from msumastro.scripts import run_astrometry
run_astrometry.main(['/my/folder/of/images'])

Find problems and create summary: `run_triage.py`¶

Usage summary¶

usage: run_triage.py [-h] [--debug] [-v] [-d DESTINATION_DIR] [-n]
                     [--quiet-console] [--silent-console] [-k KEY] [-l] [-a]
                     [-t TABLE_NAME] [-o OBJECT_NEEDED_LIST]
                     [-p POINTING_NEEDED_LIST] [-f FILTER_NEEDED_LIST]
                     [-y ASTROMETRY_NEEDED_LIST]
                     [dir [dir ...]]

Positional Arguments¶

dir: Directory to process

Named Arguments¶

--debug

Turn on very detailed logging output

Default: False

-v, --verbose

provide more information during processing

Default: False

-d, --destination-dir

Directory in which output from this script will be stored

-n, --no-log-destination

Do not write log files to destination directory

Default: False

--quiet-console

Log only errors (or worse) to console while running scripts

Default: False

--silent-console

Turn off all logging output to console

Default: False

-k, --key

FITS keyword to add to table in addition to the defaults; for multiple keywords use this option multiple times.

Default: []

-l, --list-default

Print default list keywords put into table and exit

Default: False

-a, --all

Construct table from all FITS keywords present in headers and the list of default keywords.

Default: False

-t, --table-name

Name of file in which table is saved; default is Manifest.txt

Default: “Manifest.txt”

-o, --object-needed-list

Name of file to which list of files that need object name is saved; default is NEEDS_OBJECT_NAME.txt

Default: “NEEDS_OBJECT_NAME.txt”

-p, --pointing-needed-list

Name of file to which list of files that need pointing name is saved; default is NEEDS_POINTING_INFO.txt

Default: “NEEDS_POINTING_INFO.txt”

-f, --filter-needed-list

Name of file to which list of files that need filter is saved; default is NEEDS_FILTER.txt

Default: “NEEDS_FILTER.txt”

-y, --astrometry-needed-list

Name of file to which list of files that need astrometry is saved; default is NEEDS_ASTROMETRY.txt

Default: “NEEDS_ASTROMETRY.txt”

For each directory provided on the command line create a table in that directory with one row for each FITS file in the directory. The columns are FITS keywords extracted from the header of each file.

The list of default keywords extracted is available through the command line option --list-default.

Note

This feature is available only from the command line.

For more control over the parameters see triage_fits_files()

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Invoking this script from the command line:
python run_triage.py /my/folder/of/images
Get list of default keywords included in summary table:
python run_triage.py --list-default
To work on the same folder from within python, do this:
from msumastro.scripts import run_triage
run_triage.main(['/my/folder/of/images'])
# or...
run_triage.main(['--list-default'])

DESCRIPTION¶

For each directory provided on the command line create a table in that directory with one row for each FITS file in the directory. The columns are FITS keywords extracted from the header of each file.

The list of default keywords extracted is available through the command line option --list-default.

Note

This feature is available only from the command line.

For more control over the parameters see triage_fits_files()

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

EXAMPLES¶

Invoking this script from the command line:
python run_triage.py /my/folder/of/images
Get list of default keywords included in summary table:
python run_triage.py --list-default
To work on the same folder from within python, do this:
from msumastro.scripts import run_triage
run_triage.main(['/my/folder/of/images'])
# or...
run_triage.main(['--list-default'])

Manual intervention: `quick_add_keys_to_file.py`¶

Warning

This script OVERWRITES the image files in the directories specified on the command line. There is NO WAY TO DISABLE this behavior.

Usage summary¶

usage: quick_add_keys_to_file.py [-h] [--file-list FILE_LIST]
                                 (--key-file KEY_FILE | --key-value KEY_VALUE KEY_VALUE)
                                 [files [files ...]]

Positional Arguments¶

files: Files in which to add/change keywords

Named Arguments¶

--file-list: File with list of files in which keywords are to be changed
--key-file: File with keywords and values to be set
--key-value: Keyword to add/change

Add/modify keywords in FITS files.

Add each of the keywords in either the key_file or specified on the command line to each of the files either listed in the file file_list or specified on the command line. If the keyword is already present its value is updated to the value in key_file. A HISTORY comment is added to the header for each keyword indicating which keyword was modified.

Warning

This script OVERWRITES the image files in the list specified on the command line. There is NO WAY to override this behavior.

Add/modify keywords in FITS files.

DESCRIPTION¶

Add each of the keywords in either the key_file or specified on the command line to each of the files either listed in the file file_list or specified on the command line. If the keyword is already present its value is updated to the value in key_file. A HISTORY comment is added to the header for each keyword indicating which keyword was modified.

Warning

This script OVERWRITES the image files in the list specified on the command line. There is NO WAY to override this behavior.

Sorting files based on image properties¶

Note

By default this script makes a copy of the images being sorted. There is an option for moving the files instead.

This type of sorting is handy for working with images in GUI software like AstroImageJ or MaxImDL, but will make it harder to process the data programmatically in python.

Usage summary¶

usage: sort_files.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                     [--quiet-console] [--silent-console] [--move]
                     dir [dir ...]

Positional Arguments¶

dir: Directory to process

Named Arguments¶

-v, --verbose

provide more information during processing

Default: False

-d, --destination-dir

Directory in which output from this script will be stored

--debug

Turn on very detailed logging output

Default: False

-n, --no-log-destination

Do not write log files to destination directory

Default: False

--quiet-console

Log only errors (or worse) to console while running scripts

Default: False

--silent-console

Turn off all logging output to console

Default: False

--move, -m

Move files instead of copying them.

Default: False

For the directory provided on the command line sort the FITS files in this way:
destination
    |
    |
    |
    |---'BIAS'
    |
    |---'DARK'
    |     |---exposure_time_1
    |     |---exposure_time_2, etc.
    |
    |---'FLAT'
    |      |---filter_1
    |      |    |---exposure_time_1
    |      |    |---exposure_time_2, etc.
    |      |
    |      |---filter_2, etc.
    |
    |---'LIGHT'
            |---object_1
            |       |---filter_1
            |       |    |---exposure_time_1
            |       |    |---exposure_time_2, etc.
            |       |
            |       |---filter_2, etc.
            |
            |---object_2
            |       |---filter_1
            |       |---filter_2, etc.
            |
            |---object_3, etc.
            |
            |---'no_object'
                    |---filter_1
                    |---filter_2, etc.
The names in single quotes, like ‘bias’, appear exactly as written in the directory tree created. Names like exposure_time_1 are replaced with a value, for example 30.0 if the first dark exposure time is 30.0 seconds.

The directory destination/calibration/flat/R will contain all of the FITS files that are R-band flats.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

Unless you explicitly supply a destination using the –destination-dir option the files will be copied/moved in the directory in which they currently exist. While this should not lead to information loss, since files are moved or copied but never deleted, you have been warned.

DESCRIPTION¶

For the directory provided on the command line sort the FITS files in this way:
destination
    |
    |
    |
    |---'BIAS'
    |
    |---'DARK'
    |     |---exposure_time_1
    |     |---exposure_time_2, etc.
    |
    |---'FLAT'
    |      |---filter_1
    |      |    |---exposure_time_1
    |      |    |---exposure_time_2, etc.
    |      |
    |      |---filter_2, etc.
    |
    |---'LIGHT'
            |---object_1
            |       |---filter_1
            |       |    |---exposure_time_1
            |       |    |---exposure_time_2, etc.
            |       |
            |       |---filter_2, etc.
            |
            |---object_2
            |       |---filter_1
            |       |---filter_2, etc.
            |
            |---object_3, etc.
            |
            |---'no_object'
                    |---filter_1
                    |---filter_2, etc.
The names in single quotes, like ‘bias’, appear exactly as written in the directory tree created. Names like exposure_time_1 are replaced with a value, for example 30.0 if the first dark exposure time is 30.0 seconds.

The directory destination/calibration/flat/R will contain all of the FITS files that are R-band flats.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

Unless you explicitly supply a destination using the –destination-dir option the files will be copied/moved in the directory in which they currently exist. While this should not lead to information loss, since files are moved or copied but never deleted, you have been warned.

Command-line scripts¶

Running the complete standard workflow: run_standard_header_process.py¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

Header patching: run_patch.py¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

DESCRIPTION¶

Header patching¶

Adding OBJECT keyword¶

EXAMPLES¶

Adding astrometry: run_astromtery.py¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

DESCRIPTION¶

EXAMPLES¶

Find problems and create summary: run_triage.py¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

DESCRIPTION¶

EXAMPLES¶

Manual intervention: quick_add_keys_to_file.py¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

DESCRIPTION¶

Sorting files based on image properties¶

Usage summary¶

Positional Arguments¶

Named Arguments¶

DESCRIPTION¶

EXAMPLES¶

Running the complete standard workflow: `run_standard_header_process.py`¶

Header patching: `run_patch.py`¶

Adding astrometry: `run_astromtery.py`¶

Find problems and create summary: `run_triage.py`¶

Manual intervention: `quick_add_keys_to_file.py`¶