Command-line scripts¶

Each of the command-line scripts described below is also callable from python. The details of how you call it from python are described below.

Both these ways of invoking the script (from the command line or from python) is are wrappers around the python functions that do the real work. References to those functions, which tend to provide more control over what you can do at the expense of taking more effort to understand, are provided below where appropriate.

Running the complete standard workflow: `run_standard_header_process.py`¶

Usage summary¶

usage: run_standard_header_process.py [-h]
                                      (--dest-root DEST_ROOT | --overwrite-source)
                                      [--scripts-only] [-r {a,t,p}]
                                      [-o OBJECT_LIST] [--quiet-console]
                                      [--silent-console] [--debug]
                                      [--quiet-log]
                                      source_root

Positional arguments:

source_root

All directories below this one that contain images will be processed

Options:

`--dest-root`	If set, image directories below ``source-root`` will be copied into this directory tree. Only directories that contain image files will be copied; any intermediary directories required to contain directories that contain images will also be created.
`--overwrite-source=False`
	This flag must be used to overwrite images in the course directory.
`--scripts-only=False`
	This script will write a single shell script with the name provided in this option. No images will be modified or directories created, but the script can be run to do those things.
`-r, --run-only`	Select which scripts you want to run. This can be any combination of [p]atch, [a]strometry and [t]riage. Possible choices: a, t, p
`-o, --object-list`
	Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed
`--quiet-console=False`
	Log only errors (or worse) to console while running scripts
`--silent-console=False`
	Turn off all logging output to console
`--debug=False`	Turn on very detailed logging output
`--quiet-log=False`
	Log only warnings (or worse) to FILES AND CONSOLE while running scripts

Header patching: `run_patch.py`¶

For a detailed description of which header keywords are modified see Keywords purged before further processing.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary¶

usage: run_patch.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                    [--quiet-console] [--silent-console] [-o OBJECT_LIST]
                    dir [dir ...]

Positional arguments:

dir

Directory to process

Options:

`-v=False, --verbose=False`
	provide more information during processing
`-d, --destination-dir`
	Directory in which output from this script will be stored
`--debug=False`	Turn on very detailed logging output
`-n=False, --no-log-destination=False`
	Do not write log files to destination directory
`--quiet-console=False`
	Log only errors (or worse) to console while running scripts
`--silent-console=False`
	Turn off all logging output to console
`-o=https://raw.github.com/mwcraig/feder-object-list/master/feder_object_list.csv, --object-list=https://raw.github.com/mwcraig/feder-object-list/master/feder_object_list.csv`
	Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed

DESCRIPTION ———– For each directory provided on the command line the headers all of the FITS files in that directory are modified to add information like LST, apparent object position, and more. See the full documentation for a list of the specific keywords that are modified. Header patching ^^^^^^^^^^^^^^^ This is basically a wrapper around the function :func:`patch_headers.patch_headers` with the options set so that: + “Bad” keywords written by MaxImDL 5 are purged. + ``IMAGETYP`` keyword is changed from default MaxIM DL style to IRAF style (e.g. “Bias Frame” to “BIAS”) + Additional useful times like LST, JD are added to the header. + Apparent position (Alt/Az, hour angle) are added to the header. + Information about overscan is added to the header. + Files are overwritten. For more control over what is patched and where the patched files are saved see the documentation for ``patch_headers`` at :func:`patch_headers.patch_headers`. Adding OBJECT keyword ^^^^^^^^^^^^^^^^^^^^^ ``run_patch`` also adds the name of the object being observed when appropriate (i.e. only for light files) and possible. It needs to be given a list of objects; looking up the coordinates for those objects requires an Internet connection. See For a detailed description of the object list file see :func:`Object file format <patch_headers.read_object_list>`. for a detailed description of the function that actually adds the object name see :func:`patch_headers.add_object_info`. If no object list is specified or present in the directory being processed the `OBJECT` keyword is simply not added to the FITS header. .. Note:: This script is **NOT RECURSIVE**; it will not process files in subdirectories of the the directories supplied on the command line. .. WARNING:: This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option. EXAMPLES ——– Invoking this script from the command line:: run_patch.py /my/folder/of/images To work on the same folder from within python, do this:: from msumastro.scripts import run_patch run_patch.main([‘/my/folder/of/images’]) To use the same object list for several different directories do this:: run_patch.py –object-list path/to/list.txt dir1 dir2 dir3 where ``path/to/list.txt`` is the path to your object list and ``dir1``, ``dir2``, etc. are the directories you want to process. From within python this would be:: from msumastro.scripts import run_patch run_patch.main([‘–object-list’, ‘path/to/list.txt’, ‘dir1’, ‘dir2’, ‘dir3’])

DESCRIPTION¶

For each directory provided on the command line the headers all of the FITS files in that directory are modified to add information like LST, apparent object position, and more. See the full documentation for a list of the specific keywords that are modified.

Header patching¶

This is basically a wrapper around the function patch_headers.patch_headers() with the options set so that:

“Bad” keywords written by MaxImDL 5 are purged.

IMAGETYP keyword is changed from default MaxIM DL style to IRAF style (e.g. “Bias Frame” to “BIAS”)

Additional useful times like LST, JD are added to the header.

Apparent position (Alt/Az, hour angle) are added to the header.

Information about overscan is added to the header.

Files are overwritten.

For more control over what is patched and where the patched files are saved see the documentation for patch_headers at patch_headers.patch_headers().

Adding OBJECT keyword¶

run_patch also adds the name of the object being observed when appropriate (i.e. only for light files) and possible. It needs to be given a list of objects; looking up the coordinates for those objects requires an Internet connection. See

For a detailed description of the object list file see Object file format.

for a detailed description of the function that actually adds the object name see patch_headers.add_object_info().

If no object list is specified or present in the directory being processed the OBJECT keyword is simply not added to the FITS header.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES¶

Invoking this script from the command line:

run_patch.py /my/folder/of/images

To work on the same folder from within python, do this:

from msumastro.scripts import run_patch
run_patch.main(['/my/folder/of/images'])

To use the same object list for several different directories do this:

run_patch.py --object-list path/to/list.txt dir1 dir2 dir3

where path/to/list.txt is the path to your object list and dir1, dir2, etc. are the directories you want to process.

From within python this would be:

from msumastro.scripts import run_patch
run_patch.main(['--object-list', 'path/to/list.txt',
               'dir1', 'dir2', 'dir3'])

Adding astrometry: `run_astromtery.py`¶

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary¶

usage: run_astrometry.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                         [--quiet-console] [--silent-console] [-b]
                         dir [dir ...]

Positional arguments:

dir

Directory to process

Options:

`-v=False, --verbose=False`
	provide more information during processing
`-d, --destination-dir`
	Directory in which output from this script will be stored
`--debug=False`	Turn on very detailed logging output
`-n=False, --no-log-destination=False`
	Do not write log files to destination directory
`--quiet-console=False`
	Log only errors (or worse) to console while running scripts
`--silent-console=False`
	Turn off all logging output to console
`-b=False, --blind=False`
	Turn ON blind astrometry; disabled by default because it is so slow.

DESCRIPTION ———– For each directory provided on the command line add astrometry to the light files (those with ``IMAGETYP=’LIGHT’`` in the FITS header). By default, astrometry is added only for those files with pointing information in the header (specifically, RA and Dec) because blind astrometry is fairly slow. It may be faster to insert RA/Dec into those files before doing astrometry. The functions called by this script set the WCS reference pixel to the center of the image, which turns out to make aligning images a little easier. For more control over the parameters see :func:`add_astrometry` and for even more control, :func:`call_astrometry`. .. Note:: This script is **NOT RECURSIVE**; it will not process files in subdirectories of the the directories supplied on the command line. .. WARNING:: This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option. EXAMPLES ——– Invoking this script from the command line:: run_astrometry.py /my/folder/of/images To work on the same folder from within python, do this:: from msumastro.scripts import run_astrometry run_astrometry.main([‘/my/folder/of/images’])

DESCRIPTION¶

For each directory provided on the command line add astrometry to the light files (those with IMAGETYP='LIGHT' in the FITS header).

By default, astrometry is added only for those files with pointing information in the header (specifically, RA and Dec) because blind astrometry is fairly slow. It may be faster to insert RA/Dec into those files before doing astrometry.

The functions called by this script set the WCS reference pixel to the center of the image, which turns out to make aligning images a little easier.

For more control over the parameters see add_astrometry() and for even more control, call_astrometry().

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES¶

Invoking this script from the command line:
run_astrometry.py /my/folder/of/images
To work on the same folder from within python, do this:
from msumastro.scripts import run_astrometry
run_astrometry.main(['/my/folder/of/images'])

Find problems and create summary: `run_triage.py`¶

Usage summary¶

usage: run_triage.py [-h] [--debug] [-v] [-d DESTINATION_DIR] [-n]
                     [--quiet-console] [--silent-console] [-k KEY] [-l] [-a]
                     [-t TABLE_NAME] [-o OBJECT_NEEDED_LIST]
                     [-p POINTING_NEEDED_LIST] [-f FILTER_NEEDED_LIST]
                     [dir [dir ...]]

Positional arguments:

dir

Directory to process

Options:

`--debug=False`	Turn on very detailed logging output
`-v=False, --verbose=False`
	provide more information during processing
`-d, --destination-dir`
	Directory in which output from this script will be stored
`-n=False, --no-log-destination=False`
	Do not write log files to destination directory
`--quiet-console=False`
	Log only errors (or worse) to console while running scripts
`--silent-console=False`
	Turn off all logging output to console
`-k=[], --key=[]`
	FITS keyword to add to table in addition to the defaults; for multiple keywords use this option multiple times.
`-l=False, --list-default=False`
	Print default list keywords put into table and exit
`-a=False, --all=False`
	Construct table from all FITS keywords present in headers and the list of default keywords.
`-t=Manifest.txt, --table-name=Manifest.txt`
	Name of file in which table is saved; default is Manifest.txt
`-o=NEEDS_OBJECT_NAME.txt, --object-needed-list=NEEDS_OBJECT_NAME.txt`
	Name of file to which list of files that need object name is saved; default is NEEDS_OBJECT_NAME.txt
`-p=NEEDS_POINTING_INFO.txt, --pointing-needed-list=NEEDS_POINTING_INFO.txt`
	Name of file to which list of files that need pointing name is saved; default is NEEDS_POINTING_INFO.txt
`-f=NEEDS_FILTER.txt, --filter-needed-list=NEEDS_FILTER.txt`
	Name of file to which list of files that need filter is saved; default is NEEDS_FILTER.txt

DESCRIPTION ———– For each directory provided on the command line create a table in that directory with one row for each FITS file in the directory. The columns are FITS keywords extracted from the header of each file. The list of default keywords extracted is available through the command line option ``–list-default``. .. Note:: This feature is available only from the command line. For more control over the parameters see :func:`triage_fits_files` .. Note:: This script is **NOT RECURSIVE**; it will not process files in subdirectories of the the directories supplied on the command line. EXAMPLES ——– Invoking this script from the command line:: python run_triage.py /my/folder/of/images Get list of default keywords included in summary table:: python run_triage.py –list-default To work on the same folder from within python, do this:: from msumastro.scripts import run_triage run_triage.main([‘/my/folder/of/images’]) # or... run_triage.main([‘–list-default’])

DESCRIPTION¶

For each directory provided on the command line create a table in that directory with one row for each FITS file in the directory. The columns are FITS keywords extracted from the header of each file.

The list of default keywords extracted is available through the command line option --list-default.

Note

This feature is available only from the command line.

For more control over the parameters see triage_fits_files()

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

EXAMPLES¶

Invoking this script from the command line:
python run_triage.py /my/folder/of/images
Get list of default keywords included in summary table:
python run_triage.py --list-default
To work on the same folder from within python, do this:
from msumastro.scripts import run_triage
run_triage.main(['/my/folder/of/images'])
# or...
run_triage.main(['--list-default'])

Manual intervention: `quick_add_keys_to_file.py`¶

Warning

This script OVERWRITES the image files in the directories specified on the command line. There is NO WAY TO DISABLE this behavior.

Usage summary¶

usage: quick_add_keys_to_file.py [-h] [--file-list FILE_LIST]
                                 (--key-file KEY_FILE | --key-value KEY_VALUE KEY_VALUE)
                                 [files [files ...]]

Positional arguments:

files

Files in which to add/change keywords

Options:

`--file-list`	File with list of files in which keywords are to be changed
`--key-file`	File with keywords and values to be set
`--key-value`	Keyword to add/change

Add/modify keywords in FITS files. DESCRIPTION ———– Add each of the keywords in either the ``key_file`` or specified on the command line to each of the files either listed in the file ``file_list`` or specified on the command line. If the keyword is already present its value is updated to the value in ``key_file``. A HISTORY comment is added to the header for each keyword indicating which keyword was modified. .. WARNING:: This script OVERWRITES the image files in the list specified on the command line. There is NO WAY to override this behavior.

Add/modify keywords in FITS files.

DESCRIPTION¶

Add each of the keywords in either the key_file or specified on the command line to each of the files either listed in the file file_list or specified on the command line. If the keyword is already present its value is updated to the value in key_file. A HISTORY comment is added to the header for each keyword indicating which keyword was modified.

Warning

This script OVERWRITES the image files in the list specified on the command line. There is NO WAY to override this behavior.

Sorting files based on image properties¶

Note

By default this script makes a copy of the images being sorted. There is an option for moving the files instead.

This type of sorting is handy for working with images in GUI software like AstroImageJ or MaxImDL, but will make it harder to process the data programmatically in python.

Usage summary¶

usage: sort_files.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                     [--quiet-console] [--silent-console] [--move]
                     dir [dir ...]

Positional arguments:

dir

Directory to process

Options:

`-v=False, --verbose=False`
	provide more information during processing
`-d, --destination-dir`
	Directory in which output from this script will be stored
`--debug=False`	Turn on very detailed logging output
`-n=False, --no-log-destination=False`
	Do not write log files to destination directory
`--quiet-console=False`
	Log only errors (or worse) to console while running scripts
`--silent-console=False`
	Turn off all logging output to console
`--move=False, -m=False`
	Move files instead of copying them.

DESCRIPTION ———– For the directory provided on the command line sort the FITS files in this way:: destination | | | |—’BIAS’ | |—’DARK’ | |—exposure_time_1 | |—exposure_time_2, etc. | |—’FLAT’ | |—filter_1 | | |—exposure_time_1 | | |—exposure_time_2, etc. | | | |—filter_2, etc. | |—’LIGHT’ |—object_1 | |—filter_1 | | |—exposure_time_1 | | |—exposure_time_2, etc. | | | |—filter_2, etc. | |—object_2 | |—filter_1 | |—filter_2, etc. | |—object_3, etc. | |—’no_object’ |—filter_1 |—filter_2, etc. The names in single quotes, like `’bias’`, appear exactly as written in the directory tree created. Names like `exposure_time_1` are replaced with a value, for example 30.0 if the first dark exposure time is 30.0 seconds. The directory ``destination/calibration/flat/R`` will contain all of the FITS files that are R-band flats. .. Note:: This script is **NOT RECURSIVE**; it will not process files in subdirectories of the the directories supplied on the command line. .. Warning:: Unless you explicitly supply a destination using the –destination-dir option the files will be copied/moved in the directory in which they currently exist. While this *should not* lead to information loss, since files are moved or copied but never deleted, you have been warned. EXAMPLES ——–

DESCRIPTION¶

For the directory provided on the command line sort the FITS files in this way:
destination
    |
    |
    |
    |---'BIAS'
    |
    |---'DARK'
    |     |---exposure_time_1
    |     |---exposure_time_2, etc.
    |
    |---'FLAT'
    |      |---filter_1
    |      |    |---exposure_time_1
    |      |    |---exposure_time_2, etc.
    |      |
    |      |---filter_2, etc.
    |
    |---'LIGHT'
            |---object_1
            |       |---filter_1
            |       |    |---exposure_time_1
            |       |    |---exposure_time_2, etc.
            |       |
            |       |---filter_2, etc.
            |
            |---object_2
            |       |---filter_1
            |       |---filter_2, etc.
            |
            |---object_3, etc.
            |
            |---'no_object'
                    |---filter_1
                    |---filter_2, etc.
The names in single quotes, like ‘bias’, appear exactly as written in the directory tree created. Names like exposure_time_1 are replaced with a value, for example 30.0 if the first dark exposure time is 30.0 seconds.

The directory destination/calibration/flat/R will contain all of the FITS files that are R-band flats.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

Unless you explicitly supply a destination using the –destination-dir option the files will be copied/moved in the directory in which they currently exist. While this should not lead to information loss, since files are moved or copied but never deleted, you have been warned.

Command-line scripts¶

Running the complete standard workflow: run_standard_header_process.py¶

Usage summary¶

Header patching: run_patch.py¶

Usage summary¶

DESCRIPTION¶

Header patching¶

Adding OBJECT keyword¶

EXAMPLES¶

Adding astrometry: run_astromtery.py¶

Usage summary¶

DESCRIPTION¶

EXAMPLES¶

Find problems and create summary: run_triage.py¶

Usage summary¶

DESCRIPTION¶

EXAMPLES¶

Manual intervention: quick_add_keys_to_file.py¶

Usage summary¶

DESCRIPTION¶

Sorting files based on image properties¶

Usage summary¶

DESCRIPTION¶

EXAMPLES¶

Running the complete standard workflow: `run_standard_header_process.py`¶

Header patching: `run_patch.py`¶

Adding astrometry: `run_astromtery.py`¶

Find problems and create summary: `run_triage.py`¶

Manual intervention: `quick_add_keys_to_file.py`¶