Commit afadeb12 authored by karpov-sv's avatar karpov-sv
Browse files

Transient filtering documentation added

parent 02fd9859
...@@ -333,3 +333,6 @@ apidoc_excluded_paths = ['tests'] ...@@ -333,3 +333,6 @@ apidoc_excluded_paths = ['tests']
apidoc_separate_modules = True apidoc_separate_modules = True
apidoc_toc_file = None apidoc_toc_file = None
apidoc_module_first = True apidoc_module_first = True
# Autodoc
autodoc_member_order = 'bysource'
...@@ -14,7 +14,7 @@ The model includes: ...@@ -14,7 +14,7 @@ The model includes:
and its fitting is implemented in :func:`stdpipe.photometry.match` function, and wrapped into a bit higher-level and easier to use :func:`stdpipe.pipeline.calibrate_photometry`. The routines perform an iterative fitting with rejection of pairs deviating too much (more than `threshold` sigmas) from the model that is defined like that: and its fitting is implemented in :func:`stdpipe.photometry.match` function, and wrapped into a bit higher-level and easier to use :func:`stdpipe.pipeline.calibrate_photometry`. The routines perform an iterative fitting with rejection of pairs deviating too much (more than `threshold` sigmas) from the model that is defined like that:
.. math:: .. math::
{\rm Catalogue} = {\rm Instrumental} + C\cdot{\rm color} + ZP(x, y, {\rm spatial_order}) + {\rm additive} + {\rm error} {\rm Catalogue} = {\rm Instrumental} + C\cdot{\rm color} + ZP(x, y, {\rm spatial\_order}) + {\rm additive} + {\rm error}
where where
...@@ -29,7 +29,7 @@ Zero point :math:`ZP` is a spatially varying polynomial with the degree controll ...@@ -29,7 +29,7 @@ Zero point :math:`ZP` is a spatially varying polynomial with the degree controll
Additive flux term is defined by linearizing the additional flux in every photometric aperture (e.g. due to incorrect background level determination) and has a form Additive flux term is defined by linearizing the additional flux in every photometric aperture (e.g. due to incorrect background level determination) and has a form
.. math:: .. math::
{\rm additive} = -2.5/\log(10)/10^{-0.4\cdot{\rm Instrumental}} \cdot {\rm bg_corr}(x, y, {\rm bg_order}) {\rm additive} = -2.5/\log(10)/10^{-0.4\cdot{\rm Instrumental}} \cdot {\rm bg_corr}(x, y, {\rm bg\_order})
where :math:`{\rm bg_corr}` is a flux correction inside the aperture. This term is also spatially dependent, and is controlled by `bg_order` parameter. If :code:`bg_order=None`, the fitting for this term is disabled. where :math:`{\rm bg_corr}` is a flux correction inside the aperture. This term is also spatially dependent, and is controlled by `bg_order` parameter. If :code:`bg_order=None`, the fitting for this term is disabled.
......
Transient detection and filtering
=================================
Transients may be detected either on initial *science* image as described in :ref:`Object detection and measurement`, or on the difference image after image subtraction. For the latter, you may also make use of a difference image noise model, as returned by :func:`stdpipe.subtraction.run_hotpants` with :code:`get_noise=True` - it will reduce the amount of subtraction artefacts at the position of brighter objects.
.. code-block:: python
# Get PSF model of and store it to temporary file
psf_model = psf.run_psfex(image, mask=mask, order=0, gain=gain, psffile='/tmp/psf.psf', verbose=True)
# Run SExtractor on difference image with custom noise model,
# using both image and template masks for masking
sobj = photometry.get_objects_sextractor(diff, mask=mask|tmask, err=ediff,
edge=10, wcs=wcs, aper=5.0, psf='/tmp/psf.psf', verbose=True)
# Perform forced aperture photometry in a circular aperture with 1.0*FWHM radius,
# measuring local backround in an annulus between 5.0*FWHM and 7.0*FWHM,
# and again with custom noise model and forced to zero global background level
sobj = photometry.measure_objects(sobj, diff, mask=mask|tmask, fwhm=fwhm, aper=1.0, bkgann=[5, 7],
sn=3, verbose=True, bg=0, err=ediff)
# The difference is in original image normalization, so we know photometric zero point
sobj['mag_calib'] = sobj['mag'] + m['zero_fn'](sobj['x'], sobj['y'])
sobj['mag_calib_err'] = np.hypot(sobj['magerr'], m['zero_fn'](sobj['x'], sobj['y'], get_err=True))
# We may immediately reject flagged objects as they correspond to imaging artefacts (masked regions)
sobj = sobj[sobj['flags'] == 0]
print(len(sobj), 'transient candidates found in difference image')
The transient candidates detected by such a routine will require some filtering in order to reject the subtraction artefacts, probably filter out known variable stars and maybe catalogued stars, also filter out known minor planets, etc. We have a convenience high-level routine :func:`stdpipe.pipeline.filter_transient_candidates` that uses a number of lower-level ones (:func:`stdpipe.catalogs.xmatch_objects`, :func:`stdpipe.catalogs.xmatch_skybot`, :func:`stdpipe.catalogs.xmatch_ned`). High-level routine is able to either filter out the "bad" objects from the list of candidates, or just mark the ones that should be rejected for various reasons (if :code:`remove=False`).
.. code-block:: python
# Filter out all candidates that match the objects from Pan-STARRS DR1 or AAVSO VSX catalogues
# Also filter out all Solar System minor planets known to IMCCE SkyBoT service
# Also, filter out all entries having flags 0x100 or 0x200 (thus, corresponding to objects
# with masked pixels in their footprints
candidates = pipeline.filter_transient_candidates(sobj, sr=2/3600,
flagged=True, vizier=['ps1', 'vsx'], skybot=True, time=time, verbose=True)
# ...or, we may do the same without removing the candidates:
candidates = pipeline.filter_transient_candidates(sobj, sr=2/3600, remove=False,
flagged=True, vizier=['ps1', 'vsx'], skybot=True, time=time, verbose=True)
# and remove them later manually
candidates = candidates[candidates['candidate_good']==True]
# and now just print all the remaining candidates
for i,cand in enumerate(candidates):
print('Candidate %d with mag = %.2f +/- %.2f at x/y = %.1f %.1d and RA/Dec = %.4f %.4f' %
(i, cand['mag_calib'], cand['mag_calib_err'], cand['x'], cand['y'], cand['ra'], cand['dec']))
.. autofunction:: stdpipe.pipeline.filter_transient_candidates
:noindex:
...@@ -46,6 +46,7 @@ Data processing ...@@ -46,6 +46,7 @@ Data processing
astrometry astrometry
photometry photometry
subtraction subtraction
transients
Common principles Common principles
----------------- -----------------
......
"""
Module containing the routines for handling various online catalogues.
"""
from __future__ import absolute_import, division, print_function, unicode_literals from __future__ import absolute_import, division, print_function, unicode_literals
import os, posixpath, shutil, tempfile import os, posixpath, shutil, tempfile
...@@ -207,6 +211,19 @@ def xmatch_skybot(obj, sr=10/3600, time=None, col_ra='ra', col_dec='dec', col_id ...@@ -207,6 +211,19 @@ def xmatch_skybot(obj, sr=10/3600, time=None, col_ra='ra', col_dec='dec', col_id
return xcat return xcat
def xmatch_ned(obj, sr=3/3600, col_ra='ra', col_dec='dec', col_id='id'): def xmatch_ned(obj, sr=3/3600, col_ra='ra', col_dec='dec', col_id='id'):
"""Cross-match object list with NED database entries
The routine is extremely inefficient as it has to query the objects one by one!
:param obj: astropy.table.Table with objects
:param sr: Cross-matching radius in degrees
:param col_ra: Column name in `obj` table containing Right Ascension values
:param col_dec: Column name in `obj` table containing Declination values
:param col_id: Column name in `obj` table containing some unique object identifier
:returns: The table of NED objects augmented with `id` column containing the identifiers from `col_id` columns of matched objects.
"""
xcat = [] xcat = []
# FIXME: is there more optimal way to query NED for multiple sky positions?.. # FIXME: is there more optimal way to query NED for multiple sky positions?..
......
...@@ -98,22 +98,45 @@ def refine_astrometry(obj, cat, sr=10/3600, wcs=None, order=0, ...@@ -98,22 +98,45 @@ def refine_astrometry(obj, cat, sr=10/3600, wcs=None, order=0,
return wcs return wcs
def filter_transient_candidates(obj, sr=None, pixscale=None, time=None, def filter_transient_candidates(obj, sr=None, pixscale=None, time=None,
obj_col_ra = 'ra', obj_col_dec = 'dec',
cat=None, cat_col_ra='RAJ2000', cat_col_dec='DEJ2000', cat=None, cat_col_ra='RAJ2000', cat_col_dec='DEJ2000',
vizier=['ps1', 'usnob1', 'gsc'], skybot=True, ned=False, vizier=['ps1', 'usnob1', 'gsc'], skybot=True, ned=False,
flagged=True, flagmask=0xff00, flagged=True, flagmask=0xff00,
col_id=None, get_candidates=True, remove=True, verbose=False): col_id=None, get_candidates=True, remove=True, verbose=False):
""" """Higher-level transient candidate filtering routine.
Higher-level transient candidate filtering routine.
It optionally filters out the following classes of objects: It optionally filters out the following classes of objects:
- flagged ones, i.e. with obj['flags'] != 0
- positionally coincident with stars from provided cataloge table (if cat != None) - flagged ones, i.e. with :code:`obj['flags'] != 0`
- positionally coincident with stars from Vizier catalogues provided as a list of names (if vizier is non-empty) - positionally coincident with stars from provided cataloge table (if :code:`cat != None`)
- positionally and temporally coincident with Solar system objects from SkyBoT service (if skybot = True) - positionally coincident with stars from Vizier catalogues provided as a list of names (if `vizier` is non-empty)
- positionally and temporally coincident with NED objects (if ned = True) - positionally and temporally coincident with Solar system objects from SkyBoT service (if :code:`skybot = True`)
- positionally and temporally coincident with NED objects (if :code:`ned = True`).
If get_candidates = False, it returns only the indices of "good" objects, else returning filtered object list
If get_candidates = True and remove = False, it does not filter the objects but just mark them in added columns If :code:`get_candidates = False`, it returns only the indices of "good" objects, else returning filtered object list.
If :code:`get_candidates = True` and :code:`remove = False`, it does not remove the objects but just mark them in added columns corresponding to various filters applied.
The routine will not modify the original list, but return its filtered / modified copy instead.
:param obj: Input object list
:param obj_col_ra: Column name for object Right Ascension
:param obj_col_dec: Column name for object Declination
:param sr: Matching radius in degrees
:param pixscale: Pixel scale. If provided, and `sr` is not specified, the latter will be set to half median of FWHMs of objects
:param time: Time corresponding to the object observations, as :class:`astropy.time.Time` or :class:`datetime.datetime` object.
:param cat: Input reference catalogue, as :class:`astropy.table.Table` or similar object. Optional
:param cat_col_ra: Column name for catalogue Right Ascension
:param cat_col_dec: Column name for catalogue Declination
:param vizier: List of Vizier catalogue identifiers, or their short names, to cross-match the objects with.
:param skybot: Whether to cross-match the objects with the positions of known minor planets at the time of observations (specified through `time` parameter)
:param ned: Whether to cross-match the positions of objects with NED database entries
:param flagged: Whether to filter out flagged objects, keeping only the ones with :code:`(obj['flags'] & flagmask) == 0`
:param flagmask: The mask to be used for filtering of flagged objects. Will be ANDed with object flags, so should have a bit set for every relevant mask bit
:param col_id: Column name for some unique identifier that should not appear twice in the object list. If not specified, new column `stdpipe_id` with unique integer identifiers will be created automatically
:param get_candidates: Whether to return the list of candidates, or (if :code:`get_candidates=False`) just a boolean mask of objects surviving the filters
:param remove: Whether to remove the filtered entries from the returned list. If not, a number of additional columns (all having names `candidate_*`) will be added for every filter used, with `True` set if the filter matches the object. Finally, the boolean column `candidate_good` will have `True` for the objects surviving all the filters
:param verbose: Whether to show verbose messages during the run of the function or not. May be either boolean, or a `print`-like function.
:returns: Either the copy of the list with "good" candidates, or (if :code:`get_candidates=False`) just a boolean mask of objects surviving the filters, with the same size as original objects list.
""" """
# Simple wrapper around print for logging in verbose mode only # Simple wrapper around print for logging in verbose mode only
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment