Stack Club

The LSST science collaborations’ Stack Club is learning the LSST software “stack” by writing tutorial Jupyter notebooks about it. These notebooks are organized by topic area, and can be browsed at the links below. There is also the stackclub package of useful python tools that you can import and use - click through below to learn more about them.

Tutorial Notebooks

Getting Started

Wondering how you can get started learning about the LSST software stack, by writing tutorial notebooks and contributing them to the Stack Club’s growing library? Need help getting going on the LSST Science Platform (LSP) JupyterLab? See the index table below for links to various resources, including: notes on the LSP, notebooks to walk you through the Stack Club workflow, and some help on how to explore the Stack code. Click on the “rendered” links to see the notebooks with their outputs.

Notebook	Short description	Links	Owner
Notes on Getting Started	Some brief notes on the LSST Science Platform JupyterLab set-up.	markdown	Phil Marshall
Hello World	Read about the Stack Club git/GitHub workflow, and make your first contribution to a notebook.	ipynb, rendered	Phil Marshall
Templates	A folder containing a template notebook, and a template folder README file, to help you get your project started.	link	Phil Marshall
Finding Docs	Locate the documentation for Stack code objects, including using the stackclub library where_is utility function.	ipynb, rendered	Phil Marshall
Import Tricks	Learn how to use some stackclub library utilities for importing notebooks and remote modules.	ipynb, rendered	Phil Marshall

Basics

This set of tutorial notebooks will help you explore the basic properties of the LSST software Stack data structures, classes and functions. The table contains links to the notebook code, and also to auto-rendered views of the notebooks with their outputs.

Notebook	Short description	Links	Owner
Gen-3 Butler Tutorial	Demonstrates basic data access and manipulations using the Gen-3 Butler	ipynb, rendered	Alex Drlica-Wagner
Calexp Guided Tour	Shows how to read an exposure object from a data repository, and how to access and display various parts.	ipynb, rendered	David Shupe
AFW Table Guided Tour	Shows how to read and write catalogs using the AFW Table class, and how to access and display various parts.	ipynb, rendered	Imran Hasan

Visualization

See the table below for a set of tutorial notebooks (some provided by the Project) demonstrating visualization technologies available in the LSST Science Platform notebook aspect.

Notebook	Short description	Links	Owner
Visualizing Images with AFW Display	How to access the lsst.afw.display routines, and use the LSST data Butler to access processed image data and inspect it visually.	ipynb, rendered	Brant Robertson
Firefly Visualization Demo	Introduction to the Firefly interactive plotter and image viewer.	ipynb, video	Simon Krughoff
Interactive Visualization with Bokeh, HoloViews, and Datashader	Examples of interactive visualization with the Boken, HoloViews, and Datashader plotting packages available in PyViz suite of data analysis python modules; brushing and linking with large datasets	ipynb, rendered	Keith Bechtol
Visualizing the LSST Camera Focal Plane	Create a labeled visualization of the LSST Camera including amps, detectors, rafts integrated into the full focal plane.	ipynb, rendered	Alex Drlica-Wagner
Globular Cluster Intro	General purpose tutorial including interactive Firefly visualization of a globular cluster from LSST 2018.	ipynb	Jim Bosch

Image Processing

Here, we explore the image processing routines in the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.

Notebook	Short description	Links	Owner
Brighter-Fatter Correction	Analysis of beam simulator images and the brighter-fatter correction.	ipynb, rendered	Andrew Bradshaw

Source Detection

While source detection in the LSST science pipelines is carried out (first) during the image processing step, there are subsequent detection phases - and, moreover, we are interested in how sources are detected (and how their measured properties depends on that process). See the index table below for links to tutorial notebooks exploring this.

Notebook	Short description	Links	Owner
Low Surface Brightness	Run source detection, deblending, and measurement tasks; subtract bright sources from an image; convolve image and detect low-surface brightness sources.	ipynb, rendered	Alex Drlica-Wagner
Footprints	Investigate the concept of a footprint: the region of an image used for detecting and measuring source properties.	ipynb, rendered	Imran Hasan

Deblending

This folder contains a set of tutorial notebooks exploring the deblending of LSST objects. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.

Notebook	Short description	Links	Owner
Scarlet Tutorial	Standalone introduction to the scarlet deblender, including how to configure and run it.	ipynb, rendered	Fred Moolekamp
Deblending in the Stack	Where and how deblending happens in the LSST Science Pipelines.	ipynb, rendered	Fred Moolekamp

Measurement

This folder contains a set of tutorial notebooks exploring the Object measurement routines in the LSST science pipelines. See the index table below for links to the notebook code, and an auto-rendered view of the notebook with outputs.

Notebook	Short description	Links	Owner
Asteroid Light Curve	Find an asteroid from 2015 HiTS data, create postage stamps and generate a light curve. This notebook was used in the Summer 2020 Stack Club Course Session 06.	ipynb, rendered	Bryce Kalmbach
Undersampled Moments	Examine biases introduced by undersampled moments.	ipynb, rendered	Andrew Bradshaw
Resolved Dwarf Galaxy	Explore resolved sources around a nearby dwarf galaxy	ipynb, rendered	Jeff Carlin

Validation

This set of tutorial notebooks explores the validation packages accompanying the LSST software Stack, and also contains some stand-alone notebooks useful for examining various aspects of data quality.

Notebook	Short description	Links	Owner
Image Quality Demo	Examples of image shape measurements in the Stack including PSF size and ellipticity, shape measurements with and without PSF corrections; visualizing image quality statistics aggregated with pandas; examining PSF model ellipticity residuals	ipynb, rendered	Keith Bechtol
Verify Quality Demo	Examples use of LSST verify package	ipynb, rendered	Keith Bechtol

The stackclub Package

The Stack Club tutorial Jupyter notebooks make use of a number of homegrown functions and classes, which are kept in the stackclub package for easy import. You can browse these modules below.

Finding Documentation

There are a number of good places to find information about the classes and functions in the LSST software Stack: the built-in Jupyter notebook help() function already gets us a long way, but if you want to locate and read the source code, the stackclub.where_is function can help.

where_is.where_is(object, in_the='source', assuming_its_a=None)

Print a markdown hyperlink to the source code of object.

Parameters:

• object (python object or string) – The class or function you are looking for, or the name of a python object or file.
• in_the (string, optional) – The kind of place you want to look in: [‘source’, ‘repo’, ‘technotes’]
• assuming_its_a (string, optional) – The kind of object you think you have: `[‘cmdlinetask’], default=None

Examples

>>> from stackclub import where_is
>>> from lsst.daf.persistence import Butler
>>> where_is(Butler.get, in_the='source')
>>> where_is(Butler, in_the='repo')
>>> where_is(Butler, in_the='technotes')
>>> where_is("makeDiscreteSkyMap.py", in_the="source", assuming_its_a="cmdlinetask")

Notes

See also the FindingDocs tutorial notebook for a working demo.

Importing Notebooks as Modules

Once this module has been imported, further import statements will treat Jupyter notebooks as importable modules. It’s unlikely that you will need to call any of the functions or classes in nbimport yourself - this section is just for reference.

This module was adapted from the Jupyter notebook documentation (copyright (c) Jupyter Development Team, and distributed under the terms of the Modified BSD License) for use in the stackclub package.

class nbimport.NotebookFinder

Module finder that locates Jupyter Notebooks.

Notes

Once an instance of this class is appended to sys.meta_path, the import statement will work on notebook names.

Examples

To gain the ability to import notebooks, we just import the nbimport module. The DataInventory notebook might contain a useful function - here’s how we’d import it:

>>> import stackclub
>>> import DataInventory

We can also import remote notebooks, using wimport:

>>> import stackclub
>>> dm_butler_skymap_notebook = "https://github.com/LSSTDESC/DC2-analysis/raw/master/tutorials/dm_butler_skymap.ipynb"
>>> skymapper = stackclub.wimport(dm_butler_skymap_notebook, vb=True)

The DataInventory notebook provides a live demo of this example.

find_module(fullname, path=None)

Find the notebook module and return a suitable loader.

Parameters:

• fullname (string) – Name of the notebook to be found (without ipynb extension)
• path (string) – Path of folder containing notebook (optional).

Returns:

loaders[path] – Suitable loader object for dealing with Notebook import statements.

Return type:

NotebookLoader

class nbimport.NotebookLoader(path=None)

Module Loader for Jupyter Notebooks

load_module(fullname)

Import a notebook as a module

Parameters:fullname (string) – Name of notebook (without the .ipynb extension) Returns:mod – Notebook in module form, after it has been imported (executed). Return type:module

Notes

All code cells in the notebook are executed, silently (by redirecting the standard output).

nbimport.find_notebook(fullname, path=None)

Find a notebook, given its fully qualified name and an optional path.

Parameters:

• fullname (string) – Name of the notebook to be found (without ipynb extension)
• path (string, optional) – Path of folder containing notebook.

Returns:

nb_path – File name of notebook, if found (else None)

Return type:

string

Notes

The input notebook name “foo.bar” is turned into “foo/bar.ipynb”. Tries turning “Foo_Bar” into “Foo Bar” if Foo_Bar does not exist.

nbimport.stdoutIO(stdout=None)

Catch the stdout of the imported notebook cells.

Notes

Adapted from stackoverflow.com/questions/3906232 Note that this approach does not capture any rich notebook output, e.g. from IPython.display.

Importing Modules from the Web

This is pretty experimental!

wimport.wimport(url, vb=False)

Download a module and import it.

Parameters:

• url (string) – Web address of the target module
• vb (boolean, optional) – Verbose in operation [def=False]

Returns:

globals()[modulename] – The module, as imported.

Return type:

module

Notes

wimport maintains a secret local cache of downloaded modules, hidden from the user so that they are not tempted to edit the module locally. (If they need to do that, they should clone the relevant repo.)

Examples

Suppose the stackclub library did _not_ include the where_is module: we could still download it and import it, using wimport.

>>> where_is_url = "https://github.com/LSSTScienceCollaborations/StackClub/raw/issue/79/library/stackclub/where_is.py"
>>> from stackclub import wimport
>>> so = wimport(where_is_url, vb=True)
>>> so.where_is(Butler.get, in_the='source')