Developer Guide

Contributing Guide

Work in progress!

Optional Dependencies

data-describe may make use of optional dependencies such as nltk or modin. When adding or using these optional dependencies in data-describe modules, the following patterns should be used:

_requires marks functionality that requires a dependency

Use the _requires decorator on any object function or class that needs the optional dependency. Usage:

from data_describe.compat import _requires

@_requires("nltk")
def function_that_uses_nltk():
    return

_requires should generally take the top-level package name as its sole argument. See the section on packages vs subpackages for more information.

_compat is used to lazily import from dependencies

Instead of having import statements at the top of the file, import and use the _compat object to use functionalities from the optional dependency:

from data_describe.compat import _requires, _compat

@_requires("nltk")
def function_that_uses_nltk_freqdist():
    _compat["nltk"].FreqDist()

_compat should generally take the sub-package as its key in a dictionary-style access. See the section on packages vs subpackages for more information.

packages vs subpackages

Some packages do not export all of their subpackages. For example, import statsmodels does not provide access to statsmodels.graphics.tsaplots, as the graphics subpackage is not exported.

As a result, _requires generally takes the top-level package name, as this checks if the package itself is installed. In contrast, _compat takes the subpackage to enable imports.

One exception to this are the google client libraries such as google-cloud-storage or google-cloud-bigquery. Each of these are installed individually, but they are organized as subpackages of the google namespace i.e. google.cloud.storage. In this case, _requires should instead be the specific subpackage (i.e. _requires("google.cloud.storage")) since requiring only the google package is not specific enough.

Side imports

Some packages require downloads of additional data or models to function. One example is the stopwords for nltk. Downloading of these resources is handled in data_describe/compat/_dependency.py. When adding a dependency that requires this download, adhere to the following steps:

  1. Add a function that takes the module as its sole argument, checks for the existence of the resource (i.e if it was already downloaded), and executes the download if it doesn’t exist.

  2. Add this function to the module-import mapping used to initialize _compat:

_compat = DependencyManager(
    {
        "nltk": nltk_download,
        "spacy": spacy_download,
        # "new_package": downloader_function,
    }
)

Add to extras_require in setup.py

The new dependency should be added to the extras_require in setup.py. If applicable, try to use existing tags over creating new ones. New tags should be alphabetical and short.

Add to conda environments

The new dependency should be added to all conda environment definitions. These are located in two locations: etc/*.yml and docker/*/*.yml

Docstring Checking

darglint can optionally be used to check if docstrings are outdated. It hasn’t been added to the pre-commit hooks because it can take a long time to parse.

To run darglint, use the following command:

darglint -v 2 --strictness=short <FILES>

Automatic Documentation

Documentation is generated using Sphinx. To run the typical build process, use the provided conda environment definition at etc/doc-environment.yml and run docs/make.py.

Note that sphinx-multiversion is used to build documentation for multiple versions of data-describe and only captures changes in tagged commits and/or the master branch on remote.

Notebook Update

Example notebooks are stored in the examples/ folder for easy access in the Github repository. Run docs/update_notebook_docs.py to copy these notebooks (if any updates) into the docs/source directory and update the index.

Manual Build

To test a build of the Sphinx-generated documentation without using sphinx-multiversion, you can run the following command:

sphinx-build -a -E docs/source docs/build

The generated HTML files will be in docs/build