Developer Guide¶
Contributing Guide¶
Work in progress!
Recommended Python Setup¶
We recommend the following setup for Python development when contributing to data-describe:
To create a new conda environment for development, run:
conda create -n test-env
conda env update -n test-env -f etc/test-environment.yml
Pre-commit¶
Pre-commit is also recommended for running linting and style checks. To install pre-commit:
pip install pre-commit
pre-commit install
The checks will now run on commit.
Optional Dependencies¶
data-describe may make use of optional dependencies such as nltk
or modin
. When adding or using these optional dependencies in data-describe modules, the following patterns should be used:
_requires
marks functionality that requires a dependency¶
Use the _requires
decorator on any object function or class that needs the optional dependency. Usage:
from data_describe.compat import _requires
@_requires("nltk")
def function_that_uses_nltk():
return
_requires
should generally take the top-level package name as its sole argument. See the section on packages vs subpackages for more information.
_compat
is used to lazily import from dependencies¶
Instead of having import statements at the top of the file, import and use the _compat
object to use functionalities from the optional dependency:
from data_describe.compat import _requires, _compat
@_requires("nltk")
def function_that_uses_nltk_freqdist():
_compat["nltk"].FreqDist()
_compat
should generally take the sub-package as its key in a dictionary-style access. See the section on packages vs subpackages for more information.
packages vs subpackages¶
Some packages do not export all of their subpackages. For example, import statsmodels
does not provide access to statsmodels.graphics.tsaplots
, as the graphics
subpackage is not exported.
As a result, _requires
generally takes the top-level package name, as this checks if the package itself is installed. In contrast, _compat
takes the subpackage to enable imports.
One exception to this are the google client libraries such as google-cloud-storage
or google-cloud-bigquery
. Each of these are installed individually, but they are organized as subpackages of the google
namespace i.e. google.cloud.storage
. In this case, _requires
should instead be the specific subpackage (i.e. _requires("google.cloud.storage")
) since requiring only the google
package is not specific enough.
Side imports¶
Some packages require downloads of additional data or models to function. One example is the stopwords for nltk
. Downloading of these resources is handled in data_describe/compat/_dependency.py
. When adding a dependency that requires this download, adhere to the following steps:
Add a function that takes the module as its sole argument, checks for the existence of the resource (i.e if it was already downloaded), and executes the download if it doesn’t exist.
Add this function to the module-import mapping used to initialize
_compat
:
_compat = DependencyManager(
{
"nltk": nltk_download,
"spacy": spacy_download,
# "new_package": downloader_function,
}
)
Add to extras_require
in setup.py¶
The new dependency should be added to the extras_require
in setup.py. If applicable, try to use existing tags over creating new ones. New tags should be alphabetical and short.
Add to conda environments¶
The new dependency should be added to all conda environment definitions. These are located in two locations: etc/*.yml
and docker/*/*.yml
Docstring Checking¶
darglint can optionally be used to check if docstrings are outdated. It hasn’t been added to the pre-commit hooks because it can take a long time to parse.
To run darglint, use the following command:
darglint -v 2 --strictness=short <FILES>
Automatic Documentation¶
Documentation is generated using Sphinx. To run the typical build process, use the provided conda environment definition at etc/doc-environment.yml
and run docs/make.py
.
Note that sphinx-multiversion
is used to build documentation for multiple versions of data-describe
and only captures changes in tagged commits and/or the master branch on remote.
Notebook Update¶
Example notebooks are stored in the examples/
folder for easy access in the Github repository. Run docs/update_notebook_docs.py
to copy these notebooks (if any updates) into the docs/source
directory and update the index.
Manual Build¶
To test a build of the Sphinx-generated documentation without using sphinx-multiversion
, you can run the following command:
sphinx-build -a -E docs/source docs/build
The generated HTML files will be in docs/build