Contributing Guide

Contributing to data-describe

Thanks for taking the time to contribute to data-describe!

Code of Conduct

All contributors are expected to abide by the Code of Conduct.

Getting Started - Issues

The “to do” task list for data-describe starts at the Issues page in Github. Issues generally fall under one of two categories: - Bugs: Parts of data-describe that aren’t working as they should - Enhancements: Ideas for additions or changes to data-describe to make it better

Note: Before submitting a new issue, you should search the existing issues to make sure that it hasn’t already been reported.

Filing a bug report

Bug reports are important in helping identify things that aren’t working in data-describe. Filing a bug report is straightforward: Head to the Issues page, click on the New issue button, and select the Bug Report template to get started.

The first thing you’ll want to do is to create a short, descriptive title.

Next, you’ll want to fill out the contents of the Bug Report. The contents have been pre-filled with a template with guidance on common details that should be included in a bug report. Here are a few quick tips on how to read and fill out the template:

Some example sections are provided to help guide the contents of your report: ** Header Sections **

Comment lines are added in the template to provide guidance and will not show in the final output: <!-- Comments --> :

Triple back-tick marks can be used to add code examples:

```
def python_code():
```

Filing a Feature Request (Enhancement)

Filing a feature request follows the similar steps. However, since enhancements can be more subjective in nature than bug reports, it is recommended to follow the guidelines below when filing a feature request:

1. Incremental changes: Small, incremental changes (for example, clarifying the labels on a plot, or extending configuration support for a feature) can be started by using the Feature Request template when creating a new issue. Even for small changes, it’s recommended to describe your thought process - how and/or why the enhancement would be beneficial.

2. Brand new features / Large Reworks / System Redesign: For brand new features or other enhancements that may involve large changes, a design document should be written to explain and document the scope of the changes. A template for this design document can be found in the repository at docs/designs/TEMPLATE.md. This document can be submitted in a Pull Request, which is described further below.

Finding something to work on

For first time contributors, jumping head first into squashing bugs or designing new features can be overwhelming. It may be easiest to start by searching for the good first issue label. Issues that are tagged as good first issue typically involve small changes such as fixing errors in documentation.

Claiming/assigning an issue

To avoid overlapping work on issues, you should check to make sure that nobody else is actively working on the problem already. This is indicated by the Assignees section on the right side of the page for an issue.

1. If nobody is assigned to the issue, you should post a comment requesting to tackle the issue.

2. If somebody else is already assigned to the issue, and there hasn’t been any recent activity, post a comment to ask if the current assignee is still actively working on the issue. If they are unable or unwilling to complete work on the issue, or you don’t receive any response in a reasonable time frame (~ 2 weeks), the issue may be re-assigned.

Using Git / GitHub

The following section provides a walkthrough of using git/GitHub to edit the README on the front page.

1. Fork the repository: Only the primary project team has permissions to directly edit the main data-describe repository. As an external contributor, you will need to make a personal copy of the repository (a.k.a. “fork”) to begin making changes. To create a fork, click on the Fork button at the top right of the page of the repository on Github.

This will create a copy of the data-describe repository under your own GitHub account. (Visit “Your Repositories” under your account.)

2. Clone the repository: To access and edit your fork (copy) of the repository on your computer, you will need to clone the repository. First, you will need to ensure that you have the Git software installed on your computer. Then, open the terminal/shell of your choice (e.g. Terminal for Mac, Powershell for Windows) in the location where you want the repository copy to be saved on your computer.

For example, on Windows, SHIFT + RIGHT CLICK in a folder and select Open Powershell window here. Alternatively, one can use the cd commands to navigate to the desired folder.

Enter and run the following commands:

git clone https://github.com/<YOUR-USERNAME>/data-describe.git data-describe-<YOUR-USERNAME>
cd data-describe-<YOUR-USERNAME>
git remote add upstream https://github.com/data-describe/data-describe

where <YOUR-USERNAME> is your GitHub username.

3. Checkout a new branch: While this step is optional for forked repositories, it’s generally best practice to create a new branch for each thing you’re working on. Run the following command:

git checkout -b update-readme

where update-readme is the name of the branch. (For future contributions, replace this with a name of your choosing). This command creates the new branch and checks out the branch.

4. Adding your changes: Now open (using a text editor) and edit the README.md file by adding to the title.

Now that you’ve made a change to the repository, you’ll need to stage the changes so that it’s tracked by git. Run the command git add README.md.

where the README.md is the path to the file to be added, relative to the current working directory. To confirm that the file(s) have been added, you can run git status. You should see that the file has been added to Staged Changes.

5. Committing your changes: To save this changes to Git, you must commit the files. Run the command:

git commit -m "Add emoji to README title"

where the text inside the quotes is a description of the changes you’ve made in this commit.

6. Pushing your changes: To push your changes online for the first time, run:

git push origin --set-upstream update-readme

where --set-upstream update-readme creates the same update-readme branch online, in your fork on GitHub.

If you’re pushing more changes at a later time, you only need to run git push

7. Creating a Pull Request: To submit your changes for review, you will need to create a pull request. When a new branch is pushed to GitHub, a prompt will appear to create a pull request:

Then fill out the form with (at minimum) a title and description and create the pull request. At this point, a project team member or maintainer can review your pull request, provide comments, and officially merge it when approved.

Advanced Git Practices

Rebase

If you’re working on some changes for a long period of time, it’s possible that other contributors may have submitted other changes on the same files you’re working on. To sync your branch, run:

git pull origin master --rebase

and follow the prompts to approve and/or resolve the changes that should be kept.

Updating your Fork

To update the master branch of your fork (so that new branches are created off of an up-to-date master branch), run:

git fetch upstream

Pull Request Conventions

• The pull request title is used to build the release notes. Write the title in past tense, describing the extent of the changes.

• Pull Requests should have labels to identify which category belongs to in the release notes. Use the exclude notes label if the change doesn’t make sense to document in release notes.

• Pull Requests should be linked to issues, either manually or using keywords.

Python Developer Environment

The following setup is recommended for Python development:

• IDE: VS Code

  • Extensions:

    • ms-python.python

    • njpwerner.autodocstring

• Package/Environment Manager: Conda

  • Miniconda

To create a new conda environment for development, run:

conda create -n test-env
conda env update -n test-env -f etc/test-environment.yml

then activate using conda activate test-env or select it as your default Python interpreter for VSCode.

Python Extension Configuration

data-describe uses flake8 for linting. Configure VSCode by disabling pylint and enabling flake8. data-describe uses black for auto formatting. Configure VSCode by enabling black.

Pre-commit

Pre-commit is also strongly recommended for running linting and style checks. This will warn and prevent you from committing code that does not pass certain standards. To install pre-commit:

pip install pre-commit
pre-commit install --allow-missing-config

Code checks will now run prior to any commits you make.

data-describe currently utilizes the following tools for code checks: - black: Ensure consistent formatting of Python files - mypy: Validate Python type hints - flake8: Multiple checks for - import order - syntax errors or anti-patterns - (lack of) executable flags on files - docstring validation

See .pre-commit-config.yaml for a full list of hooks used by pre-commit.

Docstring Checks

(Optional) darglint can be used to check if docstrings are outdated. It hasn’t been added to the pre-commit hooks because it can take a long time to parse files.

To run darglint, use the following command:

darglint -v 2 --strictness=short <FILES>

Automatic Documentation

Documentation is generated using Sphinx.

Example notebooks are stored in the examples/ folder for easy access in the Github repository.

Notebook Execution

Run etc/run_notebooks.py to re-run notebooks. Using this script instead of manual execute ensures that cells are executed in order and that the kernelspec metadata is not updated to a different kernel name.

Notebook Update

Note: This step is typically executed by a Github Actions workflow.

Run docs/update_notebook_docs.py to manually copy these notebooks (if any updates) into the docs/source directory and update the index page of the Sphinx-generated documentation.

Sphinx Build

To run the typical build process, use the provided conda environment definition at etc/doc-environment.yml and run docs/make.py.

Note that this uses sphinx-multiversion to build documentation for multiple versions of data-describe and only captures changes in tagged commits and/or the master branch on remote - it does not build the documentation on your local branch.

Manual Build

To test a build of the Sphinx-generated documentation without using sphinx-multiversion, you can run the following command:

sphinx-build -a -E docs/source docs/build

The generated HTML files will be in docs/build.

Testing

Unit testing uses pytest.

Test Environment

Use the conda environment etc/test-environment.yml for executing tests.

Running the full test suite

To run the full test suite, run pytest in the root directory of the repository.

Running a selected test file or function

To run selected test(s), run pytest -k <PATTERN>, where <PATTERN> matches the test file name or function name.

Running selected marked tests

To run selected groups of tests, run pytest -m <MARKER>.

The current options for <MARKER> are: - base: Core features of data-describe (excluding any optional dependencies)

Design Patterns

The following section describes design patterns used in the Python package.

Optional Dependencies

data-describe may make use of optional dependencies such as nltk or modin. When adding or using these optional dependencies in data-describe modules, the following patterns should be used:

_requires marks functionality that requires a dependency

Use the _requires decorator on any object function or class that needs the optional dependency. Usage:

from data_describe.compat import _requires

@_requires("nltk")
def function_that_uses_nltk():
    return

_requires should generally take the top-level package name as its sole argument. See the section on packages vs subpackages for more information.

_compat is used to lazily import from dependencies

Instead of having import statements at the top of the file, import and use the _compat object to use functionalities from the optional dependency:

from data_describe.compat import _requires, _compat

@_requires("nltk")
def function_that_uses_nltk_freqdist():
    _compat["nltk"].FreqDist()

_compat should generally take the sub-package as its key in a dictionary-style access. See the section on packages vs subpackages for more information.

packages vs subpackages

Some packages do not export all of their subpackages. For example, import statsmodels does not provide access to statsmodels.graphics.tsaplots, as the graphics subpackage is not exported.

As a result, _requires generally takes the top-level package name, as this checks if the package itself is installed. In contrast, _compat takes the subpackage to enable imports.

One exception to this are the google client libraries such as google-cloud-storage or google-cloud-bigquery. Each of these are installed individually, but they are organized as subpackages of the google namespace i.e. google.cloud.storage. In this case, _requires should instead be the specific subpackage (i.e. _requires("google.cloud.storage")) since requiring only the google package is not specific enough.

Side imports

Some packages require downloads of additional data or models to function. One example is the stopwords for nltk. Downloading of these resources is handled in data_describe/compat/_dependency.py. When adding a dependency that requires this download, adhere to the following steps:

1. Add a function that takes the module as its sole argument, checks for the existence of the resource (i.e if it was already downloaded), and executes the download if it doesn’t exist.

2. Add this function to the module-import mapping used to initialize _compat:

_compat = DependencyManager(
    {
        "nltk": nltk_download,
        "spacy": spacy_download,
        # "new_package": downloader_function,
    }
)

Add to extras_require in setup.py

The new dependency should be added to the extras_require in setup.py. If applicable, try to use existing tags over creating new ones. New tags should be alphabetical and short.

Add to conda environments

The new dependency should be added to all conda environment definitions. These are located in two locations: etc/*.yml and docker/*/*.yml

Example notebooks

The examples/ folder of the repository contains Jupyter notebooks that provide more detailed documentation on using features of data-describe. Ideally, all features in data-describe should have its own example notebook.

Notebook Naming Convention

Notebook names should not contain spaces.

Notebook Structure

• The first cell in an example notebook must be a level 1 markdown header with the name/title of the example: # Title

• This header will be used as the name of the page in the auto-generated Sphinx documentation.

• Ideally, a short description of how and why to use the particular feature (i.e. from the design document) should be included after the title.

• Individual examples demonstrating different methods of using the feature should be separated by level 2 (or lower) headers.

Notebook Execution

To ensure consistency in cell execution order and kernel specifications, a Python script etc/run_notebooks.py should be used. This script will use papermill to execute all notebooks.

You can optionally add the --notebook-name argument to select specific notebooks for execution. This argument can be used multiple times and, if specified, utilizes str contains logic to select notebooks.

Notebook Testing

Unit tests for notebooks are in tests/test_notebooks.py. Follow the same pattern of using pytest-notebooks to add a test for a notebook.