Contributing to the SDK

Note: The SDK currently works with Python versions 3.7 through 3.10.x. Python 3.6 is no longer supported.

Let’s build together! Please see our Contributor Guide for more information on contributing to Meltano.

We believe that everyone can contribute and we welcome all contributions. If you’re not sure what to work on, here are some ideas to get you started.

Chat with us in #contributing on Slack.

Contributors are expected to follow our Code of Conduct.

Setting up Prereqs

Make sure poetry, pre-commit and nox are installed. You can use pipx to install all of them. To install pipx:

pip3 install pipx
pipx ensurepath

With pipx installed, you globally add the required tools:

pipx install poetry
pipx install pre-commit
pipx install nox
pipx inject nox nox-poetry

Now you can use Poetry to install package dependencies:

cd sdk

# Install package and dependencies:
poetry install
# OR install in editable mode:
poetry install --no-root

Local Developer Setup

First clone, then…

Ensure you have the correct test library, formatters, and linters installed:
- poetry install
If you are going to update documentation, install the docs extras:
- poetry install -E docs
The project has pre-commit hooks. Install them with:
- pre-commit install
Most development tasks you might need should be covered by nox sessions. You can use nox -l to list all available tasks. For example:
- Run unit tests: nox -r.
  
  We use coverage for code coverage metrics.
- Run pre-commit hooks: pre-commit run --all.
  
  We use black, flake8, isort, mypy and pyupgrade. The project-wide max line length is 88.
- Build documentation: nox -rs docs
  
  We use sphinx to build documentation.

If you are using VSCode

Make sure you have also installed the Python extension.
Set interpreter to match poetry’s virtualenv: run Python: Select interpreter and select the poetry interpreter.
The pre-commit extension will allow to run pre-commit hooks on the current file from the VSCode command palette.

Testing Locally

To run tests:

# Run just the core and cookiecutter tests (no external creds required):
nox -rs tests

# Run all tests (external creds required):
nox -rs tests -- -m "external"

To view the code coverage report in HTML format:

nox -rs coverage -- html && open ./htmlcov/index.html

Platform-specific Testing

To mark a test as platform-specific, use the @pytest.mark.<platform> decorator:

import pytest

@pytest.mark.windows
def test_windows_only():
    pass

Supported platform markers are windows, darwin, and linux.

Snapshot Testing

We use pytest-snapshot for snapshot testing.

Adding a new snapshot

To add a new snapshot, use the snapshot fixture and mark the test with the @pytest.mark.snapshot decorator. The fixture will create a new snapshot file if one does not already exist. If a snapshot file already exists, the fixture will compare the snapshot to the actual value and fail the test if they do not match.

The tests/snapshots directory is where snapshot files should be stored and it’s available as the snapshot_dir fixture.

@pytest.mark.snapshot
def test_snapshot(snapshot, snapshot_dir):
    # Configure the snapshot directory
    snapshot.snapshot_dir = snapshot_dir.joinpath("test_snapshot_subdir")

    snapshot_name = "test_snapshot"
    expected_content = "Hello, World!"
    snapshot.assert_match(expected_content, snapshot_name)

Generating or updating snapshots

To update or generate snapshots, run the nox update_snapshots session

nox -rs update_snapshots

or use the --snapshot-update flag

poetry run pytest --snapshot-update -m 'snapshot'

This will run all tests with the snapshot marker and update any snapshots that have changed. Commit the updated snapshots to your branch if they are expected to change.

Testing Updates to Docs

Documentation runs on Sphinx, using ReadtheDocs style template, and hosting from ReadtheDocs.org. When a push is detected by readthedocs.org, they automatically rebuild and republish the docs. ReadtheDocs is also version aware, so it retains prior and unreleased versions of the docs for us.

To build the docs and live-reload them locally:

nox -rs docs-serve

Sphinx will automatically generate class stubs, so be sure to git add them.

Semantic Pull Requests

This repo uses the semantic-prs GitHub app to check all PRs againts the conventional commit syntax.

Pull requests should be named according to the conventional commit syntax to streamline changelog and release notes management. We encourage (but do not require) the use of conventional commits in commit messages as well.

In general, PR titles should follow the format <type>: <desc>, where type is any one of these:

ci
chore
build
docs
feat
fix
perf
refactor
revert
style
test

Optionally, you may use the expanded syntax to specify a scope in the form <type>(<scope>): <desc>. Currently scopes are:

scopes:

taps # tap SDK only
targets # target SDK only
mappers # mappers only
templates # cookiecutters

More advanced rules and settings can be found within the file .github/semantic.yml.

Workspace Development Strategies for the SDK

Universal Code Formatting

From the Black website:

By using Black, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters. Black makes code review faster by producing the smallest diffs possible. Blackened code looks the same regardless of the project you’re reading. Formatting becomes transparent after a while and you can focus on the content instead.

Pervasive Python Type Hints

Type hints allow us to spend less time reading documentation. Public modules are checked in CI for type annotations on all methods and functions.

Docstring convention

All public modules in the SDK are checked for the presence of docstrings in classes and functions. We follow the Google Style convention for Python docstrings so functions are required to have a description of every argument and the return value, if applicable.

What is Poetry and why do we need it?

For more info on Poetry and Pipx, please see the topic in our python tips guide.