Contributor Guide

Contributor Guide#

Thank you for your interest in improving this project. This project is public domain under the CC0 1.0 Universal license and welcomes contributions in the form of bug reports, feature requests, and merge requests.

Here is a list of important resources for contributors:

How to report a bug#

Report bugs on the Issue Tracker.

When filing an issue, make sure to answer these questions:

  • Which operating system and Python version are you using?

  • Which version of this project are you using?

  • What did you do?

  • What did you expect to see?

  • What did you see instead?

The best way to get your bug fixed is to provide a test case, and/or steps to reproduce the issue.

How to request a feature#

Request features on the Issue Tracker.

How to set up your development environment#

You need Python 3.11+ (matching requires-python in pyproject.toml) and uv.

  1. Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh (or pip install uv)

  2. Sync dependencies: uv sync --group dev

  3. Install pre-commit hooks: pre-commit install --install-hooks

This creates a .venv directory with all dependencies. Use uv run <command> to run commands in the environment — it needs no activation and works the same on every platform — or activate it with source .venv/bin/activate (Windows: .venv\Scripts\activate, or .venv\Scripts\Activate.ps1 under PowerShell).

Docstring style and linting#

Docstrings follow the Google convention enforced by pydocstyle (via ruff). Automatic fixes for docstring rules are disabled to prevent large refactors, so ruff --fix will not rewrite docstrings for you. When a docstring lint error appears, update the text manually until ruff reports a clean result.

Types in Args:, Attributes: and Returns: sections must be written out in full: geopandas.GeoDataFrame, not GeoDataFrame; xarray.DataArray, not xr.DataArray. Sphinx turns this text into cross-references, and a bare name either fails to resolve or resolves to the wrong object. The docs CI job enforces it with nox -s docs-check, which builds the documentation with --nitpick, so an unresolved reference blocks the merge. If a reference genuinely cannot resolve, add it to nitpick_ignore in docs/_config.yml with a comment saying why.

Always remove docs/_build before running nox -s docs-check. Sphinx builds incrementally, so a stale build directory reports success in seconds while a clean build of the same tree fails.

When the docs build fails because of the network#

docs-check downloads an intersphinx inventory from each project listed under intersphinx_mapping in docs/_config.yml. If one of those hosts is unreachable you will see:

WARNING: failed to reach any of the inventories with the following issues:

followed by a URL and a connection error, and — because the build runs with --warningiserror — the build then fails. This is a network problem, not something your change caused. Nothing in the diff will fix it: re-run the job, or if you are working offline, expect it until you have a connection again. The CI docs job retries twice for this reason, and intersphinx_timeout is set so an unreachable host fails in seconds rather than hanging.

Note that this failure also masks real nitpick warnings: with no inventory loaded, every external cross-reference becomes unresolvable at once. A sudden jump from zero warnings to dozens is the signature of a failed inventory fetch, not of dozens of new mistakes.

When a notebook’s HyRiver call falls back#

The example notebooks call USGS services through the HyRiver packages. Those calls are wrapped by hyriver_or_cached in docs/_data/nb_data.py, which falls back to the GeoParquet fixtures committed alongside it when a service is unreachable — so an outage no longer fails the docs build. A notebook that prints

!! falling back to committed fixture

ran against cached data, not live data. This is the same bargain as the Recorded data section below: committed inputs so our code can be tested when someone else’s server is not answering.

First check whether the fixtures have actually drifted from the live services:

uv run pytest tests/test_notebook_data.py -m slow

Only once that fails should you regenerate the fixtures, when the WBD or NHDPlus source data is reissued:

uv run python scripts/refresh_notebook_data.py

Run the refresh script only to refresh, not to check for drift: it rewrites all four fixture files unconditionally, and GeoParquet output is not byte-stable, so git status afterward cannot distinguish encoding churn from a real data change.

The expected feature counts are asserted in tests/test_notebook_data.py. If a count changes, that is a real change in the source data — update the test and the spec deliberately rather than adjusting the number to make the suite pass.

How to test the project#

Run the full test suite:

nox

List the available Nox sessions:

nox --list-sessions

You can also run a specific Nox session. For example, invoke the unit test suite like this:

nox --session=tests

Unit tests are located in the tests directory, and are written using the pytest testing framework.

The merge gate is offline; live-service tests are marked slow#

nox -s tests runs -m "not slow", and that selection makes no network calls. Verified by running it with HTTP_PROXY/HTTPS_PROXY pointed at a dead port: 412 passed in 77 s.

Five tests read from the climateR catalog on GitHub or gridMET over THREDDS OPeNDAP. They cost roughly 330 s of what was a 406 s run, and their latency made the CI tests job swing between 307 s and 762 s with no change in the code — enough that a timing jump was once misread as a code regression. They are marked @pytest.mark.slow (#131).

What they uniquely covered was the ClimRCatData path. That splits in two: parsing the catalog is our code and breaks when we change it, while reading the data is someone else’s server and breaks during their outages. Only the second belongs outside the gate, so tests/test_climr_catalog_offline.py keeps the first, built on tests/data/climater_catalog_subset.parquet — an 88 KB subset of the gridmet and GLDAS entries.

A committed fixture can drift from its source, so test_vendored_subset_matches_live_catalog compares the subset’s columns and row counts against the live catalog. It is itself marked slow, being the one test there that needs the network. To regenerate the subset:

import pandas as pd

cat = pd.read_parquet(
    "https://github.com/mikejohnson51/climateR-catalogs/releases/download/June-2024/catalog.parquet"
)
cat[cat["id"].isin(["gridmet", "GLDAS"])].reset_index(drop=True).to_parquet(
    "tests/data/climater_catalog_subset.parquet", index=False
)

slow tests run in the tests-full job. On develop it runs automatically; on a merge request it is a manual play button. It never blocks either way.

That is because of what a failure there means. slow tests ask “are these services reachable, and still shaped the way we expect?” — a question whose answer is usually about somebody else’s server. Blocking work on it helps no one, and the release review merge request is the sharpest case: its source branch is develop, so its head pipeline is the develop pipeline.

The question “did a code change break the method?” is asked in the merge gate instead, offline, where it does block. tests/test_climr_catalog_offline.py runs the full ClimRCatData → WeightGen → AggGen pipeline against committed gridMET data in about twenty seconds, with pinned output values.

Recorded data#

tests/data/recorded/ holds five days of real gridMET over the Delaware River Basin — 257 KB for three variables, pulled from the live THREDDS endpoint once. It is real data, not a synthetic stand-in: same values, shape, coordinate names and CRS. What it drops is the dependency on that endpoint being up.

A catalog entry can point at it because _open_catalog_dataset appends the OPeNDAP-only #fillmismatch query to URLs and not to local paths. Before that, the catalog-driven code paths appended it unconditionally and a local file could not be opened at all.

To re-record, subset each variable to the DRB bounds and write it to tests/data/recorded/gridmet_<var>_drb_1980.nc. If the values change, the pinned expectations in test_aggregated_values_are_unchanged change with them — work out why before updating them. !239 moved numbers exactly like these by changing which points got sampled, and the right response was to understand the shift, not to paste in new figures.

Notebook execution is a separate, on-request check#

tests/test_notebooks.py executes the thirteen notebooks in docs/Examples. It is marked notebooks, not slow, and is excluded from both the merge gate and tests-full. Run it when you want it:

nox -s test-notebooks

or press the manual tests-notebooks job on a pipeline. Read the Docs also executes these notebooks on every docs build after merge.

It is deliberately not automatic. Each notebook carries a 600 s timeout and drives the NHGF STAC catalog, THREDDS OPeNDAP and the ClimateR catalog end to end, so an automatic run would mostly report someone else’s outage — the failure mode tests-full was rescued from in #128.

Two things about this test are worth knowing, because both hid real breakage:

  • It executes each notebook with that notebook’s own directory as the working directory, which is what Jupyter and Read the Docs do. Without that, every relative path resolves against the repository root and correct notebooks fail with “No such file or directory”.

  • It uses allow_errors=True so a notebook runs to the end and every failing cell is reported. The consequence is that execute() does not raise on a cell error — it records the error in the cell’s outputs. Scanning those outputs is the only thing that can fail this test, so that scan must never live inside an except block. It did, which is why the test could not fail at all (#126).

PROJ network access is pinned off during tests#

tests/conftest.py disables PROJ network access for the whole test session, and the suite will not reproduce without it.

PROJ’s network setting decides which transformation operation it selects, not merely whether it can download something. With network access enabled it picks higher-accuracy grid-based operations it would otherwise skip: EPSG:4326 to EPSG:5070 offers 1 candidate operation with network access off and 48 with it on. A different operation gives slightly different coordinates, and this suite pins expected values to many decimal places. Before the pin, 13 tests failed on a clean checkout for no reason other than PROJ_NETWORK being enabled — which some PROJ builds do by default, and which users turn on deliberately so PROJ can fetch datum grids.

Both the environment variable and pyproj’s runtime toggle are set, and neither is sufficient alone: the parallel engines run through joblib, whose worker processes read PROJ_NETWORK themselves during PROJ initialization, while the runtime call is what takes effect in a process where pyproj is already imported.

To investigate the network-enabled path deliberately, set GDPTOOLS_TEST_ALLOW_PROJ_NETWORK=1. Expect failures, and read the note in tests/conftest.py before drawing conclusions from them.

Worth knowing separately: a transformation that needs a datum-shift grid it cannot fetch — behind a firewall, on an offline machine, or through a VPN that intercepts TLS — does not raise. It returns infinity, and geopandas.to_crs reports success. gdptools.utils._check_finite_geometries exists to catch that.

Branching model#

This project uses trunk-based development. develop is the single long-lived trunk — it is the default branch, the integration point, and the branch that release tags are cut from. There are no other long-lived branches.

To contribute:

  • Create a short-lived topic branch off develop.

  • Keep it small and focused; open a merge request back into develop quickly.

  • Name the branch <type>-<short-desc>, e.g. fix-issue-100-nan or docs-update-readme.

  • Delete the branch after it is merged.

How to submit changes#

Open a merge request to submit changes to this project.

Your merge request needs to meet the following guidelines for acceptance:

  • The Nox test suite must pass without errors and warnings.

  • Include unit tests. Coverage is currently around 75%, and fail_under in pyproject.toml sets the floor at 60% — new code is expected to raise the figure, not lower it.

  • If your changes add functionality, update the documentation accordingly.

Feel free to submit early, though—we can always iterate on this.

It is recommended to open an issue before starting work on anything. This will allow a chance to talk it over with the owners and validate your approach.