Aggregation Classes

Aggregation Classes#

The aggregation module provides classes for performing area-weighted statistics on gridded data. These classes transform raw gridded datasets into meaningful statistics aggregated over polygon geometries or interpolated along polyline geometries.

gdptools offers two main aggregation approaches:

  • AggGen: Area-weighted aggregation for polygon geometries using precomputed or dynamically calculated weights

  • InterpGen: Point-based interpolation and statistics along polyline geometries

Both classes support multiple processing engines (serial and parallel) and various output formats for flexible deployment in different computational environments.

Key Features#

Statistical Methods#

  • Basic statistics: mean, median, min, max, sum, count, standard deviation

  • Masked statistics: Versions that handle nodata values appropriately

  • Weighted calculations: Area-weighted statistics for accurate spatial aggregation

Processing Engines#

  • Serial: Sequential processing for smaller datasets or debugging

  • Parallel: Multi-core processing using joblib for improved performance

Output Formats#

  • CSV: Tabular data with statistics per polygon/time

  • Parquet: Efficient columnar storage for large datasets

  • NetCDF: CF-compliant format for scientific data interchange

  • JSON: Structured data for web applications and APIs

Grid-to-Polygon Aggregation (AggGen)#

The AggGen class performs area-weighted aggregation of gridded data over polygon geometries. It’s designed for climate data analysis, hydrological modeling, and other applications requiring spatially aggregated statistics.

class AggGen(user_data, stat_method, agg_engine, agg_writer, weights, out_path=None, file_prefix=None, append_date=False, precision=None, jobs=-1)[source]#

Bases: object

Performs grid-to-polygon aggregation using area-weighted statistics.

This class provides functionality to aggregate gridded data over polygon geometries using various statistical methods and processing engines.

Parameters:

  • user_data (UserData) – Input data for aggregation (e.g., UserCatData).

  • stat_method (Literal['masked_mean', 'mean', 'masked_std', 'std', 'masked_median', 'median', 'masked_count', 'count', 'masked_sum', 'sum', 'masked_min', 'min', 'masked_max', 'max']) – Statistical method to apply for aggregation.

  • agg_engine (Literal['serial', 'parallel']) – Aggregation engine to use for processing.

  • agg_writer (Literal['none', 'csv', 'parquet', 'netcdf', 'json']) – Output writer format for results.

  • weights (str | DataFrame) – Path to CSV file or DataFrame containing area weights.

  • out_path (str | None) – Directory path for output files. Required if agg_writer is not ‘none’.

  • file_prefix (str | None) – Prefix for output file names. Required if agg_writer is not ‘none’.

  • append_date (bool) – Whether to append current date to output file names.

  • precision (int | None) – Number of decimal places for output data rounding.

  • jobs (int | None) – Number of processors for the parallel engine. -1 uses all available.

Raises:

  • ValueError – If agg_writer is not ‘none’ but out_path or file_prefix is missing.

  • TypeError – If stat_method, agg_engine, or agg_writer is invalid.

Examples

Basic aggregation with CSV output:

agg = AggGen(
    user_data=my_data,
    stat_method="mean",
    agg_engine="serial",
    agg_writer="csv",
    weights="weights.csv",
    out_path="/output",
    file_prefix="results",
)
gdf, dataset = agg.calculate_agg()

Parallel processing with NetCDF output:

agg = AggGen(
    user_data=my_data,
    stat_method="masked_mean",
    agg_engine="parallel",
    agg_writer="netcdf",
    weights=weights_df,
    out_path="/output",
    file_prefix="climate_data",
    jobs=4,
)
gdf, dataset = agg.calculate_agg()

Initialize the AggGen class with configuration parameters.

Sets up the aggregation system by configuring the statistical method, processing engine, and output writer based on the provided parameters.

Parameters:

  • user_data (UserData) – Input data container with source data and target geometries.

  • stat_method (Literal['masked_mean', 'mean', 'masked_std', 'std', 'masked_median', 'median', 'masked_count', 'count', 'masked_sum', 'sum', 'masked_min', 'min', 'masked_max', 'max']) – Statistical method for aggregation (e.g., ‘mean’, ‘masked_mean’).

  • agg_engine (Literal['serial', 'parallel']) – Processing engine (‘serial’ or ‘parallel’).

  • agg_writer (Literal['none', 'csv', 'parquet', 'netcdf', 'json']) – Output format (‘none’, ‘csv’, ‘parquet’, ‘netcdf’, ‘json’).

  • weights (str | DataFrame) – Path to weights CSV file or DataFrame with precomputed weights.

  • out_path (str | None) – Output directory path. Required if agg_writer is not ‘none’.

  • file_prefix (str | None) – Prefix for output file names. Required if agg_writer is not ‘none’.

  • append_date (bool) – If True, append current date to output filenames.

  • precision (int | None) – Number of decimal places for rounding output values.

  • jobs (int | None) – Number of processors for parallel processing. -1 uses all available.

Raises:

  • ValueError – If agg_writer is not ‘none’ but out_path or file_prefix is missing.

  • TypeError – If stat_method, agg_engine, or agg_writer is invalid.

__init__(user_data, stat_method, agg_engine, agg_writer, weights, out_path=None, file_prefix=None, append_date=False, precision=None, jobs=-1)[source]#

Initialize the AggGen class with configuration parameters.

Sets up the aggregation system by configuring the statistical method, processing engine, and output writer based on the provided parameters.

Parameters:

  • user_data (UserData) – Input data container with source data and target geometries.

  • stat_method (Literal['masked_mean', 'mean', 'masked_std', 'std', 'masked_median', 'median', 'masked_count', 'count', 'masked_sum', 'sum', 'masked_min', 'min', 'masked_max', 'max']) – Statistical method for aggregation (e.g., ‘mean’, ‘masked_mean’).

  • agg_engine (Literal['serial', 'parallel']) – Processing engine (‘serial’ or ‘parallel’).

  • agg_writer (Literal['none', 'csv', 'parquet', 'netcdf', 'json']) – Output format (‘none’, ‘csv’, ‘parquet’, ‘netcdf’, ‘json’).

  • weights (str | DataFrame) – Path to weights CSV file or DataFrame with precomputed weights.

  • out_path (str | None) – Output directory path. Required if agg_writer is not ‘none’.

  • file_prefix (str | None) – Prefix for output file names. Required if agg_writer is not ‘none’.

  • append_date (bool) – If True, append current date to output filenames.

  • precision (int | None) – Number of decimal places for rounding output values.

  • jobs (int | None) – Number of processors for parallel processing. -1 uses all available.

Raises:

  • ValueError – If agg_writer is not ‘none’ but out_path or file_prefix is missing.

  • TypeError – If stat_method, agg_engine, or agg_writer is invalid.

calculate_agg()[source]#

Calculate area-weighted aggregations for target polygons.

Performs the complete aggregation workflow: interpolates source gridded data to target polygons, computes the specified statistic, and optionally writes results to the specified output format.

Returns:

A two-element tuple:

  • A GeoDataFrame with target polygons and computed statistics.

  • An xarray Dataset with aggregated values in CF-compliant format.

Return type:

tuple[geopandas.GeoDataFrame, xarray.Dataset]

Raises:

  • TypeError – If writer or engine configuration is invalid.

  • ValueError – If output path or file prefix is missing when writing is enabled.

Examples

agg = AggGen(user_data, "mean", "serial", "csv", weights_df)
gdf, dataset = agg.calculate_agg()
print(f"Processed {len(gdf)} polygons")
property agg_data: dict[str, AggData]#

Get the aggregation data collected during processing.

Returns:

A mapping from variable name to the corresponding AggData instance, which contains metadata and processed data for each variable.

Return type:

dict[str, AggData]

Notes

This property is populated only after calling calculate_agg().

InterpGen Examples#

Basic Aggregation#

# Basic aggregation with CSV output
from gdptools.agg_gen import AggGen

agg = AggGen(
    user_data=climate_data,
    stat_method="mean",
    agg_engine="serial",
    agg_writer="csv",
    weights=weights_df,
    out_path="./output",
    file_prefix="climate_stats"
)
gdf, dataset = agg.calculate_agg()

# Access aggregated data
print(f"Processed {len(gdf)} polygons")
print(f"Variables: {list(dataset.data_vars)}")

Parallel Processing with Advanced Options#

# Parallel processing with NetCDF output
agg = AggGen(
    user_data=climate_data,
    stat_method="masked_mean",
    agg_engine="parallel",
    agg_writer="netcdf",
    weights="weights.csv",
    out_path="./output",
    file_prefix="aggregated_climate",
    jobs=4,
    precision=2,
    append_date=True
)
gdf, dataset = agg.calculate_agg()

# The agg_data property contains detailed processing information
processing_info = agg.agg_data
print(f"Processed variables: {list(processing_info.keys())}")

Polyline Interpolation (InterpGen)#

The InterpGen class interpolates gridded data along polyline geometries at specified intervals and computes statistics. This is useful for analyzing data along rivers, roads, transects, or other linear features.

class InterpGen(user_data, *, pt_spacing=50, stat='all', interp_method='linear', mask_data=False, output_file=None, calc_crs, method='serial', jobs=-1)[source]#

Bases: object

Calculates grid statistics along polyline geometries.

This class provides functionality to interpolate gridded data along polyline geometries at specified point intervals and compute statistics.

Parameters:

  • user_data (UserData) – Input data container with source data and target polylines.

  • pt_spacing (float | int | None) – Spacing between interpolation points, in the linear units of calc_crs. If 0, the line’s own vertices are used instead.

  • stat (str) – Statistic to calculate (“all”, “mean”, “median”, “min”, “max”, “std”).

  • interp_method (str) – xarray interpolation method (“linear”, “nearest”, “cubic”).

  • mask_data (bool) – Whether to mask nodata values during interpolation.

  • output_file (str | None) – Path to CSV file for saving results. If None, no file is written.

  • calc_crs (str | int | CRS) – Coordinate reference system in which distances are measured. Required, with no default. Must be distance-faithful for your area – not equal-area, which is what weight_gen_crs needs. See InterpGen.__init__() for why. Can be EPSG code, WKT string, or pyproj.CRS object.

  • method (Literal['serial', 'parallel']) – Interpolation engine to use for processing.

  • jobs (int | None) – Number of processors for the parallel engine. -1 uses all available.

Raises:

  • TypeError – If method='dask' is requested; the dask engine was removed in gdptools 0.4.0.

  • ValueError – If the specified interpolation method is not supported.

Examples

Basic line interpolation:

interp = InterpGen(
    user_data=my_data,
    pt_spacing=100,
    stat="mean",
    interp_method="linear",
    calc_crs=5070,  # required; distances are measured in this CRS
)
stats, points = interp.calc_interp()

Parallel processing with custom CRS:

interp = InterpGen(
    user_data=my_data,
    pt_spacing=50,
    stat="all",
    calc_crs=5070,
    method="parallel",
    jobs=4,
)
stats, points = interp.calc_interp()

Initialize the InterpGen class with configuration parameters.

Sets up the interpolation system for calculating statistics along polyline geometries using the specified interpolation method and processing engine.

Parameters:

  • user_data (UserData) – Input data container with source gridded data and target polylines.

  • pt_spacing (float | int | None) – Distance between interpolation points, in the linear units of calc_crs – meters for a meter-based CRS such as EPSG:5070, but degrees for a geographic one, which is why calc_crs must be projected. If 0, the line’s own vertices are used instead of evenly spaced points.

  • stat (str) – Statistical method to apply (“all”, “mean”, “median”, “min”, “max”, “std”).

  • interp_method (str) – xarray interpolation method (“linear”, “nearest”, “cubic”).

  • mask_data (bool) – If True, mask nodata values during interpolation.

  • output_file (str | None) – Path to CSV file for saving results. If None, no file is written.

  • calc_crs (str | int | CRS) –

    Coordinate reference system in which distances are measured. Required – there is deliberately no default.

    This needs a distance-faithful projected CRS, which is a different requirement from weight_gen_crs elsewhere in this package: that one needs equal-area. pt_spacing is expressed in the linear units of this CRS, so a projection that distorts distance silently changes your sample spacing. Measured at 40N, 100W: a true 1000 m separation is 1103 m in EPSG:6931 and 1304 m in EPSG:3857, so pt_spacing=50 would sample every 45 m and 38 m of ground respectively. EPSG:5070 holds within about 1% between roughly 25.7N and 48.4N, loosening to about 1.4% at both edges of its declared area of use; the UTM zone containing your data is better still, within 0.1% anywhere in the zone. See Choosing calc_crs for how to choose and how to check.

  • method (Literal['serial', 'parallel']) – Processing engine (“serial” or “parallel”).

  • jobs (int | None) – Number of processors for parallel processing. -1 uses all available.

__init__(user_data, *, pt_spacing=50, stat='all', interp_method='linear', mask_data=False, output_file=None, calc_crs, method='serial', jobs=-1)[source]#

Initialize the InterpGen class with configuration parameters.

Sets up the interpolation system for calculating statistics along polyline geometries using the specified interpolation method and processing engine.

Parameters:

  • user_data (UserData) – Input data container with source gridded data and target polylines.

  • pt_spacing (float | int | None) – Distance between interpolation points, in the linear units of calc_crs – meters for a meter-based CRS such as EPSG:5070, but degrees for a geographic one, which is why calc_crs must be projected. If 0, the line’s own vertices are used instead of evenly spaced points.

  • stat (str) – Statistical method to apply (“all”, “mean”, “median”, “min”, “max”, “std”).

  • interp_method (str) – xarray interpolation method (“linear”, “nearest”, “cubic”).

  • mask_data (bool) – If True, mask nodata values during interpolation.

  • output_file (str | None) – Path to CSV file for saving results. If None, no file is written.

  • calc_crs (str | int | CRS) –

    Coordinate reference system in which distances are measured. Required – there is deliberately no default.

    This needs a distance-faithful projected CRS, which is a different requirement from weight_gen_crs elsewhere in this package: that one needs equal-area. pt_spacing is expressed in the linear units of this CRS, so a projection that distorts distance silently changes your sample spacing. Measured at 40N, 100W: a true 1000 m separation is 1103 m in EPSG:6931 and 1304 m in EPSG:3857, so pt_spacing=50 would sample every 45 m and 38 m of ground respectively. EPSG:5070 holds within about 1% between roughly 25.7N and 48.4N, loosening to about 1.4% at both edges of its declared area of use; the UTM zone containing your data is better still, within 0.1% anywhere in the zone. See Choosing calc_crs for how to choose and how to check.

  • method (Literal['serial', 'parallel']) – Processing engine (“serial” or “parallel”).

  • jobs (int | None) – Number of processors for parallel processing. -1 uses all available.

calc_interp()[source]#

Run interpolation and statistical calculations along polylines.

Performs the complete interpolation workflow: generates points along polylines at specified intervals, interpolates gridded data to these points, and computes the requested statistics.

Returns:

The statistics and the interpolated points, always as a two-element tuple. This does not vary with the stat argument – stat selects which statistics the first element carries, not whether the second element is returned.

Return type:

tuple[pandas.DataFrame, geopandas.GeoDataFrame]

Raises:

ValueError – If the specified interpolation method is not supported.

Examples

interp = InterpGen(user_data, pt_spacing=100, stat="mean", calc_crs=5070)
stats, points = interp.calc_interp()
print(f"Mean values: {stats['mean'].values}")

interp = InterpGen(user_data, pt_spacing=50, stat="all", calc_crs=5070)
stats, points = interp.calc_interp()
print(f"Generated {len(points)} interpolation points")

Usage Examples#

Basic Line Interpolation#

# Basic line interpolation
from gdptools.agg_gen import InterpGen

interp = InterpGen(
    user_data=river_data,
    pt_spacing=100,  # 100-meter intervals, measured in calc_crs
    stat="mean",
    interp_method="linear",
    calc_crs=5070,  # required -- see "Choosing calc_crs" below
)
stats, points = interp.calc_interp()  # always a two-element tuple

# Display results
print(f"Mean values along line: {stats['mean'].values}")

Comprehensive Statistics with Custom Configuration#

# Comprehensive statistics with custom CRS
interp = InterpGen(
    user_data=river_data,
    pt_spacing=50,
    stat="all",  # Returns all statistics
    calc_crs=5070,  # Conus Albers -- within ~1% over most of CONUS
    method="parallel",
    jobs=2,
    output_file="river_stats.csv"
)
stats, points = interp.calc_interp()

# Access detailed results
print(f"Generated {len(points)} interpolation points")
print(f"Statistics available: {list(stats.columns)}")

Choosing calc_crs#

pt_spacing is a distance, and InterpGen measures it in calc_crs. The line is reprojected into that CRS, its length is measured there, and a sample point is placed every pt_spacing units along it. calc_crs therefore does not merely label the output — it decides how far apart your samples actually land on the ground. A CRS that stretches distance by 10% gives you samples 10% closer together than you asked for, and it does so silently: the run succeeds and the results look entirely plausible.

That is why there is no default. There is no CRS that is right everywhere, and a wrong one here does not announce itself.

No projection preserves every distance#

Flattening a curved Earth onto a plane always distorts something, and distance is always one of the casualties somewhere. At any point a projection has a scale factor: projected distance divided by true ground distance. It varies from place to place, and in general it also varies with direction.

The projections you are likely to reach for fall into two families, and they fail in different ways.

Conformal projections — transverse Mercator (UTM), Lambert Conformal Conic, State Plane — hold the scale factor the same in every direction at a given point. Distortion is isotropic: a segment measures the same whichever way it runs. The scale factor still changes from place to place, so a conformal projection is only useful over the area it was designed for. UTM is built for exactly this job: scale factor 0.9996 on the zone’s central meridian, drifting to roughly 1.001 at the zone edge near the equator, and staying better than 0.1% anywhere in the zone.

Equal-area projections — Albers Equal Area (EPSG:5070), Lambert Azimuthal Equal Area (EPSG:6931) — hold area constant instead. Preserving area forces a trade: whatever is stretched in one direction must be compressed by the reciprocal amount in the perpendicular direction. Distortion is anisotropic, so the same 1000 m measures differently depending on its bearing. That is the right property for weight_gen_crs, where areas are the quantity being computed, and the wrong one here. An equal-area CRS is not a neutral, safe choice for calc_crs; it is a projection that has spent its accuracy budget on something else.

Measured#

A true 1000 m ground separation, swept over all azimuths, measured at the example line data in docs/Examples/ClimateR-Catalog/test_lines/ (42.73°N, 93.57°W):

calc_crs

shortest

longest

error

anisotropy

EPSG:32615 UTM 15N — conformal, correct zone

999.6 m

999.6 m

−0.04%

1.0000

EPSG:5070 Conus Albers — equal-area

994.0 m

1006.0 m

±0.60%

1.0121

EPSG:6931 EASE-Grid 2.0 North — equal-area

916.2 m

1091.4 m

−8.4% … +9.1%

1.1912

EPSG:3857 Web Mercator — pseudo-conformal

1359.3 m

1364.3 m

+36%

1.0037

Read the last column as the signature of each family. UTM is conformal, so its anisotropy is 1.0000: direction does not matter. EPSG:6931 is equal-area, so at this latitude it stretches east–west by 9.1% and compresses north–south by 8.4% — their product is 1, which is exactly what preserving area requires, and neither cancels in distance.

EPSG:3857 is the instructive middle case. Despite the name it is not conformal: WGS 84 / Pseudo-Mercator applies the spherical Mercator formulae to ellipsoidal latitudes, and EPSG classifies it as non-conformal for that reason. Its residual anisotropy of 1.0037 above is that ellipsoid-versus-sphere mismatch made visible; it grows toward the equator (1.0067 at 0°, 1.0056 at 25°, 1.0018 at 60°). Its error is also the largest here by far, and roughly predictable: the scale factor is approximately 1/cos(latitude), or 1.3614 at 42.73°N — about +36%. It is a display projection. Never use it for calc_crs.

EPSG:5070’s accuracy is not uniform across its area of use either. An Albers projection is exact on its two standard parallels, which for EPSG:5070 are 29.5°N and 45.5°N, and the measured worst-case error tracks them. The error depends only on latitude — it is identical at 118°W, 96°W and 66°W — so one row says everything:

Latitude

24.41°N

28°N

30°N

34°N

38°N

42°N

46°N

48°N

49.38°N

worst error

1.42%

0.36%

0.11%

0.75%

0.98%

0.71%

0.14%

0.83%

1.43%

The first and last columns are the north and south limits of EPSG:5070’s declared area of use (24.41°N to 49.38°N). So the projection holds within 1% between about 25.7°N and 48.4°N — which covers most, but not all, of the lower 48 — tightening to near zero on the standard parallels and loosening to about 1.4% at both edges. The northern edge is the one that catches people: the 49°N border across Washington, Idaho, Montana, North Dakota and Minnesota sits at 1.25%, worse than Brownsville, Texas at 0.96%.

It reaches your results#

Sampling one of the example stream lines at pt_spacing=100 — the same line, the same request, only calc_crs changed:

calc_crs

sample points produced

EPSG:5070

91

EPSG:6931

94

EPSG:3857

124

EPSG:4326

1

EPSG:3857 returns 36% more points than asked for. EPSG:4326 returns one point, because a geographic CRS is measured in degrees, not meters — pt_spacing=100 asks for a point every 100 degrees. Always give calc_crs a projected CRS.

Recommendations#

  • Data within one UTM zone — use that zone. It is the most distance-faithful choice generally available, and the zone is floor((longitude + 180) / 6) + 1.

  • Data spanning a region or several zones — use a conformal projection built for that region, such as a Lambert Conformal Conic with standard parallels bracketing your latitudes.

  • CONUS-wide work — EPSG:5070 is a reasonable compromise, within about 1% across the middle latitudes and about 1.4% at the edges of its area of use, and it is what these examples use. It is equal-area rather than conformal, so it is not the theoretically correct family, but its distortion is small enough to be defensible when one CRS has to cover the whole country.

  • Avoid EPSG:3857, any geographic CRS, and equal-area CRSs used far from where they are centered — EPSG:6931 is centered on the North Pole and costs 9% in Iowa.

Check your own study area#

Do not take any of the above on trust for your data. Measure it — the check is short, and it is the only way to know what your pt_spacing really means:

import math

from pyproj import Geod, Transformer

geod = Geod(ellps="WGS84")


def distance_error(lon, lat, calc_crs, true_m=1000.0):
    """Worst-case error in a true `true_m` separation at (lon, lat), any direction."""
    t = Transformer.from_crs(4326, calc_crs, always_xy=True)
    x1, y1 = t.transform(lon, lat)
    lengths = []
    for azimuth in range(0, 180, 5):
        lon2, lat2, _ = geod.fwd(lon, lat, float(azimuth), true_m)
        x2, y2 = t.transform(lon2, lat2)
        lengths.append(math.hypot(x2 - x1, y2 - y1))
    return (min(lengths) / true_m - 1) * 100, (max(lengths) / true_m - 1) * 100


# Use the centroid of your own target geometries, and check the corners too.
lo, hi = distance_error(-93.571, 42.730, 5070)
print(f"EPSG:5070 at the study site: {lo:+.2f}% .. {hi:+.2f}%")

If the worst-case error is comfortably smaller than the precision you need from pt_spacing, the CRS is fine. If it is not, the samples are not where you think they are.

Type Definitions#

The module provides several literal types for configuration options:

STATSMETHODS#

Available aggregation methods.

Options:

masked_mean: Masked mean of the data. mean: Mean of the data. masked_std: Masked standard deviation of the data. std: Standard deviation of the data. masked_median: Masked median of the data. median: Median of the data. masked_count: Masked count of the data. count: Count of the data. masked_sum: Masked sum of the data. sum: Sum of the data. masked_min: Masked minimum of the data. min: Minimum of the data. masked_max: Masked maximum of the data. max: Maximum of the data.

alias of Literal[‘masked_mean’, ‘mean’, ‘masked_std’, ‘std’, ‘masked_median’, ‘median’, ‘masked_count’, ‘count’, ‘masked_sum’, ‘sum’, ‘masked_min’, ‘min’, ‘masked_max’, ‘max’]

AGGENGINES#

Available aggregation engines.

Options:

serial: Perform area-weighted aggregation sequentially. parallel: Perform area-weighted aggregation in parallel.

alias of Literal[‘serial’, ‘parallel’]

AGGWRITERS#

Available output writers.

Options:

none: Do not write output to file. csv: Write output in CSV format. parquet: Write output in Parquet format. netcdf: Write output in NetCDF format. json: Write output in JSON format.

alias of Literal[‘none’, ‘csv’, ‘parquet’, ‘netcdf’, ‘json’]

LINEITERPENGINES#

Available line interpolation engines.

Options:

serial: Perform interpolation sequentially. parallel: Perform interpolation in parallel.

alias of Literal[‘serial’, ‘parallel’]

Best Practices#

Performance Considerations#

  • Use "serial" engine for datasets < 1GB and debugging

  • Use "parallel" engine for moderate to large datasets (1GB+) on multi-core systems

Memory Management#

  • Set appropriate jobs parameter based on available memory. If you request more workers than the machine has physical CPU cores, AggGen clamps the value and raises a warning so you know the engine throttled your request.

  • Use precision parameter to control output file sizes

  • Consider chunking large datasets before processing

Output Format Selection#

  • CSV: Human-readable, good for small to medium datasets

  • Parquet: Efficient for large datasets, preserves data types

  • NetCDF: Standard for scientific data, CF-compliant

  • JSON: Structured data for web applications

Error Handling#

  • Validate input data and geometries before processing

  • Use masked statistics (masked_*) for datasets with nodata values

  • Test with small subsets before processing large datasets

Note

All aggregation classes automatically handle coordinate reference system (CRS) transformations and ensure proper alignment between source data and target geometries.

Warning

When using the parallel engine, ensure sufficient memory is available. Large datasets may require chunking.