Data Sources — Overview
All data sources inherit from DataSource and implement a single method:
RasterData is a named tuple (array: np.ndarray, crs: str | None, transform: Affine | None).
No rasterio objects are ever exposed outside a source module.
Callers use fetch(), which the base class provides. It reads and writes the disk cache,
calls _fetch() and applies valid_range. A source becomes cacheable by also implementing
_cache_key(grid), which returns a JSON-serialisable dict identifying the request, or
None when the source is not cached.
Source catalogue
| Source | Use case |
|---|---|
ArraySource |
In-memory numpy array |
ConstantSource |
Single scalar broadcast over the grid |
RasterSource |
Local GeoTIFF file |
URLSource |
Remote GeoTIFF via HTTP |
WCSSource |
Generic OGC WCS endpoint |
PointGridSource |
Sample any callable over an N×N lat/lon grid |
MosaicSource |
Combine several sources, best source first |
DerivedSource |
Compute a layer from other sources |
Grid-aware vs self-contained sources
Self-contained sources (ArraySource, ConstantSource, RasterSource, URLSource)
ignore the grid argument. They carry their own spatial metadata.
Grid-aware sources need the reference grid to determine what geographic area to
query. They call grid.extent_wgs84() to obtain (lon_min, lat_min, lon_max, lat_max)
before making API or WCS requests. The box is computed from points along all four grid
edges, so it covers edges that curve in lon/lat, and it reaches the pole for polar grids.
A grid crossing the antimeridian gets a box spanning nearly all longitudes.
Grid-aware sources are: WCSSource, PointGridSource.
MosaicSource follows the sources it wraps. It needs the grid to align them before it
can compare them pixel by pixel, but as long as one of them is self-contained the mosaic
reports that source when asked without a grid, so an automatic grid can still be derived
from it.
DerivedSource works the same way: it aligns its inputs to the grid before calling its
function, and reports its first self-contained input when asked without a grid.
Masking sentinel values with valid_range
Every source takes valid_range=(lo, hi). Values below lo or above hi are replaced
with NaN, so they are excluded from inference instead of being discretised as real values.
Both bounds are inclusive, and either may be None to leave that side unbounded:
geobn.RasterSource("dem.tif", valid_range=(-500.0, 9000.0)) # mask both tails
geobn.RasterSource("dem.tif", valid_range=(0.0, None)) # mask negatives only
The three GeoTIFF-backed sources (RasterSource, URLSource, WCSSource) convert a
declared nodata value and any mask band to NaN on their own. valid_range covers the data
that declares nothing and encodes missing values as an extreme number, such as −9999 or
32767.
For the cached sources the range is applied to the array on its way out, so the cache holds the data as it arrived and changing the range does not trigger a new request.
Caching and freshness
URLSource, WCSSource and PointGridSource take cache_dir to store what they fetched
on disk, and cache_ttl — a timedelta or a number of seconds — for how long an entry
stays usable. Without a TTL an entry is used forever, which is what terrain and bathymetry
want; with one, an older entry is fetched again.
Each entry is an .npy array beside a .json sidecar holding its CRS, transform and the
time it was fetched. For URLSource the sidecar also holds the ETag and Last-Modified
headers the server sent, if any, so an expired entry is checked with a conditional request
and downloaded again only if the file has changed. cache_ttl is not part of the cache key, so changing it re-reads the
same entry rather than orphaning it.
If an entry has expired and the fetch then fails, the expired entry is returned with a
UserWarning naming its age, so an offline run still produces a map.
PointGridSource additionally needs name, since the callable it wraps cannot identify
itself.
DataSource ABC
DataSource
Bases: ABC
Base class for all geographic data sources.
Every source returns a plain RasterData tuple (array, crs, transform) — no rasterio objects are ever exposed outside the source module.
Subclasses implement :meth:_fetch; :meth:fetch wraps it, reads and
writes the disk cache, and applies valid_range. A source becomes
cacheable by overriding :meth:_cache_key.
Parameters
valid_range:
Optional (lo, hi) tuple. Pixels below lo or above hi are
replaced with NaN, so they are excluded from inference instead of
being discretised as real values. Both bounds are inclusive, and
either may be None to leave that side unbounded. Use this for
data that encodes missing values as extreme numbers without
declaring them as nodata, e.g. (-500, 9000) for elevation in
metres.
cache_dir:
Optional directory for caching fetched data on disk. Accepted only
by sources that can be cached.
cache_ttl:
Maximum age of a cache entry, as a :class:~datetime.timedelta or a
number of seconds. An older entry is refetched. The default
None never expires, which suits data that does not change, such
as terrain. If the refetch fails and an expired entry exists, that
entry is returned with a :class:UserWarning giving its age.
resampling:
How the source is resampled onto the reference grid when its CRS,
resolution or extent differ from the grid's:
- ``"bilinear"`` (default) interpolates between the four source
pixels around each grid pixel centre. It suits continuous data
at a similar or coarser resolution than the grid.
- ``"nearest"`` takes the source pixel under the grid pixel centre.
- ``"mode"`` takes the most frequent value among the source pixels
within each grid pixel, ties going to the smallest value. Use it,
or ``"nearest"``, for class rasters such as land cover or seabed
type, where interpolating would produce codes that are not
classes.
- ``"average"``, ``"min"`` and ``"max"`` take the mean, minimum or
maximum of the source pixels within each grid pixel. Use them
when the source is finer than the grid: ``"average"`` for
continuous data, and ``"max"`` or ``"min"`` for hazard layers,
where a small steep slope or a shallow shoal must not be averaged
away.
``"mode"``, ``"average"``, ``"min"`` and ``"max"`` ignore NaN
pixels, so a grid pixel is NaN only when none of its source pixels
has data. Where the source is coarser than the grid, each grid
pixel lies within one source pixel and these methods give the same
result as ``"nearest"``.
fetch
Fetch data aligned to grid if provided.
Serves the disk cache when one is configured and the entry is within
cache_ttl. Values outside valid_range are returned as NaN.
Parameters
grid: Reference grid context. Required by sources that need to know the spatial domain before querying (e.g. OpenMeteoSource). Ignored by sources that are self-contained (Array, Raster, URL, Constant).