Install

Install

This page covers a first-time deployment of shepard. For host sizing read System requirements first.

Stack — docker-compose

Production deployment uses infrastructure/docker-compose.yml. The non-optional services are:

Optional / behind compose profiles:

• postgis (postgis/postgis:16-3.5) under profile spatial — enables the optional spatial-data feature.
• shepard-hsds (hdfgroup/hsds:v0.9.5) under profile hdf — enables the optional HDF5 / HSDS feature (details below).
• mongoexpress for ad-hoc DB browsing.
• prometheus (prom/prometheus:v3.9.1) and grafana (grafana/grafana:12.2.1-security-01) under profile monitoring — see Observability.
• timescale-migration-preparation under profile timescale-migration-preparation for the InfluxDB → TimescaleDB migration.

Bring it up:

cd infrastructure
docker compose --env-file .env up -d
# add profiles as needed:
docker compose --env-file .env --profile spatial --profile monitoring up -d

Pre-deploy checklist

Independent of where you host:

• Rotate .env defaults. aidocs/07 H8 explicitly flags the shipped POSTGRES_*, NEO4J_PW, MONGO_PASSWORD, etc. as already-public placeholders.
• Restrict CORS. quarkus.http.cors.origins=* in the bundled application.properties is permissive; tighten before any internet-exposed deployment (aidocs/07 C2). See Configuration for the knobs.
• Decide on auth. All paths assume an OIDC provider. The simplest path is to add a Keycloak service to the same compose; a managed OIDC is less ops but shifts trust to a third party. See Authentication.
• Decide on backups. Each substrate has its own recipe; pick one before you load real data, not after. See Backup and restore.
• Pick a file-storage adapter. Default is GridFS (no extra service); Garage S3 is the production-grade option. See Storage substrate.

Hosting paths

shepard’s full stack (Quarkus + Neo4j 5 + Mongo 8 + TimescaleDB + optional PostGIS + Nuxt 3) needs roughly 8 GB RAM and 20 GB disk to run comfortably. That requirement narrows the hosting field.

Comparison matrix

The default compose under infrastructure/ bundles Caddy as a reverse proxy with automatic Let’s Encrypt; it covers the internet-exposed deploy without further work. Swap in Traefik / nginx if your operator preference runs that way; the upstream production guide on the GitLab wiki at gitlab.com/dlr-shepard/shepard is the authoritative source for production-shape deploys.

Hetzner Cloud (cheapest paid)

A Hetzner CX22 (2 vCPU, 4 GB, ~€4/month) is too small for shepard in default shape; a CX32 (4 vCPU, 8 GB, ~€7/month) is the workable floor; CCX13 (4 dedicated vCPU, 16 GB, ~€15/month) is the production-shape reference the upstream team uses for staging. Same docker-compose as everywhere else; the bundled Caddy gets you TLS without further config.

Bare-metal / lab box

Any always-on Linux box with 8 GB+ RAM and Docker installed can run the full stack. Clone the repo, copy infrastructure/.env.example to .env, rotate the secrets, then docker compose up -d. Caddy fronts the stack; supply a public hostname (or a Cloudflare Tunnel) to get a TLS cert.

Managed-services split

For a backend-only proof-of-concept where you skip the frontend and exercise shepard’s REST API:

• Neo4j AuraDB Free — 50K nodes / 175K relationships limit. Plenty for the Showcase Seed.
• MongoDB Atlas M0 — 512 MB, free forever. Good for files and structured-data payloads at small scale.
• Render Free Postgres — 1 GB, but TimescaleDB extension is not available on Render’s managed Postgres; you’d have to drop timeseries from the demo or run TimescaleDB elsewhere.

The connection strings drop into the backend container’s env directly — NEO4J_HOST, QUARKUS_MONGODB_CONNECTION_STRING, QUARKUS_DATASOURCE_JDBC_URL. See Configuration for the full list of consumed environment variables.

This path’s main draw is no infra to manage; its cost is provider sprawl (three accounts, three dashboards, three billing relationships). Not recommended unless you have a specific reason to avoid running databases yourself.

GitHub Codespaces / Gitpod

For interactive evaluation only. A 4-core Codespace with the docker-in-docker feature can run shepard’s compose end-to-end via docker compose up; access is via Codespaces’ auto-issued public HTTPS URL on the forwarded port. The instance shuts down on idle and data does not survive the codespace’s lifecycle — fine for a 30-minute demo, not for anything you’d revisit.

HDF5 (HSDS) — opt-in sidecar

The HDF5 payload kind is off by default. With the toggle off, every /v2/hdf-containers/... endpoint returns 404 Not Found and no HSDS HTTP client is ever instantiated. See docs/reference/hdf-container.md for the API surface; this section covers the operator side.

Phase 1 only. The current slice (backlog ID A5a — see aidocs/35) ships HdfContainer create / read / delete plus the HSDS sidecar itself. The per-DataObject HdfReference, the byte-identical download fallback, and the shared-Keycloak token relay are deferred to A5b – A5e. Phase 1 uses HTTP Basic between the shepard backend and the HSDS sidecar.

Install steps

1. Pick credentials and put them in .env. The compose service defaults to admin / admin — fine for an air-gapped dev box, never fine for any deployment reachable from the public internet.

   # in infrastructure/.env
   HSDS_USERNAME=hsds-admin
   HSDS_PASSWORD=<32-char-random-secret>
   HSDS_BUCKET_NAME=shepard
   

2. Mirror the credentials onto the backend. The backend speaks HTTP Basic to the sidecar; supply the same username / password as environment variables consumed by application.properties.

   # also in infrastructure/.env, then reference from docker-compose.yml
   # as env to the backend service:
   SHEPARD_HDF_ENABLED=true
   SHEPARD_HDF_HSDS_USERNAME=hsds-admin
   SHEPARD_HDF_HSDS_PASSWORD=<same-32-char-secret>
   # endpoint defaults to http://shepard-hsds:5101 (the compose service
   # name), no override needed when running inside the compose network.
   

   Then add these to the backend service’s environment: block in infrastructure/docker-compose.yml next to the other SHEPARD_* keys. Startup fails fast if SHEPARD_HDF_ENABLED=true but the credentials are blank — Phase 1 deliberately refuses to run in “ambient auth” mode.

3. Bring up the profile.

   cd infrastructure
   docker compose --env-file .env --profile hdf up -d
   

   Verify the sidecar is up:

   curl -fsS -u "$HSDS_USERNAME:$HSDS_PASSWORD" http://localhost:5101/about
   

4. Restart the backend (so it picks up the new env). On startup the shepard log emits one line per HsdsClient construction: HSDS client initialised against endpoint=http://shepard-hsds:5101 (HTTP Basic, Phase 1).

5. Smoke test from the API.

   # Create a container.
   curl -fsS -X POST http://localhost:8080/v2/hdf-containers \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer <shepard-api-key>" \
        -d '{"name":"smoke","description":"first hdf"}'
   # Returns: {"appId":"019…","hsdsDomain":"/shepard/019…/", …}
   
   # Read it back.
   curl -fsS http://localhost:8080/v2/hdf-containers/019… \
        -H "Authorization: Bearer <shepard-api-key>"
   

Capacity planning

The compose service mounts ./hsds-storage for the POSIX storage backend (HSDS’s default). Object-store backends (S3 / MinIO / Azure Blob) are opt-in via HSDS upstream env vars — see aidocs/35 §3.

Rule of thumb (per HSDS docs): plan for ~1.2× the raw HDF5 size on disk because of HSDS’s chunk-store overhead. A 100 GB HDF5 file → ~120 GB on disk.

Add ./hsds-storage (or your S3 bucket prefix, if you flip storage backends) to your backup checklist. HSDS supplies the hsadmin CLI for higher-level export / import if you need storage-backend migration.

What’s deferred

Until A5e ships the auth bridge, users authenticate to HSDS directly with the admin credentials configured here (or per-user Basic credentials you provision via hsadmin). Once A5e lands, shepard mints per-user JWTs signed by a shared Keycloak realm and the h5pyd ergonomics (“one credential, the shepard API key”) arrive — see aidocs/35 §5.

Neo4j plugins

Operators get the neosemantics (“n10s”) plugin enabled by default (NEO4J_PLUGINS=["n10s"] plus n10s.* allowed in dbms.security.procedures.{allowlist,unrestricted} in infrastructure/docker-compose.yml). The plugin backs the new SemanticRepositoryType.INTERNAL connector type — see Semantic repositories for the casual-user story.

If you customised the Neo4j service and removed the procedures allowlist line for n10s.*, you’ll need to add it back, otherwise shepard’s startup hook logs:

N10sBootstrapHook: neosemantics (n10s) procedures not registered in Neo4j.
SemanticRepositoryType.INTERNAL will report unhealthy.

That’s the only operator-visible signal — the rest of shepard (including external SPARQL/JSKOS/SKOSMOS repositories) continues to work unchanged.

The bootstrap calls n10s.graphconfig.init(...) exactly once per fresh database; it’s idempotent and safe to restart through. Override the handleVocabUris mode via the shepard.semantic.internal.handle-vocab-uris property (default IGNORE), or fully disable the bootstrap with shepard.semantic.internal.enabled=false.

shepard plugins

shepard core stays small; new payload kinds, external integrations, and identifier providers ship as shepard-plugin-* JARs that an operator drops into /deployments/plugins/ without rebaking the image. The first shipped plugin is the Helmholtz Unhide publish feed (UH1a); HDF5, video, AAS, ePIC, DataCite, GridFS-vs-S3, and others queue behind it.

Quick install:

cp shepard-plugin-foo-1.2.3.jar /deployments/plugins/
# restart shepard so PluginRegistry picks it up
docker compose restart shepard-backend

Inspect what’s loaded:

shepard-admin plugins list
# Plugin       Version   State      Source
# unhide       1.0.0     ENABLED    build classpath

Full operator runbook, lifecycle states, troubleshooting in Plugins (reference).

The design rationale (drop-in JARs via Java ServiceLoader, not compose-side sidecars or per-install Dockerfile forks) is recorded in ADR-0023.

Reverse proxy

caddy terminates TLS on ports 80 / 443 (and 443/UDP for HTTP/3). Configuration lives in infrastructure/proxy/Caddyfile; static SSL material in infrastructure/proxy/ssl. Local-development variants are under infrastructure-local/ (Keycloak + a developer-friendly compose file).

Compliance and certifications

Cloud exit strategy (Cloud-Strategie §8)

Operators at German public institutions who are required to conduct a Vorabprüfung (prior assessment) under the Cloud-Strategie des DLR (v1.0, 2019-09-26) §5.3 / point 8 may cite the following: Shepard’s docker-compose deployment satisfies the §8 Container-Lösungen + Exit-Strategie requirement structurally. All application containers are built from standard OCI images — Quarkus/JVM (backend), Node/Nuxt (frontend), Neo4j 5, TimescaleDB (PostgreSQL-compatible), MongoDB, Garage (S3-compatible object storage) — and no proprietary cloud-provider APIs are consumed. The entire stack is cloud-provider-agnostic: an operator can migrate a running deployment to any docker-compose–capable host (on-premises, a different VPS provider, or a DLR-internal server) by copying the infrastructure/docker-compose*.yml files together with the named volume data directories (Neo4j, MongoDB, TimescaleDB, Garage) and repointing the .env file. All research data is accessible via the Shepard REST API and the shepard-admin CLI at any time; no data is locked into a proprietary cloud format. The source code is MIT-licensed and self-hostable with zero runtime dependencies on any proprietary cloud service. This exit-strategy fitness was established at initial deployment and is preserved by the cloud-provider-agnostic constraint on all future dependencies (enforced via the project’s dependency-review gate in .github/dependency-review-config.yml).

Provider certifications (Hetzner Cloud)

The nuclide.systems reference deployment runs on Hetzner Cloud (Falkenstein, Germany). Hetzner Online GmbH holds the following certifications relevant to German public institution cloud-use assessments:

• ISO/IEC 27001:2022 — certified by SOCOTEC Certification Deutschland GmbH. Scope: “all hosting services and the data centers” (covers all Hetzner datacentre parks and entire infrastructure). Certificate: https://www.hetzner.com/assets/downloads/ISO-Certificate.pdf
• BSI C5:2020 Type 2 attestation — cloud-services scope; confirms implementation and effectiveness of security controls over a defined audit period (Type 2 = period-of-time coverage, not point-in-time). Technical and Organisational Measures documentation: https://docs.hetzner.com/general/security-and-identify/technical-and-organizational-measures/
• § 8a BSI-KritisV — Hetzner is classified as a KRITIS operator under German critical-infrastructure law; compliance is documented via the prescribed BSI audit and verification procedures.

The C5:2020 Type 2 attestation and BSI-KritisV compliance are specific to Hetzner’s German datacentre operations and their cloud-services product line. Operators deploying Shepard on other cloud providers should verify their own provider’s certification posture — the Cloud-Strategie §”Empfehlungen” recommends preferring ISO 27001- and BSI C5-certified providers where available. Hetzner publishes its current certification status at https://www.hetzner.com/legal/certifications/ and in its ISMS documentation.

Integrations

Third-party tools that integrate against a running shepard instance live under docs/install/ — e.g. Apache Superset for SQL-native BI against the TimescaleDB hypertables.