Provenance / activity log (reference)

Provenance reference

shepard captures a W3C PROV-O provenance trail for every mutation and (opt-in) read against entities in its graph. The log is exposed under /v2/provenance/* and rendered through one of four content types depending on the request’s Accept header.

Casual users — for the “what’s been happening in this Collection?” task, see the Monitor Collection activity page in the help track. This page is the wire-shape deep-dive.

Instance admins — for the instance-wide viewer under Admin → Activity Log, see the Instance-wide activity log help page.

What gets captured

A :Activity Neo4j node lands for every authenticated 2xx response on the mutating verbs (POST / PUT / PATCH / DELETE) — plus, when the operator flips shepard.provenance.capture-reads=true, also on GET. Fields:

Captures are best-effort observability — they never block the request and a capture failure never propagates to the user.

Read endpoints

Permissions: casual users see only their own rows. The ?scope=collection stats endpoint is gated on Read permission against the target Collection (admins bypass; missing Collection → 404; lacking Read → 403). Instance-admins see everyone’s rows across every endpoint.

Content types

Four output shapes on the same endpoints, dispatched by the request’s Accept header.

1. Plain JSON (default)

GET /v2/provenance/activities
Accept: application/json

Returns a JSON array of Activity objects matching the field table above. This is the shape the per-Collection dashboard consumes; it’s also the right shape for a quick curl | jq look.

[
  {
    "appId": "0192a4f3-7d8a-7000-8c4b-9f1e2a3b4c5d",
    "actionKind": "CREATE",
    "targetKind": "Collection",
    "targetAppId": "0192a3...",
    "agentUsername": "alice",
    "summary": "POST /v2/dataobjects",
    "startedAtMillis": 1700000000000,
    "endedAtMillis": 1700000000500
  }
]

2. W3C PROV-JSON

GET /v2/provenance/activities
Accept: application/prov+json

The W3C PROV-JSON Submission shape — blocks for prefix, activity, agent, entity, wasAssociatedWith, used, wasGeneratedBy. Suitable for tools that already speak PROV-JSON (some RO-Crate generators, OpenLineage adapters).

{
  "prefix": {
    "prov": "http://www.w3.org/ns/prov#",
    "shepard": "https://noheton.github.io/shepard/prov#"
  },
  "activity": {
    "shepard:activity/<appId>": {
      "prov:type": "shepard:CREATE",
      "prov:startTime": "2023-11-14T22:13:20Z"
    }
  },
  "agent": { "shepard:agent/alice": { "prov:type": "prov:Person" } },
  "wasAssociatedWith": {
    "_:wa0": { "prov:activity": "shepard:activity/<appId>", "prov:agent": "shepard:agent/alice" }
  }
}

3. PROV-O JSON-LD

GET /v2/provenance/activities
Accept: application/ld+json

W3C PROV-O serialised as JSON-LD. The @context is embedded inline — the consuming tool needs no network round-trip to resolve namespaces. Pipes directly into Apache Jena, RDF4J, Stardog, or any other JSON-LD-aware loader.

{
  "@context": {
    "prov": "http://www.w3.org/ns/prov#",
    "shepard": "https://noheton.github.io/shepard/prov#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "@graph": [
    {
      "@id": "shepard:agent/alice",
      "@type": ["prov:Agent", "prov:Person"],
      "shepard:username": "alice"
    },
    {
      "@id": "shepard:entity/<targetAppId>",
      "@type": ["prov:Entity"],
      "shepard:kind": "Collection"
    },
    {
      "@id": "shepard:activity/<appId>",
      "@type": ["prov:Activity"],
      "prov:startedAtTime": { "@type": "xsd:dateTime", "@value": "2023-11-14T22:13:20Z" },
      "prov:endedAtTime":   { "@type": "xsd:dateTime", "@value": "2023-11-14T22:13:20.500Z" },
      "prov:type": "shepard:CREATE",
      "prov:wasAssociatedWith": { "@id": "shepard:agent/alice" },
      "prov:generated": { "@id": "shepard:entity/<targetAppId>" }
    }
  ]
}

Field-level mapping rules:

4. metadata4ing (m4i) JSON-LD

GET /v2/provenance/activities
Accept: application/ld+json; profile="https://w3id.org/nfdi4ing/metadata4ing/"

(Short form: profile=metadata4ing.)

The NFDI4Ing metadata4ing flavour — engineering-research subtypes on top of PROV-O. Activities are dual-typed so a PROV-O-only reader still parses the document. The m4i namespace is added to the @context alongside prov / shepard / xsd.

{
  "@context": {
    "prov": "http://www.w3.org/ns/prov#",
    "shepard": "https://noheton.github.io/shepard/prov#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "m4i": "http://w3id.org/nfdi4ing/metadata4ing#"
  },
  "@graph": [
    {
      "@id": "shepard:agent/alice",
      "@type": ["m4i:Person", "prov:Agent", "prov:Person"],
      "shepard:username": "alice"
    },
    {
      "@id": "shepard:entity/<targetAppId>",
      "@type": ["m4i:InvestigatedObject", "prov:Entity"],
      "shepard:kind": "Collection"
    },
    {
      "@id": "shepard:activity/<appId>",
      "@type": ["m4i:ProcessingStep", "prov:Activity"],
      "prov:startedAtTime": { "@type": "xsd:dateTime", "@value": "2023-11-14T22:13:20Z" },
      "prov:type": "shepard:CREATE",
      "m4i:hasMethod": "shepard:method/CREATE",
      "prov:wasAssociatedWith": { "@id": "shepard:agent/alice" },
      "prov:generated": { "@id": "shepard:entity/<targetAppId>" },
      "m4i:hasOutput":  { "@id": "shepard:entity/<targetAppId>" }
    }
  ]
}

Additional m4i field mappings (in addition to all PROV-O fields, which are kept):

The m4i:hasMethod mapping is a pragmatic placeholder — the HTTP verb isn’t a real method in the engineering-research sense, but it makes the document parseable as a typed event by an m4i-aware consumer. A future slice may wire m4i:hasMethod to an actual template-defined process step (post-T1, see aidocs/54).

Profile negotiation rules

• No Accept header, or Accept: application/json (or any superset that doesn’t mention application/ld+json / application/prov+json) → plain JSON.
• Accept: application/prov+json → W3C PROV-JSON.
• Accept: application/ld+json (no profile= parameter) → PROV-O JSON-LD.
• Accept: application/ld+json; profile="https://w3id.org/nfdi4ing/metadata4ing/" (or ; profile=metadata4ing short form) → m4i JSON-LD.
• Accept: application/ld+json; profile=<unknown> → 406 Not Acceptable with an RFC 7807 application/problem+json body whose type is https://noheton.github.io/shepard/errors/provenance.unsupported-profile and whose detail cites the supported values. The response is always plain application/json (not ld+json) so a JSON-LD parser doesn’t try to ingest the error body.

The negotiation is additive — every existing client (the in-app sparkline composable, the casual curl | jq user, the shepard-py SDK) sees no wire change. The new media types are opt-in per request.

/count under JSON-LD

GET /v2/provenance/count
Accept: application/ld+json

Returns a thin wrapper carrying the integer as shepard:numberOfActivities typed xsd:nonNegativeInteger under the same @context family:

{
  "@context": {
    "prov": "http://www.w3.org/ns/prov#",
    "shepard": "https://noheton.github.io/shepard/prov#",
    "xsd": "http://www.w3.org/2001/XMLSchema#"
  },
  "shepard:numberOfActivities": {
    "@type": "xsd:nonNegativeInteger",
    "@value": "42"
  }
}

The plain JSON variant (no Accept: ld+json) keeps returning { "count": 42 } for backwards compatibility.

Filtering parameters

All three list endpoints accept the same query parameters:

Retention

:Activity rows are pruned by a nightly job (@Scheduled cron 0 42 3 * * ?). The retention window is operator-configurable via shepard.provenance.retention-days=730 (two years; negative value = keep forever). When the dashboard reports “No recorded activity in the selected window” it may be that older rows were pruned — the operator can flip the retention to your needs.

Caveats

• Read capture is opt-in. Without shepard.provenance.capture-reads=true, the log only reflects mutations.
• Request bodies + headers are never echoed. The on-row summary is server-generated from the method + target metadata.
• :Activity is the casual audit trail. F3’s tamper-evident security audit trail (aidocs/24 §3.7) carries the who-gained-what permission diff that doesn’t fit the PROV-O-shaped activity log.
• JSON-LD emission only. shepard renders JSON-LD; it doesn’t ship a JSON-LD parser dependency. Round-tripping a JSON-LD body back into shepard (e.g. for replay) isn’t on the v1 roadmap.

Design references

• aidocs/workflows/55-provenance-and-activity-overhaul.md — the PROV1 design doc (capture filter, dashboard, retention).
• aidocs/workflows/64-provenance-architecture.md §3.2 — the PROV1h / m4i content-negotiation design.
• aidocs/platform/63-architecture-decision-log.md ADR-0004 — why PROV-O over OpenLineage.