Skip to content

Indexing

Indexing periodically copies Stored Signal metadata from a Datasource into plintz, so that users can search for Signals. plintz searches that copy, in Postgres, and doesn't query the Datasource for it.

When Indexing runs

  • At startup, Indexing runs in the background for each Datasource that has never been indexed. A restart doesn't re-index a Datasource that already has an index. If you add a Datasource to plintz.yaml, recreating the backend indexes it. See What happens at startup.
  • On a schedule, Indexing runs for every Datasource at each indexing_interval_seconds. The default is five minutes. The first scheduled run starts one interval after the backend starts.
  • On demand. An Administrator runs Indexing for one Datasource with Index now on the Datasources tab of the Administration page, or calls POST /api/indexing/run to index every configured Datasource now, or just one with ?datasource=<id>. It returns when it is done. Only Administrators may call it; anyone else gets 403. See below.

plintz runs only one Indexing at a time. A request waits if another run is active. Each run indexes Datasources in sequence, so a slow Datasource delays the rest.

Running Indexing on demand

The call needs a session, so copy the plintz_session cookie from a browser, as for the smoke test:

curl -X POST -b "plintz_session=$PLINTZ_SESSION" http://localhost:8080/api/indexing/run

To index only one Datasource, add its id: /api/indexing/run?datasource=synthetic. An id that isn't configured gets 404.

The response has one result per Datasource. Each result has a status of succeeded with the Stored Signal count, or failed with an error message:

{"indexed_signals": 7, "datasources": [{"datasource": "synthetic", "status": "succeeded", "signal_count": 7, "indexed_at": "2026-10-19T08:18:31.880328Z", "error": null}]}

Indexing a Datasource

Indexing reads each Stored Signal's Data ID, name, description, unit of measure, Interpolation Method and Maximum Interpolation from the Datasource. If the Datasource does not provide an Interpolation Method, Indexing uses Step. If it does not provide a Maximum Interpolation, Indexing uses 40 hours. An unrecognized unit of measure makes the Stored Signal unitless.

  • Indexing finds new Stored Signals and changes to their metadata on its next run. It reads new Samples for already indexed Stored Signals directly from the Datasource, without Indexing.
  • Overrides are kept. An Administrator's overrides of a Stored Signal's properties stay as they are, and replace the values Indexing copies.
  • If a Stored Signal is missing from the Datasource, Indexing moves it to the Trash. A later Indexing restores it if it reappears. A Stored Signal that an Administrator moved to the Trash stays there. See Trash. The Audit Trail records each Stored Signal that Indexing moves to the Trash or restores, with the actor Indexing, as one insert for each, in the same transaction as the Indexing of its Datasource.

Each successful Indexing records how many Stored Signals it stored and how many it added and left out compared to the Indexing before it. The Configuration tab of the Administration page shows them for each Datasource. A Stored Signal an Administrator moved to the Trash still counts as stored; one that is missing from the Datasource, or that fails alone as below, counts as left out.

Indexing doesn't touch the block cache. When data changes in a Datasource, see Cache invalidation.

When a Datasource fails

A failed Indexing of one Datasource doesn't stop the others. The Datasource keeps its previous index, unchanged, and plintz logs Indexing of Datasource 'x' failed with the cause, in the traceback's last line. See Logs. The next scheduled Indexing tries it again, and so does a restart if it has never been indexed. An on-demand run reports failed with the error. One cause is that a Datasource's Connector returned the same Data ID twice. Others come from the Datasource's system, such as being unreachable.

A single Stored Signal can fail on its own:

  • If a Stored Signal has an Interpolation Method other than Step, Linear or PILinear (matched case-insensitively), or a Maximum Interpolation that is not zero or more seconds, Indexing leaves it out of the Datasource's signal_count and moves it to the Trash. plintz logs a warning naming the Signal.
  • If a Databricks metadata row has no Data ID or name, or has a Maximum Interpolation that is not a duration such as 2day or 40h, Indexing leaves that Stored Signal out in the same way.

Indexing processes the rest of the Datasource. Fix the metadata at the Datasource, and the next Indexing restores the Stored Signal.

Removing and adding back a Datasource

Remove a Datasource's entry from datasources in plintz.yaml, and recreate the backend. At the next Indexing, its Stored Signals go to the Trash. They can no longer be found by search, and while the Datasource isn't configured, their Samples can't be read.

Adding the entry back, with the same id, restores the Stored Signals at the next Indexing, except those an Administrator had moved to the Trash. Their overrides are kept.

Neither happens at startup, which indexes only Datasources that were never indexed. Wait for the next scheduled run, or run Indexing on demand.