Skip to content

0.4.0

proTES now processes tasks with FOCA's plugin engine instead of its own middlewares, on FOCA 0.19: plugins check or change tasks, choose the TES instances and their order, and record the provenance of each task, which users read through the API. The image ships the routing plugins of foca-plugins (tes.routing.distance and generic.routing.random), so that, as in 0.3.0, tasks go to the TES instance closest to their input data, with random order as the fallback. The release is built from locked, hash-checked dependencies, and removes forwarded_to from task logs. Several changes are breaking: read the upgrade steps first. See the documentation for details.

Breaking changes

  • Plugins replace the middlewares: new tasks pass through a pipeline of FOCA plugins (plugins.pipeline), and custom.middlewares is refused at startup. The middleware management API and its collection are removed, and custom middlewares must be ported to plugins. Tasks that no TES instance accepts fail with 503 (was 500), requests that already passed proTES with 508 Loop Detected (if server.public_url or plugins.gateway_id is set), and tasks with tags under gateway. other than those of the clients' choices with 400 (see Plugins) (#333, #346; #315, #320; #316, #317)
  • proTES runs on FOCA 0.19 (was 0.17): custom configs must apply the config changes of FOCA 0.18 and 0.19, and the image and the Helm chart of this release must be deployed together (see FOCA 0.18 and 0.19) (#331, #338; #359, #360)
  • TES instances are listed in custom.tes.backends (in the Helm chart, tes.backends) instead of custom.tes.service_list and custom.tes.discovery (see TES instance list) (#332, #340)
  • API: logs[].metadata.forwarded_to is removed without a deprecation period, also from tasks stored by earlier versions; clients read the TES instance and the remote identifier from the gateway tags gateway.backend and gateway.backend_task_id. custom.storeLogs is refused at startup (see Forwards in task logs) (#324, #374)
  • Access control: custom lists of security.access_control.policies must add the rules users-list-plugins, users-get-plugins, owners-get-provenance and admins-get-provenance, or users get 403 (#322, #352; #218, #357)
  • Database: proTES uses the new collections plugin_builds, pipelines, failed_runs and plugin_store, and creates indexes on startup: its database user needs the right createIndex (see Task provenance) (#334, #349)

Features

  • The image ships the routing plugins tes.routing.distance and generic.routing.random of foca-plugins, pinned and hash-checked in uv.lock, and the default pipeline is distance routing with random order as the fallback again, as in 0.3.0. With the default settings, the hosts of inputs are looked up at DB-IP and Nominatim; set provider: offline and a GeoIP database, or FOCA's foca.routing.ordered, to avoid third parties (see Routing) (#379, #381)
  • Task provenance: proTES stores the provenance of each task's plugin pipeline run and serves it with GET /tasks/{id}/provenance to the task's owner and to admins; new tasks have FOCA's gateway tags (gateway.*) that summarize it (#334, #349; #322, #352)
  • Clients may choose the plugins and the TES instances of a task with the tags gateway.pipeline (if the admin enables it) and gateway.backends, and list what they may choose with GET /plugins; each choice applies to its task only (#218, #357)
  • Helm chart: values for the plugins (plugins) and for the settings of the identity provider's issuer (#335, #348; #337, #341)

Security

  • Requests to TES instances discovered in service registries go through FOCA's guarded HTTP client, which checks the address of each connection, so that a host whose addresses change after discovery can't lead requests with proTES's token to private addresses (#339, #347)
  • .env files, PEM, key and PKCS#12 files are excluded from the Docker build context everywhere, also below pro_tes/ (#361, #367)
  • The dev-image push job, which holds the Docker Hub credentials, now derives the version with build requirements installed from uv.lock with hash checks, exported by a digest-pinned uv image (#376, #378)

Fixes

  • The forwarding of a task, with its state request, takes at most 55 seconds with the default settings (plugins.defaults.forward_timeout: 20, was FOCA's 60), within the 60 seconds of common proxies' read timeouts; other work, e.g., registry reads, token requests and database writes, comes on top. Tokens of service registries are no longer logged (#364, #369)
  • Middlewares kept the order of the TES instances, and the changes of failed middlewares were undone, before the plugins replaced the middlewares (#314, #319; #313, #318)

Documentation

  • The upgrade path from 0.3.0 (#362, #363, #368)
  • A recipe for images derived from the image of proTES with other plugins, pinned by hash, tested in CI with a test plugin (#323, #355)
  • The org-wide community files replace those of the repository, including the contributing guide (#325, #326; #370, #372)
  • The README logo, the Zenodo DOI and the docs badge (#350, #351; #307, #309, #311)

Maintenance

  • CI's lint, test, chart and docs jobs install the hash-checked versions from a new uv.lock (uv sync --locked), and Dependabot updates pyproject.toml and the lock together (#375, #377)
  • CI passes in forks, uses the org's shared actions for the CI gate and for scanning images, checks pull request titles under the gate, and checks the chart before signing anything; workflow and job names are aligned and prepared for the repository name protes (#327, #328; #330, #342; #329, #343; #371, #373; #353, #354; #310, #312; #356, #358)
  • Sourcery skips rule-test fixtures (#306)
  • Dependabot updates of the pre-commit hooks and of the GitHub Actions (#344, #345)