Skip to content

0.5.0

proTES now serves TES 1.1 and relays tasks to TES instances of both TES 1.0 and 1.1, adapting each task to the version of the instance that runs it. It moves to FOCA 0.20 and an image without pip, whose dependencies are locked and hash-checked. TES instances can set their own API path, so TESP-API works without a rewriting proxy, and the view FULL now serves the logs of the TES instance. Lookups of addresses at third parties become opt-in. Tested in CI against real instances of Funnel 0.11.14 and TESP-API 0.2.0, but not against TESK or on Rahti. Several changes are breaking: read the upgrade steps first. See the documentation for details.

Breaking changes

  • API: after a request to cancel it, a task stays CANCELING until its TES instance reports a final state, where it was CANCELED right away before; tasks may also be PREEMPTED, so clients that accept only the states of TES 1.0 (e.g., py-tes before 1.1) need an update (#241, #392, #405)
  • Tasks with ignore_error: true, strict backend_parameters or wildcards in output paths no longer go to TES instances of TES 1.0: they get 400 if no TES 1.1 instance is selected, or 503 if an instance is unconfirmed (see TES versions) (#393, #394, #405)
  • Funnel and other TES instances that report no valid TES version count as TES 1.0, and need the label tes.version: "1.1"; Funnel 0.11.14 reports an empty type.version (see TES versions) (#393, #405)
  • API: outputs with wildcards in their path and without path_prefix get 400, as TES 1.1 requires it, also if the path means *, ? or [...] literally (e.g., /data/sample[1].txt), which TES 1.1 can't escape: set path_prefix for them (#418, #421)
  • API: tasks with a path over 4096 bytes get 400 (path of inputs and outputs, path_prefix, volumes, and workdir, stdin, stdout and stderr of executors) (#408, #410)
  • The default pipeline is random routing, without lookups at third parties; tes.routing.distance is no longer available unless it is listed in plugins.available, and pipelines that use it without that stop the app at startup (see the upgrade steps) (#384, #385)
  • proTES runs on FOCA 0.20 and foca-plugins 0.2.0. The image has no pip, so pip install in derived images and init scripts fails, and the recipe for derived images changes; plugins that cap foca<0.20 no longer resolve; records in failed_runs are keyed by run-<run ID> (see Plugin images) (#399, #336, #407)
  • Helm chart: the chart sets max_page_size_full to 6 instead of the app's 64, as one page of the view FULL could exceed the memory limit of 256 MiB, so clients get smaller pages. config.custom: null and helm upgrade --reuse-values keep 64 (#414, #420)
  • Helm chart: the chart uses foca-common 0.4.0, whose rendering fails on some values that it accepted before (see the upgrade steps) (#398, #406)
  • The filters tag_key and tag_value of GET /tasks need MongoDB 5.0 or later; other requests are not affected (#392, #405)

Features

  • proTES serves TES 1.1.0, with the filters state, tag_key and tag_value of GET /tasks and path_prefix in outputs (#241, #392, #405)
  • proTES reads the TES version of each TES instance from its service info or the label tes.version, and adapts tasks to it (#393, #394, #405)
  • TES instances can set their API path (api_path, apiPath in the chart), e.g., /v1 for TESP-API (#411, #417)
  • The view FULL serves the standard output and error and the system logs of the TES instance, stored once a task has finished, at most post_task.max_log_size per task (1 MiB by default, 0 turns it off), so stored tasks and pages of the view FULL get larger; tasks that finished before keep what they have (#413, #419)
  • custom.service_info.list_backends leaves the TES instances out of the public service info (#396, #366, #401)
  • Helm chart: values for the plugin store's key secret, for several pods of the app and for its probes (#398, #406)

Security

  • A crafted output path could block a worker, as the check for wildcards took quadratic time; it now takes linear time (#408, #410)
  • The hosts of inputs are looked up at DB-IP and Nominatim only if the admin enables tes.routing.distance (#384, #385)
  • The image installs exactly what uv.lock locks, hash-checked (#365, #389)
  • The image has no pip or uv, and CI checks that; the accepted vulnerabilities of packages vendored by pip are gone (#336, #383, #386, #407)
  • Startup errors of the config no longer show the invalid input, and URLs of the identity provider with user info stop the app at startup, as they would end up in the logs (#399, #407)

Fixes

  • Tasks of TES instances that add fields to tasks, e.g., author of TESP-API, stayed UNKNOWN forever; they are now tracked to their final state (#409, #412)
  • A cancel that the TES instance refuses because the task has finished or is gone gets 200; other refusals and failures get 502 instead of 500 (#418, #421)
  • Further log entries of a TES instance (e.g., of retries) no longer lose the logs and outputs of a task (#418, #421)
  • Discovered TES instances fail over when a TLS handshake fails or a request is invalid, as nothing was sent (#399, #407)

Documentation

  • Which TES implementations were tested, and notes on them: Funnel 0.11.14 and TESP-API 0.2.0 ran in CI; TESK and a deployment on Rahti were not tested (#427, #428; #429, #431)
  • Deployment on OpenShift (#418, #421)
  • The upgrade path from 0.4.0 (#387, #388; #429, #431)

Maintenance

  • CI runs tasks on real instances of Funnel and TESP-API with Pulsar, in the job Integration, which is not part of the gate (#422, #426)