Skip to content

0.3.0

The first release, with the app image, the Helm chart and these notes published together. It changes almost everything since the last deployed version (images tagged by date, FOCA 0.12) and the legacy charts (2.x and 3.0.0): proTES now requires FOCA 0.17.0 (or a later 0.17 release) and Python 3.13 or later, requires access tokens, gives each task to the user who created it, calls TES instances with credentials of its own, can discover them in a service registry, forwards each task exactly once, tracks tasks without blocking workers, and has a hardened chart and image. Many of the changes are breaking: read the upgrade steps first. See the documentation for details.

Known limitations of this release:

  • per-user limits on stored or unfinished tasks are not available yet (#285); limit requests at the ingress controller (see "Rate limits" on the Configuration page of the documentation);
  • finished tasks are kept until they are deleted by hand: retention is not available yet (#286);
  • each task adds two owner rules to the access control database, and FOCA 0.17 loads all rules for each request, so requests slow down as the number of tasks grows; FOCA 0.18 will cache the policy enforcer;
  • once CANCELED is stored, the resume sweep of the workers ignores the task: if its tracking is lost afterwards, its final logs may be missing (see "Task tracking" on the Operations page of the documentation).

Breaking changes

  • API: all TES endpoints except GET /service-info require an access token of a trusted issuer (security.auth.issuers) for a configured audience (security.auth.audience), and a role from the groups of the identity provider (security.access_control.claim_groups, group_mapping); before, anyone could create, list, read and cancel all tasks. HTTP Basic credentials of callers are no longer passed through to TES instances (see Authentication) (#212, #240)
  • API: bearer tokens must be access tokens as per RFC 9068 (typ header at+jwt); other tokens, such as ID tokens, are rejected (401). The new option security.auth.clients limits the accepted tokens to those of certain clients (see Authentication) (#214, #245)
  • API: each task belongs to the user who created it: users list, read and cancel only their own tasks, and admins can no longer see tasks; they only manage the access control rules. Tasks of others and unknown tasks get 403 (was 404 for unknown tasks). The rule users-get-cancel-tasks is replaced by owners-get-cancel-tasks, and the access control model has owner roles and no admin bypass. Tasks stored by earlier versions have no owner, so nobody sees them until operators give them to their users (see Task ownership) (#153, #215, #250)
  • API: errors are problem details (title, status, detail, sent as application/json) instead of objects with message and code; request bodies are limited to 1 MiB (413) (see Errors) (#203, #219)
  • API: times of tasks (creation_time, and start_time and end_time of the logs) are UTC RFC 3339 date-times instead of local-time strings; proTES sets creation_time and rejects tasks that set it (400), as the TES specification marks it read-only (see Task times and remote identifiers) (#208, #238)
  • API: GET /tasks returns an empty next_page_token on the last page (it returned the last task's identifier, so clients kept paging) and an empty page if no task matches; invalid page tokens and page sizes are rejected (400); pages have 256 tasks by default (was 5) and at most 2047; name_prefix is taken literally, not as a regular expression (see Task listing) (#165, #208, #238)
  • API: pages of the view FULL have at most 64 tasks (custom.controllers.list_tasks.max_page_size_full), also by default, so clients that list full tasks must follow next_page_token; inline contents of inputs (inputs[].content) over 128 KiB (custom.controllers.post_task.max_inline_content) are rejected with 413 Content Too Large (see Task listing) (#284, #292)
  • API: tasks that no TES instance accepts (500), and tasks that fail validation after the middlewares (400), are deleted with their owner's roles instead of being kept as SYSTEM_ERROR (see Stored tasks) (#284, #292)
  • API: proTES sends each task to a TES instance with a single request, to <url>/ga4gh/tes/v1 only (earlier versions also tried <url>/v1 and <url>), without following redirects, and no longer fails over to the next path or TES instance when the outcome is unknown (read timeouts, connections lost after sending, 5xx): such tasks fail with SYSTEM_ERROR and 502 Bad Gateway, as the TES instance may run them (see TES instances and their credentials) (#274, #282)
  • API: unfinished tasks whose identifier at their TES instance is unknown can't be canceled (409 Conflict; was 500, or 200 without effect for tasks in state UNKNOWN); tasks in state UNKNOWN are now canceled at their TES instance. New tasks are stored with the state that their TES instance reports, instead of UNKNOWN (see Task times and remote identifiers) (#275, #291)
  • API: tasks with credentials in the URLs of their inputs or outputs (https://user:password@host/...) are rejected (400), for all TES instances; custom.controllers.post_task.url_credentials: strip removes the credentials instead (see TES instances and their credentials) (#213, #251)
  • API: logs[].metadata.forwarded_to is a list of forwards (id, url, forward_time) instead of a nested tesNextTes object; tasks stored by earlier versions are served with their forwards as lists (see Stored tasks) (#173, #219)
  • API: the middleware management API is no longer registered, as its controllers don't exist yet (see Middleware management API) (#196; #203, #219)
  • Configuration: custom configs must apply the config changes of FOCA 0.14 to 0.17 (e.g., exceptions.mapping; see the FOCA release notes and migration guides). The API specification (tes) and the access control rules are named entries, which overlays change by name; the service info is configured in service_info.service and no longer stored in the database (see Configuration) (#203, #219; #214, #245)
  • Configuration: the identity provider is deployment configuration, without defaults: proTES refuses to start without trusted issuers, an audience and a group of the role admin (unless custom.access_control.require_admin is false), without enforced access control, or with service clients, which it doesn't support yet (see Authentication) (#212, #240; #215, #250)
  • Configuration: unknown keys in custom are errors: the app and the Celery worker refuse to start, naming them, instead of ignoring them. Remove the keys that had no effect from config overlays: custom.celery, custom.controllers.celery and custom.controllers.post_task.timeout (post, poll, job); requests to TES instances time out after custom.tes.timeout seconds instead (default 15, was a hard-coded 5) (see Configuration) (#276, #281; #274, #282)
  • Configuration: entries of custom.tes.service_list are objects with the URL of the TES instance (url) and how proTES authenticates at it (auth: client_credentials by default, basic or none), instead of URLs; with client_credentials, the app and the Celery worker need PROTES_CLIENT_ID and PROTES_CLIENT_SECRET and refuse to start without them. URLs of TES instances must be https URLs without credentials (unless custom.tes.url_schemes allows http); tasks are only forwarded to configured (or discovered) TES instances. Custom middlewares that read custom.tes.service_list get these entries instead of URLs (see TES instances and their credentials) (#213, #251)
  • Configuration: the base config lists no TES instances (it listed public ones), and the default service info describes the app with placeholders on reserved example domains (RFC 2606; id is org.example.protes); deployments set their own, and the app warns on startup while the placeholders are set (see TES instances and their credentials) (#229, #251, #263; #205, #220)
  • Configuration: custom.controllers.post_task.polling.wait and attempts must be positive, and wait and the new max_wait below 1800 seconds; custom.controllers.list_tasks.default_page_size must not exceed max_page_size (see Task tracking) (#277, #278, #279, #288; #208, #238)
  • Database: task identifiers are unique: on startup, proTES creates a unique index on task.id and drops the index of earlier versions; it refuses to start while stored tasks share identifiers, and names them (see Task identifiers) (#248, #258)
  • Database: on startup, proTES removes the HTTP Basic credentials of callers that earlier versions stored with each task (basic_auth), and credentials in the URLs of inputs and outputs of stored tasks (see TES instances and their credentials) (#213, #251)
  • Celery: the task that tracks tasks no longer takes credentials, and tasks are tracked by short polls that schedule the next one, so the new worker rejects tracking jobs that earlier versions queued: upgrade while no tasks are being submitted. Unfinished tasks stored by earlier versions keep their last state (see Task tracking) (#213, #251; #277, #278, #279, #288)
  • Celery: after polling.attempts consecutive failed polls, a task is UNKNOWN until its TES instance answers again, instead of failing with a final SYSTEM_ERROR; workers no longer set INITIALIZING when they start tracking a task (see Task tracking) (#278, #288)
  • Helm chart: the chart is replaced by a thin chart over FOCA's library chart foca-common 0.2.0, with camelCase values (e.g., hostName instead of applicationDomain), other resource names and selectors, Secrets of the deployment for MongoDB, the broker and Flower instead of credentials in the values (the broker's user was guest:guest, Flower's admin:admin), Flower off by default, pinned images, NetworkPolicies, and an Ingress for all paths (was /ga4gh/tes/v1 only). Releases of the legacy charts can't be upgraded: uninstall them and install the new chart (see Helm chart) (#216, #252; #246, #270)
  • Helm chart: the chart version follows the app version, so chart versions restart below 3.0.0; the chart attached to a release deploys the image of that release (protes.tag defaults to the chart's appVersion), and the chart of this release needs images of this release (see Image) (#44, #230)
  • Image: latest is the highest stable release (not alpha, beta or release candidate) on main; images of commits on dev are tagged dev and sha-<commit>. Date-tagged images are no longer pushed (see Image) (#44, #230)
  • Image: the image installs proTES as a package, runs as a non-root user, and serves the app with gunicorn --config python:pro_tes.gunicorn_config 'pro_tes.app:init_app()'; /app/pro_tes, gunicorn.py and wsgi.py are gone, and GUNICORN_PROCESSES and GUNICORN_THREADS are replaced by GUNICORN_CMD_ARGS. The Celery worker runs celery --app pro_tes.celery_worker worker (see Image) (#204, #223)
  • Docker Compose: MongoDB 7.0 (was 3.6, which can't be upgraded to 7.0 directly), named volumes instead of PROTES_DATA_DIR, whose data is not migrated, ports published on 127.0.0.1 only, and Flower opt-in (see Docker Compose) (#171, #219; #204, #223)
  • Python: proTES requires Python 3.13 or later, the Python version of the image (based on elixircloud/foca:0.17.0-py3.13), the only one it is tested with (see Configuration) (#214, #245; #246, #270)
  • Middlewares: the distance-based middleware no longer uses the unmaintained package ip2geotools: it looks up IP addresses with DB-IP's API over HTTPS (was HTTP) and geocodes places with Nominatim, times out after 10 seconds (was 62), and no longer locates addresses that are not global, nor places without city or country, for which it falls back to the next middleware (see Distance-based task distribution) (#214, #245)

Features

  • Task ownership: users with the role user create tasks and become their owners (FOCA's owner roles); admins manage the access control rules via FOCA's permissions API (#153, #215, #250)
  • Credentials of proTES for TES instances: an access token of proTES's own client (OAuth 2.0 client credentials grant, from custom.tes.client_credentials, cached and refreshed), HTTP Basic credentials from environment variables, or none, per TES instance (#213, #251)
  • Discovery of TES instances in a service registry with verified services (GA4GH Service Registry API, e.g., Cloud Registry): custom.tes.discovery, off by default, without a default registry. Only verified TES services with public https addresses are used, and the active TES instances are listed in the service info (backends) (#202, #260)
  • Task tracking with short polls that schedule the next one, so that a few worker processes track many tasks; polls are acknowledged after they ran, and workers resume the tracking of stored unfinished tasks when they start and periodically, so restarts and upgrades of the workers no longer stop the tracking (#277, #278, #279, #288)
  • Helm chart: Gateway API HTTPRoute (with an optional redirect to HTTPS), OpenShift Route, Ingress with an optional cert-manager certificate, external MongoDB, protes.extraEnv for the app and the worker (e.g., the client secret), Gunicorn settings, probes, and checks that fail rendering for values of the legacy chart and for missing security settings (#216, #252)
  • ignore_error of executors is relayed to TES instances (#176, #219)
  • Configuration: local overrides (config.local.yaml, docker-compose.override.yaml) are ignored by Git (#266, #269)

Security

  • Signed images and chart: images (releases, dev and sha-<commit>) come with SBOM and provenance attestations, a GitHub artifact attestation and a keyless Sigstore signature; the chart attached to the release has a GitHub artifact attestation and a Sigstore bundle (protes-0.3.0.tgz.sigstore.json). The published images are the scanned ones: images are scanned for known vulnerabilities with Trivy in CI, before a release is published, and weekly (latest). See "Verifying images and charts" on the Operations page of the documentation (#234, #256)
  • Task identifiers can't be shared: a request that draws an identifier in use gets another one, and never gets a role on another user's task (#248, #258)
  • Discovered TES instances must resolve to public addresses only, checked on discovery and before each request; requests to them follow no redirects (#202, #260)

Fixes

  • Tasks that their TES instance reports as PREEMPTED are final (they were polled forever), and CANCELING tasks are polled until they are canceled (#261, #262)
  • Tasks are sent to TES instances exactly once (#274, #282)
  • New tasks are stored with the state that their TES instance reports, and a cancel request while a task is being forwarded keeps it CANCELED (#275, #291)
  • A cancel request no longer replaces a final state that the tracking stored while the request was sent to the TES instance (#296, #301)
  • Tasks whose create request was cut off (e.g., by a rolling upgrade) no longer stay UNKNOWN for good: the resume sweep of the workers sets them to SYSTEM_ERROR after custom.controllers.post_task.max_create_time seconds (new, default 3600, at most a year). On upgrade, this also applies to tasks of earlier versions: UNKNOWN tasks without any remote identifier that are older than that become SYSTEM_ERROR (see Task times and remote identifiers) (#297, #303)
  • Failed polls no longer end the tracking of tasks that are still running, and the tracking survives worker restarts (#277, #278, #279, #288)
  • Task listings iterate the database cursor instead of loading all matching tasks, and the view BASIC leaves out the logs it should (#284, #292; #208, #238)
  • Each TES instance gets its own copy of the task; the remote identifier is stored in tes_endpoint.task_id, so that tasks can also be canceled with custom.storeLogs.execution_trace: False (#208, #238)
  • Startup no longer loops forever with jsonschema > 3.2.0, and no longer fetches the GA4GH service info schemas from GitHub (#173, #203, #219)
  • Output sizes reported by TES instances as integers no longer fail the tracking of finished tasks (#203, #219)
  • The tracking task uses the database named by MONGO_DBNAME, as the app does (#203, #219)
  • The examples of TES instances in the docs and the chart use the current config format (#229, #263)
  • Release workflow: the previous release is found on first releases, and releases are published after their assets are uploaded, as immutable releases require (#272, #273; #280, #287)

Documentation

  • A documentation site on Read the Docs (MkDocs Material), with pages on getting started, configuration, the API, access control, middlewares, deployment, operations, development and upgrading (#209, #228; #210, #221)
  • A code of conduct, a contributing guide, a security policy, CODEOWNERS and citation metadata (#211, #232, #243; #233, #242; #235, #244; #247, #249)
  • Rate limits: how to limit requests at the ingress controller (#284, #292)
  • The served API specification declares the 409 (cancel) and 413 (create) responses of proTES as extensions (#298, #302)
  • The size of FULL pages: a page can reach about 64 MiB by default, so memory-limited deployments (the chart's API pod has 256 MiB) should lower custom.controllers.list_tasks.max_page_size_full (#298, #302)

Maintenance

  • Releases are published from signed version tags by a release workflow, with checks of the tag, the citation metadata, the release notes and the pinned dependencies, also before tagging; the notes of the first release list all its pull requests (#44, #230; #264, #267; #265, #268; #289, #293)
  • The release version comes from one source, the tag (and, before tagging, the citation metadata): the docs and the chart's README get it from there, and a test fails if it is hard-coded elsewhere (#290, #299)
  • Packaging with pyproject.toml and setuptools-scm, and FOCA 0.17.0 and foca-common 0.2.0 releases pinned (#204, #223; #246, #270)
  • Checks with pre-commit (ruff, mypy, semgrep, ast-grep, gitleaks, actionlint), Google-style docstrings and complete type hints, and tests against a real MongoDB at 100% coverage (#227; #206, #231; #142, #149, #168, #207, #236; #159, #239)
  • Pull requests target dev by default, also those of Dependabot (#257, #259)
  • Ported fixes of cloud-registry's first release: Gunicorn keepalive 100 s, generic-repository and docs-URL tests, Dependabot groups, unused Swagger 2 specs removed (#266, #269)
  • Citation metadata are set safely (file mode kept) (#253, #255)