Operations¶
Components¶
A deployment of proTES consists of:
- the app (the TES API), served by Gunicorn;
- one or more Celery workers, which track forwarded tasks at the TES instances (see Task tracking);
- RabbitMQ, the broker between the app and the workers;
- MongoDB, which stores the tasks (database
taskStore, collectiontasks), their provenance (see Task provenance) and the entries of the plugin store (see Plugin store); - optionally Flower, a web UI to monitor the Celery workers.
The Docker Compose setup and the Helm chart run all of them.
Logs¶
The app and the workers log to standard error (section log of the
config): each request to the API, the plugins loaded and failed, the TES
instances that a task is forwarded to, and errors of the TES instances. Read them with
docker compose logs or kubectl logs.
Task tracking¶
The state of a task is the state last tracked from the TES instance that runs
it (see Tasks). The Celery workers poll each unfinished task
at its TES instance every polling.wait seconds (see
Tasks): each poll is a short Celery task, which
schedules the next one, so no worker process is held for the runtime of a
task, and long tasks don't keep the others from being tracked. Once a task has
finished, its state, logs and outputs are stored (tracking writes no other
fields of a task, so its tags are kept), and tracking ends; a canceled
task stays CANCELING until its TES instance has finished it, or answers that
it no longer has it (404; then CANCELED), or until the first poll more than
polling.max_cancel_time seconds after the cancel (default 3600; later if its
tracking has to be resumed first): then, e.g., if the TES instance was
decommissioned, keeps failing or keeps reporting the task as running, the task
is set to SYSTEM_ERROR, its tracking ends, and a warning is logged (not
canceled within ... seconds); it may still run at the TES instance, so check
there. After
polling.attempts consecutive failed polls (e.g., while the TES instance or
the identity provider is down), the state of the task is UNKNOWN until the
TES instance answers again, and polls back off to every polling.max_wait
seconds; such tasks are not marked as failed, as they may still be running.
The workers need little capacity per task: a worker process is busy only
for the duration of a poll (one request to the TES instance). As a rule of
thumb, one process tracks about as many tasks as polls fit into
polling.wait; e.g., with polls of 0.1 seconds and the default wait of 3
seconds, about 30 tasks. Raise --concurrency (processes per worker) or the
number of workers for more unfinished tasks, and watch the queue length in
Flower or RabbitMQ: if polls wait in the queue, tasks are polled less often
than configured, but are still tracked.
Polls are acknowledged after they ran (acks_late), so the broker delivers
the poll of a worker that is lost (e.g., killed after its grace period)
again, and polls that are scheduled but not yet due go back to the broker
when a worker shuts down. Polls that fail because MongoDB is unavailable
are retried, with backoff up to polling.max_wait seconds, and those that
fail because RabbitMQ is go back to it. Each
task is tracked by one chain of polls (the stored task holds the identifier
of its current poll and when it ran, tracking). When a worker starts, and
then every polling.max_wait seconds, it resumes the tracking of stored
unfinished tasks that were forwarded to a TES instance but haven't been
polled for three times polling.max_wait seconds (or ever), e.g., because
RabbitMQ lost its data. Upgrades and restarts of the workers therefore don't
stop the tracking of tasks; they only delay their next polls. Tasks being
canceled (CANCELING) are resumed too (for those canceled by versions before
0.6.0, which didn't store the time of the cancel, max_cancel_time counts
from then on); those that earlier versions stored as CANCELED are not.
Tasks whose creation was cut off (e.g., if a Gunicorn worker was stopped
while it forwarded them, during an upgrade) stay in state UNKNOWN
without identifier at a TES instance: they can be neither tracked nor
canceled. The workers set them to SYSTEM_ERROR when they resume the
tracking of tasks, once the tasks are older than
post_task.max_create_time seconds (default 3600; see
Tasks). As for tasks that a TES instance did not
confirm, the task may run at a TES instance: it is kept, and the logs of
the API name the TES instances that were tried.
Task owners¶
Only the owner of a task can read and cancel it through the API, not admins
(see Task ownership). Operators work on
the database instead (database taskStore, or the one in MONGO_DBNAME),
e.g., with mongosh; the owner of a task is in user_id of its document in
the collection tasks, and as owner roles (g2 rules) in the collection
policy_rules. Owners are subjects (sub claims) of the users' access
tokens; with several trusted issuers, they are namespaced by issuer
(<issuer>|<sub>, or the issuer's namespace; bare for an issuer with
namespace: ""), so <sub> below stands for that form:
// tasks of a user
db.tasks.find({ user_id: "<sub>" }, { "task.id": 1, "task.state": 1 })
// tasks without owner, e.g., created by earlier versions (not tombstones)
db.tasks.find(
{ user_id: { $exists: false }, deleted_at: { $exists: false } },
{ "task.id": 1 }
)
Tasks created before owners were recorded have no owner, so nobody can see them through the API. To give such a task to a user, add the two owner roles via the permissions API, as an admin:
for resource in "/ga4gh/tes/v1/tasks/<id>" "/ga4gh/tes/v1/tasks/<id>:cancel"; do
curl -X POST "http://localhost:8080/admin/access-control/permissions" \
-H "Authorization: Bearer <token>" -H "Content-Type: application/json" \
-d "{\"policy_type\": \"g2\", \"rule\": {\"v0\": \"user:<sub>\", \"v1\": \"owner\", \"v2\": \"${resource}\"}}"
done
and set user_id of the task to the user's sub in the database. To purge
tasks, delete their documents and their owner roles, e.g., those without
owner (leave tombstones, deleted_at, in place: see below):
To delete finished tasks regularly, set a retention of finished
tasks instead: it removes their
owner roles, and leaves tombstones (documents with task.id and
deleted_at only) that keep their identifiers from being used again.
Tombstones are not tasks: never delete them, and leave them out of queries of
tasks (deleted_at: { $exists: false }), as above.
Owner roles left by deleted tasks don't give access to other tasks (their
owners get 404 Not Found, and their identifiers are not used for new
tasks), but are kept in memory by each instance of proTES; delete them
together with the documents, via the permissions API, or in the database
(db.policy_rules.deleteMany({ ptype: "g2", v2: { $in: [...] } }), with the
paths of the deleted tasks, and those ending in :cancel).
Rules of access control changed in the database directly (e.g., with
mongosh) apply only once proTES is restarted (all instances; e.g.,
kubectl rollout restart deployment/<app> with the Helm chart, <app> as in its README, Names and labels), as
proTES keeps the rules in memory and notices only the changes made through
FOCA, e.g., via the permissions API (see FOCA's migration guide to
0.18).
Abusive tasks¶
Admins can't cancel the tasks of others through the API. To stop a task,
cancel it at the TES instance that runs it: its URL is in
tes_endpoint.host of the task document, and its identifier there in
tes_endpoint.task_id; the operators of the TES instance can also cancel
it on their side. If a worker still tracks the task, proTES records its new
state; otherwise, set task.state of the document to CANCELED. To keep a
user from creating more tasks, remove them from the groups with the role
user at the identity provider; to cap the number of tasks that each user
can have, set task limits. To keep proTES
from sending tasks to a TES instance, remove it from custom.tes.backends, or, if it was discovered
in a service registry, have a registry admin remove its verification (it is
no longer used once the registry has been read again, see refresh_interval
in Discovery in a service
registry).
Usage records¶
With accounting of resource
usage, each task that
finished at its TES instance has a record in the collection usage (by
task identifier, _id), with its owner (user_id), and admins get the
totals per user at GET /usage. The records hold the
subject identifier of the owner, which is personal data: they are kept
until you delete them, also after their tasks are deleted (by the
retention of finished tasks
or by hand), so delete them once they are no longer needed for accounting,
e.g., with mongosh:
// records of tasks that finished before a date
db.usage.deleteMany({ finished_at: { $lt: ISODate("2026-01-01T00:00:00Z") } })
// records of a user (e.g., on request)
db.usage.deleteMany({ user_id: "<sub>" })
To delete records automatically some time after their tasks finished, add
a TTL index on finished_at (e.g., after 400 days), in a config
overlay, which replaces the indexes of the collection,
so repeat the others:
db:
dbs:
taskStore:
collections:
usage:
indexes:
- keys:
finished_at: 1
options:
expireAfterSeconds: 34560000
- keys:
user_id: 1
finished_at: 1
MongoDB doesn't change the options of an existing index: drop the index
finished_at_1 first (db.usage.dropIndex("finished_at_1")), then restart
proTES, which creates the indexes of the config. Records whose task
finished at an unknown time are deleted by when they were recorded.
Task provenance¶
Each new task has the provenance of its plugin pipeline in
provenance of its document (FOCA's foca.plugins.ProvenanceStore; the
fields are described in FOCA's
documentation): the
versions of FOCA and proTES, the request identifier, the pipeline (its
source, e.g., default, and the hash of the resolved pipeline), the
plugins that ran (by build hash), the hook calls with their outcomes and
the paths they changed, the TES instances before and after routing, the
client's choice of TES instances, the forwarding attempts (TES instance,
outcome, reason, remote identifier, times) and the final reason, if no TES
instance accepted the task. The record is stored with the task (after
routing), and replaced, with the attempts, once a TES instance accepted the
task (or may have, SYSTEM_ERROR). Records hold no IP addresses, tokens,
header values or claims, and take at most 1 MiB. Owners and admins get
them with GET /tasks/{id}/provenance (see Provenance),
and tasks carry a summary in their tags (gateway.*, see Gateway
tags). Tasks created by earlier versions have none.
Records are deleted with the task. They add no personal data to the task
besides the request identifier, as long as the plugins that ran add none:
records hold the resolved pipeline with the parameters of each plugin
(pipeline.resolved, also for pipelines chosen by clients) and what plugins
annotate. Plugins must declare secret parameters as pydantic.SecretStr
(shown as ********** in the records) and must not annotate personal
data; check this when you review a plugin (see FOCA's
documentation), and
don't put secrets in parameters of presets or the default pipeline
otherwise. Owners and admins get the records, with the parameters.
Shared parts are stored once, by hash (_id), in two more collections of
the database of the tasks, and the records refer to them:
plugin_builds: the builds of the plugins that ran (plugin, package, version, entry point, digest of the installed files, and the image digest, ifFOCA_IMAGE_DIGESTis set);pipelines: the default and preset pipelines (resolved); pipelines chosen by clients stay in the records.
If they can't be written, the records keep them (builds, resolved) and
have stored: false; creating the task doesn't fail. Neither collection
is cleaned up: they get a document per plugin build (new ones with each
image, if the image digest is set) and per pipeline only, and documents
are small.
Requests whose pipeline run failed (e.g., tasks rejected by a plugin, or
accepted by no TES instance) leave no task. With
plugins.provenance.record_failed: true (off by default, so that no record
of rejected input is kept; see Plugins), their
records go to the collection failed_runs, with the time (failed_at) and
the error (type, status, slug), keyed by the run's random identifier
(run-<run ID>), with the request identifier in the field request_id (a
generated one, unless server.request_id trusts clients' X-Request-ID). Clients can't replace records, but each
rejected request adds one. Look up a request by request_id, and add an
index on it if you do that often. Requests refused before the pipeline started (e.g.,
invalid tasks, loops, or invalid tags under gateway.) have no record, nor
have tasks whose response a plugin failed: they are kept, and run at their
TES instance.
A TTL index deletes the records of failed runs 30 days after they failed
(FOCA's default; db section of config.yaml). To keep them for another
time, change the index in place, and set its expireAfterSeconds in the
config to the same value (else proTES refuses to start, as the configured
index conflicts with the existing one), e.g., for 7 days, in the database
of the tasks (use taskStore, or the one in MONGO_DBNAME):
db.runCommand({
collMod: "failed_runs",
index: { keyPattern: { failed_at: 1 }, expireAfterSeconds: 604800 },
})
To look up the provenance of a task, or recent failed runs (in the same database):
// provenance of a task, with its pipeline and plugin builds
const record = db.tasks.findOne({ "task.id": "<id>" }).provenance
db.pipelines.findOne({ _id: record.pipeline.resolved_hash })
db.plugin_builds.find({ _id: { $in: Object.keys(record.plugins) } })
// failed runs of the last day
db.failed_runs.find({ failed_at: { $gt: new Date(Date.now() - 864e5) } })
Plugin store¶
Plugins may keep small data across requests in FOCA's key-value store
(ctx.store, e.g., caches and per-user counters; see FOCA's
documentation).
proTES itself stores nothing there. The entries are in the collection
plugin_store of the app's first database in db.dbs, the database of the
tasks (taskStore, or the one in MONGO_DBNAME), shared by all instances,
with a namespace (ns) per plugin distribution and an expiry
(expires_at): entries live at most max_ttl seconds (by default 30 days;
see Plugins), and MongoDB removes expired
entries about once a minute. Plugins whose store is turned off
(enabled: false) keep their entries in the memory of each process
instead, without size limit: give them a short max_ttl (e.g., an hour).
proTES creates the indexes of the collection on startup (a TTL index on
expires_at, one on ns), so its database user needs the right to create
indexes (createIndex, part of the role readWrite). If that fails,
proTES starts anyway and logs a warning (an error, with the cause, for
conflicting indexes, failed authentication or missing rights), and writes
of plugins to the store fail until the indexes exist. On each start, proTES
also cuts the expiry of stored entries to the current max_ttl; if that
fails (logged), they keep their expiry until a later start succeeds. During
a rolling update that lowers max_ttl, instances with the old setting
still write entries with the old max_ttl: once all instances have the new
setting, restart them once more (e.g., kubectl rollout restart
deployment/<app> with the Helm chart), so that all entries expire within
the new max_ttl.
What each plugin keeps there is listed in the plugin listing (data_stored,
with the mode of its store, storage: shared in the database, local in
the memory of each process, or off; see Choices of
clients). Entries keyed by users are
personal data, kept at most max_ttl. To remove the entries of a plugin
distribution earlier (e.g., one that is no longer installed), delete them
in the database of the tasks:
Verifying images and charts¶
Images on Docker Hub (elixircloud/protes) of releases (X.Y.Z, from
version tags) and of commits on dev (dev, sha-<commit>) are built by
this repository's workflows on GitHub Actions, with an SBOM and the build
provenance attached as attestations, a GitHub artifact
attestation
of their provenance, and a keyless Sigstore
signature (images tagged by date, of earlier deployments, are not). To check
that an image was built by the release workflow, run (with the GitHub
CLI or
cosign 3 or
later, which reads the signatures in the Sigstore bundle format):
gh attestation verify oci://docker.io/elixircloud/protes:0.6.0 \
--repo elixir-cloud-aai/protes \
--signer-workflow elixir-cloud-aai/protes/.github/workflows/release.yml \
--source-ref refs/tags/v0.6.0
cosign verify docker.io/elixircloud/protes:0.6.0 \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity https://github.com/elixir-cloud-aai/protes/.github/workflows/release.yml@refs/tags/v0.6.0
These are the commands for release 0.6.0: replace it with the
version to verify. --source-ref and the ref in
--certificate-identity check that the workflow ran for the release tag,
not for another branch. The workflow files were renamed after 0.3.0:
releases after 0.3.0 are signed by release.yml instead of release.yaml.
The signatures and attestations name the repository as it was called when
the workflow ran, and cosign compares the identity exactly, also its case.
The repository's name changes from proTES to protes (GitHub ignores the
case in URLs, so links of both forms work): releases up to 0.3.0 are signed
as elixir-cloud-aai/proTES, releases after the rename as
elixir-cloud-aai/protes (in --repo, --signer-workflow and
--certificate-identity), for images and charts alike. The commands for
images above and for charts below use the form of release
0.6.0.
Images of commits on dev (dev, sha-<commit>) are signed and attested
by the CI workflow instead (by checks.yaml instead of ci.yml before the
workflow files were renamed). These commands accept both forms of the
repository's name and of the workflow file, for images built before and after
the renames:
gh attestation verify oci://docker.io/elixircloud/protes:dev \
--owner elixir-cloud-aai \
--cert-identity-regex '^https://github\.com/elixir-cloud-aai/(proTES|protes)/\.github/workflows/(ci\.yml|checks\.yaml)@refs/heads/dev$' \
--source-ref refs/heads/dev
cosign verify docker.io/elixircloud/protes:dev \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp '^https://github\.com/elixir-cloud-aai/(proTES|protes)/\.github/workflows/(ci\.yml|checks\.yaml)@refs/heads/dev$'
To show an image's SBOM or provenance, run:
docker buildx imagetools inspect docker.io/elixircloud/protes:0.6.0 \
--format '{{ json .SBOM }}'
docker buildx imagetools inspect docker.io/elixircloud/protes:0.6.0 \
--format '{{ json .Provenance }}'
The Helm chart attached to a release (protes-0.6.0.tgz) has a GitHub
artifact attestation of its provenance and a Sigstore bundle with its
signature (protes-0.6.0.tgz.sigstore.json, attached too):
gh release download v0.6.0 --repo elixir-cloud-aai/protes \
--pattern 'protes-0.6.0.tgz*'
gh attestation verify protes-0.6.0.tgz \
--repo elixir-cloud-aai/protes \
--signer-workflow elixir-cloud-aai/protes/.github/workflows/release.yml \
--source-ref refs/tags/v0.6.0
cosign verify-blob protes-0.6.0.tgz \
--bundle protes-0.6.0.tgz.sigstore.json \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity https://github.com/elixir-cloud-aai/protes/.github/workflows/release.yml@refs/tags/v0.6.0
Images are scanned for known vulnerabilities with Trivy
in CI and again before a release is published (both fail on fixable critical
and high vulnerabilities), and the latest image weekly. The repository's
code scanning alerts show the findings that have a fix, which the maintainers
can act on; findings without a fix (often OS packages of the base image)
appear there once a fix is released. The weekly scan also keeps a full report,
those findings included, as a workflow artifact of each run. The published
images (releases and dev) are the scanned ones: the workflows push each
image by digest, untagged, scan exactly that digest, and only if the scan
passes (no fixable critical or high vulnerabilities, except accepted ones)
sign and tag it.