Overview

Ingestion candidates

Before ingestion, ingestion candidates are found by searching through files in a staging area. An ingestion candidate is defined as data with associated metadata.

The following methods of doing this search are implemented:

The ingestion state machine

Each ingestion proceeds through states of the ingestion state machine.

The states of the ingestion state machine are as follows:

stateDiagram-v2 state "STAGING" as staging state "FAILED_INGEST_REQUEST" as failed_ingest_request state "PROCESSING" as processing state "FAILED_PROCESSING" as failed_processing state "PROCESSED" as processed staging --> processing staging --> failed_ingest_request: Failed ingest request processing --> processed: Processing successful processing --> failed_processing: Processing failed

Ingestion states

STAGING

The STAGING ingestion state marks the beginning of the ingestion process where an ingestion has not yet been requested.

FAILED_INGEST_REQUEST

The request for ingestion (i.e. the preprocessing) failed.

PROCESSING

Data is ingested into an “ingestion backend” and metadata is added.

FAILED_PROCESSING

The processing stage failed.

PROCESSED

The processing stage was successful.

Ingestion backends

To accommodate different distributed data management solutions (referred to as “ingestion backends”), the upload and metadata setting functionality has been abstracted. The following ingestion backends are supported:

  • Rucio deterministic RSE (--ingestion-backend-name=rucio)

  • Rucio non-deterministic RSE (--ingestion-backend-name=rucio-non-det)

Metadata

Metadata is first validated against a schema (etc/schemas/metadata.json) before the data associated with it is ingested.

For Rucio data ingestion, the required fields are:

  • namespace: The Rucio scope in which the new file should be located.

  • name: The name of the file within Rucio - the scope:name together is the Data Identifier (DID).

  • lifetime: The lifetime in seconds for the new file to be retained.

Additionally, the following optional fields can be specified:

  • dataset_name: The Rucio dataset name the file will be attached to. The dataset scope will be the same as that specified in namespace.

  • meta: An object containing science metadata fields, which will be set against the ingested file.

Install

For Rucio backends

This service should be run inside a dockerised environment. This is to ensure that the necessary storage management tools e.g. gfal (Rucio only), are included. A dockerfile (and Makefile target make build-image-rucio) is included for this purpose. This image is also built as part of the CI pipeline and is available in the container registry.

Credentials are passed through to the container by environment variables. This can be done by either:

  • volume mounting an existing access token to /tmp/access_token, or

  • setting OIDC_CLIENT_ID, OIDC_CLIENT_SECRET and OIDC_TOKEN_ENDPOINT, or

  • setting OIDC_AGENT_AUTH_CLIENT_CFG_VALUE and OIDC_AGENT_AUTH_CLIENT_CFG_PASSWORD, or

  • setting OIDC_ACCESS_TOKEN

The order of precedence for checking is as listed above.

For long-term use, a mechanism to refresh access tokens is necessary. This can be done either by passing in an encoded oidc-agent session with included refresh token (option 3), using a service client that can obtain a token when required via the client_credentials grant (option 2), or by setting up a volume mount to a token and handling the refresh yourself either using oidc-agent or something similar (option 1).

You may also want to specify a volume mount for the ingestion staging area (which should mount to /tmp/ingest in the container) if you want the staging directory accessible from the host.

Using docker-compose

Deployment by docker-compose is possible using the script supplied. Before calling, be sure to set the necessary environment variables (including sensitive ones shown in the .env.template file). If you are using a service client that can obtain a token via the client_credentials grant, ensure that the RUCIO_CFG_CLIENT_OIDC_SCOPE docker compose environment variable includes offline_access.

The service can be started by running the Makefile target make start-ingest-service-rucio and stopped by running make stop-ingest-service-rucio.

Additional targets are provided for development environments (with overriden values defined in docker-compose.rucio.dev.yml); to use these run the Makefile target make start-ingest-service-rucio-dev and stop with make stop-ingest-service-rucio-dev.

Using Helm charts

Deployment by the Helm is also possible using the charts provided. Before running an install, be sure all variables are set appropriately in the values.yaml.

You will also need to have populated variables related to authn/z. The chart currently supports authn/z either by use of a service client or oidc-agent session. The former needs to be setup by an IAM administrator.

Using FluxCD

In your Vault create the secret:

  • engine: low-aa

  • path: production/srcnet/data_ingestion

    • keys:

      • client_id: IAM Client ID

      • client_secret: IAM Client Secret If you want to test data staging, enable the dummyFileGenerator by enabling it in the etc/helm/values.yaml file

dummyFileGenerator:
  enabled: false  # <-- set this to true
  namespace: sp5203_ingestion # <-- target Rucio namespace for the tests
  lifetime: 60 # <-- lifetime of the file on the Rucio server
  cadence: 5 # <-- test file staging cadence in minutes

kubectl create namespace srcnet-ingestion
kubectl apply -f etc/flux/bootstrap/

The minikube cannot resolve ghcr.io so before running the script add the repo manually

sudo docker pull ghcr.io/fluxcd/image-reflector-controller:v0.34.0
minikube image load ghcr.io/fluxcd/image-reflector-controller:v0.34.0

docker pull ghcr.io/fluxcd/image-automation-controller:v0.40.0
minikube image load ghcr.io/fluxcd/image-automation-controller:v0.40.0

sudo docker pull ghcr.io/fluxcd/source-controller:v1.5.0
minikube image load ghcr.io/fluxcd/source-controller:v1.5.0

sudo docker pull ghcr.io/fluxcd/kustomize-controller:v1.5.1
minikube image load ghcr.io/fluxcd/kustomize-controller:v1.5.1

sudo docker pull ghcr.io/fluxcd/helm-controller:v1.2.0
minikube image load ghcr.io/fluxcd/helm-controller:v1.2.0

Monitoring

If you want to send ingestion events to an existing elasticsearch instance, specify the ELASTICSEARCH_URL and ELASTICSEARCH_INDEX environment variables (optionally also ELASTICSEARCH_USER and ELASTICSEARCH_PASSWORD if behind http basic auth).

Usage

rucio Backend - Rucio Deterministic Ingestion

When the service is run with the rucio backend (environment variable INGESTION_BACKEND_NAME=rucio in the docker-compose.rucio.yml file), files placed in the staging area (with a corresponding metadata file) are moved into the processing area as described in the Ingestion States section above.

From here, the files are uploaded to the specified Rucio ingestion site via the Rucio client.

If this step completes successfully, the files are moved to the processed state. From this point, the ingested files have been copied into the Rucio ingestion RSE, and can be safely removed from the processed area (e.g. when needed to recover disk space).

Demonstration

{width=100% height=100%}

rucio-non-det Backend - Rucio Non-Deterministic Ingestion

When the service is run with the rucio-non-det backend (environment variable INGESTION_BACKEND_NAME=rucio-non-det in the docker-compose.rucio.yml file), the staging area itself must correspond to a space on a disk which is managed by a non-deterministic Rucio Storage Element - see the Rucio documentation for more details on the distinction between deterministic and non-deterministic RSEs. Within this space, the same state machine directories will be created (notably staging and processed) when the service is started for the first time. Now when files are detected in the staging area (with a corresponding metadata file), they are moved straight into the processed area. Because of this, when ingesting files with the rucio-non-det backend, not only must the service have read/write access to the non-deterministic RSE physical storage (to detect new data), but an additional env var RUCIO_PFN_BASEPATH must be set (e.g. <protocol>://<host>:<port>/url/for/ingest/area), identifying the URL that matches the monitored staging area.

From here, the files are registered in-place with the Rucio server. This means that all of the information fields about the files, including a data identifier (DID), file lifetime, Adler-32 checksum, and physical file name (PFN), are added into the Rucio server database, allowing the files to be replicated to further sites.

If the registration step completes successfully, the files will remain in the processed state directory. From this point, these ingested files are managed by Rucio, with further replication and deletion on expiry being controlled via the central Rucio server.

Notes

To check that Flux has correctly updated to the latest image