Async SaaS integrations

Overview

Fides offers support for async integrations where an access or erasure request cannot be fulfilled immediately. Two async strategies are supported: callback and polling.

• Callback: The vendor calls back to Fides when the job is complete. Fides waits passively.
• Polling: Fides submits the request and then actively polls the vendor's status endpoint on a scheduled interval until the job is complete.

Prerequisite: Enable DSR 3.0

DSR 3.0 task scheduling added the ability to run tasks in parallel and introduced task persistence to the application database to support async integrations. To enable DSR 3.0, you must be running a celery worker as part of your Fides deployment, in addition to the environment variables below:

FIDES__CELERY__TASK_ALWAYS_EAGER=false
FIDES__EXECUTION__USE_DSR_3_0=true

Further, you can optionally update these configuration variables which specify how long data lives in your system and the time to wait between status updates. See Configuration variables for more information.

FIDES__REDIS__DEFAULT_TTL_SECONDS=604800
FIDES__EXECUTION__REQUEST_TASK_TTL=604800
FIDES__EXECUTION__STATE_POLLING_INTERVAL=30

Callback strategy

In the callback strategy, the third-party service is responsible for POSTing results back to Fides when the async job completes. The integration config must denote strategy: callback and include reply-to and reply-to-token headers so Fides can pass callback instructions to the vendor.

Example Async SaaS Config Endpoint

In the example below, we describe how a third-party custom integration can denote its callback async strategy, as well as add reply-to and reply-to-token values to request headers to tell the third party service how to send the results asynchronously back to Fides.

saas_config:
  fides_key: saas_async_config
  name: Async Callback Example Custom Connector
  type: async_callback_example
  description: Test Async Config
  version: 0.0.1

connector_params:
    - name: domain
    - name: api_token
      label: API token

client_config:
    protocol: http
    host: <domain>
    authentication:
      strategy: bearer
      configuration:
        token: <api_token>

test_request:
    method: GET
    path: /

endpoints:
    - name: user
      requests:
        read:
          method: GET
          path: /api/v1/user
          query_params:
            - name: query
              value: <email>
          param_values:
            - name: email
              identity: email
          async_config:
            strategy: callback
          headers:
            - name: reply-to
              value: <reply_to>
            - name: reply-to-token
              value: <reply_to_token>

Asynchronous Access Results

Asynchronous access results can be posted back to the Fides reply-to endpoint as a list of rows, using the reply-to-token as the Bearer Token. Note that these results may be filtered and returned to the end user and/or used to make downstream dependent requests.

    curl -X 'POST' \
      'http://localhost:8080/api/v1/request-task/callback' \
      -H 'accept: application/json' \
      -H 'Authorization: Bearer <reply_to_token>' \
      -H 'Content-Type: application/json' \
      -d '{\
          "access_results": [\
            {"id": "user_id", "system_id": "test_system_id", "state": "test_state"}\
          ]
      }'

Polling strategy

In the polling strategy, Fides submits the initial request to the vendor and then actively checks a vendor-provided status endpoint on a recurring schedule until the job is confirmed complete. Use this strategy when the vendor exposes a status check endpoint but cannot — or will not — POST results back to Fides.

How it works

1. Fides sends the initial access or erasure request to the vendor endpoint.
2. Fides extracts a correlation ID from the response body using the correlation_id_path field. This ID uniquely identifies the async job at the vendor.
3. Fides stores the job as a sub-request and enters a waiting state.
4. On each scheduled check (controlled by FIDES_EXECUTION_ASYNC_POLLING_INTERVAL_HOURS, default every 1 hour), Fides calls the status_request endpoint, substituting <correlation_id> in the path or parameters.
5. Fides reads the status field at status_path in the response. When it matches status_completed_value, the job is considered done.
6. For access requests, Fides then calls the result_request endpoint to retrieve the user's data. For erasure requests, no result fetch is needed — completion of the status check is sufficient.
7. If the job does not complete within FIDES__EXECUTION__ASYNC_POLLING_REQUEST_TIMEOUT_DAYS (default: 30 days), the task is marked as errored.

Example Async Polling SaaS Config

The example below shows a connector with both an access (read) and an erasure (delete) endpoint configured for polling.

saas_config:
  fides_key: saas_async_polling_config
  name: Async Polling Example Custom Connector
  type: async_polling_example
  description: Example SaaS config using the polling async strategy
  version: 0.0.1

connector_params:
    - name: domain
    - name: api_token
      label: API token
      sensitive: true

client_config:
    protocol: https
    host: <domain>
    authentication:
      strategy: bearer
      configuration:
        token: <api_token>

test_request:
    method: GET
    path: /

endpoints:
    - name: user
      requests:
        read:
          method: GET
          path: /api/access-package
          param_values:
            - name: email
              identity: email
          correlation_id_path: request_id     # JSON path in the response to extract the job ID
          async_config:
            strategy: polling
            configuration:
              status_request:
                method: GET
                path: /api/access-package/status
                status_path: status           # JSON path in the status response to evaluate
                status_completed_value: completed  # value that signals the job is done
              result_request:                 # required for access requests; omit for erasure
                method: GET
                path: /api/access-package/result
        delete:
          method: DELETE
          path: /api/anonymize-user/<email>
          param_values:
            - name: email
              identity: email
          correlation_id_path: correlation_id
          async_config:
            strategy: polling
            configuration:
              status_request:
                method: GET
                path: /api/anonymize-user/<correlation_id>/status
                status_path: status
                status_completed_value: completed
              # result_request is omitted for erasure operations

Polling configuration field reference

* Required unless using request_override for custom status-check logic.