Fides Documentation – Ethyca

Dataset annotation reference

This page serves as a comprehensive reference for all dataset annotations available in Fides. Dataset annotations are defined in YAML and help describe your data architecture for privacy request processing.

Dataset properties Permalink for this section

Property	Type	Required	Description
fides_key	string	Yes	A unique identifier for the dataset
organization_fides_key	string	No	The Fides key of the organization that owns the dataset
name	string	No	A human-readable name for the dataset
description	string	No	A description of what the dataset represents
data_categories	sequence	No	Array of FidesLang data categories that apply to this dataset
collections	sequence	Yes	Array of collections/tables within the dataset
tags	sequence	No	Array of tags for the dataset
property_ids	sequence	No	Array of property IDs that this dataset is associated with. When set, this dataset is only included in privacy requests that specify a matching `property_id`. Datasets with no property IDs are treated as universal and included in all requests. See properties for details.
fides_meta	YAML collection	No	Additional metadata about the dataset

- fides_key: {{fides_key}}
  organization_fides_key: {{organization_fides_key}}
  tags: {{tags}}
  name: {{dataset_name}}
  description: {{dataset_description}}
  data_categories:
    - {{data_category}}
      {{data_category}}
  fides_meta: ...

Dataset fides_meta properties Permalink for this section

Property	Type	Required	Description
resource_id	string	No	The resource ID of the dataset
after	sequence	No	A list of collections that should be processed before this dataset
namespace	YAML collection	No	Namespace configuration for the dataset
namespace.connection_type	string	No	Integration type for the namespace (e.g. "bigquery", "postgres", "snowflake", "rds_postgres", "google_cloud_sql_postgres")
namespace.dataset_id	string	No	Dataset identifier (BigQuery)
namespace.project_id	string	No	Project identifier (BigQuery)
namespace.database_name	string	No	Database name (PostgreSQL, Snowflake, RDS PostgreSQL, Google Cloud SQL PostgreSQL)
namespace.schema	string	No	Schema name (PostgreSQL, Snowflake, RDS PostgreSQL, Google Cloud SQL PostgreSQL)
namespace.database_instance_name	string	No	RDS instance identifier (RDS PostgreSQL)

fides_meta:
  resource_id: {{resource_id}}
  after:
    - {{collection_id}}
  namespace:
    connection_type: {{connection_type}}
    # ... fields depend on connection type, see examples below

Namespace fields vary by connection type:

# BigQuery
namespace:
  connection_type: bigquery
  dataset_id: {{dataset_id}}
  project_id: {{project_id}}

# PostgreSQL
namespace:
  connection_type: postgres
  database_name: {{database_name}}
  schema: {{schema}}

# Google Cloud SQL PostgreSQL
namespace:
  connection_type: google_cloud_sql_postgres
  database_name: {{database_name}}
  schema: {{schema}}

# Snowflake
namespace:
  connection_type: snowflake
  database_name: {{database_name}}
  schema: {{schema}}

# RDS PostgreSQL
namespace:
  connection_type: rds_postgres
  database_instance_name: {{database_instance_name}}
  database_name: {{database_name}}
  schema: {{schema}}

Collection properties Permalink for this section

Property	Type	Required	Description
name	string	Yes	Name of the collection/table
description	string	No	Description of the collection's purpose
data_categories	sequence	No	Array of FidesLang data categories that apply to this collection
fields	sequence	Yes	Array of fields within the collection
fides_meta	YAML collection	No	Additional metadata about the collection

collections:
  - name: {{collection_name}}
    description: {{collection_description}}
    data_categories:
      - {{data_category}}
        {{data_category}}
    fields: ...
    fides_meta: ...

Collection fides_meta properties Permalink for this section

Property	Type	Required	Description
skip_processing	boolean	No	If true, this collection will be skipped during privacy request processing
after	sequence	No	A list of collections that should be processed before this collection
erase_after	sequence	No	A list of collections that should process erasures before this collection
masking_strategy_override	YAML collection	No	The masking strategy to use for the collection
masking_strategy_override.strategy	string	No	The masking strategy to use for the collection. Valid values are "delete" or "mask"
partitioning	YAML collection	No	The partitioning strategy to use for the collection
partitioning.where_clauses	sequence	No	A list of where clauses to use for the collection

fides_meta:
  skip_processing: true|false
  after:
    - {{collection_id}}
      {{collection_id}}
  erase_after:
    - {{collection_id}}
      {{collection_id}}
  masking_strategy_override:
    strategy: mask|delete
  partitioning:
    where_clauses:
      - {{where_clause}}

Field properties Permalink for this section

Property	Type	Required	Description
name	string	Yes	Name of the field
description	string	No	Description of the field's contents
data_categories	sequence	No	FidesLang data categories that apply to this field
fides_meta	YAML collection	No	Additional metadata used by Fides for privacy operations
fields	sequence	No	For JSON datasets, nested data are represented as fields of fields

fields:
  - name: {{field_name}}
    description: {{field_description}}
    data_categories:
      - {{data_category}}
        {{data_category}}
    fields: ...

Field references properties Permalink for this section

Property	Type	Required	Description
dataset	string	Yes	The name of the dataset that contains the referenced collection-field relationship
field	string	Yes	The name of the field in the referenced collection/table
direction	string	No	The direction of the relationship (e.g. "to", "from")

references:
  - dataset: {{dataset_id}}
    field: {{collection_id}}.{{field_id}}
    direction: {{direction}}

Field fides_meta properties Permalink for this section

Property	Type	Required	Description
identity	string	At least 1 per privacy request pipeline	Specify the field that should be used as the identity key
references	sequence	No	References to other fields for joins
primary_key	boolean	No	If true, indicates this field is a primary key
data_type	string	No	The data type of the field (e.g. "string", "integer")
length	integer	No	Maximum length for string/text fields
read_only	boolean	No	If true, field cannot be modified
return_all_elements	boolean	No	If true, field will return all elements in a collection
custom_request_field	string	No	The custom field in a privacy request used to associate with this field
automated_processing	boolean	No	If true, field is used in automated decision making

fides_meta:
  identity: {{field_name}}
  references: ...
  primary_key: true|false
  data_type: {{data_type}}
  length: {{length}}
  read_only: true|false
  return_all_elements: true|false
  custom_request_field: {{custom_request_field}}
  automated_processing: true|false