# Annotate Complex Fields

Fides can retrieve and mask data from complex objects and arrays. This involves annotating your dataset files to let Fides know about your complex data.

## Declare an object field [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#declare-an-object-field/index.html)

To declare an `object` field, define nested fields underneath that field. In the example below, `workplace_info` is an object field with two nested fields: `employer` and `position`.

Data categories cannot be specified at the `object` level due to potential conflicts with nested fields. Instead, annotate the scalar fields within the object field. Here, the `workplace_info.position` field has `data_category: user.job_title`.

```yaml
dataset:
  - fides_key: mongo_nested_object_example
    name: Mongo Example with Nested Objects
    description: Example of a Mongo dataset that contains 'details' about customers defined in the 'postgres_example_test_dataset'
    collections:
      - name: customer_details
        fields:
          - ...
          - name: workplace_info
            fides_meta:
                data_type: object
            fields:
              - name: employer
                fides_meta:
                  data_type: string
              - name: position
                data_categories: [ user.job_title ]
                fides_meta:
                  data_type: string
              - name: id
```

## Reference a nested field [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#reference-a-nested-field/index.html)

To define a relationship between a field on one collection and a nested field on another collection, use dot notation in the `fides_meta` references for as many levels are necessary.

In the example below, this field is denoted by `<collection_name>.<field_name>.<sub_field>` name, or `customer_details.workplace_info.id`.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: customer
        fields:
          - name: workplace_id
            data_categories: [system.operations]
            fides_meta:
              references:
                - dataset: mongo_nested_object_example
                  field: customer_details.workplace_info.id
                  direction: to
          ...
```

## Declare an array field [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#declare-an-array-field/index.html)

There is no official `array` type. Instead, an array is represented by a `[]` flag on a field.

### Declare an array of scalar values [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#declare-an-array-of-scalar-values/index.html)

In this example, the `mydatabase:customer` collection has a `travel_identifiers` field that is an string array,
described by `data_type: string[]`. An array of integers would be described by `data_type: integer[]`.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: customer
        fields:
          - ...
          - name: travel_identifiers
            fides_meta:
              data_type: string[]
              data_categories: [system.operations]
```

### Declare a nested array [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#declare-a-nested-array/index.html)

In this example, the `mydatabase:customer` collection has a nested `workplace_info.direct_reports` string array.

We define `direct_reports` as a subfield under `workplace_info`, as well as add the data_type `string[]` to `direct_reports`.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: customer
        fields:
          - name: workplace_info
            fides_meta:
              data_type: object
            fields:
              - name: employer
                fides_meta:
                  data_type: string
              - name: position
                data_categories: [ user.job_title ]
                fides_meta:
                  data_type: string
              - name: direct_reports
                data_categories: [ user.name ]
                fides_meta:
                  data_type: string[]
```

### Declare an array of objects [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#declare-an-array-of-objects/index.html)

In this example, the `mydatabase:customer` collection has an `emergency_contacts` object array field, or
embedded documents, denoted by `data_type: object[]`. Each object in the `emergency_contacts` array can contain a
`name`, `relationship`, and `phone` field.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: customer
        fields:
          - name: emergency_contacts
            fides_meta:
              data_type: object[]
            fields:
              - name: name
                data_categories: [ user.name ]
                fides_meta:
                  data_type: string
              - name: relationship
                fides_meta:
                  data_type: string
              - name: phone
                data_categories: [ user.contact.phone_number ]
                fides_meta:
                  data_type: string
```

### Reference an array [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#reference-an-array/index.html)

Reference an array field as if it is any other field. You cannot currently reference a specific index in an array field, but Fides will search an array field for matches.

In this example, `mydatabase:flights.plane` is an integer field that will be used to lookup records that match an integer
in the `mydatabase:aircraft.planes` array.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: flights
        fields:
          - ...
          - name: passenger_information
            fields:
              - name: passenger_ids
                fides_meta:
                  data_type: string[]
          - name: plane
            data_categories: [ system.operations ]
            fides_meta:
              data_type: integer
      - name: aircraft
        fields:
          - name: _id
            data_categories: [ system.operations ]
            fides_meta:
              primary_key: True
              data_type: object_id
          - name: planes
            data_categories: [ system.operations ]
            fides_meta:
              data_type: integer[]
              references:
                - dataset: mydatabase
                  field: flights.plane
                  direction: from
          - name: model
            data_categories: [ system.operations ]
            fides_meta:
              data_type: string
```

In this more complicated example, a field in an array of objects is used to look up a different field in an array of
objects in another collection. Potentially multiple values from `mydatabase:customer.comments.comment_id` can be used
to query for corresponding values in `mydatabase:conversations.thread.comment`. Because this field is in an array of objects,
multiple matches may be found.

```yaml
dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: customer
        fields:
          - name: comments
            fides_meta:
              data_type: object[]
            fields:
              - name: comment_id
                fides_meta:
                  data_type: string
                  references:
                    - dataset: mydatabase
                      field: conversations.thread.comment
                      direction: to
      - name: conversations
        fides_meta:
          data_type: object[]
        fields:
          - name: thread
            fields:
              - name: comment
                fides_meta:
                  data_type: string
              - name: message
                fides_meta:
                  data_type: string
              - name: chat_name
                data_categories: [ user.name ]
                fides_meta:
                  data_type: string
```

## Query an array [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#query-an-array/index.html)

There are some assumptions made with array querying that may or may not fit with how your data is structured. If an array
is an entrypoint into a collection (i.e., one collection references its array field), there is ambiguity around how
the queries should be built. (e.g., AND versus OR, and whether only the matched indices or matched embedded documents within arrays should be considered).

### Assumptions [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#assumptions/index.html)

1. If an array is the entry point into a node, Fides will search for corresponding matches across the entire array. You cannot specify a certain index.
2. Fides operates on "OR" queries. Data returned from multiple array fields will be flattened before being passed into the next collection.
1. For example, Collection A returned values \[1, 2, 3\], and Collection B returned values \[4, 5, 6\]. Collection C has an array field that depends on both Collection A and Collection B. Fides will search Collection C's array field to return any record that contains one of the values \[1, 2, 3, 4, 5, 6\] in the array.
3. By default, if an array field is an entry point to a node, only matching indices in that array are considered, both for access and erasures, as well as for subsequent queries on dependent collections where applicable.
1. For example, a query on Collection A only matched indices 0 and 1 in an array. Only the data located at indices 0 and 1 will be returned, and used to query data on dependent collection C.
2. This can be overridden by specifying `return_all_elements: true` on an entrypoint array field, in which case, the query will return the entire array and/or mask the entire array.
4. Individual array elements are masked, not the entire array, e.g. \["MASKED", "MASKED", "MASKED"\]

### Example query traversal [Permalink for this section](/content/docs/privacy-requests/guides/complex-fields#example-query-traversal/index.html)

This is an example traversal created from the test `postgres_example` and `mongo_test` datasets. Multiple collections point to or from complex objects and arrays. See the `mongo_example_test_dataset.yml` for more information.