Submodules

airbyte_cdk.sources.declarative.transformations.add_fields module

class airbyte_cdk.sources.declarative.transformations.add_fields.AddFields(fields: List[airbyte_cdk.sources.declarative.transformations.add_fields.AddedFieldDefinition], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation

Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate indices with null values). So if you add a field at index 5 to the array [“value”], it will become [“value”, null, null, null, null, “new_value”].

This transformation has access to the following contextual values:

record: the record about to be output by the connector config: the input configuration provided to a connector stream_state: the current state of the stream stream_slice: the current stream slice being read

Examples of instantiating this transformation via YAML: - type: AddFields

fields:

# hardcoded constant - path: [“path”]

value: “static_value”

# nested path - path: [“path”, “to”, “field”]

value: “static”

# from config - path: [“shop_id”]

value: “{{ config.shop_id }}”

# from state - path: [“current_state”]

value: “{{ stream_state.cursor_field }}” # Or {{ stream_state[‘cursor_field’] }}

# from record - path: [“unnested_value”]

value: {{ record.nested.field }}

# from stream_slice - path: [“start_date”]

value: {{ stream_slice.start_date }}

# by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/# - path: [“two_times_two”]

value: {{ 2 * 2 }}

fields

A list of transformations (path and corresponding value) that will be added to the record

Type

List[AddedFieldDefinition]

fields: List[airbyte_cdk.sources.declarative.transformations.add_fields.AddedFieldDefinition]
parameters: dataclasses.InitVar[Mapping[str, Any]]
transform(record: airbyte_cdk.sources.declarative.types.Record, config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → airbyte_cdk.sources.declarative.types.Record

Transform a record by adding, deleting, or mutating fields.

Parameters

  • record – The input record to be transformed

  • config – The user-provided configuration as specified by the source’s spec

  • stream_state – The stream state

  • stream_slice – The stream slice

Returns

The transformed record

class airbyte_cdk.sources.declarative.transformations.add_fields.AddedFieldDefinition(path: List[str], value: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: object

Defines the field to add on a record

parameters: dataclasses.InitVar[Mapping[str, Any]]
path: List[str]
value: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]
class airbyte_cdk.sources.declarative.transformations.add_fields.ParsedAddFieldDefinition(path: List[str], value: airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: object

Defines the field to add on a record

parameters: dataclasses.InitVar[Mapping[str, Any]]
path: List[str]
value: airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString

airbyte_cdk.sources.declarative.transformations.remove_fields module

class airbyte_cdk.sources.declarative.transformations.remove_fields.RemoveFields(field_pointers: List[List[str]], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation

A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.

If an input field pointer references an item in a list (e.g: [“k”, 0] in the object {“k”: [“a”, “b”, “c”]}) then the object at that index is set to None rather than being not entirely removed from the list. TODO change this behavior.

It’s possible to remove objects nested in lists e.g: removing [“.”, 0, “k”] from {“.”: [{“k”: “V”}]} results in {“.”: [{}]}

Usage syntax:

```yaml
my_stream:

<other parameters..> transformations:

  • type: RemoveFields field_pointers:

    • [“path”, “to”, “field1”]

    • [“path2”]

```

field_pointers

pointers to the fields that should be removed

Type

List[FieldPointer]

field_pointers: List[List[str]]
parameters: dataclasses.InitVar[Mapping[str, Any]]
transform(record: Mapping[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]
Parameters

record – The record to be transformed

Returns

the input record with the requested fields removed

airbyte_cdk.sources.declarative.transformations.transformation module

class airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation

Bases: object

Implementations of this class define transformations that can be applied to records of a stream.

abstract transform(record: Mapping[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Transform a record by adding, deleting, or mutating fields.

Parameters

  • record – The input record to be transformed

  • config – The user-provided configuration as specified by the source’s spec

  • stream_state – The stream state

  • stream_slice – The stream slice

Returns

The transformed record

Module contents

class airbyte_cdk.sources.declarative.transformations.AddFields(fields: List[airbyte_cdk.sources.declarative.transformations.add_fields.AddedFieldDefinition], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation

Transformation which adds field to an output record. The path of the added field can be nested. Adding nested fields will create all necessary parent objects (like mkdir -p). Adding fields to an array will extend the array to that index (filling intermediate indices with null values). So if you add a field at index 5 to the array [“value”], it will become [“value”, null, null, null, null, “new_value”].

This transformation has access to the following contextual values:

record: the record about to be output by the connector config: the input configuration provided to a connector stream_state: the current state of the stream stream_slice: the current stream slice being read

Examples of instantiating this transformation via YAML: - type: AddFields

fields:

# hardcoded constant - path: [“path”]

value: “static_value”

# nested path - path: [“path”, “to”, “field”]

value: “static”

# from config - path: [“shop_id”]

value: “{{ config.shop_id }}”

# from state - path: [“current_state”]

value: “{{ stream_state.cursor_field }}” # Or {{ stream_state[‘cursor_field’] }}

# from record - path: [“unnested_value”]

value: {{ record.nested.field }}

# from stream_slice - path: [“start_date”]

value: {{ stream_slice.start_date }}

# by supplying any valid Jinja template directive or expression https://jinja.palletsprojects.com/en/3.1.x/templates/# - path: [“two_times_two”]

value: {{ 2 * 2 }}

fields

A list of transformations (path and corresponding value) that will be added to the record

Type

List[AddedFieldDefinition]

fields: List[airbyte_cdk.sources.declarative.transformations.add_fields.AddedFieldDefinition]
parameters: dataclasses.InitVar[Mapping[str, Any]]
transform(record: airbyte_cdk.sources.declarative.types.Record, config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → airbyte_cdk.sources.declarative.types.Record

Transform a record by adding, deleting, or mutating fields.

Parameters

  • record – The input record to be transformed

  • config – The user-provided configuration as specified by the source’s spec

  • stream_state – The stream state

  • stream_slice – The stream slice

Returns

The transformed record

class airbyte_cdk.sources.declarative.transformations.RecordTransformation

Bases: object

Implementations of this class define transformations that can be applied to records of a stream.

abstract transform(record: Mapping[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Transform a record by adding, deleting, or mutating fields.

Parameters

  • record – The input record to be transformed

  • config – The user-provided configuration as specified by the source’s spec

  • stream_state – The stream state

  • stream_slice – The stream slice

Returns

The transformed record

class airbyte_cdk.sources.declarative.transformations.RemoveFields(field_pointers: List[List[str]], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation

A transformation which removes fields from a record. The fields removed are designated using FieldPointers. During transformation, if a field or any of its parents does not exist in the record, no error is thrown.

If an input field pointer references an item in a list (e.g: [“k”, 0] in the object {“k”: [“a”, “b”, “c”]}) then the object at that index is set to None rather than being not entirely removed from the list. TODO change this behavior.

It’s possible to remove objects nested in lists e.g: removing [“.”, 0, “k”] from {“.”: [{“k”: “V”}]} results in {“.”: [{}]}

Usage syntax:

```yaml
my_stream:

<other parameters..> transformations:

  • type: RemoveFields field_pointers:

    • [“path”, “to”, “field1”]

    • [“path2”]

```

field_pointers

pointers to the fields that should be removed

Type

List[FieldPointer]

field_pointers: List[List[str]]
parameters: dataclasses.InitVar[Mapping[str, Any]]
transform(record: Mapping[str, Any], config: Optional[Mapping[str, Any]] = None, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]
Parameters

record – The record to be transformed

Returns

the input record with the requested fields removed