Submodules

airbyte_cdk.sources.declarative.extractors.dpath_extractor module

class airbyte_cdk.sources.declarative.extractors.dpath_extractor.DpathExtractor(field_path: List[Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]], config: Mapping[str, Any], parameters: dataclasses.InitVar[Mapping[str, Any]], decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder())

Bases: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor

Record extractor that searches a decoded response over a path defined as an array of fields.

If the field path points to an array, that array is returned. If the field path points to an object, that object is returned wrapped as an array. If the field path points to an empty object, an empty array is returned. If the field path points to a non-existing path, an empty array is returned.

Examples of instantiating this transform: ```

extractor:
type: DpathExtractor field_path:

“root”

“data”

```

```

extractor:

type: DpathExtractor field_path:

“root”

“{{ parameters[‘field’] }}”

```

```

extractor:: type: DpathExtractor field_path: []

```

field_path

Path to the field that should be extracted

Type: Union[InterpolatedString, str]

config

The user-provided configuration as specified by the source’s spec

Type: Config

decoder

The decoder responsible to transfom the response in a Mapping

Type: Decoder

config: Mapping[str, Any]

decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder()

extract_records(response: requests.models.Response) → List[Mapping[str, Any]]: Selects records from the response :param response: The response to extract the records from :return: List of Records extracted from the response

field_path: List[Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]]

parameters: dataclasses.InitVar[Mapping[str, Any]]

airbyte_cdk.sources.declarative.extractors.http_selector module

class airbyte_cdk.sources.declarative.extractors.http_selector.HttpSelector

Bases: object

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

abstract select_records(response: requests.models.Response, stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[airbyte_cdk.sources.declarative.types.Record]: Selects records from the response :param response: The response to select the records from :param stream_state: The stream state :param records_schema: json schema of records to return :param stream_slice: The stream slice :param next_page_token: The paginator token :return: List of Records selected from the response

airbyte_cdk.sources.declarative.extractors.record_extractor module

class airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor

Bases: object

Responsible for translating an HTTP response into a list of records by extracting records from the response.

abstract extract_records(response: requests.models.Response) → List[Mapping[str, Any]]: Selects records from the response :param response: The response to extract the records from :return: List of Records extracted from the response

airbyte_cdk.sources.declarative.extractors.record_filter module

class airbyte_cdk.sources.declarative.extractors.record_filter.RecordFilter(parameters: dataclasses.InitVar[Mapping[str, Any]], config: Mapping[str, Any], condition: str = '')

Bases: object

Filter applied on a list of Records

config (Config): The user-provided configuration as specified by the source’s spec condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False

condition: str = ''

config: Mapping[str, Any]

filter_records(records: List[Mapping[str, Any]], stream_state: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[Mapping[str, Any]]

parameters: dataclasses.InitVar[Mapping[str, Any]]

airbyte_cdk.sources.declarative.extractors.record_selector module

class airbyte_cdk.sources.declarative.extractors.record_selector.RecordSelector(extractor: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor, config: typing.Mapping[str, typing.Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], schema_normalization: airbyte_cdk.sources.utils.transform.TypeTransformer, record_filter: typing.Optional[airbyte_cdk.sources.declarative.extractors.record_filter.RecordFilter] = None, transformations: typing.List[airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation] = <factory>)

Bases: airbyte_cdk.sources.declarative.extractors.http_selector.HttpSelector

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

extractor

The record extractor responsible for extracting records from a response

Type: RecordExtractor

schema_normalization

The record normalizer responsible for casting record values to stream schema types

Type: TypeTransformer

record_filter

The record filter responsible for filtering extracted records

Type: RecordFilter

transformations

The transformations to be done on the records

Type: List[RecordTransformation]

config: Mapping[str, Any]

extractor: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor

parameters: dataclasses.InitVar[Mapping[str, Any]]

record_filter: Optional[airbyte_cdk.sources.declarative.extractors.record_filter.RecordFilter] = None

schema_normalization: airbyte_cdk.sources.utils.transform.TypeTransformer

select_records(response: requests.models.Response, stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[airbyte_cdk.sources.declarative.types.Record]: Selects records from the response :param response: The response to select the records from :param stream_state: The stream state :param records_schema: json schema of records to return :param stream_slice: The stream slice :param next_page_token: The paginator token :return: List of Records selected from the response

transformations: List[airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation]

Module contents

class airbyte_cdk.sources.declarative.extractors.DpathExtractor(field_path: List[Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]], config: Mapping[str, Any], parameters: dataclasses.InitVar[Mapping[str, Any]], decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder())

Bases: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor

Record extractor that searches a decoded response over a path defined as an array of fields.

If the field path points to an array, that array is returned. If the field path points to an object, that object is returned wrapped as an array. If the field path points to an empty object, an empty array is returned. If the field path points to a non-existing path, an empty array is returned.

Examples of instantiating this transform: ```

extractor:
type: DpathExtractor field_path:

“root”

“data”

```

```

extractor:

type: DpathExtractor field_path:

“root”

“{{ parameters[‘field’] }}”

```

```

extractor:: type: DpathExtractor field_path: []

```

field_path

Path to the field that should be extracted

Type: Union[InterpolatedString, str]

config

The user-provided configuration as specified by the source’s spec

Type: Config

decoder

The decoder responsible to transfom the response in a Mapping

Type: Decoder

config: Mapping[str, Any]

decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder()

extract_records(response: requests.models.Response) → List[Mapping[str, Any]]: Selects records from the response :param response: The response to extract the records from :return: List of Records extracted from the response

field_path: List[Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]]

parameters: dataclasses.InitVar[Mapping[str, Any]]

class airbyte_cdk.sources.declarative.extractors.HttpSelector

Bases: object

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

abstract select_records(response: requests.models.Response, stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[airbyte_cdk.sources.declarative.types.Record]: Selects records from the response :param response: The response to select the records from :param stream_state: The stream state :param records_schema: json schema of records to return :param stream_slice: The stream slice :param next_page_token: The paginator token :return: List of Records selected from the response

class airbyte_cdk.sources.declarative.extractors.RecordFilter(parameters: dataclasses.InitVar[Mapping[str, Any]], config: Mapping[str, Any], condition: str = '')

Bases: object

Filter applied on a list of Records

config (Config): The user-provided configuration as specified by the source’s spec condition (str): The string representing the predicate to filter a record. Records will be removed if evaluated to False

condition: str = ''

config: Mapping[str, Any]

filter_records(records: List[Mapping[str, Any]], stream_state: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[Mapping[str, Any]]

parameters: dataclasses.InitVar[Mapping[str, Any]]

class airbyte_cdk.sources.declarative.extractors.RecordSelector(extractor: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor, config: typing.Mapping[str, typing.Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], schema_normalization: airbyte_cdk.sources.utils.transform.TypeTransformer, record_filter: typing.Optional[airbyte_cdk.sources.declarative.extractors.record_filter.RecordFilter] = None, transformations: typing.List[airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation] = <factory>)

Bases: airbyte_cdk.sources.declarative.extractors.http_selector.HttpSelector

Responsible for translating an HTTP response into a list of records by extracting records from the response and optionally filtering records based on a heuristic.

extractor

The record extractor responsible for extracting records from a response

Type: RecordExtractor

schema_normalization

The record normalizer responsible for casting record values to stream schema types

Type: TypeTransformer

record_filter

The record filter responsible for filtering extracted records

Type: RecordFilter

transformations

The transformations to be done on the records

Type: List[RecordTransformation]

config: Mapping[str, Any]

extractor: airbyte_cdk.sources.declarative.extractors.record_extractor.RecordExtractor

parameters: dataclasses.InitVar[Mapping[str, Any]]

record_filter: Optional[airbyte_cdk.sources.declarative.extractors.record_filter.RecordFilter] = None

schema_normalization: airbyte_cdk.sources.utils.transform.TypeTransformer

select_records(response: requests.models.Response, stream_state: Mapping[str, Any], records_schema: Mapping[str, Any], stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → List[airbyte_cdk.sources.declarative.types.Record]: Selects records from the response :param response: The response to select the records from :param stream_state: The stream state :param records_schema: json schema of records to return :param stream_slice: The stream slice :param next_page_token: The paginator token :return: List of Records selected from the response

transformations: List[airbyte_cdk.sources.declarative.transformations.transformation.RecordTransformation]