Subpackages

Submodules

airbyte_cdk.sources.declarative.requesters.paginators.limit_paginator module

class airbyte_cdk.sources.declarative.requesters.paginators.default_paginator.DefaultPaginator(pagination_strategy: airbyte_cdk.sources.declarative.requesters.paginators.strategies.pagination_strategy.PaginationStrategy, config: Mapping[str, Any], url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], parameters: dataclasses.InitVar[Mapping[str, Any]], decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder(), page_size_option: Optional[airbyte_cdk.sources.declarative.requesters.request_option.RequestOption] = None, page_token_option: Optional[Union[airbyte_cdk.sources.declarative.requesters.request_path.RequestPath, airbyte_cdk.sources.declarative.requesters.request_option.RequestOption]] = None)

Bases: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token

Examples

1. * fetches up to 10 records at a time by setting the “limit” request param to 10 * updates the request path with “{{ response._metadata.next }}” ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: request_parameter field_name: limit

page_token_option:
type: RequestPath path: “location”

pagination_strategy:
type: “CursorPagination” cursor_value: “{{ response._metadata.next }}” page_size: 10

```

2. * fetches up to 5 records at a time by setting the “page_size” header to 5 * increments a record counter and set the request parameter “offset” to the value of the counter ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: header field_name: page_size

pagination_strategy:
type: “OffsetIncrement” page_size: 5

page_token_option:
option_type: “request_parameter” field_name: “offset”

```

3. * fetches up to 5 records at a time by setting the “page_size” request param to 5 * increments a page counter and set the request parameter “page” to the value of the counter ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: request_parameter field_name: page_size

pagination_strategy:
type: “PageIncrement” page_size: 5

page_token_option:
type: RequestOption option_type: “request_parameter” field_name: “page”

```

page_size_option

the request option to set the page size. Cannot be injected in the path.

Type: Optional[RequestOption]

page_token_option

the request option to set the page token

Type: Optional[RequestPath, RequestOption]

pagination_strategy

Strategy defining how to get the next page token

Type: PaginationStrategy

config

connection config

Type: Config

url_base

endpoint’s base url

Type: Union[InterpolatedString, str]

decoder

decoder to decode the response

Type: Decoder

config: Mapping[str, Any]

decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder()

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, str]: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Mapping[str, Any]]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

page_size_option: Optional[airbyte_cdk.sources.declarative.requesters.request_option.RequestOption] = None

page_token_option: Optional[Union[airbyte_cdk.sources.declarative.requesters.request_path.RequestPath, airbyte_cdk.sources.declarative.requesters.request_option.RequestOption]] = None

pagination_strategy: airbyte_cdk.sources.declarative.requesters.paginators.strategies.pagination_strategy.PaginationStrategy

parameters: dataclasses.InitVar[Mapping[str, Any]]

path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

reset() → None: Reset the pagination’s inner state

url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]

class airbyte_cdk.sources.declarative.requesters.paginators.default_paginator.PaginatorTestReadDecorator(decorated: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator, maximum_number_of_pages: int = 5)

Bases: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

In some cases, we want to limit the number of requests that are made to the backend source. This class allows for limiting the number of pages that are queried throughout a read command.

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Union[Mapping[str, Any], str]]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, str]: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Mapping[str, Any]]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

reset() → None: Reset the pagination’s inner state

airbyte_cdk.sources.declarative.requesters.paginators.no_pagination module

class airbyte_cdk.sources.declarative.requesters.paginators.no_pagination.NoPagination(parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

Pagination implementation that never returns a next page.

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Union[Mapping[str, Any], str]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, str]: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Mapping[str, Any]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

parameters: dataclasses.InitVar[Mapping[str, Any]]

path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

reset() → None: Reset the pagination’s inner state

airbyte_cdk.sources.declarative.requesters.paginators.paginator module

class airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

Bases: abc.ABC, airbyte_cdk.sources.declarative.requesters.request_options.request_options_provider.RequestOptionsProvider

Defines the token to use to fetch the next page of records from the API.

If needed, the Paginator will set request options to be set on the HTTP request to fetch the next page of records. If the next_page_token is the path to the next page of records, then it should be accessed through the path method

abstract next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Mapping[str, Any]]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

abstract path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

abstract reset() → None: Reset the pagination’s inner state

Module contents

class airbyte_cdk.sources.declarative.requesters.paginators.DefaultPaginator(pagination_strategy: airbyte_cdk.sources.declarative.requesters.paginators.strategies.pagination_strategy.PaginationStrategy, config: Mapping[str, Any], url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], parameters: dataclasses.InitVar[Mapping[str, Any]], decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder(), page_size_option: Optional[airbyte_cdk.sources.declarative.requesters.request_option.RequestOption] = None, page_token_option: Optional[Union[airbyte_cdk.sources.declarative.requesters.request_path.RequestPath, airbyte_cdk.sources.declarative.requesters.request_option.RequestOption]] = None)

Bases: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

Default paginator to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token

Examples

1. * fetches up to 10 records at a time by setting the “limit” request param to 10 * updates the request path with “{{ response._metadata.next }}” ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: request_parameter field_name: limit

page_token_option:
type: RequestPath path: “location”

pagination_strategy:
type: “CursorPagination” cursor_value: “{{ response._metadata.next }}” page_size: 10

```

2. * fetches up to 5 records at a time by setting the “page_size” header to 5 * increments a record counter and set the request parameter “offset” to the value of the counter ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: header field_name: page_size

pagination_strategy:
type: “OffsetIncrement” page_size: 5

page_token_option:
option_type: “request_parameter” field_name: “offset”

```

3. * fetches up to 5 records at a time by setting the “page_size” request param to 5 * increments a page counter and set the request parameter “page” to the value of the counter ```

paginator:
type: “DefaultPaginator” page_size_option:

type: RequestOption inject_into: request_parameter field_name: page_size

pagination_strategy:
type: “PageIncrement” page_size: 5

page_token_option:
type: RequestOption option_type: “request_parameter” field_name: “page”

```

page_size_option

the request option to set the page size. Cannot be injected in the path.

Type: Optional[RequestOption]

page_token_option

the request option to set the page token

Type: Optional[RequestPath, RequestOption]

pagination_strategy

Strategy defining how to get the next page token

Type: PaginationStrategy

config

connection config

Type: Config

url_base

endpoint’s base url

Type: Union[InterpolatedString, str]

decoder

decoder to decode the response

Type: Decoder

config: Mapping[str, Any]

decoder: airbyte_cdk.sources.declarative.decoders.decoder.Decoder = JsonDecoder()

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, str]: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Mapping[str, Any]]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

page_size_option: Optional[airbyte_cdk.sources.declarative.requesters.request_option.RequestOption] = None

page_token_option: Optional[Union[airbyte_cdk.sources.declarative.requesters.request_path.RequestPath, airbyte_cdk.sources.declarative.requesters.request_option.RequestOption]] = None

pagination_strategy: airbyte_cdk.sources.declarative.requesters.paginators.strategies.pagination_strategy.PaginationStrategy

parameters: dataclasses.InitVar[Mapping[str, Any]]

path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

reset() → None: Reset the pagination’s inner state

url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]

class airbyte_cdk.sources.declarative.requesters.paginators.NoPagination(parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator

Pagination implementation that never returns a next page.

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Union[Mapping[str, Any], str]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, str]: Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Mapping[str, Any]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

parameters: dataclasses.InitVar[Mapping[str, Any]]

path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

reset() → None: Reset the pagination’s inner state

class airbyte_cdk.sources.declarative.requesters.paginators.PaginationStrategy

Bases: object

Defines how to get the next page token

abstract get_page_size() → Optional[int]

Returns: page size: The number of records to fetch in a page. Returns None if unspecified

abstract property initial_token: Optional[Any]: Return the initial value of the token

abstract next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Any]

Parameters

response – response to process
last_records – records extracted from the response

Returns

next page token. Returns None if there are no more pages to fetch

abstract reset() → None: Reset the pagination’s inner state

class airbyte_cdk.sources.declarative.requesters.paginators.Paginator

Bases: abc.ABC, airbyte_cdk.sources.declarative.requesters.request_options.request_options_provider.RequestOptionsProvider

Defines the token to use to fetch the next page of records from the API.

If needed, the Paginator will set request options to be set on the HTTP request to fetch the next page of records. If the next_page_token is the path to the next page of records, then it should be accessed through the path method

abstract next_page_token(response: requests.models.Response, last_records: List[airbyte_cdk.sources.declarative.types.Record]) → Optional[Mapping[str, Any]]

Returns the next_page_token to use to fetch the next page of records.

Parameters

response – the response to process
last_records – the records extracted from the response

Returns

A mapping {“next_page_token”: <token>} for the next page from the input response object. Returning None means there are no more pages to read in this response.

abstract path() → Optional[str]

Returns the URL path to hit to fetch the next page of records

e.g: if you wanted to hit https://myapi.com/v1/some_entity then this will return “some_entity”

Returns: path to hit to fetch the next request. Returning None means the path is not defined by the next_page_token

abstract reset() → None: Reset the pagination’s inner state

class airbyte_cdk.sources.declarative.requesters.paginators.PaginatorTestReadDecorator(decorated: airbyte_cdk.sources.declarative.requesters.paginators.paginator.Paginator, maximum_number_of_pages: int = 5)