Subpackages

Submodules

airbyte_cdk.sources.declarative.requesters.http_requester module

class airbyte_cdk.sources.declarative.requesters.http_requester.HttpRequester(name: str, url_base: typing.Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], path: typing.Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], config: typing.Mapping[str, typing.Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], authenticator: typing.Optional[airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator] = None, http_method: typing.Union[str, airbyte_cdk.sources.declarative.requesters.requester.HttpMethod] = HttpMethod.GET, request_options_provider: typing.Optional[airbyte_cdk.sources.declarative.requesters.request_options.interpolated_request_options_provider.InterpolatedRequestOptionsProvider] = None, error_handler: typing.Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler] = None, disable_retries: bool = False, message_repository: airbyte_cdk.sources.message.repository.MessageRepository = <airbyte_cdk.sources.message.repository.NoopMessageRepository object>)

Bases: airbyte_cdk.sources.declarative.requesters.requester.Requester

Default implementation of a Requester

name

Name of the stream. Only used for request/response caching

Type

str

url_base

Base url to send requests to

Type

Union[InterpolatedString, str]

path

Path to send requests to

Type

Union[InterpolatedString, str]

http_method

HTTP method to use when sending requests

Type

Union[str, HttpMethod]

request_options_provider

request option provider defining the options to set on outgoing requests

Type

Optional[InterpolatedRequestOptionsProvider]

authenticator

Authenticator defining how to authenticate to the source

Type

DeclarativeAuthenticator

error_handler

Error handler defining how to detect and handle errors

Type

Optional[ErrorHandler]

config

The user-provided configuration as specified by the source’s spec

Type

Config

authenticator: Optional[airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator] = None
config: Mapping[str, Any]
deduplicate_query_params(url: str, params: Optional[Mapping[str, Any]]) → Mapping[str, Any]

Remove query parameters from params mapping if they are already encoded in the URL. :param url: URL with :param params: :return:

disable_retries: bool = False
error_handler: Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler] = None
get_authenticator() → airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator

Specifies the authenticator to use when submitting requests

get_method() → airbyte_cdk.sources.declarative.requesters.requester.HttpMethod

Specifies the HTTP method to use

get_path(*, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]], next_page_token: Optional[Mapping[str, Any]]) → str

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “some_entity”

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Union[Mapping[str, Any], str]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

get_url_base() → str
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “https://myapi.com/v1/”

http_method: Union[str, airbyte_cdk.sources.declarative.requesters.requester.HttpMethod] = 'GET'
interpret_response_status(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Specifies conditions for backoff, error handling and reporting based on the response from the server.

By default, back off on the following HTTP response statuses:

  • 429 (Too Many Requests) indicating rate limiting

  • 500s to handle transient server errors

Unexpected but transient exceptions (connection timeout, DNS resolution failed, etc..) are retried by default.

property logger: logging.Logger
property max_retries: Optional[int]
message_repository: airbyte_cdk.sources.message.repository.MessageRepository = <airbyte_cdk.sources.message.repository.NoopMessageRepository object>
name: str
parameters: dataclasses.InitVar[Mapping[str, Any]]
classmethod parse_response_error_message(response: requests.models.Response) → Optional[str]

Parses the raw response object from a failed request into a user-friendly error message. By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.

Parameters

response –

Returns

A user-friendly message that indicates the cause of the error

path: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]
request_options_provider: Optional[airbyte_cdk.sources.declarative.requesters.request_options.interpolated_request_options_provider.InterpolatedRequestOptionsProvider] = None
send_request(stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Optional[Union[Mapping[str, Any], str]] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) → Optional[requests.models.Response]

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it’s used to log the performed request and response. If it’s not provided, no logging is performed.

url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]

airbyte_cdk.sources.declarative.requesters.request_option module

class airbyte_cdk.sources.declarative.requesters.request_option.RequestOption(field_name: str, inject_into: airbyte_cdk.sources.declarative.requesters.request_option.RequestOptionType, parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: object

Describes an option to set on a request

field_name

Describes the name of the parameter to inject

Type

str

inject_into

Describes where in the HTTP request to inject the parameter

Type

RequestOptionType

field_name: str
inject_into: airbyte_cdk.sources.declarative.requesters.request_option.RequestOptionType
parameters: dataclasses.InitVar[Mapping[str, Any]]
class airbyte_cdk.sources.declarative.requesters.request_option.RequestOptionType(value)

Bases: enum.Enum

Describes where to set a value on a request

body_data = 'body_data'
body_json = 'body_json'
header = 'header'
request_parameter = 'request_parameter'

airbyte_cdk.sources.declarative.requesters.requester module

class airbyte_cdk.sources.declarative.requesters.requester.HttpMethod(value)

Bases: enum.Enum

Http Method to use when submitting an outgoing HTTP request

GET = 'GET'
POST = 'POST'
class airbyte_cdk.sources.declarative.requesters.requester.Requester

Bases: airbyte_cdk.sources.declarative.requesters.request_options.request_options_provider.RequestOptionsProvider

abstract get_authenticator() → airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator

Specifies the authenticator to use when submitting requests

abstract get_method() → airbyte_cdk.sources.declarative.requesters.requester.HttpMethod

Specifies the HTTP method to use

abstract get_path(*, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]], next_page_token: Optional[Mapping[str, Any]]) → str

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “some_entity”

abstract get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

abstract get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

abstract get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

abstract get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

abstract get_url_base() → str
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “https://myapi.com/v1/”

abstract interpret_response_status(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Specifies conditions for backoff, error handling and reporting based on the response from the server.

By default, back off on the following HTTP response statuses:

  • 429 (Too Many Requests) indicating rate limiting

  • 500s to handle transient server errors

Unexpected but transient exceptions (connection timeout, DNS resolution failed, etc..) are retried by default.

abstract send_request(stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Optional[Union[Mapping[str, Any], str]] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) → Optional[requests.models.Response]

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it’s used to log the performed request and response. If it’s not provided, no logging is performed.

Module contents

class airbyte_cdk.sources.declarative.requesters.HttpRequester(name: str, url_base: typing.Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], path: typing.Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str], config: typing.Mapping[str, typing.Any], parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], authenticator: typing.Optional[airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator] = None, http_method: typing.Union[str, airbyte_cdk.sources.declarative.requesters.requester.HttpMethod] = HttpMethod.GET, request_options_provider: typing.Optional[airbyte_cdk.sources.declarative.requesters.request_options.interpolated_request_options_provider.InterpolatedRequestOptionsProvider] = None, error_handler: typing.Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler] = None, disable_retries: bool = False, message_repository: airbyte_cdk.sources.message.repository.MessageRepository = <airbyte_cdk.sources.message.repository.NoopMessageRepository object>)

Bases: airbyte_cdk.sources.declarative.requesters.requester.Requester

Default implementation of a Requester

name

Name of the stream. Only used for request/response caching

Type

str

url_base

Base url to send requests to

Type

Union[InterpolatedString, str]

path

Path to send requests to

Type

Union[InterpolatedString, str]

http_method

HTTP method to use when sending requests

Type

Union[str, HttpMethod]

request_options_provider

request option provider defining the options to set on outgoing requests

Type

Optional[InterpolatedRequestOptionsProvider]

authenticator

Authenticator defining how to authenticate to the source

Type

DeclarativeAuthenticator

error_handler

Error handler defining how to detect and handle errors

Type

Optional[ErrorHandler]

config

The user-provided configuration as specified by the source’s spec

Type

Config

authenticator: Optional[airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator] = None
config: Mapping[str, Any]
deduplicate_query_params(url: str, params: Optional[Mapping[str, Any]]) → Mapping[str, Any]

Remove query parameters from params mapping if they are already encoded in the URL. :param url: URL with :param params: :return:

disable_retries: bool = False
error_handler: Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler] = None
get_authenticator() → airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator

Specifies the authenticator to use when submitting requests

get_method() → airbyte_cdk.sources.declarative.requesters.requester.HttpMethod

Specifies the HTTP method to use

get_path(*, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]], next_page_token: Optional[Mapping[str, Any]]) → str

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “some_entity”

get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Union[Mapping[str, Any], str]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

get_url_base() → str
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “https://myapi.com/v1/”

http_method: Union[str, airbyte_cdk.sources.declarative.requesters.requester.HttpMethod] = 'GET'
interpret_response_status(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Specifies conditions for backoff, error handling and reporting based on the response from the server.

By default, back off on the following HTTP response statuses:

  • 429 (Too Many Requests) indicating rate limiting

  • 500s to handle transient server errors

Unexpected but transient exceptions (connection timeout, DNS resolution failed, etc..) are retried by default.

property logger: logging.Logger
property max_retries: Optional[int]
message_repository: airbyte_cdk.sources.message.repository.MessageRepository = <airbyte_cdk.sources.message.repository.NoopMessageRepository object>
name: str
parameters: dataclasses.InitVar[Mapping[str, Any]]
classmethod parse_response_error_message(response: requests.models.Response) → Optional[str]

Parses the raw response object from a failed request into a user-friendly error message. By default, this method tries to grab the error message from JSON responses by following common API patterns. Override to parse differently.

Parameters

response –

Returns

A user-friendly message that indicates the cause of the error

path: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]
request_options_provider: Optional[airbyte_cdk.sources.declarative.requesters.request_options.interpolated_request_options_provider.InterpolatedRequestOptionsProvider] = None
send_request(stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Optional[Union[Mapping[str, Any], str]] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) → Optional[requests.models.Response]

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it’s used to log the performed request and response. If it’s not provided, no logging is performed.

url_base: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str]
class airbyte_cdk.sources.declarative.requesters.RequestOption(field_name: str, inject_into: airbyte_cdk.sources.declarative.requesters.request_option.RequestOptionType, parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: object

Describes an option to set on a request

field_name

Describes the name of the parameter to inject

Type

str

inject_into

Describes where in the HTTP request to inject the parameter

Type

RequestOptionType

field_name: str
inject_into: airbyte_cdk.sources.declarative.requesters.request_option.RequestOptionType
parameters: dataclasses.InitVar[Mapping[str, Any]]
class airbyte_cdk.sources.declarative.requesters.Requester

Bases: airbyte_cdk.sources.declarative.requesters.request_options.request_options_provider.RequestOptionsProvider

abstract get_authenticator() → airbyte_cdk.sources.declarative.auth.declarative_authenticator.DeclarativeAuthenticator

Specifies the authenticator to use when submitting requests

abstract get_method() → airbyte_cdk.sources.declarative.requesters.requester.HttpMethod

Specifies the HTTP method to use

abstract get_path(*, stream_state: Optional[Mapping[str, Any]], stream_slice: Optional[Mapping[str, Any]], next_page_token: Optional[Mapping[str, Any]]) → str

Returns the URL path for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “some_entity”

abstract get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a non-JSON payload.

If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

abstract get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Optional[Mapping[str, Any]]

Specifies how to populate the body of the request with a JSON payload.

At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.

abstract get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → Mapping[str, Any]

Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.

abstract get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) → MutableMapping[str, Any]

Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.

E.g: you might want to define query parameters for paging if next_page_token is not None.

abstract get_url_base() → str
Returns

URL base for the API endpoint e.g: if you wanted to hit https://myapi.com/v1/some_entity then this should return “https://myapi.com/v1/”

abstract interpret_response_status(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Specifies conditions for backoff, error handling and reporting based on the response from the server.

By default, back off on the following HTTP response statuses:

  • 429 (Too Many Requests) indicating rate limiting

  • 500s to handle transient server errors

Unexpected but transient exceptions (connection timeout, DNS resolution failed, etc..) are retried by default.

abstract send_request(stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None, path: Optional[str] = None, request_headers: Optional[Mapping[str, Any]] = None, request_params: Optional[Mapping[str, Any]] = None, request_body_data: Optional[Union[Mapping[str, Any], str]] = None, request_body_json: Optional[Mapping[str, Any]] = None, log_formatter: Optional[Callable[[requests.models.Response], Any]] = None) → Optional[requests.models.Response]

Sends a request and returns the response. Might return no response if the error handler chooses to ignore the response or throw an exception in case of an error. If path is set, the path configured on the requester itself is ignored. If header, params and body are set, they are merged with the ones configured on the requester itself.

If a log formatter is provided, it’s used to log the performed request and response. If it’s not provided, no logging is performed.