Subpackages

Submodules

airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy module

class airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy.BackoffStrategy

Bases: object

Backoff strategy defining how long to wait before retrying a request that resulted in an error.

abstract backoff(response: requests.models.Response, attempt_count: int) → Optional[float]: Return time to wait before retrying the request. :param response: response received for the request to retry :param attempt_count: number of attempts to submit the request :return: time to wait in seconds

airbyte_cdk.sources.declarative.requesters.error_handlers.composite_error_handler module

class airbyte_cdk.sources.declarative.requesters.error_handlers.composite_error_handler.CompositeErrorHandler(error_handlers: List[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler

Error handler that sequentially iterates over a list of `ErrorHandler`s

Sample config chaining 2 different retriers:

error_handler:

type: “CompositeErrorHandler” error_handlers:

response_filters:

predicate: “{{ ‘codase’ in response }}” action: RETRY

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 5

response_filters:

http_codes: [ 403 ] action: RETRY

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 10

error_handlers

list of error handlers

Type: List[ErrorHandler]

error_handlers: List[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler]

interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

property max_time: Optional[int]: Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.

parameters: dataclasses.InitVar[Mapping[str, Any]]

airbyte_cdk.sources.declarative.requesters.error_handlers.default_error_handler module

class airbyte_cdk.sources.declarative.requesters.error_handlers.default_error_handler.DefaultErrorHandler(parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], config: typing.Mapping[str, typing.Any], response_filters: typing.Optional[typing.List[airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter.HttpResponseFilter]] = None, max_retries: typing.Optional[int] = <property object>, max_time: int = 600, backoff_strategies: typing.Optional[typing.List[airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy.BackoffStrategy]] = None)

Bases: airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler

Default error handler.

By default, the handler will only retry server errors (HTTP 5XX) and too many requests (HTTP 429) with exponential backoff.

If the response is successful, then return SUCCESS Otherwise, iterate over the response_filters. If any of the filter match the response, then return the appropriate status. If the match is RETRY, then iterate sequentially over the backoff_strategies and return the first non-None backoff time.

Sample configs:

1. retry 10 times `

error_handler:
max_retries: 10

` 2. backoff for 5 seconds `

error_handler:

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 5

` 3. retry on HTTP 404 `

error_handler:

response_filters:

http_codes: [ 404 ] action: RETRY

` 4. ignore HTTP 404 `

error_handler:

response_filters:

http_codes: [ 404 ] action: IGNORE

` 5. retry if error message contains retrythisrequest! substring `

error_handler:

response_filters:

error_message_contain: “retrythisrequest!” action: IGNORE

` 6. retry if ‘code’ is a field present in the response body `

error_handler:

response_filters:

predicate: “{{ ‘code’ in response }}” action: IGNORE

`

7. ignore 429 and retry on 404 `

error_handler: - http_codes: [ 429 ]

action: IGNORE

http_codes: [ 404 ] action: RETRY

`

response_filters

response filters to iterate on

Type: Optional[List[HttpResponseFilter]]

max_retries

maximum retry attempts

Type: Optional[int]

backoff_strategies

list of backoff strategies to use to determine how long

Type: Optional[List[BackoffStrategy]]

to wait before retrying

DEFAULT_BACKOFF_STRATEGY: alias of airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.exponential_backoff_strategy.ExponentialBackoffStrategy

backoff_strategies: Optional[List[airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy.BackoffStrategy]] = None

config: Mapping[str, Any]

interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

max_time: int = 600

parameters: dataclasses.InitVar[Mapping[str, Any]]

response_filters: Optional[List[airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter.HttpResponseFilter]] = None

airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler module

class airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler

Bases: object

Defines whether a request was successful and how to handle a failure.

abstract interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

abstract property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

abstract property max_time: Optional[int]: Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.

airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter module

class airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter.HttpResponseFilter(action: Union[airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction, str], config: Mapping[str, Any], parameters: dataclasses.InitVar[Mapping[str, Any]], http_codes: Optional[Set[int]] = None, error_message_contains: Optional[str] = None, predicate: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_boolean.InterpolatedBoolean, str] = '', error_message: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str] = '')

Bases: object

Filter to select HttpResponses

action

action to execute if a request matches

Type: Union[ResponseAction, str]

http_codes

http code of matching requests

Type: Set[int]

error_message_contains

error substring of matching requests

Type: str

predicate

predicate to apply to determine if a request is matching

Type: str

error_message

error message to display if the response matches the filter

Type: Union[InterpolatedString, str

DEFAULT_RETRIABLE_ERRORS = {429, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599}

TOO_MANY_REQUESTS_ERRORS = {429}

action: Union[airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction, str]

config: Mapping[str, Any]

error_message: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str] = ''

error_message_contains: str = None

http_codes: Set[int] = None

matches(response: requests.models.Response, backoff_time: Optional[float] = None) → Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus]

parameters: dataclasses.InitVar[Mapping[str, Any]]

predicate: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_boolean.InterpolatedBoolean, str] = ''

airbyte_cdk.sources.declarative.requesters.error_handlers.response_action module

class airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction(value)

Bases: enum.Enum

Response statuses for non retriable responses

FAIL = 'FAIL'

IGNORE = 'IGNORE'

RETRY = 'RETRY'

SUCCESS = 'SUCCESS'

airbyte_cdk.sources.declarative.requesters.error_handlers.response_status module

airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.FAIL: Final[airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus] = <airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus object>: Response is unsuccessful, but can be ignored. No need to retry

class airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus(response_action: Union[airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction, str], retry_in: Optional[float] = None, error_message: str = '')

Bases: object

ResponseAction amended with backoff time if an action is RETRY

property action: The ResponseAction to execute when a response matches the filter

property error_message: str: The message to be displayed when an error response is received

classmethod retry(retry_in: Optional[float]) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Returns a ResponseStatus defining how long to backoff before retrying

Parameters: retry_in – how long to backoff before retrying. None if no wait required
Returns: A response status defining how long to backoff before retrying

property retry_in: Optional[float]: How long to backoff before retrying a response. None if no wait required.

airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.SUCCESS: Final[airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus] = <airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus object>: Response is unsuccessful. The failure needs to be handled

Module contents

class airbyte_cdk.sources.declarative.requesters.error_handlers.BackoffStrategy

Bases: object

Backoff strategy defining how long to wait before retrying a request that resulted in an error.

abstract backoff(response: requests.models.Response, attempt_count: int) → Optional[float]: Return time to wait before retrying the request. :param response: response received for the request to retry :param attempt_count: number of attempts to submit the request :return: time to wait in seconds

class airbyte_cdk.sources.declarative.requesters.error_handlers.CompositeErrorHandler(error_handlers: List[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler], parameters: dataclasses.InitVar[Mapping[str, Any]])

Bases: airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler

Error handler that sequentially iterates over a list of `ErrorHandler`s

Sample config chaining 2 different retriers:

error_handler:

type: “CompositeErrorHandler” error_handlers:

response_filters:

predicate: “{{ ‘codase’ in response }}” action: RETRY

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 5

response_filters:

http_codes: [ 403 ] action: RETRY

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 10

error_handlers

list of error handlers

Type: List[ErrorHandler]

error_handlers: List[airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler]

interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

property max_time: Optional[int]: Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.

parameters: dataclasses.InitVar[Mapping[str, Any]]

class airbyte_cdk.sources.declarative.requesters.error_handlers.DefaultErrorHandler(parameters: dataclasses.InitVar[typing.Mapping[str, typing.Any]], config: typing.Mapping[str, typing.Any], response_filters: typing.Optional[typing.List[airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter.HttpResponseFilter]] = None, max_retries: typing.Optional[int] = <property object>, max_time: int = 600, backoff_strategies: typing.Optional[typing.List[airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy.BackoffStrategy]] = None)

Bases: airbyte_cdk.sources.declarative.requesters.error_handlers.error_handler.ErrorHandler

Default error handler.

By default, the handler will only retry server errors (HTTP 5XX) and too many requests (HTTP 429) with exponential backoff.

If the response is successful, then return SUCCESS Otherwise, iterate over the response_filters. If any of the filter match the response, then return the appropriate status. If the match is RETRY, then iterate sequentially over the backoff_strategies and return the first non-None backoff time.

Sample configs:

1. retry 10 times `

error_handler:
max_retries: 10

` 2. backoff for 5 seconds `

error_handler:

backoff_strategies:

type: “ConstantBackoff” backoff_time_in_seconds: 5

` 3. retry on HTTP 404 `

error_handler:

response_filters:

http_codes: [ 404 ] action: RETRY

` 4. ignore HTTP 404 `

error_handler:

response_filters:

http_codes: [ 404 ] action: IGNORE

` 5. retry if error message contains retrythisrequest! substring `

error_handler:

response_filters:

error_message_contain: “retrythisrequest!” action: IGNORE

` 6. retry if ‘code’ is a field present in the response body `

error_handler:

response_filters:

predicate: “{{ ‘code’ in response }}” action: IGNORE

`

7. ignore 429 and retry on 404 `

error_handler: - http_codes: [ 429 ]

action: IGNORE

http_codes: [ 404 ] action: RETRY

`

response_filters

response filters to iterate on

Type: Optional[List[HttpResponseFilter]]

max_retries

maximum retry attempts

Type: Optional[int]

backoff_strategies

list of backoff strategies to use to determine how long

Type: Optional[List[BackoffStrategy]]

to wait before retrying

DEFAULT_BACKOFF_STRATEGY: alias of airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.exponential_backoff_strategy.ExponentialBackoffStrategy

backoff_strategies: Optional[List[airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategy.BackoffStrategy]] = None

config: Mapping[str, Any]

interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

max_time: int = 600

parameters: dataclasses.InitVar[Mapping[str, Any]]

response_filters: Optional[List[airbyte_cdk.sources.declarative.requesters.error_handlers.http_response_filter.HttpResponseFilter]] = None

class airbyte_cdk.sources.declarative.requesters.error_handlers.ErrorHandler

Bases: object

Defines whether a request was successful and how to handle a failure.

abstract interpret_response(response: requests.models.Response) → airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus

Evaluate response status describing whether a failing request should be retried or ignored.

Parameters: response – response to evaluate
Returns: response status

abstract property max_retries: Optional[int]: Specifies maximum amount of retries for backoff policy. Return None for no limit.

abstract property max_time: Optional[int]: Specifies maximum total waiting time (in seconds) for backoff policy. Return None for no limit.

class airbyte_cdk.sources.declarative.requesters.error_handlers.HttpResponseFilter(action: Union[airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction, str], config: Mapping[str, Any], parameters: dataclasses.InitVar[Mapping[str, Any]], http_codes: Optional[Set[int]] = None, error_message_contains: Optional[str] = None, predicate: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_boolean.InterpolatedBoolean, str] = '', error_message: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str] = '')

Bases: object

Filter to select HttpResponses

action

action to execute if a request matches

Type: Union[ResponseAction, str]

http_codes

http code of matching requests

Type: Set[int]

error_message_contains

error substring of matching requests

Type: str

predicate

predicate to apply to determine if a request is matching

Type: str

error_message

error message to display if the response matches the filter

Type: Union[InterpolatedString, str

DEFAULT_RETRIABLE_ERRORS = {429, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599}

TOO_MANY_REQUESTS_ERRORS = {429}

action: Union[airbyte_cdk.sources.declarative.requesters.error_handlers.response_action.ResponseAction, str]

config: Mapping[str, Any]

error_message: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_string.InterpolatedString, str] = ''

error_message_contains: str = None

http_codes: Set[int] = None

matches(response: requests.models.Response, backoff_time: Optional[float] = None) → Optional[airbyte_cdk.sources.declarative.requesters.error_handlers.response_status.ResponseStatus]

parameters: dataclasses.InitVar[Mapping[str, Any]]

predicate: Union[airbyte_cdk.sources.declarative.interpolation.interpolated_boolean.InterpolatedBoolean, str] = ''