Submodules
airbyte_cdk.sources.declarative.stream_slicers.cartesian_product_stream_slicer module
- class airbyte_cdk.sources.declarative.stream_slicers.cartesian_product_stream_slicer.CartesianProductStreamSlicer(stream_slicers: List[airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer], parameters: dataclasses.InitVar[Mapping[str, Any]])
Bases:
airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer
Stream slicers that iterates over the cartesian product of input stream slicers Given 2 stream slicers with the following slices: A: [{“i”: 0}, {“i”: 1}, {“i”: 2}] B: [{“s”: “hello”}, {“s”: “world”}] the resulting stream slices are [
{“i”: 0, “s”: “hello”}, {“i”: 0, “s”: “world”}, {“i”: 1, “s”: “hello”}, {“i”: 1, “s”: “world”}, {“i”: 2, “s”: “hello”}, {“i”: 2, “s”: “world”},
]
- stream_slicers
Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
- Type
List[StreamSlicer]
- get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”
At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.
- get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Optional[Mapping]
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.
- get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
- get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
- parameters: dataclasses.InitVar[Mapping[str, Any]]
- stream_slicers: List[airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer]
- stream_slices() Iterable[Mapping[str, Any]]
Defines stream slices
- Returns
List of stream slices
airbyte_cdk.sources.declarative.stream_slicers.datetime_stream_slicer module
airbyte_cdk.sources.declarative.stream_slicers.list_stream_slicer module
airbyte_cdk.sources.declarative.stream_slicers.single_slice module
airbyte_cdk.sources.declarative.stream_slicers.stream_slicer module
- class airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer
-
Slices the stream into a subset of records. Slices enable state checkpointing and data retrieval parallelization.
The stream slicer keeps track of the cursor state as a dict of cursor_field -> cursor_value
See the stream slicing section of the docs for more information.
- abstract stream_slices() Iterable[Mapping[str, Any]]
Defines stream slices
- Returns
List of stream slices
airbyte_cdk.sources.declarative.stream_slicers.substream_slicer module
Module contents
- class airbyte_cdk.sources.declarative.stream_slicers.CartesianProductStreamSlicer(stream_slicers: List[airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer], parameters: dataclasses.InitVar[Mapping[str, Any]])
Bases:
airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer
Stream slicers that iterates over the cartesian product of input stream slicers Given 2 stream slicers with the following slices: A: [{“i”: 0}, {“i”: 1}, {“i”: 2}] B: [{“s”: “hello”}, {“s”: “world”}] the resulting stream slices are [
{“i”: 0, “s”: “hello”}, {“i”: 0, “s”: “world”}, {“i”: 1, “s”: “hello”}, {“i”: 1, “s”: “world”}, {“i”: 2, “s”: “hello”}, {“i”: 2, “s”: “world”},
]
- stream_slicers
Underlying stream slicers. The RequestOptions (e.g: Request headers, parameters, etc..) returned by this slicer are the combination of the RequestOptions of its input slicers. If there are conflicts e.g: two slicers define the same header or request param, the conflict is resolved by taking the value from the first slicer, where ordering is determined by the order in which slicers were input to this composite slicer.
- Type
List[StreamSlicer]
- get_request_body_data(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Specifies how to populate the body of the request with a non-JSON payload.
If returns a ready text that it will be sent as is. If returns a dict that it will be converted to a urlencoded form. E.g. {“key1”: “value1”, “key2”: “value2”} => “key1=value1&key2=value2”
At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.
- get_request_body_json(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Optional[Mapping]
Specifies how to populate the body of the request with a JSON payload.
At the same time only one of the ‘request_body_data’ and ‘request_body_json’ functions can be overridden.
- get_request_headers(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Return any non-auth headers. Authentication headers will overwrite any overlapping headers returned from this method.
- get_request_params(*, stream_state: Optional[Mapping[str, Any]] = None, stream_slice: Optional[Mapping[str, Any]] = None, next_page_token: Optional[Mapping[str, Any]] = None) Mapping[str, Any]
Specifies the query parameters that should be set on an outgoing HTTP request given the inputs.
E.g: you might want to define query parameters for paging if next_page_token is not None.
- parameters: dataclasses.InitVar[Mapping[str, Any]]
- stream_slicers: List[airbyte_cdk.sources.declarative.stream_slicers.stream_slicer.StreamSlicer]
- stream_slices() Iterable[Mapping[str, Any]]
Defines stream slices
- Returns
List of stream slices
- class airbyte_cdk.sources.declarative.stream_slicers.StreamSlicer
-
Slices the stream into a subset of records. Slices enable state checkpointing and data retrieval parallelization.
The stream slicer keeps track of the cursor state as a dict of cursor_field -> cursor_value
See the stream slicing section of the docs for more information.
- abstract stream_slices() Iterable[Mapping[str, Any]]
Defines stream slices
- Returns
List of stream slices