Subpackages

Submodules

airbyte_cdk.sources.abstract_source module

class airbyte_cdk.sources.abstract_source.AbstractSource

Bases: airbyte_cdk.sources.source.Source, abc.ABC

Abstract base class for an Airbyte Source. Consumers should implement any abstract methods in this class to create an Airbyte Specification compliant Source.

check(logger: logging.Logger, config: Mapping[str, Any]) → airbyte_protocol.models.airbyte_protocol.AirbyteConnectionStatus: Implements the Check Connection operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#check.

abstract check_connection(logger: logging.Logger, config: Mapping[str, Any]) → Tuple[bool, Optional[Any]]

Parameters

logger – source logger
config – The user-provided configuration as specified by the source’s spec. This usually contains information required to check connection e.g. tokens, secrets and keys etc.

Returns

A tuple of (boolean, error). If boolean is true, then the connection check is successful and we can connect to the underlying data source using the provided configuration. Otherwise, the input config cannot be used to connect to the underlying data source, and the “error” object should describe what went wrong. The error object will be cast to string to display the problem to the user.

property continue_sync_on_stream_failure: bool

This function is in-development which means it is subject to change. Use at your own risk.

By default, a source should raise an exception and stop the sync when it encounters an error while syncing a stream. This method can be overridden on a per-source basis so that a source will continue syncing streams other streams even if an exception is raised for a stream.

Type: WARNING

discover(logger: logging.Logger, config: Mapping[str, Any]) → airbyte_protocol.models.airbyte_protocol.AirbyteCatalog: Implements the Discover operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#discover.

property message_repository: Union[None, airbyte_cdk.sources.message.repository.MessageRepository]

property name: str: Source name

property per_stream_state_enabled: bool

property raise_exception_on_missing_stream: bool

read(logger: logging.Logger, config: Mapping[str, Any], catalog: airbyte_protocol.models.airbyte_protocol.ConfiguredAirbyteCatalog, state: Optional[Union[List[airbyte_protocol.models.airbyte_protocol.AirbyteStateMessage], MutableMapping[str, Any]]] = None) → Iterator[airbyte_protocol.models.airbyte_protocol.AirbyteMessage]: Implements the Read operation from the Airbyte Specification. See https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/.

abstract streams(config: Mapping[str, Any]) → List[airbyte_cdk.sources.streams.core.Stream]

Parameters: config – The user-provided configuration as specified by the source’s spec.

Any stream construction related operation should happen here. :return: A list of the streams in this source connector.

airbyte_cdk.sources.config module

class airbyte_cdk.sources.config.BaseConfig

Bases: pydantic.main.BaseModel

Base class for connector spec, adds the following behaviour:

resolve $ref and replace it with definition
replace all occurrences of anyOf with oneOf
drop description

classmethod schema(*args, **kwargs) → Dict[str, Any]: We’re overriding the schema classmethod to enable some post-processing

airbyte_cdk.sources.source module

class airbyte_cdk.sources.source.BaseSource

Bases: airbyte_cdk.connector.BaseConnector[airbyte_cdk.connector.TConfig], abc.ABC, Generic[airbyte_cdk.connector.TConfig, airbyte_cdk.sources.source.TState, airbyte_cdk.sources.source.TCatalog]

abstract discover(logger: logging.Logger, config: airbyte_cdk.connector.TConfig) → airbyte_protocol.models.airbyte_protocol.AirbyteCatalog: Returns an AirbyteCatalog representing the available streams and fields in this integration. For example, given valid credentials to a Postgres database, returns an Airbyte catalog where each postgres table is a stream, and each table column is a field.

abstract read(logger: logging.Logger, config: airbyte_cdk.connector.TConfig, catalog: airbyte_cdk.sources.source.TCatalog, state: Optional[airbyte_cdk.sources.source.TState] = None) → Iterable[airbyte_protocol.models.airbyte_protocol.AirbyteMessage]: Returns a generator of the AirbyteMessages generated by reading the source with the given configuration, catalog, and state.

abstract read_catalog(catalog_path: str) → airbyte_cdk.sources.source.TCatalog

abstract read_state(state_path: str) → airbyte_cdk.sources.source.TState

class airbyte_cdk.sources.source.Source

Bases: airbyte_cdk.connector.DefaultConnectorMixin, airbyte_cdk.sources.source.BaseSource[Mapping[str, Any], Union[List[airbyte_protocol.models.airbyte_protocol.AirbyteStateMessage], MutableMapping[str, Any]], airbyte_protocol.models.airbyte_protocol.ConfiguredAirbyteCatalog], abc.ABC

classmethod read_catalog(catalog_path: str) → airbyte_protocol.models.airbyte_protocol.ConfiguredAirbyteCatalog

classmethod read_state(state_path: str) → Union[List[airbyte_protocol.models.airbyte_protocol.AirbyteStateMessage], MutableMapping[str, Any]]: Retrieves the input state of a sync by reading from the specified JSON file. Incoming state can be deserialized into either a JSON object for legacy state input or as a list of AirbyteStateMessages for the per-stream state format. Regardless of the incoming input type, it will always be transformed and output as a list of AirbyteStateMessage(s). :param state_path: The filepath to where the stream states are located :return: The complete stream state based on the connector’s previous sync

Module contents

class airbyte_cdk.sources.AbstractSource