singer_sdk.sql.SQLStream

class singer_sdk.sql.SQLStream

Bases: Stream, ABC

Base class for SQLAlchemy-based streams.

connector_class

alias of SQLConnector

classmethod __new__(*args, **kwargs)

__init__(tap, catalog_entry, connector=None)

Initialize the database stream.

If connector is omitted, a new connector will be created.

Parameters:

• tap (Tap) – The parent tap object.

• catalog_entry (dict) – Catalog entry dict.

• connector (SQLConnector | None) – Optional connector to reuse.

Return type:

None

apply_query_filters(query, table, *, context=None)

Apply WHERE and ORDER BY clauses to the query.

By default, this method applies a replication filter to the query and orders the results by the replication key, if a replication key is set.

Parameters:

• query (selectable.Select) – The SQLAlchemy Select object.

• table (sa.Table) – The SQLAlchemy Table object.

• context (Context | None) – The context object.

Returns:

A SQLAlchemy Select object.

Return type:

selectable.Select

apply_query_limit(query)

Apply LIMIT clause to the query.

By default, this method applies a limit filter to the query if the stream has an ABORT_AT_RECORD_COUNT value set.

The ABORT_AT_RECORD_COUNT limit is incremented by 1 to ensure that the MaxRecordsLimitException exception is properly raised by caller Stream._sync_records() if more records are available than can be processed.

Parameters:

query (selectable.Select) – The SQLAlchemy Select object.

Returns:

A SQLAlchemy Select object.

Return type:

selectable.Select

build_query(*, context=None)

Build a SQLAlchemy Select object for the stream.

• Apply WHERE and ORDER BY clauses to the query.

• Apply a LIMIT clause to the query.

Parameters:

context (Context | None) – The context object.

Returns:

A SQLAlchemy Select object.

Return type:

selectable.Select

get_records(context)

Return a generator of record-type dictionary objects.

If the stream has a replication_key value defined, records will be sorted by the incremental key. If the stream also has an available starting bookmark, the records will be filtered for values greater than or equal to the bookmark value.

Parameters:

context (Context | None) – If partition context is provided, will read specifically from this data slice.

Yields:

One dict per record.

Raises:

NotImplementedError – If partition is passed in context and the stream does not support partitioning.

Return type:

t.Iterable[Record]

get_selected_schema()

Return a copy of the Stream JSON schema, dropping any fields not selected.

Returns:

A dictionary containing a copy of the Stream JSON schema, filtered to any selection criteria.

Return type:

dict

property connector: SQLConnector

Connector object.

property effective_schema: dict

Effective schema for the stream.

property fully_qualified_name: FullyQualifiedName

Generate the fully qualified version of the table name.

Returns:

The fully qualified name.

Raises:

ValueError – If table_name is not able to be detected.

property is_sorted: bool

Expect stream to be sorted.

When True, incremental streams will attempt to resume if unexpectedly interrupted.

Returns:

True if stream is sorted. Defaults to False.

property metadata: MetadataMapping

Singer metadata.

Metadata from an input catalog will override standard metadata.

property primary_keys: Sequence[str]

Primary keys from the catalog entry definition.

property schema: dict

JSON Schema dictionary.

Metadata from an input catalog will override standard metadata.

supports_nulls_first: bool = False

Whether the database supports the NULLS FIRST/LAST syntax.

property tap_stream_id: str

Unique ID used by the tap to identify this stream.

Generally, this is the same value as in Stream.name.

In rare cases, such as for database types with multi-part names, this may be slightly different from Stream.name.