singer_sdk.BatchSink

class singer_sdk.BatchSink(target: singer_sdk.plugin_base.PluginBase, stream_name: str, schema: Dict, key_properties: Optional[List[str]])

Base class for batched record writers.

abstract process_batch(context: dict)None

Process a batch with the given batch context.

This method must be overridden.

If process_record() is not overridden, the context[“records”] list will contain all records from the given batch context.

If duplicates are merged, these can be tracked via tally_duplicate_merged().

Parameters

context – Stream partition or context dictionary.

process_record(record: dict, context: dict)None

Load the latest record from the stream.

Developers may either load to the context dict for staging (the default behavior for Batch types), or permanently write out to the target.

If this method is not overridden, the default implementation will create a context[“records”] list and append all records for processing during process_batch().

If duplicates are merged, these can be tracked via tally_duplicate_merged().

Parameters
  • record – Individual record in the stream.

  • context – Stream partition or context dictionary.

start_batch(context: dict)None

Start a new batch with the given context.

The SDK-generated context will contain batch_id (GUID string) and batch_start_time (datetime).

Developers may optionally override this method to add custom markers to the context dict and/or to initialize batch resources - such as initializing a local temp file to hold batch records before uploading.

Parameters

context – Stream partition or context dictionary.