SDK Implementation Details

This section documents certain behaviors and expectations of the SDK framework.

Implementation Details

• Command Line Reference
  • Common CLI Options
    • --help
    • --version
    • --about
      • --format
    • --config
      • --config ENV
  • Tap-Specific CLI Options
    • --state
    • --catalog
      • Input Catalog Stream Selection
    • --test
      • --test schema
    • --discover
  • Target-Specific CLI Options
    • --input
• SDK Implementation Details - Catalog Discovery
  • Additional Discovery Mode References
• SDK Implementation Details - Catalog Metadata
  • Additional Singer Metadata References
• SDK Implementation Details - Record Metadata
• SDK Implementation Details - Tap Metrics
  • Customization options
    • metrics_log_level
    • SINGER_SDK_LOG_CONFIG
  • Additional Singer Metrics References
• SDK Implementation Details - Logging
  • Logging levels
  • Default log format
  • Custom logging configuration
• SDK Implementation Details - Stream State
  • Important Notice Regarding State
  • Standard State Format
  • Per-Stream STATE Properties
  • State Message Frequency
  • Backwards Compatibility
  • Partitioned State
  • Record Duplication
  • Advanced Use Cases
    • Partitioned State Example
  • Replication Key is Singular
  • The Impact of Sorting on Incremental Sync
    • Dealing with Unsorted Streams
    • Replication Key Signposts
  • Additional Singer State References
• SDK Implementation Details - “At Least Once” Delivery Promise
  • Causes of Record Duplication
    • Cause #1: Greater-than-or-equal-to comparisons
    • Cause #2: Replication Key Signposts
    • Cause #3: Stream interruption
  • Recommended Deduplication Strategies
    • Strategy #1: Removing duplicates using primary keys in the target
    • Strategy #2: Removing duplicates using dbt transformations
      • Sample dedupe implementation using dbt:
  • Future proposals to mitigate record duplication

How to use the implementation reference material

Note: You should not need to master all of the details here in order to build your tap, and the behaviors described here should be automatic and/or intuitive. For general guidance on tap development, please instead refer to our Dev Guide.

The specifications provided in this section are documented primarily to support advanced use cases, behavior overrides, backwards compatibility with legacy taps, debugging unexpected behaviors, or contributing back to the SDK itself.