Tap and target performance

The following sections provide different strategies to improve the performance of your tap or target.

Use BATCH mode

See the BATCH messages page for more information.

Use a different message writer or reader

Starting from version 0.45.0, the Meltano Singer SDK supports different message writers and readers.

For example, to update your tap to use a message writer based on the msgspec serialization library:

from singer_sdk.contrib.msgspec import MsgSpecWriter


class MyTap(Tap):
    message_writer_class = MsgSpecWriter

To update your target:

from singer_sdk.contrib.msgspec import MsgSpecReader


class MyTarget(Target):
    message_reader_class = MsgSpecReader

Measuring performance

We’ve had success using viztracer to create flame graphs for SDK-based packages and find if there are any serious performance bottlenecks.

You can start doing the same in your package. Start by installing viztracer.

Poetry

$ poetry add --group dev viztracer

uv

$ uv add --group dev viztracer

Then simply run your package’s CLI as normal, preceded by the viztracer command

Poetry

$ poetry run viztracer my-tap
$ poetry run viztracer -- my-target --config=config.json --input=messages.json

uv

$ uv run viztracer my-tap
$ uv run viztracer -- my-target --config=config.json --input=messages.json

That command will produce a result.json file which you can explore with the vizviewer tool.

Poetry

$ poetry run vizviewer result.json

uv

$ uv run vizviewer result.json

The output should look like this

Note: Chrome seems to work best for running the vizviewer app.