Migrate your REST tap to use the new pagination classes!#

This guide will walk you through the process of migrating your REST tap to use the new pagination classes.

What’s changed?#

In a future release of the SDK, the RESTStream.get_next_page_token method will no longer be called as part of the stream pagination process. It is replaced by the RESTStream.get_new_paginator.

In the meantime, the get_next_page_token method will still be called if you define it in your stream class, but a deprecation warning will be logged, if warnings are enabled.

Note

If you have not overridden the get_next_page_token method in your stream class, you do not need to make any changes.

Why is this change being made?#

The new pagination classes are designed to be more flexible, easier to understand, and easier to test. They also align better with the Single Responsibility Principle.

How to migrate#

Example: Offset pagination#

Another common pattern is to use an offset parameter to indicate the starting point of the next page of results. The BaseOffsetPaginator class can be used to handle this pattern.

# New implementation

from singer_sdk.pagination import BaseOffsetPaginator

class MyPaginator(BaseOffsetPaginator):
    def has_more(self, response):
        data = response.json()
        return data.get("has_more", False)


class MyStream(RESTStream):
    def get_new_paginator(self):
        return BaseOffsetPaginator(start_value=0, page_size=250)

    def get_url_params(self, context, next_page_token):
        params = {}

        # Next page token is an offset
        if next_page_token:
            params["offset"] = next_page_token

        return params