singer_sdk.RESTStream

class singer_sdk.RESTStream(tap: singer_sdk.plugin_base.PluginBase, name: Optional[str] = None, schema: Optional[Union[Dict[str, Any], singer.schema.Schema]] = None, path: Optional[str] = None)

Abstract base class for REST API streams.

property authenticator

Return or set the authenticator for managing HTTP auth headers.

If an authenticator is not specified, REST-based taps will simply pass http_headers as defined in the stream class.

Returns

Authenticator instance that will be used to authenticate all outgoing requests.

get_next_page_token(response: requests.models.Response, previous_token: Optional[Any])Any

Return token identifying next page or None if all records have been read.

Parameters
  • response – A raw requests.Response object.

  • previous_token – Previous pagination reference.

Returns

Reference value to retrieve next page.

get_records(context: Optional[dict])Iterable[Dict[str, Any]]

Return a generator of row-type dictionary objects.

Each row emitted should be a dictionary of property names to their values.

Parameters

context – Stream partition or context dictionary.

Yields

One item per (possibly processed) record in the API.

get_url(context: Optional[dict])str

Get stream entity URL.

Developers override this method to perform dynamic URL generation.

Parameters

context – Stream partition or context dictionary.

Returns

A URL, optionally targeted to a specific partition or context.

get_url_params(context: Optional[dict], next_page_token: Optional[Any])Dict[str, Any]

Return a dictionary of values to be used in URL parameterization.

If paging is supported, developers may override with specific paging logic.

Parameters
  • context – Stream partition or context dictionary.

  • next_page_token – Token, page number or any request argument to request the next page of data.

Returns

Dictionary of URL query parameters to use in the request.

property http_headers

Return headers dict to be used for HTTP requests.

If an authenticator is also specified, the authenticator’s headers will be combined with http_headers when making HTTP requests.

Returns

Dictionary of HTTP headers to use as a base for every request.

next_page_token_jsonpath: Optional[str] = None

Optional JSONPath expression to extract a pagination token from the API response. Example: “$.next_page”

parse_response(response: requests.models.Response)Iterable[dict]

Parse the response and return an iterator of result rows.

Parameters

response – A raw requests.Response object.

Yields

One item for every item found in the response.

prepare_request(context: Optional[dict], next_page_token: Optional[Any])requests.models.PreparedRequest

Prepare a request object.

If partitioning is supported, the context object will contain the partition definitions. Pagination information can be parsed from next_page_token if next_page_token is not None.

Parameters
  • context – Stream partition or context dictionary.

  • next_page_token – Token, page number or any request argument to request the next page of data.

Returns

Build a request with the stream’s URL, path, query parameters, HTTP headers and authenticator.

prepare_request_payload(context: Optional[dict], next_page_token: Optional[Any])Optional[dict]

Prepare the data payload for the REST API request.

By default, no payload will be sent (return None).

Developers may override this method if the API requires a custom payload along with the request. (This is generally not required for APIs which use the HTTP ‘GET’ method.)

Parameters
  • context – Stream partition or context dictionary.

  • next_page_token – Token, page number or any request argument to request the next page of data.

Returns

Dictionary with the body to use for the request.

records_jsonpath: str = '$[*]'

JSONPath expression to extract records from the API response.

request_records(context: Optional[dict])Iterable[dict]

Request records from REST endpoint(s), returning response records.

If pagination is detected, pages will be recursed automatically.

Parameters

context – Stream partition or context dictionary.

Yields

An item for every record in the response.

Raises

RuntimeError – If a loop in pagination is detected. That is, when two consecutive pagination tokens are identical.

property requests_session

Get requests session.

Returns

The requests.Session object for HTTP requests.

abstract property url_base

//api.mysite.com/v3/’.

Type

Return the base url, e.g. ‘https