singer_sdk.RESTStream¶
-
class
singer_sdk.RESTStream(tap: singer_sdk.plugin_base.PluginBase, name: Optional[str] = None, schema: Optional[Union[Dict[str, Any], singer.schema.Schema]] = None, path: Optional[str] = None)¶ Abstract base class for REST API streams.
-
property
authenticator¶ Return or set the authenticator for managing HTTP auth headers.
If an authenticator is not specified, REST-based taps will simply pass http_headers as defined in the stream class.
- Returns
Authenticator instance that will be used to authenticate all outgoing requests.
-
get_next_page_token(response: requests.models.Response, previous_token: Optional[Any]) → Any¶ Return token identifying next page or None if all records have been read.
- Parameters
response – A raw requests.Response object.
previous_token – Previous pagination reference.
- Returns
Reference value to retrieve next page.
-
get_records(context: Optional[dict]) → Iterable[Dict[str, Any]]¶ Return a generator of row-type dictionary objects.
Each row emitted should be a dictionary of property names to their values.
- Parameters
context – Stream partition or context dictionary.
- Yields
One item per (possibly processed) record in the API.
-
get_url(context: Optional[dict]) → str¶ Get stream entity URL.
Developers override this method to perform dynamic URL generation.
- Parameters
context – Stream partition or context dictionary.
- Returns
A URL, optionally targeted to a specific partition or context.
-
get_url_params(context: Optional[dict], next_page_token: Optional[Any]) → Dict[str, Any]¶ Return a dictionary of values to be used in URL parameterization.
If paging is supported, developers may override with specific paging logic.
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Dictionary of URL query parameters to use in the request.
-
property
http_headers¶ Return headers dict to be used for HTTP requests.
If an authenticator is also specified, the authenticator’s headers will be combined with http_headers when making HTTP requests.
- Returns
Dictionary of HTTP headers to use as a base for every request.
-
next_page_token_jsonpath: Optional[str] = None¶ Optional JSONPath expression to extract a pagination token from the API response. Example: “$.next_page”
-
parse_response(response: requests.models.Response) → Iterable[dict]¶ Parse the response and return an iterator of result rows.
- Parameters
response – A raw requests.Response object.
- Yields
One item for every item found in the response.
-
prepare_request(context: Optional[dict], next_page_token: Optional[Any]) → requests.models.PreparedRequest¶ Prepare a request object.
If partitioning is supported, the context object will contain the partition definitions. Pagination information can be parsed from next_page_token if next_page_token is not None.
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Build a request with the stream’s URL, path, query parameters, HTTP headers and authenticator.
-
prepare_request_payload(context: Optional[dict], next_page_token: Optional[Any]) → Optional[dict]¶ Prepare the data payload for the REST API request.
By default, no payload will be sent (return None).
Developers may override this method if the API requires a custom payload along with the request. (This is generally not required for APIs which use the HTTP ‘GET’ method.)
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Dictionary with the body to use for the request.
-
records_jsonpath: str = '$[*]'¶ JSONPath expression to extract records from the API response.
-
request_records(context: Optional[dict]) → Iterable[dict]¶ Request records from REST endpoint(s), returning response records.
If pagination is detected, pages will be recursed automatically.
- Parameters
context – Stream partition or context dictionary.
- Yields
An item for every record in the response.
- Raises
RuntimeError – If a loop in pagination is detected. That is, when two consecutive pagination tokens are identical.
-
property
requests_session¶ Get requests session.
- Returns
The requests.Session object for HTTP requests.
-
abstract property
url_base¶ //api.mysite.com/v3/’.
- Type
Return the base url, e.g. ‘https
-
property