singer_sdk.RESTStream¶
- class singer_sdk.RESTStream(tap: singer_sdk.plugin_base.PluginBase, name: Optional[str] = None, schema: Optional[Union[Dict[str, Any], singer.schema.Schema]] = None, path: Optional[str] = None)¶
Abstract base class for REST API streams.
- property authenticator: Optional[singer_sdk.authenticators.APIAuthenticatorBase]¶
Return or set the authenticator for managing HTTP auth headers.
If an authenticator is not specified, REST-based taps will simply pass http_headers as defined in the stream class.
- Returns
Authenticator instance that will be used to authenticate all outgoing requests.
- backoff_handler(details: dict) None¶
Adds additional behaviour prior to retry.
By default will log out backoff details, developers can override to extend or change this behaviour.
- Parameters
details – backoff invocation details https://github.com/litl/backoff#event-handlers
- backoff_max_tries() int¶
The number of attempts before giving up when retrying requests.
Setting to None will retry indefinitely.
- Returns
limit
- Return type
int
- backoff_runtime(*, value: Callable[[Any], int]) Generator[int, None, None]¶
Optional backoff wait generator that can replace the default backoff.expo.
It is based on parsing the thrown exception of the decorated method, making it possible for response values to be in scope.
- Parameters
value – a callable which takes as input the decorated function’s thrown exception and determines how long to wait.
- Yields
The thrown exception
- backoff_wait_generator() Callable[[...], Generator[int, Any, None]]¶
The wait generator used by the backoff decorator on request failure.
See for options: https://github.com/litl/backoff/blob/master/backoff/_wait_gen.py
And see for examples: Code Samples
- Returns
The wait generator
- extra_retry_statuses: List[int] = [429]¶
Response code reference for rate limit retries
- get_next_page_token(response: requests.models.Response, previous_token: Optional[Any]) Any¶
Return token identifying next page or None if all records have been read.
- Parameters
response – A raw requests.Response object.
previous_token – Previous pagination reference.
- Returns
Reference value to retrieve next page.
- get_records(context: Optional[dict]) Iterable[Dict[str, Any]]¶
Return a generator of row-type dictionary objects.
Each row emitted should be a dictionary of property names to their values.
- Parameters
context – Stream partition or context dictionary.
- Yields
One item per (possibly processed) record in the API.
- get_url(context: Optional[dict]) str¶
Get stream entity URL.
Developers override this method to perform dynamic URL generation.
- Parameters
context – Stream partition or context dictionary.
- Returns
A URL, optionally targeted to a specific partition or context.
- get_url_params(context: Optional[dict], next_page_token: Optional[Any]) Dict[str, Any]¶
Return a dictionary of values to be used in URL parameterization.
If paging is supported, developers may override with specific paging logic.
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Dictionary of URL query parameters to use in the request.
- property http_headers: dict¶
Return headers dict to be used for HTTP requests.
If an authenticator is also specified, the authenticator’s headers will be combined with http_headers when making HTTP requests.
- Returns
Dictionary of HTTP headers to use as a base for every request.
- next_page_token_jsonpath: Optional[str] = None¶
Optional JSONPath expression to extract a pagination token from the API response. Example: “$.next_page”
- parse_response(response: requests.models.Response) Iterable[dict]¶
Parse the response and return an iterator of result rows.
- Parameters
response – A raw requests.Response object.
- Yields
One item for every item found in the response.
- prepare_request(context: Optional[dict], next_page_token: Optional[Any]) requests.models.PreparedRequest¶
Prepare a request object.
If partitioning is supported, the context object will contain the partition definitions. Pagination information can be parsed from next_page_token if next_page_token is not None.
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Build a request with the stream’s URL, path, query parameters, HTTP headers and authenticator.
- prepare_request_payload(context: Optional[dict], next_page_token: Optional[Any]) Optional[dict]¶
Prepare the data payload for the REST API request.
By default, no payload will be sent (return None).
Developers may override this method if the API requires a custom payload along with the request. (This is generally not required for APIs which use the HTTP ‘GET’ method.)
- Parameters
context – Stream partition or context dictionary.
next_page_token – Token, page number or any request argument to request the next page of data.
- Returns
Dictionary with the body to use for the request.
- records_jsonpath: str = '$[*]'¶
JSONPath expression to extract records from the API response.
- request_decorator(func: Callable) Callable¶
Instantiate a decorator for handling request failures.
Uses a wait generator defined in backoff_wait_generator to determine backoff behaviour. Try limit is defined in backoff_max_tries, and will trigger the event defined in backoff_handler before retrying. Developers may override one or all of these methods to provide custom backoff or retry handling.
- Parameters
func – Function to decorate.
- Returns
A decorated method.
- request_records(context: Optional[dict]) Iterable[dict]¶
Request records from REST endpoint(s), returning response records.
If pagination is detected, pages will be recursed automatically.
- Parameters
context – Stream partition or context dictionary.
- Yields
An item for every record in the response.
- Raises
RuntimeError – If a loop in pagination is detected. That is, when two consecutive pagination tokens are identical.
- property requests_session: requests.sessions.Session¶
Get requests session.
- Returns
The requests.Session object for HTTP requests.
- response_error_message(response: requests.models.Response) str¶
Build error message for invalid http statuses.
- Parameters
response – A requests.Response object.
- Returns
The error message
- Return type
str
- property timeout: int¶
Return the request timeout limit in seconds.
The default timeout is 300 seconds, or as defined by DEFAULT_REQUEST_TIMEOUT.
- Returns
The request timeout limit as number of seconds.
- abstract property url_base: str¶
Return the base url, e.g.
https://api.mysite.com/v3/.
- validate_response(response: requests.models.Response) None¶
Validate HTTP response.
Checks for error status codes and wether they are fatal or retriable.
In case an error is deemed transient and can be safely retried, then this method should raise an
singer_sdk.exceptions.RetriableAPIError. By default this applies to 5xx error codes, along with values set in:extra_retry_statusesIn case an error is unrecoverable raises a
singer_sdk.exceptions.FatalAPIError. By default, this applies to 4xx errors, excluding values found in:extra_retry_statusesTap developers are encouraged to override this method if their APIs use HTTP status codes in non-conventional ways, or if they communicate errors differently (e.g. in the response body).
- Parameters
response – A requests.Response object.
- Raises
FatalAPIError – If the request is not retriable.
RetriableAPIError – If the request is retriable.