Batch requests and jobs

Batch#

POST /v1/batch runs up to 100 GET requests in one call. Each item names a path and its parameters, exactly as you would send them alone:

curl -X POST "https://api.socialscrape.io/v1/batch" \
  -H "x-api-key: $SOCIALSCRAPE_API_KEY" \
  -H "content-type: application/json" \
  -H "Idempotency-Key: 6f1c2a9e-0b7d-4c55-9a39-2f4f0f6a1c01" \
  -d '{"requests":[
        {"path":"/v1/github/user","params":{"handle":"torvalds"}},
        {"path":"/v1/github/user","params":{"handle":"gaearon"}}
      ]}'

The answer has one full envelope per item, in order, in data.results, each with its own request_id, charge and status:

{
  "ok": true,
  "credits_charged": 2,
  "data": {
    "object": "batch",
    "results": [
      { "path": "/v1/github/user", "status": 200, "response": { "ok": true, "credits_charged": 1, "data": { } } },
      { "path": "/v1/github/user", "status": 404, "response": { "ok": false, "error": { "code": "not_found" }, "credits_charged": 0 } }
    ]
  }
}

• The batch's credits_charged is the sum of its items'. Each item is priced and cached as if you had called it alone, so failures and cached repeats cost nothing.
• A failing item never fails the others. Check each item's ok.
• Items run concurrently on our side and each counts toward your rate limit.
• Credits are held up front, so concurrent items cannot overdraw the balance.

See Batch requests in the reference.

Retrying safely: Idempotency-Key#

Send an Idempotency-Key header on a POST you might retry (a network error leaves you not knowing if it ran). The same key with the same request returns the first answer again, with x-idempotent-replay: true, and charges nothing a second time. Keys are remembered for 24 hours.

• The same key with a different request answers 409 conflict.
• A retry that arrives while the first is still running answers a retryable 503; wait and retry.
• Server errors, 429 and 402 answers are not stored, so they are safe to retry with the same key after you fix the cause.

Use a fresh random value (a UUID) per logical operation.

Jobs: requests that answer later#

Most requests answer in a second or two. A request still running after about 25 seconds (AI transcription of a long video, a slow platform) answers 202 instead, with a job:

{
  "ok": true,
  "credits_charged": 0,
  "credits_remaining": 5000,
  "cached": false,
  "cached_at": null,
  "request_id": "req_0123456789abcdef01234567",
  "data": {
    "object": "job",
    "job_id": "job_0123456789abcdef01234567",
    "status": "running",
    "poll_url": "https://api.socialscrape.io/v1/jobs/job_0123456789abcdef01234567"
  }
}

Poll data.poll_url with your key. Polling is free. While the job runs you get the same 202; when it finishes you get the final envelope with the result and the charge, once. A failed job returns its error with credits_charged: 0. Job results are kept for 24 hours.

import os
import time
import requests

headers = {"x-api-key": os.environ["SOCIALSCRAPE_API_KEY"]}
res = requests.get("https://api.socialscrape.io/v1/transcript", headers=headers, params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}, timeout=60)

while res.status_code == 202:
    time.sleep(3)
    res = requests.get(res.json()["data"]["poll_url"], headers=headers, timeout=60)

body = res.json()

Look up a job by id with Job result.