The job
A study needs repeatable collection. Page through a feed with its cursor, keep the raw response, and record when each page was fetched.
Endpoints
| Endpoint | Returns | Cost |
|---|---|---|
GET /v1/youtube/search | YouTube search | 1 credit |
GET /v1/youtube/video/comments | YouTube video comments | 1 credit |
GET /v1/tiktok/search/hashtag | TikTok videos by hashtag | 1 credit |
GET /v1/bluesky/profile/posts | Bluesky profile posts | 1 credit |
GET /v1/telegram/channel/posts | Telegram public channel posts | 1 credit |
Failed requests and cached repeats cost 0 credits. Send max_age_days to accept a recent cached copy. See credits and caching.
Recipe
- Define the query and the date window.
- Page with the cursor until has_more is false.
- Store the raw response and the fetch time.
Preview
The API is in early access. Calls use
api.socialscrape.io and return the shapes in the reference.Python
import os
import requests
API = "https://api.socialscrape.io"
HEADERS = {"x-api-key": os.environ["SOCIALSCRAPE_API_KEY"]}
def get(path, **params):
res = requests.get(API + path, headers=HEADERS, params=params, timeout=60)
res.raise_for_status()
return res.json()["data"]
# YouTube search
page = get("/v1/youtube/search", query="climate policy", type="video", upload_date="month")
# YouTube video comments
comments = get("/v1/youtube/video/comments", url=page["items"][0]["url"], sort="top")
print(page["has_more"], page["cursor"])
print(len(comments["items"]))TypeScript
const API = "https://api.socialscrape.io";
const headers = { "x-api-key": process.env.SOCIALSCRAPE_API_KEY! };
async function get(path: string, params: Record<string, string | number | boolean>) {
const query = new URLSearchParams(Object.entries(params).map(([k, v]) => [k, String(v)]));
const res = await fetch(`${API}${path}?${query}`, { headers });
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
return (await res.json()).data;
}
// YouTube search
const page = await get("/v1/youtube/search", { query: "climate policy", type: "video", upload_date: "month" });
// YouTube video comments
const comments = await get("/v1/youtube/video/comments", { url: page.items[0].url, sort: "top" });
console.log(page.has_more, page.cursor);
console.log(comments.items.length);Good to know
A cached response carries cached_at, so you can tell fresh collection from a repeat. Check your ethics board and each platform's terms for your use.