Scrape Instagram profile embed HTML with Python

Preview

The API is in early access. Ask for a key; the examples use api.socialscrape.io.

Set up

You need Python 3 and the requests package.

python -m venv .venv
source .venv/bin/activate
pip install requests
export SOCIALSCRAPE_API_KEY=ss_live_...

The script

Save as scrape.py and run python scrape.py. It calls GET /v1/instagram/profile/embed.

scrape.py

import json
import os
import time

import requests

URL = "https://api.socialscrape.io/v1/instagram/profile/embed"
HEADERS = {"x-api-key": os.environ["SOCIALSCRAPE_API_KEY"]}
PARAMS = {
    "handle": "nasa",
}


def pick(obj, *path):
    """Follow a path through nested dicts; None if any step is missing or null."""
    for key in path:
        obj = obj.get(key) if isinstance(obj, dict) else None
    return obj


def get(params):
    """One request. Failures cost nothing; retry the ones the API marks retryable."""
    for attempt in range(3):
        body = requests.get(URL, headers=HEADERS, params=params, timeout=60).json()
        if body["ok"]:
            return body
        err = body["error"]
        if not err["retryable"] or attempt == 2:
            raise RuntimeError(f'{err["code"]}: {err["message"]}')
        time.sleep(2**attempt)


body = get(PARAMS)
data = body["data"]

with open("out.json", "w", encoding="utf-8") as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

print("url:", str(pick(data, "url"))[:200])
print(f"saved out.json, credits left: {body['credits_remaining']}")

How it works

get() sends one request and retries the failures the API marks retryable. A failed request costs nothing.
The result is one object in data.
pick() reads nested fields and gives None instead of crashing when the platform returned null.
The whole object goes to out.json and a few fields are printed.

What you get back

These fields come from a recorded sample. The full field list is on the endpoint page.

Fields in the scriptSample

{
  "url": "https://www.instagram.com/nasa/"
}

The short version

Without paging, saving or retries, the request is this.

import os
import requests

params = {
    "handle": "nasa",
}
res = requests.get(
    "https://api.socialscrape.io/v1/instagram/profile/embed",
    headers={"x-api-key": os.environ["SOCIALSCRAPE_API_KEY"]},
    params=params,
    timeout=60,
)
body = res.json()

if body["ok"]:
    print(body["data"])
else:
    print(body["error"]["code"], body["error"]["message"])

With the SDK

The Python client does the retries for you. It is in preview and not published yet.

from socialscrape import SocialScrape

client = SocialScrape()  # reads SOCIALSCRAPE_API_KEY

res = client.instagram.profile_embed(handle="nasa")
print(res.data)
print(res.credits_charged, res.credits_remaining)

What it costs

1 credit. A failed request costs nothing. Add max_age_days to reuse a cached result for 0 credits. Results are kept 30 days.

The same request in nine languages.
Instagram profile embed HTML API: sample response and every parameter.
The JavaScript version.
All Instagram endpoints.

How to scrape Instagram profile embed HTML with Python