Blocked by HTML

The error#

If the content-type contains text/html and the status is 400 or higher:

Blocked by Cloudflare: {status} - {remaining}/{limit} - {username}
Blocked by HTML: {status} - {remaining}/{limit} - {username}

The Cloudflare line is chosen when the response has a cf-ray header. Otherwise the word is HTML. The client then raises AbortReqError with no message.

A different AbortReqError is raised when a new x-client-transaction-id still gets HTTP 404 after three tries:

Faield to get XClIdGen. See: https://github.com/vladkens/twscrape/issues/248

The spelling Faield is in the source.

What triggers it#

An HTML body on an error status, which is the bot check or a block page rather than GraphQL JSON. The XClId line is three 404s while refreshing the client transaction id.

How to confirm#

The log starts with Blocked by Cloudflare or Blocked by HTML. The exception type is AbortReqError. If the text is the Faield to get XClIdGen sentence, it is the transaction-id failure, not the HTML block.

Fixes#

AbortReqError stops that request. The HTML block is the IP or the client being challenged. The library does not solve a Cloudflare challenge in _check_rep. A proxy is available on the account context (proxy= is passed into the client). The transaction-id failure points at the GitHub issue named in the message. Updating twscrape is how that generator stays current. This page was read against 0.20.1.

SocialScrape#

Profile is GET /v1/twitter/profile. One post is GET /v1/twitter/tweet. A profile's posts are GET /v1/twitter/profile/tweets. These read what X shows logged out.

Failed requests cost 0 credits. New accounts get 100 credits on signup, no card. Suspended accounts come back as removed, unknown ones as not_found.