silkworm.httpcache

On-disk HTTP response cache for fast, polite development re-runs.

Wrap the engine’s HTTP client with an HttpCache so repeated runs of a spider (while writing selectors, for example) read pages from disk instead of downloading them again:

run_spider(MySpider, http_cache=HttpCache(".silkworm/httpcache"))

Entries are keyed by request_fingerprint(), so the same method, canonical URL, and body share one entry regardless of headers or metadata. Cached responses carry an x-silkworm-cache: hit header.

class silkworm.httpcache.HttpCache

Bases: object

Configuration and storage for cached HTTP responses.

Parameters:

• directory – Cache directory; created when missing.

• expiration – Maximum entry age (seconds or timedelta); None keeps entries forever.

• ignore_statuses – Response statuses that are never stored (e.g. 5xx).

• methods – HTTP methods eligible for caching.

Set request.meta["dont_cache"] = True to bypass the cache for one request.

__init__(directory, *, expiration=None, ignore_statuses=(500, 502, 503, 504, 522, 524, 408, 429), methods=('GET', 'HEAD'))

Parameters:

• directory (str | os.PathLike[str])

• expiration (float | timedelta | None)

• ignore_statuses (Iterable[int])

• methods (Iterable[str])

Return type:

None

wrap(client)

Return a client that serves client’s responses from this cache.

Parameters:

client (FetchClient)

Return type:

CachingHttpClient

is_cacheable(request)

Return whether request may be read from or written to the cache.

Parameters:

request (Request)

Return type:

bool

load(request)

Return (metadata, body) for a fresh entry, else None.

Parameters:

request (Request)

Return type:

tuple[dict[str, object], bytes] | None

store(request, response)

Write response for request; return whether it was stored.

Parameters:

• request (Request)

• response (Response)

Return type:

bool

class silkworm.httpcache.CachingHttpClient

Bases: object

HTTP client wrapper that reads and writes an HttpCache.

Created by HttpCache.wrap(); exposes the same interface as the wrapped client so the engine can use it transparently.

__init__(client, cache)

Parameters:

• client (FetchClient)

• cache (HttpCache)

Return type:

None

property concurrency: int

Return the wrapped client’s concurrency.

property html_max_size_bytes: int

Return the wrapped client’s HTML parsing limit.

async fetch(req)

Return a cached response for req or fetch and store it.

Parameters:

req (Request)

Return type:

Response

async close()

Close the wrapped client and log cache effectiveness.

Return type:

None