silkworm.exceptions

exception silkworm.exceptions.SilkwormError[source]

Bases: Exception

Base exception for the framework.

exception silkworm.exceptions.HttpError[source]

Bases: SilkwormError

Raised when an HTTP request fails.

exception silkworm.exceptions.HttpTimeoutError[source]

Bases: HttpError

Raised when an HTTP request exceeds its timeout; usually transient.

exception silkworm.exceptions.HttpConnectionError[source]

Bases: HttpError

Raised when the transport fails (DNS, connect, reset); usually transient.

exception silkworm.exceptions.ResponseTooLargeError[source]

Bases: HttpError

Raised when a response body exceeds the configured size limit.

exception silkworm.exceptions.SpiderError[source]

Bases: SilkwormError

Raised when a spider callback errors.

exception silkworm.exceptions.BatchPipelineError[source]

Bases: SilkwormError

Raised when a backend rejects one or more items from a bulk request.

__init__(pipeline, *, total, failed)[source]
Parameters:
Return type:

None

exception silkworm.exceptions.SelectorError[source]

Bases: SilkwormError

Raised when CSS/XPath selector evaluation fails.

exception silkworm.exceptions.MarkdownConversionError[source]

Bases: SilkwormError

Raised when HTML to Markdown conversion fails.

exception silkworm.exceptions.IgnoreRequest[source]

Bases: SilkwormError

Raise from a request or response middleware to drop a request silently.

Ignored requests are counted under ignored_requests (labelled by reason) and are neither errors nor passed to errbacks.

Parameters:

  • message – Human-readable explanation used in debug logs.

  • reason – Short, low-cardinality label used in statistics.

__init__(message='', *, reason='ignored')[source]
Parameters:
Return type:

None

exception silkworm.exceptions.DropItem[source]

Bases: SilkwormError

Raise from an item pipeline to discard an item.

Later pipelines are skipped, the item is counted under items_dropped (labelled by reason), and emit() returns normally.

Parameters:

  • message – Human-readable explanation used in logs.

  • reason – Short, low-cardinality label used in statistics.

__init__(message='', *, reason='dropped')[source]
Parameters:
Return type:

None

exception silkworm.exceptions.CloseSpider[source]

Bases: SilkwormError

Raise from a callback, middleware, or pipeline to stop the crawl gracefully.

Pending requests are discarded (or kept for resuming when a job directory is configured), in-flight requests finish, and the crawl result reports reason as its close reason.

__init__(reason='close_spider')[source]
Parameters:

reason (str)

Return type:

None

exception silkworm.exceptions.CrawlFailedError[source]

Bases: SilkwormError

Raised by Engine.run() when the crawl violates its failure policy.

result

The complete CrawlResult, including the policy violations in result.failures.

__init__(result)[source]
Parameters:

result (CrawlResult)

Return type:

None