silkworm.settings

Layered engine settings for runners and the command line.

Engine options resolve from four layers, lowest precedence first:

  1. Engine defaults.

  2. Environment variables named SILKWORM_<OPTION> (for example SILKWORM_CONCURRENCY=32 or SILKWORM_MAX_DEPTH=3).

  3. The spider’s custom_settings entries whose keys name engine options.

  4. Options passed explicitly to a runner or on the command line.

Only scalar options (numbers, booleans, strings, and paths) can come from the environment or custom_settings; middlewares, pipelines, and clients are always passed in code. Values are validated and converted, so a typo such as SILKWORM_MAX_DEPTH=three fails with a clear error instead of being ignored. Engine itself never reads these layers; use resolve_options() (as the runners do) to apply them.

silkworm.settings.coerce_setting(name, value, *, source='setting')[source]

Validate and convert value for engine option name.

Parameters:

  • name (str) – Engine option name, e.g. "max_depth".

  • value (object) – Raw value (a string from the environment or command line, or a JSON value from custom_settings).

  • source (str) – Where the value came from, used in error messages.

Raises:

  • KeyError – If name is not a configurable engine option.

  • ValueError – If the value cannot be converted.

Return type:

object

silkworm.settings.resolve_options(spider, options, *, environ=None)[source]

Merge the settings layers into the options passed to Engine.

Parameters:

  • spider (Spider) – Spider whose custom_settings form the third layer.

  • options (EngineOptions | Mapping[str, object]) – Explicit options; these always win.

  • environ (Mapping[str, str] | None) – Environment mapping; defaults to os.environ.

Return type:

EngineOptions

silkworm.settings.settings_from_env(environ=None)[source]

Return engine options configured through SILKWORM_<OPTION> variables.

Parameters:

environ (Mapping[str, str] | None)

Return type:

dict[str, object]

silkworm.settings.settings_from_spider(spider)[source]

Return engine options configured in spider.custom_settings.

Parameters:

spider (Spider)

Return type:

dict[str, object]