Orchestrate Web Crawls with Scrapyd and Automated Data Enrichment

How it works This workflow runs a spider job in the background via Scrapyd, using a YAML config that defines selectors and parsing rules. When triggered, it schedules the spider with parameters (query, project ID, page limits, etc.). The workflow polls Scrapyd until the job finishes. Once complete, it fetches the output items, enriches them (parse JSONL, deduplicate, extract ID/part number/make/model/part name, normalize price), sorts results, and returns structured JSON. Optional debug outputs such as logs, HTML dumps, and screenshots are also collected.

How to use Use the manual trigger for testing, or replace it with webhook, schedule, or other triggers. Adjust the runtime parameters (q, project_id, pages, etc.) directly in the workflow when running. The background spider config (YAML and spider code) must be updated separately — this workflow only orchestrates and enriches results, it does not define scraping logic.

Requirements

Scrapyd service for job scheduling & status tracking

A deployed spider with a valid YAML config (adjust selectors there)

JSON Lines output (items.jl) from the spider

Endpoints for optional artifacts (logs, HTML, screenshots)

n8n with HTTP, Wait, Code, and Aggregate nodes enabled

Customising this workflow

Update the YAML config if the target website structure changes

Modify the enrichment code to extract different fields (e.g., categories, ratings)

Adjust deduplication (cheapest, newest, or other logic)

Toggle debug retrieval depending on performance/storage needs

Extend webhook response to integrate with databases, APIs, or downstream workflows