Templates/News Crawler Pipeline

News Crawler Pipeline

Data PipelineProductionData harvesting

Scheduled scraper pipeline with URL-frontier queuing, document storage, full-text indexing, and CDN-cached read API. Use for aggregators, content trackers, or data harvesting systems.

Recommended for: Teams building content aggregation pipelines

9 nodes10 connectionsFrontier queuingIndexed retrievalFailure isolation

Scope Notes

Intended for Teams building content aggregation pipelines at Production complexity and Data harvesting scope. Treat the graph as a starting shape rather than a fixed stack: keep the topology and the signals below, then swap providers to match your platform.

Frontier queuingIndexed retrievalFailure isolation

Review Checklist

0 of 12 reviewed

Use Case

Web crawlers, content aggregation, search indexing

Best Fit Scenarios

  • News and feed aggregation
  • Content monitoring and archival
  • Search indexes built from external pages

Stack Breakdown

SchedulerWorker ServiceRedis QueueMongoDBElasticsearchFastAPI

Architecture Layers

1Read path (client & CDN)
2Scheduler & workers
3URL frontier
4Document storage
5Search index
6Query API & failure isolation

Components by Category

async

SchedulerRedis QueueDead Letter Queue

backend

Worker ServiceFastAPI

database

MongoDBElasticsearch

infra

CDN

frontend

Reader Client

Why This Topology Works

The scheduler feeds a Redis-backed frontier that workers claim from, so crawl pressure is bounded and resumable. MongoDB stores raw articles, Elasticsearch serves retrieval, and the CDN caches the read API for readers and RSS consumers.

Scaling Notes

Workers scale with frontier depth. MongoDB absorbs document writes while Elasticsearch handles search traffic, and the CDN shields the query API from read spikes.

Observability

Track crawl success rate per domain, frontier depth, and indexing lag between MongoDB and Elasticsearch. Unfetchable tasks accumulate in the DLQ for inspection.

Typical Bottlenecks

  • Per-domain rate limits and robots rules
  • Duplicate documents across sources
  • Search indexing lag behind crawl volume

Async Flow and Reliability

Crawling is fully asynchronous: the scheduler enqueues, workers claim and fetch, and storage/indexing happen after the fetch returns. Failed fetches park in the DLQ instead of blocking the frontier.

Upgrade Path

Add an egress/politeness boundary and a dedupe stage that canonicalises URLs and content hashes before storage, then move raw HTML to object storage with MongoDB holding only canonical documents.