article.md — crawling-frequency
Ctrl + F search
“How often should we crawl?” is usually the first question in price monitoring. There is no universal number, because refresh rate depends on category volatility, assortment size, and anti-bot constraints.
1) A simple frequency framework
- KVI / Top SKUs: crawl more often (e.g., 1–6 times/day), especially for highly competitive items.
- Long tail SKUs: crawl less frequently (e.g., 1–4 times/week) to save budget and reduce blocks.
- Promotions: increase frequency during promo windows or specific days of week.
2) What defines “big webshop” crawling
- Large SKU sets (hundreds of thousands to millions)
- Need for stable crawling under rate limits and blocks
- Continuous operation (scheduler, retries, observability)
3) Typical crawl modes
In practice, teams use at least two modes:
- FULL: deep crawl / full refresh (e.g., 1× every few days)
- PARTIAL: category-based or incremental updates (higher frequency)
4) Practical examples
A common setup is to split the workload into two lists:
- KVI list (must-update items): frequent crawling + stricter monitoring
- Everything else: less frequent crawling, but consistent long-term coverage
Note: the right panel shows “crawl runs” as an illustrative proof block. It is intentionally presented in a terminal / HTOP-like style (no images), so it can be embedded into an SEO article as a code-like reference.