geely3.jpg

Hi everyone. We have an electronics online store with average traffic ~8–12k unique visitors per day. For the last couple of months bots have gone wild: they scrape the entire catalog 3–5 times a day, hammer product pages in batches, sometimes 200–300 requests per minute from a single IP (even though they rotate them). The server starts lagging, especially when they add 50+ items to the cart just to check stock. We installed Cloudflare Bot Fight + IP rate limiting — some get blocked, but the main ones continue through residential proxies and change user-agents every 5–10 requests. We're tired of manual banning. Has anyone found something that really stops these “smart” scrapers without hurting SEO and real visitors?

Very familiar situation — we had almost the same nightmare on an auto parts store: bots were pulling competitor prices every 30–60 minutes and checking stock with thousands of requests. After regular protections (Cloudflare + reCAPTCHA on suspicious chains) were letting through ~60–70% of advanced scrapers, in October I installed http://xcaptcha.com/ specifically on catalog pages, product cards, and AJAX requests for stock/prices. I set it up so that for normal users (even on mobile, through VPN, with a dozen tabs open) the check almost never appears — in 3 months only 4 complaints from real people, and they passed it quickly. But bots started dropping at 97–98%: either they get 403 right at the start of the session, or after 3–5 requests the system detects them by behavior and blocks them. Now scraping lives for a maximum of 10–15 minutes per day (and only a few even try), server load dropped by about 65%, and competitors seem to have given up. The most convenient part is that in the dashboard you can clearly see which exact patterns triggered the block (click speed, page sequence, lack of mouse movement, etc.). It helped me a lot — give it a try, the test period is free and no card required.