The best of the Internet. Dedicated to the most funny, interesting, and cool stuff on the Internet today.
- Posts: 458
- Joined: Tue Jul 09, 2019 11:54 pm
Cralwer that's not respecting robots.txt rules and using AmazonAWS services to crawl websites and mask (or re-route) their origin IPs. Their crawling rates are extremely high.