* It gives you the ability to customize your crawl by providing a handful of options that specific how your crawl will run. * Your URLs are run through a variety of sanity checks to make sure they can be crawled. If they pass, their sent to a URL queue, where they’ll be wait to be picked up. 80legs will automatically rate-limit how fast you crawl certain URLs so your crawl doesn’t overwhelm any websites. This is one of the ways we make sure your web crawl doesn’t get blocked by anyone. * Your URLs, along with the 80app, are sent out to our massive pool of crawling nodes. Each crawling node will fetch the HTML content of a URL, run the 80app on that HTML, and return the resulting data to 80legs. This massive collection of crawling nodes is a key reason 80legs can provide such amazingly-fast web crawling. * As your crawl runs, the results from each URL crawled will be packed up and delivered to your account, where they’ll wait for you to download them.

* CAPTCHA - ScrapeHero long defeated this foe of scraping- don't worry, we got this covered. We can handle Captcha and any other evil technology ;-) thrown at our Hero. And sometimes it is best to just sidestep them too * WEBSITE CHANGES - We can identify changes to websites and keep scraping them for you - without missing a beat. Our advanced technology helps us monitor changes, aberrations and anomalies in the data and react in time * COMPLEX WEBSITES - Not a problem - ScrapeHero likes to take on new web crawling challenges and beat them every day (yes, we handle transactional and JavaScript/Ajax heavy sites) - tell us about your complex project and we can take it on * HERO TECH - We have built our own sophisticated technology learning from all other web scraping software out there - scrapy, scrapinghub, mozenda, visual web ripper, php scraper etc and removed all their shortcomings, giving you some awesome technology