* It gives you the ability to customize your crawl by providing a handful of options that specific how your crawl will run. * Your URLs are run through a variety of sanity checks to make sure they can be crawled. If they pass, their sent to a URL queue, where they’ll be wait to be picked up. 80legs will automatically rate-limit how fast you crawl certain URLs so your crawl doesn’t overwhelm any websites. This is one of the ways we make sure your web crawl doesn’t get blocked by anyone. * Your URLs, along with the 80app, are sent out to our massive pool of crawling nodes. Each crawling node will fetch the HTML content of a URL, run the 80app on that HTML, and return the resulting data to 80legs. This massive collection of crawling nodes is a key reason 80legs can provide such amazingly-fast web crawling. * As your crawl runs, the results from each URL crawled will be packed up and delivered to your account, where they’ll wait for you to download them.

Optimized web browsers: Web browsers are used to load and parse web pages, and Content Grabber has a range of different browsers to achieve maximum performance in every scenario - from a fully dynamic web browser to the ultra-fast HTML5 parser only browser. Fine tuning: All web scraping tools spend most of their time waiting for new web pages to load, so it’s important to optimize this process. Debugging: Content Grabber has one of the best debuggers of any web automation software, and this will help you build reliable agents where all issues that can be resolved at design time are resolved at design time. Error handling: Many web scraping errors are unavoidable even with the best designed agents, and this is where error handling comes into play. One example could be an unreliable website that suddenly starts returning only error pages, and requires a web browser restart to start functioning again. Ease-of-Use: The Content Grabber agent editor has a typical point and click user interface where you click on the content you want to extract, or on the buttons and links you want to follow. Data: Data is everything when it comes to web scraping. Content Grabber allows you to load data from any source and use it in your agents for anything you need. You can also export extracted data to almost anywhere. This flexibility is key - enabling your technology to grow with your business. Scripting: No one wants to write scripts to get things done and with Content Grabber you rarely have to. However, if you have some unusual requirements, or you need to fine tune some process, it's nice to know the ability is there.