* It gives you the ability to customize your crawl by providing a handful of options that specific how your crawl will run. * Your URLs are run through a variety of sanity checks to make sure they can be crawled. If they pass, their sent to a URL queue, where they’ll be wait to be picked up. 80legs will automatically rate-limit how fast you crawl certain URLs so your crawl doesn’t overwhelm any websites. This is one of the ways we make sure your web crawl doesn’t get blocked by anyone. * Your URLs, along with the 80app, are sent out to our massive pool of crawling nodes. Each crawling node will fetch the HTML content of a URL, run the 80app on that HTML, and return the resulting data to 80legs. This massive collection of crawling nodes is a key reason 80legs can provide such amazingly-fast web crawling. * As your crawl runs, the results from each URL crawled will be packed up and delivered to your account, where they’ll wait for you to download them.

* Point-and-Click Agent Creation: Easily create extraction agents simply by browsing websites – no coding required. * JavaScript Injection: Connotate automatically handles complex navigation, such as selecting menu items and options in drop-down controls. * Visual Content Tagging: Extract only the precise content you need by tagging it as you browse webpages, reducing downstream processing requirements. * Database Extraction: Integrate webpage and database content, including content from SQL databases and MongoDB. * Connotate-Optimized Browser: Automatically extract over 95% of sites without programming, including complex JavaScript-based dynamic site technologies, such as Ajax. * Full Story Extract: Automatically extract full stories simply by selecting headlines. * User Behavior Recording: Agents learn how to navigate websites simply by observing how users navigate them when creating agents. * Multi-Page Story Extraction: Easily extract content that spans multiple pages by following next/more links. * Intelligent Machine Learning: Agents adapt automatically to most website changes, reducing maintenance costs by more than 90%. * Language-Agnostic: Extract content from sites in any language * Automated Login: Easily pass credentials to websites to access protected content. * PDF Retrieval: Automatically download PDFs and other files from websites. * Form and Parameter Filling: Easily extract dynamically generated content by automatically filling in forms and other parameters taken from databases or spreadsheets. * SDK: Extend Connotate to extract content from any site, no matter how complex. * Intelligent Site Navigation: Up to 10 times better extraction performance, as well as lower footprints on sites being extracted.