I was initially hired by a company in California to create a web scraper for them that scraped a single website for contact information. That first version can be seen below.



The client was so pleased with the speed of the multi-threaded scraper, its ease of use, and its accuracy that they hired me to do more than 70 additional pages.

To do this, a plugin engine was created for the scraper to allow sites to be added by simply dragging-and-dropping a plugin for the particular site.

It retained all the same great features as the first version--singleton execution, error reporting, multi-threaded scraping, high accuracy--and added the ability to select which sites were to be scraped and to add new sites in seconds.

Selecting plugins is a breeze:



Once your plugins are selected, you get 2 progress bars instead of the single progress bar shown in the original. One shows which plugin is being processed, the other shows the progress on that plugin.

In addition to the plugins for the scraper, there was an additional project awarded to create a site that allowed granular searching of the scraped data, and the attractive display of the search results.