BigDataTraining.IN OpenCrawler Platform

We are excited to announce our OpenCrawler Platform. With collective efforts the platform pushes the workload to the OpenCrawler agents that runs on volunteer machines, thus updating our HBase Server. At present we are crawling at the rate of 7 mn web pages per day.

We developed this to support the Proof of Concept Project post Training, to deal with internet scale data problems, which requires heavy crawling, which needs a collective effort – OpenCrawler Platform makes it possible.

Be a part of this OpenCrawler Initiative,  mail to to request early developer beta access.