Spidermon: Scrapinghub’s Now Open Sourced Spider Monitoring Library

Spidermon makes it easy to validate data, monitor spider statistics and send notifications to everyone when things don’t go well in an easy and extensible way. Installing Spidermon is just as straightforward as any other Python library:

Once installed, to use Spidermon in your project, you first need to enable it in the settings.py file:

To start monitoring your spiders with Spidermon the key concepts you need to understand are the Monitor and the MonitorSuite. For example, this monitor will only pass if no items have validation errors:

When something goes wrong with our spiders, we want to be notified (e.g., by e-mail, on Slack, etc) so we can take corrective actions to solve the problem.

Source: blog.scrapinghub.com