crawler
  1. crawler

In Scope

Summary

Issues: Unresolved

Key Summary Due Date
New Feature CRAWLER-20 URL priorization
Improvement CRAWLER-21 Additional page parameters in LinkGraph
New Feature CRAWLER-22 Support robots.txt

View Issues

Issues: Updated recently

Key Summary Updated
Improvement CRAWLER-26 Enhance the command line tool to support http proxy authentication
New Feature CRAWLER-25 Event system with information about the crawling process
Task CRAWLER-24 Lucene example: Increment indexing by caching page information (last modified header)

View Issues