Crawly is optimized for multi-threaded requests and parsing. While slower pages are loading, faster ones are being parsed.
If your website is anywhere fast, thousands of pages will be parsed within just a few minutes. True parallel parsing.
Parsing results can be used in any way you want. Use one of the existing result formatters or write your one - it's that easy.
For your convenience we prepares a bunch of formatters for common tasks. Exporting site map? Checking for broken links? It's all done.