Quick and effortless parsing

Crawly is optimized for multi-threaded requests and parsing. While slower pages are loading, faster ones are being parsed.

If your website is anywhere fast, thousands of pages will be parsed within just a few minutes. True parallel parsing.

Multiple export formatters

Parsing results can be used in any way you want. Use one of the existing result formatters or write your one - it's that easy.

For your convenience we prepares a bunch of formatters for common tasks. Exporting site map? Checking for broken links? It's all done.

Clean code

First thing you want and last one you expect when using third party library is a clean, well structured and easy-to-understand code.

We tried our best to keep the code simple, self-documenting and easy to read.

Fast parsing

Crawly analyses average page size and adjusts parsing speed to stay within memory limits and work fast.

Simultaneous parsing and parsed content analyse allows for getting things done quick.

Extendible and customisable

Use crawly standalone via example shell script, within your application as a composer library or via public API.

Every approach fits different needs. Pick the one that fits best for your use case.

Formatters overview

Site map exporter

Formatters, made to create website map in XML or HTML format will create a valid site map for your website or any online service.

Sections analyser

Websites naturally have sections - news, articles, products. Use our formatter to find out the list of sections and their contents.

Broken links finder

Broken links are no good for SEO. Your users click them, see 404 page and leave. Find and fix them with one of our default formatter.

Page weight distribution

Formatters, made to create website map in XML or HTML format will create a valid site map for your website or any online service.

Write your own

Formatters have clear input values and a lot of different examples. Creating one for your personal needs should be pretty easy!

Contribute

Please contribute any formatter you write for your own needs - it will likely be helpful to others!

Want to contribute?

New formatters, bugfixes and improvements are very welcome.