Crawling a page with Blackfire Player

By Grégoire Pineau, on Mar 06, 2017

In January, we released the new version of the Blackfire Player. Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses. This blog series gives concrete examples of how the Player features can be used.

It integrates with Blackfire in order to automate performance testing, but can also cover a lot more use cases, without necessarily involving any sort of performance management.

Let’s have a look at a use case on the crawling capacities of Blackfire Player. With the simple DSL below, we can crawl each Symfony version tagged on GitHub, and return the results in a JSON file.


     endpoint ""
     set releases []
     set github_tags_url ""

     # get the latest Symfony releases
     while github_tags_url
         visit github_tags_url
             wait 200
             expect status_code() == 200
             set releases merge(releases, json('[*].name'))
             set github_tags_url regex('/<([^>]+)>; rel="next"/', header('Link'))

Executed by this:

blackfire-player run github.bkf --json

Gives this:

     "1": {
         "releases": [
             # not showing all values for the sake of readability in this article
         "github_tags_url": null


Grégoire Pineau

Grégoire worked as a (Symfony-certified!) developer at He's also involved in Symfony development and several other Open Source projects, and is the Swiss Army Knife of DevOps. It would only be him, most of the projects he worked on would be named after savanna animals.