Getting started with the Blackfire test suite: part 5 of a series

That’s right—it’s time for the fifth and final part of our Blackfire test suite series. 

Throughout this journey in performance testing, we went from writing our first test, then our first metrics, to defining the scenarios of the critical user journeys and evaluating them with Blackfire Builds. That’s quite an achievement!

In the third part, I mentioned writing custom metrics and using automation as the Jedi way of using Blackfire. Does this mean we are Jedi yet? Well, not just yet. 

There is one more thing to master: integrating performance tests with your CI/CD pipelines. 

Periodic Builds constitute a unique Synthetic Monitoring technique. They allow you to regularly simulate a given set of actions or path that an end-user would take on your site. Each scenario step triggering back-end code execution automatically generates a profile of that code. Each profile is being tested against your custom assertions as well as Blackfire recommendations.

We’ve seen in part 4 that we can trigger Builds with Blackfire Player. Blackfire Player is a convenient way to work with local and development environments. This final article will show you how to schedule and configure Blackfire Builds directly.

Triggering Builds with the UI

The simplest way to trigger a build is by … clicking the “Start a Build” button.  Yes, indeed. Provided the defined endpoint is reachable by Blackfire servers, all the URLs listed at all the steps of every scenario will be profiled, at the matching assertions evaluated. Easy peasy.

That’s super cool! Let’s add incrementally more automation. Let’s schedule periodic builds as a first step in our testing automation journey. To do so, click the configuration icon (marker #2) in the picture above. We can then update the schedule of those builds (1h, 6h, 12h, or 24h), or cancel them altogether.

With Periodic Builds in place, we are now ensuring the performance of our application is matching our requirements multiple times per day. This is complementary to Blackfire Monitoring as we evaluate our critical user journeys as a whole when Live Monitoring provides more granular insights.

Programmatically trigger Builds

The next step towards Blackfire Builds automation is to trigger them with Blackfire CLI or a cURL request. In any case, the requirement is to generate a specific build token from the UI.

Then, a new build can be triggered using the blackfire build-trigger  command or cURL. In both cases, we have the possibility to override the default endpoint, and we need to provide the UUID of the Blackfire Environment in which the builds report has to be sent to, and the token allowing the operation. A specific title can also be provided to explain the context in which that build was triggered.

blackfire build-trigger <ENDPOINT> --env=<ENV-UUID> --token=<TOKEN-VALUE> --title=<A TITLE FOR YOUR BUILD>

curl -X POST https://blackfire.io/api/v2/builds/env/<ENV-UUID>/webhook \
    --user "<TOKEN-VALUE>" \
    -d "endpoint=<ENDPOINT>" \
    -d "title=Build title"

The documentation provides more information for all different options. There is notably the possibility to define a parent build ID, which allows for building comparison functions when writing assertions. Yes, you got that right. You can even write assertions evaluating the evolution of measurements between two builds, therefore controlling the change performance in time. How powerful is that?

"percent(main.wall_time) < 10%"
"diff(metrics.sql.queries.count) < 2"

Custom Integration with CI/CD pipelines

Surely, you already know your way to integrate the blackfire build-trigger  command (or a cURL request) as an extra step in your already configured deployment pipeline. You could then block the deployment or prevent a pull request from being merged.

We also have integrations with GitHub Actions, Travis CI, Jenkins, and Platform.sh. For the last one there, the documentation on Platform.sh integration provides the command to trigger once to set the integration with your Platform.sh project. Select the accurate Blackfire Environment with the drop-down menu to have the credentials and UUID.

platform integration:add --type=webhook --url='https://********:********@blackfire.io/api/v2/builds/env/<ENV_UUID>/platformsh'

You will be prompted to answer a couple of questions to define which events to listen to, which states (pending, in_progress, or complete) to act on, which Platform.sh environments to include, or to exclude.

Here is an example of the outcome after configuring the integration for all project environments. A build will be triggered whenever we push to an environment, merge, or redeploy it:

As always, let’s test the configuration by pushing code to an environment of that project. A Build notice immediately appears, proving the integration is working as expected.

GitHub Actions

Last but not least, let’s have a look at the integration with GitHub Actions. The documentation provides multiple examples of jobs using Blackfire.

Do not forget to define some secret variables in your project settings to set up Blackfire on GitHub Actions. This example may get you started:

name: Blackfire.io


on: push


jobs:
    setup:
        name: Setup build
        runs-on: ubuntu-latest
        outputs:
            url: '${{ steps.await-env.outputs.env_url }}'
            name: '${{ steps.await-env.outputs.integration_name }}'
            status: '${{ steps.await-env.outputs.commit_status }}'
            cases: '${{ steps.get-tests.outputs.cases }}'
        steps:
            - uses: actions/checkout@v2
            - name: Wait for deployment to complete
              id: await-env
              uses: ./.github/actions/platformsh-wait
              with:
                  github-token: '${{ secrets.GITHUB_TOKEN }}'


    build:
        runs-on: ubuntu-latest
        name: Run build
        needs: setup
        steps:
            - uses: actions/checkout@v2
            - name: Setup Blackfire via setup-php Action
              uses: shivammathur/setup-php@v2
              with:
                  extensions: 'blackfire, :xdebug'
                  tools: 'blackfire, blackfire-player'
              env:
                  BLACKFIRE_SERVER_ID: '${{ secrets.BLACKFIRE_SERVER_ID }}'
                  BLACKFIRE_SERVER_TOKEN: '${{ secrets.BLACKFIRE_SERVER_TOKEN }}'
                  BLACKFIRE_CLIENT_ID: '${{ secrets.BLACKFIRE_CLIENT_ID }}'
                  BLACKFIRE_CLIENT_TOKEN: '${{ secrets.BLACKFIRE_CLIENT_TOKEN }}'
            - name: List vars
              run: printenv
            - name: Trigger a Blackfire Build
              run: blackfire build-trigger ${{ needs.setup.outputs.url }} --env=${{ secrets.BLACKFIRE_ENV_UUID }} --token=${{ secrets.BLACKFIRE_BUILD_TOKEN }}
              # --title --external-id (commit hash, for example) --external-parent-id (parent commit hash?) --external-url (pr url?)

Now, you should be as close as possible to becoming a Blackfire Tests Jedi Master. You are now in a situation where you can fully test the performances of your applications, write custom metrics, and make sure all the critical user journeys are under tight control.

I’m looking forward to reading your performance testing stories. 

Find us on Reddit and Twitter, and share with the community your take on performance testing. We’ll be happy to continue that conversation with you. Kudos to all Jedis and Padawans.

Happy Performance Testing!

Read the full series:

• Part 1 – your first test
• Part 2 – custom metrics
• Part 3 – your first scenario
• Part 4 – in-depth scenarios
• Part 5 – automation and CI/CD pipelines (you are here)