The hitchhiker’s guide to observability – part 2

In the first installment of this hitchhiker’s guide to observability, we have seen that observability is as powerful as the habit we can develop. It is necessary to rely on a powerful solution such as Blackfire. But here’s the catch: it is not sufficient.

Observability only works if you actually use it. The more you integrate it into your development and deployment process, the more effective it becomes. We have begun exploring a list of short tasks you could squeeze into any agenda to help you get started on the right foot and build an healthy observability happy.

4/ Write a first short test

You have now determined your priorities and are faced with two options. You could decide to start optimizing one of those critical endpoints you identified. That’s a bold move. Go for it! To start, take 5 minutes to read this piece, “Taming the beast: performance optimization of unfamiliar applications with Blackfire”. It might help you on your quest. Even the fearless heroes need allies.

Another option is to secure those endpoints or scripts (yes, CLI commands can also be monitored) by writing performance tests. While waiting for an opportunity to start working on improving them, you can ensure the performance of those critical parts won’t be silently degraded.

You can start by reading the “Getting started with the Blackfire test suite” blog post series. Each entry might take 3 to 5 minutes to complete and is designed to guide you through your first steps with performance testing.

Performance tests should evaluate what makes an endpoint fast rather than how fast it is. Time is volatile. You should consider the number of SQL queries made, the number of entities hydrated, or how many times you go through that greedy function. In short, consider the cause, not the consequences.

You could even start with the most minimalistic test with a single assertion: the SQL query count. Iteratively, and as you get accustomed to the exercise, you will improve and enrich your test. Those are defined into a .blackfire.yaml file whose very first version might look like this:

tests:
    "The homepage should be fast":
        path: "/"
        assertions:
            - "metrics.sql.queries.count <= 5"

5/ Enable synthetic monitoring

Congratulations on implementing your first performance test. The few minutes you invested in it should pay off in hours saved in the future.

Every time profiles are triggered, all the matching assertions are automatically evaluated. The tab “assertions” displays the results of all the performance tests matching the profiled request or CLI command.

As you might not want to frequently manually trigger a profile for each of your application’s critical endpoints, it is worth considering automating a bit more. Synthetic Monitoring lets you describe the critical user journeys to have them automatically profiled periodically or on demand. One trigger to profile them all!

This blog post can guide you through your discovery of Synthetic Monitoring. Scenarios can also be described into your .blackfire.yaml file. As for your first test, done is way better than perfect. Start small and improve over time. We trust you; you got this.

# Read more about writing scenarios at https://docs.blackfire.io/builds-cookbooks/scenarios
scenarios: |
   #!blackfire-player

   scenario
   	name 'My most vital user journey'

   	visit url('/')
   	visit url('/my')
   	visit url('/crital')
   	visit url('/endpoints')

6/ Automate your tests

Let’s continue on this journey and ensure the time you invested so far bears its fruit. You are now in a situation where you can trigger a profile for each endpoint related to your critical user journeys. That’s an achievement you should be proud of. This is referred to as a Blackfire Build.

Yet, at this point, you still have to trigger those builds manually. You can possibly schedule periodic builds that can be triggered every 1, 6, 12, or 24 hours. This allows a periodical check-up on your application performance and early warning if needed.

Even better, you could integrate them with your CI/CD pipeline and/or PaaS, so your tests are evaluated against every pull request and environment created. You could project the impact of changes before they reach production and never ever release a version of your application that degrades its performance.

Take a few more minutes to schedule periodic builds and integrate your tests with your pipelines.

7/ Activate alerting

Let’s continue our effort to make the most out of this group of small 5-minutes tasks we carried-out so far. Those are meant to help you get started on a right and fast track. At the same time, they are meant to be investments for your future. We want future-you to be in the best position possible to face any ordeal.

Let’s invest 5 more minutes in futureproofing your application and processes. Automated performance tests are meant to inform you of any regression introduced in your code. Blackfire Alerting is specifically designed to give you a head start whenever a crisis unfolds. Set up your first alerts and benefit from a more secure application.

42/ Repeat

You have hacked you way into observability by splitting the learning curve into many snackable chunks. You invested in your application and its futures in a way to build a sort of observability passive income. You created simple tests you automated. Slowly, but surely, you increase your grasp over your application’s performance.

Just like that, you figured out the answer to the great question of life, the universe and everything observability. This answer is repetition. The greatest heroes of that realm do not have to battle demons and wildfires.

Here’s to the wise and savvy ones who slowly and steadily created the conditions for never having to fight those battles. They push the world forward. They have more time to build great futures when others keep putting out fires.

To better observability and beyond

The “Hitchhiker’s guide to observability” series:

• part 1
• part 2 and final (you are here)