ownCloud: Recent Performance Optimizations

Editor’s note: This is a guest blog post by Morris Jobke and Jos Poortvliet from ownCloud. In this blog post, they explain how Blackfire helped them improve the performance of various parts of ownCloud.

A little under two weeks ago, Blackfire started to be used by community members working on ownCloud. With over 2 million users and hundreds of developers, ownCloud is the largest open source file sync & share technology. It comes with a powerful extension system for which many apps are available and is written in PHP. A recent performance improvement was accompanied by a graph showing the difference in code execution time. Since that first graph, many more have been created and much code has been optimized.

The first Optimizations

The easy to read and intuitive Blackfire images quickly spread over Github and led to a speedup in the AppStore, thanks to caching the responses from apps.owncloud.com. While the first fetching data from the app store takes a few seconds, subsequent page loads can rely on the cache, making a big perceptible difference for the end user, easily speeding up operations 20 times. The developer making the changes, Lukas, provided a nice graph showing the difference.

Getting Serious

Lukas, who ‘discovered’ Blackfire initially, started to simply upload files and use the profiler to see what took a lot of time. This resulted in a series of changes to file related operations, collected inthis Github issue — starting small with a 4% performance gain on the processing ofnormalizeUnicode, adding a simplification to a check on paths and ending with a performance boost of 24% for simple searches.

Speed-up File Deletion

Another extensive profiling session resulted in even more impressive results after Thomas noticed that deleting a folder with 3,000+ files in it took a huge amount of time to get processed. In total over 35 minutes, showing in the profiler as this graph. It shows clearly that the bottleneck are nearly 40,000 SQL requests which are causing over 70% of the total request time.

In the first iteration the request time went down to 10 minutes by reducing the SQL queries by a factor of 10. Now, another interesting bottleneck showed up: the PHP method strlen was called over 60,000 times by a component from a third party library that ownCloud uses, ZendSearch, from Zend Framework. Focus moved to this library and it became clear that strlenwas used to just decide if a string has a defined length. Somebody came up with the idea of simply using the native language construct isset rather than a function like strlen. Instead of doing this…

if (strlen($name) > 49) {
    ...
}

… you can do this which results in the exact same behavior, except that it is a lot faster:

if (isset($name[49])) {
    ...
}

We provided a patch upstream which reduces the load for this code path greatly.

For the deletion use case, the improvement moved the instant processing to update the search index to a background job. This results in the execution time going down to just 1 minute! There is still room for further improvements but it is a great first step and it took only a moderate amount of effort to achieve. The change from strlen to isset turned out to also be relevant in ownCloud code, bringing a nice 20% increase in speed for processing a million path lookups.

Four Punic Contributions

A debugging session led to finding another performance bottleneck with very large folders and another set of upstream improvements. The first patch going in the Punic localization library wascaching getTimezoneNameNoLocationSpecific, followed by this and this. Familiar by now might be a patch to use isset() instead of strlen()! On the ownCloud side, reusing the array key of mimetypes helped improve performance and the session also found a bug which wassubsequently fixed. More ideas are still coming in while this issue is tackled.

Architectural Impact

Not all performance issues can be relatively easily fixed. Lukas discovered that the customOC_Autoloader in ownCloud is quite a bit slower than the Composer loader we are transitioning to. These loaders bring in functionality from third party libraries ownCloud uses when needed in a piece of code. The OC_Autoloader is the ‘old’ way ownCloud has been handling that and the fact that OC_Autoloader wasn’t fast had been noted before. ownCloud 8 will introduce theComposer loader, a dependency manager strongly inspired by Node’s npm and Ruby’s bundler. Clearly, it not only saves the ownCloud team the work of maintaining their own loader but also offers performance improvements! Moving completely to the Composer loader is already on theagenda for ownCloud 8.1.

Another change coming in ownCloud 8.1 is related to memcache, something Lukas bumped into when going through code with Blackfire. It is a relatively big change but there is already code and some testing has been done. Due to the impact, however, it will have to wait until the next release.

Impact on Reporting of Issues

Blackfire is also starting to be used by users who report performance issues. When asked to run a profiling run, the results prompted a first optimization already and more are in the pipeline.

Nice Things about Blackfire

As most of us here at ownCloud work remotely (as in “not in one city at all”) this is a nice way to share the results with other contributors. Blackfire makes it really easy to inspect all the calls that are made during a request and to spot bottlenecks in our application even if you didn’t run the profiler.

It has an easy to use user interface that just works and the profiler has a minimal setup effort. Running an agent, loading a PHP module and clicking on a browser plugin button. That’s all you need to get some nice graphs with really valuable metrics about your request. Of course we have some wish list items already, like the ability to more easily grab images from the result graphs (or export them to other formats) and a Blackfire Firefox plugin! Last but not least, it would be great if it could do the performance optimizations automatically. For this, integration with Skynet was suggested.

Enthusiasm around the tool is becoming very visible, with a large number of performance related pull requests showing up with Blackfire graphs. We expect more improvements in ownCloud performance thanks to Blackfire – a great tool!

Editor’s note: We are always looking for great stories about how Blackfire helped our users find bottlenecks and improve the performance of their apps. Feel free to contact us if you are interested in sharing your experience with Blackfire.