Report on the TYPO3 Neos Performance Code Sprint February 2014

Categories: Development, Community created by Karsten Dambekalns
In the second week of February 2014 a code sprint focusing on the performance of TYPO3 Neos took place. Hosted by TechDivision in their Lübeck office it brought together eight people for three days of intense work.
Rens Admiraal, Karsten Dambekalns, Dominique Feyer, Sebastian Heuer, Berit Hlubek, Christopher Hlubek, Robert Lemke, Christian Müller had the overall goal of speeding up Neos. This involved some profiling and tests first, before performance improvements were approached from a number of directions. The results are partly merged already as I write this, partly they are still under review. Below the outcome is grouped by product, but keep in mind that any improvement in Flow benefits Neos directly, of course!

Neos improvements

The most promising result in terms of performance improvement is the TypoScript Content Cache for Neos. It is a content cache with nesting support integrated into TypoScript. That means that cache identifiers and tags for flushing can be configured in TypoScript and rendered content will be re-used across documents. If an inner cache entry is flushed, the entry will be re-created as lazy as possible (no need for a complete document re-render). This not only results in the highest possible speedup (first measurements in the demo site show response times < 80ms in Production, further improved by skipping routing and controller invocation with an HTTP component) but also means that caching can be active during editing without side effects (and of course it supports workspaces). The change is still under review. TYPO3CR now has an in-memory first level cache for Nodes, eliminating queries to the ORM for nodes that have already been fetched.
The FlowQuery "children" operation will now optimize some common filters internally to reduce the amount of nodes fetched and to produce better queries.
The Menu implementation was improved and now correctly calculates item states alongside some new features (see change for details). the use of ExtJS was dropped long ago, Neos was still using Ext.Direct for communication with the server. The removal of that dependency was on the agenda for a long time now. During the code sprint we managed to have a replacement change ready for review. It is a start on further optimizations to implement a RESTful API in Neos.
The site export in Neos works quite fast even for huge sites, but importing is slowing down to a crawl for a lot of nodes. This is improved by directly talking to the database without involving the ORM layer. The change is still under review (and right now would break importing for PostgreSQL users due to a PHP bug). really performance related is the support for content dimensions. We decided to work on it anyway, as it changes a lot in the internals of TYPO3CR - and further performance improvements are easier to be based on the new code. Another result of working on content dimensions during the performance sprint was that even though there is a lot of new code and complexity involved, there is almost no performance penalty coming with the new feature. Focus FTW!

Flow improvements

When using Doctrine one can now use the Doctrine query result cache on a per query basis. This can help speed up applications. While this is not used in Neos so far, it can be enabled on a global basis. Combined with the TransientMemoryBackend this results in a per-request query cache that can speed up Neos - depending on the site structure and size, YMMV. The class loader now performs faster and avoids lookups through the use of a map with known proxy classes. On top of that, it now supports all autloading mechanisms supported in Composer (PSR-0, PSR-4, Classmap, Files). Who'd have thought, eh. The loading of YAML files can be sped up by installing the php-yaml extension - if it is detected Flow will use it instead of the Symfony YAML parser. The only drawback: the extension is a lot stricter on the YAML syntax, so you might need to fix your configuration files… The Redis cache backend was more or less rewritten from scratch (still under review). The new backend also implements IterableBackendInterface and FreezableBackendInterface, which allows usage for various caches (like Flow_Session_* or Flow_Reflection_*) that could not use it so far.

What else?

The Plumber (and PhpProfiler) tools were finally tagged and are now installable via Composer without the need for a dev-master version. A node generator tool for Neos was created, to produce test data of various sizes and characteristics. It can be used by anyone who needs to fill Neos with loads of data - as in the (experimental) performance testing distribution started by Dominique.
In the TYPO3 Media package we discovered some pretty useless code that leads to the image file being loaded whenever media is used - which of course can have a huge performance impact. A fix is under review.

Thanks to our sponsors!