Benchmarking and tuning the parts
Baseline data
In this section we discuss various factors that can influence our website's performance. We introduce some commonly used tuning parameters and see whether they do effect the performance. The chapters order goes from low-level to high-level tuning. If you think one factor is not relevant for you (because you can't change it, or you know it's not a bottleneck), simply skip the section and move on.
We started our benchmarking with a worst-case scenario: A freshly installed Debian Linux server, with a default Apache1 and MySQL configuration, mod_php4 and a fresh 3.8 Quickstart package. Because we knew we should be optimizing for less-than-optimal situations, we did choose to benchmark with TYPO3 caching turned off, just as a baseline, so we could be sure that real-world performance would probably be better. For all testing stages, we used a second host as a client, which was located on the same 100MBit switched network as the server.
In order to have a comparable value, our first test was to request a 24 kByte sized static file (so TYPO3 was not involved at this time) from the server. This gave us an approximation of the maximum speed of our server.
We fired up Apachebench with 1000 requests and a concurrency of 100. Note that the concurrency was the actual crucial, the number of requests was just set to a higher number so that our averages were more robust.
$ab -n 1000 -c 100 yourserver/static%20file.txt _blank>http://yourserver/static_file.txt
The average time of one request was about 3ms. That meant more than 300 concurrent requests per second (300 Req/s).
The next step was to fetch some TYPO3 pages:
$ab -n 1000 -c 100 yourserver/index.php;http://yourserver/index.php?no_cache=1
Be warned before you try to play with this kind of test: Chances are your webserver will get desperately overloaded after a few seconds, "DOSing" yourself.
The result in our environment was, well, somewhat disappointing 4 Req/s. Compared to the static test, that was nothing. Time for some tuning.
Operating System
The choice of operating systems does in many cases make a difference in performance. Some popular benchmarking comparisons stressed the fact that,
Linux 2.6 performs better than 2.4, for both Apache and MySQL. See
www.2cpu.com/articles/98_1.html and www-128.ibm.com/developerworks/linux/library/l-web26/Linux scales better than *BSD, not to mention Windows or Solaris. See
bulk.fefe.de/scalability/
Many of these benchmarks were either low-level or made on extreme hardware (heavy SMP machines), so we checked if simply replacing the stock 2.4.x kernel with 2.6.14.x and activating Native Posix Threads Library (NPTL) did enhance performance: No, it did not significantly. The overall effect was hard to measure and not even stable, still with a result of 4 Req/s.
Conclusion: Does that mean tweaking the OS is useless? No, but in our case threading and scheduling were simply not the bottleneck. If you find out that spawning threads and processes is actually bad on your system, feel free to tune and change it. Especially if you use Apache2, NPTL is a must have to use threads instead of processes. Compared to processes, threads significantly reduce the load Apache2 causes on your machine. But anyway, this does not mean you would automatically serve your pages faster.
One more hint on kernel stuff: In one of our tests, we somehow managed to synflood the network stack of the 2.6 kernel. The result was that incoming TCP connections were dropped. Since the test results within that synflood were totally unreliable, we advise you to check your kernel logs for TCP drops from time to time.
Database - tuning MySQL
A frequently suspected cause for bad performance is the database. We know TYPO3 is pretty DB-intensive. It does multiple queries on each and every request, like getting data from the cache, updating sessions, etc. So the database is heavily used. MySQL optimization is a science by itself, and there are a lot of myths and useful recipes floating around on how to tune it. Note that there's a detailed chapter in the official <link dev.mysql.com/doc/refman/4.1/en/optimization.html>MySQL documentation</link> about that topic. We mainly concentrated on tuning the <link dev.mysql.com/doc/refman/4.1/en/server-parameters.html>server parameters</link> by altering the configuration and measuring the effect:
We raised max_connections to more than 100 - since we were using (because it is default) persistent connections, the number roughly corresponded with the number of MaxClients in Apache (see below). Setting this number up did not cost performance and gave us some reserves for additional requests.
A next step was to tweak query_cache_size, query_cache_limit, key_buffer_size and table_cache to higher values, since we had enough spare RAM - this seemed to be quite useful for tuning because TYPO3 does a lot of repetitive queries, so
caching queries should work well.Then we switched off the bin-log, because we did not use replication and our backup strategy was not based on logging every single write query.
Although tweaking I/O for the DB (by using a dedicated RAID, a different file system, or noatime options) was even mentioned in the
TYPO3 wiki, we did not spend time on that. The reason was, that we had enough RAM to do memory caching, which was generally faster than disk access.
After tuning our database, the result was, again, nothing significantly changed: 4 Req/s. Checking the MySQL stats (using mysqladmin status), we saw that caching worked quite well, so where was the problem?
It turned out that the database was simply not the bottleneck in our setup, for several reasons:
In a setup with one website, we did not have that many simultaneous connections, so spawning threads for queries is not a problem. This would probably change once we serve several websites.
The Quickstart site served (mostly) static content, which was perfectly MySQL-cacheable. This would probably change once the amount of dynamic content grows.
Conclusion: MySQL tuning was not very useful in our case, but could help in other cases. The impact of the database grows with the number of sites and the amount of dynamic, non-cacheable content you serve.
Webserver - Apache and alternatives
The next of the usual suspects was the webserver. For a number of reasons, Apache is the standard web server for most TYPO3 installations, therefore we concentrated mainly on tuning and comparing the Apache 1.3 and 2.0 series. Additionally, we had a look at the relatively new lightweight webserver <link www.lighttpd.net/>lighttpd</link> which seemed to be the first of the smaller server that had all the features needed for TYPO3 hosting.
Tuning Apache 1 for TYPO3 was not really rocket science, because we were mainly fiddling with the parameters related to forking new processes with requests. The main issue was the number of MaxClients, or maximum child processes spawned. Before any tuning, the default number of MaxClients was 150. This did not cause problems when serving only static files. But once TYPO3 was involved, each of those Apache processes included mod_php and code execution, so they did consume quite an amount of CPU and memory (remember that 32MB memory limit you set in php.ini?). Our machine was not powerful enough to handle that amount of simultaneous requests and all server power was sucked up in seconds. The solution was to decrease that value, so that less processes were forked and the host was again able to breathe. The drawback was that we could not accommodate all requests at the same time if there were more parallel requests than MaxClients. They did have to wait in the queue and, in the worst case, timed out.
So we had a trade-off between being nice to everybody (and spawn processes like hell) and risk that our server burns in no time, or limit the number of processes and decrease load and request time for the accepted connections. This required some adjustment and trial and error. However, we found that, at this stage of our tests, we could do almost as many requests per second with MaxClients 32 than with 64, with considerably lower load (so that our ssh session still reacted without much delay).
Another solution was to use another process model, introduced with Apache2, that did not spawn expensive processes but cheap threads. This "worker" module is <link www.zeuscat.com/andrew/work/aprbench/>reported</link> to be significantly less resource hungry and scalable. However, it was not marked stable with mod_php at the time of our research and could not even been installed without much hassle on Debian. Therefore, we only tested the Apache2 "mpm-prefork" module (without threading), which unsurprisingly gave approximately the same performance as plain Apache1.
Another option, similar to the Apache2 worker approach, was to use a lightweight non-forking webserver. We knew that Apache1 was not the fastest option for pure static file throughput. There is a long <link litespeedtech.com/benchmark.html>list of small webservers</link>, both free and commercial, that outperform Apache1 in this discipline. The trouble was we were not serving only static files, and most of those servers had bad support for dynamic webpages. One of the better-looking alternatives is lighttpd, which fully supports the <link www.fastcgi.com/>FastCGI</link> interface for long-running PHP processes.
We compared lighttpd+FastCGI+PHP with Apache+mod_php. The results were the following:
Lighttpd did a substantially better job than Apache when serving static files.
Lighttpd did, however, not perform better than Apache when serving TYPO3 sites. In fact, it was a little slower (serving 20 Req/s compared to Apaches 23 Req/s, both with eAccelerator running). We couldn't confirm the 20 percent speed increase reported by others, maybe because TYPO3 was not simply outputting phpinfo().
Lighttpd, in our configuration, caused a lot less load on the server when serving TYPO3. The four (!) FastCGI-processes did almost the same work using less resources.
At a first glance, lighttpd seems to be a good option because of its lean design, neat configuration and scalability (e.g. the distribution of FastCGI processes on different machines and built-in loadbalancing).
Conclusion: Be careful when configuring Apache if you expect high peak load. Adjust the value of MaxClients for your personal needs by trial-and-error. Have a look at lighttpd if you like smaller tools and are able to live without the modules and support Apache offers.
There were a lot of other parameters that we could have tweaked to enhance Apache's performance, like disabling AllowOverride, HostnameLookups or access logs, compress output. But again, since we were mostly dealing with relatively few, but expensive dynamic page requests, these issues would effect our performance only marginally. Just like MySQL - tune if you like, it may be worthwhile if you grow really big. For further details on Apache performance, we recommend two articles, one on the official <link httpd.apache.org/docs/1.3/misc/perf-tuning.html>Apache website</link> and another at <link www.xs4all.nl/%7Ethomas/apachecon/PerformanceTuning.html>http://www.xs4all.nl/~thomas/apachecon/PerformanceTuning.html</link>.
PHP
The next candidate for optimization was PHP, and a very suitable candidate it was. In fact, PHP optimization was so trivial and effective that we hardly dare to tell you: We installed a PHP accelerator, such as eAccelerator or Zend Optimizer, and got huge speed improvements with TYPO3. The reason for this was that TYPO3 generally does a lot of computation, code lookups and expensive calls, so that caching the byte-code payed off really well.
At the time of our research, there were at least four <link meta.wikimedia.org/wiki/PHP%20caching%20and%20optimization>acceleration tools</link> for PHP. Since our aim is not to compare competing products, but show the impact of PHP acceleration, we concentrated on one product. Our choice was eAccelerator, simply because it's was free, GPL licensed and available as a <link typo3.sunsite.dk/software/debian/>Debian package</link>.
First of all, let's have a look on how eAccelerator works in principle. Installed as a PHP module, it gets activated and configured in php.ini. There are two major components to speed up the execution of PHP scripts: Code optimization and caching. eAccelerator first analyses the compiled bytecode and then tries to optimize it for speed enhancements. After that, it gets cached as an object in the shared memory and/or on your hard disk. Once cached, far less operations (and thus time) are needed to execute the script code. With regard to the amount of code in TYPO3, it's easy to understand, why eAccelerator comes with an enormous performance boost.
Now let's have a look at it in practice. Our first step was to find out, which component had more impact on performance. Both optimization and caching work independently and can be turned on/off in php.ini:
; Turn on caching
eaccelerator.enable=1
We had two tests, in each case turning on only one of both. The result was clear, optimization did not significantly speed up page generation, but caching did. Simply activating caching without any further tweaking gave us 21 Req/s for FC Bigfeet, that is five times the performance. Although we could not measure changes when using only bytecode optimization, combining both seemed to be the best practice. Of course, the improvement might vary with the complexity of your TYPO3 site, but since there are practically no drawbacks, using a PHP accelerator is a must for every TYPO3 site.
Since caching was so useful, we had a closer look on memory usage. We used a simple usage statistic tool (eaccelerator.php), which comes along with eAccelerator. Among other things, it provided information about the allocated memory, the number of cached files plus the amount of memory each cached file needed. Our question then was how much memory did eAccelerator consume with TYPO3. The goal was to provide a basic rule. We quickly clicked through all the FE pages of FC Bigfeet and got a total amount of 8 Mbyte. Rushing through the BE, the amount increased to 26 Mbyte. Additionally testing some other websites on the server, we found out the following:
The more extensions and websites we had, the more memory we needed to spend.
To find out a good value of how much memory to spend, we had to test our requirements over and over. The more extensions we added, the more memory we needed to share. In most cases, it was possible to measure the total amount of needed memory for a website. As a rule, 32 MByte for one average site was a minimum:
eaccelerator.shm_size="32"
As we had more than one website, we could minimize the memory consumption by using the same source directory for all sites. This is a must for multihosting server!
Reaching the limit of available memory, we found two solutions:
Remove less frequently used scripts from the cache by short time to live (TTL) values:
eaccelerator.shm_ttl="1800"
Turn off eAccelerator for the BE, by adding the following line in ./typo3/.htaccess:
php_flag eaccelerator.enable 0
Although the eAccelerator website did miss some user docs in its documentation section, the software is shipped with a README file, where we found all necessary information about how to install, configure and use it. On the top of it all, eAccelerator came along with an API to integrate accelerating functions into ones own scripts.
Conclusion: A PHP accelerator is a must have. With eAccelerator, the load on our server decreased significantly because less script parsing and compilation was involved. So we could raise Apache's MaxClients back to 64, a nice side effect.
Notice that low-level tweaking (Linux Kernel, MySQL) did not improve the performance of the Quickstart site, but remember that its page contents are basically static. The more you increase the share of dynamic content in your website, the more the database could become a limiting factor. Caching was the most prominent tuning candidate: One one hand, by using eAccelerator to cache precompiled PHP bytecode, speeding up the execution of PHP scripts. On the other hand by using the caching ability of TYPO3, storing once rendered pages as HTML in the database.
Fortunately, most of the TYPO3 sites we know serve mostly static content, so the case of a completely uncacheable site is quite rare. In fact, caching increases performance so much that, unless you need real-time user-specific content, even extremely short caching intervals pay off immediately.
Finally, here come our recommendations: