Report on the Server Admin Team sprint in Stuttgart (March 2016)

The location for this sprint was sponsored by <link http: www.schommer-media.de _blank>Schommer Media and we are thankful for sharing their desks with us in their office, located in the heart of Stuttgart. In total, 10 people from the TYPO3 community participated in the sprint. During the sprint, the following topics were covered, of course, besides many smaller cleanups and could you just fix this and that thing tasks:

By now, the registrars of our own domains like typo3.org or t3con.eu where spread amongst many different agencies. This made it especially difficult to change settings like ownership names or DNS servers over all domains. Therefore, we decided to move all domains to our own reseller account and use our own DNS server everywhere as well. Beside two edge cases with special TLDs, we could finish the whole migration at this sprint. If you're in charge of other TYPO3 related domains which should be registered trough the association, please send us their transfer code as well as the DNS zone.

The vast majority of TYPO3 infrastructure runs in OpenVZ-based containers. Unfortunately, this project is no longer a valid option, as recent distributions dropped support for it. As a replacement, it was decided during the last sprint to move towards <link http: www.linux-kvm.org page main_page _blank>KVM virtualization. Further, it was decided to take this fundamental change in the infrastructure also as a reason to change one particular architectural aspect of our infrastructure: We will move all servers to one data center. Until now, our five physical servers are sponsored by different TYPO3 agencies, which is on the one hand a wonderful gesture of our open source community, on the other hand a big blocker for moving a VM from one machine to another or when sensitive data is sent unencrypted and unauthenticated over the Internet (which we don't do..).

Given these two architectural changes, many discussions and preparations happened in the meantime, so that we were now able to provision this new setup to two (new) machines. In the following, we briefly summarize its main building blocks:

• KVM for virtualization.
• Proxy server (nginx + haproxy) with HTTPS-offloading in front of all services. No application server will have a public IP.
• <link https: www.pfsense.org _blank>pfSense firewall appliance for outbound traffic, as well as for admins that need VPN access into the internal network.
• <link https: www.tinc-vpn.org _blank>Tinc-based site-to-site VPN between old and new physical servers, to benefit early from the private network that we build up, i.e. the LDAP server port will not be exposed to the public.

Both proxies and firewalls are running on two different servers redundantly. Incoming traffic is distributed using DNS load balancing between proxies, which also helps to overcome single node failures. Redundancy of outbound connections is provided by pfsense's <link https: en.wikipedia.org wiki common_address_redundancy_protocol _blank>CARP implementation (first hop redundancy). We hope to finish more and more hosts to that new infrastructure in the upcoming months. Until then, we thankfully acknowledge the sponsoring of our servers by <link https: punkt.de _blank>punkt.de, <link http: www.speedpartner.de _blank>Speedpartner/<link http: www.arrabiata.de _blank>Arrabiata, <link https: jweiland.net _blank>Jochen Weiland, and <link http: www.dkd.de _blank>DKD.

During the almost 4.5 years since we started using <link https: chef.io _blank>Chef for provisioning of servers, we learned a lot. One big issue was the shift between a cookbook-centric to an application-centric paradigm (librarian-chef vs. <link http: berkshelf.com _blank>berkshelf), which the complete community made and so did we. Many refactorings happened in the past months so that we can proudly state that we are finally finished with this transition. This, includes that every cookbook that sets up a certain application, e.g., Gerrit, Redmine, OTRS and was successful locally (using the <link http: kitchen.ci _blank>test-kitchen tool), also works in production context without running into tens of potential side-effects, e.g., contradictory version constraints. To synchronize all of us, we had a discussion of lessons learned and guide lines for how to work with Chef, based on a (living) <link https: github.com typo3-cookbooks site-skeletontypo3org tree develop presentation _blank>slide set.

Further, steps to build up a continuous integration pipeline for Chef cookbooks using Jenkins were undertaken.Therefore, we started working on a Jenkins Server that automatically tests and uploads our cookbooks to the Chef server. In order to let others participate in the development of the Jenkins infrastructure, we kick-started a Vagrant environment that can be found <link https: github.com typo3-infrastructure chef-jenkins _blank>here - do not hesitate to open up pull requests if you want to join the development.

The idea behind our Jenkins setup is that it automatically scans the <link https: github.com typo3-cookbooks _blank>TYPO3-cookbooks organization on Github and creates a build pipeline for each of the cookbooks automatically. We therefore use two Jenkins plugins: the <link https: wiki.jenkins-ci.org display jenkins _blank>Jobs DSL Plugin and the <link https: wiki.jenkins-ci.org display jenkins _blank>Pipeline Plugin. The cookbooks will be built, syntax checked, integration tested with Test-Kitchen and then uploaded to the Chef server.

Take a look at the <link https: github.com typo3-infrastructure chef-jenkins blob master readme.md _blank>README file of our Jenkins repository to see where we want to get and what the current issues are.

TYPO3.org user accounts are widespread across multiple systems nowadays. A fact that makes managing accounts and adding new account for consumers like services, websites or unix tools harder than necessary. Our goal was to prepare a central LDAP service infrastructure to be used as global authentication service and user repository - and to import all existing users and groups.

Based on previous research and evaluation, we discussed various aspects and options of every package available and decided to use OpenLDAP as the LDAP server in the end. We then started to create and test a replicated setup that could be easily build with Chef cookbooks and is designed to provide failover services by a mirror replication setup load-balanced through HA Proxy.

When the proof of concept, built of three virtual machines replicating the directory to each other, worked reliable, we started to code migration scripts and successfully tested an import of all user accounts of TYPO3.org in the end. TYPO3-specific attributes have been added to the underlying LDAP standard schemas of each account. The LDAP tree structure is designed from an organizational view of TYPO3 with options to include subtrees for typo3.org users, association members or websites like the TYPO3 shop or T3CON sites.

Next steps are to finish the Chef Cookbooks, refactor the import script and provide a web based tool for account management and self-service. We expect the service as well as the management tool to be ready for production till May 2016.

Good news: DOORS opened and the time of "Waiting for the S..erver" is over. The current documentation server has a very old system and is mainly manually maintained. It's more a "proof of concept" than a real production server. Though it has become already indispensable for most developers. But now the time has come for a move. We, the Documentation Team, are aiming at a new server with an up to date Debian and a fresh and clean setup. We've made already plans and Ansible playbooks for automated provisioning. As the Server Team is busy with reorganizing the server infrastructure this sprint is the perfect moment to coordinate everything and make the new machine for documentation reality. We achieved that! Plus: It's a time of intense communication about how to do things best: We did that! With many stimulation and inispring ideas of how to do things better. "Paving the way" was the plan for the sprint, and we achieved that. The real work can start now!

We started to implement a Chef cookbook to provision PHP-based web servers. This cookbook should install and configure the complete PHP Stack including Apache2, PHP and MySQL. This is still work in progress and will be finished in the next weeks. If you want to participate you can create a PR <link https: github.com obi12341 chef-php _blank>here.

Final Words

Besides socializing and the bummer that there is no breakfast in Stuttgart city on Sunday morning, we were happy that  - Markus Blaschke and Florian Tatzel, both from Stuttgart, joined us for an afternoon and presented their <link https: webdevops.io projects typo3-docker-boilerplate _blank>TYPO3 Boilerplate to us.

To summarize, we are happy to see that several legacy components (OpenVZ, lack of central user directory) were tackled and - once finished - will provide the TYPO3 project and us a more robust and flexible infrastructure. If you want to join our activities, just contact us in the #typo3-server-team channel on Slack and maybe also join our next sprint, which will happen from May 20th-22nd.

The costs of this sprint were covered by the TYPO3 Association.