Report on the combined Server Admin and typo3.org Team Sprint in Würzburg (November 2016)
November 26, 2016
The combined code sprints of the Server Admin Team and the typo3.org team took place at the University of Würzburg over the weekend from November 11-13. In total, 13 participants worked on concepts and code for the (new) typo3.org web site, as well as the projects server infrastructure.
typo3.org - Conceptional Work
The team reviewed the typo3.org user survey which was conducted some weeks ago. We structured the results into different areas (like design & usability, content, functions) and defined conclusions for each area. One result is that we have to improve the connection between our existing sites, especially between typo3.org and typo3.com. The sites are addressing different target groups, but will still have to share some content (mainly about our products). So we want to tell our stories from different angles:
- typo3.com: Sell the product. Describe product features, case studies and show cases.
- typo3.org: Help to use the product. Inform about the products in a more technical way. Be the central information hub for the TYPO3 universe.
The third step was to define the goals, we want to reach with our site:
- typo3.org should provide its users easy and fast access to informations we want to transport
- typo3.org should be the central information hub for our community
- Increase number of TYPO3 users, community members, contributors and Association members
- typo3.org should represent TYPO3 in a modern and professional way
- typo3.org should be likeable and friendly, like its community is
typo3.org - Technical Setup
The deployment for the new typo3.org web site, as well as other sibling sites will be based on GitLab CI and was set up on https://git-t3o.typo3.org. To gain more flexibility, the TER extension which implements the TYPO3 Extension Repository was extracted from typo3.org and will soon run on a dedicated TYPO3 7.6 installation. The new address for it will be https://extensions.typo3.org.
LDAP and my.typo3.org
The Single Sign-On (SSO) functionality is one central part of our infrastructure. To reduce complexity, the SSO service will be decoupled from typo3.org and will move to my.typo3.org. Additionally, LDAP will be used instead of the current very specific solution. The Server Team and T3O Team created a concept to migrate SSO services smoothly in the background. As a first step, all user data will be stored additionally on the LDAP server. Once the user migration is finished, the LDAP will become the leading system for authentication and all data modifications will be done on my.typo3.org. For this task, the team has started to create the necessary functions:
- OpenLDAP Server running and tested
- ACLs are set and tested
- LDAP groups based on TYPO3 BE groups
- Frontend users can edit their data, which is stored in LDAP, while some features are still missing, e.g. double opt-in, spam control
- Various accounts with different access levels for administrative access have been created
- Mapping of TYPO3 account status »disabled« (during opt-in process or manually) has been implemented
- Process for account migration has been discussed and defined
To provide a smooth migration experience of all accounts to LDAP, methods will hook in the T3O account create/update/delete processes and transfer account data to LDAP. It is planned to send messages to all users to enforce the update and finish the migration until the end of the year. As soon as all accounts have been migrated to LDAP, the LDAP server may be used as full source of accounts/authentication for any other TYPO3 service. The code has been integrated into the extension EXT:ajaxlogin which is already used on typo3.org. Any account creation, update, password change or removal on T3O (Frontend and Backend) will be pushed to LDAP in the background over a secure connection including login credentials. Additionally, a custom extension for LDAP user migrations (EXT:t3o_ldap) was created and tested. It is integrated into the backend with a data handler hook which pushes all updates to LDAP too.
As we use Docker more and more, we automated the installation and configuration of Docker-based machines using Chef.
More and more applications require a SMTP server to send mails directly. We set up an internal mail relay consisting of two servers in different places to send out emails from our different applications trough a central place. While doing this, we also changed our SPF record on typo3.org where we removed our old, include based configuration which did not work as expected on some servers, as we hit some lookup limits.
LEMP Stack for PHP Hosting
While many of the *.typo3.org sub-sites are PHP-based, we unfortunately lack a modern stack for this kind of application. Therefore, time was invested to make a modern Linux/Nginx/MariaDB/PHP setup available for us and the community. We combined our existing Chef cookbooks to configure the base infrastructure (hostnames, DNS, backup, ...) with existing Puppet modules to manage websites. While this seems quite odd at the beginning, it allows us to use the appropriate Chef configuration as its done on all out other servers, while configure standard websites optimized for TYPO3 with existing & established Puppet modules. With this, we are able to provide standard hostings for different subprojects on a defined and current infrastructure. By now, we use Debian Jessie (PHP 5.6). The option to run PHP 7 based projects will follow with Debian Stretch as soon as it is released next year.
KVM "golden image"
Our new server infrastructure is based on KVM machines. By now, we always (automatically) run through the Debian installer to spin up new VMs. To speed up the VM creation process, a complete base image, which can be dumped directly into LVM was created. This can be achieved with the tool diskimage-builder from OpenStack, which we adjusted to our needs. An early preview can be viewed here. It is not yet complete and still needs some skeleton configs for cloud-init. We use docker to build the image, which allows us, to integrate the "build image" task into Jenkins easily.
As we move our services into our new data center infrastructure, we prioritized the Gerrit server based on demands by the development team. The Chef setup for Gerrit was adjusted to the meet new environments and the newer operating system. As the migration tests were successful, this service can be moved to a more powerful server. The migration has been scheduled for Dec 5 and will be announced separately...
The next sprint will be again co-located among both teams. The date has been fixed and the sprint will take place on January 16-18, 2017 (Mon-Wed) in Basel, Switzerland.
We like to thank the University of Würzburg and our valued team member Steffen Gebert for being our host for 3 days, and the TYPO3 Association for covering all travel and accommodation costs.