Here are some insights in the process, the reasons why the infrastructure was changed, and the benefits summary for TYPO3.
History
Before the change, docs.typo3.org was actually a development system for the Documentation Team. It was never meant to be a production environment. The system was cluttered with many different scripts and solutions by different team members, amassed over the years.
Some may recognize this situation and think about the technical complexities. The existing infrastructure made it hard for us to change certain parts of docs.typo3.org. E.g., many 404 Not Found requests were generated once manuals and sections were moved around. Users of the ViewHelper Reference have seen that, especially in December 2018 and January 2019. Those problems were nearly impossible to solve for us as a team, due to the legacy system.
The system also prevented us from using Docker. Right now we provide a Docker Container for rendering. This Container could not be used on production, due to the old system.
There was a need for change.
Requirements
The Documentation Team (Martin Bless and Daniel Siepmann) headed to the headquarters of TYPO3 Inc. and met with Mathias Schreiber and Christian Kuhn. We sat together for a day and discussed the future of docs.typo3.org. During this meeting the requirements were defined, and some ideas noted on how to meet these requirements. The requirements were:
Support Composer
TYPO3 as a project wants to improve the support of Composer in third party extensions.
This is solved by requiring every project (manuals and extensions) to immediately provide a valid composer.json in order to be rendered.
Scalability
docs.typo3.org needs to scale. This point consists of multiple self enclosed requirements. The Documentation Team consists of three people, who already have a lot of work to do. In order to scale, some recurring procedures need to be automatic or easy to solve:
-
Make it easy to auto render documentation.
Now everyone can include a hook in his GitHub, GitLab and Bitbucket repository. Also "custom" hosters are supported. Each push can trigger the rendering via a web hook. If you encounter any issue, inform the Documentation Team.
-
Allow easy previewing of rendered documentation before release.
Some extension authors already know this pain. A new release is done, and there is an issue inside of the Documentation. The authors can now render the documentation locally upfront using the provided Docker Container.
Now they can also push changes to a branch documentation-draft to generate a non indexed preview of the documentation. -
Allow managing of redirects. Now it's possible to add redirects for whole manuals and single URLs through a web UI.
Also larger rewrites can be provided by configuring the nginx web server.
Why do we need to worry about scale? TYPO3 plans to expand in further countries where more people would read and contribute to docs.typo3.org. Also more extensions might be developed, requiring additional support for their documentation..
A Self Documented Environment
We now have a self-documented environment. This is done by using Bamboo and Intercept. Bamboo contains the CI and defines how documentation gets rendered. People can have a look into the bash script and know exactly what's going on.
Intercept, the middleware developed by TYPO3 Inc., is also providing the connection between all systems. The whole source code is open source. Everyone can have a look and provide pull requests.
There is no more hidden source code, or odd cron jobs within a server, hiding knowledge from the community.
Multi-language Support
Translation of manuals already was supported, but rarely worked. This is now solved with a clean new URL structure. Authors are able to provide specific language by locales, e.g., en-US or de-AT, in their documentation.
Make Contributions Easier
Some might want to contribute to Documentation through the rendering chain, deployment, or further logic. This is now possible, as all parts are publicly available and self documented. More contributors can assist with their particular expertise.
Knowing the Contents of docs.typo3.org
This is done by Intercept. Being the single point of truth, Intercept knows about every single request for rendering documentation. Also all results from Bamboo are reported back to Intercept. Thus Intercept is able to know the content of docs.typo3.org. The Documentation Team can delete and re-render any available documentation through a simple web ui.
This web UI is publicly available. Anonymous users won't have the opportunity to trigger some actions, but still can search through all available manuals.
Support Surf and Fluid Documentation
TYPO3 provides not only the CMS and third party extensions, but also Surf as a deployment solution and Fluid as a template engine. Both products need a place to provide their own documentation.
This is now done through the exact same workflows, except they have a custom build plan within Bamboo. Using this system Surf and Fluid ViewHelper References are already rendered.
Later this year, official documentation for Fluid may follow.
Work
It turns out that we had a lot of requirements that were time consuming to solve. Luckily, TYPO3 Inc. could jump in and invested the necessary effort. They blended the existing knowledge of Documentation Team into TYPO3’s current systems. TYPO3 Inc. has handed the system over to the Documentation Team, and it is in full control.
Outcome
We now have a state of the art infrastructure. Every author can add a Webhook to their repository to trigger the rendering of his documentation. Only version tags in the form of Major.Minor.Patch are rendered, together with master Branch and documentation-draft Branch.
The Documentation Team now has a modern web UI to solve typical use cases like adding redirects or investing rendering issues. All requests through webhooks and all responses from Bamboo are logged and are accessible through the web UI.
The new docs.typo3.org also provides a new canonical routing in the form of /type/vendor/name/version/locale, e.g., /p/typo3/form/9.5/en-US.
NOTE: There are no patch level versions available anymore.
Call to Action
In order to allow the rendering for your extension, add a valid composer.json to your project. This has to provide a dependency on typo3/cms-core with supported versions. Add the new webhook to your repository (remove the old one if it exists).
The Documentation Team also welcomes contributions from the entire TYPO3 community. You can help improve our documentation by editing and proofreading existing docs. Check out the previous blog post regarding contributions.
Copyediting and Proofreading: Tony Lush