The service has been up and running since April 2018 and, since then, we’ve created a database of websites running TYPO3. The database includes statistics about which TYPO3 major and patch versions are in use.
In order to analyze and identify trends from the data collected by t3versions, Ronald Meeuwissen from the TYPO3 marketing team kicked off the “Market Intelligence Team” project. To begin the project, the TYPO3 Marketing Team and TYPO3 GmbH defined several goals and milestones. As part of the first milestone, the following tasks have been outlined and accomplished.
View the latest TYPO3 Statistics
Regularly Check Existing Websites
In order to visualize statistics and trends, existing data must be updated on a regular basis. Therefore all existing websites in the database will be checked at least once every three months.
Increase Data Coverage
There are currently two ways a newly launched TYPO3 website gets added to the t3versions database. Either a user manually enters the website domain into www.t3versions.com or it’s added when we perform a mass scan of domain lists (with millions of domains), which we acquire once per year.
We discussed creating an API endpoint that performs TYPO3 version checks as a way of adding new TYPO3 websites to the database, and added this functionality to the first version of our API.
Provide Custom Data Analysis
To identify trends (e.g., increasing or decreasing usage of TYPO3 in a country), the TYPO3 Association and TYPO3 GmbH needed the ability to perform custom queries on the data available for TYPO3 websites. This task was completed by creating an API.
Creating the T3versions API
Developing the t3versions API was the largest piece of work for our first milestone. The API was created using the popular Django REST Framework, which provides many necessary functions, like authentication, permissions, and rate-limiting.
The first version of the API contains endpoints that perform the following actions:
- Check domains
- Queue a given domain to be checked for TYPO3 usage
- Query the result of a queued check
- Get and query database websites
- Get a list of all TYPO3 websites in the database including a limited set of data (domain, major version, last check).
- Use several search filters to query the full set of data (excluding patch level version)
- Site changes
- Use several search filters to query monitored site changes (e.g. site updated TYPO3 version)
All API endpoints require user authentication and specific permissions.
We’re currently working on providing 3rd party services access to endpoints that perform site checks. This will make it possible to automate your own checks for TYPO3 version detection and may also increase the data coverage on t3versions.
Stabilize Version Analysis
While developing the API, we identified the following two problems in the t3versions scan engine:
- When a check discovered that a known TYPO3 website was offline or its domain was unreachable, it did not remove the site from the database.
- When a check got redirected to a non-TYPO3 website from a known website, it did not remove the original site from the database
Both problems have been fixed and a rescan of all available websites resulted in a removal of 51635 TYPO3 websites from the database.
Make Scan Results and Statistics More User-Friendly
When T3versions was created, technical tasks related to the backend took priority over the website’s layout and visual design.
Since completing the first milestone, we replaced the website’s initial CSS with the Bulma CSS Framework. Colors are now more TYPO3-like and the website content—statistics, scan results, etc.—is more readable and user-friendly.
The check result page was also improved, so it’s now clear whether a website is running a community-supported TYPO3 version or not.
Next Steps
The work accomplished in the first milestone of the project is now the basis for upcoming improvements and changes to t3versions. Together with TYPO3 GmbH, we will define and add new trend graphs to the website as soon as usable data is available. We’d also appreciate your feedback. Please feel free to shoot us a message and suggest potential new features—we’ll prioritize them with our stakeholders!
