Direct Mail

Copyright © by Ivan Kartolo <ivan.kartolo@no spam pleasedkd.de>
Published under the Open Content License available from http://www.opencontent.org/opl.shtml

Table Of Contents

• 1. Direct Mail
  • 1.1. Extension Manuals
  • 1.2. Introduction
    • What does it do?
  • 1.3. The Direct Mail extension is an advanced newsletter mailer system with sophisticated options for personalization of emails including response statistics.
  • 1.4. This extension handles the backend generation and mailing of newsletters, but does not provide any subscription tool. It is intended to be use in conjunction with frontend subscription plugins such as Direct Mail Subscription (direct_mail_subscription), Front End User Registration (sr_feuser_register) and Email Address Subscription (sr_email_subscribe).
    • Requirements
  • 1.5. Extension Direct Mail (extension key: direct_mail) requires TYPO3 4.0.0+ and PHP 4.1.0+.
  • 1.6. Version 1.1.0 is the last version of this extension that will work with TYPO3 3.8.
  • 1.7. Extension Address list (tt_address) is a prerequisite.
  • 1.8. Please read the , before you're upgrading your direct_mail installation.
  • 1.9. Users Manual
    • Summary of the Direct Mail process
  • 1.10. The Direct Mail process let you create newsletter pages which can then be emailed to people on a recipient list. The following are the main steps of the Direct mail process:
  • 1.11. At any point, you may, if needed, use the module context sensitive help. Backend forms for editing direct mails and recipient lists also provide context sensitive help on each field.
    • Accessing the Direct Mail module
  • 1.12. Upon the installation of the Direct Mail extension there will be a new section, called “Direct Mail”  in the backend menu. In this section you will find 5 modules:
  • 1.13. Extension Manuals
  • 1.14. Extension Manuals
    • The Direct Mail Wizard
  • 1.15. The Direct Mail Wizard is located in the first module (Direct Mail). After choosing a Direct Mail SysFolder, the first step is shown. Four possible sources of a Direct Mail is shown: internal page (TYPO3 page), external page, quickmail and from existing Direct Mail.
  • 1.16. In the second step, the detail information of the direct mail is shown and you can manually change this, e.g. insert  attachments, by clicking the edit button. In this step the content of of the direct mail is automatically fetched and an information on the fetching is also shown (successful, warning or error). If there is an error, you should leave the wizard , correct this error (e.g. by clearing FE-cache) and continue the sending process by choosing from the list of existing Direct Mail (fourth Option in the first step).
  • 1.17. Extension Manuals
  • 1.18. The third step is the categorizing of the content. This step will be shown if the direct mail's source is a TYPO3 step. If the direct mail's source is other than TYPO3 page, then it will be skipped to step four. After choosing the categories, click the “update category settings” button to save the assignment of the categories.
  • 1.19. Extension Manuals
  • 1.20. To continue the sending process, click the next button. The fourth step will be shown.
  • 1.21. Extension Manuals
  • 1.22. Extension Manuals
  • 1.23. Extension Manuals
  • 1.24. The fourth step is sending test mails. If UIDs of tt_address or direct mail groups in the configuration module are given, they will be shown and click the name will send email to them and this step will be shown again. If there is no UID given in the configuration module, there will be only a text field, where you can give an email address.
  • 1.25. After finish testing the direct mail, click on the next button will show the fifth and final step.
  • 1.26. Extension Manuals
  • 1.27. Extension Manuals
  • 1.28. Extension Manuals
  • 1.29. The last step is mass sending the direct mail. After choosing a recipient lists from a drop down lists, you can choose time, when the mailer engine start sending it. The button next to the input field will show a pop up calendar. You can choose date and time. By clicking the “send to all subscribers in the recipient list” button, the direct mail is released to sending.
  • 1.30. The number in the brackets shows the amount of the recipients in the list.
  • 1.31. If you check the “Send this as test newsletter” box, the subject of the email will be prepended with the text, which you can define in the configuration module (in the “additional module options” section).
    • Personalizing direct mail content
  • 1.32. You may personalize the direct mails that will be sent to each recipient by inserting the following markers in content elements of the newsletter page. The markers will be substituted with the corresponding value found in each recipient record, whenever available. This is the list of predefined markers:
  • 1.33. Additional fields from the recipient table (e.g. defined by an extension) can be configured in the Extension Manager. You have to switch to "Information" view of EXT:direct_mail and set a comma-separated list of DB fields in "Additional DB fields of the recipie..." (addRecipFields)  to make more markers available. These can be used according to the pattern above like ###USER_<some field>###.
  • 1.34. Personalization only works for recipient lists of type "From pages" (and "From other recipient lists", if these contain "From pages" lists).
    • Types of recipient lists
  • 1.35. There are five types of Recipient lists:
  • 1.36. For more information on the individual fields found on the creation/editing form of recipient lists, please use the contextual help attached to each such field.
    • Importing a csv list of email addresses
  • 1.37. The Recipient Lists module gives you the option to import a csv (comma-separates list) of address records and create a recipient list containing the imported records.
  • 1.38. To make it easier to import csv records, there is a wizard, which guide you through the process.
  • 1.39. In the first step you can choose to upload a csv file or paste the records into a text field. You can use comma (;), semicolon (;), or colon (:) as field delimiter. This can be configured in the second step.
  • 1.40. Extension Manuals
  • 1.41. By clicking the next button, the csv file or csv records are uploaded and the second step is shown. In this step you can specify the detail information of the csv data, such as field delimiter, field encapsulation, and field name in the first line. You can also specify the SysFolder, where the records should be imported to, the uniqueness of the records, rename or update the records if a similar record is found, or to empty the SysFolder before importing.
  • 1.42. ATTENTION! If you set the field “remove all Addresses in the storage folder before importing”, all records in this SysFolder WILL be physically deleted.
  • 1.43. Extension Manuals
  • 1.44. After specifying the configuration you can start mapping the fields. There are 3 columns in the mapping step. The description column shows the first row of the csv records (if you set in the configuration that the first row is the field names) or shows only field_xx (where xx is continuous number).
  • 1.45. The mapping column shows only the list of field, which are part of tt_address table. You must at least map the field “Name” and “Email”.
  • 1.46. The value column shows the first up to three rows from the csv records. They should help you to map the field.
  • 1.47. Extension Manuals
  • 1.48. Extension Manuals
  • 1.49. Extension Manuals
  • 1.50. There are some new feature in the importer. You can now set HTML flag or categories to all records you are importing. In the select box, which contains the field names of tt_adress, there's a new entry called “categories”. This entry can be mapped to a comma-separated list of direct mail category IDs. This value will overwrite whatever categories you selected in the “Additional options” section.
  • 1.51. Extension Manuals
  • 1.52. After mapping the fields you are ready to start the import process. To start the import click the import button. After the importing,  a list of new imported, invalid email, updated and doublet records will be shown.
  • 1.53. Extension Manuals
  • 1.54. Extension Manuals
  • 1.55. Configuration
    • Installing the extension
  • 1.56. Import and install the extension using the Extension Manager.
  • 1.57. You may be required to install extension Address list (tt_address) which is  a prerequisite of extension Direct Mail.
  • 1.58. Some updates to the database structure may be needed. Press the “Make updates” button to make these updates.
  • 1.59. When the extension is installed, a “Direct Mail” Section and five new modules will appear above the Help section of the backend menu, provided that the user is granted access using the User Admin tool.
  • 1.60. The Extension Manager installation dialog allows to set the following extension configuration variables:
  • 1.61. If you are upgrading to version 2.0+ from an earlier version, an additional entry may be presented in the function drop-down menu of the extension manager: UPDATE! This option provides a function to convert some tables of your database. You should read the section below about upgrading to version 2.0+.
    • Creating a Direct Mail folder
  • 1.62. All operations of the Direct Mail module are performed from a specifically configured SysFolder. You may configure multiple Direct Mail folders.
  • 1.63. To create a Direct Mail folder:
  • 1.64. Once the Direct Mail folder is created, its title should appear in the list of available Direct Mail folders when using backend menu option Web > Direct Mail. Clicking on the folder title should display a page with the module drop-down function menu.
    • Configuring the Direct Mail module in Page TSConfig
  • 1.65. The Direct Mail configuration properties are set in the Page TSConfig of the Direct Mail folder under key mod.web_modules.dmail.
  • 1.66. Note that all these properties may conveniently be set using the Direct Mail module function “Module configuration”.
  • 1.67. The following properties set default values for corresponding properties of direct mails. These properties of direct mails determine the headers inserted in the direct mail messages.
  • 1.68. Extension Manuals
  • 1.69. The following properties set default values for corresponding properties of direct mails. These properties of direct mails determine the format of the content of direct mail messages.
  • 1.70. Extension Manuals
  • 1.71. The following properties set default values for corresponding properties of direct mails. These properties of direct mails specify parameters used to fetch the content of the direct mails.
  • 1.72. Extension Manuals
  • 1.73. The following properties specify the content transfer encodings and character sets to use when sending mails.
  • 1.74. Extension Manuals
  • 1.75. The following properties specify how links in mail content are processed.
  • 1.76. Extension Manuals
  • 1.77. The following properties specify parameters for the operations of various functions of the Direct Mail module.
    • Configuring the Direct Mail importer Module in Page TSConfig
  • 1.78. The Direct Mail configuration properties are set in the Page TSConfig of the Direct Mail folder under key mod.web_modules.dmail.importer.
  • 1.79. The following properties set default values for corresponding properties of the direct mail's importer. These properties of direct mail's importer determine the configurations.
  • 1.80. Extension Manuals
    • Configuring the Direct Mail Wizard in User TSConfig
    • Configuring the Calendar of the Direct Mail Wizard
  • 1.81. Extension Manuals
  • 1.82. The calendar used in the fifth step of the Direct Mail send Wizard is based on the Date Selector Library Extension (rlmp_dateselectlib) from Robert Lemke.
  • 1.83. The following properties are all children of mod.web_modules.dmail.calConf and it is used in the Page TSConfig of the Direct Mail SysFolder.
  • 1.84. Extension Manuals
    • Configuring the TS template for newsletter pages
  • 1.85. Newsletter pages are just normal pages. Their rendering is configured by the TS template. However, take the following into consideration:
    • Configuring plain text rendering
  • 1.86. It is good practice to always include a plain text version in email messages. Some email clients are not able to present the html content of email messages, but will present the plain text version if available. Users of many email clients may also have the option to set their preference as to the format that they wish to see.
  • 1.87. In order to configure the rendering f your newsletter pages in plain text, include static template “Direct Mail Plain Text” on the TS Template applicable to the pages. This static template contains a pre-defined template for plain text content rendering and makes it accessible as page type 99. You may thereafter configure the template to your needs in the template.
  • 1.88. Note: In versions previous to version 2.0, plain text rendering was achieved by including static template "plugin.alt.plaintext (99)". This static template may still be used in version 2.0+.
  • 1.89. The “Direct Mail Plain Text” static template is as follows:
    • Constants
    • Setup
    • Configuring plain text rendering of News records
  • 1.90. If you insert News records in your newsletter page, using the “Insert records” content element type, you may configure the plain text rendering of these News rendering. Simply add static template “Direct Mail News Plain Text” in your TS template. Note that order is important: “Direct Mail News Plain Text” should come after or below “Direct Mail Plain Text”. You may then tailor the template to your specific needs.
  • 1.91. The “Direct Mail Plain Text” static template is as follows:
    • Setup
    • Configuring the use of categories
  • 1.92. If you intend to make use of Direct Mail categories, you should do the following.
  • 1.93. First, add static template “Direct Mail Content Boundaries” to the TS template of the Direct Mail folder. This will ensure that content boundaries are inserted on the page when it is rendered whenever categories assignments are found on the content elements of the page. The static template ensures that content boundaries will be inserted in both HTML and plain text content. Insertion of boundaries is specified by setting TS setup property:
  • 1.94. This setting is already included in static template  “Direct Mail Content Boundaries” .
  • 1.95. Secondly, categories that may be assigned to content elements, to address records, to FE users or to recipient lists are determined by Page TSConfig. Therefore, you should configure the following properties in the Page TsConfig of the Direct Mail folder and, perhaps, of any page whose subtree may contain records of the corresponding types and which may be used in Direct Mail operations:
  • 1.96. where pid_list is the list of page id's on which categories may be found that may be assigned to records of the given type.
  • 1.97. Finally, when the use of categories is thus configured on a page and its subtree, you may also want the categories assignment field to be displayed in backend forms. This is achieved by setting the following properties in Page TSConfig of the same pages:
  • 1.98. The “Direct Mail Content Boundaries”  static template is as follows:
    • Setup
    • Enabling click statistics
  • 1.99. To enable the click statistics of the Direct Mail module, enable the following checkbox in the "Module configuration” function:
  • 1.100. Extension Manuals
  • 1.101. Extension Manuals
  • 1.102. This will register all links in the email messages and count all clicks on them.
  • 1.103. Note that form url's are not directed through the jumpurl feature but rather directly to the target page.
  • 1.104. A nice trick is to place a little clear-gif image in your HTML template and put the parameter dmailerping=”1” in the tag. This will force the capture function to set the url of this image absolute through the jumpurl registration. This means in other words that when this mail is opened it will be registered. This is an additional feature to the regular feature which registers all links clicked in the mail.
  • 1.105. Example:
  • 1.106. Note that the result of the jumpurl setting on the above HTML line is that the src attribute will be replaced by one that refers to the address of a script. Such attribute in HTML content of email messages may be disabled by SPAM filtering software.
    • Automatically invoke the mailer engine.
  • 1.107. There are some way to automatically invoking the direct mail mailer engine.
  • 1.108. Extension Manuals
  • 1.109. It's recommended to use the CLI option to automatically invoking the mailer engine.
    • Configuring the mailer engine cron task
  • 1.110. The mailer engine may be invoked from the Direct Mail backend module. However, if your newsletters are sent to numerous recipients, this extension provides a mailer engine script that may be configured as a cron task on the server. In this example setting, the mailer engine will be invoked every 5 minutes and will check if any mails need to be sent:
  • 1.111. Note that the absolute path to the script must be specified.
  • 1.112. Note also that dmailerd.phpcron is a shell script and requires the availability of a PHP binary, "/usr/bin/php”. Depending on your server configuration, you may have to edit the first line of the script to refer to the location of the PHP binary.
    • Configuring the CLI script
  • 1.113. Since TYPO3 4.x there is a CLI mode for TYPO3. The direct_mail CLI script uses the new CLI-API, which is available  since TYPO3 4.1.x.
  • 1.114. Before writing a cron task in your crontab, a BE-user with the name of “_cli_direct_mail” has to be created. This user must have no administrator right. After creating the BE-user, you can write the following line in the crontab:
  • 1.115. Extension Manuals
  • 1.116. This will call the CLI-script with two parameters: the extension's key (direct_mail) and a task (masssend).
    • Configuring direct_mail on Gabriel
  • 1.117. Please refer to the documentation of Gabriel. The direct_mail task will be shown in the list of task automatically.
    • Using Sendmail deferred delivery mode
  • 1.118. The following option applies only when Sendmail is used in Unix/Linux environments.
  • 1.119. When sending a large number of emails with the mailer engine, processing delays may be caused by attempts on the part of the mail server to locate the destination domain of each email. As a consequence, timeouts may occur causing failure of the cron ask. Or the cron task may be triggered before the previous instance has terminated. In the latter case, some emails may be sent multiple times.
  • 1.120. Such problems may be avoided by using the Sendmail deferred delivery mode. In this mode, each email is simply put on a queue for later processing by Sendmail. This results in much faster response to the cron task.
  • 1.121. You may enable the deferred delivery mode by setting the corresponding variable on the installation screen of this extension. The option applies only to emails sent by the mailer engine cron task of the Direct Mail extension.
  • 1.122. The following variables must be set in the Sendmail configuration file (values shown here are just examples):
  • 1.123. These example settings have the following meaning: the email queue will be emptied every 10 minutes. If a problem is encountered while trying to send an email, it will be put on another queue and delivery will be attempted again every 30 minutes.
  • 1.124. For more information on Sendmail configuration, see: http://www.sendmail.org/.
    • Configuring the analysis of returned mails
  • 1.125. There are probably many ways to configure analysis of returned mails. We propose an approach based on open Source program fetchmail. For more information about fetchmail, see .
  • 1.126. Extension Manuals
    • Configuring SMTP
  • 1.127. To send the Newsletter with SMTP, you only have to enable this feature from the extension manager (Ext Manager > Direct Mail).
  • 1.128. Extension Manuals
  • 1.129. Before enabling this feature, you have to make sure, that PEAR:mail is installed on your server. To install PEAR:mail you can use one of the following extensions:
  • 1.130. This feature is kindly provided by .
  • 1.131. There is a bug in PEAR, please see “Known Problem” Section.
  • 1.132. Upgrading
    • Upgrading to version 2.6.4
  • 1.133. A new feature (see issue at , thanks to Sonja Scholz for providing the patch), which needs an update of the fe_users table is integrated in this version. A new column called activate newsletter (module_sys_dmail_newsletter) has been added to the fe_users table. If you're sending newsletter to fe_users, this field needs to be marked.
  • 1.134. This feature enables the fe_user to unsubscribe the newsletter without having to delete its own account
    • Upgrading to version 2.5.0
  • 1.135. To make sure that your cron task and analysis of the returned mail working, you should check the path to the scripts in the cron task. All scripts can be found in the folder res/scripts.
    • Upgrading to version 2.0+
  • 1.136. The two main considerations when upgrading to version 2.0+ from a previous version are:
  • 1.137. When upgrading to version 2.0+, it is necessary to convert pre-existing categories in Page TSConfig and create corresponding Direct Mail Categories in the Direct Mail folders. If any categories were assigned to Content elements, Addresses , FE Users or Recipient Lists, these assignments also need to be converted to refer to the newly created Direct Mail Categories.
  • 1.138. Conversion of categories is NOT revertible. Therefore, it would be prudent to take a backup of the TYPO3 database before this conversion is performed.
  • 1.139. When upgrading to version 2.0, an additional option is presented in the function drop-down menu of the Extension Manager: UPDATE! The additional option is presented if a Direct Mail folder already exists and until at least one Direct Mail Category has been created.
  • 1.140. If you select the UPDATE! option, pre-existing categories defined in Page TSConfig will be converted into new Direct Mail Categories. The Direct Mail categories will be created in each of the Direct Mail folders for which category definitions may be found in Page TSConfig.
  • 1.141. Conversion of categories assignments in Content elements, Addresses, FE Users and Recipient Lists will also be attempted. If the use of categories was not yet configured, this part of the conversion process will fail, but may re-attempted a later time, using the “Categories conversion” function of the Direct Mail module. The next step is thus to configure the use of categories: see the section of this document on this subject.
  • 1.142. Note that the conversion of categories assignments is only simulated, until you effectively confirm the conversion.  Once confirmed, the records are updated and the conversion cannot be undone without restoring the table with some database backup.
  • 1.143. When the use of categories is configured, you may re-attempt the conversion of categories assignments using the “Categories conversion” option of the function menu of the Direct Mail module. The function will report statistics on the conversion being performed, as well as conversion already done. It will also list the records that could not be converted: this exception would normally be due to the fact that the use of categories is not configured for this type of record in the page subtree in which it is found.
  • 1.144. Although it may not be undone, categories conversion may be re-attempted as many times as required without causing any harm.
  • 1.145. Troubleshooting
    • Direct mails are not sent
  • 1.146.  Here is a checklist to work through:
    • Mass mail is only delivered after sending another mail from same MAC OS X server
  • 1.147. Problem description:
  • 1.148. Invoking a massmail results in getting an admin-mail which says that the procedure has started.
  • 1.149. Hours later I didn't got any newsletter, nor an admin-mail which says that sending has stopped.
  • 1.150. After invoking a form-mail from another domain on the same server, I instantly get all those messages, i.e. the admin mail and the newsletter. They all got the same timestamp as the first received admin mail.
  • 1.151. I send a newsletter to a small group of users in tt_address. There was only one (1) user in this list.
  • 1.152. Discussion:
  • 1.153. Asking Google brought the hint that there's some authorization problem with sendmail. chmod'ing and chown'ing is supposed to help.
  • 1.154. Reference: http://www.entropy.ch/software/macosx/php/welcome_de.html (german)
  • 1.155. Solution:
  • 1.156. I had to change the group of the "postfix"-directory as described in the reference, i.e.
  • 1.157. sudo chmod g-w /
  • 1.158. and
  • 1.159. sudo chgrp smmsp /var/spool/postfix
  • 1.160. Source:
  • 1.161. http://bugs.typo3.org/view.php?id=2570
  • 1.162. Known Problems
  • 1.163. Please see/report problems in the TYPO3 Bugtracker under project .
  • 1.164. You may get support in the use of this extension by subscribing to .
  • 1.165. To-Do List
  • 1.166. Please see/submit feature request sin the TYPO3 Bugtracker under project .
  • 1.167. You may sponsor further development of this extension by contacting the of this document or any of the contributors mentioned at the beginning of this document.
  • 1.168. Change Log
  • 1.169. For a complete change log, please refer to the ChangeLog file.
  • 1.170. Extension Manuals

This document is related to version 2.6.7 of the extension direct_mail.