Archiving Emails from Linux-based Email Servers

Synchronizing Users

Linux-based email servers support various database types for managing users. Additionally to the legacy user database know as the passwd file, LDAP-based databases such as OpenLDAP or Active Directory as well as SQL-based databases are often used. Instruction on setting up user synchronization with MailStore Server can be found in the following chapters of the MailStore Server manual:

Note: When batch-archiving IMAP mailboxes MailStore Server automatically creates non-existing users. The authentication type of those users will be set to MailStore-integrated.

Archiving Individual Mailboxes

Setting up archiving processes for IMAP or POP3 mailboxes is done using archiving profiles. General information about archiving profiles is available in chapter Working with Archiving Profiles.

Users can only archive their own mailboxes to their personal user archive. To archive the emails of other users, you have to be logged on to MailStore Client as MailStore administrator. Only MailStore administrators can archive the emails of other users.
In MailStore, click on Archive Email.
To create a new archiving profile, select Other Server via IMAP/POP3 from the Email Servers list in the Create Profile area of the application window.
A wizard opens guiding you through the setup process.
Select Single Mailbox and click on OK.
Fill out the fields Email Address, Host, Access via, User Name and Password. The Email Address is used to label the folder that is created in the archive by this profile. Click on Test to verify the data entered.
For the IMAP-TLS and IMAP-SSL protocols only: If the certificate provided by the remote host cannot be verified (e.g. self-signed or signed by an unknown certificate authority), enable the option Accept all certificates to allow MailStore to establish a connection. As this option leads to an insecure configuration, warnings may appear in the summary and/or the dashboard.
Click on Next.
Customize the list of folders to be archived (IMAP only), the filter (IMAP only) and the deletion rules. By default, no emails will be deleted from a mailbox. The timeout value only has to be adjusted as needed (e.g. with very slow servers).

Click on Next.
If logged on to MailStore Server as administrator, the target archive can be specified at the next step. Select the archive of the user for whom the mailbox is to be archived.
At the last step, a name for the new archiving profile can be specified. After clicking on Finish, the archiving profile will be listed under Saved Profiles and can be run immediately, if desired.

Archiving Multiple Mailboxes in One Step

MailStore provides several methods for archiving multiple mailboxes in one step, which can be used with common IMAP servers on Linux-based email servers.

While the method Multiple IMAP Mailboxes (CSV File) does not require any special configuration on the IMAP server, it is normally necessary to modify the IMAP server's configuration before the 'Multiple IMAP Mailboxes (Master Password) method can be used.

Configuration of the IMAP Server

In the following you find instructions for the most common IMAP server on Linux servers, explaining how to enable access to the mailboxes using the PLAIN SASL method as described in RFC 4616.

Important notice:The following sections describe configuration changes that should only be performed by experienced administrators. The instructions do not cover configuration characteristics of the MTAs on different Linux distributions.

Dovecot IMAP

TODO

Cyrus IMAP

TODO

Configuration of MailStore Server

To learn more about archiving multiple mailboxes in one step, please refer to the chapter Batch-archiving_IMAP_Mailboxes of the MailStore Server manual.

Archiving Incoming and Outgoing Emails Directly

Archiving of all incoming and outgoing email directly requires that the used mail transfer agent (MTA) delivers copies of all email into a dedicated mailbox (called multidrop mailbox in the following). From this multidrop mailbox, emails can be archived by using a profile of type [Archiving_IMAP_and_POP3_Multidrop_Mailboxes|Multidrop Mailbox]] in MailStore Server.

Please notice: Emails can and should be automatically deleted after they have successfully been archived from a multidrop mailbox. You can configure this in the corresponding archiving profile.

Configuration of the Mail Transfer Agent

In the following it is explained how to set up the required copy function in the most popular mail transfer agent on Linux servers.

Important notice:The following sections describe configuration changes that should only be performed by experienced administrators. The instructions do not cover configuration characteristics of the MTAs on different Linux distributions.

Exim

First create a new mailbox that should act as a multidrop mailbox.

To deliver copies of all emails into a multidrop mailbox, it is necessary to add a new router as well as a new transport to the Exim configuration.

The configuration of the new Exim router should look like this:

mailstore_router:
  no_expn
  no_verify
  transport = mailstore_transport
  driver = accept
  unseen

Please make sure that the mailstore_router is passed through before the router responsible for delivering the messages to the actual user mailboxes. It is strongly recommended to enter it first in the Exim router configuration. Depending on your anti-spam or anti-virus protection it may also be suitable to put it right after the routers handling those actions.

The delivery into a multidrop mailbox is performed by the corresponding Exim transport, which is to be defined in the transport section of the Exim configuration. In this example, a transport called mailstore_transport is to be created. A new transport can be put anywhere in the Exim transport configuration. The following example shows the configuration of an Exim transport storing all emails in Maildir format in the directory /var/mailstore/Maildir. To store messages in a different location, please refer to the Exim manual.

mailstore_transport:
  driver = appendfile
  delivery_date_add
  envelope_to_add
  return_path_add
  maildir_format = true
  create_directory = true
  directory = /var/mailstore/Maildir

After making these configuration changes, the Exim daemon must be restarted.

Postfix

First create a new mailbox that should act as a multidrop mailbox.

To have Postfix send a copy of all emails to this mailbox, you need to add the following line to the file main.cf:

 always_bcc = [email protected]

After making these configuration changes, the Postfix daemon must be restarted.

Sendmail

Because Sendmail is one of the most flexible mail transfer agent available, there is no best practice guide for setting up the functionality described above. How Sendmail is configured to send a copy of all emails to an additional mailbox mainly depends on your system environment. Since Sendmail 8.10 it is recommended to use the libmilter interface to add features to Sendmail.

A database with all available Milter plug-ins is available at https://www.milter.org. There is also a special section with archiving solutions. The listed plug-ins differ substantially in the way they are integrated into Sendmail and the way they work.

Qmail

First create a new mailbox that should act as a multidrop mailbox.

To have Postfix send a copy of all emails to this mailbox certain changes to be made before compiling the Qmail source code. The following guide is an adapted version of the article "Life with Qmail FAQ":

QUEUE_EXTRA is a compile-time configuration variable that specifies an additional recipient that will be added to every delivery. This is used primarily for logging; the FAQ describes how to use QUEUE_EXTRA to keep copies of all incoming and outgoing messages.

To use QUEUE_EXTRA, edit extra.h specifying the additional recipient in the format Trecipient\0, and the length of the QUEUE_EXTRA string in QUEUE_EXTRALEN (the "\0" counts as one character). For example:

 #define QUEUE_EXTRA "Tmailstore\0"
 #define QUEUE_EXTRALEN 11

Recompile the Qmail source code and restart the Qmail daemon to apply the changed.

Detailed information regarding Qmail can be found on the [Qmail Hompage] and [Life with qmail].

Configuration of MailStore Server

Setting up archiving processes for multidrop mailboxes is done using archiving profiles. General information about archiving profiles is available in chapter Working with Archiving Profiles.

Please proceed as follows:

Log on as MailStore administrator using MailStore Client.
In MailStore, click on Archive Email.
To create a new archiving profile, select Other Server via IMAP/POP3 from the Email Server list in the Create Profile area of the application window.
A wizard opens guiding you through the setup process.
Select Multidrop Mailbox and click OK.
Please note: To be able to archive a multidrop mailbox, MailStore Server users along with their email addresses must exist in the MailStore Server user management. If this is not the case, MailStore Server will offer to set up and run the directory synchronization at this point. Once completed, the wizard will resume.
Alternatively, you can cancel the wizard and create users manually as described the in chapter User Management.
Fill out the fields Access via, Host, Username and Password. Click on Test to verify the data entered.

For the TLS and SSL protocols only: If the certificate provided by the remote host cannot be verified (e.g. self-signed or signed by an unknown certificate authority), enable the option Accept all certificates to allow MailStore to establish a connection. As this option leads to an insecure configuration, warnings may appear in the summary and/or the dashboard.
Adjust any further settings such as how to handle emails with unknown addresses or asking MailStore to delete emails after they have been archived. The latter option is especially sensible when dealing with mailboxes that are exclusively used for archiving.
The placeholders {u-email} or {h-email} can be used under Target Folders. {u-email} represents a user's primary email address and {h-email} is the email address found in the email header. Should a user have multiple aliases, using {h-email} will result in emails sent to different aliases of a user being archived in different folders, while using {u-email} will file all emails below the primary email address.

Notice: If you are additionally archiving messages from the users mailboxes, you have to set the target folder names here to the folder names that match the names that were created by the user mailbox archiving profiles. Otherwise, additional or similar folders could be created in the users' archives. The folder names are case-sensitive.
If the option Synchronize with Directory Services before archiving is enabled, the MailStore user list will be synchronized with configured directory service before the archiving process actually runs. This has the advantage that, for example, new employees will be created as MailStore users before archiving which enables MailStore to sort their emails into the correct archives.
Click on Next.
The timeout value only has to be adjusted on a case-by-case basis (e.g. with very slow servers).
Click on Next.
At the last step, select a name for the new archiving profile. After clicking on Finish, the archiving profile will be listed under Saved Profiles and can be run immediately or automatically, if desired.