Bulk Import of Email Files

A checked version of this page, approved on 26 April 2023, was based on this revision.

When archiving email files, MailStore only allows archiving of individual email files (PST, MBOX) or directories which contain email files of one user (EML, MSG).

The following describes how to archive all PST-, MBOX-, EML- or MSG-files of all users in a few steps, using a bulk import script written in Python.

The bulk import script expects the email files in folders that are named after the corresponding MailStore users. Such a folder structure can easily be created by another script included in the Python scripting package.

If you want to archive PST- or MBOX-files of one user only, you can use the bulk import scripts from the batch scripting package; the installation of Python is not required in that case.

Preparing Python

The provided scripts are written in Python. Python 3.2 or newer is required as well as the Python scripting package. The Python API-Wrapper has to be installed.

Preparing the Folder Structure

The scripting package contains the createUserFolders.py script which prepares the folders by fetching a list of existing MailStore Server users and creating folders with the same names in the filesystem. The actual bulk archiving script uses these folder names to assign the enclosed files to the corresponding users in MailStore.

The MailStore Server Administration API must be enabled in the MailStore Server Service Configuration.

After installing Python, the script can be modified by right-clicking on it, then selecting IDLE, the supplied Python editor. Adapt the values of USER, PASS, HOST and PORT to your installation.

The rootpath variable contains the path where the folders will be created.

Press F5 to run the script. A security question follows.

Preparing the Archiving Profile

Before executing the bulk import script, a new archiving profile that corresponds to the email files (PST, MBOX, EML/MSG) to be archived must be created manually in MailStore Server.

The profile must of the type Single User. The settings in this archiving profile apply to all read files. For instance, if your exclude or include folders or define filters, they will be applied to all files you archive from.

The path you specify in the profile to the PST or MBOX file, or the folder containing EML/MSG files, is irrelevant, as it gets overwritten by the bulk import script. Additionally, sub folders are recognized when archiving EML/MSG files, whereas MailStore headers in EML files are always ignored.

Important Notice: If you want to archive PST files, Microsoft Outlook must be installed on the machine where the archiving profile is run.

Depending on the file type to be archived, the created profiles must be named templateBulkImportEMLMSG, templateBulkImportPST or templateBulkImportMBOX.

Filling the Folders

For each user, put the email files to be archived into the corresponding folders under the rootpath. When archiving PST files, these can be located in subfolders. A simultaneous import of different file types is not supported. If you want to archive PST and MBOX files, a separate run is necessary for each.

Archiving

Archiving is done by the bulkImport.py script. Before running it must be adapted to your MailStore installation by using IDLE. Since this script uses the MailStore Client to connect to the server, rather than connecting to the MailStore Server Administration API you have to use slightly other values.

The path to MailStoreCmdSilent.exe, the host name, credentials of an administrator and the rootpath have to be adjusted.

When importing into MailStore Server the HOST has to be in the format computer:port. When importing into a MailStore SPE instance the format has to be https://computer:port/instanceid.

Similar to the createUserFolders.py script, rootpath points to the folder that contains the actual user folders. The filetype variable can have one of the following values according to the file type to archive: pst, mbox or emlmsg.

In IDLE, press the F5 key to run the script.

Logging

While the archiving process runs, its progress can be monitored in the status window. The message exitcode: 0 means that the file or folder has been archived successfully. A different output would show the error. The task logs of the procedures are stored in the rootpath folder and are named as follows:

 Date-Time-User-File.log (PST,MBOX)

or

 Date-Time-User.log (EML/MSG)

The table Recent Results in MailStore Client gives you an overview of all run archiving profiles. Failed runs can by easily identified by their start time as the file names of the log files begin with the same value.

Migrating on-premises Exchange Archive Mailboxes into MailStore

MailStore is not able to access on-premises Exchange archive mailboxes directly. To archive messages contained therein, they have to be copied to a location that MailStore is able to read from. The messages could either be moved back to the original mailbox by the end user through Outlook and then be archived with MailStore using Exchange or Outlook archiving profiles. Alternatively, an Exchange administrator exports the archive mailboxes into PST files. The latter can be done in one central location and fully automated. Afterward, the PST files can be archived by using the procedure described above.

To create PST files from archive mailboxes, follow these steps:

Create a network share that can be reached from the Exchange Server.
Grant the domain group Exchange Trusted Subsystem write permissions on this share.
Open a Microsoft Exchange Management Shell session in the Microsoft Exchange Server.
Execute the following command in the Microsoft Exchange Management Shell to grant export privileges to [email protected]. Adjust the user name according to your environment:

  New-ManagementRoleAssignment -Role "Mailbox Import Export" -User administrator@example.com

Define a new variable in the Exchange Management Shell session that points to the network share:

  $ExportPath = "\\NAS\PSTs"

Execute the following script in the Exchange Management Shell to create the PST files. The Exchange Management Shell has to be started by the user to whom have been granted export privileges. You might have to restart the Exchange Management Shell session:

  foreach ($mailbox in (Get-Mailbox))
 {
	 $directory = ("$ExportPath\{0}" -f $mailbox.Alias).toLower()
	 if (-not (test-path $directory) )
	 {
		 New-Item -ItemType directory -Path $directory
	 }
	 New-MailboxExportRequest -Mailbox $mailbox.Alias -FilePath ("{0}\{1}-Archive.pst" -f $directory,$mailbox.Alias) -IsArchive
 }

Note: The New-MailboxExportRequest cmdlet only accepts exporting to UNC paths. Local paths are not supported. When the parameter -IsArchive is omitted, the regular mailbox will be copied into the PST file.

The export processes are running in the background. The state can be checked with the following command:

  Get-MailboxExportRequest | Get-MailboxExportRequestStatistics

The role assignments can be removed with the following commands:

 Get-ManagementRoleAssignment -Role "Mailbox Import Export" | select Name
 Remove-ManagementRoleAssignment "Mailbox Import Export-Administrator"

Once the export is finished, the PST files can be archived with the procedure described above.