Difference between revisions of "Storage Locations"

[unchecked revision]

[checked revision]

Latest revision as of 15:53, 28 September 2021

A MailStore archive physically (i.e. on the storage) consists of individual archive stores, each with its own database, search indexes and data containers.
To MailStore users these archive stores are fully transparent; they have access to all archived emails in all active archive stores and thus get a logical view on the archive according to their respective privileges.

Under Administrative Tools > Storage > Storage Locations you can configure the parameters of the archive stores' auto-create, manually create new archive stores and manage the archive's existing archive stores. You can also view the location of the master database here.

Changing the Storage Location of the Master Database

The storage location of the master database can only be viewed here. In case the MailStore Client is started on the MailStore Server machine, clicking on Change... closes the MailStore Client and starts the MailStore Server Service Configuration.

Archive Store Basics

In MailStore, there are two types of archive stores: Internal Archive Stores and External Archive Stores.

Internal archive stores are stored entirely in configurable file system directories and can be created, managed and backed up automatically by MailStore. For most environments using internal archive stores is recommended.

External archive stores offer storage in SQL databases, but have several limitations. Information about external archive stores is available in chapter Using External Archive Stores.

Both internal and external archive stores always consist of the following three components:

Folder Information and Meta Data
Contains all data needed for the construction of the folder structure and the email list.
Email Headers and Contents
Contains the actual payload of the archive.
Full Text Index
Contains all data needed for searching through emails and attachments. The full text index can be reconstructed at any time. Additional information is available in chapter Search Indexes.

MailStore archive stores as a whole as well as their individual components can be put on different physical storage locations, including network based storages. For additional information, please refer to the article Using Network Attached Storage (NAS).

Important notice: When choosing the physical storage system, please attend to the System Requirements.

Creating Archive Stores

In MailStore Server, internal archive stores are created automatically and additionally can be created manually. External archive stores can only be created manually.

Creating Internal Archive Stores Automatically

From time to time, MailStore Server automatically creates a new internal archive store for optimal system performance and stability. The maximum size and amount of emails are determined by MailStore Server internally.

If needed, you can limit the size that an archive store can reach at most before a new one is created to align with your backup routine. You can also configure the storage location where new archive stores will be created automatically.

Important notice: The maximum size should not be configured manually without good reason, so that MailStore Server can determine the optimal time automatically for creating a new archive store.

To change these settings, please proceed as follows:

Below the list of archive stores, click on the Create automatically... button.
The Auto-Create Archive Stores dialog opens.
Customize the settings as preferred.
Under Base directory enter the directory below which you want MailStore Server to create the new internal archive stores. The internal archive stores that are created automatically by MailStore Server, including any subfolders, follow the naming scheme year-month, e.g. 2021-10.
Optional: With the option Use different base directories for databases, content and search indexes you can configure separate directories for the individual components of an archive store. For example, you can put database and search index on a fast storage to accelerate folder operations and the MailStore search while leaving the email content files on a slower storage.
Click on OK.

Creating an Internal Archive Store Manually

To create a new internal archive store manually, please proceed as follows:

Below the list of archive stores, click on the Create... button.
The Create New Archive Store wizard opens.
Select the option Internal Archive Store and click on Next.
Enter a name for the new internal archive store in the Name field, e.g. 2021-10.
If you don't want MailStore to archive new emails in the new archive store, deselect the option Archive new messages here.
MailStore Server derives a base directory for the new internal archive store from the name entered and the path of the master database. By default, MailStore Server stores all components of an archive store in a folder structure that is created automatically below the base directory. You can optionally either change the proposed path manually or select an existing directory; an existing directory must not contain any files or subfolders.
Optional: With the option Use different base directories for databases, content and search indexes you can configure separate directories for the individual components of an archive store. For example, you can put database and search index on a fast storage to accelerate folder operations and the MailStore search while leaving the email content files on a slower storage.
Click on Finish to create the internal archive store.

Creating an External Archive Store

External archive stores can only be created manually. Creating a new external archive store is described in chapter Using External Archive Stores.

Managing Archive Stores

Setting the Status

Right-click on an archive store and select the status from the Set Status submenu. As an alternative, you can select an archive store and set the status via the drop-down list in the details pane on the right of the archive store list. You can set the following status:

Archive here
In MailStore Server, there can only be one archive store with this status. All newly archived emails are written into this archive store. The emails are available to all MailStore users and can be located by searching and through the folder structure. Emails in this archive store can be deleted or moved according to configured user privileges and compliance settings.

Notice: If a new internal archive store has been created automatically, its status is initially set to Archive here.
If you set an external archive store's status to Archive here, the automatic creation of internal archive stores will be suspended until you set an internal archive store to this status.
Normal
Emails in archive stores with status Normal are available to all MailStore users and can be located by searching and through the folder structure. Emails in this archive store can be deleted or moved according to configured user privileges and compliance settings.
Write-Protected
Irrespective of user privileges and compliance settings, emails in write-protected archive stores can only be accessed read-only. The emails are available to all MailStore users and can be located by searching and through the folder structure. However, emails in such archive stores cannot be deleted or moved.

Notice: Please note that file system write access to the directory of the archive store is still required and that this status prevents automatic processing of retention policies.
Disabled
Disabling an archive store allows you to make changes to its configuration. This may be necessary after Moving the Archive, for example. While an archive store is disabled, the emails contained therein are not available to the archive.

Notice: Please note that this status prevents the execution of archiving profiles.

Editing

Through the context menu item Edit... you can change the name and the directories of an archive store with status Disabled.

Renaming

Through the context menu item Rename... you can change the name of an archive store irrespective of its status.

Detaching

Through the context menu item Detach you can detach an archive store from the archive, for example if all email contained therein do no longer have to be archived. The detached archive store and the emails contained therein are no longer available to the archive.

Attaching

A detached archive store can be reattached to the archive using the Attach... button. If the archive store cannot be decrypted automatically, you will be asked for the respective recovery key. For example, this could be the case for archive stores that belonged to another MailStore installation or which have been moved from another machine. You can find more information on the recovery key in section Security and Encryption of chapter MailStore Server Service Configuration.

Unlocking

If an archive store cannot be decrypted automatically, it will be listed as Locked in the list of archive stores. Through the context menu item Unlock or by changing the status you will be asked for the respective recovery key. You can find more information on the recovery key in section Security and Encryption of chapter MailStore Server Service Configuration.

Maintenance of Archive Stores

All available maintenance commands can be accessed through the context menu of the list of archive stores. Alternatively, you can select an archive store and click on the Maintenance drop-down list in the details pane. The following commands are available:

Cleanup (Compact)
Optimizes the data structures while compacting the data.
Check Data Integrity
Verifies the data integrity between "Folder Information and Meta Data" as well as "Email Headers and Contents".
Maintain All FDB Files
Maintains the master database and all databases of internal archive stores.
Recalculate all statistics
Recalculates the statistics (number of emails per archive store) for all archive stores.

Maintenance commands can also be scheduled for automatic execution via Jobs.

@@ Line 1: / Line 1: @@
-= Structure of the MailStore Database =
+A MailStore archive physically (i.e. on the storage) consists of individual archive stores, each with its own database, search indexes and data containers.<br/>To MailStore users these archive stores are fully transparent; they have access to all archived emails in all active archive stores and thus get a logical view on the archive according to their respective privileges.
-A MailStore database consists of the following:
-== The Master Database ==
-Every MailStore Server installation has exactly one master database where general information such as users, email folders and settings are stored. Compared with file groups (see below), the master database has a very small storage space requirement.
-The master database is included in the setup of MailStore Server and is installed and set up automatically. Through the MailStore Server Configuration, the storage location of the master database can be determined and changed, if desired. Please keep in mind that the database has to be stored on a local drive; network drives on other computers are not supported.
-The master database is integrated in MailStore Server (embedded database) and can be run concurrently with other database systems.
-== File Groups ==
-The actual data - the archived emails - is stored in file groups. During the initial installation of MailStore Server, a first file group is created automatically. To distribute the archive among several storage locations, any number of additional file groups can be added at any time. In addition to allowing for flexible management of the storage space, the creation of new file groups has a positive influence on the performance of the archive.
-A file group consists of:
-'''Database''' - The database stores a list of the individual emails, including header information, as well as a list of indexes.
-'''Content''' - The actual email content is stored in .DAT files. Generally, these files have a manageable size of approximately 8 MB each, but individual files can vary considerably in size.
-'''Index''' - One index per user is created, which makes extremely fast searches possible within MailStore. In addition, indexes are used for navigation within the tree structure.
-More information about this topic is available in the following chapter: Managing Storage Locations.
-= Managing Storage Locations =
-In Storage Locations Management, the location of the master database can be viewed and the file groups of the archive can be managed. The file groups contain the actual data, the archived emails; by creating new file groups the complete archive can be distributed among different storage locations (e.g. different hard drives); existing file groups can always be moved at a later time.
-Learn more about the master database, file groups and the differences between these types of storage in the previous chapter: Structure of the MailStore Database.
-== Accessing Storage Locations Management ==
-Log on to MailStore Client as administrator. Click on ''Administrative Tools'' and then on ''Storage Locations''.
-[[File:tech_storageloc_01.png|center|500px]]
+Under ''Administrative Tools > Storage > Storage Locations'' you can configure the parameters of the archive stores' auto-create, manually create new archive stores and manage the archive's existing archive stores. You can also view the location of the master database here.
 == Changing the Storage Location of the Master Database ==
+The storage location of the master database can only be viewed here. In case the MailStore Client is started on the MailStore Server machine, clicking on ''Change...'' closes the MailStore Client and starts the [[MailStore Server Service Configuration]].
-Here, the storage location of the master database can only be viewed. By clicking on ''Change'', only a summary of the steps (as described below) required to change the location is displayed:
+== Archive Store Basics ==
+In MailStore, there are two types of archive stores: ''Internal Archive Stores'' and ''External Archive Stores''.
-Start the MailStore Server Base Configuration which is located in the MailStore Server program folder in the Windows Start menu (on the MailStore Server PC). Select the storage location of an existing master database (e.g. to restore a backup). If an empty directory is chosen, a new master database will be created. After each change, restart the MailStore Server Windows service by clicking on ''Restart'' in the same window.
-== Store Newly Archived Email In... ==
-Below File groups, there is the option to Store Newly Archived Emails In. Select the file group into which new emails are to be archived. Only file groups that are not write-protected can be chosen; their status can be changed at runtime.
-== Creating a New File Group ==
-To create a new file group, click on ''New'' in the menu bar at the bottom of the window. Select an empty directory and click on ''OK''.
-== Write-Protecting a File Group ==
-Select a file group from the list and, in the menu bar on the bottom of the window, click on Write Protect.
-The emails stored in a write-protected file group remain fully available to MailStore users and can be located through the folder structure or by running a search. However, neither can new emails be archived into this file group nor can existing ones be deleted from it. Please keep in mind that the file system still requires write access to the file group.
-After a file group has been write-protected, it is marked in the list with a lock symbol next to it.
-To remove the write-protection, select the appropriate file group and click on Write Protect again.
-== Attaching and Detaching File Groups ==
-Existing file groups can be detached from the archive: Simply select a file group from the list and click on Detach in the menu bar on the bottom of the window. Once detached, the file group and the emails contained therein are no longer available in the archive. This feature can be used for taking old parts of the archive out of storage, for example.
-A detached file group can be reattached to the archive at any time: by clicking Attach, the file group becomes fully available again. Please note: Only file groups which originated from the same archive can be attached; file groups from external archives cannot be integrated.
-== Moving File Groups ==
-To move a file group, please proceed as follows:
-*Detach the file group to be moved: Select the appropriate group from the list and click on ''Detach''.
-*Use Windows Explorer to move the file group to a different directory on any local(!) network.
-*Reattach the file group: Click on ''Attach'' and select the new storage location of the file group. Click on ''OK'' to confirm.
-== Maintenance of the Storage Locations ==
-The following features are available through Maintenance in the menu bar on the bottom of the window:
-*'''Master Database - Cleanup (FB Sweep)'''<br/>Simple cleanup of the master database.
-*'''Master Database - Rebuild (FB Backup+Restore)'''<br/>Complete rebuilding of the master database (e.g. if structural problems occur).
-*'''File Group - Free Unused Disk Space'''
-*'''File Group - Check Data Integrity'''
-*'''File Group - Cleanup (FB Sweep)'''<br/>Simple cleanup of a file group database.
-*'''File Group - Rebuild (FB Backup+Restore)'''<br/>Complete rebuilding of a file group database (e.g. if structural problems occur).
-*'''File Group - Recalculate Statistics of all File Groups'''
-== Creating File Groups Automatically ==
-MailStore Server can be configured to create and activate new file groups in regular intervals, e.g. monthly or quarterly. Please proceed as follows:
-*Start MailStore Client and log on as MailStore administrator (admin).
-*Click on Management Shell.
-*Enter the following:
- schedule filegroup-create-auto --basedir="D:\FileGroups"
-Instead of "D:\FileGroups" enter the directory in which new file groups are to be created. File groups, including subdirectories, that are created automatically by MailStore are named using the format Year-Month, for example 2009-04.
-*A dialog window appears.
-*Click on ''Other Trigger''.
-*Click on ''OK''.
-*In the window ''Schedule'' click on ''New''.
-*Under ''Schedule Task'' select ''Monthly''.
-*If new file groups are to be created quarterly, click on ''Schedule Task Monthly'' and select ''only January, April, July and October'', for example.
-*Click on ''OK'' and follow the directions on the screen.
-== Storage Strategies ==
-'''Performance:''' For every 500,000 emails, a new file group should be created. This ensures a consistently high access speed when searching emails.
-'''One-time backup:''' Older file groups can be write-protected (see above). These file groups remain available to users (with the exception of moving or deleting emails) but do no longer have to be backed up constantly. Write-protected file groups can be kept on cost-efficient storage media without any risks.
-= Administration of the Full-Text Search =
-MailStore Server offers users an extremely fast full-text search. All emails a user has read-access to are searched, in most cases in only fractions of a second. To ensure this remarkable speed, MailStore Server sets up so-called search indexes during archiving. They work in a way similar to the indexes often found in the back of books: looking up something in an index gets results significantly faster than searching each single page.
-MailStore Server maintains one index file each
-*per file group and
-*per user.
-Normally, the creation, maintenance and usage of search indexes is completely transparent meaning that neither administrators nor MailStore users need to know of their existence or their internal workings. In some cases, however, some maintenance may become necessary. For example:
+Internal archive stores are stored entirely in configurable file system directories and can be created, managed and backed up automatically by MailStore. For most environments using internal archive stores is recommended.
-*because of power outage,
+External archive stores offer storage in SQL databases, but have several limitations. Information about external archive stores is available in chapter [[Using External Archive Stores]].
-*because of unexpected termination of the MailStore Server service
-*because of missing network connectivity (only with storage on an NAS) or
-*because of changes to the index configuration (see below).
-In these cases it may occur that archive, index and settings are no longer synchronous and that individual indexes must be rebuilt. Generally, a corresponding error message will be displayed.
+Both internal and external archive stores always consist of the following three components:
+{{Archive Stores Structure}}
+MailStore archive stores as a whole as well as their individual components can be put on different physical storage locations, including network based storages. For additional information, please refer to the article [[Using Network Attached Storage (NAS)]].
-Typical tasks regarding indexes are described in the following sections.
+<p class="msnote">'''Important notice: '''When choosing the physical storage system, please attend to the [[System Requirements]].</p>
-== Setting Up Indexing for the Contents of File Attachments ==
+== Creating Archive Stores ==
+In MailStore Server, internal archive stores are created automatically and additionally can be created manually. External archive stores can only be created manually.
-In the standard configuration, MailStore Server includes the file names of file attachments in the search indexes but not their contents. To enable MailStore Server to search the contents of file attachments, it has to be configured accordingly. Please proceed as follows:
+=== Creating Internal Archive Stores Automatically ===
+From time to time, MailStore Server automatically creates a new internal archive store for optimal system performance and stability. The maximum size and amount of emails are determined by MailStore Server internally.
-*Start ''MailStore Client'' and log on as administrator (''admin'').
+If needed, you can limit the size that an archive store can reach at most before a new one is created to align with your backup routine. You can also configure the storage location where new archive stores will be created automatically.
-*Click on ''Administrative Tools''.
-*Open the page ''Search Indexes''.
-*Click on ''Options''.
-*Separated by space, enter the file types (file extensions) whose contents MailStore Server is to include in the search indexes.
-*Click on OK to save the settings.
-*The new settings apply to all emails that are archived as of now. To apply the settings to already archived emails, rebuild the appropriate search indexes as described below.
-[[File:tech_index_02.png|center|450px]]
+<p class="msnote">'''Important notice: '''The maximum size should not be configured manually without good reason, so that MailStore Server can determine the optimal time automatically for creating a new archive store.</p>
-MailStore Server can index all file types for which a so-called IFilter driver is installed. When archiving, the drivers must be installed on the MailStore Client machine, when rebuilding indexes they must be installed on MailStore Server. Typically, IFilters exist at least for all applications which are installed on the respective machines.
+To change these settings, please proceed as follows:
-If, for example, Microsoft Office 2007 is installed, the corresponding IFilter drivers for Mirosoft Office documents are installed as well. IFilter drivers can also be installed independently from their applications. To learn more, please enter IFilter into any search engine.
+* Below the list of archive stores, click on the ''Create automatically...'' button.
+* [[File:tech_storageauto_01.png|right|350px]]The ''Auto-Create Archive Stores'' dialog opens.
+* Customize the settings as preferred.
+* Under ''Base directory'' enter the directory below which you want MailStore Server to create the new internal archive stores. The internal archive stores that are created automatically by MailStore Server, including any subfolders, follow the naming scheme ''year-month'', e.g. ''2021-10''.
+* ''Optional: ''With the option ''Use different base directories for databases, content and search indexes'' you can configure separate directories for the individual components of an archive store. For example, you can put database and search index on a fast storage to accelerate folder operations and the MailStore search while leaving the email content files on a slower storage.
+* Click on ''OK''.<br clear=all />
-For reasons of stability and performance, MailStore Server processes the following file types directly, regardless of the IFilter drivers that are installed:
+=== Creating an Internal Archive Store Manually ===
+To create a new internal archive store manually, please proceed as follows:
-*Text files (TXT),
+* Below the list of archive stores, click on the ''Create...'' button.
-*HTML files (HTM and HTML)
+* The ''Create New Archive Store'' wizard opens.
-*PDF files (PDF).
+* Select the option ''Internal Archive Store'' and click on ''Next''.
+* [[File:tech_storageloc_int_01.png|right|350px]]Enter a name for the new internal archive store in the ''Name'' field, e.g. ''2021-10''.
+* If you don't want MailStore to archive new emails in the new archive store, deselect the option ''Archive new messages here''.
+* MailStore Server derives a base directory for the new internal archive store from the name entered and the path of the master database. By default, MailStore Server stores all components of an archive store in a folder structure that is created automatically below the base directory. You can optionally either change the proposed path manually or select an existing directory; an existing directory must not contain any files or subfolders.
+* ''Optional: ''With the option ''Use different base directories for databases, content and search indexes'' you can configure separate directories for the individual components of an archive store. For example, you can put database and search index on a fast storage to accelerate folder operations and the MailStore search while leaving the email content files on a slower storage.
+* Click on ''Finish'' to create the internal archive store.<br clear=all />
-== Rebuilding Search Indexes ==
+=== Creating an External Archive Store ===
+External archive stores can only be created manually. Creating a new external archive store is described in chapter [[Using External Archive Stores]].
-If new indexing settings are to be applied to all existing emails or a message appears indicating that search indexes need to be rebuilt, please proceed as follows:
+== Managing Archive Stores ==
+[[File:tech_storageloc_01.png|center|550px]]
+=== Setting the Status ===
+Right-click on an archive store and select the status from the ''Set Status'' submenu. As an alternative, you can select an archive store and set the status via the drop-down list in the details pane on the right of the archive store list. You can set the following status:
-*Start MailStore Client and log on as administrator (admin).
+* ''Archive here''<br/>In MailStore Server, there can only be one archive store with this status. All newly archived emails are written into this archive store. The emails are available to all MailStore users and can be located by searching and through the folder structure. Emails in this archive store can be deleted or moved according to configured user privileges and compliance settings.<br/><br/><p class="msnote">'''Notice: '''If a new internal archive store has been created automatically, its status is initially set to ''Archive here''.<br/>If you set an external archive store's status to ''Archive here'', the automatic creation of internal archive stores will be suspended until you set an internal archive store to this status.</p>
-*Click on Administrative Tools.
+* ''Normal''<br/>Emails in archive stores with status ''Normal'' are available to all MailStore users and can be located by searching and through the folder structure. Emails in this archive store can be deleted or moved according to configured user privileges and compliance settings.
-*Open the page Search Indexes.
+* ''Write-Protected''<br/>Irrespective of user privileges and compliance settings, emails in write-protected archive stores can only be accessed read-only. The emails are available to all MailStore users and can be located by searching and through the folder structure. However, emails in such archive stores cannot be deleted or moved.<br/><br/><p class="msnote">'''Notice: '''Please note that file system write access to the directory of the archive store is still required and that this status prevents automatic processing of retention policies.</p>
-*Check all search indexes to be rebuilt. Indexes with the status Please Rebuild are already checked for your convenience.
+* ''Disabled''<br/>Disabling an archive store allows you to make changes to its configuration. This may be necessary after [[Moving the Archive]], for example. While an archive store is disabled, the emails contained therein are not available to the archive.<br/><br/><p class="msnote">'''Notice: '''Please note that this status prevents the execution of archiving profiles.</p>
-*Click on Rebuild Search Indexes.
-[[File:tech_index_01.png|center|450px]]
+=== Editing ===
+Through the context menu item ''Edit...'' you can change the name and the directories of an archive store with status ''Disabled''.
-== Frequently Asked Questions Regarding Search Indexes ==
+=== Renaming ===
+Through the context menu item ''Rename...'' you can change the name of an archive store irrespective of its status.
-'''Why are the contents of file attachments not indexed in the standard configuration?'''
+=== Detaching ===
+Through the context menu item ''Detach'' you can detach an archive store from the archive, for example if all email contained therein do no longer have to be archived. The detached archive store and the emails contained therein are no longer available to the archive.
-Indexing the contents of file attachments slows down the archiving process and increases the size of the search indexes significantly. Because of this, the contents of file attachments are not indexed in the standard configuration.
+=== Attaching ===
+A detached archive store can be reattached to the archive using the ''Attach...'' button. If the archive store cannot be decrypted automatically, you will be asked for the respective recovery key. For example, this could be the case for archive stores that belonged to another MailStore installation or which have been moved from another machine. You can find more information on the recovery key in section  [[MailStore_Server_Service_Configuration#Security_and_Encryption|Security and Encryption]] of chapter [[MailStore Server Service Configuration]].
-'''Everything was set up correctly but I still cannot find some text within the archived file attachments. What could be the reason?'''
+=== Unlocking ===
+If an archive store cannot be decrypted automatically, it will be listed as ''Locked'' in the list of archive stores. Through the context menu item ''Unlock'' or by changing the status you will be asked for the respective recovery key. You can find more information on the recovery key in section  [[MailStore_Server_Service_Configuration#Security_and_Encryption|Security and Encryption]] of chapter [[MailStore Server Service Configuration]].
-First, this may be because, due to performance and stability, MailStore Server only indexes the first 10,000 Words (not characters!) of file attachments. In most cases, keywords are contained within the first 10,000 Words. If an email has multiple file attachments, at most 10,000 words are indexed overall. Of course, the file attachments themselves are archived in their entirety and can be searched in full with the appropriate application.
+== Maintenance of Archive Stores ==
+All available maintenance commands can be accessed through the context menu of the list of archive stores. Alternatively, you can select an archive store and click on the ''Maintenance'' drop-down list in the details pane. The following commands are available:
-Another reason could be that MailStore Server was unable to read the corresponding file because no IFilter driver was available or because the file was DRM protected (copy protection), for example.
+* ''Cleanup (Compact)''<br/>Optimizes the data structures while compacting the data.
+* ''Check Data Integrity''<br/>Verifies the data integrity between "Folder Information and Meta Data" as well as "Email Headers and Contents".
+* ''Maintain All FDB Files''<br/>Maintains the master database and all databases of internal archive stores.
+* ''Recalculate all statistics''<br/>Recalculates the statistics (number of emails per archive store) for all archive stores.
-'''In the Status column, ''OK'' (Compatibility mode) is displayed. What should be done?'''
+Maintenance commands can also be scheduled for automatic execution via [[Jobs]].
-The search index was created with an older version of MailStore Server and has an internal structure different from indexes which were built using the current version. No action is required. However, these indexes can be upgraded by rebuilding them. This significantly increases indexing and search performance.
 [[de:Speicherorte]]
+[[en:Storage Locations]]