Indexing Mail
Mailpile is built around a search engine and before you can read or sort your e-mail, Mailpile has to add it to the local search index. Usually this will happen automatically once you have configured Mailpile with the details of one or more accounts.
However, many power users already have existing e-mail on their hard drives or have specific opinions about how they would like their mail to be processed. If you are one of those users, read on to learn a bit more about how Mailpile processes mail and how you can customize and control the experience.
In this guide:
Important: Low-level configuration tools do not yet exist for the web interface of Mailpile; in order to benefit from this guide you will need to make use of the Mailpile command-line interface. Changing these settings is NOT without risk - you can break your Mailpile if you get things wrong here!
Discovering and indexing new mail happens in a few stages.
- Configuration: create mail sources
- Mailbox discovery: apply discovery policies
- Message copying: update local mailboxes
- Message indexing: record metadata and search terms, apply tags
Configuration is accomplished either by using the Add Account wizard, or manually as described in this document. This tells Mailpile to look for mail and what to do with the mail it finds. You will generally create one or more mail sources, each of which describes a remote IMAP or POP3 account or a set of mailboxes already on your local computer.
Mailbox discovery is a process whereby Mailpile will automatically discover which folders exist on an IMAP server or which mailboxes exist in a local directory. This happens periodically, so if a new folder or mailbox is created, Mailpile can automatically react and make that mail available for use. How exactly this happens is decided by the discovery policy of the relevant mail source.
Message copying is the act of downloading mail from a remote server to Mailpile's (usually encrypted) local mail store. This step may be skipped, but is generally recommended.
Indexing is the final stage, when each e-mail is individually read and added to Mailpile's index. The metadata index will store things like the date, subject and sender. Tags will be applied based on global filter rules, auto-tagging, as well as any configuration of the mail source itself. The search index will be updated using keywords extracted from the message header, body and attachments. This stage may involve decrypting the mail or processing large attachments, and may be somewhat resource intensive as a result.
Indexing happens on a per-mail-source basis and each mail source keeps its own schedule depending on the type of mailbox, network connectivity and other factors.
The status of each mail source is recorded in the Event Log; eventually there should be nice interfaces to browse this data, but for now you can take a look either by visiting /api/0/eventlog/incomplete/ in your browser or by running the eventlog incomplete command at the mailpile> prompt. In the latter case, piping through less is advised:
mailpile> pipe less eventlog incomplete
...
If you are impatient and want a round of indexing to happen right now, you can trigger it using the rescan command:
# Rescan everything, not just mail
mailpile> rescan full
# Rescan only mail sources
mailpile> rescan sources
...
# Trigger a rescan of mailbox 000b only
mailpile> rescan mailbox:000b
...
Manually configuring mail sources is a large topic, arguably one of the most complex parts of Mailpile.
Mail source settings live in two sections of the Mailpile configuration tree; sys.mailbox and sources. You will need to examine both of these structures to get a full understanding of how your Mailpile is configured. An example is presented and discussed below.
First, it's useful to get a short list of what sources exist:
mailpile> print -flat sources
{
"sources": {
"abcdefg": "{ ... }",
"1234xyz": "{ ..(Local mailboxes).. }",
"999000x": "{ ..(Personal GMail).. }"
}
}
Next, let's examine one in more detail, using less:
mailpile> pipe less print sources.abcdefg
{
"sources.abcdefg": {
"name": "Personal GMail",
"profile": "random-vcard-id-string",
"enabled": true,
"host": "imap.gmail.com",
"interval": 300,
"keepalive": false,
"auth_type": "password",
"username": "[email protected]"
"password": "(SUPPRESSED)",
"port": 993,
"post_command": "",
"pre_command": "",
"protocol": "imap_ssl",
Above we have settings pertaining to the source of mail; this is an IMAP source (GMail) which is linked to a profile (an account). The source checks for new mail every 300 seconds (interval: 300) and disconnects after each batch (keepalive: false).
Next we see the discovery policy. Whenever a new folder is discovered on the IMAP server, an entry is added to the mailboxes section below and it is configured in accordance with this policy. In this case, the default is to apply the policy "unknown" which effectively ignores the new folder:
"discovery": {
"apply_tags": {},
"create_tag": true,
"guess_tags": true,
"local_copy": true,
"parent_tag": "27",
"paths": {},
"policy": "unknown",
"process_new": true,
"visible_tags": false
},
"mailbox": {
"000w": {
"apply_tags": {},
"local": "/home/USER/.local/share/Mailpile/default/mail/12345",
"name": "MyGmailTag",
"path": "@000w",
"policy": "unknown",
"primary_tag": "Personal GMail/Baronsstigur39",
"process_new": true
},
Above we can see an IMAP folder with the id 000w, below is another with the ID 0011. The first is unconfigured (policy: unknown), in accordance with the discovery policy.
The second (the INBOX) has however been manually configured for reading mail, keeping local copies at a specific path, applying both tags 28 and a to all mail, and processing mail as "new", which means the unread flag will be set if necessary.
"0011": {
"apply_tags": {
"0000": "a"
},
"local": "/home/USER/.local/share/Mailpile/default/mail/67890",
"name": "INBOX",
"path": "@0011",
"policy": "read",
"primary_tag": "28",
"process_new": true
}
}
}
}
Finally, we can examine some of the resources mentioned in the settings above:
# Examine the sys.mailbox entry for the INBOX
mailpile> print sys.mailbox.0011
{
"sys.mailbox.0011": "src:sdpslfj/INBOX"
}
# Examine tag "a":
mailpile> print tags.a
...
# Examine the VCard for the linked profile
mailpile> vcard random-vcard-id-string
...