I managed to dig up some information on the IMAP.MRK file, for anyone brave enough to want to modify it pragmatically. The format is actually fairly simple, one header followed by zero or more message records.
If a MSG file has no corresponding record, MDaemon will update the IMAP.MRK file the next time an IMAP client or WorldClient user touches the folder, and the message will be treated as UNSEEN and UNREAD
The header is defined as follows:
struct IMAPMrkHeader
{
unsigned HeaderVersion;
unsigned UIDValidity;
unsigned UIDNext;
unsigned LastWriteCounter;
unsigned Filler0;
unsigned Filler1;
unsigned Filler2;
unsigned Filler3;
unsigned CRLF;
};
Each MSG file will have one record, which is defined as follows:
struct IMAPMrkMessage {
char Filename[MAX_IMAP_FILENAME];
unsigned char Flags;
unsigned UID;
unsigned Size;
time_t Date;
};#define FLAG_SEEN 32
#define FLAG_ANSWERED 16
#define FLAG_FLAGGED 8
#define FLAG_DELETED 4
#define FLAG_DRAFT 2
#define FLAG_RECENT 1
#define MAX_IMAP_FILENAME 23
#define IMAP_RECORD_SIZE 36
So what do all those fields mean?
- HeaderVersion is a tag to identify what file format the IMAP.MRK is using. The current value is 1. UIDValidity is the IMAP UID Validity for the folder. UIDNext is the UID that will be used for the next message added to the folder.
- LastWriteCounter is a value that changes whenever something has changed in the file. This was added when IDLE support was added to the IMAP server, so that there’s a quick way to check for changes (by comparing this one value rather than doing a compare of the entire file’s contents).When changing records in the file, lock the IMAP folder, increment the LastWriteCounter, change the records, then unlock the folder.
- UIDNext is the next UID to be assigned. To add records, lock the IMAP folder, increment the LastWriteCounter, and for each record you’re adding, use the current UIDNext value as its UID and then increment UIDNext.
So how do you lock a folder?
- Lock an IMAP folder by creating a file “IMAP-foldername-email.lck”, where foldername is a “filename-safe” version of the folder name (” and ‘/’ characters replaced with ‘_’) and email is the owner’s email address (use “public” for public folders). If the lock file already exists, something is currently operating on the IMAP.MRK so you should wait until it is unlocked before you lock it and continue.
Hope this helps someone.
IMAP.MRK file format by Dave Warren (everything-mdaemon.com) is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
very usefull thanx, but i got one question.
How whould i read the imap.mrk ?
In case i would like to store somewhere the information for a user, if a message is unread or read , deleted or forwarded.
In mdaemon rules there isn’t any command for reading the imap.mrk file. So how i can open it, and then fetch the information from inside? ( I already try the notepad, but probably is somehow Db File)
Thanx for your time.
There aren’t any tools included with MDaemon (other than MDaemon and WorldClient itself) that can read IMAP.MRK.
In fact, the point of this article is, in part, to give you the information you need to write your own such tool.
If you just want to view a mailbox, share it from the MDaemon GUI and use IMAP or WorldClient to access it.
In case i need to write a program that to read the flag information from this file, it could be difficult ? can you provide some hint ? some source code? or any available website that can help with this ?
Thank you
There’s nothing particularly difficult about it, the blog article contains the header information in C format “struct”
One other thought: the IMAP standard has no “Forwarded” flag, so although you can get “Replied” or not, you can’t get forwarded status as this isn’t known to MDaemon.
Extremely interesting information that I could not find anywhere else, thanks Dave! One question, do you have an idea what might be the issue if MDaemon stopped writing or reading the flags correctly, with the result that messages stay ‘unread’ forever? The IMAP.MRK is being updated, just the messages are not getting flagged. As this happens only from a specific message onwards, there seems to be a glitch that maybe can be ironed out somehow, anything that “historically” comes to your mind that could be tried in such cases? Many thanks!
Hi Chris,
One thing that comes to mind, are you using the normal filename format for messages, or did you rename them? While you don’t need to use the standard name, if you exceed the size of the MAX_IMAP_FILENAME 23 field, MDaemon will truncate the filename when it writes flags to IMAP.MRK, and then the next time the folder is scanned, the (untruncated) filename isn’t found and therefore the message gets the UNREAD and UNSEEN flags (and the truncated filename found in IMAP.MRK doesn’t exist as a file on disk, so it’s removed from the IMAP.MRK automatically)
tl;dr: MDaemon forgets flags for messages with filenames longer than 23 characters. MDaemon won’t create these files itself, but if you manipulate the filesystem yourself (for example, to merge folders without filename conflicts), you can create names like this yourself.
If this describes your situation, there are tons of methods/tools to fix it, but I use Bulk Rename Utility for this type of task. It’s a monster to get started with, but you could set “Name (2)” to “XX” and then “Numbering (10)” to mode Suffix to append a sequential number. Just take care to not rename MDaemon’s MD*.MSG files or you’ll wipe the flags for all messages.
Hopefully at least some of that makes sense, if not, let me know?
Hi Dave,
thank you, very interesting insight, but unfortunately, the files and directories are untouched, so do not go beyond this filename’s length limit…. but do you know if there is a limit of how many messages the IMAP:MRK can handle? The mailbox contains approx. 500k messages…
Thanks!
I don’t know if any hard limit, but 500K is a lot (especially at the NTFS level), it wouldn’t surprise me if there is some point where it breaks.
You could test by copying the MSG files to a temporary account and see if you can reproduce it, if so, delete 100K of messages that do work and see if you suddenly gain the ability to set flags.
Stupid question, but just to make sure, can you star or mark an older message and does it stick?
I tried to move 200k messages out of the folder, but this unfortunately didn’t solve it – they are still not being flagged. And I tried to mark/star an older message from times where flagging still worked, and it also isn’t being flagged now. But, such older messages/flags are still showing correctly eg. as SEEN, they just cannot be changed anymore to eg. ANSWERED. The weird thing is, that new messages are constantly being added to the imap.mrk as they come in, but only flagging isn’t working (I was wondering about this initially as I was assuming NTFS permissions might had broken down at some point) …
I’m at a bit of a loss. It wouldn’t be hard to set up a test account with 500K messages to see if I can reproduce. Want me to give it a try, or did you find a workaround/solution?