mailing list archives

Now I'm curious about the various mailbox hierarchy conventions that
may exist.

I'm reading a lot on the Web about Apple Mail and its notion that all
mailboxes be sub-mailboxes of INBOX, apparently because a popular IMAP
server does it that way.

I'm re-exporting an MH mail folder setup, with a structure that looks a lot
like

   ...
   DL/python
   DL/python/dev
   DL/python/email
   ...
   work/meetings
   work/papers
   work/projects
   work/projects/imap-server
   ...

and so on and so forth.  The MH message ID numbers are the UIDs for IMAP,
I'm using a PyLucene index of all the mail for searching, etc.

My plan was to basically represent each of these MH folders as an IMAP
mailbox, and add another top-level mailbox called "INBOX", which would
normally be empty (there's a procmail daemon that sorts the mail).  So
there's a single namespace and there's no root for reference names.

  C:  2 list "" ""
  S:  * LIST (\Noselect) "/" ""
  S:  2 OK LIST completed
  C:  3 list "INBOX" ""
  S:  * LIST (\Noselect) "/" ""
  S:  3 OK LIST completed

This seems like a valid setup, from RFC 3501.  But am I setting myself
up for grief from clients?  Would life be simpler if I made everything
be a sub-mailbox of INBOX?

Bill

On Sun, 11 Mar 2007, Bill Janssen wrote:
> I'm reading a lot on the Web about Apple Mail and its notion that all
> mailboxes be sub-mailboxes of INBOX, apparently because a popular IMAP
> server does it that way.

I suggest that you concern yourself with complying with IMAP, and not 
try to program for the whims of an individual client.  I did not find 
Apple Mail to be a satifactory IMAP client.

> I'm re-exporting an MH mail folder setup

One caution is that UIDs need to be immutable, or at least you need to 
change the UIDVALIDITY when they change.  This is a problem if you use MH 
message ID numbers for UIDs, because the mh pack and sort operations will 
renumber the message ID numbers.

As distributed, UW imapd simply returns a new UIDVALIDITY with each new 
session for an mh mailbox.  This is a problem for disconnected clients 
which use the UIDs to determine if it must download the message or if it 
already has it cached.

Lauren Weinstein has a hacked version of UW imapd, although with a hacked 
version of mh, that I helped him with that has sticky UIDs and changes the 
UIDVALIDITY only when a pack/sort is done.  I don't know his current 
status; he was using a mobile client that was quite embryonic and had lots 
of growing pains, thus there were multiple unknowns when debugging all 
this.

> My plan was to basically represent each of these MH folders as an IMAP
> mailbox, and add another top-level mailbox called "INBOX", which would
> normally be empty (there's a procmail daemon that sorts the mail).  So
> there's a single namespace and there's no root for reference names.

As I understand it, this is alright.

Have you thought about trying UW imapd, even if you plan to deploy your 
own implementation, just so you have reference to compare?

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On 12.3.2007, at 1.48, Bill Janssen wrote:

> I'm reading a lot on the Web about Apple Mail and its notion that all
> mailboxes be sub-mailboxes of INBOX, apparently because a popular IMAP
> server does it that way.

I'm not completely sure what you mean by this. If the server lists  
all mailboxes under INBOX and you haven't set "IMAP Path Prefix" to  
INBOX/ in Apple Mail, then it will list all the mailboxes under  
INBOX. But I don't think this is the intended way to use it.

My server doesn't put mailboxes under INBOX, and I'm using Apple Mail  
just fine. The non-INBOXes are listed under a separate "tss@iki.fi"  
item.

One Apple Mail bug that I have heard of is that it uses '/' as the  
hierarchy separator even though server uses something else. But I  
haven't been able to reproduce that bug for some reason.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 193 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070312/3b5807f9/attachment.sig>

Mark Crispin writes:
> On Sun, 11 Mar 2007, Bill Janssen wrote:
> > I'm reading a lot on the Web about Apple Mail and its notion that all
> > mailboxes be sub-mailboxes of INBOX, apparently because a popular IMAP
> > server does it that way.
> 
> I suggest that you concern yourself with complying with IMAP, and not 
> try to program for the whims of an individual client.  I did not find 
> Apple Mail to be a satifactory IMAP client.

I certainly understand your feelings here, but Apple Mail is the
client I'm trying to support, with all its imperfections.  As for
complying with IMAP, sure, I do indeed want to do just that; but where
there are unspecified things (like how to arrange mailboxes), I'd just
as soon do it the way that some other folks do it, to smooth the path.

> One caution is that UIDs need to be immutable, or at least you need to 
> change the UIDVALIDITY when they change.  This is a problem if you use MH 
> message ID numbers for UIDs, because the mh pack and sort operations will 
> renumber the message ID numbers.

Thanks for that reminder.

> Have you thought about trying UW imapd, even if you plan to deploy your 
> own implementation, just so you have reference to compare?

Good idea.  I hadn't realized that it supported MH folders.  I'll do that.

Bill

On Sun, 11 Mar 2007, Bill Janssen wrote:
> I certainly understand your feelings here, but Apple Mail is the
> client I'm trying to support, with all its imperfections.

I guess that I wasn't clear.  The problem with implementing a server for a 
particular client, as opposed to the specification, is that the server 
ends up complying with the server author's understanding of what the 
client does.  That is not necessarily the same as what the client actually 
does; and it tends to compound mistakes.

> where
> there are unspecified things (like how to arrange mailboxes), I'd just
> as soon do it the way that some other folks do it, to smooth the path.

This is a good example.  What you stated was your understanding (and 
perhaps others' understanding) of what Apple Mail does.  As unsatisfactory 
as I find Apple Mail to be, the fact is that if it really worked that way 
it would not work at all with multiple widely-deployed servers.

Put another way, you expressed a misunderstanding.  This happens all the 
time.

That's why I advise that you disregard all such (mis)understandings and 
implement only according to the specification.  You'll find that the 
majority of the (mis)understandings are overtaken by events -- the "that's 
strong, I thought it did such-and-such and I was going to have to do this 
workaround for it.  Well, it seems to work the way it is, so I'm leaving 
it alone!" scenario.

Undoubtably, when you finish the server there will be interoperability 
issues.  However, I recommend that you first consult with this mailing 
list, since it might be a bug at your end.  Only after we get through all 
of that, then you can start thinking about workaround for a particular 
client.

It's the whole notion of avoiding going into the swamp unh

>> Have you thought about trying UW imapd, even if you plan to deploy your
>> own implementation, just so you have reference to compare?
> Good idea.  I hadn't realized that it supported MH folders.  I'll do that.

Yup.  mh names are in the #mh/ namespace, and the #mhinbox (with snarfing 
from the spool directory) is #mhinbox.  It's armed by the .mh_profile 
file.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Sun, 2007-03-11 at 17:37 -0700, Mark Crispin wrote:
> Undoubtably, when you finish the server there will be interoperability 
> issues.  However, I recommend that you first consult with this mailing 
> list, since it might be a bug at your end.  Only after we get through all 
> of that, then you can start thinking about workaround for a particular 
> client.

My server allows enabling client workarounds individually (default is
only outlook-idle, which is the same as what UW-IMAP also does). There
aren't all that many, here's a list of all of them:

  #   delay-newmail:
  #     Send EXISTS/RECENT new mail notifications only when replying to NOOP
  #     and CHECK commands. Some clients ignore them otherwise, for example
  #     OSX Mail. Outlook Express breaks more badly though, without this it
  #     may show user "Message no longer in server" errors. Note that OE6 still
  #     breaks even with this workaround if synchronization is set to
  #     "Headers Only".
  #   outlook-idle:
  #     Outlook and Outlook Express never abort IDLE command, so if no mail
  #     arrives in half a hour, Dovecot closes the connection. This is still
  #     fine, except Outlook doesn't connect back so you don't see if new mail
  #     arrives.
  #   netscape-eoh:
  #     Netscape 4.x breaks if message headers don't end with the empty "end of
  #     headers" line. Normally all messages have this, but setting this
  #     workaround makes sure that Netscape never breaks by adding the line if
  #     it doesn't exist. This is done only for FETCH BODY[HEADER.FIELDS..]
  #     commands. Note that RFC says this shouldn't be done.
  #   tb-extra-mailbox-sep:
  #     With mbox storage a mailbox can contain either mails or submailboxes,
  #     but not both. Thunderbird separates these two by forcing server to
  #     accept '/' suffix in mailbox names in subscriptions list.

The "OSX Mail" in delay-newmail was added quite a long time ago. I don't
know if it has been fixed since, maybe I should check.

One workaround that there used to be was that several clients (kmail,
Thunderbird at least) didn't like if UID field wasn't the first field in
FETCH replies. After a while I gave up and just put it back to begin
first.

I'm not aware of any other client bugs that can be worked around.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070312/37cae1b0/attachment.sig>

Thanks, that's a nice list.

> One workaround that there used to be was that several clients (kmail,
> Thunderbird at least) didn't like if UID field wasn't the first field in
> FETCH replies. After a while I gave up and just put it back to begin
> first.

Didn't like it, how?  I'm sending it last (on a UID FETCH if it's not
explicitly specified), and Thunderbird seems happy.  I send the specified
items on a FETCH in the order they were requested.

Bill

On 12.3.2007, at 5.56, Bill Janssen wrote:

> Thanks, that's a nice list.
>
>> One workaround that there used to be was that several clients (kmail,
>> Thunderbird at least) didn't like if UID field wasn't the first  
>> field in
>> FETCH replies. After a while I gave up and just put it back to begin
>> first.
>
> Didn't like it, how?  I'm sending it last (on a UID FETCH if it's not
> explicitly specified), and Thunderbird seems happy.  I send the  
> specified
> items on a FETCH in the order they were requested.

Hmm. It probably was enough if the UID was in the "* FETCH" line. At  
least if the message body was before UID it didn't like that. And by  
not liking I mean it somehow mixed it up with previous message,  
causing at least the message's size field to be wrong.

Looks like I'm now sending the items also in the requested order,  
except message headers and body come last always.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 193 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070312/f2fcbdb8/attachment.sig>

Timo Sirainen writes:
> Hmm. It probably was enough if the UID was in the "* FETCH" line.

No, someone else reported problems with that a couple of years ago. It 
has to be at (or near?) the start.

I haven't investigated. I'd rather put UID first than Investigate 
Outlook bugs. ?This may or may not be what's going on: life is too 
short to spend debugging Intel parts so I really don't care to 
investigate further.?

Arnt

On Mon Mar 12 06:14:24 2007, Timo Sirainen wrote:
> On 12.3.2007, at 5.56, Bill Janssen wrote:
> 
>> Thanks, that's a nice list.
>> 
>>> One workaround that there used to be was that several clients 
>>> (kmail,
>>> Thunderbird at least) didn't like if UID field wasn't the first  
>>> field in
>>> FETCH replies. After a while I gave up and just put it back to 
>>> begin
>>> first.
>> 
>> Didn't like it, how?  I'm sending it last (on a UID FETCH if it's 
>> not
>> explicitly specified), and Thunderbird seems happy.  I send the  
>> specified
>> items on a FETCH in the order they were requested.
> 
> Hmm. It probably was enough if the UID was in the "* FETCH" line. 
> At  least if the message body was before UID it didn't like that. 
> And by  not liking I mean it somehow mixed it up with previous 
> message,  causing at least the message's size field to be wrong.
> 
> Looks like I'm now sending the items also in the requested order,  
> except message headers and body come last always.

I've understood the following to be legal, but I suspect it'd cause 
much breakage:

C: tag UID FETCH u BODY[1]
S: * s FETCH BODY [1] {n}
S: ...
S: * s FETCH UID u
S: tag OK

Certainly in my client's case I'm not sure how well the code-path 
that copes with the first will work. It'd handle it if it happened to 
know which UID message number s had already, but otherwise I'm really 
not sure. Mind you, I'm also not sure that my client could ever get a 
body part without knowing the UID.

I'm pretty sure that sending any implicit UID first, then everything 
else all on the same FETCH response, is not going to break anyone.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 2007-03-12 at 09:53 +0000, Dave Cridland wrote:
> > Hmm. It probably was enough if the UID was in the "* FETCH" line. 
> > At  least if the message body was before UID it didn't like that. 
> > And by  not liking I mean it somehow mixed it up with previous 
> > message,  causing at least the message's size field to be wrong.
> > 
> > Looks like I'm now sending the items also in the requested order,  
> > except message headers and body come last always.
> 
> I've understood the following to be legal, but I suspect it'd cause 
> much breakage:
> 
> C: tag UID FETCH u BODY[1]
> S: * s FETCH BODY [1] {n}
> S: ...
> S: * s FETCH UID u
> S: tag OK

I'm not sure about "FETCH .. (UID BODY[1])", but for "UID FETCH" I think
that's pretty clearly invalid:

      The number after the "*" in an untagged FETCH response is always a
      message sequence number, not a unique identifier, even for a UID
      command response.  However, server implementations MUST implicitly
      include the UID message data item as part of any FETCH response
      caused by a UID command, regardless of whether a UID was specified
      as a message data item to the FETCH.

Anyway what I meant with my "* FETCH line" comment was that I used to do
this:

C: tag FETCH s (UID BODY[])
S: * s FETCH (BODY[] {n}
S: ... UID u)
S: tag OK

And that was breaking clients. I suppose when they saw BODY[] they
wanted to immediately associate it with some UID before reading anything
after the literal.

> I'm pretty sure that sending any implicit UID first, then everything 
> else all on the same FETCH response, is not going to break anyone.

Right. :)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070312/64343842/attachment.sig>