mailing list archives

I am seeking clarification on the following paragraph that appears in 
the LIST command section of RFC3501:

"The special name INBOX is included in the output from LIST, if INBOX is 
supported by this server for this user and if the uppercase string 
"INBOX" matches the interpreted reference and mailbox name arguments 
with wildcards as described above. The criteria for omitting INBOX is 
whether SELECT INBOX will return failure; it is not relevant whether the 
user's real INBOX resides on this or some other server."

 From this, can one conclude that a server implementation MAY not return 
the INBOX in a LIST such as:

C: a004 LIST "" Inbox
S: a004 OK LIST completed

when it does return it in a LIST such as:

C: a004 LIST "" INBOX
S: * LIST (\Unmarked) "/" INBOX
S: a004 OK LIST completed

Secondly, take a scenario where there are three mailboxes: INBOX, INBOX1 
and InBox1, and the client does a LIST:

C: a004 LIST "" In%B%

Assuming the implementation is case-sensitive, what must the server 
return? More importantly, can it NOT return the INBOX? I realize this 
example is a bit contrived, but it may help in the clarification.

Thank you,

Nevon

In the case of
 	a004 LIST "" Inbox
the case-insensitivity of INBOX overrides everything else.

However, in the case of
 	a004 LIST "" In%B%
it is arguable given the wording in RFC 3501 that this could be 
interpreted as not matching INBOX.  However, I think that an 
implementation should.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Nevon,

Yes, I think it's better to discuss these things here.

The example I sent (with In%B%) was just illustrative.

Please note that the LIST parameter is NOT a mailbox name, it's a pattern 
(though the standard uses an informal language - "mailbox name with possible 
wildcards").

The standard is pretty clear about these things:

       The special name INBOX is included in the output from LIST, if
       INBOX is supported by this server for this user and if the
       uppercase string "INBOX" matches the interpreted reference and
       mailbox name arguments with wildcards as described above.


The "upper case string "INBOX"" does NOT match neither the "Inbox" pattern, 
not the "In%b%" pattern. As a result, it will not be included into the list 
issued by an IMAP server that follows the published standard.

On the other hand, the name MAY be included in the output, if the server 
uses case-insenstive names (as you could observe using a CommuniGate Pro 
server running under Windows OS) - but in that case the INBOX name is 
included as a regular mailbox name, not as a "special name INBOX" which is 
included only under the conditions specified above.



Sincerely,
Vladimir


On Mon, 10 Apr 2006 13:00:53 -0230
  Nevon Brake <nevon_brake@consilient.com> wrote:
> I am seeking clarification on the following paragraph that appears in the 
>LIST command section of RFC3501:
> 
> "The special name INBOX is included in the output from LIST, if INBOX is 
>supported by this server for this user and if the uppercase string "INBOX" 
>matches the interpreted reference and mailbox name arguments with wildcards 
>as described above. The criteria for omitting INBOX is whether SELECT INBOX 
>will return failure; it is not relevant whether the user's real INBOX 
>resides on this or some other server."
> 
> From this, can one conclude that a server implementation MAY not return 
> the INBOX in a LIST such as:
> 
> C: a004 LIST "" Inbox
> S: a004 OK LIST completed
> 
> when it does return it in a LIST such as:
> 
> C: a004 LIST "" INBOX
> S: * LIST (\Unmarked) "/" INBOX
> S: a004 OK LIST completed
> 
> Secondly, take a scenario where there are three mailboxes: INBOX, INBOX1 
>and InBox1, and the client does a LIST:
> 
> C: a004 LIST "" In%B%
> 
> Assuming the implementation is case-sensitive, what must the server 
>return? More importantly, can it NOT return the INBOX? I realize this 
>example is a bit contrived, but it may help in the clarification.
> 
> Thank you,
> 
> Nevon
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> https://mailman1.u.washington.edu/mailman/listinfo/imap-protocol

Vladimir's interpretation is valid.  The LIST argument is definitely not a 
mailbox name.

However, I disagree about it being useful or desirable compared to other 
interpretations.  In particular, I do not think that it is useful or 
desirable for a server to respond to
 	tag LIST "" inbox
with no data because of case-mismatch with INBOX.

The case of
 	tag LIST "" In%B%
is less clear, particularly given that the specification says "uppercase 
string" instead of "case-insensitive" string.

However, the kicker is that (as Vladimir notes) INBOX may be included in 
the output with a case-insensitive server.  I disagree with Vladimir's 
interpretation that the specification forces INBOX to be case-sensitive in 
LIST even on a case-insensitive server, as well as the notion of a 
"regular mailbox named INBOX" vs. a "special name INBOX" (that is an 
implementation detail).

Fortunately, though, this discussion is more academic than practical, for 
the following reasons:

The specification uses "INBOX" throughout as the canonical form of the 
name, and hammers on the point that this is case-insensitive.  Postel's 
robustness principle (this being a correct use!) says that clients should 
always send the name as "INBOX", and servers should always do a 
case-insensitive match even when the specification allows a loophole.

In the case of a client sending
 	tag LIST "" inbox
and getting nothing back, the primary onus is on the client for not using 
the canonical form.  The server, arguably, is being naughty too; but the 
loophole exists.

Although the In%B% case is more troubling, I observe it is not something 
that generally occurs in real life.  A client which actually uses such a 
pattern is probably looking for names other than INBOX; but I don't think 
that I've ever seen a client use such a pattern.

I think that, although a server should match INBOX to In%B%, Vladimir is 
correct that it does not have to.  Even with Vladimir's interpretation, it 
is ambiguous (due to underlying implementation details) whether or not the 
server would return it.  So a client should not depend upon the behavior 
either way.

I think that we all agree that the following patterns all match INBOX:
 	*
 	%
 	INBOX
 	INBOX%
 	INBOX*
and the following patterns do NOT match INBOX:
 	%/%
 	foo
 	foo/%
 	INBOX/*
 	INBOX/%

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

In conclusion, I consider Vladimir's interpretation to be an interesting 
quirk in the protocol.  It is surprising, and perhaps not what most people 
would choose; but nonetheless it is valid.

But, fortunately, this quirk is primarily of academic interest since a 
client should not do anything that would get it into trouble with this 
quirk.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, 10 Apr 2006 11:14:08 -0700 (PDT)
  Mark Crispin <mrc@cac.washington.edu> wrote:
> Vladimir's interpretation is valid.  The LIST argument is definitely not a 
>mailbox name.
> 
> However, I disagree about it being useful or desirable compared to other 
>interpretations.  In particular, I do not think that it is useful or 
>desirable for a server to respond to
> 	tag LIST "" inbox
> with no data because of case-mismatch with INBOX.

One of the biggest problem with any protocol are "useful exceptions". It has 
been proven too many times that whereever a problem (or misunderstanding) 
can emerge, it will emerge.
  
> The case of
> 	tag LIST "" In%B%
> is less clear, particularly given that the specification says "uppercase 
>string" instead of "case-insensitive" string.
> 
> However, the kicker is that (as Vladimir notes) INBOX may be included in 
>the output with a case-insensitive server.  I disagree with Vladimir's 
>interpretation that the specification forces INBOX to be case-sensitive in 
>LIST even on a case-insensitive server

I did not say that. I've said that (according to the written standard):

a) when interpreting a command with a mailbox name parameter, the server 
should interpret any combination of the InBoX letters as a reference to the 
INBOX name (which is supposed to be a [virtual] name for the mailbox 
accepting incoming mail).

b) when interpreting the LIST command and its argument (which is a pattern, 
not a mailbox name), the special "incoming" mailbox name matches it if the 
name INBOX (all caps) matches the specified pattern.

If a server happens to use case-insensitive names, and, as a result, all 
operations (including the pattern match) are case insensitive, then search 
results can be different when compared to case-sensitive systems.

The IMAP protocol does not explicitly specify what the REAL NAME of the 
"incoming mailbox" is. If it would specify that the name is INBOX, it would 
eliminate a lot of misunderstanding. Unfortunately, it does not.

> as well as the notion of a "regular 
>mailbox named INBOX" vs. a "special name INBOX" (that is an implementation 
>detail).

>Fortunately, though, this discussion is more academic than practical, for 
>the following reasons:

Unfortunately, it's not the case. The initial request was sent to our 
support group because the author of the question used "Inbox" as the 
reference to the incoming mailbox. And while it works fine with all SELECT & 
Co commands, it worked OK with LIST only because it was used on a 
case-insensitive platform. Switching to a case-sensitive platform revealed 
the problem.
  
> The specification uses "INBOX" throughout as the canonical form of the 
>name, and hammers on the point that this is case-insensitive.  Postel's 
>robustness principle (this being a correct use!) says that clients should 
>always send the name as "INBOX", and servers should always do a 
>case-insensitive match even when the specification allows a loophole.

This is exactly the answer we have given: using any name instead of INBOX is 
a call for trouble. BUT: there is nothing in the IMAP specs that explicitly 
says so. And many people who do not know (and have no reason to know) about 
all historical details behind the name "INBOX" in all caps, may conclude 
that it's OK to use Inbox as that special mailbox name, especially because 
many UI applications render it this way.

> In the case of a client sending
> 	tag LIST "" inbox
> and getting nothing back, the primary onus is on the client for not using 
>the canonical form.  The server, arguably, is being naughty too; but the 
>loophole exists.
> 
> Although the In%B% case is more troubling, I observe it is not something 
>that generally occurs in real life.  A client which actually uses such a 
>pattern is probably looking for names other than INBOX; but I don't think 
>that I've ever seen a client use such a pattern.
> 
> I think that, although a server should match INBOX to In%B%, Vladimir is 
>correct that it does not have to.  Even with Vladimir's interpretation, it 
>is ambiguous (due to underlying implementation details) whether or not the 
>server would return it.  So a client should not depend upon the behavior 
>either way.

I'd suggest we put all that "reasonable" stuff aside: the standards are the 
standards are the standards. They are created to switch from "reasonable" 
(which is, by definition, different for different souls) to [well-defined] 
"standard" .

The question here is simple:
if
a LIST "" "%n%ox"
should show any name for the incoming mailbox should it
show INBOX? Or Inbox? Or InBox? Or InBOx?

Consider the following:
a) the server should always return the mailbox name in the same "case". If 
the server itself returns INBOX in one response and Inbox in others, it will 
only create confusion (again, a note that this special mailbox DOES have a 
well-defined name, and that name is INBOX would help a lot). The answer - if 
it should return anything, it should return INBOX.

b) If (a) is correct, then the question is:

should
a LIST "" "%n%ox"
return
* INBOX

There is no reason why it should - on a case-sensitive system (on a 
case-insensitive - it can and should). The only special rule about the INBOX 
name that can be used here explicitly says that the upper-case word INBOX 
should be compared against the pattern. On a case-senstive system it will 
not match the pattern. So, the answer is "no".

c) If (b) is correct, then the answer to:
should
a LIST "" "Inbox"
return
* INBOX

is clear: on a case-insensitive system - yes, on a case-sensitive - no. The 
reason is simple: "Inbox" in the LIST command is a pattern, not a name. So, 
see (b). Also, anything that matches the "Inbox" patten, should match the
"%n%ox" pattern, otherwise we need to add "reasonable exceptions" to the "%" 
handling, too. I hope we won't.


> I think that we all agree that the following patterns all match INBOX:
> 	*
> 	%
> 	INBOX
> 	INBOX%
> 	INBOX*
> and the following patterns do NOT match INBOX:
> 	%/%
> 	foo
> 	foo/%
> 	INBOX/*
> 	INBOX/%

Yes. But servers that do not follow strict and well-defined rules do cause 
much more trouble that good. We did observe those servers returning "Inbox" 
when List "Inbox" was called, thus causing clients to display 2 different 
INBOXes, and we saw servers returning "INBOX" for List "i%" ("paged" mailbox 
list display) causing clients to loop (though it's a client problem, as on a 
case-insenstive system this later case is legal).

The "special case" of INBOX is a terrible thing, causing many problems in 
any case, so we should try to be as strict as possible about it, and not add 
additional "reasonable exceptions".

Clients should carefully process these names, NOT assuming anything not 
already specified in the standard.

More,
a) what about:
  INBOX/mailbox
  InBox/mailbox
  INBOX/mailBOX
  InBox/mailBOX
(our interpretation - 1 and 2 are the same mailbox, 3 and 4 are the same 
mailbox, they all are the same mailbox on a case-insensitive system).

b) what about:
  ~user@domain/INBOX
  ~user@domain/INBoX
  ~user@domain/INBOX/Mailbox
  ~user@domain/INBoX/Mailbox

or whatever notation is used with that server to access "foreign" mailboxes.
(our interpretation - 1 and 2, 3 and 4 they are the same, the INBOX name is 
a special name not on the top of the mailbox name hierearchy, but on the top 
of the hierarchy for a particular "mailbox set" or "account").

The standard does not really say anything about using INBOX in 
LSUB/SUBSCRIBE - fortunately, almost no client uses mailbox subscriptions 
these days, otherwise it would create even more problems.

Bottom line - we all agree here, I hope:

always use INBOX as the name of the "incoming mail" mailbox.
  
> -- Mark --

Sincerely,
Vladimir

Given the wording, agreed. I was primarily concerned that the section in 
question seemed to contradict the spirit of INBOX being 
case-insensitive, but given this discussion I can understand Vladimir's 
rationale.

Cheers,

Nevon

On Mon, 10 Apr 2006, Vladimir A. Butenko wrote:
> One of the biggest problem with any protocol are "useful exceptions". It has 
> been proven too many times that whereever a problem (or misunderstanding) can 
> emerge, it will emerge.

I agree.

This is where Postel's robustness principle comes in.  Sadly, many people 
misunderstand it to mean that protocol engines should be tolerant of 
syntax errors as opposed to being a razor for dealing with ambiguous 
interpretations.

> If a server happens to use case-insensitive names, and, as a result, all 
> operations (including the pattern match) are case insensitive, then search 
> results can be different when compared to case-sensitive systems.

I forget why we decided to say "uppercase" instead of "case-insensitive" 
for pattern matching with INBOX.  But the wording leaves no doubt that we 
did decide that, and the result is ambiguity in the search results.

> The IMAP protocol does not explicitly specify what the REAL NAME of the 
> "incoming mailbox" is. If it would specify that the name is INBOX, it would 
> eliminate a lot of misunderstanding. Unfortunately, it does not.

Ummm...well...

I consider the difference between "real name" and INBOX to be an 
implementation detail; in the protocol itself, there is no difference.

> The initial request was sent to our support 
> group because the author of the question used "Inbox" as the reference to the 
> incoming mailbox. And while it works fine with all SELECT & Co commands, it 
> worked OK with LIST only because it was used on a case-insensitive platform. 
> Switching to a case-sensitive platform revealed the problem.
> [snip]
> This is exactly the answer we have given: using any name instead of INBOX is 
> a call for trouble.

I agree completely!

> BUT: there is nothing in the IMAP specs that explicitly 
> says so. And many people who do not know (and have no reason to know) about 
> all historical details behind the name "INBOX" in all caps, may conclude that 
> it's OK to use Inbox as that special mailbox name, especially because many UI 
> applications render it this way.

Hmm.  What do other people think?  Should there be a statement in the LIST 
command to this effect?

> The question here is simple:
> if
> a LIST "" "%n%ox"
> should show any name for the incoming mailbox should it
> show INBOX? Or Inbox? Or InBox? Or InBOx?

My personal opinion (NOT "Mr. IMAP Protocol") says that it should always 
return "INBOX", and this is the one true canonical name.

My personal option also says that a LIST pattern of "Inbox" or "%n%ox" 
matches INBOX.

Mr. IMAP Protocol, on the other hand, says that it does not matter what is 
returned; "INBOX" and "InBoX" and "Inbox" are all the same name and that 
the client can make no assumptions based on the case of the pattern when 
it comes to INBOX.

Mr. IMAP Protocol also says that the client can not assume that "Inbox" or 
"%n%ox" will match INBOX.

After Mark and Mr. IMAP Protocol strangle each other, they come out with a 
joint statement of "it's ambiguous that way, so clients shouldn't do it."

> The "special case" of INBOX is a terrible thing, causing many problems in any 
> case, so we should try to be as strict as possible about it, and not add 
> additional "reasonable exceptions".

I don't think that I was adding any "reasonable exceptions".  Rather, I 
was simply trying to apply Postel's robustness principle.  The important 
thing here is for the client to "be conservative" since there is no way to 
guarantee particular server performance.

> Clients should carefully process these names, NOT assuming anything not 
> already specified in the standard.

Agreed.

> More,
> a) what about:
> INBOX/mailbox
> InBox/mailbox
> INBOX/mailBOX
> InBox/mailBOX
> (our interpretation - 1 and 2 are the same mailbox, 3 and 4 are the same 
> mailbox, they all are the same mailbox on a case-insensitive system).

My interpretation:

I agree that they are all the same mailbox on a case-insensitive system.

I don't know how many mailboxes (2 or 4) it may be on a case-sensitive 
system; I think that it is implementation-dependent.  Had I thought about 
it at the time, I would have prohibited INBOX from being a "dual-use" 
name.  I agree that your interpretation is superior.

> b) what about:
> ~user@domain/INBOX
> ~user@domain/INBoX
> ~user@domain/INBOX/Mailbox
> ~user@domain/INBoX/Mailbox
> or whatever notation is used with that server to access "foreign" mailboxes.
> (our interpretation - 1 and 2, 3 and 4 they are the same, the INBOX name is a 
> special name not on the top of the mailbox name hierearchy, but on the top of 
> the hierarchy for a particular "mailbox set" or "account").

In this case, I definitely do not know.  I think that it's implementation 
dependent.  I don't know if "~" showing up in NAMESPACE makes a 
difference.

> Bottom line - we all agree here, I hope:
> always use INBOX as the name of the "incoming mail" mailbox.

Yes!

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Hi all,

> >More,
> >a) what about:
> >INBOX/mailbox
> >InBox/mailbox
> >INBOX/mailBOX
> >InBox/mailBOX
> >(our interpretation - 1 and 2 are the same mailbox, 3 and 4 are the same 
> >mailbox, they all are the same mailbox on a case-insensitive system).
> 
> My interpretation:
> 
> I agree that they are all the same mailbox on a case-insensitive system.
> 
> I don't know how many mailboxes (2 or 4) it may be on a case-sensitive 
> system; I think that it is implementation-dependent.  Had I thought about 
> it at the time, I would have prohibited INBOX from being a "dual-use" 
> name.  I agree that your interpretation is superior.

This might be implementation-dependent, but IMHO any implementation
choosing to see 4 mailboxes in the example above will create some
problems.

RFC3501 says:
      If the server's hierarchy separator character appears elsewhere in
      the name, the server SHOULD create any superior hierarchical names
      that are needed for the CREATE command to be successfully
      completed.  In other words, an attempt to create "foo/bar/zap" on
      a server in which "/" is the hierarchy separator character SHOULD
      create foo/ and foo/bar/ if they do not already exist.

That is, creating "InBox/mailBOX" SHOULD create "InBox". If the
implementation chooses to not identify "InBox/mailBOX" with
"INBOX/mailBOX", it 
  o either does ignore this paragraph 
  o or it will at creation time of this folder also create "InBox". 

In the latter case, how would a client access InBox, even worse,
delete InBox?

After playing around with this for a while, I for myself decided, that
the only way of implementing the INBOX magig without running into all
kinds of nightmares is to stick to the upper-case variant. I.e., not
identifying "InBox/foo" with "INBOX/foo" is crying out loud for trouble.

Therefore I strongly vote for clarifying this case in the RFC. Not the
_word_ INBOX should be special, but the hierarchical name.

I agree that protocoll-wise this restriction is not needed and that
it's up to the implementation to make the desicion. But I belive there
are only two feasible solutions (being case-insensitive or having only
one _hierarchical_ name matching [Ii][Nn][Bb][Oo][Xx]) and pinning
this down in the RFC would save some trouble.

Not this discussion, though, so yes, I'm a bit off topic.

Best regards,
Ingo

Our system is case sensitive. We treat 1 and 2 as the same mailbox, and
3 and 4 as the same mailbox.

	Tony Hansen
	tony@att.com

Mark Crispin wrote:
>> More,
>> a) what about:
>> INBOX/mailbox
>> InBox/mailbox
>> INBOX/mailBOX
>> InBox/mailBOX
>> (our interpretation - 1 and 2 are the same mailbox, 3 and 4 are the
>> same mailbox, they all are the same mailbox on a case-insensitive
>> system).
> 
> My interpretation:
> 
> I agree that they are all the same mailbox on a case-insensitive system.
> 
> I don't know how many mailboxes (2 or 4) it may be on a case-sensitive
> system; I think that it is implementation-dependent.  Had I thought
> about it at the time, I would have prohibited INBOX from being a
> "dual-use" name.  I agree that your interpretation is superior.

Ingo Schurr writes:
> This might be implementation-dependent, but IMHO any implementation 
> choosing to see 4 mailboxes in the example above will create some 
> problems.

Been there, accidentally done that, you're absolutely right. Down that 
path lies madness

> Therefore I strongly vote for clarifying this case in the RFC. Not the 
> _word_ INBOX should be special, but the hierarchical name.

And what about the actual inbox? Suppose your INBOX is also available as 
"Other Users.ingo.INBOX", as it might be on some server? (Was it 
cyrus?) In that case, are "Other Users" and "ingo" case sensitive, 
while "INBOX" is not? Given the following six commands, which should 
succeed?

a CREATE "Other users.ingo.inbox.a"
b CREATE "Other user.inbo.inbox.A"
c CREATE "Other users.ingo.INBOX.a"
d CREATE INBOX.A
e CREATE inbox.B
f CREATE inbox.b

I don't really care, I just wanted to point it out ;)

Arnt

On Tue, 11 Apr 2006, Tony Hansen wrote:
> Our system is case sensitive. We treat 1 and 2 as the same mailbox, and
> 3 and 4 as the same mailbox.

Of course, this is the sensible interpretation.  However, are we certain 
that there is nobody who is nonsensical enough to treat these as four 
different names?

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Arnt Gulbrandsen writes:
> Suppose your INBOX is also available as "Other Users.ingo.INBOX", as 
> it might be on some server? (Was it cyrus?)

(No, not Cyrus. I can't remember.)

On 11/04/06 11:45 +0200, Arnt Gulbrandsen wrote:
> >Therefore I strongly vote for clarifying this case in the RFC. Not the 
> >_word_ INBOX should be special, but the hierarchical name.
> 
> And what about the actual inbox? Suppose your INBOX is also available as 
> "Other Users.ingo.INBOX", as it might be on some server? (Was it 
> cyrus?) In that case, are "Other Users" and "ingo" case sensitive, 
> while "INBOX" is not?

Tricky. I would say, "Other Users.ingo.INBOX" is case sensitive.

> Given the following six commands, which should succeed?

This, I would say, _is_ implementation dependend. Does 
a CREATE "Other users.ingo.foo"
create folder "foo"? That is, is "Other users.ingo" a link to the root
of user ingo? And if so, who is allowed to create a folder there?

IMHO, if "Other users.ingo" is indeed a link to the root of user ingo,
it should behave so. That is:

> a CREATE "Other users.ingo.inbox.a"
-> Creates INBOX.a

> b CREATE "Other user.inbo.inbox.A"
-> Creates INBOX.A

> c CREATE "Other users.ingo.INBOX.a"
-> Fails, as INBOX.a already exists ;)

> d CREATE INBOX.A
-> Fails, as INBOX.A already exists

> e CREATE inbox.B
-> Creates INBOX.B

> f CREATE inbox.b
-> Creates INBOX.b

(Assuming the server is case sensitive.)

Let's see what Mr. IMAP protocol has to say about this... ;)

Best
Ingo

When 3501bis gets written, it'd be worth adding a note about this case.

	Tony Hansen
	tony@att.com

Mark Crispin wrote:
> On Tue, 11 Apr 2006, Tony Hansen wrote:
>> Our system is case sensitive. We treat 1 and 2 as the same mailbox, and
>> 3 and 4 as the same mailbox.
> 
> Of course, this is the sensible interpretation.  However, are we certain
> that there is nobody who is nonsensical enough to treat these as four
> different names?

Mark Crispin writes:
> On Tue, 11 Apr 2006, Tony Hansen wrote:
>> Our system is case sensitive. We treat 1 and 2 as the same mailbox, 
>> and 3 and 4 as the same mailbox.
>
> Of course, this is the sensible interpretation.  However, are we 
> certain that there is nobody who is nonsensical enough to treat these 
> as four different names?

I na?vely wrote code implementing the RFC as written, and discovered 
that the code treated all four as different. Rather surprised me. I 
would not bet that everyone else has avoided the mistake.

Arnt

On Tue, 11 Apr 2006, Ingo Schurr wrote:
> Let's see what Mr. IMAP protocol has to say about this... ;)

I presume that you don't want to hear "INBOX should not be dual use".

;-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue, 11 Apr 2006, Tony Hansen wrote:
> When 3501bis gets written, it'd be worth adding a note about this case.

I'm not convinced that we can, unless we can guarantee that there is no 
implementation out there that does the nonsensical thing.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue, 11 Apr 2006, Arnt Gulbrandsen wrote:
>>> Our system is case sensitive. We treat 1 and 2 as the same mailbox,
>>> and 3 and 4 as the same mailbox.
>> Of course, this is the sensible interpretation.  However, are we
>> certain that there is nobody who is nonsensical enough to treat these
>> as four different names?
> I naively wrote code implementing the RFC as written, and discovered
> that the code treated all four as different. Rather surprised me. I
> would not bet that everyone else has avoided the mistake.

I don't think that the RFC "as written" requires that all four be treated 
as different.  Rather, the RFC simply doesn't take a position.

To be honest, I would have preferred to prohibit case-sensitivity in 
mailbox names.  However, at the time there was a lot of pushback from the 
UNIX community as to why case-sensitivity was so critically important for 
the modern UNIX world, and how the future would be case-sensitive forever.

Today, I note that Mac OS X's default filesystem is case-insensitive (as 
is Windows XP).  I actually used it for several months before I realized 
that it was case-insensitive; and if I hadn't tried to unpack a tarball 
with two files that differed only by case, I probably would still not have 
noticed.

I have this lingering nightmare that the day will come that I will be 
flamed for case-sensitivity in mailbox names, and being too stupid to 
recognize the obvious benefits of case-insensitivity (remember that I came 
from the PDP-10 world...).  *Shudder*

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin writes:
> I presume that you don't want to hear "INBOX should not be dual use".

I wonder how MrSam would react if you were to put that into 3501bis?

Arnt

It can easily be put in as a SHOULD.

	Tony

Mark Crispin wrote:
> On Tue, 11 Apr 2006, Tony Hansen wrote:
>> When 3501bis gets written, it'd be worth adding a note about this case.
> 
> I'm not convinced that we can, unless we can guarantee that there is no
> implementation out there that does the nonsensical thing.

On Tue Apr 11 20:46:19 2006, Mark Crispin wrote:
> On Tue, 11 Apr 2006, Tony Hansen wrote:
>> When 3501bis gets written, it'd be worth adding a note about this 
>> case.
> 
> I'm not convinced that we can, unless we can guarantee that there 
> is no implementation out there that does the nonsensical thing.

I don't think it's actually possible, because unless you do the 
sensible thing, then the result is a broken server.

Let's assume that we have two servers. Both consider only "INBOX" to 
be case insensitive, everything else is strictly case sensitive. Our 
client makes the same foolish assumption.

For one, we'll make INBOX not be a dual-use mailbox. This should make 
Mark happy. We'll call this A.

For the other, we'll make INBOX a dual-use mailbox. We shall call 
this B.

If they do the silly thing, then they are in the unfortunate position 
that should a client CREATE "InBox/foo", they have to return a LIST 
response for INBOX, and a list response for InBox, with different 
attributes.

Assuming this is not illegal in and of itself, does the client then 
have to treat INBOX and InBox as different names, one of which is 
case-insensitive?

In summary, I think you're safe to deprecate the practise.

Dave.
-- 
           You see things; and you say "Why?"
   But I dream things that never were; and I say "Why not?"
    - George Bernard Shaw

On Tue, 11 Apr 2006 14:25:39 -0700 (Pacific Daylight Time)
  Mark Crispin <MRC@cac.washington.edu> wrote:


> I have this lingering nightmare that the day will come that I will be 
>flamed for case-sensitivity in mailbox names, and being too stupid to 
>recognize the obvious benefits of case-insensitivity (remember that I came 
>from the PDP-10 world...).  *Shudder*

It would be beneficial if a client can learn AT LEAST if the server 
namespace is case-sensitive or not. Because any client does have to switch 
its internal routines to the case-[in]sensitive mode. Something along the 
lines used to discover the path separator -
  a (CASE) extension to the special-case LIST "" "" command?
it may also help to have:
  A001 LIST (CASE) "" "INBOX/"
that will provide an answer to how to treat
INBOX/aaa
and
INboX/aaa
cases.

But, as usual, the Pandora box is happy to be opened again:
If the server says that it's case-insenstive, than I, as a user, would 
definitely expect that not only
    Mailbox
and
    mailboX

will be same box, but that

   ????
and
   ????

will be the same mailbox, too. The underlying OS file system may handle this 
case properly - Windows does, I'm sure MacOSX does so too. But it won't help 
-
as an IMAP server cannot use that OS feature, because it cannot deal 
directly with non-Latin names: UTF-7 comes in and destroyes everything ;-(

Decoding UTF-7 back into UTF-8 and using it internally is ugly, as you can 
get a completely different name in LIST results.

That's why "strict case-sensitivity" is a GOOD thing from the protocol point 
of view, and all IMAP servers MUST be case-sensitive to avoid confusion in 
clients (but not in users). Unfortunately, many IMAP servers do map mailbox 
names into OS file names, and we have all these "semi-case-insensitivity" 
problems today.

The solution would be to get rid of UTF-7 and switch to the plain UNENCODED 
UTF-8 for mailbox names.  If a client can learn that the server supports 
that (via those special LIST or NAMESPACE responses), then the transition 
may be not as impossible as it looks now. And any server that stores 
mailboxes as UTF-7-encoded OS files can support UTF-8 clients now, as it's 
very easy to detect a UTF-8 name: simple presence of the 8th bit in any name 
byte is an indication of the "new" naming conventions.

But - there would be a lot of hidden problems in such a transition...

> -- Mark --
> 
> http://staff.washington.edu/mrc
> Science does not emerge from voting, party politics, or public debate.

Political Science does :-). There is a lot one can learn by watching 
primates voting. Or debating. Especially publicly :-)

> Si vis pacem, para bellum.

and when you wish for war, prepare for a long boring peace? :-)


Sincerely,
Vladimir

On Tue, 11 Apr 2006, Arnt Gulbrandsen wrote:
> Mark Crispin writes:
>> I presume that you don't want to hear "INBOX should not be dual use".
> I wonder how MrSam would react if you were to put that into 3501bis?

I prefer not to find out.  :-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue, 11 Apr 2006, Dave Cridland wrote:
> If they do the silly thing, then they are in the unfortunate position that 
> should a client CREATE "InBox/foo", they have to return a LIST response for 
> INBOX, and a list response for InBox, with different attributes.

Sorry, that Rubicon has already been crossed.

The server can hide InBox.  Thus, if you create "InBox/foo", you will 
never see any such name unless you explicitly enter Inbox/ prior to any 
wildcards.  Put another way, "InBox" does not exist, but "InBox/" does.

UW imapd does that if the client ignores the \NoInferiors that is 
returned with the name INBOX.

Because someone demanded it, and I got tired of being screamed at. 
That's why.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue, 11 Apr 2006, Vladimir A. Butenko wrote:
> It would be beneficial if a client can learn AT LEAST if the server namespace 
> is case-sensitive or not. Because any client does have to switch its internal 
> routines to the case-[in]sensitive mode.

Well, the attitude of the past was that the client shouldn't do any such 
thing; that is, that it could not (and should not) make any assumptions 
about how the server works.

I wouldn't be opposed to an extension of this nature, but it may be 
difficult to implement.  A UNIX based server could not assume that it is 
case-sensitive; it would have to determine this on a filesystem basis and 
it may not have a good way of knowing (other than empirical testing) if a 
remote filesystem is case-sensitive or not.

> But, as usual, the Pandora box is happy to be opened again:

You may be interested to know that your Cyrillic example actually 
displayed correctly in my Japanese environment!  :-)

> The underlying OS file system may handle this 
> case properly - Windows does, I'm sure MacOSX does so too. But it won't help -
> as an IMAP server cannot use that OS feature, because it cannot deal directly 
> with non-Latin names: UTF-7 comes in and destroyes everything ;-(

For what it's worth, I just implemented support for Unicode 
case-insensitivity in the development sources of UW imapd.  Actually, I 
cheated and canonicalized everything to titlecase.

Decomposition is next.

> Decoding UTF-7 back into UTF-8 and using it internally is ugly, as you can 
> get a completely different name in LIST results.

Don't the rules for modified UTF-7 define a completely reversible 
transform?

> That's why "strict case-sensitivity" is a GOOD thing from the protocol point 
> of view, and all IMAP servers MUST be case-sensitive to avoid confusion in 
> clients (but not in users). Unfortunately, many IMAP servers do map mailbox 
> names into OS file names, and we have all these "semi-case-insensitivity" 
> problems today.

I don't follow you as to why all IMAP server "MUST be case-sensitive", 
since clearly there are examples of servers which are case-insensitive, 
and I don't see why a server could not decide to map Unicode case for 
M-UTF7 names.

> The solution would be to get rid of UTF-7 and switch to the plain UNENCODED 
> UTF-8 for mailbox names.

I agree.  The whole point of modified UTF-7 in 1996 was to put a halt to 
the (then-common) practice of "just send 8-bits" for mailbox names in 
local character sets.

The people who did (do?) so have had 9 1/2 years warning.  That should be 
enough.  We should progress to UTF-8 mailbox names.

>> Science does not emerge from voting, party politics, or public debate.
> Political Science does :-). There is a lot one can learn by watching primates 
> voting. Or debating. Especially publicly :-)

As I say on my web page, any field of study which has "science" in its 
name is not a science...  ;-)

>> Si vis pacem, para bellum.
> and when you wish for war, prepare for a long boring peace? :-)

I don't know, as only fascists wish for war; and (as the world learned 60+ 
years ago) the only thing that you can do with fascists is defeat them.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin wrote:
> On Tue, 11 Apr 2006, Arnt Gulbrandsen wrote:
>> Mark Crispin writes:
>>> I presume that you don't want to hear "INBOX should not be dual use".
>>
>> I wonder how MrSam would react if you were to put that into 3501bis?
> 
> I prefer not to find out.  :-)

I think the ship has sailed on this one.  Cyrus has had a dual-use INBOX 
for about ten years.

Tim

On Tue, 11 Apr 2006 18:37:39 -0700 (Pacific Daylight Time)
  Mark Crispin <MRC@CAC.Washington.EDU> wrote:
> On Tue, 11 Apr 2006, Vladimir A. Butenko wrote:
>> It would be beneficial if a client can learn AT LEAST if the server 
>>namespace 
>> is case-sensitive or not. Because any client does have to switch its 
>>internal 
>> routines to the case-[in]sensitive mode.
> 
> Well, the attitude of the past was that the client shouldn't do any such 
>thing; that is, that it could not (and should not) make any assumptions 
>about how the server works.

A user asked the client to show all mailboxes with names starting with 
"Mark" and all those ending with "CRISPIN":

There is a mailbox called "Mark Crispin". The system is case-insensitive.

Now (removing all unrelated parts of the protocol):

a LIST "Mark%"
* LIST "Mark Crispin"
a OK
b LIST "" "%CRISPIN"
* LIST "MARK CRISPIN"

The later is a questionable practice, but it definitely has more merits than 
returning unmodified "Mark Crispin" as our server (and, I guess, yours) is 
doing now. The "Mark Crispin" string would confuse a client that does not 
expect that string to match the "%CRISPIN" pattern.

If there is no "fixed" name for INBOX, it's just a "case-insensitive" name, 
why would "Mark Crispin" be a "real" name for that mailbox of a 
case-insensitive system? So, if the server is free to return "Inbox" on LIST 
"Inbox", it's free to return "MARK CRISPIN" on "%CRISPIN".

And then the client will, obviously, display these mailboxes as different 
ones, and all the troubles will trouble us there..

> I wouldn't be opposed to an extension of this nature, but it may be 
>difficult to implement.  A UNIX based server could not assume that it is 
>case-sensitive; it would have to determine this on a filesystem basis and 
>it may not have a good way of knowing (other than empirical testing) if a 
>remote filesystem is case-sensitive or not.

Delete the .mark file, create a .Mark file, try to read ".mark" - if you do 
that once, during start-up of your server, that would be enough. It's more 
difficult when you have to serve several millions of users, where the 
storage can be distributed to many NFS and CFS file systems - but believe me 
- I have never seen any real-life installation that is at least 100,000 
users strong (leave alone 5,000,000 users strong) that used different file 
systems for different users.

>> But, as usual, the Pandora box is happy to be opened again:
> 
> You may be interested to know that your Cyrillic example actually 
>displayed correctly in my Japanese environment!  :-)

All Cyrillic characters are included into the ISO-2022-JP charset.
I'm pretty sure this is a preparation for occupation (uups, liberation), but 
I'm not that sure about the expected direction of that occupation... :-)

But - there is no low-case/upper-case problem with the Japanese charsets, so 
you may want to play with mailboxes in Roman, but non-Latin alphabets, - 
French, Spanish, German etc. Umlauts are tough boys to fight with in the 
upper-lowercase battles...

>> with non-Latin names: UTF-7 comes in and destroyes everything ;-(
> 
>For what it's worth, I just implemented support for Unicode 
>case-insensitivity in the development sources of UW imapd.  Actually, I 
>cheated and canonicalized everything to titlecase.
> 
> Decomposition is next.
> 
>> Decoding UTF-7 back into UTF-8 and using it internally is ugly, as you can 
>> get a completely different name in LIST results.
> 
> Don't the rules for modified UTF-7 define a completely reversible 
>transform?

I'm afraid there is a misunderstanding here. How should I specify the ??%? 
pattern in the LIST command? "&hjhj-%&jkjk-" where "&hjhj-" is "??", and 
"&jk-" is "?"?
I should, because - "&hjhj%jkjk-" would be illegal, right?

But I have no mailboxes with names starting with "&hjhj-", all of them start 
with "&hjhjkkk". So, the server already need to be "smart" and not use UTF-7 
in  patterns, but convert the pattern to the UTF-8/Unicode form and to 
compare it not with mailbox names, but with mailbox names converted from 
UTF-7 into Unicode.

And this will result in LIST responses that have almost NOTHING in common 
with the supplied pattern, if both are treated as ASCII strings.

So, it's the same question as above: should the client expect that if it 
wants "Inbo%", it will get "Inbogus town", but not INBOX? Or should the 
client NOT assume anything and treat ANY response from the server as 
correctly matching the pattern the client has provided - matching according 
to the rules known to the server only, not the client?

If someone suggests to use the later answer, then it's a call for trouble:

then if a client makes a call
A LIST %
and gets
* LIST ZZZ

and then it makes a call
A LIST ZZZ/%
(as many clients do to deal with mailbox hierarchies)
then the client HAS ABSOLUTELY NO RIGHT to expect that all returned names 
will start with "ZZZ/", and thus - can be displayed as "ZZZ" "subtree".

Sounds strange, right? But we are saying that when a client sends 
"&hj-%&HJ-" in the LIST command, it should not expect to see any name that 
ends on "HJ-", right?

So, we conclude that Client "sometimes" can assume that the LIST results 
will be matching the LIST pattern when both strings are interpreted as ASCII 
strings, and sometimes the client cannot assume that? That's a mess, isn't 
it?
  
>> That's why "strict case-sensitivity" is a GOOD thing from the protocol 
>>point 
>> of view, and all IMAP servers MUST be case-sensitive to avoid confusion in 
>> clients (but not in users). Unfortunately, many IMAP servers do map 
>>mailbox 
>> names into OS file names, and we have all these "semi-case-insensitivity" 
>> problems today.
> 
> I don't follow you as to why all IMAP server "MUST be case-sensitive", 
>since clearly there are examples of servers which are case-insensitive,

Because case-insensitive servers create a mess. The protocol becomes 
ill-defined (read: broken, incomplete, unusable - select your own favourite, 
the most offending word :-).

If the servers would impose strict case-sensitivity (including that for 
INBOX), the IMAP protocol (at least within its mailbox-name related part) 
becomes a well-defined, "real", "professional" (select the most pleasing 
word here) protocol. But it would be harder to implement on the systems with 
case-insensitive file systems (OS/2 and OS/3 aka Windows, MacOSX, VMS) - if 
the server chooses to implement mailboxes as file system objects.

> and 
>I don't see why a server could not decide to map Unicode case for M-UTF7 
>names.


Because then a client may assume virtually nothing about the server 
responses return for its wild-carded requests. And without being able to 
assume these things, the client cannot do anything with mailboxes other than 
showing them as a linear list.

>> The solution would be to get rid of UTF-7 and switch to the plain 
>>UNENCODED 
>> UTF-8 for mailbox names.
> 
> I agree.  The whole point of modified UTF-7 in 1996 was to put a halt to 
>the (then-common) practice of "just send 8-bits" for mailbox names in local 
>character sets.
> 
> The people who did (do?) so have had 9 1/2 years warning.  That should be 
>enough.  We should progress to UTF-8 mailbox names.

But before you do, please investigate all these case-sensitivity problems 
with non-LATIN (and non-Japanese ;-) alphabets. And, while doing this, 
please remember that the number of people who were shocked to learn than 
they could not create a folder named "12/12/2005" is much higher than the 
number of people knowing what a "hierarchy separator" is. But that number is 
still smaller than the number of people who did succeed to create such a 
mailbox and then asked their ISP/IT support about "that funny mailbox '12' 
and some strange symbols around it". I.e. escape symbols would be good, or, 
of you choose to use UTF-8/Unicode, you may want to use some "unprintable" 
character as the path separator.

>>> Science does not emerge from voting, party politics, or public debate.
>> Political Science does :-). There is a lot one can learn by watching 
>>primates 
>> voting. Or debating. Especially publicly :-)
> 
> As I say on my web page, any field of study which has "science" in its 
>name is not a science...  ;-)

Sure. I totally agree, and that's why I was hated by all Computer Science 
departments :-) Their only competitors where teachers of the Scientific 
Communism - but they were weaker, as they did not believe in their books 
themselves. CS departments were tougher... :-)

>>> Si vis pacem, para bellum.
>> and when you wish for war, prepare for a long boring peace? :-)
> 
> I don't know, as only fascists wish for war;

Mark, I just wanted to make a quite innocent joke. Looks like I've failed, 
and let's not turn it into a Political Science exercise. Let's continue to 
play within the Computer Science barrack :-)

> -- Mark --
> 
> http://staff.washington.edu/mrc
> Science does not emerge from voting, party politics, or public debate.
> Si vis pacem, para bellum.

Sincerely,
Vladimir

On Tue, 11 Apr 2006, Vladimir A. Butenko wrote:
> There is a mailbox called "Mark Crispin". The system is case-insensitive.
> a LIST "Mark%"
> * LIST "Mark Crispin"
> a OK
> b LIST "" "%CRISPIN"
> * LIST "MARK CRISPIN"
> The later is a questionable practice, but it definitely has more merits than 
> returning unmodified "Mark Crispin" as our server (and, I guess, yours) is 
> doing now. The "Mark Crispin" string would confuse a client that does not 
> expect that string to match the "%CRISPIN" pattern.

Actually, I believe that a case-independent server should have a canonical 
case form -- all-upper, all-lower, as-created, or whatever -- and that 
form should always be returned by LIST regardless of the pattern.

> If there is no "fixed" name for INBOX, it's just a "case-insensitive" name, 
> why would "Mark Crispin" be a "real" name for that mailbox of a 
> case-insensitive system? So, if the server is free to return "Inbox" on LIST 
> "Inbox", it's free to return "MARK CRISPIN" on "%CRISPIN".

I agree that the canonical case form of INBOX is "INBOX".  That was always 
the intention from Day 1.

> Delete the .mark file, create a .Mark file, try to read ".mark" - if you do 
> that once, during start-up of your server, that would be enough.

Yes, but that's only if there is a single filesystem on the server.

> It's more 
> difficult when you have to serve several millions of users, where the storage 
> can be distributed to many NFS and CFS file systems - but believe me - I have 
> never seen any real-life installation that is at least 100,000 users strong 
> (leave alone 5,000,000 users strong) that used different file systems for 
> different users.

I don't know if that's safe any more.  We can assume that UNIX based IMAP 
servers are probably not going to export Windows filesystems.  However, 
Mac OS X has really muddied the waters.  My own Mac has a case-sensitive 
and a case-insensitive filesystem.

I'm also not confortable about assuming that what is commonplace and good 
practice today would remain the same 5-10 years from now.  I've had too 
many painful lessons...

> But - there is no low-case/upper-case problem with the Japanese charsets, so 
> you may want to play with mailboxes in Roman, but non-Latin alphabets, - 
> French, Spanish, German etc. Umlauts are tough boys to fight with in the 
> upper-lowercase battles...

We can use Unicode titlecase.  I think that an update to RFC 3501 should 
probably specify that case-insensitivity in searching means "same 
titlecase" as opposed to "same uppercase" or "same lowercase".

> I'm afraid there is a misunderstanding here. How should I specify the $B'A'k(B%$B'\(B 
> pattern in the LIST command? "&hjhj-%&jkjk-" where "&hjhj-" is "$B'A'k(B", and 
> "&jk-" is "$B'\(B"?

Ah, good point.  The only wildcards that work well with M-UTF7 names are 
within the ASCII part of the name.

Fortunately, most clients only use wildcards for an entire hierarchy 
level.  But this is a good argument to move to UTF-8 names.

> Or should the 
> client NOT assume anything and treat ANY response from the server as 
> correctly matching the pattern the client has provided - matching according 
> to the rules known to the server only, not the client?

This is effectively what IMAP says.

> If someone suggests to use the later answer, then it's a call for trouble:
>
> then if a client makes a call
> A LIST %
> and gets
> * LIST ZZZ
>
> and then it makes a call
> A LIST ZZZ/%
> (as many clients do to deal with mailbox hierarchies)
> then the client HAS ABSOLUTELY NO RIGHT to expect that all returned names 
> will start with "ZZZ/", and thus - can be displayed as "ZZZ" "subtree".

I don't understand why.  There's no M-UTF7 here.  The only thing that may 
different is that with a case-insensitive server that it may return names 
that start with zzz/ or Zzz/ etc.

> Sounds strange, right? But we are saying that when a client sends "&hj-%&HJ-" 
> in the LIST command, it should not expect to see any name that ends on "HJ-", 
> right?

I agree that M-UTF7 patterns with embedded wildcards don't work.

>> I don't follow you as to why all IMAP server "MUST be case-sensitive", 
>> since clearly there are examples of servers which are case-insensitive,
> Because case-insensitive servers create a mess. The protocol becomes 
> ill-defined (read: broken, incomplete, unusable - select your own favourite, 
> the most offending word :-).

That's only if the client expects to know what the server will do for any 
particular LIST command if it has earlier done a LIST of *.

IMAP makes no such promises.  It's implementation dependent, as are the 
semantics of the naming hierarchy.

This was a mistake.  We all acknowledge it to have been a mistake. 
However, the discussion about naming that took place in the early 1990s 
wasted at least 18 months of everybody's time (and probably reduced all of 
our lifespans by a few years due to high blood pressure).  What came up 
was a wretched compromise, but at least it let us do our work.

I disagree that case-insensitive servers are ill-defined; any 
case-insensitive server is very well defined.  The problem is that there 
are multiple definitions, and no authority to declare one definition to be 
correct.

> If the servers would impose strict case-sensitivity (including that for 
> INBOX), the IMAP protocol (at least within its mailbox-name related part) 
> becomes a well-defined, "real", "professional" (select the most pleasing word 
> here) protocol.

Uhh...I don't know about that.  Consider all the repetitions of the Latin 
alphabet in Unicode.

You'd have a better case if you said "if all mailbox names were solely a 
sequence of unique binary octets, with no human interpretation, then the 
IMAP protocol become..."  ;-)

>> We should progress to UTF-8 mailbox names.
> But before you do, please investigate all these case-sensitivity problems 
> with non-LATIN (and non-Japanese ;-) alphabets.

Unicode titlecase.  It's not always right for all languages, but at least 
it is something that is well-defined.

> And, while doing this, please 
> remember that the number of people who were shocked to learn than they could 
> not create a folder named "12/12/2005" is much higher than the number of 
> people knowing what a "hierarchy separator" is. But that number is still 
> smaller than the number of people who did succeed to create such a mailbox 
> and then asked their ISP/IT support about "that funny mailbox '12' and some 
> strange symbols around it". I.e. escape symbols would be good, or, of you 
> choose to use UTF-8/Unicode, you may want to use some "unprintable" character 
> as the path separator.

Had URLs existed at the time, that is what IMAP should have used.  That at 
least is a hierarchy that people understand.

>> As I say on my web page, any field of study which has "science" in its name 
>> is not a science...  ;-)
> Sure. I totally agree, and that's why I was hated by all Computer Science 
> departments :-) Their only competitors where teachers of the Scientific 
> Communism - but they were weaker, as they did not believe in their books 
> themselves. CS departments were tougher... :-)

Indeed.

>>>> Si vis pacem, para bellum.
>>> and when you wish for war, prepare for a long boring peace? :-)
>> I don't know, as only fascists wish for war;
> Mark, I just wanted to make a quite innocent joke. Looks like I've failed, 
> and let's not turn it into a Political Science exercise. Let's continue to 
> play within the Computer Science barrack :-)

Oh, I understood the joke.  However, it's worth remembering that 61 years 
ago, our parents sacrificed a great deal so that we could grow up to play 
with computers...

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Vladimir A. Butenko writes:
> a LIST "Mark%"
> * LIST "Mark Crispin"
> a OK
> b LIST "" "%CRISPIN"
> * LIST "MARK CRISPIN"
>
> The later is a questionable practice, but it definitely has more 
> merits than returning unmodified "Mark Crispin" as our server (and, I 
> guess, yours) is doing now. The "Mark Crispin" string would confuse a 
> client that does not expect that string to match the "%CRISPIN" 
> pattern.

Do you know that for a fact? Which client(s)?

Arnt

On Tue, 11 Apr 2006 22:56:33 -0700 (PDT)
  Mark Crispin <mrc@CAC.Washington.EDU> wrote:

>> correctly matching the pattern the client has provided - matching 
>>according to the rules known to the server only, not the client?

[...]
>> then the client HAS ABSOLUTELY NO RIGHT to expect that all returned names 
>> will start with "ZZZ/", and thus - can be displayed as "ZZZ" "subtree".
> 
> I don't understand why.  There's no M-UTF7 here. 

Who cares? You've just said that I (being a client) cannot assume ANYTHING 
about your (server's) responses, so I cannot plan my life depending on you.

Now you are backing up and saying "sorry, I may not be reliable at coming 
home at 19:35 every evening, but if you ask me to bring home a steak, I may 
actually bring you a fish (you've asked about Inbox, I gave you INBOX), or a 
well-baked skunk (INboX, as I like that name better), but no, no, I won't 
bring any toilet paper, you can rely on me here!".

>The only thing that may 
>different is that with a case-insensitive server that it may return names 
>that start with zzz/ or Zzz/ etc.

As we saw with non-ASCII names, it's more complex than that, and the number 
of variations is endless (though it's a countable set, not a continuum :-).

There are some freeware IMAP servers using "." as the path separator. At the 
same time, "." is part of "Modified-UTF-7 BASE64" encoding. So, some name 
can look as &ADB.r985-, because it was "$B8@8l(B" (it's just an example, not a 
real encoding of this japanese word). So, what would

    LIST "%.%" command return?

I'd say that the client really cannot know here, and the hierarchy it will 
present to the user would be quite different from what some users expect.

>> in the LIST command, it should not expect to see any name that ends on 
>>"HJ-", 
>> right?
> 
> I agree that M-UTF7 patterns with embedded wildcards don't work.

The problem is that it means they do not work at all :-(. I.e. a client may 
expect SOMETHING if it uses wildcards only at the end of some level, i.e. 
doing "nesting loops" like "INBOX/%/%", - and making an *assumption* that 
everything returned by such a request will be really a subdirectory of the 
"INBOX" mailbox, but if the separator is ".", then such a request 
(INBOX.%.%" can suddently return INBOX.&ADB-.&r985-, which is a INBOX.$B8@8l(B and 
belongs to a different level.
  
> That's only if the client expects to know what the server will do for any 
>particular LIST command if it has earlier done a LIST of *.
> 
> IMAP makes no such promises.  It's implementation dependent, as are the 
>semantics of the naming hierarchy.

And that's exactly the problem.
  
> This was a mistake.  We all acknowledge it to have been a mistake. 
>However, the discussion about naming that took place in the early 1990s 
>wasted at least 18 months of everybody's time (and probably reduced all of 
>our lifespans by a few years due to high blood pressure).  What came up was 
>a wretched compromise, but at least it let us do our work.
>
> I disagree that case-insensitive servers are ill-defined; any 
>case-insensitive server is very well defined.  The problem is that there 
>are multiple definitions, and no authority to declare one definition to be 
>correct.

You are misinterpreting me. I had absolutely zero intention to point fingers 
and say how bad some decision WAS. If anyone can take the best step, taking 
into account all future developments till his own death, the best step would 
be to hang himself. What I'm trying to do here is quite different:

a) I'm trying to point to the problem with the current design.
b) Because I'm getting a feeling that some people, including Mr.Crispin MAY 
find time to do some revision of the IMAP protocol in the nearest future, 
I'm trying to remember ALL bad things we have met with - not even the 
protocol - but with this particular part of the protocol - and dump it on 
Mr.Crispin's head, so we can contribute - at least in this way - to the 
process of improving the protocol.
  
> Had URLs existed at the time, that is what IMAP should have used.  That at 
>least is a hierarchy that people understand.

It all depends on the amount of time you have. If I were you (if I may), I'd 
drop all concerns about the "compatibility issues" - and would create a 
clean and nice IMAP5 (NOT IMAP4revX). Making all the changes you would want 
to see there. And THEN see how it can be bridged to the existing standard so 
the existing clients and servers won't stop talking to each other. While 
this approarch seems scary and unrealistic, one usually discovers that the 
"bridge" to older versions is not as long and shaky as one would think.

But - that's just my not so humble opinion, this protocol is your kid, it's 
up to you to decide if it's a time for a Revolution, or smooth evolution is 
a better thing to do.
  
>>>>> Si vis pacem, para bellum.
>>>> and when you wish for war, prepare for a long boring peace? :-)
>>> I don't know, as only fascists wish for war;
>> Mark, I just wanted to make a quite innocent joke. Looks like I've failed, 
>> and let's not turn it into a Political Science exercise. Let's continue to 
>> play within the Computer Science barrack :-)
> 
> Oh, I understood the joke.  However, it's worth remembering that 61 years 
>ago, our parents sacrificed a great deal so that we could grow up to play 
>with computers...

Yes, it's worth remembering that it was 65 years rather than 61, though the 
death of those how died at the very end of that war when the outcome was 
clear was, probably, even more misfortunate.

Nothing is my sentense, though, had any intended ref. to that Great War. 
Neither did the creaters of ParaBellum(c) know anything about that upcoming 
event... There are many wars around. And not only on "battlefields".

  
> -- Mark --
>
> http://panda.com/mrc
> Democracy is two wolves and a sheep deciding what to eat for lunch.
> Liberty is a well-armed sheep contesting the vote.

Sincerely,
Vladimir

On Wed, 12 Apr 2006 13:29:26 +0200
  Arnt Gulbrandsen <arnt@gulbrandsen.priv.no> wrote:
> Vladimir A. Butenko writes:
>> a LIST "Mark%"
>> * LIST "Mark Crispin"
>> a OK
>> b LIST "" "%CRISPIN"
>> * LIST "MARK CRISPIN"
>>
>> The later is a questionable practice, but it definitely has more 
>> merits than returning unmodified "Mark Crispin" as our server (and, I 
>> guess, yours) is doing now. The "Mark Crispin" string would confuse a 
>> client that does not expect that string to match the "%CRISPIN" 
>> pattern.
> 
> Do you know that for a fact? Which client(s)?

It would be really beneficial - *IMHO* - if when rejecting some "nonexistent 
Science such as computer science" one would not reject any basics of any 
science at all. The protocols are written not to explain "convenient ways to 
do this or that", but to SPECIFY the way of interaction between entities 
involved into certain activity. If someone thinks differently, then please 
ask Mr.Crispin to degrade the status of his document from "STANDARD" to 
"INFORMATIONAL" or "BCP". And I will have some much needed relaxation time 
reading his response to such a proposal.

The whole approach to the protocol problems based on "the fact that no 
client has any problem with this or that" is WRONG - as long as we are 
talking about the standards, and not articles in some tabloids (which 
include computer science text books :-).

If I'd write some client, then I would expect the SPECIFIED, and not 
REASONABLE (from someone's point of view) response from a server. If the 
server does not return what is expected - based on the SPECIFIED STANDARD, 
then that server is *broken*.

Nothing in the SPECIFIED standard says anything about "%CRISPIN" matching 
"Mark Crispin". The server retuning such a response is broken - by 
definition of the word "standard".

Is Mark's server broken? Yes.
Is our server broken? Yes.
Does Mark plan to fix it? I doubt it.
Do we plan to fix it? I doubt it.

The moral? If the law is violated by too many people, probably it's time to 
change the law. Till then - it's the violation of the law.

My point (and provocation, as usual) was not to point the finger onto some 
"broken implementations": almost all of the existing IMAP implementations 
are broken in this respect. It was an illustration of a need to revise a 
protocol - a long overdue task, which is, unfortunately, not a simple task.

Finally, the answer to your question ("Which clients?") - was the part of 
the original phrase: "a client that does not expect that string [Mark 
Crispin] to match the "%CRISPIN" pattern." Any client has a right not to 
expect it.

And I do NOT care if such a client really exists or not - as long as we use 
this list to talk about protocols, and not about "the simplest way to 
install the XXX server on the YYY OS with a hard drive polluted by my ZZZZ 
pet". That would be Computer Science...


> Arnt

Sincerely,
Vladimir

On Wed, 12 Apr 2006, Vladimir A. Butenko wrote:
>>> correctly matching the pattern the client has provided - matching 
>>> according to the rules known to the server only, not the client?
>>> then the client HAS ABSOLUTELY NO RIGHT to expect that all returned names 
>>> will start with "ZZZ/", and thus - can be displayed as "ZZZ" "subtree".
>> I don't understand why.  There's no M-UTF7 here. 
> Who cares? You've just said that I (being a client) cannot assume ANYTHING 
> about your (server's) responses, so I cannot plan my life depending on you.

But that's no different than doing a wildcard in the shell on a system. 
The response is according to that system's rules.

I had to learn (very painfully I might add) that
 	mv *.c *.c.bak
on UNIX did not do what I expected it to do.

Mac OS X is particularly bizarre, although it makes sense when you 
consider how it is implemented.  A non-wildcard ls matches a name 
case-insensitive, but a wildcard ls is case-sensitive.

IMAP never tried to define an ideal mail store which had one (and only 
one) set of rules.  Rather, IMAP tried to be a means to export many 
different types of existing mail stores, often with very different rules.

>> The only thing that may different is that with a case-insensitive server 
>> that it may return names that start with zzz/ or Zzz/ etc.
> As we saw with non-ASCII names, it's more complex than that, and the number 
> of variations is endless (though it's a countable set, not a continuum :-).

Yes, but your example was ASCII.  I agree that once we get into non-ASCII 
names things become much messier.  But case-sensitivity is not the 
solution.  You have to be completely octet-sensitive if that's the goal.

> There are some freeware IMAP servers using "." as the path separator. At the 
> same time, "." is part of "Modified-UTF-7 BASE64" encoding.

"." can not be a hierarchy delimiter if it is within an M-UTF7 sequence. 
Perhaps you can't use "." as a hierarchy delimiter with M-UTF7.  Note that 
M-UTF7 did get out of the way of "/".  But yes, a server that uses "." as 
a hierarchy delimiter has a problem with M-UTF7.

Most people agree that it was a mistake for IMAP to have server-defined 
hierarchy delimiters; and that "/" should be the one and only hierarchy 
delimiter.  Hindsight is always easier.

Everybody agrees that M-UTF7 was unfortunate.  It was necessary at the 
time to abolish "just send 8-bits".  Too bad that punycode didn't exist at 
the time, since punycode is far superior.

Also too bad that people did "just send 8-bits" for IMAP mailbox names; 
had that not happened, we might have been able to define UTF-8 back then.

> So, some name can 
> look as &ADB.r985-, because it was "$B8@8l(B" (it's just an example, not a real 
> encoding of this japanese word). So, what would
>   LIST "%.%" command return?
> I'd say that the client really cannot know here, and the hierarchy it will 
> present to the user would be quite different from what some users expect.

I agree.  Life is tough for client authors some times.  Life is tough for 
server authors some times too.

We don't live in a perfect world.  I don't think that it's useful to 
complain about imperfections because perfection is unattainable.  It's 
more useful to try to determine what is the most reasonable thing to do 
in the fact of imperfections that exist and that other people will 
continuously create.

Most of these particular problems will be presently overtaken by events. 
At some point, we will adopt UTF-8 mailbox names, and M-UTF7 will be first 
a compatibility mode for old broken clients and later a bad memory of the 
past.

>> IMAP makes no such promises.  It's implementation dependent, as are the 
>> semantics of the naming hierarchy.
> And that's exactly the problem.

I don't consider that problem to be solvable.  The more restrictive that a 
protocol becomes in the face of implementation options, the less suitable 
that protocol is for various applications.

> a) I'm trying to point to the problem with the current design.

We all know these problems.

> b) Because I'm getting a feeling that some people, including Mr.Crispin MAY 
> find time to do some revision of the IMAP protocol in the nearest future

You are too optimistic.

So far, IMAP has been a 20 year process, the past 10 of which have 
supposedly been refining the current revision.  A new revision would open 
up a Pandora's box, as *everybody* will see it as their opportunity to 
get their own personal hot-button issue address.

I'm not volunteering to open that box.  I would be one of the people 
fighting tooth and nail to keep that box slammed shut.

IMAP is not the only IETF protocol which is having problem with 
internationalized identifiers.  It is not the only protocol in which the 
first attempts at internationalized identifiers have proven flawed.

There are a great many people who are wrestling with internationalization 
problems; and I am happy to await their guidance.

Be thankful that, unlike NNTP, IMAP took an early stand against "just send 
8-bits".  NNTP is a mess, and the proposed new specification isn't doing 
anything to solve that mess; it defends every behavior of every server 
forever.

> It all depends on the amount of time you have. If I were you (if I may), I'd 
> drop all concerns about the "compatibility issues" - and would create a clean 
> and nice IMAP5 (NOT IMAP4revX).

Compatibility with IMAP2 isn't the problem; in fact, hardly anyone worries 
about that any more.  Creeping featurism is the problem.

As attractive as it may sound to build an IMAP5 that removes the current 
known warts, I can guarantee that such a protocol would have much more 
added than removed.  It would not be completed before I retire.  It 
probably would not be completed during my lifetime.

And...it would have brand new warts, that won't be generally-recognized as 
warts until years later.

> But - that's just my not so humble opinion, this protocol is your kid, it's 
> up to you to decide if it's a time for a Revolution, or smooth evolution is a 
> better thing to do.

My goal is for IMAP to evolve to a state where it can be called "done", 
and then let the next generation worry about its successor.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.