mailing list archives

A JavaMail user encountered the following situations when another
client expunged a message being accessed by JavaMail.

First, fetching the content of the message failed:

A5 FETCH 1 (BODYSTRUCTURE)
* 1 FETCH (BODYSTRUCTURE ("TEXT" "PLAIN" ("charset" "UTF-8") NIL NIL "7BIT" 23 1 
NIL NIL NIL) FLAGS (\Deleted))
A5 OK FETCH completed.
A6 FETCH 1 (BODY.PEEK[TEXT]<0.23>)
A6 NO Some messages in the mailbox had previously been expunged and could not be 
returned.

With no opportunity to report an untagged EXPUNGED response, the command
simply failed with a NO response.

Second, fetching the BODYSTRUCTURE of an expunged message simply
failed to return the requested information although the command succeeded:

A5 FETCH 1 (BODYSTRUCTURE)
* 1 FETCH (FLAGS (\Deleted))
A5 OK FETCH completed.

Is this allowed behavior for an IMAP server?

Assuming it is, what is the recommended way for a client to detect this
situation so that it can distinguish a "message has been expunged" failure
from any other random failure?  Should the client follow such failures
with a NOOP to allow the server to report that the message has been
expunged?

It might be nice if there were a response code to indicate that the message
has been expunged, e.g.,

A6 FETCH 1 (BODY.PEEK[TEXT]<0.23>)
A6 NO [EXPUNGED 1] Some messages in the mailbox had previously been expunged and 
could not be returned.

A5 FETCH 1 (BODYSTRUCTURE)
* 1 FETCH (FLAGS (\Deleted))
A5 OK [EXPUNGED 1] FETCH completed.

Bill,

Part of the server behavior that caused you such problems is described in 
RFC 2180 section 4.1.2.

I objected to that section being including in RFC 2180.  In my opinion, 
the behaviors of RFC 2180 sections 4.1.1 and 4.1.4 are the only acceptable 
ones for the expunged message scenario.  A server with an inferior mail 
store may be forced to exhibit the 4.1.3 behavior; but this should not be 
considered "acceptable" behavior.

The best behavior is section 4.1.1, and this is what the mbx and mix mail 
format support in UW imapd does.

In any case, I consider any server that returns NO to a FETCH to be broken 
in spite of RFC 2180 section 4.1.2.  I deeply regret ever listing NO as a 
possible response to a FETCH in the base specification; it was there "for 
completeness" at a time before it was understood that some commands didn't 
have to have all three responses.

However, this is definitely a server bug:
 	A5 FETCH 1 (BODYSTRUCTURE)
 	* 1 FETCH (FLAGS (\Deleted))
 	A5 OK FETCH completed.
since the server replied OK without returning a BODYSTRUCTURE (not even a 
fake RFC 2180 section 4.1.3 one).

Just about the only reasonable thing that you can do, following unexpected 
responses to a FETCH, is to do a NOOP (which would allow untagged EXPUNGE 
events) and see if that makes the world become sane again.

As for an EXPUNGED response code, I would much rather abolish RFC 2180 
section 4.1.2 (and preferably also section 4.1.3) entirely than add 
another complication to an already overly-complex protocol.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin wrote:
> Bill,
> 
> Part of the server behavior that caused you such problems is described 
> in RFC 2180 section 4.1.2.
> 
> I objected to that section being including in RFC 2180.  In my opinion, 
> the behaviors of RFC 2180 sections 4.1.1 and 4.1.4 are the only 
> acceptable ones for the expunged message scenario.  A server with an 
> inferior mail store may be forced to exhibit the 4.1.3 behavior; but 
> this should not be considered "acceptable" behavior.

I agree, 4.1.3 is the worst by far.  There's just no reasonable way for
the client to know what's going on.

> The best behavior is section 4.1.1, and this is what the mbx and mix 
> mail format support in UW imapd does.
> 
> In any case, I consider any server that returns NO to a FETCH to be 
> broken in spite of RFC 2180 section 4.1.2.  I deeply regret ever listing 
> NO as a possible response to a FETCH in the base specification; it was 
> there "for completeness" at a time before it was understood that some 
> commands didn't have to have all three responses.
> 
> However, this is definitely a server bug:
>     A5 FETCH 1 (BODYSTRUCTURE)
>     * 1 FETCH (FLAGS (\Deleted))
>     A5 OK FETCH completed.
> since the server replied OK without returning a BODYSTRUCTURE (not even 
> a fake RFC 2180 section 4.1.3 one).

Ok, I'll ask the customer to report it.  The customer claims the server is:
Microsoft Exchange Server 2003 IMAP4rev1 server version 6.5.6944.0

> Just about the only reasonable thing that you can do, following 
> unexpected responses to a FETCH, is to do a NOOP (which would allow 
> untagged EXPUNGE events) and see if that makes the world become sane again.

Yup, that looks like the best approach.

> As for an EXPUNGED response code, I would much rather abolish RFC 2180 
> section 4.1.2 (and preferably also section 4.1.3) entirely than add 
> another complication to an already overly-complex protocol.

4.1.2 is tolerable since at least the client has some clue that something
is wrong, although an EXPUNGED response code would still be helpful in
this case.

4.1.3 is completely unacceptable.  If this remains allowed behavior, an
EXPUNGED response code should be *required* for this case.

Thanks for your help.

Hi Bill -

I appreciate your feelings about 4.1.3 being worse than 4.1.2; but 
empirical evidence over more than a decade shows that 4.1.3 actually 
causes less problems for clients than 4.1.2.

The problem with 4.1.2 is that most clients lack the slightest clue as to 
what to do with a NO response to a FETCH.  Some will loop, repeating the 
FETCH.  Others will crash.

Clients generally don't handle 4.1.3 well either.  But they don't loop or 
crash.

For what it's worth, one of the mail stores supported by UW imapd exhibits 
4.1.3 behavior.  That particular mail store has been deprecated for over a 
decade.  Almost all UW imapd sites use mail stores that do either 4.1.4 or 
(in the case of modern mail stores) 4.1.1.

However, rather than argue the point, let's agree to say that both 4.1.2 
and 4.1.3 are broken behaviors, and neither should be allowed.  Since (in 
my opinion, anyway) 4.1.2 and 4.1.3 are behaviors of old broken mail 
stores, adding an EXPUNGED code won't help since old broken code isn't 
going to get fixed.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Mark Crispin wrote:
> Hi Bill -
> 
> I appreciate your feelings about 4.1.3 being worse than 4.1.2; but 
> empirical evidence over more than a decade shows that 4.1.3 actually 
> causes less problems for clients than 4.1.2.

Most clients are incredibly stupid.  Most of my time spent supporting
JavaMail is spent finding bugs in servers that don't cause problems
for any other clients, which of course leads people to believe the
bug is in JavaMail.  Most clients just treat IMAP as a glorified POP.

4.1.3 is worst for clients that try to do the right thing.  I guess
4.1.2 is worst for stupid clients, which sadly is the majority.

I'd be happy to have both 4.1.2 and 4.1.3 declared as not allowed
behavior.

What a shame there isn't a test suite to test conformance with the
IMAP spec...

> What a shame there isn't a test suite to test conformance with the
> IMAP spec...

Conformance isn't the problem.  Some of the worst behaviour imaginable is 
exhibited by fully compliant clients and servers.

--lyndon

On Tue, 12 Sep 2006, Bill Shannon wrote:
> Most clients are incredibly stupid.  Most of my time spent supporting
> JavaMail is spent finding bugs in servers that don't cause problems
> for any other clients, which of course leads people to believe the
> bug is in JavaMail.  Most clients just treat IMAP as a glorified POP.

All true, especially the last point.

> 4.1.3 is worst for clients that try to do the right thing.  I guess
> 4.1.2 is worst for stupid clients, which sadly is the majority.

Actually, 4.1.2 is troublesome for many clients simply because a NO 
response to a FETCH isn't particularly meaningful or sensible.

4.1.3 has a compliance problem, since it generally creates an 
inconsistency with sizes that were reported earlier.  But it's less 
difficult for many clients to cope with.

> I'd be happy to have both 4.1.2 and 4.1.3 declared as not allowed
> behavior.

Sold!  ;-)

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Bill Shannon writes:
> I'd be happy to have both 4.1.2 and 4.1.3 declared as not allowed behavior.

That would leave us with only 4.1.1 and 4.1.4, which make it practically 
impossible to delete a message. I don't like the security implications 
of that.

> What a shame there isn't a test suite to test conformance with the 
> IMAP spec...

Indeed.

Arnt

Lyndon Nerenberg wrote:
>> What a shame there isn't a test suite to test conformance with the
>> IMAP spec...
> 
> 
> Conformance isn't the problem.  Some of the worst behaviour imaginable 
> is exhibited by fully compliant clients and servers.

It's the difference between "good" and "correct".  I'd start with
"correct" and work up to "good".

On Tue, 12 Sep 2006, Arnt Gulbrandsen wrote:
> Bill Shannon writes:
>> I'd be happy to have both 4.1.2 and 4.1.3 declared as not allowed behavior.
> That would leave us with only 4.1.1 and 4.1.4, which make it practically 
> impossible to delete a message. I don't like the security implications of 
> that.

It would require an extremely abusive client to cause that result; one 
that was active but only did FETCH, STORE, and SEARCH commands for an 
extended period of time, thus blocking the untagged EXPUNGE.

To date, this has not been a problem.

If this were to become a problem, a server would be perfectly within its 
rights to set a maximum time that this could go on, and when the time is 
exceeded it issues an untagged BYE.

I would not mind adding text to that effect to the base specification if 
there is concensus that such text is a good idea.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

> It's the difference between "good" and "correct".  I'd start with
> "correct" and work up to "good".

I'd start with defining "good."  The problem with a protocol as complex 
(some would say bloated) as IMAP is that -- from the client's perspective 
-- there is no one "right" way to use it.  Different client 
implementations have different goals, and thus different demands of the 
server.  In some cases it makes sense for the client to (say) talk to the 
server using a restricted feature set that, when examined without context, 
makes it look like the client is just a glorified POP engine.

And if you do ever manage to define "good" you then get to solve the 
problem of determining what constitutes "compliant" behaviour when faced 
with the n! permutations of extensions interacting with and without each 
other.

This isn't going to happen in my lifetime.

--lyndon

Mark Crispin writes:
> On Tue, 12 Sep 2006, Arnt Gulbrandsen wrote:
>> Bill Shannon writes:
>>> I'd be happy to have both 4.1.2 and 4.1.3 declared as not allowed behavior.
>> That would leave us with only 4.1.1 and 4.1.4, which make it 
>> practically impossible to delete a message. I don't like the 
>> security implications of that.
>
> It would require an extremely abusive client to cause that result; one 
> that was active but only did FETCH, STORE, and SEARCH commands for an 
> extended period of time, thus blocking the untagged EXPUNGE.

Yes.

> To date, this has not been a problem.

Sure. The average case is OK, what I dislike is the lack of a proper 
transaction. Or you could say I don't like the fuzzy semantics of the 
EXPUNGE command. When a client issues EXPUNGE and the server issues a 
tagged OK, what is the server promising the client?

If that OK means "the messages have been deleted", I think that's a fine 
promise. A transaction has been committed and that's how it is.

If the OK means "the messages have been deleted, but some other active 
clients can see the old version of the mailbox for as much as two 
minutes from now", I think that's still good. Not ideal, but still 
clear and firm.

If the OK means "the messages will be deleted, except that a 
nasty-minded client can make me keep them around forever", I don't like 
that. It hollows out EXPUNGE too much (even though as you say, noone's 
been nasty-minded so far). Do you understand my concern?

> If this were to become a problem, a server would be perfectly within 
> its rights to set a maximum time that this could go on, and when the 
> time is exceeded it issues an untagged BYE.
>
> I would not mind adding text to that effect to the base specification 
> if there is concensus that such text is a good idea.

IMO,  it depends on the precise text. Requiring that all clients offer 
the server an opportunity to send EXPUNGE every so often can't be all 
that bad, but how often? "EXPUNGE may not take effect completely until 
half an hour after the tagged OK" sounds overly long to me. "Clients 
must give the server the opportunity to send EXPUNGE at two-minute 
intervals or more often" sounds much too short. Ten minutes sounds both 
too short and too long. All partly subjective of course.

Arnt

Lyndon Nerenberg wrote:
>> It's the difference between "good" and "correct".  I'd start with
>> "correct" and work up to "good".
> 
> 
> I'd start with defining "good."  The problem with a protocol as complex 
> (some would say bloated) as IMAP is that -- from the client's 
> perspective -- there is no one "right" way to use it.  Different client 
> implementations have different goals, and thus different demands of the 
> server.  In some cases it makes sense for the client to (say) talk to 
> the server using a restricted feature set that, when examined without 
> context, makes it look like the client is just a glorified POP engine.

Exactly, "good" is a value judgement, which makes it harder to define.

The spec should be very clear on what "correct" behavior is.

> And if you do ever manage to define "good" you then get to solve the 
> problem of determining what constitutes "compliant" behaviour when faced 
> with the n! permutations of extensions interacting with and without each 
> other.
> 
> This isn't going to happen in my lifetime.

Neither the spec nor the test suite need to be perfect to deliver great
value.  I see a lot of basic functionality that's broken.  A test suite
that tested those cases would make a huge difference.  Specifying, and
testing, the esoteric corner cases can be done later.

On Tue, 12 Sep 2006, Lyndon Nerenberg wrote:
> I'd start with defining "good."  The problem with a protocol as complex (some 
> would say bloated) as IMAP is that -- from the client's perspective -- there 
> is no one "right" way to use it.

I would disagree... ;-)

RFC 1064 and 1176 had some text outlining the right way to go about 
looking at the roles of client and server in IMAP.  But I took so much 
criticism about how I did not "understand" the architecture of IMAP(!) 
that I removed that text in RFC 1730.  [The critics in question are no 
longer active members of the IMAP community.]

Briefly:

There is a mailbox "state", of which the server has full knowledge.  The 
server has the ability, at will, to update and add to the client's state. 
The client can also request the server to send updates/additions, and the 
server sends back a done response when it has finished doing so.

Once the client establishes a session with a mailbox on the server, the 
server immediately starts sending some updates (e.g., EXISTS).  The 
client, on an as-needed basis, references data from its state and when a 
hole is encountered requests a state addition from the server.  The client 
may also request state additions on an anticipatory basis.  Think of how 
page fault handling works in a demand-paged system.

In the online model, a client starts with empty state.  In the 
disconnected model, the client has some remembered state and has to 
synchronize that remembered state with the server's state.


Anyway, most clients don't do all this.  They just download-and-delete 
(pure offline model POP-style).  A few are online, but don't keep a state 
so they are constantly re-FETCHing the same static data.

> Different client implementations have 
> different goals, and thus different demands of the server.  In some cases it 
> makes sense for the client to (say) talk to the server using a restricted 
> feature set that, when examined without context, makes it look like the 
> client is just a glorified POP engine.

True.  However, even such clients can still implement the underlying IMAP 
architecture properly.  I have to wonder when I see clients that 
repeatedly re-fetch the same static data in a single session.

> And if you do ever manage to define "good" you then get to solve the problem 
> of determining what constitutes "compliant" behaviour when faced with the n! 
> permutations of extensions interacting with and without each other.
> This isn't going to happen in my lifetime.

Sadly, all true.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Thu, 14 Sep 2006, Arnt Gulbrandsen wrote:
> Sure. The average case is OK, what I dislike is the lack of a proper 
> transaction. Or you could say I don't like the fuzzy semantics of the EXPUNGE 
> command. When a client issues EXPUNGE and the server issues a tagged OK, what 
> is the server promising the client?

You are looking at it the wrong way.

Instead of looking at it from the protocol viewpoint, you are looking at 
it from the viewpoint of how some server implementation may implement it.

More importantly, you are betraying an unrealistic expectation of what the 
IMAP protocol can do: to wit, that an IMAP client can force destruction of 
a message in other clients.  Doesn't happen.  Can't happen.

The best that can happen is a mutually-cooperative effort; and perhaps an 
entity that refuses to be cooperative can be expelled from the effort.

In IMAP, a message exists in as many as four places:
  (1) the mailbox on the server
  (2) server state in a client/server session
  (3) client state in a client/server session
  (4) client cache of the mailbox

When a client issues an expunge command and gets back an untagged EXPUNGE 
result, then the message has been expunged from (1).  It is also expunged 
from (2) IN THAT SESSION.  If the client implements IMAP correctly, it is 
also expunged from (3).  IMAP has no control over (4), although one would 
presume that a well-implemented client would expunge it in (4) as well.

What is confusing you is (2) and (3) IN OTHER SESSIONS, and (4) in other 
clients; and somehow you have the issue that this renders the semantics of 
EXPUNGE "fuzzy".  The semantics of EXPUNGE are not fuzzy at all; they are 
quite precise.

You need to get it out of your head that the IMAP protocol has any control 
over (4).  There is no way that a forced removal of (4) can ever happen. 
A sufficiently recalictrant client can prevent a forced removal of (3) as 
well, although eventually it'll have to move it to (4).

A server implementation MAY be able to force the issue for (2) in the face 
of a recalcitrant client.  However, the proper way to do that is not by 
pulling the rug from under a well-behaved client that will presently do a 
NOOP or other EXPUNGE-synchronize operation.  A proper way would be a 
forced BYE on a badly-behaved client.

> IMO,  it depends on the precise text. Requiring that all clients offer the 
> server an opportunity to send EXPUNGE every so often can't be all that bad, 
> but how often? "EXPUNGE may not take effect completely until half an hour 
> after the tagged OK" sounds overly long to me. "Clients must give the server 
> the opportunity to send EXPUNGE at two-minute intervals or more often" sounds 
> much too short. Ten minutes sounds both too short and too long. All partly 
> subjective of course.

In my opinion, the minimum period should be between 5 and 30 minutes.  NAT 
has pretty much made it non-viable to go idle for more than 5 minutes.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue, 12 Sep 2006, Bill Shannon wrote:
> The spec should be very clear on what "correct" behavior is.

The base spec supposedly is very clear on that point.  Most of the 
arguments are not about what is correct, but rather about what is good.

> Neither the spec nor the test suite need to be perfect to deliver great
> value.  I see a lot of basic functionality that's broken.  A test suite
> that tested those cases would make a huge difference.  Specifying, and
> testing, the esoteric corner cases can be done later.

Unfortunately, "the esoteric corner cases can be done later" covers a 
great many sins.  What is an "esoteric corner case" to some people is an 
essential functionality to others.

Much of the problem comes from individuals who claim that IMAP is too 
complicated to implement from the spec, and thus it is alright to look at 
the examples and implement empirically to work with whatever server they 
have access to (client implementors) or Outlook and Netscape (server 
implementors).

A common server defies the specification in multiple places; when the 
violations are pointed out its author's answer is "yawn" or accusations of 
FUD-spreading.  There isn't much can be done in that case.  If a test 
suite was produced, such people would attack the motivations of the 
authors of the test suite rather than fix the problems.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue Sep 12 22:25:59 2006, Mark Crispin wrote:
> RFC 1064 and 1176 had some text outlining the right way to go about 
> looking at the roles of client and server in IMAP.  But I took so 
> much criticism about how I did not "understand" the architecture of 
> IMAP(!) that I removed that text in RFC 1730.  [The critics in 
> question are no longer active members of the IMAP community.]
> 
> 
If it makes you feel better, I went back to those descriptions when 
designing Polymer, and found them very useful.

> In the online model, a client starts with empty state.  In the 
> disconnected model, the client has some remembered state and has to 
> synchronize that remembered state with the server's state.

FWIW, Polymer does both, in as much as it's an online client that 
manages to persist its cache between sessions. It still doesn't do 
anticipatory fetching, although I keep meaning to start putting that 
in.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Mark Crispin writes:
> On Thu, 14 Sep 2006, Arnt Gulbrandsen wrote:
>> Sure. The average case is OK, what I dislike is the lack of a proper 
>> transaction. Or you could say I don't like the fuzzy semantics of 
>> the EXPUNGE command. When a client issues EXPUNGE and the server 
>> issues a tagged OK, what is the server promising the client?
>
> You are looking at it the wrong way.
>
> Instead of looking at it from the protocol viewpoint, you are looking 
> at it from the viewpoint of how some server implementation may 
> implement it.

I'd say: I'm looking at what the promises made by IMAP standard to 
client authors, which standards-conformant servers must fulfill. In 
particular what promises made about the effects of the EXPUNGE command.

> More importantly, you are betraying an unrealistic expectation of what 
> the IMAP protocol can do: to wit, that an IMAP client can force 
> destruction of a message in other clients.  Doesn't happen.  Can't 
> happen.

Of course it can't.

My issue is rather the opposite - can a client force destruction of a 
message on a server, such that other IMAP clients cannot access the 
message?

Seven hundred other clients may have cached the message already, and 
even logged out. That's a given. Time moves forwards, after all. All 
the EXPUNGE command can affect is the ability of other clients to fetch 
the messages after the time of the EXPUNGE command.

Even though EXPUNGE cannot affect the past, I think its effect on the 
future are worth discussing.

(Btw, I changed our code today to use 4.1.2+loopbreaker. The first time 
a client fetches a message which another client has expunged, but whose 
expunge has not yet been reported in this session, the server issues a 
NO as in 4.1.2, and makes a note of the UID(s). If any of those UIDs 
are fetched again before the server can report the expunge, the server 
issues a BYE. When it reports expunges, it clears its UID set. I think 
that's as good as 4.1.1+period.)

>> IMO,  it depends on the precise text. Requiring that all clients 
>> offer the server an opportunity to send EXPUNGE every so often can't 
>> be all that bad, but how often? "EXPUNGE may not take effect 
>> completely until half an hour after the tagged OK" sounds overly 
>> long to me. "Clients must give the server the opportunity to send 
>> EXPUNGE at two-minute intervals or more often" sounds much too 
>> short. Ten minutes sounds both too short and too long. All partly 
>> subjective of course.
>
> In my opinion, the minimum period should be between 5 and 30 minutes.  
> NAT has pretty much made it non-viable to go idle for more than 5 
> minutes.

Then five minutes.

Arnt

On Fri Sep 15 02:57:15 2006, Mark Crispin wrote:
> In my opinion, the minimum period should be between 5 and 30 
> minutes.  NAT has pretty much made it non-viable to go idle for 
> more than 5 minutes.

I'm not sure that's true anymore. Specifically, I know that Linux's 
NAT implementation, which accounts for several of the common DSL 
routers, doesn't timeout a silent connection until a matter of days.

It might be interesting to talk to the NAT implementors, and see what 
timeouts they're actually using these days.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Mark Crispin wrote:
> On Tue, 12 Sep 2006, Bill Shannon wrote:
> 
>> The spec should be very clear on what "correct" behavior is.
> 
> 
> The base spec supposedly is very clear on that point.  Most of the 
> arguments are not about what is correct, but rather about what is good.

I'm sure that's true on this mailing list, but the problems I deal
with day to day are about correctness.

>> Neither the spec nor the test suite need to be perfect to deliver great
>> value.  I see a lot of basic functionality that's broken.  A test suite
>> that tested those cases would make a huge difference.  Specifying, and
>> testing, the esoteric corner cases can be done later.
> 
> 
> Unfortunately, "the esoteric corner cases can be done later" covers a 
> great many sins.  What is an "esoteric corner case" to some people is an 
> essential functionality to others.

A test suite that tested the consensus essential functionality would
find a great many server bugs and would be of great benefit to customers
evaluating and using IMAP servers.

> Much of the problem comes from individuals who claim that IMAP is too 
> complicated to implement from the spec, and thus it is alright to look 
> at the examples and implement empirically to work with whatever server 
> they have access to (client implementors) or Outlook and Netscape 
> (server implementors).
> 
> A common server defies the specification in multiple places; when the 
> violations are pointed out its author's answer is "yawn" or accusations 
> of FUD-spreading.  There isn't much can be done in that case.  If a test 
> suite was produced, such people would attack the motivations of the 
> authors of the test suite rather than fix the problems.

I already don't even waste time debugging problems related to *that*
server, because they always turn out to be bugs in the server.  A test
suite would allow customers to apply some pressure to the server vendor.

On 15.9.2006, at 16.53, Arnt Gulbrandsen wrote:

> (Btw, I changed our code today to use 4.1.2+loopbreaker. The first  
> time a client fetches a message which another client has expunged,  
> but whose expunge has not yet been reported in this session, the  
> server issues a NO as in 4.1.2, and makes a note of the UID(s). If  
> any of those UIDs are fetched again before the server can report  
> the expunge, the server issues a BYE. When it reports expunges, it  
> clears its UID set. I think that's as good as 4.1.1+period.)

Has this been working? I was thinking about changing my current  
behavior of simply disconnecting the client immediately when this  
happens (which is annoying when it happens). There's currently no  
other real alternatives for me than this 4.1.2+loopbreaker or 4.1.3.  
Making it admin-configurable is probably a bad idea too.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 193 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20061228/3a111cf6/attachment.sig>

On 15.9.2006, at 16.53, Arnt Gulbrandsen wrote:

>>> IMO,  it depends on the precise text. Requiring that all clients  
>>> offer the server an opportunity to send EXPUNGE every so often  
>>> can't be all that bad, but how often? "EXPUNGE may not take  
>>> effect completely until half an hour after the tagged OK" sounds  
>>> overly long to me. "Clients must give the server the opportunity  
>>> to send EXPUNGE at two-minute intervals or more often" sounds  
>>> much too short. Ten minutes sounds both too short and too long.  
>>> All partly subjective of course.
>>
>> In my opinion, the minimum period should be between 5 and 30  
>> minutes.  NAT has pretty much made it non-viable to go idle for  
>> more than 5 minutes.
>
> Then five minutes.

5 minutes sounds pretty short to me. Have you tried adding such a  
check yet and see what it does to clients? I was thinking something  
like 15-30 minutes.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 193 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20061228/9093580d/attachment.sig>

On Fri, 15 Sep 2006, Arnt Gulbrandsen wrote:
>> Instead of looking at it from the protocol viewpoint, you are looking at it 
>> from the viewpoint of how some server implementation may implement it.
> I'd say: I'm looking at what the promises made by IMAP standard to client 
> authors, which standards-conformant servers must fulfill. In particular what 
> promises made about the effects of the EXPUNGE command.

It's not much different from what promises are made by operating systems 
when a process deletes a file when some other process has the file open. 
On UNIX, the file is deleted; on Windows NT, the delete is rejected.

Neither pulls the rug out from the other process.  Specifically, on UNIX 
the other process continues to access the file and its data for as long as 
it wants as if it wasn't deleted.

> My issue is rather the opposite - can a client force destruction of a message 
> on a server, such that other IMAP clients cannot access the message?

It can stop new access; that is, a new SELECT from accessing the message.

What it can not do is stop IMAP clients which are already accessing the 
message (which is what SELECT does) from accessing the message.  It can 
only get those other clients to de-access the message at defined points in 
their session.  Those points are normally quite frequent in an IMAP 
session.

> (Btw, I changed our code today to use 4.1.2+loopbreaker. The first time a 
> client fetches a message which another client has expunged, but whose expunge 
> has not yet been reported in this session, the server issues a NO as in 
> 4.1.2, and makes a note of the UID(s). If any of those UIDs are fetched again 
> before the server can report the expunge, the server issues a BYE. When it 
> reports expunges, it clears its UID set. I think that's as good as 
> 4.1.1+period.)

I think that 4.1.2 is a bug, and servers that do it are broken.

I think that the security issues are specious (otherwise, we wouldn't be 
using UNIX).  It's all a matter of 4.1.1/4.1.4 being difficult to 
implement with maildir-style one-file/one-message mail store and 
4.1.2/4.1.3 being easy.

When a client author complains to me about such servers (and they do 
complain), my response is "complain to that server's vendor."

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Fri, 15 Sep 2006, Dave Cridland wrote:
>> In my opinion, the minimum period should be between 5 and 30 minutes.  NAT 
>> has pretty much made it non-viable to go idle for more than 5 minutes.
> I'm not sure that's true anymore. Specifically, I know that Linux's NAT 
> implementation, which accounts for several of the common DSL routers, doesn't 
> timeout a silent connection until a matter of days.

I would expect that Linux plays nice.

That doesn't apply to the non-Linux DSL routers and standalone NAT boxes 
out there.  An attacker can make a Linksys box toss its NAT mappings in a 
matter of seconds from previous activity.

You can't assume that they're all Linux, so all is well.  Linux based 
boxes are the minority.  FWIW, Linksys used Linux for a while, and then 
went back to a proprietary OS.  So it isn't clear what the trend actually 
is.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Tue, 12 Sep 2006, Bill Shannon wrote:
> A test suite that tested the consensus essential functionality would
> find a great many server bugs and would be of great benefit to customers
> evaluating and using IMAP servers.

Ah, but if the "essential functionality" is anything less than the full 
capability of RFC 3501 you will have to find "concensus".  I think that 
you will encounter vigorous objection to any subset, no matter what you 
pick from RFC 3501 to leave out.  I'd be one of the objectors...

>> A common server defies the specification in multiple places; when the 
>> violations are pointed out its author's answer is "yawn" or accusations of 
>> FUD-spreading.  There isn't much can be done in that case.  If a test suite 
>> was produced, such people would attack the motivations of the authors of 
>> the test suite rather than fix the problems.
> I already don't even waste time debugging problems related to *that*
> server, because they always turn out to be bugs in the server.

Ha!  Note how nobody needs to name the server in question, we just all 
know which one it is.  That server is the Voldemort of the IMAP world...

> A test
> suite would allow customers to apply some pressure to the server vendor.

Perhaps.  I'm a bit more pessimistic.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

Timo Sirainen writes:
> On 15.9.2006, at 16.53, Arnt Gulbrandsen wrote:
>
>>  (Btw, I changed our code today to use 4.1.2+loopbreaker. The first
>>  time a client fetches a message which another client has expunged,
>>  but whose expunge has not yet been reported in this session, the
>>  server issues a NO as in 4.1.2, and makes a note of the UID(s). If
>>  any of those UIDs are fetched again before the server can report
>>  the expunge, the server issues a BYE. When it reports expunges, it
>>  clears its UID set. I think that's as good as 4.1.1+period.)
>
> Has this been working?

We haven't had a single complaint or bug report, and more than two 
months have passed, so it can't be terrible.

> I was thinking about changing my current behavior of simply 
> disconnecting the client immediately when this happens (which is 
> annoying when it happens). There's currently no other real 
> alternatives for me than this 4.1.2+loopbreaker or 4.1.3.

All rather difficult, isn't it?

> Making it admin-configurable is probably a bad idea too.

I agree.

Arnt

Timo Sirainen writes:
> On 15.9.2006, at 16.53, Arnt Gulbrandsen wrote:
>>>>  IMO,  it depends on the precise text. Requiring that all clients
>>>>  offer the server an opportunity to send EXPUNGE every so often
>>>>  can't be all that bad, but how often? "EXPUNGE may not take
>>>>  effect completely until half an hour after the tagged OK" sounds
>>>>  overly long to me. "Clients must give the server the opportunity
>>>>  to send EXPUNGE at two-minute intervals or more often" sounds
>>>>  much too short. Ten minutes sounds both too short and too long.
>>>>  All partly subjective of course.
>>>
>>>  In my opinion, the minimum period should be between 5 and 30
>>>  minutes.  NAT has pretty much made it non-viable to go idle for
>>>  more than 5 minutes.
>>
>>  Then five minutes.
>
> 5 minutes sounds pretty short to me. Have you tried adding such a 
> check yet and see what it does to clients? I was thinking something 
> like 15-30 minutes.

I think I'll follow rather than lead on this one. Now that I have the 
loopbreaker it's not much of a problem from my viewpoint.

Arnt

Mark Crispin wrote:
> On Tue, 12 Sep 2006, Bill Shannon wrote:
> 
>> A test suite that tested the consensus essential functionality would
>> find a great many server bugs and would be of great benefit to customers
>> evaluating and using IMAP servers.
> 
> 
> Ah, but if the "essential functionality" is anything less than the full 
> capability of RFC 3501 you will have to find "concensus".  I think that 
> you will encounter vigorous objection to any subset, no matter what you 
> pick from RFC 3501 to leave out.  I'd be one of the objectors...

I was suggesting a consensus on what to include.  Perhaps I should've
said "majority opinion".  Any reasonable decision making process that
avoids deadlock will do.  If there's something you want in but no one
else wants in, you lose.  But at least we end up with a test suite that
has stuff that "everyone" wants in, which is infinitely better than the
complete lack of a test suite we have today.

>> A test
>> suite would allow customers to apply some pressure to the server vendor.
> 
> 
> Perhaps.  I'm a bit more pessimistic.

Test suites have been very beneficial to the success of Java technology.