mailing list archives

Hi,

I recently got a report from a user connecting to our imap server about an IDLE-connection dropping with an error message. Client was Outlook Express which showed an error like "lost connection to imap server". After some investigation I found the reason in a modern AVM FritzBox NAT router, which seems to close inactive connections after about 15 minutes.

Now, as IMAP-IDLE allows for about 30 minutes inactivity by design, this behavior of the NAT router is not nice at all, and as I could not change this behavior by any setting on the router, I was rather stuck.

So, I changed the server-implementation of the IDLE command like the following: If the user is IDLE for more than 290secs (near to 5 minutes, the least I could imagine for a NAT router dropping a connection), I repeat the last EXISTS to keep the connection open. E.g.

[C] abrr SELECT "INBOX"
[S] * 1 EXISTS
[S] * 0 RECENT
[S] [...]
[S] abrr OK [READ-WRITE] SELECT completed
[C] brum IDLE
[S] + idling
[S] * 2 EXISTS   // new message arrives
[S] * 1 RECENT   // new message arrives (2)
[S] * 2 EXISTS   // 290secs inactivity
[S] * 2 EXISTS   // another 290secs inactivity
[C] DONE
[S] brum OK IDLE terminated

Now - is this protocol compliant (I think yes). Is there any scenario where this may confuse clients? Any other problem imaginable?

Christof
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20060918/b7887c3b/attachment.html>

At 2006-09-18 09:07:10 +0200, drescher@pro-image.de wrote:
>
> Now - is this protocol compliant (I think yes). Is there any scenario
> where this may confuse clients? Any other problem imaginable?

It's compliant, and it works reasonably well.

-- ams

You are lucky if your NAT box waits 15 minutes before closing inactive 
connections.  I've encountered NAT boxes that do it in less than 5, and 
can be made to dump connections even faster than that.

It isn't necessary for the server to send an untagged EXISTS as a NAT 
keepalive (although it's certainly permitted); an untagged OK will do just 
as well.

Also of concern is that clients from a certain large and very well-known 
software house in Redmond, WA will fail to exit out of IDLE before the 30 
minute autologout expires.  This causes the session to be disconnected in 
a non-obvious way to the user, who then complains that he isn't getting 
notified about new mail.

The workaround for this problem is, after 29 minutes with no new mail, to 
send a fake EXISTS that alleges that there is one new message.  Then, when 
the client issues a DONE to exit IDLE mode, send a fake EXPUNGE to revoke 
that "new message".  This will cause that client to issue a new idle.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon Sep 18 16:22:40 2006, Mark Crispin wrote:
> The workaround for this problem is, after 29 minutes with no new 
> mail, to send a fake EXISTS that alleges that there is one new 
> message.  Then, when the client issues a DONE to exit IDLE mode, 
> send a fake EXPUNGE to revoke that "new message".  This will cause 
> that client to issue a new idle.

Whilst I'm impressed at the devious nature of that workaround, it 
occurs to be that you'd have to be fairly careful, since:

1) You'd have to ensure the EXPUNGE was sent prior to the tagged OK 
of the IDLE. Easy enough, but:

2) You'd also have to increment UIDNEXT etc in order to avoid 
confusing the client later.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 18 Sep 2006, Dave Cridland wrote:
> Whilst I'm impressed at the devious nature of that workaround,

Yes, "devious" is definitely the word for it.  I might even say "evil".

> 1) You'd have to ensure the EXPUNGE was sent prior to the tagged OK of the 
> IDLE. Easy enough, but:
> 2) You'd also have to increment UIDNEXT etc in order to avoid confusing the 
> client later.

Well, yes and no.

Strictly speaking, you are correct.  However, it is one certain client 
that ever allows an IDLE to go for so long, and we can be confident that 
client will never use UIDNEXT.

IMHO, a client that
  (1) ignores the timeout rules of the IDLE specification
yet
  (2) cares about UIDNEXT
deserves to lose.  ;-)

Actually, a client that just does (1) deserves to lose.  Sadly, however, 
numerous individuals use that certain client.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin writes:
> However, it is one certain client that ever allows an IDLE to go for 
> so long, and we can be confident that client will never use UIDNEXT.

Uh. Mine allows IDLE to go on for 50 seconds longer and uses UIDNEXT.

It's not a great matter, but I urge you to reconsider this. It is the 
reference implementation, after all.

Arnt

On Mon, 18 Sep 2006, Arnt Gulbrandsen wrote:
>> However, it is one certain client that ever allows an IDLE to go for so 
>> long, and we can be confident that client will never use UIDNEXT.
> Uh. Mine allows IDLE to go on for 50 seconds longer and uses UIDNEXT.

RFC 2177 states:

    [...] clients using IDLE are advised to terminate the IDLE and
    re-issue it at least every 29 minutes to avoid being logged off.

You like cutting things that close?

Do you really mean to say that your client increments its copy of UIDNEXT 
every time EXISTS shows a new message (since UIDNEXT is normally NOT 
presented except at SELECT time!), and then issues an error message if a 
new message has a UID that is not .GE. that client calculated value?

Or, do you mean to say that your client increments its calculated value, 
and then issues an error message if UIDNEXT in a subsequent session is not 
.GE. that client calculated value?

> It's not a great matter, but I urge you to reconsider this. It is the 
> reference implementation, after all.

The protocol violation is the kludge to prevent IDLE from ever timing out. 
Removing that kludge would be better, if only ********* would fix its 
broken client.

It's also not clear, given the definition of UIDNEXT, that it is even a 
violation.  The text says "when new messages are added to a mailbox, even 
if those new messages are subsequently expunged."  After all, no messages 
were actually added to the mailbox; the EXISTS/EXPUNGE pair is a fake.

OK, granted, that's a slimy argument to make, but no slimier than other 
arguments made by other people in the past.  Don't I get to wallow in the 
slime too, just once?  ;-)

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mark Crispin writes:
> On Mon, 18 Sep 2006, Arnt Gulbrandsen wrote:
>>> However, it is one certain client that ever allows an IDLE to go for 
>>> so long, and we can be confident that client will never use 
>>> UIDNEXT.
>> Uh. Mine allows IDLE to go on for 50 seconds longer and uses UIDNEXT.
>
> RFC 2177 states:
>
>    [...] clients using IDLE are advised to terminate the IDLE and
>    re-issue it at least every 29 minutes to avoid being logged off.
>
> You like cutting things that close?

Yes. In my professional capacity I need to add workarounds and so on. 
Privately I like to follow the protocol and expect the other guy to do 
the same.

> Do you really mean to say that your client increments its copy of 
> UIDNEXT every time EXISTS shows a new message (since UIDNEXT is 
> normally NOT presented except at SELECT time!), and then issues an 
> error message if a new message has a UID that is not .GE. that client 
> calculated value?

No. MSN/UID arithmetic. The following sequence is somewhat contrived, 
but it's short and I expect you get the idea:

S: * 1000 EXISTS
S: * OK [UIDNEXT 2001]
S: * 1001 EXISTS
S: * 1001 EXPUNGE
S: * 1001 EXISTS
C: a UID FETCH 2002 ENVELOPE

There is fallback code to handle unexpected UIDNEXT increases, but not 
decreases.

>> It's not a great matter, but I urge you to reconsider this. It is the 
>> reference implementation, after all.
>
> The protocol violation is the kludge to prevent IDLE from ever timing 
> out. Removing that kludge would be better, if only ********* would 
> fix its broken client.
>
> It's also not clear, given the definition of UIDNEXT, that it is even 
> a violation.  The text says "when new messages are added to a 
> mailbox, even if those new messages are subsequently expunged."  
> After all, no messages were actually added to the mailbox; the 
> EXISTS/EXPUNGE pair is a fake.
>
> OK, granted, that's a slimy argument to make, but no slimier than 
> other arguments made by other people in the past.  Don't I get to 
> wallow in the slime too, just once?  ;-)

I was going to say "can't you do that wearing your pine hat?"  but after 
a night's sleep I'm not so sure. Here are two different answers. My 
opinion is somewhere in between, wavering.

1. Your server is the reference implementation. As I understand it, a 
reference implementation is one that implements the protocol 
completely, accurately, meticulously, and above all, without dubious 
hacks.

2. Your server has moved towards being an ordinary IMAP server for many 
years. For example, there isn't any command-line option to serve 
IMAP4rev1 without extensions if a client wishes to test that that 
works. A bit of a hack in the name of interoperability does veeeery 
little additional harm.

But wouldn't it be better to extend the 30-minute timeout than to throw 
a spanner in the works of clients that do MSN/UID arithmetic?

Arnt

On Tue, 19 Sep 2006, Arnt Gulbrandsen wrote:
>>    [...] clients using IDLE are advised to terminate the IDLE and
>>    re-issue it at least every 29 minutes to avoid being logged off.
>> You like cutting things that close?
> Yes. In my professional capacity I need to add workarounds and so on. 
> Privately I like to follow the protocol and expect the other guy to do the 
> same.

That wasn't the point.  Allowing an IDLE to go on for 29 minutes and 50 
seconds poses a terrible risk of losing the race with only 10 seconds of 
leeway.

> No. MSN/UID arithmetic. The following sequence is somewhat contrived, but 
> it's short and I expect you get the idea:
> S: * 1000 EXISTS
> S: * OK [UIDNEXT 2001]
> S: * 1001 EXISTS
> S: * 1001 EXPUNGE
> S: * 1001 EXISTS
> C: a UID FETCH 2002 ENVELOPE
> There is fallback code to handle unexpected UIDNEXT increases, but not 
> decreases.

There's a problem with your example.  UIDNEXT is only sent at SELECT time. 
It is not sent during UID FETCH or other routine operations.  Also, 
there's no reason to believe that that UID FETCH would work; UIDNEXT is 
only a prediction and not a promise.

So you can only be affected if you
  (1) keep a mathematical calculation of minimum UIDNEXT as a result of
      EXISTS
  (2) use that calculation between sessions.

> 1. Your server is the reference implementation. As I understand it, a 
> reference implementation is one that implements the protocol completely, 
> accurately, meticulously, and above all, without dubious hacks.

It would be easy to achieve this.  Elect me Dictator of the World, so I 
can outlaw broken IMAP clients that make such kludges necessary!!  As 
Dictator of the World, I promise a world of only perfect IMAP!  ;-)

Of course, there's the small problem that, like Koko in Gilbert & 
Sullivan's "The Mikado", "I have a little list"... ;-)

> But wouldn't it be better to extend the 30-minute timeout than to throw a 
> spanner in the works of clients that do MSN/UID arithmetic?

I might extend the timeout to 32 minutes if I had evidence that certain 
broken clients wouldn't screw up in other ways.

But the question is if any client is actually affected.  As noted above, 
UIDNEXT is only sent at SELECT time.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Tue Sep 19 16:58:03 2006, Mark Crispin wrote:
> There's a problem with your example.  UIDNEXT is only sent at 
> SELECT time. It is not sent during UID FETCH or other routine 
> operations.  Also, there's no reason to believe that that UID FETCH 
> would work; UIDNEXT is only a prediction and not a promise.
> 
> So you can only be affected if you
>  (1) keep a mathematical calculation of minimum UIDNEXT as a result 
> of
>      EXISTS
>  (2) use that calculation between sessions.
> 
> 
True, all true. Except that Arnt is not the only one who uses the 
property of UIDNEXT that it is at least 1 higher than the maximal UID 
ever used, and usually precisely that - as in, it will increase by 
precisely one on the addition of any message, and will never decrease.

I certainly use a combination of UIDNEXT values and counted EXPUNGE 
messages to detect whether there have been unwitnessed EXPUNGEs 
between SELECTs.


>> 1. Your server is the reference implementation. As I understand 
>> it, a reference implementation is one that implements the protocol 
>> completely, accurately, meticulously, and above all, without 
>> dubious hacks.
> 
> It would be easy to achieve this.  Elect me Dictator of the World, 
> so I can outlaw broken IMAP clients that make such kludges 
> necessary!!  As Dictator of the World, I promise a world of only 
> perfect IMAP!  ;-)
> 
> 
I'll mention it to NOMCOM.

> But the question is if any client is actually affected.  As noted 
> above, UIDNEXT is only sent at SELECT time.

I think mine might be under certain circumstances. I don't think it 
would actually choke, but I believe it might cause an additional 
round-trip that proved to be a waste of effort. It'd almost certainly 
pipeline a UID SEARCH with the terminating DONE, in any case, and 
then it would generate another - I think - on a subsequent SELECT of 
the same mailbox.

I have to say that I'm generally unconvinced if timeouts of this 
nature are the correct thing to do anyway. I think maybe sending 
untagged OK responses and seeing whether they work, and do not block, 
might be a more useful thing to do, but I fully admit to not having 
thought this through.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade