mailing list archives

Maybe my hubris got the better of me, but I didn't bargain for a complete surprise.  Well, anyway, we know now why Apple does not implement IMAP IDLE in iOS.  I've clearly been spending too much time around the IETF, to find Mr. Job's explanation to be completely incomprehensible. :-)

(Please let me know if the Message/RFC822 part didn't come through right - thanks.)

I want to ask anybody who feels strongly to the contrary to please not attack the sender (and the messenger either if you can help it :-) ).  I guess I'm stuck waiting 15 minutes for new mail notifications, and running my battery down.  I'm not forwarding my mail anywhere or running Exchange (or a clone).  The latter, in particular, is a power-hungry option ...

Cheers,
Sabahattin
-------------- next part --------------
An embedded message was scrubbed...
From: Steve Jobs <sjobs@apple.com>
Subject: Re: iOS IMAP IDLE (Standard "Push Email") Deficiency, Explanation?
Date: Mon, 04 Oct 2010 07:58:15 -0700
Size: 2611
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/e96c54cf/attachment.eml>

On 04/10/2010 18:18, Sabahattin Gucukoglu wrote:

> Maybe my hubris got the better of me, but I didn't bargain for a complete surprise.  Well, anyway, we know now why Apple does not implement IMAP IDLE in iOS.  I've clearly been spending too much time around the IETF, to find Mr. Job's explanation to be completely incomprehensible. :-)
> 
> (Please let me know if the Message/RFC822 part didn't come through right - thanks.)
> 
> I want to ask anybody who feels strongly to the contrary to please not attack the sender (and the messenger either if you can help it :-) ).  I guess I'm stuck waiting 15 minutes for new mail notifications, and running my battery down.  I'm not forwarding my mail anywhere or running Exchange (or a clone).  The latter, in particular, is a power-hungry option ...

This makes complete sense. In order to use IMAP Idle on a phone, it
would have to keep a TCP connection open, and therefore a 3G connection
open. There's a reason why phone makers advertise separate stand by, and
call times for battery usage. If IMAP idle were being used, the phone
would never enter stand by mode, and would eat the battery within a few
hours.

I considered building a system for Android that works the same way that
push mail on the iPhone works. When an email comes in, an SMS would be
sent to the phone. That SMS would be hidden from the user, but would
advise the phone to wake up and poll for new email. The only problem is,
it costs money to send SMS's.

-- 
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog -              https://secure.grepular.com/
Follow me on Twitter -           http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

The answer is Android.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Mike Cardwell wrote:
> On 04/10/2010 18:18, Sabahattin Gucukoglu wrote:
>
>   
>> Maybe my hubris got the better of me, but I didn't bargain for a complete surprise.  Well, anyway, we know now why Apple does not implement IMAP IDLE in iOS.  I've clearly been spending too much time around the IETF, to find Mr. Job's explanation to be completely incomprehensible. :-)
>>
>> (Please let me know if the Message/RFC822 part didn't come through right - thanks.)
>>
>> I want to ask anybody who feels strongly to the contrary to please not attack the sender (and the messenger either if you can help it :-) ).  I guess I'm stuck waiting 15 minutes for new mail notifications, and running my battery down.  I'm not forwarding my mail anywhere or running Exchange (or a clone).  The latter, in particular, is a power-hungry option ...
>>     
>
> This makes complete sense. In order to use IMAP Idle on a phone, it
> would have to keep a TCP connection open, and therefore a 3G connection
> open. There's a reason why phone makers advertise separate stand by, and
> call times for battery usage. If IMAP idle were being used, the phone
> would never enter stand by mode, and would eat the battery within a few
> hours.
>   
I'm using the K9 client on my Sprint EVO.  AFAIK, K9 is the only android 
client that supports IDLE.  I have 3 accounts on my phone.  All have a 
constant IDLE connection.  I get easily 12-14 hours of battery life.  I 
plug it in when I get home, so that's plenty.

(off topic)  Using the GPS is what kills my battery.  With constant GPS 
usage, I can kill a battery in 3 hours.
> I considered building a system for Android that works the same way that
> push mail on the iPhone works. When an email comes in, an SMS would be
> sent to the phone. That SMS would be hidden from the user, but would
> advise the phone to wake up and poll for new email. The only problem is,
> it costs money to send SMS's.
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/3b158677/attachment.html>

On Mon, 4 Oct 2010, Mike Cardwell wrote:
> This makes complete sense. In order to use IMAP Idle on a phone, it
> would have to keep a TCP connection open, and therefore a 3G connection
> open.

Say what?

Is that a UMTS requirement?  It certainly is not a TCP requirement.  TCP
is built on IP which is completely datagram.  I have built on-demand
networks which shut down until signaled back on, and then happily resumed
all the active TCP sessions even though the "network connection" had been
powered off for days.  The same signaling mechanism that is used for SMS
can be used to restart the mobile device radio.

That is how push works on BlackBerry.

The real problem with IDLE is that it is NAT-unfriendly.  Most NAT boxes
drop the mapping after a period of client inactivity, and send a TCP RST
to the server if the server resumes activity before the client.  Most
users don't see this, since if the client resumes activity first the same
mapping is created and no RST is sent.

No desktop client should ever use IDLE; it's unnecessary unless you using
radio.  Unfortunately, most desktop clients do implement IDLE, and then
any user who uses that client behind NAT complains about losing their
connection to the server.  A certain client that shall go unnamed (but for
which you should look out) just stops announcing new mail and shows a very
inconspicuous icon that indicates that there isn't a connection to the
server.

That, in turn, leads either to removing IDLE from the server or doing a
workaround in IDLE that makes IDLE pointless.

Notwithstanding any of the above, I stand by my previous statement.  It's
a waste of time and effort to worry overmuch about iOS4.  Android is the
way forward.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon Oct  4 18:31:08 2010, Mike Cardwell wrote:
> On 04/10/2010 18:18, Sabahattin Gucukoglu wrote:
> 
> > Maybe my hubris got the better of me, but I didn't bargain for a  
> complete surprise.  Well, anyway, we know now why Apple does not  
> implement IMAP IDLE in iOS.  I've clearly been spending too much  
> time around the IETF, to find Mr. Job's explanation to be  
> completely incomprehensible. :-)
> >
> > (Please let me know if the Message/RFC822 part didn't come  
> through right - thanks.)
> >
> > I want to ask anybody who feels strongly to the contrary to  
> please not attack the sender (and the messenger either if you can  
> help it :-) ).  I guess I'm stuck waiting 15 minutes for new mail  
> notifications, and running my battery down.  I'm not forwarding my  
> mail anywhere or running Exchange (or a clone).  The latter, in  
> particular, is a power-hungry option ...
> 
> This makes complete sense. In order to use IMAP Idle on a phone, it
> would have to keep a TCP connection open, and therefore a 3G  
> connection
> open. There's a reason why phone makers advertise separate stand  
> by, and
> call times for battery usage. If IMAP idle were being used, the  
> phone
> would never enter stand by mode, and would eat the battery within a  
> few
> hours.
> 
> 
That's somewhat ill-informed, I'm afraid. I did quite a bit of  
research into this for XMPP, and I captrued most of the findings in  
XEP-0286.

The "3G session" does need to stay open, but it can stay in Idle, or  
PCH, modes. These cost in the region of 8mA.

Small notifications - including the EXISTS and FETCH FLAGS that IDLE  
typically emits from the server - will only raise the 3G session to  
FACH mode. If the handset is forced into FACH mode constantly,  
this'll drain the battery in around 7 hours, using around 140mA.

Only if the data size reaches a certain (small) threshold - about 128  
octets typically - will the radio rise to DCH mode, where the drain  
is around 380mA - sufficient to drain the battery in three hours.


> I considered building a system for Android that works the same way  
> that
> push mail on the iPhone works. When an email comes in, an SMS would  
> be
> sent to the phone. That SMS would be hidden from the user, but would
> advise the phone to wake up and poll for new email. The only  
> problem is,
> it costs money to send SMS's.

That will cost considerably more power, since it requires  
establishment of the 3G association, which will cost much more power  
(and time) than simply raising the state.

I'd also note that Apple/Yahoo's push notification system still, as  
far as I'm aware, contains a truly pathetic security hole.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 4 Oct 2010, John Snow wrote:
> I'm using the K9 client on my Sprint EVO.  AFAIK, K9 is the only android
> client that supports IDLE.  I have 3 accounts on my phone.  All have a
> constant IDLE connection.  I get easily 12-14 hours of battery life.  I
> plug it in when I get home, so that's plenty.

Of course, SPRINT would be using EV-DO or WiMax, not UMTS.

> (off topic)  Using the GPS is what kills my battery.  With constant GPS
> usage, I can kill a battery in 3 hours.

I wonder if it's the GPS usage or the map updating?

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On 4 Oct 2010, at 18:56, Mark Crispin wrote:
> The real problem with IDLE is that it is NAT-unfriendly.  Most NAT boxes
> drop the mapping after a period of client inactivity, and send a TCP RST
> to the server if the server resumes activity before the client.  Most
> users don't see this, since if the client resumes activity first the same
> mapping is created and no RST is sent.

Yes, indeed, this argument is the most convincing I've actually seen against IDLE, and the compromise is to set a time interval that gets around the NAT's braindead behaviour.  Since most clients are happy to allow users to specify a polling frequency that determines the maximum time between checks, it's not a stretch to turn this into a between DONE-IDLE timer.  The solution is IPv6, of course.

Cheers,
Sabahattin

On 04/10/2010 21:03, Dave Cridland wrote:

>> This makes complete sense. In order to use IMAP Idle on a phone, it
>> would have to keep a TCP connection open, and therefore a 3G connection
>> open. There's a reason why phone makers advertise separate stand by, and
>> call times for battery usage. If IMAP idle were being used, the phone
>> would never enter stand by mode, and would eat the battery within a few
>> hours.
>>
> That's somewhat ill-informed, I'm afraid. I did quite a bit of research
> into this for XMPP, and I captrued most of the findings in XEP-0286.
>
> The "3G session" does need to stay open, but it can stay in Idle, or
> PCH, modes. These cost in the region of 8mA.
>
> Small notifications - including the EXISTS and FETCH FLAGS that IDLE
> typically emits from the server - will only raise the 3G session to FACH
> mode. If the handset is forced into FACH mode constantly, this'll drain
> the battery in around 7 hours, using around 140mA.
> 
> Only if the data size reaches a certain (small) threshold - about 128
> octets typically - will the radio rise to DCH mode, where the drain is
> around 380mA - sufficient to drain the battery in three hours.

Ok, I stand corrected. Thanks for the detailed response. I tried to use
IDLE with K-9 on my Android phone a while ago and it seemed to drain the
battery quite quickly so I made some wrong assumptions. Android runs on
a separate CPU to the rest of the phone, and that CPU goes to sleep
unless an application is using it. I guess when in IDLE mode, K-9 needs
to keep a thread running waiting for incoming data, and therefore needs
to keep a wake lock on the CPU. I suspect that might be what is draining
the battery more quickly.

I do some Android development myself, and I can't see how you could
develop an app within the constraints of the Android sandbox which keeps
a tcp connection alive without keeping a wake lock on the cpu and
draining it's battery much more quickly.

Either way, it seems to last a whole lot longer if I just set it to poll
for new mail every 10 minutes and I'm fine with that.

-- 
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog -              https://secure.grepular.com/
Follow me on Twitter -           http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

Mark Crispin wrote:
> On Mon, 4 Oct 2010, John Snow wrote:
>> I'm using the K9 client on my Sprint EVO.  AFAIK, K9 is the only android
>> client that supports IDLE.  I have 3 accounts on my phone.  All have a
>> constant IDLE connection.  I get easily 12-14 hours of battery life.  I
>> plug it in when I get home, so that's plenty.
>
> Of course, SPRINT would be using EV-DO or WiMax, not UMTS.
In my case, it's all EV-DO.  I'm not in range of any WiMax towers.
>
>> (off topic)  Using the GPS is what kills my battery.  With constant GPS
>> usage, I can kill a battery in 3 hours.
>
> I wonder if it's the GPS usage or the map updating?
I'm pretty sure it's the GPS chip.  The software was using map data stored
on the phone.  I guess it could just be processing of map data.  I'll 
have to
run a test of on phone vs on the net map data.
>
> -- Mark --
>
> http://panda.com/mrc
> Democracy is two wolves and a sheep deciding what to eat for lunch.
> Liberty is a well-armed sheep contesting the vote.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/310a5d84/attachment.html>

On Mon Oct  4 19:32:53 2010, Sabahattin Gucukoglu wrote:
> On 4 Oct 2010, at 18:56, Mark Crispin wrote:
> > The real problem with IDLE is that it is NAT-unfriendly.  Most  
> NAT boxes
> > drop the mapping after a period of client inactivity, and send a  
> TCP RST
> > to the server if the server resumes activity before the client.   
> Most
> > users don't see this, since if the client resumes activity first  
> the same
> > mapping is created and no RST is sent.
> 
> Yes, indeed, this argument is the most convincing I've actually  
> seen against IDLE, and the compromise is to set a time interval  
> that gets around the NAT's braindead behaviour.  Since most clients  
> are happy to allow users to specify a polling frequency that  
> determines the maximum time between checks, it's not a stretch to  
> turn this into a between DONE-IDLE timer.  The solution is IPv6, of  
> course.

Mobile operators are catching up in this respect. I'm told than 5-10  
minutes is fine for keepalives, now. In fact, the pressure from  
"real" mobile developers on application protocol developers has been  
in reducing keepalives, and placing them under the control of clients  
(so they can be synchronized to reduce power requirements).

As more push protocols are developed - and with WebSockets on the  
horizon, that's a given - then it's likely that users will discover  
that applications give them longer battery life on networks with a  
longer NAT timeout, and this may turn into a marketable feature.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 4 Oct 2010, John Snow wrote:
>> Of course, SPRINT would be using EV-DO or WiMax, not UMTS.
> In my case, it's all EV-DO.  I'm not in range of any WiMax towers.

I'm on Verizon, so EV-DO until LTE arrives.

> I'm pretty sure it's the GPS chip.  The software was using map data
> stored on the phone.  I guess it could just be processing of map data.
> I'll have to run a test of on phone vs on the net map data.

Interesting, since a Garmin handheld seems to contrive to work for much
longer.

However, it could be simply a matter of many devices these days being a
Jack of all trades and master of none.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, 4 Oct 2010, Dave Cridland wrote:
> Mobile operators are catching up in this respect. I'm told than 5-10
> minutes is fine for keepalives, now.

Oh, I agree.  But the issue isn't what the mobile operators do with NAT.
It's that you must "look out" for users that run an IMAP client on desktop
and laptop computers behind a cheap Linksys NAT box.

There is no reason for any desktop client to use IDLE, and negligible
reason for a laptop client to do so.  Typically, when I tether a laptop to
a mobile phone, I feed the phone power over USB and keep the data
connection up (via a ping if necessary).  I don't use a laptop tethered
mobile phone at all like I use an application on a mobile phone.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon Oct  4 21:52:14 2010, Mark Crispin wrote:
> On Mon, 4 Oct 2010, Dave Cridland wrote:
>> Mobile operators are catching up in this respect. I'm told than  
>> 5-10
>> minutes is fine for keepalives, now.
> 
> Oh, I agree.  But the issue isn't what the mobile operators do with  
> NAT.
> It's that you must "look out" for users that run an IMAP client on  
> desktop
> and laptop computers behind a cheap Linksys NAT box.
> 
> There is no reason for any desktop client to use IDLE, and  
> negligible
> reason for a laptop client to do so.  Typically, when I tether a  
> laptop to
> a mobile phone, I feed the phone power over USB and keep the data
> connection up (via a ping if necessary).  I don't use a laptop  
> tethered
> mobile phone at all like I use an application on a mobile phone.

Yes, except that the near constant DCH-level, plus the additional  
cost of actually transmitting over that, could quite easily cost more  
than the 500mA USB will feed it. :-)

Even laptops can benefit from reasonably power-efficient coding on  
WiFi, and as long as either clients reconnect sensibly, and/or use  
relatively short timeouts, it should be no problem. FWIW, I've had no  
problem with IDLE and no keepalive on all sorts of networks, but I'll  
be humble and bow to your experience if you insist that must be down  
to my excellent client progamming skills.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

Mark Crispin wrote:
>
> There is no reason for any desktop client to use IDLE, and negligible
> reason for a laptop client to do so. 
> -- Mark --
>

I'm surprised to hear you say that.  IDLE gives me instant notification of
new mail. Anything else relies on the client to poll periodically.  That's
definitely less than instant.   Is there a better way? 

snow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/d43cf8d5/attachment.html>

On 4.10.2010, at 21.52, Mark Crispin wrote:

> There is no reason for any desktop client to use IDLE, and negligible
> reason for a laptop client to do so.

Of course, people mostly use the same software in laptops and in desktops. And desktops are getting more and more power efficient all the time too. Which I'd think is good, with the world using too much power and all.

But yeah, having a server wake up an IDLEing client every 2 minutes isn't much better for power efficiency than sending a NOOP every 2 minutes. I've been once in a while thinking about making the "still here" notifications more dynamic (per-user/network) by automatically tracking how long a connection could stay up without traffic, but that's annoyingly much work too.

On Mon, 4 Oct 2010, Dave Cridland wrote:
> Yes, except that the near constant DCH-level, plus the additional
> cost of actually transmitting over that, could quite easily cost more
> than the 500mA USB will feed it. :-)

Hmm.  When I tether, the phone actually seems to charge albeit rather more
slowly than being idle.  On the other hand, I tether with EV-DO much more
than with UMTS, so maybe there is a different there?

> FWIW, I've had no
> problem with IDLE and no keepalive on all sorts of networks, but I'll
> be humble and bow to your experience if you insist that must be down
> to my excellent client progamming skills.

It's basically the Client That Shall Not Be Named that you must look out
for, crappy Linksys NAT boxes, and users who complain that the "server
isn't updating my email" and can't be bothered to click on that little
broken connection icon to reestablish it.  The latter tend to be admin
types who like to order that something be "fixed" to their liking
regardless of the consequences.

The crappy Linksys boxes were guilty for dropping the NAT mapping if the
session was idle too much.  This was ameliorated by having the server send
an untagged OK every 2 minutes.

The Client That Shall Not Be Named was guilty of a lot more.  First, it
used IDLE even though it had no particular reason to do so.  Second, it
was smart enough to notice that the connection got dropped, and update the
icon, but somehow it never occurred to its developers to reestablish the
connection automatically.  Last, and not least, it would not issue the
DONE after 29 minutes as described in the IDLE RFC, so even if the session
was still alive it would die in the 30th minute.

That last problem required a truly horrible and utterly disgusting kludge.
At the 28th minute, a fake untagged EXISTS would be sent announcing new
mail.  The Client That Shall Not Be Named would issue a DONE.  Then, at
once, the server would issue an untagged EXPUNGE for that fake message,
and The Client That Shall Not Be Named never noticed that nothing changed
in the UID regime as a result of that message being "added" then
"expunged"!

Nor, for that matter, did any other client.  The users of The Client That
Shall Not Be Named stopped complaining, so I called it good.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, 4 Oct 2010, Dave Cridland wrote:
>
> Yes, except that the near constant DCH-level, plus the additional cost of
> actually transmitting over that, could quite easily cost more than the 500mA
> USB will feed it. :-)

That has certainly been my experience tethering a laptop to a 3G GSM
handset in the UK. (Apple MacBook, Nokia 6500c, Orange)

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
HUMBER THAMES DOVER WIGHT PORTLAND: NORTH BACKING WEST OR NORTHWEST, 5 TO 7,
DECREASING 4 OR 5, OCCASIONALLY 6 LATER IN HUMBER AND THAMES. MODERATE OR
ROUGH. RAIN THEN FAIR. GOOD.

On Mon, 4 Oct 2010, John Snow wrote:
>> There is no reason for any desktop client to use IDLE, and negligible
>> reason for a laptop client to do so.
> I'm surprised to hear you say that.  IDLE gives me instant notification of
> new mail. Anything else relies on the client to poll periodically.  That's
> definitely less than instant.   Is there a better way?

That assumes that the IMAP server notifies instantly, instead of polling
internally at a fixed interval.  This, in turn, assumes some form of
message passing between the MDA and the IMAP server.

I find it remarkable that anyone can get bent out of shape because it may
take as much as a minute or two to be notified about a newly arrived
email; particularly as it may have taken some ungodly amount of time for
the message to have travelled from the senders MSA and the recipient's
MDA.

We have a perfectly good facility called instant messaging for real-time
communication.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On 4.10.2010, at 22.20, Mark Crispin wrote:

> Last, and not least, it would not issue the
> DONE after 29 minutes as described in the IDLE RFC, so even if the session
> was still alive it would die in the 30th minute.

If only it were the only one. It seems that just about all the widely used clients either do that or used to do it at some point. So nowadays my server never disconnects clients on IDLE because they haven't sent anything. That fixes the whole thing. I still send "* OK Still here" events every 2 minutes, so connections that are actually gone will get disconnected pretty quickly.

> That last problem required a truly horrible and utterly disgusting kludge.
> At the 28th minute, a fake untagged EXISTS would be sent announcing new
> mail.  The Client That Shall Not Be Named would issue a DONE.  Then, at
> once, the server would issue an untagged EXPUNGE for that fake message,
> and The Client That Shall Not Be Named never noticed that nothing changed
> in the UID regime as a result of that message being "added" then
> "expunged"!

This doesn't help clients that use one IDLEing connection that does nothing but IDLE, and use another connection for actually fetching the new mails. Although maybe that thing died already that had the problem, but I'm not entirely sure. Anyway, my above solution seems easiest for current and future clients..

Mark Crispin wrote:
> On Mon, 4 Oct 2010, John Snow wrote:
>>> There is no reason for any desktop client to use IDLE, and negligible
>>> reason for a laptop client to do so.
>> I'm surprised to hear you say that.  IDLE gives me instant 
>> notification of
>> new mail. Anything else relies on the client to poll periodically.  
>> That's
>> definitely less than instant.   Is there a better way?
>
> That assumes that the IMAP server notifies instantly, instead of polling
> internally at a fixed interval.  This, in turn, assumes some form of
> message passing between the MDA and the IMAP server.
That's true.  I took that as am implied requirement for implementing IDLE.
>
> I find it remarkable that anyone can get bent out of shape because it may
> take as much as a minute or two to be notified about a newly arrived
> email; particularly as it may have taken some ungodly amount of time for
> the message to have travelled from the senders MSA and the recipient's
> MDA.
>
We live in a world full of impatient people.

> We have a perfectly good facility called instant messaging for real-time
> communication.
>
Or a phone, where this discussion started.
> -- Mark --
>
> http://panda.com/mrc
> Democracy is two wolves and a sheep deciding what to eat for lunch.
> Liberty is a well-armed sheep contesting the vote.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/ddc27353/attachment.html>

On 10-10-04 5:56 PM, Mark Crispin wrote:
> That assumes that the IMAP server notifies instantly, instead of polling
> internally at a fixed interval.  This, in turn, assumes some form of
> message passing between the MDA and the IMAP server.

That's a reasonable assumption these days. I did a test at work a month
ago: send mail from my laptop to myself:

1) compose in mail.app on my macbook (office lan in victoria)
2) submission with starttls (via orthanc.ca, living on the US east coast)
3) orthanc spamassassins the message, then forwards to the gmail cloud
which hosts our $work mail
4) gmail crunches and does whatever filtering it does, then dumps in my
mailbox
5) gmail imap server notices new mail and breaks out of imap idle to
notify mail.app on my laptop

End-to-end time: < 10s.

As a kid who grew up on dialup UUCP, I continue to be amazed.  But only
two people I share the office with have ever seen a modem.


> I find it remarkable that anyone can get bent out of shape because it may
> take as much as a minute or two to be notified about a newly arrived
> email; particularly as it may have taken some ungodly amount of time for
> the message to have travelled from the senders MSA and the recipient's
> MDA.

Sorry Mark, but reality has overtaken us. BGP is the new pathalias.

That the SMTP/IMAP delivery path can go end-to-end in well under a
minute is a testament to the protocols and the systems that implement
them. If they can manage (say) <60s 99.999% of the time, I think we won.

Mail and IM solve two different problems.  IM in absolutely instant, but
if you're not there, it's gone.  Mail provides nearly-instant
gratification with fallback queueing.  This is all good stuff, even if
we need to beat people over the head sometimes to make themunderstand
the (dis)advantages of each.

> We have a perfectly good facility called instant messaging for real-time
> communication.

Yet we still have letters to compliment personal conversations. Email is
far from dead.

IMAP without IDLE is like read() without blocking.

--lyndon

On Tue, 5 Oct 2010, Timo Sirainen wrote:
> So nowadays
> my server never disconnects clients on IDLE because they haven't sent
> anything. That fixes the whole thing. I still send "* OK Still here"
> events every 2 minutes, so connections that are actually gone will get
> disconnected pretty quickly.

Be careful.  If you don't get a disconnect event, you'll presently end up
filling the TCP window and then block.

> This doesn't help clients that use one IDLEing connection that does
> nothing but IDLE, and use another connection for actually fetching the
> new mails.

Clients actually do this?  Deep sigh.  Doesn't anyone have a clue as to
how to write software any more?  Or "if it ain't in the Java cookbook, it
can't be done"???

http://www.generationaldynamics.com/cgi-bin/D.PL?d=ww2010.i.java080701

:(

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, 4 Oct 2010, John Snow wrote:
>> That assumes that the IMAP server notifies instantly, instead of polling
>> internally at a fixed interval.  This, in turn, assumes some form of
>> message passing between the MDA and the IMAP server.
> That's true.  I took that as am implied requirement for implementing IDLE.

There's no enforcement of such a requirement.  If IDLE is simply a
checklist requirement (which it usually is) then the simplest way to get
it checked is via an internal poll.

After all, who's gonna know?

> We live in a world full of impatient people.

How will they know?  This is a day and age when the vast majority of users
don't even have access to Received headers, much less can interpret them.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Tue Oct  5 05:46:54 2010, Lyndon Nerenberg wrote:
> Mail and IM solve two different problems.  IM in absolutely  
> instant, but
> if you're not there, it's gone.

See XEP-0160 - these days, people expect simple messages to persist  
and be fed to them when they come back online.

Incidentally, XMPP on smartphones - well, on real smartphones and not  
those which place shiny over substance - does mostly the same thing  
as IMAP IDLE - it maintains a long-lived TCP session which both ends  
can use to push data over. On shiny but castrated smartphones,  
though, a developer must buy in (technically at least, financially I  
don't know) into a proprietary unified push system. Which, rumour has  
it, runs over XMPP, using a persistent TCP connection...

>   Mail provides nearly-instant
> gratification with fallback queueing.  This is all good stuff, even  
> if
> we need to beat people over the head sometimes to make  
> themunderstand
> the (dis)advantages of each.
> 
> 
It also handles Big Things, and - when IMAP's in play at least -  
offers excellent server-side archival under the user's control.

It's not mere traditionalism that we're having this conversation over  
email and not in an IM chatroom, after all - even the XSF uses  
mailing lists.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On 5.10.2010, at 2.08, Mark Crispin wrote:

> On Tue, 5 Oct 2010, Timo Sirainen wrote:
>> So nowadays
>> my server never disconnects clients on IDLE because they haven't sent
>> anything. That fixes the whole thing. I still send "* OK Still here"
>> events every 2 minutes, so connections that are actually gone will get
>> disconnected pretty quickly.
> 
> Be careful.  If you don't get a disconnect event, you'll presently end up
> filling the TCP window and then block.

Luckily I don't do any blocking reads or writes.

>> This doesn't help clients that use one IDLEing connection that does
>> nothing but IDLE, and use another connection for actually fetching the
>> new mails.
> 
> Clients actually do this?  Deep sigh.  Doesn't anyone have a clue as to
> how to write software any more?  Or "if it ain't in the Java cookbook, it
> can't be done"???

I remember when I first heard of this it was a plugin (or something) that simply woke up Mail.app, because it didn't support IDLE itself. Nowadays Mail.app supports IDLE, so this plugin is no longer used. But I can't remember if that was the only problematic case.

On Tue Oct  5 02:08:06 2010, Mark Crispin wrote:
> On Tue, 5 Oct 2010, Timo Sirainen wrote:
>> So nowadays
>> my server never disconnects clients on IDLE because they haven't  
>> sent
>> anything. That fixes the whole thing. I still send "* OK Still  
>> here"
>> events every 2 minutes, so connections that are actually gone will  
>> get
>> disconnected pretty quickly.
> 
> Be careful.  If you don't get a disconnect event, you'll presently  
> end up
> filling the TCP window and then block.
> 
> 
Also, you erode power saving efficiencies on mobiles - those states  
are for sending and receiving data.

I'm afraid broken NAT middleboxes are a lost cause, some of them are  
truly awful. Nokia did a study (still ongoing) about how bad some of  
them are, because they're at the point where people are  
special-casing and working-around LAN and Broadband much more than  
Mobile.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On 5.10.2010, at 2.18, Mark Crispin wrote:

> There's no enforcement of such a requirement.  If IDLE is simply a
> checklist requirement (which it usually is) then the simplest way to get
> it checked is via an internal poll.
> 
> After all, who's gonna know?

I often force a refresh by marking some existing message unseen and then back again to seen while waiting for a message that I know is going to arrive, clicking this key combination often enough until it I see the mail. So it is possible to know, although that assumes the server notices new messages during flag changes.

(I realized only during this thread that this is actually a server problem while using Mail.app, because I'm using Solaris that doesn't have Linux-like inotify. I guess I should either increase my polling timeout to 1 second or implement some IPC of my own so at least I'd get rid of this habit half the time I read mails.)

Mark Crispin wrote:
> On Mon, 4 Oct 2010, John Snow wrote:
>>> That assumes that the IMAP server notifies instantly, instead of 
>>> polling
>>> internally at a fixed interval.  This, in turn, assumes some form of
>>> message passing between the MDA and the IMAP server.
>> That's true.  I took that as am implied requirement for implementing 
>> IDLE.
>
> There's no enforcement of such a requirement.  If IDLE is simply a
> checklist requirement (which it usually is) then the simplest way to get
> it checked is via an internal poll.
>
> After all, who's gonna know?
>
>> We live in a world full of impatient people.
>
> How will they know?  This is a day and age when the vast majority of 
> users
> don't even have access to Received headers, much less can interpret them.
>
They know via the phone.  These impatient people call to see if the 
recipient
has read the mail that they just sent. They expect the mail to arrive as 
soon as it's
sent.
> -- Mark --
>
> http://panda.com/mrc
> Democracy is two wolves and a sheep deciding what to eat for lunch.
> Liberty is a well-armed sheep contesting the vote.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20101004/21a546bc/attachment.html>

On Tue, 5 Oct 2010, Dave Cridland wrote:
>
> On shiny but castrated smartphones, though, a developer must buy in
> (technically at least, financially I don't know) into a proprietary
> unified push system. Which, rumour has it, runs over XMPP, using a
> persistent TCP connection...

It certainly uses the standard XMPP port 5223:
http://support.apple.com/kb/HT3576

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
HUMBER THAMES DOVER WIGHT PORTLAND: NORTH BACKING WEST OR NORTHWEST, 5 TO 7,
DECREASING 4 OR 5, OCCASIONALLY 6 LATER IN HUMBER AND THAMES. MODERATE OR
ROUGH. RAIN THEN FAIR. GOOD.

On Tue, 5 Oct 2010, Timo Sirainen wrote:
> I often force a refresh by marking some existing message unseen and then
> back again to seen while waiting for a message that I know is going to
> arrive, clicking this key combination often enough until it I see the
> mail. So it is possible to know, although that assumes the server
> notices new messages during flag changes.

Alpine's next message command while at the end of the mailbox does an IMAP
NOOP command which ought to refresh without the side effects of message
flagging.

A server is required to notice new messages (and flag changes) in the
selected mailbox as part of any IMAP command.  NOOP is not the command to
notice new messages; NOOP is the command to do nothing other than notice
new messages and flag changes.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, 4 Oct 2010, John Snow wrote:
> They know via the phone.  These impatient people call to see if the
> recipient has read the mail that they just sent. They expect the mail to
> arrive as soon as it's sent.

So they touch their email client, and at once it shows the message.

Email is not instantaneous.  As more policy is imposed, it will become
even less so; including such things as email delivery being withheld
pending review by compliance.

Not even BlackBerry is instantaneous any more.

Live with it.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Tue Oct  5 12:51:24 2010, Tony Finch wrote:
> On Tue, 5 Oct 2010, Dave Cridland wrote:
> >
> > On shiny but castrated smartphones, though, a developer must buy  
> in
> > (technically at least, financially I don't know) into a  
> proprietary
> > unified push system. Which, rumour has it, runs over XMPP, using a
> > persistent TCP connection...
> 
> It certainly uses the standard XMPP port 5223:
> http://support.apple.com/kb/HT3576

That's a special definition of "standard" as "formally deprecated in  
2003", of course. (See XEP-0035).

This cutting-edge technology of Apple's is marvellous, isn't it?

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade