mailing list archives

I'm currently implementing an IMAP4rev1 server and am a little stuck on
section 5.5 of rfc3501:

http://tools.ietf.org/html/rfc3501#section-5.5

I understand that the client can send additional commands before
receiving the command completion result, and that the server can process
and respond to those commands in any order (subject to ambiguity
considerations)

In reality, do any servers actually do that though, or do they all tend
to run the commands synchronously and in order? Also, do any clients
take advantage of this ability or do they tend to use multiple
connections instead if they want to run multiple commands asynchronously?

I'm trying to gauge whether or not it would be worth the extra work and
complexity to make the server run multiple commands at the same time on
a single connection. Also, are there any clients that are known to break
when receiving server responses "out of order" ?

Also, is there an exhaustive set of ambiguity rules listed anywhere
which need to be looked out for when executing multiple commands at the
same time? The RFC suggests there are lots of cases, then provides only
a few examples. I find it confusing how the examples are written as
well. Eg, the following is apparently "invalid" :

FETCH + NOOP + STORE

What does that mean in human readable terms? If the examples consisted
of two commands, then I would assume that meant command 2 has to wait
for command 1 to finish before starting, but most of the examples
consist of 3 or more commands..?

-- 
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog -              https://secure.grepular.com/
Follow me on Twitter -           http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

Few, if any, IMAP servers will execute commands out of order.

Nor has there ever been much of a use case for executing commands out of
order.  The claimed use case ~25 years ago was that SEARCH may take a
while because it may require the operator to mount a 9-track archive tape
to get the data on large archives.

That claim was demonstrable nonsense even back then.  But, even back then,
the silly "IMAP will die unless it has this feature I want" argument (it's
like calling a political opponent "bigot" or "mean-spirited") was babbled
just as much as it is today.

The valid use case for multiple commands in progress was, and remains,
command pipelining in order to save RTTs.  Pipelining is particularly
important in SMTP, but there are some cases where IMAP clients can benefit
from it.  In pipelining, commands are executed in the order in the
pipeline.

There is no particular requirement for an IMAP server to do anything to
support pipelining other than it MUST NOT clear its command input buffer
between commands.  [SMTP, on the other hand, has a PIPELINING extension;
and a server which does not have that extension MAY do such clearing.
Some SMTP servers do that as an anti-spam measure, as most spamware SMTP
blasters assume a pipelining server and do not check the extension.]

An IMAP client, on the other hand, MUST follow certain rules if it decides
to pipeline (many clients do not, and thus need not worry about these
rules).  These rules boil down to: "don't do anything whose operation may
differ depending upon the operation of a previous command."

For example, the "FETCH + NOOP + STORE" example that you mentioned is
non-pipelineable because an EXPUNGE event can happen during the NOOP, thus
altering the interpretation of the message sequence in the STORE.

Most pipelining rules involve the possibility of an EXPUNGE event.  A
command that allows an EXPUNGE event must be the last command in any
pipeline that involves message sequences in any command.

Even if a server implemented execute pipelined commands out of order (a
possibility that IMHO was stillborn 20+ years ago), it must nonetheless
result "as if" they were executed in order.  This quickly makes the entire
exercise pointless once you work out the few pipelined cases that can be
done out of order with "as if" results and see that there is no benefit to
doing those cases out of order.

Similarly, command tagging should properly be seen as a way of ensuring
synchronization in pipelined response processing (as opposed to just
counting responses ala SMTP) and NOT as a way to execute commands out of
order.

Although some clients open multiple sessions to the same mailbox on the
same server to do multiple commands at once, there is no benefit to this
practice.  If something causes an IMAP command to bottleneck, that same
something will affect a simultaneous IMAP session from that client to that
same server+mailbox.

Bottom line:

As a server implementor, you can disregard section 5.5 entirely, with the
exception that you MUST NOT clear the command input buffer after reading a
command (because there may be a pipelined command following the current
command).  Section 5.5 should be considered to be solely implementation
advice for clients that wish to pipeline.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On 29/08/2010 19:31, Mark Crispin wrote:

> Bottom line:
> 
> As a server implementor, you can disregard section 5.5 entirely, with the
> exception that you MUST NOT clear the command input buffer after reading a
> command (because there may be a pipelined command following the current
> command).  Section 5.5 should be considered to be solely implementation
> advice for clients that wish to pipeline.

Excellent. You just gave me the exact answer I was hoping for. Thanks.

-- 
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog -              https://secure.grepular.com/
Follow me on Twitter -           http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

On Sun Aug 29 19:31:58 2010, Mark Crispin wrote:
> Few, if any, IMAP servers will execute commands out of order.
> 
> 
I don't know of any for certain, but if anyone's implementation does,  
it'll be Arnt/AMS's.

> As a server implementor, you can disregard section 5.5 entirely,  
> with the
> exception that you MUST NOT clear the command input buffer after  
> reading a
> command (because there may be a pipelined command following the  
> current
> command).  Section 5.5 should be considered to be solely  
> implementation
> advice for clients that wish to pipeline.

FWIW, my client can cope with out-of-order execution. My ACAP server  
does ACAP commands out of order (where the protocol support is  
simpler).

The only case where it would be useful is SEARCH - searches can take  
time on larger mailboxes, and being able to execute a simple FETCH of  
a bodypart during the search would be beneficial. To allow for this,  
Polymer assumes that searches as a whole are executed concurrently,  
in the absence of ESEARCH - ESEARCH includes a tag in the results,  
effectively mimicking ACAP's tagged intermediate responses. Polymer  
uses SEARCH quite extensively, especially if filtered views are in  
heavy use, and one of the reasons I never put in multiple views of a  
single mailbox is that the SEARCHes in one view would simply stall  
the other view.

(I imagine this also applies to THREAD and SORT).

As Mark says, in most implementations the bottleneck would be such  
that servers are unlikely to be able to do much, but I suspect that  
there are some cases where it would be possible, in some servers.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On 30/08/2010 09:22, Dave Cridland wrote:

> FWIW, my client can cope with out-of-order execution. My ACAP server
> does ACAP commands out of order (where the protocol support is simpler).

I'd never heared of ACAP until you mentioned it in this email; even
though I've been running a CommuniGate Pro server for 4 years which
supports it... JOOI, what client implementations are there for ACAP?
Wondering if it's worth adding to my desirable feature list...

-- 
Mike Cardwell - Perl/Java/Web developer, Linux admin, Email admin
Read my tech Blog -              https://secure.grepular.com/
Follow me on Twitter -           http://twitter.com/mickeyc
Hire me - http://cardwellit.com/ http://uk.linkedin.com/in/mikecardwell

On Mon, 2010-08-30 at 09:22 +0100, Dave Cridland wrote:
> On Sun Aug 29 19:31:58 2010, Mark Crispin wrote:
> > Few, if any, IMAP servers will execute commands out of order.
> > 
> I don't know of any for certain, but if anyone's implementation does,  
> it'll be Arnt/AMS's.

If a command starts blocking on client output (like a large enough
FETCH) then Dovecot starts running the next command. There are a few
advantages, although possibly small enough that they don't matter:

1. Less latency, because any disk I/O is started while the previous
output is still being sent to client.

2. Kernel might be able to reorder disk I/O better since it knows more
what data is going to be read.

3. Maybe the client wanted to do a couple of STATUSes in the middle of a
large fetch, and could update the UI faster while FETCH is still going
on..

Of course, 1-2 can also be implemented by starting the commands but
delaying/buffering client output, and clients can do 3 with another IMAP
connection.

On Tue, Aug 31, 2010 at 1:16 AM, Mike Cardwell
<imap-protocol@lists.grepular.com> wrote:
> supports it... JOOI, what client implementations are there for ACAP?

http://dave.cridland.net/acap/polymer.html

though I've never used it.
-- 
http://gr.linkedin.com/in/yiorgos

On Tue Aug 31 16:16:12 2010, Timo Sirainen wrote:
> On Mon, 2010-08-30 at 09:22 +0100, Dave Cridland wrote:
> > On Sun Aug 29 19:31:58 2010, Mark Crispin wrote:
> > > Few, if any, IMAP servers will execute commands out of order.
> > >
> > I don't know of any for certain, but if anyone's implementation  
> does,
> > it'll be Arnt/AMS's.
> 
> If a command starts blocking on client output (like a large enough
> FETCH) then Dovecot starts running the next command. There are a few
> advantages, although possibly small enough that they don't matter:
> 
> 1. Less latency, because any disk I/O is started while the previous
> output is still being sent to client.
> 
> 2. Kernel might be able to reorder disk I/O better since it knows  
> more
> what data is going to be read.
> 
> 3. Maybe the client wanted to do a couple of STATUSes in the middle  
> of a
> large fetch, and could update the UI faster while FETCH is still  
> going
> on..
> 
> Of course, 1-2 can also be implemented by starting the commands but
> delaying/buffering client output, and clients can do 3 with another  
> IMAP
> connection.

Isode M-Box delays writing over the network until either:

a) The buffer is "big".

b) There are no commands left to process.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon Aug 30 23:20:01 2010, Yiorgos Adamopoulos wrote:
> On Tue, Aug 31, 2010 at 1:16 AM, Mike Cardwell
> <imap-protocol@lists.grepular.com> wrote:
> > supports it... JOOI, what client implementations are there for  
> ACAP?
> 
> http://dave.cridland.net/acap/polymer.html

Polymer's an aquired taste... :-)

Mulberry supports it, kind of, as does Wanderlust (an Emacs thing),  
and Eudora. None of them support the standardized datasets proposed  
in I-D forms for configuration, though, as I recall.

Speaking as someone who's implemented both ends of it, and still uses  
it in production, I have to say I wouldn't bother now. There are  
several features of ACAP I do think are highly beneficial, including  
the ability to push configuration (etc) updates to aware clients, and  
the promise of interoperability with basic configuration data, but  
there are shortfalls in the protocol's infrastructure which I think  
modern data-sharing protocols address much better, most especially  
the initial discovery and bootstrap.

In particular, I'd be inclined to use XMPP, and more specifically  
PEP/PubSub, in its private data store mode, if I were doing all this  
again. I'm intending tackling this one relatively shortly. The  
benefit there is that you're using a protocol you can bootstrap into  
purely off a jid and a password, say "I care about X Y and Z", and  
have the server not only send you your data, but everyone else's who  
wants to share it with you.

However, should we ever tackle an IMAP5, I would definitely mine ACAP  
for various features of its protocol syntax. It's a very nicely  
designed protocol, and the data model (in respect to searching etc)  
is very neat.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade