mailing list archives

Are there any limits which might prevent a client from sending a single 1GB MULTIAPPEND request (for example, 100,000 10K messages to be added to the same quotaless mailbox), especially if LITERAL+ is available?  With the "atomic transaction" requirement, this could potentially be a headache.

In practice, do real-world IMAP clients attempt to break such large potential MULTIAPPENDs into smaller batches, or do they rely on the server to process the APPENDed messages as they go?

Is it acceptable for a server to queue the APPENDed messages and process them once the end of the MULTIAPPEND is reached?

Dan Karp wrote:

>Is it acceptable for a server to queue the APPENDed messages and process them once the end of the MULTIAPPEND is reached?
>  
>
Yes. Why not?

On Tue, 27 Mar 2007, Dan Karp wrote:
> Are there any limits which might prevent a client from sending a single 
> 1GB MULTIAPPEND request (for example, 100,000 10K messages to be added 
> to the same quotaless mailbox), especially if LITERAL+ is available? 
> With the "atomic transaction" requirement, this could potentially be a 
> headache.

There are no limits in the protoocol.

Servers are free to impose any limit they want:
 	C: tag APPEND blurdybloop {2345765221}
 	S: tag NO Our policy prohibits messages that large

 	C: tag APPEND blurdybloop {10000}
 	S: + Ready for message
 	C: [10,000 bytes of message...] {10000}
 	S: + Ready for another message
 	C: [10,000 bytes of message...] {10000}
 	S: tag NO Disk quota exceeded

Note that in both cases, nothing is appended.

I don't think that a client should use LITERAL+ with MULTIAPPEND.  Both 
LITERAL+ and MULTIAPPEND are optimizations to reduce RTTs, but MULTIAPPEND 
is better for the APPEND case.  Although it sounds attractive to eliminate 
all per-message RTTs by using both simultaneously (and IMAP specifically 
does NOT preclude doing so), any failure will cause both client and server 
to go to a lot of work for nothing.

Put another way, the "headache" you note is real, but only if the client 
puts itself in that position.  IMHO, one RTT/message is OK since that RTT 
actually provides a benefit.  The problem with base specification APPEND 
is that there were two RTTs/message plus a mailbox open/lock operation, 
and no way to undo an extended operation if there is a failure.

> In practice, do real-world IMAP clients attempt to break such large 
> potential MULTIAPPENDs into smaller batches, or do they rely on the 
> server to process the APPENDed messages as they go?

Neither, I hope.

In general, it is quite unlikely that any client is going to APPEND 
100,000 10K messages other than some process for migrating mailboxes 
(which is a different issue from client operation).  I don't think that a 
client should attempt such a thing without external knowledge that the 
server will accept it.

Breaking up an APPEND like that is acceptable only if you do not want 
atomicity; that is, you *want* a partial APPEND on failure *AND* perhaps 
other intervening APPENDs happening (such as new mail delivery).

The problem is, a lot of kidlets aren't likely to understand the issue, 
and specifically why atomicity is a feature and thus should not be 
sabotaged.  There are a lot of kidlets out there who think that they are 
programmers; and alas a lot of companies are hiring them.  I just wasted 
hours yesterday because McAfee hired an obvious kidlet...  ;-(

> Is it acceptable for a server to queue the APPENDed messages and process 
> them once the end of the MULTIAPPEND is reached?

Yes.  That's how most servers work.  Remember that the final OK determines 
success/failure for the entire MULTIAPPEND.

> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> https://mailman1.u.washington.edu/mailman/listinfo/imap-protocol
>

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Tue Mar 27 18:57:02 2007, Mark Crispin wrote:
> I don't think that a client should use LITERAL+ with MULTIAPPEND.  
> Both LITERAL+ and MULTIAPPEND are optimizations to reduce RTTs, but 
> MULTIAPPEND is better for the APPEND case.  Although it sounds 
> attractive to eliminate all per-message RTTs by using both 
> simultaneously (and IMAP specifically does NOT preclude doing so), 
> any failure will cause both client and server to go to a lot of 
> work for nothing.

Just to pick up on that point, Polymer assumes (rightly in all the 
few cases I tried) that the server will refuse an APPEND as early as 
possible. I don't think there's anything encouraging this in RFC3501 
or RFC3502 (although maybe there is), but I think it's good behaviour 
as well as common.

Polymer chooses to exploit this by not using LITERAL+ only for the 
first literal in any command which exceeds a certain size, to allow 
the server to reject it. So typical small text-only messages will end 
up using non-synchronizing literals, whereas the first large 
attachment will use a synchronizing literal, so that Polymer knows 
earlier when to send it. This seems to work better than avoiding 
LITERAL+ with APPEND entirely, although there's no doubt it could do 
with some refinement - in particular, several thousand small messages 
will all be sent using non-synchronizing literals, which is an 
obvious problem.

It's also worth noting that the same applies to APPENDs whether 
MULTIAPPEND or CATENATE is used or not - Polymer's doing this for all 
commands, it's merely that unless it encounters seriously 
impressively sized mailbox names, it's unlikely to occur elsewhere.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade