mailing list archives

Due to my current back-end architecture, adding a mail message to a
folder can take a considerable amount of time, up to 10 minutes or so,
depending on whether attachment content has to be OCR'ed for the
Lucene index.  It looks to me as if the semantics of APPEND, this may
be unacceptable:

      If the mailbox is currently selected, the normal new message
      actions SHOULD occur.  Specifically, the server SHOULD notify the
      client immediately via an untagged EXISTS response.

Nothing's going to happen "immediately".  Of course, I presume that
"immediately" in this context means, "whenever the server first
notices the new message".  I can hang the connection for N minutes or
so till the result of the attempted message incorporation is known;
the client can always open another connection in the meantime.  Then
I can respond once the APPEND is complete.

Comments?

Bill

The APPEND command should return success to the client as soon as the 
server has accepted and taken responsibility for having received the new 
message(s) (note plural in the case of MULTIAPPEND).

A server that delays a response for a period of minutes IS compliant; 
however, it is likely to provoke the client into declaring the session 
dead, disconnecting, and perhaps retrying the APPEND.  Even if the client 
is not so provoked, a human user of the client would likely take the 
provocation with the same results.  So this would be perceived as a 
lower-quality server implementation.

The text that you quoted:
>      If the mailbox is currently selected, the normal new message
>      actions SHOULD occur.  Specifically, the server SHOULD notify the
>      client immediately via an untagged EXISTS response.
is a quality of implementation issue (note SHOULD instead of MUST).  As 
such, if your implementation can not fufill that requirement, it has the 
option of not doing so (at the peril of being perceived as a lower-quality 
server implementation).

I would suggest a third way.  If it is possible for a mailbox view in IMAP 
to incorporate both the mailbox itself, and the queue of messages accepted 
by APPEND, but pending final incorporation (e.g., the time-consuming OCR 
step), that may allow you to fufill both sets of requirements and be 
perceived as a higher-quality server implementation.

Remember that the server *is* allowed to do things under the sheets; what 
the client sees the server doing is NOT necessarily the same as what the 
server is doing.  IMAP only dictates client perceptions in this case and 
not server data store action; what the server does under the sheets to its 
own data store is its own business, as long as what the client sees 
complies with IMAP.

Put another way, as I am fond of saying, "a difference which makes no 
difference *is* no difference."

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Mon Mar 12 13:19:45 2007, Bill Janssen wrote:
> Due to my current back-end architecture, adding a mail message to a
> folder can take a considerable amount of time, up to 10 minutes or 
> so,
> depending on whether attachment content has to be OCR'ed for the
> Lucene index.  It looks to me as if the semantics of APPEND, this 
> may
> be unacceptable:
> 
>       If the mailbox is currently selected, the normal new message
>       actions SHOULD occur.  Specifically, the server SHOULD notify 
> the
>       client immediately via an untagged EXISTS response.
> 
> Nothing's going to happen "immediately".  Of course, I presume that
> "immediately" in this context means, "whenever the server first
> notices the new message".  I can hang the connection for N minutes 
> or
> so till the result of the attempted message incorporation is known;
> the client can always open another connection in the meantime.  Then
> I can respond once the APPEND is complete.
> 
> Comments?

Yes. :-)

An APPEND should never take more than a couple of seconds. There's no 
point in scanning images at this point, instead, have your system 
append the message instantly, and then scan through the messages 
later to classify them. They're not going to be spam, since they've 
been explicitly added by the user.

For delivery, it's different, but for IMAP, a client finding an 
APPEND hanging for more than a few seconds isn't going to wait 
patiently or open a new connection, it'll most likely hang, or cut 
the connection and make a new one.

FWIW, users expect that a delivery is pretty well instant too, in 
fact, so I think you're doomed to failure here.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@jabber.org
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

I suppose I could stick a crippled substandard version of the message
into the server's data store immediately, report APPEND success
immediately, and have that message suddenly disappear if the full
incorporation later fails, and be replaced by a real message if it
later succeeds.  I'm a bit concerned about client-side caching of
the data.

> An APPEND should never take more than a couple of seconds. There's no 
> point in scanning images at this point, instead, have your system 
> append the message instantly, and then scan through the messages 
> later to classify them. They're not going to be spam, since they've 
> been explicitly added by the user.

Well, by the user's client.  The problem is that this approach makes
those not-yet-scanned messages invisible to the search mechanism, so
that it becomes (to the user) flaky.  "I just put that message in
here!  Where is it!"  Not sure whether I want an apparently slow
server, or an apparently flaky search system.  Frankly, I'd rather
just hang the connection till it's done.  Seems more compliant to me.
Shame there's not some untagged hinting the APPEND command could
generate to estimate a time-to-incorporation.

But if the clients won't tolerate it, they won't tolerate it.

> FWIW, users expect that a delivery is pretty well instant too, in 
> fact, so I think you're doomed to failure here.

I get that a lot :-).

Bill

On Mon, 2007-03-12 at 15:55 -0700, Bill Janssen wrote:

> I suppose I could stick a crippled substandard version of the message
> into the server's data store immediately, report APPEND success
> immediately, and have that message suddenly disappear if the full
> incorporation later fails, and be replaced by a real message if it
> later succeeds.  I'm a bit concerned about client-side caching of
> the data.

If you're going to replace the message, change its UID as well.

> Well, by the user's client.  The problem is that this approach makes
> those not-yet-scanned messages invisible to the search mechanism, so
> that it becomes (to the user) flaky.  "I just put that message in
> here!  Where is it!"  Not sure whether I want an apparently slow
> server, or an apparently flaky search system.  Frankly, I'd rather
> just hang the connection till it's done.  Seems more compliant to me.
> Shame there's not some untagged hinting the APPEND command could
> generate to estimate a time-to-incorporation.

How about hanging SEARCH until the message is indexed? Clients are used
to long running SEARCH, and the user most likely is doing SEARCHes a lot
less than APPENDs (and SEARCHes after APPEND even less often).

If you don't want the UID replacing, you could hang FETCH too I suppose.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/6e1282c4/attachment.sig>

On Tue, 13 Mar 2007, Timo Sirainen wrote:
>> I suppose I could stick a crippled substandard version of the message
>> into the server's data store immediately, report APPEND success
>> immediately, and have that message suddenly disappear if the full
>> incorporation later fails, and be replaced by a real message if it
>> later succeeds.  I'm a bit concerned about client-side caching of
>> the data.
> If you're going to replace the message, change its UID as well.

I second this comment.  Messages are static; they MUST NOT be changed. 
The only want to change a message on the server is to add a new message 
with the change, and delete/expunge the old copy.

This is very imporant and bears emphasis: messages are static; they MUST 
NOT be changed.

> How about hanging SEARCH until the message is indexed? Clients are used
> to long running SEARCH, and the user most likely is doing SEARCHes a lot
> less than APPENDs (and SEARCHes after APPEND even less often).

I certainly am not used to long-running SEARCH.  I expect even a text 
SEARCH to return quickly, or at least as fast as the data can be read from 
the disk.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

> If you're going to replace the message, change its UID as well.

I think I only need to do this if the underlying changes are visible
through the IMAP protocol.  The only real difference is that they
would be returned by BODY or TEXT searches that previously didn't

> How about hanging SEARCH until the message is indexed? Clients are used
> to long running SEARCH, and the user most likely is doing SEARCHes a lot
> less than APPENDs (and SEARCHes after APPEND even less often).

The main reason I'm not using an existing IMAP server is that I can't
stand the long-running searches, so that doesn't appeal to me.

Could I just not do APPEND?  What's it used for, anyway?  Drafts,
copying messages from folders on other mail servers, etc.

Bill

On Mon, 12 Mar 2007, Bill Janssen wrote:
>> If you're going to replace the message, change its UID as well.
> I think I only need to do this if the underlying changes are visible
> through the IMAP protocol.  The only real difference is that they
> would be returned by BODY or TEXT searches that previously didn't

I fail utterly to resolve your first sentence with the second.  Put 
another way, what you describe in your second sentence sure sounds like a 
"change visible through the IMAP protocol".

> The main reason I'm not using an existing IMAP server is that I can't
> stand the long-running searches, so that doesn't appeal to me.

Which server has "long-running searches"?  Sounds like a poorly-configured 
server installation to me.  UW imapd does a text search through 14,000 
messages in about 3-4 seconds, and that's with all the Unicode conversion 
gubbish that has to be done for a good-quality search.

> Could I just not do APPEND?  What's it used for, anyway?  Drafts,
> copying messages from folders on other mail servers, etc.

APPEND is mandatory to implement.  Besides the uses you list, APPEND is 
the only way that an IMAP client can modify a message on the server (e.g., 
to save a copy of the message with attachments removed).

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

> On Mon, 12 Mar 2007, Bill Janssen wrote:
> >> If you're going to replace the message, change its UID as well.
> > I think I only need to do this if the underlying changes are visible
> > through the IMAP protocol.  The only real difference is that they
> > would be returned by BODY or TEXT searches that previously didn't
> 
> I fail utterly to resolve your first sentence with the second.  Put 
> another way, what you describe in your second sentence sure sounds like a 
> "change visible through the IMAP protocol".

I was thinking that the message looked the same -- it remains static,
as far as the protocol can see.  Same content, same UID.  The change
is to the mailbox.

Bill

On Mon, 12 Mar 2007, Bill Janssen wrote:
>>>> If you're going to replace the message, change its UID as well.
>>> I think I only need to do this if the underlying changes are visible
>>> through the IMAP protocol.  The only real difference is that they
>>> would be returned by BODY or TEXT searches that previously didn't
>> I fail utterly to resolve your first sentence with the second.  Put
>> another way, what you describe in your second sentence sure sounds like a
>> "change visible through the IMAP protocol".
> I was thinking that the message looked the same -- it remains static,
> as far as the protocol can see.  Same content, same UID.  The change
> is to the mailbox.

A change to the behavior of SEARCH for that message is a change to that 
message.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.