mailing list archives

Hello,

What's the proper way to notify a client of deleted/expunged message by another agent?

Went through RFC 2180 "IMAP4 Multi-Accessed Mailbox Practice", but I still have some doubts.

Background: in this scenario, all the mailboxes and messages are managed server-side. The client can only select/fetch things in a read-only fashion. The server itself actively manages all the mailboxes and their content, including deletion of messages.

Say one has an "inbox" which always shows the latest thousand messages. Messages flow constantly in the inbox, but only the latest 1,000 are shown in the inbox. 

How to properly notify the client when the message windows change?

For example:

c: abc select inbox
s: * 1000 exist
...
s: abc ok [read-only] select

c: abc fetch 1:*
s: * 1 fetch (uid 82542)
...
s: * 1000 fetch (uid 83541)
s: abc ok fetch

Now a new message arrives, shifting the content of the inbox by one from UID 82542-83541 to UID 82543-83542.

How should the server notify the client?

Would the following unilateral untagged notification be enough?

s: * 1 expunge 
s: * 1000 exists

Thanks in advance.

--

On 7.12.2011, at 11.27, Petite Abeille wrote:

> Now a new message arrives, shifting the content of the inbox by one from UID 82542-83541 to UID 82543-83542.
> 
> How should the server notify the client?
> 
> Would the following unilateral untagged notification be enough?
> 
> s: * 1 expunge 
> s: * 1000 exists

This is correct, but there are restrictions when you can send EXPUNGE notifications:

>       An EXPUNGE response MUST NOT be sent when no command is in
>       progress, nor while responding to a FETCH, STORE, or SEARCH
>       command.  This rule is necessary to prevent a loss of
>       synchronization of message sequence numbers between client and
>       server.  A command is not "in progress" until the complete command
>       has been received; in particular, a command is not "in progress"
>       during the negotiation of command continuation.
> 
>            Note: UID FETCH, UID STORE, and UID SEARCH are different
>            commands from FETCH, STORE, and SEARCH.  An EXPUNGE
>            response MAY be sent during a UID command.

So in these situations either you can't send a new 1000 EXISTS, or you must temporarily allow more than 1000 messages to be accessible.

On Wed, Dec 7, 2011, at 11:55 AM, Timo Sirainen wrote:
> On 7.12.2011, at 11.27, Petite Abeille wrote:
> 
> > Now a new message arrives, shifting the content of the inbox by one from UID 82542-83541 to UID 82543-83542.
> > 
> > How should the server notify the client?
> > 
> > Would the following unilateral untagged notification be enough?
> > 
> > s: * 1 expunge 
> > s: * 1000 exists
> 
> This is correct, but there are restrictions when you can send EXPUNGE notifications:
> 
> >       An EXPUNGE response MUST NOT be sent when no command is in
> >       progress, nor while responding to a FETCH, STORE, or SEARCH
> >       command.  This rule is necessary to prevent a loss of
> >       synchronization of message sequence numbers between client and
> >       server.  A command is not "in progress" until the complete command
> >       has been received; in particular, a command is not "in progress"
> >       during the negotiation of command continuation.
> > 
> >            Note: UID FETCH, UID STORE, and UID SEARCH are different
> >            commands from FETCH, STORE, and SEARCH.  An EXPUNGE
> >            response MAY be sent during a UID command.
> 
> So in these situations either you can't send a new 1000 EXISTS, or you must temporarily allow more than 1000 messages to be accessible.


Nothing stopping you returning an error if someone tries to access the earlier messages
of course - you just need to retain their "position" in the msgid list until the
protocol allows you to send notice of the expunge.

-- 
  Bron Gondwana
  brong@fastmail.fm

On Wed, 7 Dec 2011, Bron Gondwana wrote:
> Nothing stopping you returning an error if someone tries to access the earlier messages
> of course - you just need to retain their "position" in the msgid list until the
> protocol allows you to send notice of the expunge.

Although a server is permitted to do that by the specification, it should
not.

Clients don't expect such errors.

It is MUCH better if the server keeps the message data accessible in the
session until such time as an EXPUNGE can be sent. My implementation calls
these "ghost messages", because their only existance is as ephemerons in
that session (new sessions do not see them) and only until that session
sends an EXPUNGE or is terminated.

Think of how horrible it would be if, while you had a file open, suddenly
read() were to fail because someone did an unlink() on the file.

This should not be a substantial burden on the server, although it may
require a bit more effort on the part of the server author to get it
right.

The convenience of a server author, to be able to do something quick and
dirty, should take second place to the need of a client author to have
reliable server behavior.

Also, in case you worry about ephemeral messages constipating your server:

No client should keep a server on the lurch with ephemerons for more than
a short period of time. A client may do several consecutive commands in a
pipeline without giving the server an EXPUNGE opportunity, but it should
NEVER go idle on the command stream before doing so.

In my opinion, it is alright for a server to protect itself from abusive
client behavior by disconnecting, etc. if it's stuck with ephemerons for
more than a reasonable amount of time. Given that IDLE explicitly disposes
of these, it's only an issue with a NOOP-polling client.

Most NOOP-polling clients do so every 1-3 minutes, and IMAP's maximum
inactivity period is 30 minutes. Certainly a client that wakes up at
minute 29, but does not allow an EXPUNGE opportunity, is abusive and
deserves to lose its session. 15 minutes as the maximum "let me get rid of
my ephemerons or I disconnect you" time is probably quite good enough.

Remember that this isn't an issue with IDLE; it's only for polling
clients. Most polling clients will not go idle that long since that would
be likely to provoke a NAT into disconnecting.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Dec 7, 2011, at 6:13 PM, Mark Crispin wrote:

> The convenience of a server author, to be able to do something quick and
> dirty, should take second place to the need of a client author to have
> reliable server behavior.

Thanks for all the answers.

In my specific scenario, there are always, say, 1,000 sequences in the mailbox. It's just that what messages they represent varies over time. No messages are deleted per se. They just fail out of the "recent messages" window.

In any case, perhaps a simple setup is to freeze the message window upon mailbox selection, until a noop request comes along, at which point the appropriate expunge notification is communicated to the client and the new messages are published. That should guaranty fetch consistency until the server is in a position to communicate the message changes to the client, no?

On Thu, 8 Dec 2011, Petite Abeille wrote:
> In my specific scenario, there are always, say, 1,000 sequences in the
> mailbox. It's just that what messages they represent varies over time.
> No messages are deleted per se. They just fail out of the "recent
> messages" window.

I assume that this is something in an IMAP server? That is, the server
always shows 1000 messages, but from time to time messages vanish in the
existing sequence space, and new messages take their place at the top of
the space?

> In any case, perhaps a simple setup is to freeze the message window upon
> mailbox selection, until a noop request comes along, at which point the
> appropriate expunge notification is communicated to the client and the
> new messages are published. That should guaranty fetch consistency until
> the server is in a position to communicate the message changes to the
> client, no?

If I understand what you are doing correctly, yes.

So, if the bottommost three messages vanish, and are replaced by three new
messages, the server sequence on the EXPUNGE-valid point will be:
 	* 1 EXPUNGE
 	* 1 EXPUNGE
 	* 1 EXPUNGE
 	* 1000 EXISTS

Remember that the EXPUNGE response has immediate effect on the sequence
space. If your server transmits:
 	* 1 EXPUNGE
 	* 2 EXPUNGE
 	* 3 EXPUNGE
that would remove original messages 1,3,5, not 1:3.

However, note that:
 	* 3 EXPUNGE
 	* 2 EXPUNGE
 	* 1 EXPUNGE
is exactly equivalent to:
 	* 1 EXPUNGE
 	* 1 EXPUNGE
 	* 1 EXPUNGE

If you understand why this is the case, then you can be confident that you
understand how sequence numbers work.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Dec 8, 2011, at 9:52 PM, Mark Crispin wrote:

> I assume that this is something in an IMAP server? That is, the server
> always shows 1000 messages, but from time to time messages vanish in the
> existing sequence space, and new messages take their place at the top of
> the space?

Exactly.

> So, if the bottommost three messages vanish, and are replaced by three new
> messages, the server sequence on the EXPUNGE-valid point will be:
> 	* 1 EXPUNGE
> 	* 1 EXPUNGE
> 	* 1 EXPUNGE
> 	* 1000 EXISTS
> 
> Remember that the EXPUNGE response has immediate effect on the sequence
> space. If your server transmits:
> 	* 1 EXPUNGE
> 	* 2 EXPUNGE
> 	* 3 EXPUNGE
> that would remove original messages 1,3,5, not 1:3.
> 
> However, note that:
> 	* 3 EXPUNGE
> 	* 2 EXPUNGE
> 	* 1 EXPUNGE
> is exactly equivalent to:
> 	* 1 EXPUNGE
> 	* 1 EXPUNGE
> 	* 1 EXPUNGE

Good point. So exactly as, say, array/list manipulation: either remove the first element and shift the rest one position down, or remove it from the top.

> If you understand why this is the case, then you can be confident that you
> understand how sequence numbers work.

I think I got the gist of it. Thanks for the explanation.