mailing list archives

Hi,
I've received a report that Courier 4.5.0 sometimes doesn't return UIDNEXT when syncing a mailbox, violating RFC 3501. Trojita used to hit an assert in such case which was clearly my bug. This has now been fixed.

My code has always treated missing EXISTS, UIDNEXT or UIDVALIDITY (either in what was currently received from a mailbox or what we have in a persistent cache since the last time) as a reason to throw away anything in the cache and fall back to a full sync, similar to what happens when UIDVALIDITY changes. I wonder if this is really required -- on one hand, the IMAP server clearly violates the spec, but on the other hand, it might be tempting to work around this situation by at least preserving the cached message parts data so that users don't have to download everything again. The UID-seq mapping will have to be reconstructed anyway, but that's unavoidable.

Is it really required or recommended to discard the cached immutable data for messages in a mailbox when the server doesn't send UIDNEXT?

With kind regards,
Jan

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Quoting Jan Kundr?t <jkt@flaska.net>:

> Hi,
> I've received a report that Courier 4.5.0 sometimes doesn't return  
> UIDNEXT when syncing a mailbox, violating RFC 3501. Trojita used to  
> hit an assert in such case which was clearly my bug. This has now  
> been fixed.

Courier definitely does this, and it is not against spec.  From RFC 3501:

          OK [UIDNEXT <n>]
                      The next unique identifier value.  Refer to section
                      2.3.1.1 for more information.  If this is missing,
                      the client can not make any assumptions about the
                      next unique identifier value.

For cache purposes, you can workaround a missing UIDNEXT response by  
doing a 'UID FETCH * (UID)' in the mailbox.  For empty mailboxes, use  
the returned UID value (which is defined as the UIDNEXT value).   
Otherwise, you can add 1 to the returned value and use this as the  
UIDNEXT-ish value.

michael

On Fri, Dec 28, 2012, at 07:27 PM, Michael M Slusarz wrote:
> Quoting Jan Kundr?t <jkt@flaska.net>:
> 
> > Hi,
> > I've received a report that Courier 4.5.0 sometimes doesn't return  
> > UIDNEXT when syncing a mailbox, violating RFC 3501. Trojita used to  
> > hit an assert in such case which was clearly my bug. This has now  
> > been fixed.
> 
> Courier definitely does this, and it is not against spec.  From RFC 3501:
> 
>           OK [UIDNEXT <n>]
>                       The next unique identifier value.  Refer to section
>                       2.3.1.1 for more information.  If this is missing,
>                       the client can not make any assumptions about the
>                       next unique identifier value.

And yet:

B. Changes from RFC 2060

   34) Clarify UNSEEN, PERMANENTFLAGS, UIDVALIDITY, and UIDNEXT
   responses in SELECT and EXAMINE.  They are required now, but weren't
   in older versions.

Sounds like it's supposed to be there in 3501.  Also listed with REQUIRED.

> For cache purposes, you can workaround a missing UIDNEXT response by  
> doing a 'UID FETCH * (UID)' in the mailbox.

Ouch.

> For empty mailboxes, use  
> the returned UID value (which is defined as the UIDNEXT value).   
> Otherwise, you can add 1 to the returned value and use this as the  
> UIDNEXT-ish value.

The more interesting question is: what do you need to know it for anyway?

Bron.
-- 
  Bron Gondwana
  brong@fastmail.fm

On Friday, 28 December 2012 19:27:25 CEST, Michael M Slusarz wrote:
> Courier definitely does this, and it is not against spec.

Looks like that part of 3501 was not updated from 2060, then -- the UIDNEXT is listed as REQUIRED in the description of the SELECT command, as Bron pointed out.

The safest thing to assume is probably that one is talking to a 2060-compliant server when UIDNEXT is missing. The text you quote indicates that UIDVALIDITY was not required in 2060, either; my code will work in such case, but the cache will be invalidated on next reconnect.

With kind regards,
Jan

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Quoting Bron Gondwana <brong@fastmail.fm>:

> And yet:
>
> B. Changes from RFC 2060
>
>   34) Clarify UNSEEN, PERMANENTFLAGS, UIDVALIDITY, and UIDNEXT
>   responses in SELECT and EXAMINE.  They are required now, but weren't
>   in older versions.
>
> Sounds like it's supposed to be there in 3501.  Also listed with REQUIRED.

Agreed that some (most) of the text indicates that this should be  
required.  But the text I quoted previously explicitly indicates that  
UIDNEXT might not be available.  So there is, at worst, an ambiguity.

 From a practical perspective: at least one (somewhat widely used)  
IMAP server fails to issue the response, so a competent client should  
be able to work around its absence regardless of whether it is  
semantically incorrect.

>> For cache purposes, you can workaround a missing UIDNEXT response by
>> doing a 'UID FETCH * (UID)' in the mailbox.
>
> Ouch.

I assume this means this is a slow operation server-side (at least in  
Cyrus).  I personally don't care since it is a viable way of working  
around a possible RFC violation.  Performance isn't an issue here.

>> For empty mailboxes, use
>> the returned UID value (which is defined as the UIDNEXT value).
>> Otherwise, you can add 1 to the returned value and use this as the
>> UIDNEXT-ish value.
>
> The more interesting question is: what do you need to know it for anyway?

To determine if the mailbox has changed to prevent an issuance of an  
unneeded FETCH/SORT command.  Flag changes are irrelevant (if  
CONDSTORE isn't available), since clients can be designed to not care  
about these changes

michael

On Friday, 28 December 2012 20:33:10 CEST, Bron Gondwana wrote:
> The more interesting question is: what do you need to know it for anyway?

It's needed for an old heuristic -- if UIDVALIDITY is the same and (exists - old_exists) == (uidnext - old_uidnext), then the mailbox has only seen new arrivals since the last time, so the client doesn't have to fetch the whole UID-seq mapping once again, just the new arrivals. (If the first difference is higher, then the server is broken or the local cache was corrupted, so a revert to a dumb sync.)

Which is also the reason why my code checks for missing UIDNEXT -- when it's not available, this heuristic cannot work and we have to go back to full UID syncing.

Cheers,
Jan

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Quoting Jan Kundr?t <jkt@flaska.net>:

> Looks like that part of 3501 was not updated from 2060, then -- the  
> UIDNEXT is listed as REQUIRED in the description of the SELECT  
> command, as Bron pointed out.

I reported this as an errata to RFC 3501, with the recommendation that  
the "assumption" sentence from the UIDNEXT text be removed in order to  
eliminate the ambiguity between this text (which implies that the  
UIDNEXT response may be optional) and the REQUIRED requirement located  
in Sections 6.3.1 & 6.3.2.

michael

On Fri, Dec 28, 2012, at 09:01 PM, Michael M Slusarz wrote:
> Quoting Bron Gondwana <brong@fastmail.fm>:
> >> For cache purposes, you can workaround a missing UIDNEXT response by
> >> doing a 'UID FETCH * (UID)' in the mailbox.
> >
> > Ouch.
> 
> I assume this means this is a slow operation server-side (at least in  
> Cyrus).  I personally don't care since it is a viable way of working  
> around a possible RFC violation.  Performance isn't an issue here.

No, actually - it should be pretty quick.  I don't think it even needs to
do a search of the address space, it should just read the last item if
* matches uid_last.  Otherwise it will do a binary search of the index
map.

But of course Cyrus always returns UIDNEXT anyway.

> > The more interesting question is: what do you need to know it for anyway?
> 
> To determine if the mailbox has changed to prevent an issuance of an  
> unneeded FETCH/SORT command.  Flag changes are irrelevant (if  
> CONDSTORE isn't available), since clients can be designed to not care  
> about these changes

Fair enough I guess, if you really don't need the data from those intermediate
records.  Otherwise you may as well just do fetch $olduid+1:* (stuff you need),
which will be the same number of round trips as your solution in the trivial
case, and one fewer in the case where there is new data to know.

Let's play make believe for a second with Cyrus though:

. select inbox
* 2 EXISTS
* 2 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
* OK [UNSEEN 1] Ok
* OK [UIDVALIDITY 1356732706] Ok
* OK [UIDNEXT 3] Ok
* OK [HIGHESTMODSEQ 5] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
. OK [READ-WRITE] Completed
. fetch 1:* (uid)     
* 1 FETCH (UID 1)
* 2 FETCH (UID 2)
. OK Completed (0.000 sec)
. fetch * (uid)
* 2 FETCH (UID 2)
. OK Completed (0.000 sec 
-- 
  Bron Gondwana
  brong@fastmail.fm

>> Looks like that part of 3501 was not updated from 2060, then -- the
>> UIDNEXT is listed as REQUIRED in the description of the SELECT command, as
>> Bron pointed out.
>
> I reported this as an errata to RFC 3501, with the recommendation that the
> "assumption" sentence from the UIDNEXT text be removed in order to eliminate
> the ambiguity between this text (which implies that the UIDNEXT response may
> be optional) and the REQUIRED requirement located in Sections 6.3.1 & 6.3.2.

Um...

But this erratum is wrong.  You jumped to premature conclusions.

Look at the paragraph above the responses in Section 6.3.1.  It says this:

      The SELECT command selects a mailbox so that messages in the
      mailbox can be accessed.  Before returning an OK to the client,
      the server MUST send the following untagged data to the client.
      Note that earlier versions of this protocol only required the
      FLAGS, EXISTS, and RECENT untagged data; consequently, client
      implementations SHOULD implement default behavior for missing data
      as discussed with the individual item.

The text that you're suggesting removing is there very purposefully,
and should NOT be removed.  I intend to mark the erratum as
"Rejected", but I'll give us some time to discuss it further first.

Barry, Applications AD

On Fri, Dec 28, 2012, at 11:13 PM, Bron Gondwana wrote:
> On Fri, Dec 28, 2012, at 09:01 PM, Michael M Slusarz wrote:
> > Quoting Bron Gondwana <brong@fastmail.fm>:
> > >> For cache purposes, you can workaround a missing UIDNEXT response by
> > >> doing a 'UID FETCH * (UID)' in the mailbox.
> > >
> > > Ouch.
> > 
> > I assume this means this is a slow operation server-side (at least in  
> > Cyrus).  I personally don't care since it is a viable way of working  
> > around a possible RFC violation.  Performance isn't an issue here.
> 
> No, actually - it should be pretty quick.  I don't think it even needs to
> do a search of the address space, it should just read the last item if
> * matches uid_last.  Otherwise it will do a binary search of the index
> map.
> 
> But of course Cyrus always returns UIDNEXT anyway.
> 
> > > The more interesting question is: what do you need to know it for anyway?
> > 
> > To determine if the mailbox has changed to prevent an issuance of an  
> > unneeded FETCH/SORT command.  Flag changes are irrelevant (if  
> > CONDSTORE isn't available), since clients can be designed to not care  
> > about these changes
> 
> Fair enough I guess, if you really don't need the data from those intermediate
> records.  Otherwise you may as well just do fetch $olduid+1:* (stuff you need),
> which will be the same number of round trips as your solution in the trivial
> case, and one fewer in the case where there is new data to know.
> 
> Let's play make believe for a second with Cyrus though:
> 
> . select inbox
> * 2 EXISTS
> * 2 RECENT
> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
> * OK [UNSEEN 1] Ok
> * OK [UIDVALIDITY 1356732706] Ok
> * OK [UIDNEXT 3] Ok
> * OK [HIGHESTMODSEQ 5] Ok
> * OK [URLMECH INTERNAL] Ok
> * OK [ANNOTATIONS 65536] Ok
> . OK [READ-WRITE] Completed
> . fetch 1:* (uid)     
> * 1 FETCH (UID 1)
> * 2 FETCH (UID 2)
> . OK Completed (0.000 sec)
> . fetch * (uid)
> * 2 FETCH (UID 2)
> . OK Completed (0.000 sec 

Sorry - cat on keyboard failure.

)
. store 2 +flags \Deleted
* 2 FETCH (FLAGS (\Recent \Deleted))
. OK Completed
. expunge
* 2 EXPUNGE
. OK [HIGHESTMODSEQ 7] Completed
. fetch * (uid)
* 1 FETCH (UID 1)
. OK Completed (0.000 sec)

That definitely doesn't work!

. uid fetch * (uid)
. OK Completed (0.000 sec)

And that contains nothing.  You can't know from that if stuff got added and
expunged since your last request though.

Consider:

UIDs 1 3 4 5 7 
EXISTS 5
UIDNEXT 10

APPEND 11 12
EXPUNGE 4 12

UIDs 1 3 5 7 11
EXISTS 5
UIDNEXT 13

Your query couldn't tell the difference between state A and state B.

Bron.
-- 
  Bron Gondwana
  brong@fastmail.fm

Quoting Barry Leiba <barryleiba@computer.org>:

>>> Looks like that part of 3501 was not updated from 2060, then -- the
>>> UIDNEXT is listed as REQUIRED in the description of the SELECT command, as
>>> Bron pointed out.
>>
>> I reported this as an errata to RFC 3501, with the recommendation that the
>> "assumption" sentence from the UIDNEXT text be removed in order to eliminate
>> the ambiguity between this text (which implies that the UIDNEXT response may
>> be optional) and the REQUIRED requirement located in Sections 6.3.1 & 6.3.2.
>
> Um...
>
> But this erratum is wrong.  You jumped to premature conclusions.

Sigh.  I *KNEW* that UIDNEXT was not required for some reason - I had  
just forgotten the logic & explanation.  For some reason I forgot that  
RFC 2060 is also IMAP4rev1, NOT IMAP4, so there are several backward  
compatibility concerns between the two documents.

> Look at the paragraph above the responses in Section 6.3.1.  It says this:
>
>       The SELECT command selects a mailbox so that messages in the
>       mailbox can be accessed.  Before returning an OK to the client,
>       the server MUST send the following untagged data to the client.
>       Note that earlier versions of this protocol only required the
>       FLAGS, EXISTS, and RECENT untagged data; consequently, client
>       implementations SHOULD implement default behavior for missing data
>       as discussed with the individual item.

I'd still argue that labeling the PERMANENTFLAGS/UIDNEXT responses an  
unqualified REQUIRED in the SELECT/EXAMINE description is, at best,  
confusing - this thread being proof of this fact. (I will note the  
UNSEEN response has already been modified in an errata to handle a  
similar case where UNSEEN is REQUIRED... unless it isn't.)  However,  
this critique is more of an extremely tardy editorial comment rather  
than an errata.

> The text that you're suggesting removing is there very purposefully,
> and should NOT be removed.  I intend to mark the erratum as
> "Rejected", but I'll give us some time to discuss it further first.

I don't think further discussion is necessary.  You are correct - it  
should be rejected.  Wishing there was an "unsubmit" button on the  
submission page...

Sorry for the noise,
michael

Quoting Bron Gondwana <brong@fastmail.fm>:

>> > > The more interesting question is: what do you need to know it  
>> for anyway?
>> >
>> > To determine if the mailbox has changed to prevent an issuance of an
>> > unneeded FETCH/SORT command.  Flag changes are irrelevant (if
>> > CONDSTORE isn't available), since clients can be designed to not care
>> > about these changes
>>
>> Fair enough I guess, if you really don't need the data from those  
>> intermediate
>> records.  Otherwise you may as well just do fetch $olduid+1:*  
>> (stuff you need),
>> which will be the same number of round trips as your solution in the trivial
>> case, and one fewer in the case where there is new data to know.

If UIDNEXT doesn't change, why would you be doing a fetch?

>> Let's play make believe for a second with Cyrus though:
>>
>> . select inbox
>> * 2 EXISTS
>> * 2 RECENT
>> * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)] Ok
>> * OK [UNSEEN 1] Ok
>> * OK [UIDVALIDITY 1356732706] Ok
>> * OK [UIDNEXT 3] Ok
>> * OK [HIGHESTMODSEQ 5] Ok
>> * OK [URLMECH INTERNAL] Ok
>> * OK [ANNOTATIONS 65536] Ok
>> . OK [READ-WRITE] Completed
>> . fetch 1:* (uid)
>> * 1 FETCH (UID 1)
>> * 2 FETCH (UID 2)
>> . OK Completed (0.000 sec)
>> . fetch * (uid)
>> * 2 FETCH (UID 2)
>> . OK Completed (0.000 sec
>
> Sorry - cat on keyboard failure.

I have a dog, so I tend to not have these kind of issues :).

> )
> . store 2 +flags \Deleted
> * 2 FETCH (FLAGS (\Recent \Deleted))
> . OK Completed
> . expunge
> * 2 EXPUNGE
> . OK [HIGHESTMODSEQ 7] Completed
> . fetch * (uid)
> * 1 FETCH (UID 1)
> . OK Completed (0.000 sec)
>
> That definitely doesn't work!

Why not?  The stored UIDNEXT != the cached UIDNEXT.  So something is  
up and the client should issue FETCH's to retrieve further  
information.  UIDNEXT's have nothing to do with the actual next  
assigned UID, so I don't care if it is incrementing or decrementing.   
It only matters if it is different.

> . uid fetch * (uid)
> . OK Completed (0.000 sec)

I may have mis-typed before.  A UID FETCH * is not useful - it should  
be a FETCH * (UID) used instead to determine the last UID in the  
mailbox.  And if the mailbox is empty, I agree that the actual UIDNEXT  
value can't be determined.  In our client, we clear the UIDNEXT and  
cache so the next time a message is seen the cache is restart/rebuilt.

michael

On 31 Dec 2012 20:32, "Michael M Slusarz" <slusarz@curecanti.org> wrote:
> Why not?  The stored UIDNEXT != the cached UIDNEXT.  So something is up
and the client should issue FETCH's to retrieve further information.
 UIDNEXT's have nothing to do with the actual next assigned UID, so I don't
care if it is incrementing or decrementing.  It only matters if it is
different.
>

Kind of. If size less cached size equals uidnext less cached uidnext, there
have been no expunges.

>> . uid fetch * (uid)
>> . OK Completed (0.000 sec)
>
>
> I may have mis-typed before.  A UID FETCH * is not useful - it should be
a FETCH * (UID) used instead to determine the last UID in the mailbox.  And
if the mailbox is empty, I agree that the actual UIDNEXT value can't be
determined.  In our client, we clear the UIDNEXT and cache so the next time
a message is seen the cache is restart/rebuilt.

Uid fetch * (uid) is the same as Fetch * (uid).

For updating a msgno/uid mapping, you want uid search. Ideally with return
(all), though in practise,  I doubt there are any servers that don't do
qresync but still do esearch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130101/c91cb975/attachment.html>