mailing list archives

Hi,


I'm trying to find out if it's possible to issue multiple non-waiting UID SEARCH commands like this:


1 UID SEARCH UNSEEN
2 UID SEARCH FLAGGED
3 UID SEARCH DELETED



RFC 3501 says this:


UID SEARCH + UID SEARCH may be valid or invalid as a non-waiting
      command sequence, depending upon whether or not the second UID
      SEARCH contains message sequence numbers.


As none of the commands contain message sequence numbers, I would say this should be allowed.


With Gmail and Courier-IMAP everything seems to work fine, but with Dovecot, although both the tagged and untagged responses have the correct order within their category in my testing, sometimes an untagged response is sent before the tagged response of the preceding command:


1 UID SEARCH UNSEEN
2 UID SEARCH FLAGGED
3 UID SEARCH DELETED
* SEARCH 33085
1 OK Search completed (0.000 secs).
* SEARCH
* SEARCH
2 OK Search completed (0.000 secs).
3 OK Search completed (0.000 secs).


But this should be fine as long as I can rely on the order of the untagged responses. So my questions are:


1. Is my interpretation correct that the above commands are valid according to the RFC?
2. Do you know of any servers that don't respond in the order of the commands?


Thanks!
Lasse


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130821/97f9b285/attachment.html>

On Wed, Aug 21, 2013 at 03:26:50AM -0700, Lasse Jansen wrote:
> 1. Is my interpretation correct that the above commands are valid according to the RFC?

Borderline. But I'd say it's okay.

> 2. Do you know of any servers that don't respond in the order of the commands?

No. And I've looked.

The belt-and-braces crowd would suggest implementing ESEARCH: If the
server supports that the untagged response will be tied explicitly to
the appropriated command.

May I ask what you're doing?

Arnt

On 21.8.2013, at 13.26, Lasse Jansen <lasse@lasselog.com> wrote:

> UID SEARCH + UID SEARCH may be valid or invalid as a non-waiting
>       command sequence, depending upon whether or not the second UID
>       SEARCH contains message sequence numbers.
> 
> 
> As none of the commands contain message sequence numbers, I would say this should be allowed.

It is. However it doesn't say anything about the order in which the replies should arrive. So it would make sense only when you just want to combine multiple SEARCHes into one results.

> With Gmail and Courier-IMAP everything seems to work fine, but with Dovecot, although both the tagged and untagged responses have the correct order within their category in my testing, sometimes an untagged response is sent before the tagged response of the preceding command:
> 
> 1 UID SEARCH UNSEEN
> 2 UID SEARCH FLAGGED
> 3 UID SEARCH DELETED
> * SEARCH 33085
> 1 OK Search completed (0.000 secs).
> * SEARCH
> * SEARCH
> 2 OK Search completed (0.000 secs).
> 3 OK Search completed (0.000 secs).
> 
> But this should be fine as long as I can rely on the order of the untagged responses.

For the above SEARCH commands it's very likely that you receive the untagged responses in the correct order, but I don't think it's guaranteed. Dovecot attempts to process SEARCH commands 0.1 seconds at a time, then see if there's some other work to do, handle that and go back to processing SEARCH. So for example if the first SEARCH was using BODY/TEXT in a large enough mailbox, the 2 and 3 SEARCH replies could be finished and returned first. The tagged replies are always returned in order, because Mail.app used to break if they weren't.

FETCH command is handled in somewhat similar way. It gets processed until the client output buffer gets full, then Dovecot sees if there's some other work to be done while waiting, possibly starting another FETCH command that starts mixing the results with the first one. With FETCH this shouldn't cause any trouble though.

On 21/08/2013 11:26, Lasse Jansen wrote:
> Hi,
>
> I'm trying to find out if it's possible to issue multiple non-waiting 
> UID SEARCH commands like this:
>
> 1 UID SEARCH UNSEEN
> 2 UID SEARCH FLAGGED
> 3 UID SEARCH DELETED
>
> RFC 3501 says this:
>
> UID SEARCH + UID SEARCH may be valid or invalid as a non-waiting
>        command sequence, depending upon whether or not the second UID
>        SEARCH contains message sequence numbers.
> As none of the commands contain message sequence numbers, I would say this should be allowed.
>
> With Gmail and Courier-IMAP everything seems to work fine, but with Dovecot, although both the tagged and untagged responses have the correct order within their category in my testing, sometimes an untagged response is sent before the tagged response of the preceding command:
Both are legal.
>
> 1 UID SEARCH UNSEEN
> 2 UID SEARCH FLAGGED
> 3 UID SEARCH DELETED
> * SEARCH 33085
> 1 OK Search completed (0.000 secs).
> * SEARCH
> * SEARCH
> 2 OK Search completed (0.000 secs).
> 3 OK Search completed (0.000 secs).
> But this should be fine as long as I can rely on the order of the untagged responses. So my questions are:
>
> 1. Is my interpretation correct that the above commands are valid according to the RFC?
> 2. Do you know of any servers that don't respond in the order of the commands?
Servers should not reorder responses to such commands or they will be in 
violation of RFC 3501. So yes, you can rely on the order.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130821/1299f09c/attachment.html>

On Wednesday, 21 August 2013 12:26:50 CEST, Lasse Jansen wrote:
> As none of the commands contain message sequence numbers, I 
> would say this should be allowed.

In my understanding of the RFC, even though you are allowed to send this, 
the result you get might not match your expectations, as others have 
explained in this thread.

The command to implement here is ESEARCH which:

a) solves your pipelining problem neatly and properly,
b) allows for further compression of the result via the min:max syntax,
c) is nowadays supported even by GMail

Furthermore, if your IMAP server supports CONDSTORE (again, GMail nowadays 
does), you can save yourself some traffic during the flag resynchronization 
by using FETCH CHANGEDSINCE.

There is one more uncertainity with the SEARCH response -- some sources 
(see the whole thread at [1]) support the interpretation that it is OK to 
actually send the SEARCH data for a single command split into multiple 
responses which are to be joined by the client. I wonder if this is valid. 
My understanding of RFC3501 wording is that such behavior is *not* correct 
and that there should always be exactly one SEARCH response to each SEARCH 
command, but my client will join them anyway. I wasn't able to pinpoint 
some authoritative source which made me change this, unfortunately.

With kind regards,
Jan

[1] http://www.mail-archive.com/imap@u.washington.edu/msg02127.html

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

We are currently working on optimizing the IMAP sync of our email client Unibox [1]. I've looked at other clients and most of them use things like UID FETCH 1:* (UID FLAGS) or similar to update flags of already-synced emails, but this sends quite a lot of redundant data. As we only care about very few tags anyway (seen, deleted, flagged) we implemented the flag-updating with UID SEARCH. Being able to pipeline this would speed things up.


[1] http://www.uniboxapp.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130821/56df87f0/attachment.html>

>> But this should be fine as long as I can rely on the order of the untagged
>> responses. So my questions are:
>>
>> 1. Is my interpretation correct that the above commands are valid according
>> to the RFC?
>>
>> 2. Do you know of any servers that don't respond in the order of the
>> commands?
>
> Servers should not reorder responses to such commands or they will be in
> violation of RFC 3501. So yes, you can rely on the order.

Really?  Can you show us where in 3501 it's required that untagged
responses be in any particular order?  I don't think it does.

Barry

Ok, cool. So it looks like my fallback option will stay non-pipelined UID SEARCH and I'll need to decide if I use pipelined ESEARCH or CONDSTORE as optimized version.


Thanks all!
Lasse

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130821/3f8720f5/attachment.html>

On 21/08/2013 15:47, Barry Leiba wrote:
>>> But this should be fine as long as I can rely on the order of the untagged
>>> responses. So my questions are:
>>>
>>> 1. Is my interpretation correct that the above commands are valid according
>>> to the RFC?
>>>
>>> 2. Do you know of any servers that don't respond in the order of the
>>> commands?
>> Servers should not reorder responses to such commands or they will be in
>> violation of RFC 3501. So yes, you can rely on the order.
> Really?  Can you show us where in 3501 it's required that untagged
> responses be in any particular order?  I don't think it does.
I was looking at:

5.5.    Multiple Commands in Progress

    The client MAY send another command without waiting for the
    completion result response of a command, subject to ambiguity rules
    (see below) and flow control constraints on the underlying data
    stream.  Similarly, a server MAY begin processing another command
    before processing the current command to completion, subject to
    ambiguity rules.  However, any command continuation request responses
    and command continuations MUST be negotiated before any subsequent
    command is initiated.

    The exception is if an ambiguity would result because of a command
    that would affect the results of other commands.  Clients MUST NOT
    send multiple commands without waiting if an ambiguity would result.
    If the server detects a possible ambiguity, it MUST execute commands
    to completion in the order given by the client.

(The last sentence) But I suppose it doesn't cover cases when there is 
no ambiguity, for example if commands are truly independent.

So I suppose I was not quite correct. In reality though, I don't know of 
any server that would execute multiple SEARCHes out of order. I am 
curious to learn if any such server exists.

Dovecot does apparently if a search command takes longer that 0.1 seconds as Timo mentioned before. I tried his suggestion to force this behavior with a BODY search and it does indeed happen:


1 UID SEARCH BODY "NOT IN MY MESSAGES"
2 UID SEARCH UNSEEN
* SEARCH 33115
* SEARCH
1 OK Search completed (7.811 secs).
2 OK Search completed (0.000 secs).



The unseen message is reported before the empty search result for the body search.


Lasse

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130821/e43ffbe4/attachment.html>

On 21.8.2013, at 21.00, Arnt Gulbrandsen <arnt@gulbrandsen.priv.no> wrote:

> From hazy memory: In case of ambiguity, the server must process the commands in the order received?

I think this is the main point of it:

> The exception is if an ambiguity would result because of a command that would affect the results of other commands.

Running two SEARCH commands in reverse order don't affect the results of each others, just the order in which they're received by the client.

On 21.8.2013, at 18.35, Lasse Jansen <lasse@lasselog.com> wrote:

> Dovecot does apparently if a search command takes longer that 0.1 seconds as Timo mentioned before. I tried his suggestion to force this behavior with a BODY search and it does indeed happen:
> 
> 1 UID SEARCH BODY "NOT IN MY MESSAGES"
> 2 UID SEARCH UNSEEN
> * SEARCH 33115
> * SEARCH
> 1 OK Search completed (7.811 secs).
> 2 OK Search completed (0.000 secs).
> 
> The unseen message is reported before the empty search result for the body search.

I don't remember if that was really intentional though, since the main idea was to be able to send other commands during a long SEARCH, not really to be able to send another SEARCH. It requires adding extra code to prevent that. I think I'll probably add that, since although I'm not convinced the current way is violating RFC 3501, it's also not very helpful behavior for clients. But of course it'll take 10 years before most of the existing servers are upgraded..

> I don't remember if that was really intentional though, since the main idea was to
> be able to send other commands during a long SEARCH, not really to be able to
> send another SEARCH. It requires adding extra code to prevent that. I think I'll
> probably add that, since although I'm not convinced the current way is violating
> RFC 3501, it's also not very helpful behavior for clients. But of course it'll take
> 10 years before most of the existing servers are upgraded..

I don't think any SEARCH pipelining is helpful to clients -- that's a
large part of the reason we did ESEARCH.  Because the search results
come in untagged, unlabelled responses that can't be correlated with
the search criteria that match them, any client that tries to pipeline
SEARCH commands is misguided, and is asking for trouble.

So I don't think you should spend your time on "fixing" this.

Barry

On 21/08/2013 17:10, Barry Leiba wrote:
>> I don't remember if that was really intentional though, since the main idea was to
>> be able to send other commands during a long SEARCH, not really to be able to
>> send another SEARCH. It requires adding extra code to prevent that. I think I'll
>> probably add that, since although I'm not convinced the current way is violating
>> RFC 3501, it's also not very helpful behavior for clients. But of course it'll take
>> 10 years before most of the existing servers are upgraded..
> I don't think any SEARCH pipelining is helpful to clients -- that's a
> large part of the reason we did ESEARCH.  Because the search results
> come in untagged, unlabelled responses that can't be correlated with
> the search criteria that match them, any client that tries to pipeline
> SEARCH commands is misguided, and is asking for trouble.
>
> So I don't think you should spend your time on "fixing" this.
+1.