mailing list archives

I basically want to be able to do searches across my entire mail
database, instead of just folder-by-folder, and I'm wondering how to
plaster that into the protocol.  I suppose one way would be to present
all the mail as one tricky folder which normally contains one message,
but responds to SEARCH over it by suddenly filling itself with the
results of the search, sending an untagged EXISTS, and sending them
back to the client.  The mailbox would gradually fill up with search
results; I'd have to expire them and send EXPUNGES periodically.

Are there any protocol extensions I should know about which support
this kind of thing?

Bill

On Mon, 12 Mar 2007, Bill Janssen wrote:
> I basically want to be able to do searches across my entire mail
> database, instead of just folder-by-folder, and I'm wondering how to
> plaster that into the protocol.

There is an unpublished extension, implemented by UW imapd, called SCAN 
which is a LIST with a third argument that consists of text to be looked 
up in the mailbox, and which acts as a filter on the LIST results.  Only 
the mailboxes which contain that text are returned in untagged LIST 
responses.

However, SCAN is a disgusting hack that was done to create the equivalent 
of "greping through mail files" since there were oldtimers who refused to 
use IMAP unless they could "grep over IMAP" (actually, "fgrep" since they 
didn't actually use regular expressions and neither does SCAN).

The problem with SCAN is that, like [f]grep, it makes no distinction 
between mailbox metadata and message data.  Thus, the command
 	tag SCAN "" % "mbx"
will match all mailboxes on a system which uses mbx-format as the mail 
store, since every mbx-format mailbox starts with "*mbx*".

A much better thing to do would be to implement to corresponding 
functionality in LISTEXT.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

On Mon, 2007-03-12 at 16:29 -0700, Bill Janssen wrote:
> I basically want to be able to do searches across my entire mail
> database, instead of just folder-by-folder, and I'm wondering how to
> plaster that into the protocol.  I suppose one way would be to present
> all the mail as one tricky folder which normally contains one message,
> but responds to SEARCH over it by suddenly filling itself with the
> results of the search, sending an untagged EXISTS, and sending them
> back to the client.  The mailbox would gradually fill up with search
> results; I'd have to expire them and send EXPUNGES periodically.

That's a pretty nice idea, I think I'll do something similar if/when I
implement support for searching multiple mailboxes :) The way I'd do it
is:

 - Have one "Search all folders" mailbox
 - It has zero messages initially
 - SEARCH done inside it
   1) EXPUNGEs all the existing messages if there are any (although
unless UID SEARCH is done, the EXPUNGEs can't be sent to client yet)
   2) Adds all the matched messages virtually to the mailbox, and
returns EXISTS with the new message count (includes the old messages
with non-UID SEARCH)
   3) Send SEARCH result after the EXISTS reply with listing all the new
messages.

Hmm. I guess this could be generalized to be a virtual folder where the
search rule gets updated by the SEARCH command.

There also exists a draft for searching multiple mailboxes:
draft-melnikov-imapext-multimailbox-search-01.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/ccc0cc5d/attachment.sig>

> The problem with SCAN is that, like [f]grep, it makes no distinction 
> between mailbox metadata and message data.

Yep, that's not good.

>  - Have one "Search all folders" mailbox
>  - It has zero messages initially
>  - SEARCH done inside it
>    1) EXPUNGEs all the existing messages if there are any (although
> unless UID SEARCH is done, the EXPUNGEs can't be sent to client yet)
>    2) Adds all the matched messages virtually to the mailbox, and
> returns EXISTS with the new message count (includes the old messages
> with non-UID SEARCH)
>    3) Send SEARCH result after the EXISTS reply with listing all the new
> messages.

Yep, that's what I was thinking.  I think I'd keep one message in it
-- not sure how empty folders are regarded by clients.

> Hmm. I guess this could be generalized to be a virtual folder where the
> search rule gets updated by the SEARCH command.

That's how I did it a few years ago with a Verity/mh-e indexing
system.  Every search got saved in a "past searches" namespace, and
you could visit it and look over the "folders".  Opening one re-ran
the search and presented the results.  Could do the same thing here.
Is there an out-of-band way of announcing a new mailbox?

Bill

> There also exists a draft for searching multiple mailboxes:
> draft-melnikov-imapext-multimailbox-search-01.txt

Thanks for the poiner, I like this.  Seems reasonable to implement.
Any clients support it?

Bill

tss@iki.fi wrote:
> On Mon, 2007-03-12 at 16:29 -0700, Bill Janssen wrote:
>   
>> I basically want to be able to do searches across my entire mail
>> database, instead of just folder-by-folder, and I'm wondering how to
>> plaster that into the protocol.  I suppose one way would be to present
>> all the mail as one tricky folder which normally contains one message,
>> but responds to SEARCH over it by suddenly filling itself with the
>> results of the search, sending an untagged EXISTS, and sending them
>> back to the client.  The mailbox would gradually fill up with search
>> results; I'd have to expire them and send EXPUNGES periodically.
>>     
>
> That's a pretty nice idea, I think I'll do something similar if/when I
> implement support for searching multiple mailboxes :) The way I'd do it
> is:
>
>  - Have one "Search all folders" mailbox
>  - It has zero messages initially
>  - SEARCH done inside it
>    1) EXPUNGEs all the existing messages if there are any (although
> unless UID SEARCH is done, the EXPUNGEs can't be sent to client yet)
>    2) Adds all the matched messages virtually to the mailbox, and
> returns EXISTS with the new message count (includes the old messages
> with non-UID SEARCH)
>    3) Send SEARCH result after the EXISTS reply with listing all the new
> messages.
>
> Hmm. I guess this could be generalized to be a virtual folder where the
> search rule gets updated by the SEARCH command.
>
> There also exists a draft for searching multiple mailboxes:
> draft-melnikov-imapext-multimailbox-search-01.txt
>   

How about combining the multiple mailbox search with a select command to 
create a temporary mailbox view that contains the search result?  Since 
it doesn't exists as a listable mailbox, then it can't be the 
destination for a copy command, avoiding any issue of which underlying 
mailbox should be the destination.  The untagged exists response 
response would show the number of hits to the search query.  Also, the 
result set could be referenced by sequence number instead of the 
individual UIDs as returned by the standard search command.

snow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/5d0c9329/attachment.html>

On Mon, 2007-03-12 at 17:16 -0700, Bill Janssen wrote:
> 
> 
> Yep, that's what I was thinking.  I think I'd keep one message in it
> -- not sure how empty folders are regarded by clients.

I don't think there are any problems with empty mailboxes.

> Is there an out-of-band way of announcing a new mailbox?

Sending an untagged * LIST reply (and LSUB maybe). I doubt there are
many (any?) clients that pick it up though.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: This is a digitally signed message part
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/dc6606da/attachment.sig>

> How about combining the multiple mailbox search with a select command to 
> create a temporary mailbox view that contains the search result?  Since 
> it doesn't exists as a listable mailbox, then it can't be the 
> destination for a copy command, avoiding any issue of which underlying 
> mailbox should be the destination.  The untagged exists response 
> response would show the number of hits to the search query.  Also, the 
> result set could be referenced by sequence number instead of the 
> individual UIDs as returned by the standard search command.

How would the client know to SELECT it?

I was actually thinking of doing what Timo reminded me of yesterday,
using a CREATE/SELECT combo, which would create a new mailbox the name
of which would be the query, and would then rescan the query if it was
selected.  They would be persistent in the server, but since they would
be smart virtual mailboxes, this wouldn't be a big cost.

Bill

John Snow wrote:

> How about combining the multiple mailbox search with a select command 
> to create a temporary mailbox view that contains the search result?  
> Since it doesn't exists as a listable mailbox, then it can't be the 
> destination for a copy command, avoiding any issue of which underlying 
> mailbox should be the destination.  The untagged exists response 
> response would show the number of hits to the search query.  Also, the 
> result set could be referenced by sequence number instead of the 
> individual UIDs as returned by the standard search command.

This looks similar to VFOLDER extension: 
<http://tools.ietf.org/html/draft-ietf-lemonade-vfolder-01>

Bill Janssen wrote:
>> How about combining the multiple mailbox search with a select command to 
>> create a temporary mailbox view that contains the search result?  Since 
>> it doesn't exists as a listable mailbox, then it can't be the 
>> destination for a copy command, avoiding any issue of which underlying 
>> mailbox should be the destination.  The untagged exists response 
>> response would show the number of hits to the search query.  Also, the 
>> result set could be referenced by sequence number instead of the 
>> individual UIDs as returned by the standard search command.
>>     
>
> How would the client know to SELECT it?
>
> I was actually thinking of doing what Timo reminded me of yesterday,
> using a CREATE/SELECT combo, which would create a new mailbox the name
> of which would be the query, and would then rescan the query if it was
> selected.  They would be persistent in the server, but since they would
> be smart virtual mailboxes, this wouldn't be a big cost.
>
> Bill
>   

The client would know to use this method the same way it would know to 
do any other extension, it would be written to take advantage of the 
advertised capability.

Is there a way to identify a mailbox as read-only?  What happens when a 
client tries to copy or append a new message to your "search all 
folders" mailbox?  I suppose you'd have to respond NO, but there should 
be a way for the client to know before trying such an operation.

snow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/96b60da1/attachment.html>

alexey.melnikov@isode.com wrote:
> John Snow wrote:
>
>> How about combining the multiple mailbox search with a select command 
>> to create a temporary mailbox view that contains the search result?  
>> Since it doesn't exists as a listable mailbox, then it can't be the 
>> destination for a copy command, avoiding any issue of which 
>> underlying mailbox should be the destination.  The untagged exists 
>> response response would show the number of hits to the search query.  
>> Also, the result set could be referenced by sequence number instead 
>> of the individual UIDs as returned by the standard search command.
>
> This looks similar to VFOLDER extension: 
> <http://tools.ietf.org/html/draft-ietf-lemonade-vfolder-01>
>
Sort of.   In vfolder, the search criteria is stored to create a view 
that is a subset of an existing mailbox. In that case, it can easily be 
treated the same as any other mailbox.  My hope is to expand that beyond 
a single mailbox.  I think creating a server stored view is 
unnecessary.  The extension aware client can create the combined view on 
the fly.  I see it more as an extension to search than a new virtual 
mailbox.   In a standard search, the results are only updated when the 
client issues a new search and not pushed from the server.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070314/f64aa600/attachment.html>

> The client would know to use this method the same way it would know to 
> do any other extension, it would be written to take advantage of the 
> advertised capability.

Oh, OK.  I was trying to think of ways to do it that wouldn't require
an extension.

> Is there a way to identify a mailbox as read-only?  What happens when a 
> client tries to copy or append a new message to your "search all 
> folders" mailbox?  I suppose you'd have to respond NO, but there should 
> be a way for the client to know before trying such an operation.

From RFC 3501:

      If the client is not permitted to modify the mailbox but is
      permitted read access, the mailbox is selected as read-only, and
      the server MUST prefix the text of the tagged OK response to
      SELECT with the "[READ-ONLY]" response code.

But my search folders are really Presto folders (see section 4.2 of
http://www2.parc.com/csl/projects/placeless/papers/tochi-presto.pdf,
page 9), though the system isn't based on the Presto codebase.  So
I could support COPY and/or APPEND.

Bill

John Snow writes:
> Sort of.   In vfolder, the search criteria is stored to create a view 
> that is a subset of an existing mailbox. In that case, it can easily 
> be treated the same as any other mailbox.  My hope is to expand that 
> beyond a single mailbox.

Vfolder will do that by using Alexey's multimailbox search, at least 
that's the vague intention.

> I think creating a server stored view is unnecessary.  The extension 
> aware client can create the combined view on the fly.

FWIW, vfolder is the way it so so that unextended clients see the data, 
too. The folder can be set up by an extended client or outside IMAP and 
accessed by anything.

Arnt

>   
>> Is there a way to identify a mailbox as read-only?  
>>     
>
> From RFC 3501:
>
>       If the client is not permitted to modify the mailbox but is
>       permitted read access, the mailbox is selected as read-only, and
>       the server MUST prefix the text of the tagged OK response to
>       SELECT with the "[READ-ONLY]" response code.
>
>   

Yep, forgot about that.  thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070313/f5ec3c76/attachment.html>

arnt@gulbrandsen.priv.no wrote:
> John Snow writes:
>> Sort of.   In vfolder, the search criteria is stored to create a view 
>> that is a subset of an existing mailbox. In that case, it can easily 
>> be treated the same as any other mailbox.  My hope is to expand that 
>> beyond a single mailbox.
>
> Vfolder will do that by using Alexey's multimailbox search, at least 
> that's the vague intention.
>
>> I think creating a server stored view is unnecessary.  The extension 
>> aware client can create the combined view on the fly.
>
> FWIW, vfolder is the way it so so that unextended clients see the 
> data, too. The folder can be set up by an extended client or outside 
> IMAP and accessed by anything.
>
> Arnt
That's fine, but what happens if you try to copy into a  vfolder that 
was created using multimailbox search.  What underlying mailbox is the 
true destination?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070314/753a847c/attachment.html>

>>> Is there a way to identify a mailbox as read-only?  
>>>     
>>
>> >From RFC 3501:
>>
>>       If the client is not permitted to modify the mailbox but is
>>       permitted read access, the mailbox is selected as read-only, and
>>       the server MUST prefix the text of the tagged OK response to
>>       SELECT with the "[READ-ONLY]" response code.
>>
>>   
>
> Yep, forgot about that.  thanks.
>   
I take that back.  This response doesn't help with the copy problem.  
This read-only indicator is only available after the mailbox is 
selected, not when it's listed. 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070314/60eb7751/attachment.html>

John Snow writes:
> That's fine, but what happens if you try to copy into a  vfolder that 
> was created using multimailbox search. What underlying mailbox is the 
> true destination?

In my implementation it is not possible to grant 'i' or 'p' for virtual 
mailboxes, so the question doesn't arise.

We decided to revisit the issue if/when someone complains. Hasn't happened yet.

Arnt

John Snow wrote:
>
>
> arnt@gulbrandsen.priv.no wrote:
>> John Snow writes:
>>> Sort of.   In vfolder, the search criteria is stored to create a 
>>> view that is a subset of an existing mailbox. In that case, it can 
>>> easily be treated the same as any other mailbox.  My hope is to 
>>> expand that beyond a single mailbox.
>>
>> Vfolder will do that by using Alexey's multimailbox search, at least 
>> that's the vague intention.
>>


If we were to combine the multimailbox search with vfolder, what is 
expected to be used for UIDs in a view the includes multiple mailboxes?  
In the normal vfolder, the UID is the same one that exists in the 
underlying mailbox.  But since UIDs are only required to be unique 
within the mailbox, combining multiples and using the underlying UID for 
the message could result in duplicate UIDs.  So, that's not right. 

snow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20070322/9f6e2ddd/attachment.html>

> I take that back.  This response doesn't help with the copy problem.  
> This read-only indicator is only available after the mailbox is 
> selected, not when it's listed. 

I see what you mean.  It can only be sent for selected mailboxes, even
as a response code for BAD.

Bill

John Snow wrote:
> If we were to combine the multimailbox search with vfolder, what is 
> expected to be used for UIDs in a view the includes multiple 
> mailboxes?  In the normal vfolder, the UID is the same one that 
> exists in the underlying mailbox.  But since UIDs are only required 
> to be unique within the mailbox, combining multiples and using the 
> underlying UID for the message could result in duplicate UIDs.  So, 
> that's not right.
> snow.
UIDs are not simply unique, but strictly increasing.
In a singly-backed vfolder, the uids can be the same if and only if 
messages cannot dynamically appear other than bby delivery.
Dave.