mailing list archives

Hi,

As the subject states, is there actually any valid use case these days for
COPY to just copy messages instead of being a poor substitute for MOVE
(that is COPY+EXPUNGE)?

If an IMAP server would mark COPYied messages with \Delete and
expunge these immediately after a message has been copied, would it break
any real-use expectations?

Why I'm asking is that I'm building a database backed email server (
https://wildduck.email), we have a moderately sized cluster of emails
(100k+ users, ~50TB+ of data, few hundred million emails) and when an IMAP
client tries to copy all messages from one large folder to another then
copying takes a lot of time (eg 'COPY 1:* target' where * is 10 000) as
listing the database entries and copying these around takes time. And as
there is no response until messages have been fully copied the client might
think that TCP connection has been lost and retries the same action, ending
up doing multiple COPY calls.

So I was wondering if we could simply delete the already copied message
from the source folder, as most probably the client would do it anyway once
COPY is fully completed. Basically COPY would be an alias for MOVE.
Obviously non-standard behavior but would we actually break something
client side by doing this?

Regards,
Andris Reinman
https://wildduck.email
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20200909/7621c931/attachment.html>

On 09/09/2020 17:27, Andris Reinman wrote:
> Hi,
> 
> As the subject states, is there actually any valid use case these days 
> for COPY to just copy messages instead of being a poor substitute for 
> MOVE (that is COPY+EXPUNGE)?
> 
I often copy bunches of email to other folders, to other users accounts, 
and to various archive accounts, after which the messages are *not* 
deleted, so I would say, "Yes, there are valid use cases for COPY doing 
*exactly* what the word says."

	Cheers,
		Gary	B-)

What does the db data model look like?  Is the email data in the database, are there pointers?

> On Sep 9, 2020, at 3:28 AM, Andris Reinman <andris.reinman@gmail.com> wrote:
> 
> ?
> Hi,
> 
> As the subject states, is there actually any valid use case these days for COPY to just copy messages instead of being a poor substitute for MOVE (that is COPY+EXPUNGE)?
> 
> If an IMAP server would mark COPYied messages with \Delete and expunge these immediately after a message has been copied, would it break any real-use expectations?
> 
> Why I'm asking is that I'm building a database backed email server (https://wildduck.email), we have a moderately sized cluster of emails (100k+ users, ~50TB+ of data, few hundred million emails) and when an IMAP client tries to copy all messages from one large folder to another then copying takes a lot of time (eg 'COPY 1:* target' where * is 10 000) as listing the database entries and copying these around takes time. And as there is no response until messages have been fully copied the client might think that TCP connection has been lost and retries the same action, ending up doing multiple COPY calls.
> 
> So I was wondering if we could simply delete the already copied message from the source folder, as most probably the client would do it anyway once COPY is fully completed. Basically COPY would be an alias for MOVE. Obviously non-standard behavior but would we actually break something client side by doing this?
> 
> Regards,
> Andris Reinman
> https://wildduck.email
> 
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20200909/03a5adbf/attachment.html>

On 2020-09-09 01:27, Andris Reinman wrote:
> As the subject states, is there actually any valid use case these days 
> for COPY to just copy messages instead of being a poor substitute for 
> MOVE (that is COPY+EXPUNGE)?

The use case is copying a message, rather than moving it.

I admit many/most users just move messages and in fact some mail 
interfaces don't have an easy copy button, some omit it completely. If 
you control the interface and want to drop the copy button, I wouldn't 
be thrilled but I could accept it. But I believe you're talking about 
silently and unexpectedly deleting a user's email.

I will sometimes copy a bunch of messages into a temporary folder for a 
specific purpose. Perhaps they're related to something I am working on 
but not part of a single thread (or selected parts of a much larger 
thread), perhaps they're all requiring action/attention, maybe I'm going 
to share that folder. But I *always* want the original content sorted in 
the longterm place they live and I'll nuke that temporary folder once 
I'm done with it. And admittedly this is not something I do especially 
frequently either.

If my messages were silently discarded from their original folder, I 
would be both logging it as a defect and very quickly switching to a 
email service that doesn't lose email unexpectedly.

(Full disclosure: $DAYJOB has a product that only stores a single 
instance of a message within a mailbox -- But there is no risk because 
there is no copy action that fails silently, it simply doesn't exist, 
and nothing other than an obvious "X" delete button will delete a message).

I also use COPY to COPY and would be upset if the messages in my
source folder disappeared withou my asking them to :)

I think you'd be better off implementing https://tools.ietf.org/html/rfc6851
even though it relies on clients to notice and use the implementation.

On Wednesday, 09 September 2020 at 10:27, Andris Reinman wrote:
> Hi,
> 
> As the subject states, is there actually any valid use case these days for
> COPY to just copy messages instead of being a poor substitute for MOVE
> (that is COPY+EXPUNGE)?
> 
> If an IMAP server would mark COPYied messages with \Delete and
> expunge these immediately after a message has been copied, would it break
> any real-use expectations?
> 
> Why I'm asking is that I'm building a database backed email server (
> https://wildduck.email), we have a moderately sized cluster of emails
> (100k+ users, ~50TB+ of data, few hundred million emails) and when an IMAP
> client tries to copy all messages from one large folder to another then
> copying takes a lot of time (eg 'COPY 1:* target' where * is 10 000) as
> listing the database entries and copying these around takes time. And as
> there is no response until messages have been fully copied the client might
> think that TCP connection has been lost and retries the same action, ending
> up doing multiple COPY calls.
> 
> So I was wondering if we could simply delete the already copied message
> from the source folder, as most probably the client would do it anyway once
> COPY is fully completed. Basically COPY would be an alias for MOVE.
> Obviously non-standard behavior but would we actually break something
> client side by doing this?
> 
> Regards,
> Andris Reinman
> https://wildduck.email

> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

On 2020-09-09 09:49, Gary R. Schmidt wrote:
> On 09/09/2020 17:27, Andris Reinman wrote:
>> Hi,
>> 
>> As the subject states, is there actually any valid use case these days 
>> for COPY to just copy messages instead of being a poor substitute for 
>> MOVE (that is COPY+EXPUNGE)?
>> 
> I often copy bunches of email to other folders, to other users
> accounts, and to various archive accounts, after which the messages
> are *not* deleted, so I would say, "Yes, there are valid use cases for
> COPY doing *exactly* what the word says."

I frequently COPY mail which has been filtered into a folder on delivery 
back to my inbox if there is something odd that I need to investigate 
and/or action.

Roundcube has "drag and drop" for MOVE, and Shift+"drag and drop" for 
COPY, which is quite convenient of it.

> On Sep 9, 2020, at 1:49 AM, Gary R. Schmidt <grschmidt@acm.org> wrote:
> 
> On 09/09/2020 17:27, Andris Reinman wrote:
>> Hi,
>> As the subject states, is there actually any valid use case these days for COPY to just copy messages instead of being a poor substitute for MOVE (that is COPY+EXPUNGE)?
> I often copy bunches of email to other folders, to other users accounts, and to various archive accounts, after which the messages are *not* deleted, so I would say, "Yes, there are valid use cases for COPY doing *exactly* what the word says."

For the latter two examples, a copy to another account would almost certainly be an APPEND, and not actually a COPY.

But, I can't imagine that you wouldn't break someone's workflow almost immediately if you implemented this "COPY as alias for MOVE" behavior.

- Phillip
> 
> 	Cheers,
> 		Gary	B-)
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

> What does the db data model look like?  Is the email data in the database,
> are there pointers?
>

It is a sharded MongoDB cluster. Messages are parsed into structured
documents, attachments are decoded and deduplicated and stored separately.
All email documents are stored in the same large collection (currently at
250M documents) and include a "mailbox id" value that is used to filter out
messages that belong to the same IMAP mailbox.

Around 30M FETCH commands are run every day against that collection so it
is quite busy as well.

Doing a copy for messages means
1. getting a cursor for the result set of matching messages
2. processing the cursor one message at a time
3. reading the entire message entry from DB to application (without
attachments as these are stored separately)
4. modifying the "folder id" and UID, MODSEQ etc values of the record in
memory
5. inserting the record to the collection as a new document

Copying each message also triggers notifications about added messages to
IMAP clients, modseq updates for the mailbox etc.
So obviously this is not very fast and may take a lot of time (in computer
terms).

I wanted to use a different approach at first where there would be just a
single email document and that document would contain a list of mailbox ids
where it is currently stored (and also what are the UID/MODSEQ values in
these mailboxes). I was not able to figure out proper database indexing for
that so went with the more simplistic approach where every email document
is dedicated to a specific mailbox.

Regards,
Andris Reinman

Kontakt Aaron Burrow (<burrows.labs@gmail.com>) kirjutas kuup?eval K, 9.
september 2020 kell 22:45:

>
> On Sep 9, 2020, at 3:28 AM, Andris Reinman <andris.reinman@gmail.com>
> wrote:
>
> ?
> Hi,
>
> As the subject states, is there actually any valid use case these days for
> COPY to just copy messages instead of being a poor substitute for MOVE
> (that is COPY+EXPUNGE)?
>
> If an IMAP server would mark COPYied messages with \Delete and
> expunge these immediately after a message has been copied, would it break
> any real-use expectations?
>
> Why I'm asking is that I'm building a database backed email server (
> https://wildduck.email), we have a moderately sized cluster of emails
> (100k+ users, ~50TB+ of data, few hundred million emails) and when an IMAP
> client tries to copy all messages from one large folder to another then
> copying takes a lot of time (eg 'COPY 1:* target' where * is 10 000) as
> listing the database entries and copying these around takes time. And as
> there is no response until messages have been fully copied the client might
> think that TCP connection has been lost and retries the same action, ending
> up doing multiple COPY calls.
>
> So I was wondering if we could simply delete the already copied message
> from the source folder, as most probably the client would do it anyway once
> COPY is fully completed. Basically COPY would be an alias for MOVE.
> Obviously non-standard behavior but would we actually break something
> client side by doing this?
>
> Regards,
> Andris Reinman
> https://wildduck.email
>
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20200909/342aafb5/attachment.html>

I'll second what Dave says below.

99% of the time I want move semantics. But then there IS that 1% where I truly do want to make a copy of a message into a second folder, and it had better be a copy and the original had better not get lost from wherever else it might happen to live.

	Tony

?On 9/9/20, 10:46 PM, "Imap-protocol on behalf of Dave Warren" <imap-protocol-bounces@mailman13.u.washington.edu on behalf of dw@thedave.ca> wrote:

    On 2020-09-09 01:27, Andris Reinman wrote:
    > As the subject states, is there actually any valid use case these days 
    > for COPY to just copy messages instead of being a poor substitute for 
    > MOVE (that is COPY+EXPUNGE)?

    The use case is copying a message, rather than moving it.

    I admit many/most users just move messages and in fact some mail 
    interfaces don't have an easy copy button, some omit it completely. If 
    you control the interface and want to drop the copy button, I wouldn't 
    be thrilled but I could accept it. But I believe you're talking about 
    silently and unexpectedly deleting a user's email.

    I will sometimes copy a bunch of messages into a temporary folder for a 
    specific purpose. Perhaps they're related to something I am working on 
    but not part of a single thread (or selected parts of a much larger 
    thread), perhaps they're all requiring action/attention, maybe I'm going 
    to share that folder. But I *always* want the original content sorted in 
    the longterm place they live and I'll nuke that temporary folder once 
    I'm done with it. And admittedly this is not something I do especially 
    frequently either.

    If my messages were silently discarded from their original folder, I 
    would be both logging it as a defect and very quickly switching to a 
    email service that doesn't lose email unexpectedly.

    (Full disclosure: $DAYJOB has a product that only stores a single 
    instance of a message within a mailbox -- But there is no risk because 
    there is no copy action that fails silently, it simply doesn't exist, 
    and nothing other than an obvious "X" delete button will delete a message).

There is MOVE support but usage between COPY and MOVE is pretty even. For
example, I checked, and yesterday 3,500 distinct users issued 31,000 MOVE
commands and 3,300 distinct users issued 32,000 COPY commands, so it is
around 50%-50%. Users on the latest iOS Mail App use MOVE and users on
Outlook use COPY + UID STORE \Deleted.

Kontakt Brendan Cully (<brendan@kublai.com>) kirjutas kuup?eval N, 10.
september 2020 kell 09:08:

> I also use COPY to COPY and would be upset if the messages in my
> source folder disappeared withou my asking them to :)
>
> I think you'd be better off implementing
> https://tools.ietf.org/html/rfc6851
> even though it relies on clients to notice and use the implementation.
>
> On Wednesday, 09 September 2020 at 10:27, Andris Reinman wrote:
> > Hi,
> >
> > As the subject states, is there actually any valid use case these days
> for
> > COPY to just copy messages instead of being a poor substitute for MOVE
> > (that is COPY+EXPUNGE)?
> >
> > If an IMAP server would mark COPYied messages with \Delete and
> > expunge these immediately after a message has been copied, would it break
> > any real-use expectations?
> >
> > Why I'm asking is that I'm building a database backed email server (
> > https://wildduck.email), we have a moderately sized cluster of emails
> > (100k+ users, ~50TB+ of data, few hundred million emails) and when an
> IMAP
> > client tries to copy all messages from one large folder to another then
> > copying takes a lot of time (eg 'COPY 1:* target' where * is 10 000) as
> > listing the database entries and copying these around takes time. And as
> > there is no response until messages have been fully copied the client
> might
> > think that TCP connection has been lost and retries the same action,
> ending
> > up doing multiple COPY calls.
> >
> > So I was wondering if we could simply delete the already copied message
> > from the source folder, as most probably the client would do it anyway
> once
> > COPY is fully completed. Basically COPY would be an alias for MOVE.
> > Obviously non-standard behavior but would we actually break something
> > client side by doing this?
> >
> > Regards,
> > Andris Reinman
> > https://wildduck.email
>
> > _______________________________________________
> > Imap-protocol mailing list
> > Imap-protocol@u.washington.edu
> > http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20200910/597ba6ea/attachment.html>