mailing list archives

Copy a message with UID 1267 from Inbox to Mbox and deleted it in Inbox. 
With Inbox selected:

C: aaa UID COPY 1267 "Mbox"
S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
C: bbb UID store 1267 +Flags (\Deleted)
:

This works fine and message appears gone from Inbox and is now visible 
in Mbox at UID 1007. Now with Mbox selected, attempt to copy the message 
UID 1007 back to Inbox:

C: ccc UID COPY 1007 "Inbox"
S: ccc OK [COPYUID 987654321 1007 1267] UID COPY completed
C: ddd UID store 1007 +Flags (\Deleted)
:

This occurs with no error and the message appears gone from Mbox. 
However, it does not appear in Inbox. The problem is that the server 
doesn't use the UIDNEXT of Inbox, which is 1270. Instead it reuses and 
copies to the UID that the message originally had in Inbox, 1267. Also, 
the \deleted flag on 1267 remains set and does not get cleared after the 
copy, causing the message to remain not visible in Inbox.

Does this seem like acceptable imap server behavior?

Note: If Inbox is expunged before the message is copied back, the server 
copies to Inbox's UIDNEXT (1270) and the the message appears correctly 
in Inbox with \delete flag cleared.

Thanks,

-gene

On Sun, Oct 8, 2017 at 1:58 PM, Gene Smith <gds@chartertn.net> wrote:

> Copy a message with UID 1267 from Inbox to Mbox and deleted it in Inbox.
> With Inbox selected:
>
> C: aaa UID COPY 1267 "Mbox"
> S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
> C: bbb UID store 1267 +Flags (\Deleted)
>


> C: ccc UID COPY 1007 "Inbox"
> S: ccc OK [COPYUID 987654321 1007 1267] UID COPY completed
> C: ddd UID store 1007 +Flags (\Deleted)
>


> Does this seem like acceptable imap server behavior?
>

No, this is a violation. Both messages have UID 1267, and if there is no
intervening EXPUNGE, the first message still exists when the second one
overwrites it. Neither behavior is remotely permissible.

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20171008/2e02a247/attachment.html>

On 10/8/17 5:43 PM, Tim Showalter wrote:
> On Sun, Oct 8, 2017 at 1:58 PM, Gene Smith <gds@chartertn.net 
> <mailto:gds@chartertn.net>> wrote:
> 
>     Copy a message with UID 1267 from Inbox to Mbox and deleted it in
>     Inbox. With Inbox selected:
> 
>     C: aaa UID COPY 1267 "Mbox"
>     S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
>     C: bbb UID store 1267 +Flags (\Deleted)
> 
>     C: ccc UID COPY 1007 "Inbox"
>     S: ccc OK [COPYUID 987654321 1007 1267] UID COPY completed
>     C: ddd UID store 1007 +Flags (\Deleted)
> 
>     Does this seem like acceptable imap server behavior?
> 
> 
> No, this is a violation. Both messages have UID 1267, and if there is no 
> intervening EXPUNGE, the first message still exists when the second one 
> overwrites it. Neither behavior is remotely permissible.
> 
> Tim
> 

I think the server is thinking, "its the same message I am copying back 
to Inbox (how does it knows? Same Message-ID or maybe checksum?) so I 
don't have to do anything." If the server had at least reset the 
\deleted flag on message at UID 1267 in Inbox I think it would be OK. 
The rfc says COPY must preserve the flags. It also says messages added 
to a mailbox must have UID >= UIDNEXT. But if the server is copying in a 
message that is already present but just flagged as \deleted, maybe RFC 
doesn't say you can't COPYUID to the original UID which is < UIDNEXT? 
But I think the server should reset \deleted at the destination.

FYI, the server is a product called "openwave" used by a fairly large 
ISP (charter/spectrum).

While working on thunderbird bug 1408610 (*) I attempted to do an 
"authenticate LOGIN" to the imap.mail.yahoo.com server with unsuccessful 
results (tb = thunderbird, yh = yahoo):

tb:  2 authenticate LOGIN
yh:  +                     <-- no Username: "challenge" sent here.
tb:   <base64 encoded uid>
yh:  + <base64 encoded string saying auth string is bad>
tb:  <base64 encoded password>
yh:  2 BAD [AUTHENTICATIONFAILED] LOGIN Invalid credentials

I get this result when sent by tb or when sent manually using openssl. I 
have tried several variants to see if it would work such as sending 
separate base64 encoded and spaced separated uid and pwd right after the 
first yh + response. Also tried base64 encoded "\0<uid>\0<pwd>", that 
works for PLAIN, after yh + response and others variations to no avail.

Authenticate PLAIN and simple imap login with normal ascii uid and pwd 
text work OK. Noticed that few imap servers even support "authenticate 
LOGIN". One that I did find that supports it, after providing the uid 
and pwd, reports RENEGOTIATING and then openssl crashes.

Only official documentation I could find on "authenticate login" is
https://tools.ietf.org/html/draft-murchison-sasl-login-00 but it only 
provides an example for smtp.

-gene
P/S: The yahoo server reports a capability AUTH=LOGIN
(*) https://bugzilla.mozilla.org/show_bug.cgi?id=1408610

On Sun, Oct 8, 2017 at 3:26 PM, Gene Smith <gds@chartertn.net> wrote:
>
>     C: aaa UID COPY 1267 "Mbox"
>>     S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
>>     C: bbb UID store 1267 +Flags (\Deleted)
>>
>>     C: ccc UID COPY 1007 "Inbox"
>>     S: ccc OK [COPYUID 987654321 1007 1267] UID COPY completed
>>     C: ddd UID store 1007 +Flags (\Deleted)
>
>
I think the server is thinking, "its the same message I am copying back to
> Inbox (how does it knows? Same Message-ID or maybe checksum?) so I don't
> have to do anything." If the server had at least reset the \deleted flag on
> message at UID 1267 in Inbox I think it would be OK. The rfc says COPY must
> preserve the flags. It also says messages added to a mailbox must have UID
> >= UIDNEXT. But if the server is copying in a message that is already
> present but just flagged as \deleted, maybe RFC doesn't say you can't
> COPYUID to the original UID which is < UIDNEXT? But I think the server
> should reset \deleted at the destination.
>

Recovering a used UID is never permitted*. If there is no EXPUNGE, there
should be two copies of the message in Inbox in this example. (Setting
\Deleted is reversible.)

The design of the specification is concerned with what happens when a
separate client connects before and after this exchange. Because UIDNEXT
has not been used correctly, clients that have cached state wouldn't see
changes to the mailbox correctly.

I guess the server might have an optimization (aka hack) for this, but it
doesn't appear to be properly protocol compliant.

* unless the UIDVALIDITY is reset, which trashes everyone's caches anyway.

Tim
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20171008/831bdddd/attachment.html>

On 10/8/17 9:40 PM, Tim Showalter wrote:
> On Sun, Oct 8, 2017 at 3:26 PM, Gene Smith <gds@chartertn.net 
> <mailto:gds@chartertn.net>>?wrote:
> 

Tim,
Thanks for your help.

> Recovering a used UID is never permitted*.?If there is no EXPUNGE, there 
> should be two copies of the message in Inbox in this example. (Setting 
> \Deleted is reversible.)

Yes, for other IMAP servers I have tested, if I copy message "A" 10 
times into Inbox I see 10 identical copies of message A in Inbox. For 
this openwave server, I see only 1 (and uid fetch 1:* (FLAGS) shows only 1).

> 
> The design of the specification is concerned with what happens when a 
> separate client connects before and after this exchange. Because UIDNEXT 
> has not been used correctly, clients that have cached state wouldn't see 
> changes to the mailbox correctly.

If the server would clear the \deleted flag at the destination uid and 
did no actual copy in response to this uid copy it would produce an 
acceptable result (message visible in client again). This would be no 
different than changing flags with uid store it would seem (e.g., what 
happen when a deleted message is undeleted with crtl-z in client). Not 
sure how other/separate clients would be confused by this.

> 
> I guess the server might have an optimization (aka hack) for this, but 
> it doesn't appear to be properly protocol compliant.

So in summary, you are saying that the server should always use the 
UIDNEXT or greater when messages are copied to a mbox, even if the 
server determines there is an identical message at a lower UID. To do 
otherwise violates rfc 3501.

-gene

Gene Smith writes:
> Yes, for other IMAP servers I have tested, if I copy message 
> "A" 10 times into Inbox I see 10 identical copies of message A 
> in Inbox. For this openwave server, I see only 1 (and uid fetch 
> 1:* (FLAGS) shows only 1).

Have you tested with gmail?

The server doesn't seem to reuse a UID, if I understand you correctly. The 
message has a UID before the COPY command and nothing changes. The UID is 
in continuous use for a particular message, so there's no reuse going on, 
and the ban on reuse does not apply.

Can you find out whether flags apply to the message or to references? Ie. 
if you copy a message into two mailboxes and set flag "foobar" on the 
message in one mailbox, does the flag appear on the message in the other 
mailbox?

Arnt

On 10/9/17 10:39 AM, Arnt Gulbrandsen wrote:
> Gene Smith writes:
>> Yes, for other IMAP servers I have tested, if I copy message "A" 10 
>> times into Inbox I see 10 identical copies of message A in Inbox. For 
>> this openwave server, I see only 1 (and uid fetch 1:* (FLAGS) shows 
>> only 1).
> 
> Have you tested with gmail?

Just tried gmail and and if I copy a message multiple times there 
remains only one copy of the message in the destination folder. I 
haven't looked at the imap transactions to to see what is actually 
happening.

With imap.outlook.com it does allow multiple duplicate messages to be 
copied in so I assume it is assigning new and higher UID to each duplicate.

imap.yahoo seems different. When I copy in a duplicate, at the 
destination folder that message seems to disappear in my client for 
about 1/2 second or so and then come back. I did look at the imap 
transaction for yahoo to see what is happening. Yahoo is assigning a new 
and higher UID each time the duplicate message is copied to a 
destination but it seems to be silently expunging the previous UID for 
the message. I see nothing being "stored", such as \deleted, to the 
message's previous UID and I see no expunge responses. As multiple 
duplicate copies occur, the gap between the highest UID and the next to 
highest UID grows (based on uid fetch 1, *).

Zimbra and mDaemon allow multiple duplicates to be copied in like 
outlook. Haven't looked at transaction details.

> 
> The server doesn't seem to reuse a UID, if I understand you correctly. 
> The message has a UID before the COPY command and nothing changes. The 
> UID is in continuous use for a particular message, so there's no reuse 
> going on, and the ban on reuse does not apply.

Yes, in the mbox where the message was deleted, its UID is still present 
but with the \deleted flag set. The same message but with a different 
UID is in Trash. When I try to copy it back to the original mbox from 
Trash, the server indicates that it is copying it to the original UID 
(not >= UIDNEXT). The main problem I see is that this copy doesn't clear 
the \deleted flag for the UID in the original folder so the message 
remain invisible in the client.

> 
> Can you find out whether flags apply to the message or to references? 
> Ie. if you copy a message into two mailboxes and set flag "foobar" on 
> the message in one mailbox, does the flag appear on the message in the 
> other mailbox?

Actually, this is what happens when I delete a message from openwave 
server. In the original folder the message's UID gets marked with 
\deleted flag and the message is not expunged. In the Trash folder where 
the client copies it on delete, the \deleted flag is *not* set. So the 
exact same message resides in two folders with differing flags.

As an additional test on openwave, I copied a message from Inbox to two 
different folders. In one folder I set several attributes (Important, 
Starred, Junk, Unread). In the other folder these flags don't appear for 
the exact same message. I then copied the this highly-flagged message 
back to Inbox attempting to overwrite the original. These attributes did 
not appear in the original message in Inbox.

> 
> Arnt
> 
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

On Mon, Oct 9, 2017 at 7:39 AM, Arnt Gulbrandsen <arnt@gulbrandsen.priv.no>
wrote:

> Gene Smith writes:
>
>> Yes, for other IMAP servers I have tested, if I copy message "A" 10 times
>> into Inbox I see 10 identical copies of message A in Inbox. For this
>> openwave server, I see only 1 (and uid fetch 1:* (FLAGS) shows only 1).
>>
>
> Have you tested with gmail?


Since Gmail considers a COPY operation as an "add label" operation, it's
usually a no-op if the message already has that label, which is what would
happen in this case.

There are cases where Gmail does "reassign UID" on a COPY, ie it bumps the
UID of a message to the UIDNEXT.  It does this if the user is calling COPY
with a source/target folder that are the same, on the assumption that the
user is trying to make an actual copy of the message in the same folder.
Clients doing this tend to be confused if the message isn't the latest UID
in that case.  We also reassign UIDs when messages are moved to/from the
spam/trash folders, for similar reasons.  (Note that APPEND always bumps
the UID when it detects an attempt to APPEND a duplicate message for
similar reasons to the COPY to the same folder)

Most IMAP flags are also implemented as labels, and so are consistent
across all "copies" of a message.  The exception is \Deleted, so doing
something like a COPY/Deleted as a MOVE operation without an EXPUNGE and
then doing it back will leave both "copies" with a \Deleted flag and both
will be removed with EXPUNGE.  That said, the default is for the user to be
in auto-expunge mode, in which case storing a \Deleted flag will result in
the message being EXPUNGED (that label removed) at the next sync point.

IE, from the given example:
C: aaa UID COPY 1267 "Mbox"
S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
C: bbb UID store 1267 +Flags (\Deleted)

The UID store is a sync point, so the message will be auto-expunged at this
point, so the COPY back will get a new UID.

You'd need to change the user using the advanced IMAP options to a
different deletion mode to see the UID maintained.

Also, Gmail offers the MOVE extension, so you should really use that
instead of COPY\Deleted\EXPUNGE.

Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20171009/ea8f1474/attachment.html>

On 2017-10-09 14:41, Gene Smith wrote:
> Yes, in the mbox where the message was deleted, its UID is still present 
> but with the \deleted flag set. The same message but with a different 
> UID is in Trash. When I try to copy it back to the original mbox from 
> Trash, the server indicates that it is copying it to the original UID 
> (not >= UIDNEXT). The main problem I see is that this copy doesn't clear 
> the \deleted flag for the UID in the original folder so the message 
> remain invisible in the client.

I think it might reduce confusion if we stop saying "Message was 
deleted". "Deleted" doesn't have a clear definition in this context 
because it isn't really a thing in IMAP (although it is how IMAP clients 
present multiple different behaviours to users). At the protocol level, 
you can set the \Deleted flag or you can EXPUNGE.

Since a \Deleted flag can be trivially removed, there is no "reuse" of a 
UID, the UID was still being reused and the COPY was just returning an 
accurate (if unusual) result. Expunging and then reusing the UID 
(assuming an unchanged UIDVALIDITY) would be a very significant problem 
with regards to clients maintaining the validity of their caches, but 
simply changing a flag is perfectly valid.

On 10/9/17 7:30 PM, Brandon Long wrote:
> 
> 
> On Mon, Oct 9, 2017 at 7:39 AM, Arnt Gulbrandsen 
> <arnt@gulbrandsen.priv.no <mailto:arnt@gulbrandsen.priv.no>> wrote:
> 
>     Gene Smith writes:
> 
>         Yes, for other IMAP servers I have tested, if I copy message "A"
>         10 times into Inbox I see 10 identical copies of message A in
>         Inbox. For this openwave server, I see only 1 (and uid fetch 1:*
>         (FLAGS) shows only 1).
> 
> 
>     Have you tested with gmail?
> 
> 
> Since Gmail considers a COPY operation as an "add label" operation, it's 
> usually a no-op if the message already has that label, which is what 
> would happen in this case.
> 
> There are cases where Gmail does "reassign UID" on a COPY, ie it bumps 
> the UID of a message to the UIDNEXT.? It does this if the user is 
> calling COPY with a source/target folder that are the same, on the 
> assumption that the user is trying to make an actual copy of the message 
> in the same folder.? Clients doing this tend to be confused if the 
> message isn't the latest UID in that case.? We also reassign UIDs when 
> messages are moved to/from the spam/trash folders, for similar reasons.  
> (Note that APPEND always bumps the UID when it detects an attempt to 
> APPEND a duplicate message for similar reasons to the COPY to the same 
> folder)
> 
> Most IMAP flags are also implemented as labels, and so are consistent 
> across all "copies" of a message.? The exception is \Deleted, so doing 
> something like a COPY/Deleted as a MOVE operation without an EXPUNGE and 
> then doing it back will leave both "copies" with a \Deleted flag and 

To simulate a MOVE, wouldn't you COPY the message to a destination 
folder and then set the \Deleted on the source message/folder. So the 
\deleted flag is not on both copies? Then you would uid expunge the 
source message.

> both will be removed with EXPUNGE.? That said, the default is for the 
> user to be in auto-expunge mode, in which case storing a \Deleted flag 
> will result in the message being EXPUNGED (that label removed) at the 
> next sync point.
> 
> IE, from the given example:
> C: aaa UID COPY 1267 "Mbox"
> S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
> C: bbb UID store 1267 +Flags (\Deleted)
> 
> The UID store is a sync point, so the message will be auto-expunged at 
> this point, so the COPY back will get a new UID.

Thanks for the info on gmail. One thing I didn't understand is why gmail 
has the "auto-expunge on \Deleted" default feature. You have provide a 
rationale for it. But it seems to be a violation of rfc 3501. It also 
causes problem for clients that use the "Just mark it as deleted" delete 
method that just sets the \Deleted flag and leaves a deleted email 
summary in place but indicates it is deleted by drawing a line though 
the summary. With default gmail, the crossed-out message vanishes from 
the summary list since it auto-expunged. This can be fixed in the gmail 
imap settings, fortunately. The outlook imap server also does the 
auto-expunge but doesn't seem to have a way to disable it. I haven't yet 
seen other imap servers that do auto-expunge.

> 
> You'd need to change the user using the advanced IMAP options to a 
> different deletion mode to see the UID maintained.

Advanced IMAP option == not auto-expunge? I think that's what you mean.

OK, if I do this with auto-expunge set to false:
Copy message "test" (uid 5604) from Inbox to [Gmail]/f1
Message test is visible in Inbox and f1. Not crossed-out.
In Inbox, mark test with \deleted. It becomes crossed-out.
In f1 copy test to Inbox. Goes to uid 5604 in Inbox.
test in Inbox keeps \Deleted flag and remain crossed-out.

So gmail imap behaving same as openwave imap! It copies back to the 
original UID and doesn't clear the \Deleted flag on UID 5604 in Inbox 
and message remains crossed-out in client.

Why doesn't \Deleted on UID 5604 in Inbox get cleared after copy back?

> 
> Also, Gmail offers the MOVE extension, so you should really use that 
> instead of COPY\Deleted\EXPUNGE.

Yes, no problem when MOVE used since it expunges the message after it 
copies it to the destination. That's typically what is used.

> 
> Brandon

On 10/9/17 5:32 PM, Dave Warren wrote:
> 
> I think it might reduce confusion if we stop saying "Message was 
> deleted". "Deleted" doesn't have a clear definition in this context 
> because it isn't really a thing in IMAP (although it is how IMAP clients 
> present multiple different behaviours to users). At the protocol level, 
> you can set the \Deleted flag or you can EXPUNGE.

OK, s/Message was deleted/Message was deleted in the client program/  :)

> 
> Since a \Deleted flag can be trivially removed, there is no "reuse" of a 
> UID, the UID was still being reused and the COPY was just returning an 
> accurate (if unusual) result. Expunging and then reusing the UID 
> (assuming an unchanged UIDVALIDITY) would be a very significant problem 
> with regards to clients maintaining the validity of their caches, but 
> simply changing a flag is perfectly valid.

Openwave has no problem if the copy destination mbox is first expunged 
or if the original UID now having the \deleted flag is expunged with 
"uid expunge <uid>" since openwave has UIDPLUS capability. In that case 
the server copies to a new UID >= UIDNEXT with \deleted not set so the 
copied-in message is visible in the client program.

So I think if there is a bug in the openwave server it because it 
somewhat violates this requirement for the COPY command from rfc 3501:

"The COPY command copies the specified message(s) to the end of the 
specified destination mailbox.  The flags and internal date of the
message(s) SHOULD be preserved, and the Recent flag SHOULD be set,
in the copy."

1.The copy does not always copy messages to the "end" of specified 
destination mbox. (I don't see an explicit definition for "end" in the 
rfc but I am sure it means at UID >=UIDNEXT or at the next greater 
sequence number.)

2.When it does a copy to an existing and unexpunged UID not at the "end" 
it does not preserve the flag states from the source mbox/UID. Also, the 
Recent flag is not set. Of course, this is a SHOULD requirement so maybe 
there are valid reasons for not doing these that I'm not aware of.

-gene








> 
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

On Oct 12, 2017 9:46 PM, "Gene Smith" <gds@chartertn.net> wrote:

On 10/9/17 7:30 PM, Brandon Long wrote:


>
> On Mon, Oct 9, 2017 at 7:39 AM, Arnt Gulbrandsen <arnt@gulbrandsen.priv.no
> <mailto:arnt@gulbrandsen.priv.no>> wrote:
>
>     Gene Smith writes:
>
>         Yes, for other IMAP servers I have tested, if I copy message "A"
>         10 times into Inbox I see 10 identical copies of message A in
>         Inbox. For this openwave server, I see only 1 (and uid fetch 1:*
>         (FLAGS) shows only 1).
>
>
>     Have you tested with gmail?
>
>
> Since Gmail considers a COPY operation as an "add label" operation, it's
> usually a no-op if the message already has that label, which is what would
> happen in this case.
>
> There are cases where Gmail does "reassign UID" on a COPY, ie it bumps the
> UID of a message to the UIDNEXT.  It does this if the user is calling COPY
> with a source/target folder that are the same, on the assumption that the
> user is trying to make an actual copy of the message in the same folder.
> Clients doing this tend to be confused if the message isn't the latest UID
> in that case.  We also reassign UIDs when messages are moved to/from the
> spam/trash folders, for similar reasons.  (Note that APPEND always bumps
> the UID when it detects an attempt to APPEND a duplicate message for
> similar reasons to the COPY to the same folder)
>
> Most IMAP flags are also implemented as labels, and so are consistent
> across all "copies" of a message.  The exception is \Deleted, so doing
> something like a COPY/Deleted as a MOVE operation without an EXPUNGE and
> then doing it back will leave both "copies" with a \Deleted flag and
>

To simulate a MOVE, wouldn't you COPY the message to a destination folder
and then set the \Deleted on the source message/folder. So the \deleted
flag is not on both copies? Then you would uid expunge the source message.


both will be removed with EXPUNGE.  That said, the default is for the user
> to be in auto-expunge mode, in which case storing a \Deleted flag will
> result in the message being EXPUNGED (that label removed) at the next sync
> point.
>
> IE, from the given example:
> C: aaa UID COPY 1267 "Mbox"
> S: aaa OK [COPYUID 123456789 1267 1007] UID COPY completed
> C: bbb UID store 1267 +Flags (\Deleted)
>
> The UID store is a sync point, so the message will be auto-expunged at
> this point, so the COPY back will get a new UID.
>

Thanks for the info on gmail. One thing I didn't understand is why gmail
has the "auto-expunge on \Deleted" default feature. You have provide a
rationale for it. But it seems to be a violation of rfc 3501. It also
causes problem for clients that use the "Just mark it as deleted" delete
method that just sets the \Deleted flag and leaves a deleted email summary
in place but indicates it is deleted by drawing a line though the summary.
With default gmail, the crossed-out message vanishes from the summary list
since it auto-expunged. This can be fixed in the gmail imap settings,
fortunately. The outlook imap server also does the auto-expunge but doesn't
seem to have a way to disable it. I haven't yet seen other imap servers
that do auto-expunge.


What is the utility of the leave in folder marked as deleted?

Most clients moved to a trash folder concept, which doesn't require the
leave deleted state.

Gmail added auto-expunge as soon as we started testing imap, since people
would delete messages... And they'd still be there in the web interface.
We had no interest in adding the weird deleted still there mode to the
Gmail client.  We debated a couple different modes, and they basically
wound up as advanced settings, but just expunging made the most sense,
especially since most mobile clients at the time never called expunge, and
the main use case of imap for Gmail was for mobile clients.

It is not a violation of RFC 3501, since another client could be connected
and issue an expunge, as long as it's only visible at a sync point.

It did cause the occasional issue with clients that couldn't handle it in
all cases, but those were all client bugs.


> You'd need to change the user using the advanced IMAP options to a
> different deletion mode to see the UID maintained.
>

Advanced IMAP option == not auto-expunge? I think that's what you mean.

OK, if I do this with auto-expunge set to false:
Copy message "test" (uid 5604) from Inbox to [Gmail]/f1
Message test is visible in Inbox and f1. Not crossed-out.
In Inbox, mark test with \deleted. It becomes crossed-out.
In f1 copy test to Inbox. Goes to uid 5604 in Inbox.
test in Inbox keeps \Deleted flag and remain crossed-out.

So gmail imap behaving same as openwave imap! It copies back to the
original UID and doesn't clear the \Deleted flag on UID 5604 in Inbox and
message remains crossed-out in client.

Why doesn't \Deleted on UID 5604 in Inbox get cleared after copy back?


What is the proper thing to do?  We're mixing two mailbox models which
aren't entirely compatible.

I agree that clearing the flag is probably the right choice.  Most likely,
this is a fairly unlikely scenario that only occurs with a really small
number of clients when a very unused setting is enabled, so no one ran into
it and made a decision.  I could probably file a bug for it.

Brandon


> Also, Gmail offers the MOVE extension, so you should really use that
> instead of COPY\Deleted\EXPUNGE.
>

Yes, no problem when MOVE used since it expunges the message after it
copies it to the destination. That's typically what is used.


> Brandon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20171013/00ca0530/attachment.html>

On 10/13/17 3:19 AM, Brandon Long wrote:
> 
> 
> What is the utility of the leave in folder marked as deleted?
> 
> Most clients moved to a trash folder concept, which doesn't require the 
> leave deleted state.

Well, I prefer the move to trash concept but some Thunderbird users 
still prefer the "Just mark it as deleted" non-default mode where a 
Trash folder is not involved and deleted messages just get crossed out 
in the UI after the \Deleted flag is set.

> 
> Gmail added auto-expunge as soon as we started testing imap, since 
> people would delete messages... And they'd still be there in the web 
> interface.? We had no interest in adding the weird deleted still there 
> mode to the Gmail client.? We debated a couple different modes, and they 
> basically wound up as advanced settings, but just expunging made the 
> most sense, especially since most mobile clients at the time never 
> called expunge, and the main use case of imap for Gmail was for mobile 
> clients.
> 
> It is not a violation of RFC 3501, since another client could be 
> connected and issue an expunge, as long as it's only visible at a sync 
> point.

You're right, I looked at the rfc again and nowhere does it say an 
auto-expunge can't occur; but, it never really mentions the concept.

> 
> It did cause the occasional issue with clients that couldn't handle it 
> in all cases, but those were all client bugs.
> 

Since you can turn off auto-expunge at gmail it works around the problem 
in Thunderbird when using "just mark deleted" mode.

> 
>     OK, if I do this with auto-expunge set to false:
>     Copy message "test" (uid 5604) from Inbox to [Gmail]/f1
>     Message test is visible in Inbox and f1. Not crossed-out.
>     In Inbox, mark test with \deleted. It becomes crossed-out.
>     In f1 copy test to Inbox. Goes to uid 5604 in Inbox.
>     test in Inbox keeps \Deleted flag and remain crossed-out.
> 
>     So gmail imap behaving same as openwave imap! It copies back to the
>     original UID and doesn't clear the \Deleted flag on UID 5604 in
>     Inbox and message remains crossed-out in client.
> 
>     Why doesn't \Deleted on UID 5604 in Inbox get cleared after copy back?
> 
> 
> What is the proper thing to do?? We're mixing two mailbox models which 
> aren't entirely compatible.
> 
> I agree that clearing the flag is probably the right choice.? Most 
> likely, this is a fairly unlikely scenario that only occurs with a 
> really small number of clients when a very unused setting is enabled, so 
> no one ran into it and made a decision.? I could probably file a bug for it.
> 

This is also what I observed for the openwave server that I thought was 
a bug. So it sounds like you agree a copy back to the same UID should 
result in the same flag states at the destination that are present at 
the source.

On Oct 13, 2017 5:40 PM, "Gene Smith" <gds@chartertn.net> wrote:

On 10/13/17 3:19 AM, Brandon Long wrote:

>
>
> What is the utility of the leave in folder marked as deleted?
>
> Most clients moved to a trash folder concept, which doesn't require the
> leave deleted state.
>

Well, I prefer the move to trash concept but some Thunderbird users still
prefer the "Just mark it as deleted" non-default mode where a Trash folder
is not involved and deleted messages just get crossed out in the UI after
the \Deleted flag is set.



> Gmail added auto-expunge as soon as we started testing imap, since people
> would delete messages... And they'd still be there in the web interface.
> We had no interest in adding the weird deleted still there mode to the
> Gmail client.  We debated a couple different modes, and they basically
> wound up as advanced settings, but just expunging made the most sense,
> especially since most mobile clients at the time never called expunge, and
> the main use case of imap for Gmail was for mobile clients.
>
> It is not a violation of RFC 3501, since another client could be connected
> and issue an expunge, as long as it's only visible at a sync point.
>

You're right, I looked at the rfc again and nowhere does it say an
auto-expunge can't occur; but, it never really mentions the concept.



> It did cause the occasional issue with clients that couldn't handle it in
> all cases, but those were all client bugs.
>
>
Since you can turn off auto-expunge at gmail it works around the problem in
Thunderbird when using "just mark deleted" mode.



>     OK, if I do this with auto-expunge set to false:
>     Copy message "test" (uid 5604) from Inbox to [Gmail]/f1
>     Message test is visible in Inbox and f1. Not crossed-out.
>     In Inbox, mark test with \deleted. It becomes crossed-out.
>     In f1 copy test to Inbox. Goes to uid 5604 in Inbox.
>     test in Inbox keeps \Deleted flag and remain crossed-out.
>
>     So gmail imap behaving same as openwave imap! It copies back to the
>     original UID and doesn't clear the \Deleted flag on UID 5604 in
>     Inbox and message remains crossed-out in client.
>
>     Why doesn't \Deleted on UID 5604 in Inbox get cleared after copy back?
>
>
> What is the proper thing to do?  We're mixing two mailbox models which
> aren't entirely compatible.
>
> I agree that clearing the flag is probably the right choice.  Most likely,
> this is a fairly unlikely scenario that only occurs with a really small
> number of clients when a very unused setting is enabled, so no one ran into
> it and made a decision.  I could probably file a bug for it.
>
>
This is also what I observed for the openwave server that I thought was a
bug. So it sounds like you agree a copy back to the same UID should result
in the same flag states at the destination that are present at the source.


I think that severs could implement either, the usage isn't write to spec
or quite against.  That said, leaving \Deleted can result in the unexpected
behavior of message loss, which should be avoided, so the user friendly
choice is to not leave \Deleted in that case.

Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20171013/9aa917b0/attachment.html>

I have been researching the CONDSTORE feature in Thunderbird that is 
currently disabled by default due to some unresolved bugs. Several bug 
reports mention the QRESYNC extension to CONDSTORE as something that 
would be helpful but is not supported in Thunderbird at all. One reason 
listed is because gmail imap didn't support it and still doesn't the 
last time I checked. Pending support for QRESYNC by gmail was last 
mentioned several years ago on this list. Does anyone know the current 
status of QRESYNC support in gmail imap, i.e., are there still plans to 
support it and, if so, when?

Thanks,
Gene Smith

On 10/13/17 9:13 PM, Brandon Long wrote:
> 
> > I think that severs could implement either, the usage isn't write to 
> spec or quite against.? That said, leaving \Deleted can result in the 
> unexpected behavior of message loss, which should be avoided, so the 
> user friendly choice is to not leave \Deleted in that case.

Yeah, you're right again. The rfc only say copy SHOULD preserve the 
flags and date and SHOULD set the Recent flag. Not mandatory.

Again, thanks for the info.

-gene

I don't think there are any plans to support qresync on Gmail at this time.

Brandon

On Fri, Sep 7, 2018, 8:26 PM Gene Smith <gds@chartertn.net> wrote:

> I have been researching the CONDSTORE feature in Thunderbird that is
> currently disabled by default due to some unresolved bugs. Several bug
> reports mention the QRESYNC extension to CONDSTORE as something that
> would be helpful but is not supported in Thunderbird at all. One reason
> listed is because gmail imap didn't support it and still doesn't the
> last time I checked. Pending support for QRESYNC by gmail was last
> mentioned several years ago on this list. Does anyone know the current
> status of QRESYNC support in gmail imap, i.e., are there still plans to
> support it and, if so, when?
>
> Thanks,
> Gene Smith
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20180908/8b7a5c24/attachment.html>