mailing list archives

Hi all,

First time poster.  Quick background: I am the primary author of IMP  
(http://www.horde.org/imp/) and the related Horde IMAP client library.  
  Had a discussion a few weeks back with Timo on the dovecot list  
about some keyword issues, and he suggested I raise the topics on this  
list.

That topic is handling of keywords.  More specifically, it can be  
broken down into several parts (apologizes if I make the following  
discussion too basic - just want to make sure I am on the same page):

1. Allowing non-ASCII characters in keywords
2. Creating a method of sharing keyword->MUA (or user) defined labels  
between user agents

Starting with topic #1: We are limited by 3501 to atoms for keywords.   
Thus, no non-ASCII characters, or even spaces, which limits the  
practical usage of keywords as labels in and of themselves.  For  
example, a user may want to use a keyword to label messages in a  
folder as important for tax purposes and wants to use the label "Tax  
Forms (2011)".  Obviously, that label won't work [contains SP, (, and  
) as invalid atom characters].

MUA's can handle this issue in a variety of ways.  A "dumb" MUA must  
just tell the user they can't use that label.  A slightly smarter MUA  
might instead strip out all non-allowable atoms which generates the  
TaxForms2011 keyword.  While this works, it is not the most desirable  
response.  However, for the latter sort of MUAs, this sort of behavior  
falls apart when the label contains some sort of non-ASCII character  
(say "Quarant?ne").  Further, other MUAs may simply ignore the keyword  
if they don't know what it is.

A smart MUA would instead pick an arbitrary keyword, use this keyword  
on the server, and keep a copy of the user's preferred label locally  
to do the necessary translation.  This approach works well - as long  
as a user is only accessing that IMAP mail store with that MUA.  If  
using multiple MUA's, this approach breaks down because the other MUAs  
don't know the keyword -> label associations.  (The MUA's arbitrary  
keyword might be something very similar to the desired label - e.g.  
TaxForms2011 - to make things somewhat usable on other MUAs.  But  
again, not the most desirable situation).

Further, a smart MUA is faced with the problem that the keyword picked  
in one mailbox may not be available in all other mailboxes.  Although  
rare, it is entirely conceivable that collisions could occur across  
mailboxes.  E.g. MUA #1 creates a 'Label1' keyword in INBOX - the user  
associates this keyword with 'Foo'.  MUA #2 creates a 'Label1' keyword  
in 'OtherMailbox' - the user associates this keyword with 'Bar'.   
However, once MUA #2 accesses INBOX, it can no longer universally use  
'Label1' across all mailboxes.  It instead will need to do some  
excessive processing (e.g. scanning mailboxes accessed for the first  
time for all known local keywords, and keeping a potentially  
complicated lookup table locally) to make sure these keywords don't  
conflict.

That defines the problem.  Possible solutions?

1. Implement UTF7-IMAP (or similar) storage of keywords.
Pros: Would allow keyword & label information to be kept in a single  
location; all MUAs should be able to easily adapt since they should  
already have code necessary to do UTF7-IMAP conversions
Cons: Haven't thought about it fully, but doesn't seem like it could  
be made to be backward compatible with RFC 3501 keywords.  Doesn't  
solve problem of having one keyword meaning the same thing across all  
mailboxes.

2. Define a server METADATA entry (RFC 5464) to store keyword->label mapping
Pros: Works with old RFC 3501 servers.  Leverages an existing IMAP extension
Cons: Doesn't solve problem of having one keyword meaning the same  
thing across all mailboxes.  Doesn't really seem to be the kind of  
thing that should necessarily be a metadata entry - flags/keywords are  
a core part of the IMAP spec, so to me it just "feels" like something  
else should be used instead (really useful description, I know).

3. New commands/response codes
Pros: Works with old RFC 3501 servers.  Would be able to handle issue  
of one keyword meaning the same thing across all mailboxes.
Cons: As with all extensions, adds complexity and more commands to  
implement on the client side.

4. Combo of #2 and #3
Leverage #2 to store label lookup information, with #3 to define  
system-wide keywords.
Pros: See #3.  Only adds commands necessary that can't be done through  
other means.
Cons: Requires client/server to support METADATA, even though it would  
probably not be that much more work just to add the needed lookup bits  
to a new extension.


Hopefully I have provided a coherent explanation of my perceived  
limitations with the current specs.  Coming from a webmail developer,  
this is an important issue to me since it is quite common to be  
dealing with IMAP servers that are being accessed by other MUAs.   
Synchronization of keywords would go a long way towards making  
cross-usage of MUAs smoother.

As a last little bit, just to get it out there, I will mention that  
Timo also briefly mentioned that he has thought about commands to  
manage PERMANENTFLAGS  
(http://dovecot.org/list/dovecot/2011-May/058894.html)  Figured I  
would throw it out there while on the topic of flags/keywords in order  
to ensure all topics are covered.  Hope you don't mind Timo :)

Looking forward to hearing responses/comments/criticism.

michael

On Wed, 18 May 2011 01:16:45 -0600
Michael M Slusarz <slusarz@curecanti.org> wrote:

(Terminology: keyword, label, flag - joy)

> 1. Implement UTF7-IMAP (or similar) storage of keywords.
> Pros: Would allow keyword & label information to be kept in a single  
> location; all MUAs should be able to easily adapt since they should  
> already have code necessary to do UTF7-IMAP conversions
> Cons: Haven't thought about it fully, but doesn't seem like it could  
> be made to be backward compatible with RFC 3501 keywords.  Doesn't  
> solve problem of having one keyword meaning the same thing across all  
> mailboxes.

Lack of backwards compatibility kills this one out the gate.  You can't
break existing clients and servers.

> 2. Define a server METADATA entry (RFC 5464) to store keyword->label mapping
> Pros: Works with old RFC 3501 servers.  Leverages an existing IMAP extension
> Cons: Doesn't solve problem of having one keyword meaning the same  
> thing across all mailboxes.  Doesn't really seem to be the kind of  
> thing that should necessarily be a metadata entry - flags/keywords are  
> a core part of the IMAP spec, so to me it just "feels" like something  
> else should be used instead (really useful description, I know).

IMAP is a twisty little maze of hacks on top of the original spec already.

> 3. New commands/response codes
> Pros: Works with old RFC 3501 servers.  Would be able to handle issue  
> of one keyword meaning the same thing across all mailboxes.
> Cons: As with all extensions, adds complexity and more commands to  
> implement on the client side.

And on the server side too - unless you only want to support a couple of
servers.  Cyrus would add it, and Dovecot.  But selling it to everyone else
might be tricky.  And actually, storage in Cyrus would be slightly tricky
too, but I can figure that one out.

> 4. Combo of #2 and #3
> Leverage #2 to store label lookup information, with #3 to define  
> system-wide keywords.
> Pros: See #3.  Only adds commands necessary that can't be done through  
> other means.
> Cons: Requires client/server to support METADATA, even though it would  
> probably not be that much more work just to add the needed lookup bits  
> to a new extension.

I think once you're requiring the client and server to support a brand new
command anyway, requiring METADATA isn't a big stretch.  I believe you can
actually store a server-level METADATA item, as well as per-mailbox ones.
Makes sense to use that.  It does mean that the admin needs to add new
keywords to the system though... mapping to basic flag names underneath.

> Hopefully I have provided a coherent explanation of my perceived  
> limitations with the current specs.  Coming from a webmail developer,  
> this is an important issue to me since it is quite common to be  
> dealing with IMAP servers that are being accessed by other MUAs.   
> Synchronization of keywords would go a long way towards making  
> cross-usage of MUAs smoother.

Absolutely.  While you're at it, can you look into standardising
$NotSpam $NonSpam NotSpam NonSpam Spam Junk NotJunk, etc - all the
various magic flags used by different clients to label UBE.

This boat may well have sailed...

> As a last little bit, just to get it out there, I will mention that  
> Timo also briefly mentioned that he has thought about commands to  
> manage PERMANENTFLAGS  
> (http://dovecot.org/list/dovecot/2011-May/058894.html)  Figured I  
> would throw it out there while on the topic of flags/keywords in order  
> to ensure all topics are covered.  Hope you don't mind Timo :)

I've been thinking about that for Cyrus.  We run a "repack" on the mailbox
every so often - and we could count up the usage of flags during that, and
remove any named user flags from PERMANENTFLAGS that had no message currently
using them after a repack.

FYI: Cyrus has a hard limit of 128 user flags due to the fixed-width binary
nature of the cyrus.index format.  This makes removing stale flags slightly
more pressing for Cyrus than for other servers which don't have this limit...

Bron.

On Wed, May 18, 2011 at 12:16 AM, Michael M Slusarz
<slusarz@curecanti.org> wrote:
> Hi all,
>
> First time poster. ?Quick background: I am the primary author of IMP
> (http://www.horde.org/imp/) and the related Horde IMAP client library. ?Had
> a discussion a few weeks back with Timo on the dovecot list about some
> keyword issues, and he suggested I raise the topics on this list.
>
> That topic is handling of keywords. ?More specifically, it can be broken
> down into several parts (apologizes if I make the following discussion too
> basic - just want to make sure I am on the same page):
>
> 1. Allowing non-ASCII characters in keywords

Note this is one of the main reasons we didn't map our labels to
keywords in the Gmail IMAP implementation.  That, and the majority of
clients don't really expose keywords to the user in any real fashion.

We do store keywords as labels which aren't visible in our web
interface, mostly because IMAP clients rarely used keywords to store
anything particularly user visible.

<snip>

> 3. New commands/response codes
> Pros: Works with old RFC 3501 servers. ?Would be able to handle issue of one
> keyword meaning the same thing across all mailboxes.
> Cons: As with all extensions, adds complexity and more commands to implement
> on the client side.

And this was our eventual solution.  We don't expect clients to
implement our extensions for this, but since IMAP has become the
defacto Gmail API, it became necessary to expose some Gmail specific
data over IMAP for non-traditional applications.

http://code.google.com/apis/gmail/imap/#x-gm-labels

So, we added X-GM-LABELS which is analogous to FLAGS and can be used
in SEARCH/FETCH/STORE in a similar way, and uses UTF7 ASTRINGs as
arguments.

Obviously, this isn't backward compatible at all, but I may argue that
if the end-goal is to have real user-visible labels, then the current
keywords namespace is probably too cluttered to be able to actually
share keywords across clients.  That could be solved by yet another
prefix, though.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM

On 18 May 2011, at 10:08, Bron Gondwana wrote:

> Absolutely.  While you're at it, can you look into standardising
> $NotSpam $NonSpam NotSpam NonSpam Spam Junk NotJunk, etc - all the
> various magic flags used by different clients to label UBE.
>
> This boat may well have sailed...

The boat may have sailed, but these flags have "recently" been 
standardised by Alexey Melnikov:

	http://www.iana.org/assignments/imap-keywords/imap-keywords.xml
	http://www.ietf.org/mail-archive/web/morg/current/msg00441.html

For the record, I would (as an email client developer) also welcome 
improvements of IMAP keywords.

Kind regards,

Benny Kj?r Nielsen

On 18.5.2011, at 11.08, Bron Gondwana wrote:

> On Wed, 18 May 2011 01:16:45 -0600
> Michael M Slusarz <slusarz@curecanti.org> wrote:
> 
> (Terminology: keyword, label, flag - joy)
> 
>> 1. Implement UTF7-IMAP (or similar) storage of keywords.
>> Pros: Would allow keyword & label information to be kept in a single  
>> location; all MUAs should be able to easily adapt since they should  
>> already have code necessary to do UTF7-IMAP conversions
>> Cons: Haven't thought about it fully, but doesn't seem like it could  
>> be made to be backward compatible with RFC 3501 keywords.  Doesn't  
>> solve problem of having one keyword meaning the same thing across all  
>> mailboxes.
> 
> Lack of backwards compatibility kills this one out the gate.  You can't
> break existing clients and servers.

I think this is the only way to make it completely backwards compatible with both clients and servers. mUTF7 wouldn't work because keywords are case-insensitive, but something else would, as long as servers allow long enough keywords. The encoded keywords could begin with some yet another prefix character, such as #. And it could be required that it's used only when non-atom-chars are in the name. So it would be possible to set, for example:

 - $Spam
 - Spam
 - Work
 - #121h5jh125jh12fjs2

Or possibly assume that # (or something else) is so rarely used, that the whole keyword wouldn't have to be encoded:

 - Tax#20Forms#20#282011#29

That would still be readable with old clients that show keywords as-is, and wouldn't require anything special from server side.

On 05/18/2011 10:08 AM, Bron Gondwana wrote:
> Lack of backwards compatibility kills this one out the gate.  You can't
> break existing clients and servers.

It doesn't have to. ENABLE exists for things like this.

That said, I like X-GM-LABELS more. A new namespace for user-originated 
names.

Arnt

On Wed, 18 May 2011, Brandon Long wrote:
>> 1. Allowing non-ASCII characters in keywords
> Note this is one of the main reasons we didn't map our labels to
> keywords in the Gmail IMAP implementation.  That, and the majority of
> clients don't really expose keywords to the user in any real fashion.

The problem is that in doing so, you forked IMAP in a way that causes
trouble to compliant clients in this regard, and breaks compliant clients
due to violations of the specification elsewhere.

This is deja vu of Microsoft's 1990s "Embrace, Extend, Destroy". The
difference is that Microsoft's code has more bugs than a Manhattan
basement.  Google, on the other hand, just doesn't seem to care about
compliance.

> We do store keywords as labels which aren't visible in our web
> interface, mostly because IMAP clients rarely used keywords to store
> anything particularly user visible.

That is the case if you define "IMAP clients" as Outlook and Thunderbird.
Yes, I am aware that is the definition as used at Google.

> And this was our eventual solution.  We don't expect clients to
> implement our extensions for this, but since IMAP has become the
> defacto Gmail API, it became necessary to expose some Gmail specific
> data over IMAP for non-traditional applications.

Too bad that the Gmail server is an incompatible and non-compliant fork
that is (at best) barely usable with a standard IMAP client.  This has
obliged most uses of standard IMAP clients to give up on using the Gmail
IMAP server in favor of POP or POP-over-IMAP download/delete.

The extensions are not a problem so much as it is the other things.
However, you could done that functionality within the framework of
standard IMAP, without any extension; and then the functionality would be
used by standard clients without requiring specific knowledge of Gmail.

It just requires creativity and understanding; and the latter requires
experience.  Too bad that Google sacked Brian Reid.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Wed, 18 May 2011 11:52:27 +0300
Timo Sirainen <tss@iki.fi> wrote:

> I think this is the only way to make it completely backwards compatible with both clients and servers. mUTF7 wouldn't work because keywords are case-insensitive, but something else would, as long as servers allow long enough keywords. The encoded keywords could begin with some yet another prefix character, such as #. And it could be required that it's used only when non-atom-chars are in the name. So it would be possible to set, for example:
> 
>  - $Spam
>  - Spam
>  - Work
>  - #121h5jh125jh12fjs2
> 
> Or possibly assume that # (or something else) is so rarely used, that the whole keyword wouldn't have to be encoded:
> 
>  - Tax#20Forms#20#282011#29
> 
> That would still be readable with old clients that show keywords as-is, and wouldn't require anything special from server side.

Hmm... here's what we're setting for "the name of the folder the message
was deleted from" in our new web interface to allow "undelete" to work:

xof-KRZGC43I xof-IFZGG2DJOZSQ xof-KNSW45BAJF2GK3LT

MIME::Base32::encode($FlagFolderName);

http://www.ietf.org/rfc/rfc3548.txt

Base32 is case-insensitivity-safe, but better packing than base16.

I think it's better to just base32 the entire string rather than trying
for partial encoding - you're still going to get case folding otherwise.

So something like "if it starts with a # it's a base32 encoded UTF8 keyword"
would be fully flexible, and backwards compatible.

Bron.

On Thu, 19 May 2011 10:27 +0200, "Arnt Gulbrandsen" <arnt@gulbrandsen.priv.no> wrote:
> On 05/18/2011 10:08 AM, Bron Gondwana wrote:
> > Lack of backwards compatibility kills this one out the gate.  You can't
> > break existing clients and servers.
> 
> It doesn't have to. ENABLE exists for things like this.

Yeah, of course. But you need to suppress them unless enabled or something... and deal with issues ?ike what "store flags (x)" means if not enabled. Does it clear the flags it can't see?

> That said, I like X-GM-LABELS more. A new namespace for user-originated 
> names

Agree. It makes more sense than extending flags.

Bron
-- 
  Bron Gondwana
  brong@fastmail.fm

On Wed, May 18, 2011 at 11:44:54AM -0700, Mark Crispin wrote:
> On Wed, 18 May 2011, Brandon Long wrote:
> >And this was our eventual solution.  We don't expect clients to
> >implement our extensions for this, but since IMAP has become the
> >defacto Gmail API, it became necessary to expose some Gmail specific
> >data over IMAP for non-traditional applications.
> 
> Too bad that the Gmail server is an incompatible and non-compliant fork
> that is (at best) barely usable with a standard IMAP client.  This has
> obliged most uses of standard IMAP clients to give up on using the Gmail
> IMAP server in favor of POP or POP-over-IMAP download/delete.

What annoys me the most is no CONDSTORE/QRESYNC support, so you
can't create an efficient gmail client - you need to sync all the
flags, every time.  Gmail is big enough that I'm fairly willing to
special case it in exchange for big performance wins, but there's
just no way to be sure you have all the changes.
 
Bron.

On Wed, May 18, 2011 at 2:07 PM, Bron Gondwana <brong@fastmail.fm> wrote:
> On Wed, May 18, 2011 at 11:44:54AM -0700, Mark Crispin wrote:
>> On Wed, 18 May 2011, Brandon Long wrote:
>> >And this was our eventual solution. ?We don't expect clients to
>> >implement our extensions for this, but since IMAP has become the
>> >defacto Gmail API, it became necessary to expose some Gmail specific
>> >data over IMAP for non-traditional applications.
>>
>> Too bad that the Gmail server is an incompatible and non-compliant fork
>> that is (at best) barely usable with a standard IMAP client. ?This has
>> obliged most uses of standard IMAP clients to give up on using the Gmail
>> IMAP server in favor of POP or POP-over-IMAP download/delete.
>
> What annoys me the most is no CONDSTORE/QRESYNC support, so you
> can't create an efficient gmail client - you need to sync all the
> flags, every time. ?Gmail is big enough that I'm fairly willing to
> special case it in exchange for big performance wins, but there's
> just no way to be sure you have all the changes.

Its on our roadmap, though it requires some fairly extensive changes
to our backend store.  May not happen until next year.

CONDSTORE was published when we started, but QRESYNC wasn't.
CONDSTORE would have been a lot of problems on our existing backend,
and it didn't appear any clients implemented it.  That's one issue
we've had on whether to expend the effort on any particular extension
is knowing whether or not any clients would actually take advantage of
it.

We do try to be fast for the traditional UID FLAGS fetch, but its
still O(messages) so it can get slow for large mailboxes, and the data
transfer isn't anything to sneeze at either.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM

On Wed, 18 May 2011, Bron Gondwana wrote:
> What annoys me the most is no CONDSTORE/QRESYNC support, so you
> can't create an efficient gmail client - you need to sync all the
> flags, every time.  Gmail is big enough that I'm fairly willing to
> special case it in exchange for big performance wins, but there's
> just no way to be sure you have all the changes.

This serves to illustrate the overall problem with synchronization, and
specifically with synchronizing email.

Email synchronization is a fool's errand; but there seem to be an abundant
supply of fools that undertake it.

Thus we have miserable mobile device email clients such as Mail.app on the
iToy, BlackBerry, and the default Mail app on Android.  At least Android
has k9mail which - just barely - steps over the line into "usability".

A special place in hell exists for the authors of synchronizing email
clients that refuse to display email unless there's a session open to the
server...

All in the name of the holy religion of "email users won't tolerate
waiting for the network/not being able to access mail without network
access."  Hello kids, ever hear of webmail?

There is another way of doing client/server email.  Too bad that very few
understand it; many more that have heard of it, but do not understand it.

Sadly, as the only money to be made in email today is in making email NOT
work, not in making it work, it's not likely that we'll see much change in
the status quo.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Wed, May 18, 2011 at 02:47:38PM -0700, Brandon Long wrote:
> Its on our roadmap, though it requires some fairly extensive changes
> to our backend store.  May not happen until next year.

Fair enough.  I'm actually more interested in some of the other
stuff.  I should look at how you did the threading and the unique
message ids and see if there's any way we can make it interoperate
at all with how Cyrus does GUID (sha1 of message body) and the
conversations stuff we're doing at Opera/FastMail - which is also
super non-standard.
 
> CONDSTORE was published when we started, but QRESYNC wasn't.
> CONDSTORE would have been a lot of problems on our existing backend,
> and it didn't appear any clients implemented it.  That's one issue
> we've had on whether to expend the effort on any particular extension
> is knowing whether or not any clients would actually take advantage of
> it.

Yeah - CONDSTORE was a pain when it came in to Cyrus too, but it turns
out to be very good at making a low bandwidth replication protocol as
well, because you can avoid the downside (no memory of deletes) by
caching the modseq at which you deleted a message.  Which is exactly
what QRESYNC buys as well.

> We do try to be fast for the traditional UID FLAGS fetch, but its
> still O(messages) so it can get slow for large mailboxes, and the data
> transfer isn't anything to sneeze at either.

Yeah, the data transfer is the real problem.  At least you support
COMPRESS!

Bron.

Mark Crispin wrote:
>
> There is another way of doing client/server email.  Too bad that very few
> understand it; many more that have heard of it, but do not understand it.
>

Well?  Don't keep me in suspense.... what is it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20110518/10a27e01/attachment.html>

On Wed, May 18, 2011 at 04:15:19PM -0700, Mark Crispin wrote:
> On Wed, 18 May 2011, Bron Gondwana wrote:
> >What annoys me the most is no CONDSTORE/QRESYNC support, so you
> >can't create an efficient gmail client - you need to sync all the
> >flags, every time.  Gmail is big enough that I'm fairly willing to
> >special case it in exchange for big performance wins, but there's
> >just no way to be sure you have all the changes.
> 
> This serves to illustrate the overall problem with synchronization, and
> specifically with synchronizing email.
> 
> Email synchronization is a fool's errand; but there seem to be an abundant
> supply of fools that undertake it.

God yeah, because - y'know, people WANT it.  There's two approaches to
that.  Rail against the stupidity of people, or accept that just, perhaps,
it might be OK for people to want things.
 
> Thus we have miserable mobile device email clients such as Mail.app on the
> iToy, BlackBerry, and the default Mail app on Android.  At least Android
> has k9mail which - just barely - steps over the line into "usability".

Barely is right.  A search that could find emails older than yesterday
would be nice.

> A special place in hell exists for the authors of synchronizing email
> clients that refuse to display email unless there's a session open to the
> server...

Agree.

> All in the name of the holy religion of "email users won't tolerate
> waiting for the network/not being able to access mail without network
> access."  Hello kids, ever hear of webmail?

I don't know about you - but I'm using offlineimap+mutt as my email client
on my laptop precisely to avoid all this nonsense.  offlineimap speaks a
very minimal IMAP protocol and does the great evil of email synchronisation
(more fool me) so that I can actually access all my email when I'm, y'know,
OFFLINE.  I find myself benefiting from that multiple times per month
despite theoretically having mobile access everywhere.  And it's all on an
SSD, so searches are stupidly fast even if I wind up doing body scans.

Besides I'm not sharing my IO queue with 5,000 other connected users when
I go to actually access my email.

The downside of offlineimap is, as Brendan pointed out for gmail's case,
a 100k message mailbox will cause 6Mb of traffic just to synchronise flags,
because there's no way to say "tell me what changed since last time".  At
least I'm running a patched copy that has COMPRESS support.
http://github.com/brong/brong-offlineimap/ - it's pretty old now, but the
compress support makes it worthwhile.

> Sadly, as the only money to be made in email today is in making email NOT
> work, not in making it work, it's not likely that we'll see much change in
> the status quo.

The money is in making $FOO work the way users want, rather than telling
them what to want.  Unless you have a huge advertising budget it's much
easier to sell people something they already think they want than to make
them want something you would prefer to sell.

Bron.

On Wed, May 18, 2011 at 9:58 PM, Bron Gondwana <brong@fastmail.fm> wrote:
> The downside of offlineimap is, as Brendan pointed out for gmail's case,
> a 100k message mailbox will cause 6Mb of traffic just to synchronise flags,
> because there's no way to say "tell me what changed since last time". ?At
> least I'm running a patched copy that has COMPRESS support.
> http://github.com/brong/brong-offlineimap/ - it's pretty old now, but the
> compress support makes it worthwhile.

Looks like you forked it a couple months before we submitted a patch
to stop asking for iNTERNALDATE on every sync request.

https://github.com/brong/brong-offlineimap/blob/brondev/offlineimap/folder/IMAP.py#L138

Since INTERNALDATE never changes, and the format isn't small (41 extra
bytes per message), it greatly increases the size of the response
without any benefit... and it also causes us to fall back to a much
slower / more expensive fetch handler.  Its probably a good thing to
fix even if you aren't using Gmail.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM

On Thu, 19 May 2011, Bron Gondwana wrote:
>> Email synchronization is a fool's errand; but there seem to be an abundant
>> supply of fools that undertake it.
> God yeah, because - y'know, people WANT it.  There's two approaches to
> that.  Rail against the stupidity of people, or accept that just, perhaps,
> it might be OK for people to want things.

They "want" it because they have been told they want it, and they have
never been shown an alternative.  It's the same reason that North Koreans
support their government.

What consumers really want from email is that it works.  Synchronization
vs. non-synchronization, bombastic blurdybloops vs. wallflower sarasoops,
none of that matters.  The only thing that matters is that it work.

>> At least Android
>> has k9mail which - just barely - steps over the line into "usability".
> Barely is right.

By "barely" I mean that (unlike the default Google Mail app) it
condescends to cope with 4 mail sources, and a total of about 200
messages, without crashing on every new message.

Clearly an unreasonable demand on my part.

> A search that could find emails older than yesterday
> would be nice.

A problem that was solved nearly 40 years ago.

> I don't know about you - but I'm using offlineimap+mutt as my email client
> on my laptop precisely to avoid all this nonsense.  offlineimap speaks a
> very minimal IMAP protocol and does the great evil of email synchronisation
> (more fool me) so that I can actually access all my email when I'm, y'know,
> OFFLINE.  I find myself benefiting from that multiple times per month
> despite theoretically having mobile access everywhere.

The only time I ever find myself offline is when I am in very remote areas
of northern British Columbia or the Yukon; and that's only because I chose
not to cough up the dough for satellite.  Even in rural Alaska, where I
have to carry a firearm for bear protection on my property, I have
continuous reliable online access.

> And it's all on an
> SSD, so searches are stupidly fast even if I wind up doing body scans.

Except that at some time you had to download the body to that device.
Given the number of devices that I use, the volume of mail that I deal,
and the large attachments in that mail, that is not a viable option.

> Besides I'm not sharing my IO queue with 5,000 other connected users when
> I go to actually access my email.

If a server can't handle the I/O load for a mere 5000 connected users then
it has no business being a server for 5000 connected users.

> The downside of offlineimap is, as Brendan pointed out for gmail's case,
> a 100k message mailbox will cause 6Mb of traffic just to synchronise flags,
> because there's no way to say "tell me what changed since last time".

That is because of the stupidity of doing such synchronization in the
first place.  Unless you have such a large screen that it can show the
flags of 100K messages, there is no reason that the UI needs it.

A webmail server doesn't transmit data 100,000 messages to the web
browser.

Think about that.

> The money is in making $FOO work the way users want, rather than telling
> them what to want.  Unless you have a huge advertising budget it's much
> easier to sell people something they already think they want than to make
> them want something you would prefer to sell.

The problem is not in producing a product that people would want to use.
The problem is getting ROI on the development costs; and that in turn
requires people to care enough about email that doesn't suck to pay money
for it.

The only email software that people buy is software that is purposed to
make email work less well.  There is quite a bit of money in doing that.
They'll pay quite a bit per-seat for that, but will throw you out of the
office if you suggest that they pay a fraction as much for an email client
that does not suck.

Think about that.

All other email software is "free", or is part of a collaboration suite
(and there is one big player in that game that makes the others look like
mice by comparison).  People don't buy a collaboration suite to get email;
they buy a collaboration suite to get a collaboration suite.  Email is an
afterthought; yet having paid the costs for the collaboration suite they
aren't about to pay for separate email, no matter how badly email sucks in
the collaboration suite and/or how amazingly great the separate email may
be.

There's no ROI on email clients.  Most people use webmail anyway.  The
proof is that there are NO good email clients on mobile devices.  If there
was a market, it would be filled with high-priced competition and people
would be paying it.  It isn't and they aren't.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On 05/19/11 07:45, Mark Crispin wrote:
> The only time I ever find myself offline is when I am in very remote areas
> of northern British Columbia or the Yukon; and that's only because I chose
> not to cough up the dough for satellite.  Even in rural Alaska, where I
> have to carry a firearm for bear protection on my property, I have
> continuous reliable online access.

Hi Mark, I'm coming from Europe, where the charges for cell data roaming
still suck pretty hard. There's no way I'm going to pay for a transfer
of 1MB of data every now and then.

> That is because of the stupidity of doing such synchronization in the
> first place.  Unless you have such a large screen that it can show the
> flags of 100K messages, there is no reason that the UI needs it.

Please advise me on a way to better show information about the total
number of unread messages in a mailbox. I can't use STATUS on a SELECTed
mailbox, and I can't do anything when I see an EXPUNGE of a message for
which I haven't got flags before. Shall I decrement my unread count, or
not? To my knowledge, there's absolutely no way of doing that correctly
except knowing FLAGS of all messages in advance.

And yes, I really like the number on the left that shows me how many
unread messages are in there, nicely after each mailbox name.

Cheers,
Jan

-- 
Trojita, a fast e-mail client -- http://trojita.flaska.net/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 270 bytes
Desc: OpenPGP digital signature
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20110519/ce0cdb52/attachment.sig>

On Wed, May 18, 2011 at 10:45:15PM -0700, Mark Crispin wrote:
> The only time I ever find myself offline is when I am in very remote areas
> of northern British Columbia or the Yukon; and that's only because I chose
> not to cough up the dough for satellite.  Even in rural Alaska, where I
> have to carry a firearm for bear protection on my property, I have
> continuous reliable online access.

Bully for you.  The rest of the world is not necessarily in the same
set of circumstances as you.
 
> >And it's all on an
> >SSD, so searches are stupidly fast even if I wind up doing body scans.
> 
> Except that at some time you had to download the body to that device.
> Given the number of devices that I use, the volume of mail that I deal,
> and the large attachments in that mail, that is not a viable option.

I can do that when I have a fast link.  Again, the world is not you.
It's not a viable option FOR YOU.  Storage for the ~5Gb of email I
currently keep in offlineimap is trivially cheap, and I only had to
sync it once.

> >Besides I'm not sharing my IO queue with 5,000 other connected users when
> >I go to actually access my email.
> 
> If a server can't handle the I/O load for a mere 5000 connected users then
> it has no business being a server for 5000 connected users.

Seriously?  You expect anyone to buy the argument that a server for 5000
concurrent users should be able to give similar random IO performance to
a local dedicated disk for one user?

> >The downside of offlineimap is, as Brendan pointed out for gmail's case,
> >a 100k message mailbox will cause 6Mb of traffic just to synchronise flags,
> >because there's no way to say "tell me what changed since last time".
> 
> That is because of the stupidity of doing such synchronization in the
> first place.  Unless you have such a large screen that it can show the
> flags of 100K messages, there is no reason that the UI needs it.
> 
> A webmail server doesn't transmit data 100,000 messages to the web
> browser.
> 
> Think about that.

Thinking... done.  Gosh, that was quick.  I'm not running a webmail server,
I'm running a local cache that I want to keep in sync with the server so I
can run searches and get identical results.  I want this to work offline.

Did I mention 400ms round trips?  They are noticable compared to local
storage.  That's approximately the distance between Australia and Norway,
and the speed of light isn't that flexible about my wants.

> The only email software that people buy is software that is purposed to
> make email work less well.  There is quite a bit of money in doing that.
> They'll pay quite a bit per-seat for that, but will throw you out of the
> office if you suggest that they pay a fraction as much for an email client
> that does not suck.
> 
> Think about that.

I thought.  Turns out maybe they DO know what they want, and what they want
isn't what you're peddling.  Maybe different features are important to them
than the features that are important to you.

Think about that.
 
> There's no ROI on email clients.  Most people use webmail anyway.  The
> proof is that there are NO good email clients on mobile devices.  If there
> was a market, it would be filled with high-priced competition and people
> would be paying it.  It isn't and they aren't.

True.  So what was your point exactly?

Bron.

On Thu, 19 May 2011, Jan Kundr?t wrote:
> Please advise me on a way to better show information about the total
> number of unread messages in a mailbox. I can't use STATUS on a SELECTed
> mailbox, and I can't do anything when I see an EXPUNGE of a message for
> which I haven't got flags before. Shall I decrement my unread count, or
> not? To my knowledge, there's absolutely no way of doing that correctly
> except knowing FLAGS of all messages in advance.

SEARCH will always be less data than fetching all the messages in advance.

It can be argued that you're still in a world of hurt in a 100K mailbox
where most messages are unseen (or there's an unseen message near the
start).  But it is less of a world of hurt than fetching all the flags.
What's more, this is an edge case, not the normal use case.

[And it's not all that bad.  I've loaded threads for mailboxes that large,
via WiFi, on a bitty device.  It took a while, but not forever.
Considering that most bitty device clients toss their cookies at much less
than 1000 messages, the fact that it handled 100K messages at all was
amazing and the modest delay was excusable.]

Unseen messages in the normal use case are clustered at the top of the
INBOX.  That's not going to hurt.  This is the case that you care about,
not the edge case.

You even have warning about the edge case; the UNSEEN value from SELECT!
If you see:
 	* 100000 EXISTS
 	* OK [UNSEEN 1]
that indicates that there may be as many as 100000 unseen messages (and
even suggest that all messages are unseen).

Conversely, if you see:
 	* 100000 EXISTS
 	* OK [UNSEEN 99982]
Then you know that there are, at most, 19 unseen messages.

Depending upon how much you care about the edge case, there are some
creative things that you can do.  For example, you can do
 	tag SEARCH 1:50 UNSEEN
and found that all of the first 50 messages were unseen, that might be an
indicator that all messages are unseen.

In fact, if you do
 	tag FETCH 1 FLAGS
or
 	tag SEARCH 1 RECENT
and find that the first message is recent, that is a sure indicator of
that case.  You don't care any further.

You may also want to say that showing an unseen count is only useful if it
is under some value.  Thus if you do:
 	tag SEARCH 99900:100000 UNSEEN
and see that they are all unseen, then maybe just report "99+" as the
unseen value.  As an end user, once we get into three digit unseen counts,
we're in the world of "lots of unseen messages" that I'm probably not
going to read on a bitty device screen.

It is limited only by your imagination (a problem for some Gen-Xers) and
the number of RTTs that you're willing to spend.

There are lots of other tricks that you can do with message sequence
numbers and the very useful property of strict ascendency.  A client that
does not use MSNs at all will suck.  It can't help but suck; it is denied
most of the magic of IMAP.

Last but not least, there is an extension called ESEARCH that solves this
problem.  Unfortunately, there is ambiguity in how to interpret the
ESEARCH specification.  But for this purpose, if the server offers ESEARCH
you are home free.

In conclusion, let me emphasize that there is quite a bit of magic in IMAP
that you'll never learn if you just look at the examples and start
programming without considering the interactions.  IMAP is a (too) subtle
protocol; yet every arcane aspect has a purpose - at times quite deep and
magical.

Learn as well to distinguish between edge cases and common cases.  Don't
say "there's no point to doing the common case well because the edge case
will suck"; that will lead you to sucking in the common case.  Let the
edge case suck; you want to excel in the common case.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Thank you for a very entertaining rant.  From your simplistic cookbook
arguments, you must be Gen-X.

On Fri, 20 May 2011, Bron Gondwana wrote:
> Thinking... done.  Gosh, that was quick.  I'm not running a webmail server,
> I'm running a local cache that I want to keep in sync with the server so I
> can run searches and get identical results.  I want this to work offline.

You didn't think hard enough, nor did you grasp the point.

> Did I mention 400ms round trips?

I deal with RTTs like that all the time.  I've dealt with much higher.  I
will spare you the "back in the day" story.

It's your choice to be clouded to the other end of the world over slow
satellite link.  Those who live by the cloud, die by the cloud.

> Turns out maybe they DO know what they want, and what they want
> isn't what you're peddling.  Maybe different features are important to them
> than the features that are important to you.

I don't produce email clients.  Other than a (nicely profitable) hobby, I
don't produce email servers either.  What I produce is quite lucrative,
thank you.

I do, however, consume email clients.  For my purposes, I want the email
that I use to work well.  The current crop of email clients all suck, some
more than others.

Mobile device email clients suck worse of all.  It's a pathetic world when
the likes of k9mail on Android - a program whose failings are well-known -
is a recommended solution.

Perhaps you haven't noticed - nobody has formally dropped the term "email"
from their marketing yet - but today's industry is collaboration
(dominated by one player in particular) and messaging.  Email is a dying
medium.

Email client usage is rapidly dropping off.  Current email clients do not
do what consumers want.  As there is no remedy within email, consumers
have turned to other media for communication.  Webmail is simply a
disposable means to keep an email address for those few matters that still
require one.

It is, however, fascinating that you produce the "what consumers want"
argument to attack me; webmail has shown that consumers don't particularly
value your desire to work offline.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

Hmmm... didn't realize my seemingly innocuous email requesting  
comments would cause such a vigorous, wide-ranging discussion on a  
variety of subjects.  However, in no large part due to my own selfish  
interests, this is my attempt to summarize/expand on the feedback  
generally related to the original post.

Here's what (I think) we learned:

* Option A: Feasibility of extending keywords entirely within RFC 3501  
framework

Timo/Bron both present the idea that it would be technically possible  
to use the existing keyword limitations (atoms; case insensitive) to  
store non-ASCII data.  A solution involving using a special character  
to define a "keyword+", such as #, along with encoding UTF7/UTF8 data  
in a case-insensitive manner, e.g. base32, is suggested (I would  
actually suggest good old base16.  Since base32 requires MANDATORY '='  
padding - see RFC 4648 [6] - base16 in many situations will actually  
be shorter.  Plus, hex data is already prevalent in various contexts  
so developers would most likely have a greater comfort level in  
working with this kind of data).

The downside of this approach is that legacy clients would have  
unfettered access to these new keywords.  Although many clients ignore  
keywords they are unfamiliar with, other clients don't.  We can  
probably all agree that these clients presenting keywords such as  
'#666F' (base16 encoding of 'foo') to clients may not be the best of  
ideas.  We would be preserving backward compatibility at the expense  
of providing a label that means absolutely nothing to a user.

However, this limitation could be worked around with use of ENABLE.   
Thus, these newer keywords/labels would not be visible to legacy  
servers.

* Option B: Creating a new class of keywords called "Labels"

Brandon suggested that the Gmail method of X-GM-LABELS  
(http://code.google.com/apis/gmail/imap/#x-gm-labels) be used.
X-GM-LABELS are essentially keywords except they take ASTRINGS,  
encoded w/UTF7-IMAP, instead of atoms.  Although Google treats labels  
interchangably as folders, the concept could be used as an extension  
of keywords also.

This solution would require definition of a store modifier to the  
STORE/UID STORE commands (RFC 3501[6.4.6] - or maybe a store modifier  
under RFC 4466 [2.5]), along with new SEARCH criteria (RFC 3501  
[6.4.4]) and FETCH CRITERIA (RFC 3501 [6.4.5]).  The APPEND command  
would most likely need to be altered (or a new command defined) to  
allow for specifying labels when saving a message. (This list of  
potential spec alterations is not meant to be exhaustive.)

* Option C: Use keywords, provide keyword <-> label lookup

Another option, that did not receive much (if any) interest, would  
require NO additional extension of the IMAP protocol.  This method  
would be akin to the way that many clients work today: use regular RFC  
3501 keywords to mark a message, and then internally do a translation  
to the label when displaying to the end user.

This translation table could be made globally accessible to all mail  
clients through the METADATA extension.  A server-wide entry (RFC 5464  
[3.2.1.1, 6.2]) could potentially be defined and used as a  
standardized way to do keyword -> label lookups across various MUAs  
accessing the mail store.  This solution would require registration of  
an IANA entry, but would not require a substantive change to the IMAP  
spec.


Not sure if I am missing any other option, or another viable combo of  
the concepts listed above.

To add a bit more to the discussion, an issue from the client side  
that is important to address:

It would be useful for a client to have a list of all labels available  
on the server that can be applied to the messages.  These labels could  
then be used in a UI element to allow for easy labeling of messages,  
using labels that have already been previously defined by the user.

At a minimum, this would need to be available on a per-mailbox basis.   
This issue seems easy enough: a new response code [PERMANENTLABELS]  
(?) could be output on a SELECT/EXAMINE which lists all labels used in  
the mailbox.  This response code would be ignored by legacy clients  
pursuant to RFC 3501 [7.1].  The alternative, requiring the client to  
grab all flags/keywords in a mailbox and parse for the  
flags/keywords/labels used, is untenable for large mailboxes.

However, as a practical matter, labels should be globally available in  
all mailboxes.  For example, a TODO label would be useful in any  
mailbox.  Additionally, this would potentially facilitate client UI,  
as a client could, for example, provide a single unified page of  
labels to allow the user to edit rather than requiring editing in each  
mailbox (or requiring the client to loop through every mailbox to  
globally change a label).

Finally, and this might be overkill, but possibly implement some sort  
of MODSEQ-like counter to track label changes.  When labels change,  
the MODSEQ-like number is incremented.  This would allow a client to  
skip synchronizing the label list on every login if not needed.   
However, given that the number of labels in general might be limited  
(most people use no more than 5-10?), the cost might outweigh the  
benefit.

michael

On 20/05/11 16:26, Michael M Slusarz wrote:
>
>
> Finally, and this might be overkill, but possibly implement some sort 
> of MODSEQ-like counter to track label changes.  When labels change, 
> the MODSEQ-like number is incremented.  This would allow a client to 
> skip synchronizing the label list on every login if not needed.  
> However, given that the number of labels in general might be limited 
> (most people use no more than 5-10?), the cost might outweigh the 
> benefit.
>

If your new labels are set via the STORE/UID STORE commands, then 
RFC4551 says that setting them must update MODSEQ.  So your MODSEQ-like 
counter already exists.

-- 
Greg.

On Fri, 2011-05-20 at 00:26 -0600, Michael M Slusarz wrote:

> The downside of this approach is that legacy clients would have  
> unfettered access to these new keywords.  Although many clients ignore  
> keywords they are unfamiliar with, other clients don't.  We can  
> probably all agree that these clients presenting keywords such as  
> '#666F' (base16 encoding of 'foo') to clients may not be the best of  
> ideas.  We would be preserving backward compatibility at the expense  
> of providing a label that means absolutely nothing to a user.
> 
> However, this limitation could be worked around with use of ENABLE.   
> Thus, these newer keywords/labels would not be visible to legacy  
> servers.

You mean legacy clients I guess.

> * Option B: Creating a new class of keywords called "Labels"

Is it all that useful to have basically just another set of keywords
with duplicated list of commands/parameters to manage them?

You could do almost the same thing with using "ENABLE utf8-keywords",
which changes keywords to be UTF-8 ASTRINGs rather than ATOMs. Clients
that don't understand those can keep ignoring them. Clients that do, can
manage them using the same old keyword commands/parameters. And if you
wanted to separate them from the "old keywords" you could prefix them
with '#' or something. The only difference to "labels" is that it
wouldn't be possible to fetch only flags+keywords or only labels, but I
don't really see why any client would want to do that.

Quoting Greg Banks <gnb@fastmail.fm>:

> On 20/05/11 16:26, Michael M Slusarz wrote:
>>
>>
>> Finally, and this might be overkill, but possibly implement some  
>> sort of MODSEQ-like counter to track label changes.  When labels  
>> change, the MODSEQ-like number is incremented.  This would allow a  
>> client to skip synchronizing the label list on every login if not  
>> needed.  However, given that the number of labels in general might  
>> be limited (most people use no more than 5-10?), the cost might  
>> outweigh the benefit.
>>
>
> If your new labels are set via the STORE/UID STORE commands, then  
> RFC4551 says that setting them must update MODSEQ.  So your  
> MODSEQ-like counter already exists.

I guess I did not make myself clear.  The MODSEQ-ish value wasn't to  
catch label changes on messages.  It would be intended to catch label  
definition changes on a server.

The problem with looking at CONDSTORE MODSEQs is that you could  
perform all sorts of commands that will change this value that have  
nothing to do with label definition changes.  So MODSEQs become  
useless for a client to determine whether it needs to update its list  
of label definitions from the server.

This particular proposal seems lamer the more I think about it.

michael

Quoting Timo Sirainen <tss@iki.fi>:

> On Fri, 2011-05-20 at 00:26 -0600, Michael M Slusarz wrote:
>
>> * Option B: Creating a new class of keywords called "Labels"
>
> Is it all that useful to have basically just another set of keywords
> with duplicated list of commands/parameters to manage them?
>
> You could do almost the same thing with using "ENABLE utf8-keywords",
> which changes keywords to be UTF-8 ASTRINGs rather than ATOMs. Clients
> that don't understand those can keep ignoring them. Clients that do, can
> manage them using the same old keyword commands/parameters. And if you
> wanted to separate them from the "old keywords" you could prefix them
> with '#' or something. The only difference to "labels" is that it
> wouldn't be possible to fetch only flags+keywords or only labels, but I
> don't really see why any client would want to do that.

After typing up my message last night, and dwelling a bit on the  
contents, I'm beginning to think this is the better option.  I  
personally am fine with legacy clients not being able to show these  
newer keywords because that is already the status quo for many clients  
(i.e. not displaying keywords they don't recognize).  Once we remove  
the legacy compliance component, then we don't have to rely on any  
kind of funky encoding in order to pack non-atoms in the label.

With this solution, I would disagree however with the user of a '#' or  
other prefix to distinguish "new utf-8 keywords" from "old RFC 3501  
keywords".  I think the assumption should rather be that if a keyword  
is provided in a STORE/APPEND command as an ASTRING, the intent is  
that string is a human readable representation.

E.g., with these series of client commands:

TAG1 ENABLE UTF8-KEYWORDS
[...]
TAG2 STORE 1 +FLAGS foo
[...]
TAG3 STORE 2 +FLAGS "bar"

Here, TAG 2 is taken to be a regular 3501 keyword.  It is not  
necessarily intended to be the label for that message.  A client is  
free to do keyword -> label translation on this value, and can choose  
to not display this keyword if it doesn't already know about it.

Conversely, TAG3 is intended to be the full human readable label.  A  
client should not do translation, and SHOULD directly display this  
label to the user.

(Thus, this proposed extension is more than just an ability to provide  
non-atoms in a keyword.  This extension is designed to store full  
human readable labels with a message.  Therefore, this proposed  
extension is more properly identified as something like LABELS instead  
of UTF8-KEYWORDS).

I still believe the list of utf8-keywords/labels in a mailbox should  
be something that is announced by the server.  For example, a LABELS  
untagged response code could be used to provide this information.  E.g.:

TAG1 SELECT INBOX
* OK [LABELS "Foo 1" "Bar 2" {7}
? label
]
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
[...]

A client can parse these two entries to determine which flags are  
available to ALL clients, and which flags are available only to  
clients that support the new utf8-keyword extension.  This response  
code would allow the client to determine the list of keywords intended  
to act as labels, and therefore ALWAYS be displayed to the user as-is.  
Clients could then continue to hide regular keywords (all non-system  
flags contained in PERMANENTFLAGS) that they have no knowledge of.

In this solution, we would probably need to add a new special flag to  
PERMANENTFLAGS to indicate that all labels can be permanently set  
(\AllLabels) - this would prevent the need of having to repeat all  
labels in PERMANENTFLAGS.  If a certain subset of labels could not be  
set (I could only imagine this to be a rare occurrence), the  
\AllLabels flag would be omitted and the permanent labels would have  
to be enumerated in the PERMANENTFLAGS list.

michael

On Fri, 2011-05-20 at 12:19 -0600, Michael M Slusarz wrote:

> With this solution, I would disagree however with the user of a '#' or  
> other prefix to distinguish "new utf-8 keywords" from "old RFC 3501  
> keywords".  I think the assumption should rather be that if a keyword  
> is provided in a STORE/APPEND command as an ASTRING, the intent is  
> that string is a human readable representation.
> 
> E.g., with these series of client commands:
> 
> TAG1 ENABLE UTF8-KEYWORDS
> [...]
> TAG2 STORE 1 +FLAGS foo
> [...]
> TAG3 STORE 2 +FLAGS "bar"
> 
> Here, TAG 2 is taken to be a regular 3501 keyword.  It is not  
> necessarily intended to be the label for that message.  A client is  
> free to do keyword -> label translation on this value, and can choose  
> to not display this keyword if it doesn't already know about it.
> 
> Conversely, TAG3 is intended to be the full human readable label.  A  
> client should not do translation, and SHOULD directly display this  
> label to the user.

ASTRING means atom or string. So what you're really saying is that atoms
should be treated as old keywords, while strings should be treated as
new keywords. I think many IMAP parsers hide that distinction, and such
distinction isn't all that clear to humans either. If you want to keep
them separated, '#' or similar rarely used prefix would work basically
the same.

> I still believe the list of utf8-keywords/labels in a mailbox should  
> be something that is announced by the server.  For example, a LABELS  
> untagged response code could be used to provide this information.  E.g.:
> 
> TAG1 SELECT INBOX
> * OK [LABELS "Foo 1" "Bar 2" {7}
> ? label
> ]
> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
> [...]

This would get solved automatically if there was no keyword/label
separation:

* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft "#Foo 1" "#Bar 2" {9}
#? label \*)]

Not having an explicit keyword/label separation would make this easy for
servers to implement it (and clients too, I'd think!). All I'd have to
do is remove some restrictions and add a bit of code to show a keyword
only if it's a valid atom or if utf8-keywords is ENABLEd.

On Fri, May 20, 2011 at 10:00:54PM +0300, Timo Sirainen wrote:
> Not having an explicit keyword/label separation would make this easy for
> servers to implement it (and clients too, I'd think!). All I'd have to
> do is remove some restrictions and add a bit of code to show a keyword
> only if it's a valid atom or if utf8-keywords is ENABLEd.

You haven't addressed the case of "STORE FLAGS" (not +FLAGS) here, where
you would have to decide if you implicitly clear a flag based on the
ENABLE command.  That adds more complexity to the server.

Bron ( of course, few clients actually do that )

Quoting Timo Sirainen <tss@iki.fi>:

> On Fri, 2011-05-20 at 12:19 -0600, Michael M Slusarz wrote:
>
>> With this solution, I would disagree however with the user of a '#' or
>> other prefix to distinguish "new utf-8 keywords" from "old RFC 3501
>> keywords".  I think the assumption should rather be that if a keyword
>> is provided in a STORE/APPEND command as an ASTRING, the intent is
>> that string is a human readable representation.
>>
>> E.g., with these series of client commands:
>>
>> TAG1 ENABLE UTF8-KEYWORDS
>> [...]
>> TAG2 STORE 1 +FLAGS foo
>> [...]
>> TAG3 STORE 2 +FLAGS "bar"
>>
>> Here, TAG 2 is taken to be a regular 3501 keyword.  It is not
>> necessarily intended to be the label for that message.  A client is
>> free to do keyword -> label translation on this value, and can choose
>> to not display this keyword if it doesn't already know about it.
>>
>> Conversely, TAG3 is intended to be the full human readable label.  A
>> client should not do translation, and SHOULD directly display this
>> label to the user.
>
> ASTRING means atom or string. So what you're really saying is that atoms
> should be treated as old keywords, while strings should be treated as
> new keywords. I think many IMAP parsers hide that distinction, and such
> distinction isn't all that clear to humans either. If you want to keep
> them separated, '#' or similar rarely used prefix would work basically
> the same.

I am really against the idea of using something like '#' as a prefix,  
simply because, however rare it may be, it is still part of a valid  
keyword that might already be used by somebody, somewhere.  Example:  
'#1todolist'.

What about declaring as a system flag instead?  In other words, define  
'\#' as the prefix for all labels: "\#Foo Bar".  This would be  
guaranteed not to conflict with any existing keyword (or flag).  I  
realize that the original RFC 3501 spec didn't really intend for  
system flags to be used as a pseudo-namespace for a whole series of  
user defined keywords... but then again, the 3501 spec didn't allow  
for non-atoms in keywords either, which is the whole reason we are  
having this discussion in the first place.

I wouldn't think that parsing as a system flag (\#) vs. a keyword (#)  
would be any different  - both require scanning the prefix.  The only  
place where there might be a bit of problems is server-side: namely,  
if the code branches based on whether a flag is system or not.  But  
that doesn't seem like anything that would not be able to be worked  
around.

>> I still believe the list of utf8-keywords/labels in a mailbox should
>> be something that is announced by the server.  For example, a LABELS
>> untagged response code could be used to provide this information.  E.g.:
>>
>> TAG1 SELECT INBOX
>> * OK [LABELS "Foo 1" "Bar 2" {7}
>> ? label
>> ]
>> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)]
>> [...]
>
> This would get solved automatically if there was no keyword/label
> separation:
>
> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft "#Foo  
> 1" "#Bar 2" {9}
> #? label \*)]
>
> Not having an explicit keyword/label separation would make this easy for
> servers to implement it (and clients too, I'd think!). All I'd have to
> do is remove some restrictions and add a bit of code to show a keyword
> only if it's a valid atom or if utf8-keywords is ENABLEd.

I would agree that if we had a namespace that provides non-conflicting  
label names from keywords, that using just PERMANENTFLAGS would be  
acceptable.

michael

On 21.5.2011, at 0.43, Bron Gondwana wrote:

> On Fri, May 20, 2011 at 10:00:54PM +0300, Timo Sirainen wrote:
>> Not having an explicit keyword/label separation would make this easy for
>> servers to implement it (and clients too, I'd think!). All I'd have to
>> do is remove some restrictions and add a bit of code to show a keyword
>> only if it's a valid atom or if utf8-keywords is ENABLEd.
> 
> You haven't addressed the case of "STORE FLAGS" (not +FLAGS) here, where
> you would have to decide if you implicitly clear a flag based on the
> ENABLE command.  That adds more complexity to the server.
> 
> Bron ( of course, few clients actually do that )

Hmm. Yeah. I've never really liked that command. Internally in Dovecot I have only +flags/keywords, -flags/keywords and "remove all keywords". I wonder what the real use cases for STORE FLAGS is? Probably mainly syncing client's local state to server. For that if the client doesn't understand UTF-8 keywords, it probably shouldn't be touching them either. But yes, that makes my code at least more complex, requiring yet another "remote all atom keywords" command.

On Sat, 21 May 2011, Timo Sirainen wrote:
> Hmm. Yeah. I've never really liked that command. Internally in Dovecot I
> have only +flags/keywords, -flags/keywords and "remove all keywords". I
> wonder what the real use cases for STORE FLAGS is?

Other than removing all flags ("tag STORE n FLAGS ()"), there isn't a use
case.  That was not the opinion of a certain individual back in the day.

Be thankful that we have +FLAGS and -FLAGS.  I had to fight a certain
individual who demanded that I remove them.  The claim was that there
should only be FLAGS, so the client would know the result rather than
getting it back after the application of +FLAGS/-FLAGS.

I never found out if the client (which rather quickly faded into oblivion)
he wrote used FLAGS.  Certainly everybody else uses +FLAGS/-FLAGS.

My c-client API doesn't even have an API call for FLAGS.  The c-client
based version of imapd implements FLAGS by doing the API equivalent of
-FLAGS for all system flags + keywords, followed by a +FLAGS of the
desired list.  I wouldn't be surprised if the back end of other IMAP
servers also lack a "set flags" operator (instead of the more useful "add
flags" and "remove flags").

I used FLAGS once, in a tool that had to clear all flags for some reason
that I have thankfully forgotten.  I just remember doing it, and thinking
"gee, finally a use for FLAGS".  Needless to say, it would have been
better if that tool was inconvenienced than what we have now with every
IMAP server having to do it.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.