mailing list archives

changed the subject to not distract from the main thrust of the original thread.

On Wed, May 18, 2011 at 11:44 AM, Mark Crispin <mrc+imap@panda.com> wrote:
> On Wed, 18 May 2011, Brandon Long wrote:
>>>
>>> 1. Allowing non-ASCII characters in keywords
>>
>> Note this is one of the main reasons we didn't map our labels to
>> keywords in the Gmail IMAP implementation. ?That, and the majority of
>> clients don't really expose keywords to the user in any real fashion.
>
> The problem is that in doing so, you forked IMAP in a way that causes
> trouble to compliant clients in this regard, and breaks compliant clients
> due to violations of the specification elsewhere.

I think this is a difference of opinion and not of fact.  Obviously
Gmail and IMAP have different models of a mailbox, and our
implementation had to bridge these two models.  You're implying that
we should have had folders which had no analogue in the web interface,
and we should have exposed IMAP labels as keywords and somehow solved
the same problem that this thread was asking about, perhaps as base32
encoded keywords.  No client our users actually used would have done
anything with the keywords, and our users would have been presented
with a single folder with all of their mail in it, which for some
users would have been a million messages... and let me tell you how
well most clients handle folders of that size.

> This is deja vu of Microsoft's 1990s "Embrace, Extend, Destroy". The
> difference is that Microsoft's code has more bugs than a Manhattan
> basement. ?Google, on the other hand, just doesn't seem to care about
> compliance.

I wasn't aware that we were particularly out of compliance.  I'm sure
that we are in some subtle ways, but the IMAP spec is fairly broad in
interpretation.  There was no compliance test we could use as we were
developing the service, the best we could do was compare against the
clients we had access to, many of which have their own compliance
issues that we had to work around as best we could.  We know about
http://imapwiki.org/ImapTest, though it came out after our initial
work.  It looks like their last score for us from 3 years ago isn't
all that great, I had thought when I last looked at it we weren't that
bad, so perhaps it would behoove us to see what we can do about
improving that.

>> We do store keywords as labels which aren't visible in our web
>> interface, mostly because IMAP clients rarely used keywords to store
>> anything particularly user visible.
>
> That is the case if you define "IMAP clients" as Outlook and Thunderbird.
> Yes, I am aware that is the definition as used at Google.

At the end of the day, what our users care about is that the system
works for them.  Strictly adhering to a spec that breaks things for
our users isn't a viable option.  That said, those clients aren't
actually that high on the list, looks like they're 7th & 9th.  Mobile
clients dominate, and Thunderbird is the second most popular desktop
client behind Apple Mail.

I'm sure there is some self-reinforcing in those stats, of course,
because a client that really doesn't work well isn't likely to be
used.  Its also not easy to identify clients.  The popularity of
mobile clients is also somewhat expected as most people would use the
web interface of a web mail product.

<snip>

> The extensions are not a problem so much as it is the other things.
> However, you could done that functionality within the framework of
> standard IMAP, without any extension; and then the functionality would be
> used by standard clients without requiring specific knowledge of Gmail.

As this thread points out, its not actually possible to handle unicode
label names as keywords without a bunch of challenges.  Maybe if we'd
just started putting out ASTRINGs instead of ATOMs, no clients would
care, but that's certainly not compliant.  It would also mean a
greatly increased bandwidth to give clients something they'd never
use.  Already, the typical FETCH 1:* (UID FLAGS) command that many
clients use when they first open a folder can generate 6MB of download
on a 100k message folder, adding our labels as keywords would probably
double that.

> It just requires creativity and understanding; and the latter requires
> experience. ?Too bad that Google sacked Brian Reid.

Wow is that left field.

In any case, I'm happy to discuss our reasoning for particularly
implementation choices or incompatibilities.  I'm also willing to file
bugs in cases where we're out of compliance by accident.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM

On 05/18/2011 05:36 PM, Brandon Long wrote:
> I think this is a difference of opinion and not of fact. Obviously
> Gmail and IMAP have different models of a mailbox, and our
> implementation had to bridge these two models.  You're implying that
> we should have had folders which had no analogue in the web interface,
> and we should have exposed IMAP labels as keywords and somehow solved
> the same problem that this thread was asking about, perhaps as base32
> encoded keywords.
My experience over the past several years has been that people who use 
email clients instead of webmail tend to be a more overly conservative 
bunch of people than those who do not use them. In that vein, the people 
who would be accessing Gmail over IMAP would probably have preferred the 
traditional folder model over Gmail's label concepts; I remember my 
disappointment to have discovered that my "normal" email usage model 
just didn't well with Gmail's implementation (what? delete doesn't 
actually delete the mail?)

> No client our users actually used would have done
> anything with the keywords,
If this information were made public, I'm sure knowledgeable people 
would have asked for some kind of standardization in this regard.

P.S. Gmail is not the Google product I reserve special ire for... that 
would be Google Groups. But that is a rant well off-topic for this, er, 
mailing list.

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth

On Wed, 18 May 2011, Joshua Cranmer wrote:
> My experience over the past several years has been that people who use
> email clients instead of webmail tend to be a more overly conservative
> bunch of people than those who do not use them. In that vein, the people
> who would be accessing Gmail over IMAP would probably have preferred the
> traditional folder model over Gmail's label concepts;

+1

Don't overlook Google's "youth oriented culture".  Not exactly youth any
more since there's a new generation now.  But it is a culture that very
much has the "everything that came before us sucks and we don't care"
mentality of its founders' generation.

This is distinct from the Apple mindset, which is that of a cult.

> I remember my
> disappointment to have discovered that my "normal" email usage model
> just didn't well with Gmail's implementation (what? delete doesn't
> actually delete the mail?)

I only use Gmail for spam from Google.  I doubt that mail that touches
Google's servers ever really goes away; at best it becomes invisible to
you.

>> No client our users actually used would have done
>> anything with the keywords,
> If this information were made public, I'm sure knowledgeable people
> would have asked for some kind of standardization in this regard.

Google's sole focus in supporting IMAP is to trick Outlook into giving a
more or less Gmail web experience.  They don't care what it does to
compliant clients.

Once you understand that point, the implementation of IMAP Gmail makes
complete sense.

I once made the mistake of compiling a large set of itemized and detailed
issues of what Google, Microsoft, and Apple each did wrong, complete with
examples and citations.  I spent long days on this effort, and eventually
delivered the appropriate issue lists to each of them.  The vast majority
of those issues remain unfixed.

I won't say who, but one of the threesome actually said "this list is too
long, narrow it down to one or two."

Then there is Yahoo, whose SMTP server still has a bug that I identified
in the 1990s!  I recently talked with some folks in Venezuela who ran up
against that bug; what a blast from the past.

At some point, you have to assume that they don't give a damn.  Perhaps
they are right.  Consumers have grown accustomed to the fact that email
software sucks; for the most part they've moved away from it.  There just
is no expectation, much less demand, for email that doesn't suck.

Maybe we need product liability laws for software.  Imagine me as a
federal bureaucrat empowered to slap daily fines on these cretins for not
fixing their bugs.  When it comes to fixing bugs, and having bugs get
fixed, I can be relentless.  Oh I would have such fun.  Hah!

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Wed, May 18, 2011 at 5:16 PM, Joshua Cranmer <Pidgeot18@verizon.net> wrote:
> On 05/18/2011 05:36 PM, Brandon Long wrote:
>>
>> I think this is a difference of opinion and not of fact. Obviously
>> Gmail and IMAP have different models of a mailbox, and our
>> implementation had to bridge these two models. ?You're implying that
>> we should have had folders which had no analogue in the web interface,
>> and we should have exposed IMAP labels as keywords and somehow solved
>> the same problem that this thread was asking about, perhaps as base32
>> encoded keywords.
>
> My experience over the past several years has been that people who use email
> clients instead of webmail tend to be a more overly conservative bunch of
> people than those who do not use them. In that vein, the people who would be
> accessing Gmail over IMAP would probably have preferred the traditional
> folder model over Gmail's label concepts; I remember my disappointment to
> have discovered that my "normal" email usage model just didn't well with
> Gmail's implementation (what? delete doesn't actually delete the mail?)

Its not the default, no, but there is a setting for that.  You
currently have to enable the Advanced IMAP settings lab, and then you
can select that delete means "move to trash" or "delete forever".  It
applies to the message when its been removed from the last visible
folder.

There's also a setting to disable auto-expunge.  Both of these were
the defaults originally, but were changed when testing clearly showed
that the expected user experience matched somewhat with what Mark
said: people expected a Gmail like experience in their client.  The
main use and clients turned out to be mobile devices, the number of
users who use a desktop IMAP client to access Gmail is pretty tiny.

As for Outlook, we've had a bunch of weird issues with it, and instead
we wrote a plug-in for Outlook, which also gave us the ability to sync
contacts and calendar, no IMAP involved:
http://www.google.com/apps/intl/en/business/outlook_sync.html

I'm sure these settings aren't the full picture,  most flags/keywords
still apply to the message across folders, for example, and I'm sure
there are a couple other ones.

>> No client our users actually used would have done
>> anything with the keywords,
>
> If this information were made public, I'm sure knowledgeable people would
> have asked for some kind of standardization in this regard.
>
> P.S. Gmail is not the Google product I reserve special ire for... that would
> be Google Groups. But that is a rant well off-topic for this, er, mailing
> list.

My team has recently taken over the mail handling aspects of Groups,
if you have requests, feel free to forward them to me off list.
They're also working on the new web interface, available here:
https://groups.google.com/forum/#!overview

If you're talking about it being a Usenet gateway, then I probably
already know the rant.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM

On Wed, May 18, 2011 at 10:00 PM, Brandon Long <blong@google.com> wrote:
> My team has recently taken over the mail handling aspects of Groups,
> if you have requests, feel free to forward them to me off list.
> They're also working on the new web interface, available here:
> https://groups.google.com/forum/#!overview

That should have said "the Groups team" has been working on a new web
interface, but I'm happy to forward concerns or file bugs.

Brandon
-- 
?Brandon Long <blong@google.com>
?Staff Engineer
?Gmail Delivery TLM