mailing list archives

Hi,
one of my users has reported [1] that his IMAP server (UW-IMAPD, reporting as "2007e.404" in the initial greetings) is using the wrong [2] interpretation of the ESEARCH extension -- always reporting the UIDs as "min:max", effectively removing the huge benefit of that extension.

What is the least damaging way to work around this? Here's a couple of ideas:

a) Detect ESEARCH response passed as min:max. When the number of messages exceeds what EXISTS reports, blacklist the ESEARCH extension and try once again using plain SEARCH.

b) Resort to ugly user-agent-sniffing-alike and blacklist ESEARCH preemptively when the initial greetings contains the "IMAP4rev1 2007e.404 at" string *and* the "SCAN" capability is present.

The option b) looks like a reasonable stopgap measure with a very limited potential for false positives. It is also implementable very quickly within my client. The option a) will require a bit more work, but it looks like something more "robust" than b).

Is there some inherent drawback in -- at least temporarily -- going with b) that I might be missing? It's an obsolete, unmaintained server with not that great market share, so implementing extra workarounds just for it are not terribly high on my list of priorities...

With kind regards,
Jan

[1] https://bugs.kde.org/show_bug.cgi?id=320828
[2] http://www.ietf.org/mail-archive/web/imapext/current/msg00477.html

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

On 6 Jun 2013, at 21:15, Jan Kundr?t <jkt@flaska.net> wrote:

> Hi,
> one of my users has reported [1] that his IMAP server (UW-IMAPD, reporting as "2007e.404" in the initial greetings) is using the wrong [2] interpretation of the ESEARCH extension -- always reporting the UIDs as "min:max", effectively removing the huge benefit of that extension.
> 
> What is the least damaging way to work around this? Here's a couple of ideas:
> 
> a) Detect ESEARCH response passed as min:max. When the number of messages exceeds what EXISTS reports, blacklist the ESEARCH extension and try once again using plain SEARCH.
> 
> b) Resort to ugly user-agent-sniffing-alike and blacklist ESEARCH preemptively when the initial greetings contains the "IMAP4rev1 2007e.404 at" string *and* the "SCAN" capability is present.
> 
> The option b) looks like a reasonable stopgap measure with a very limited potential for false positives. It is also implementable very quickly within my client. The option a) will require a bit more work, but it looks like something more "robust" than b).

Option a) looks quite unreliable (and dangerous) to me.
I hope newer versions of UW don't suffer from this.

> Is there some inherent drawback in -- at least temporarily -- going with b) that I might be missing? It's an obsolete, unmaintained server with not that great market share, so implementing extra workarounds just for it are not terribly high on my list of priorities...
> 
> With kind regards,
> Jan
> 
> [1] https://bugs.kde.org/show_bug.cgi?id=320828
> [2] http://www.ietf.org/mail-archive/web/imapext/current/msg00477.html

On Thu, 06 Jun 2013 15:15:32 -0500, Jan Kundr?t wrote:
| Hi,
| one of my users has reported [1] that his IMAP server (UW-IMAPD,
| reporting as "2007e.404" in the initial greetings) is using the
| wrong [2] interpretation of the ESEARCH extension -- always
| reporting the UIDs as "min:max", effectively removing the huge
| benefit of that extension.
| 
| What is the least damaging way to work around this? Here's a couple of ideas:
| 
| a) Detect ESEARCH response passed as min:max. When the number of
| messages exceeds what EXISTS reports, blacklist the ESEARCH
| extension and try once again using plain SEARCH.
| 
| b) Resort to ugly user-agent-sniffing-alike and blacklist ESEARCH
| preemptively when the initial greetings contains the "IMAP4rev1
| 2007e.404 at" string *and* the "SCAN" capability is present.
| 
| The option b) looks like a reasonable stopgap measure with a very
| limited potential for false positives. It is also implementable very
| quickly within my client. The option a) will require a bit more
| work, but it looks like something more "robust" than b).
| 
| Is there some inherent drawback in -- at least temporarily -- going
| with b) that I might be missing? It's an obsolete, unmaintained
| server with not that great market share, so implementing extra
| workarounds just for it are not terribly high on my list of
| priorities...

Take a look at Panda IMAP at

  http://github.com/jonabey/panda-imap

that repo includes the very last version of Mark Crispin's personal
fork of UW IMAP that he did after he was laid off from UW.  I know he
fixed a lot of non-security related stuff in his code that was never
back-ported into UW IMAP.

If it turns out that the bug you're describing is still in this code,
I'd be happy to work on fixing it / accepting patches to fix it.

Thanks,

 Jon

-- 
-------------------------------------------------------------------------------
Jonathan Abbey 				              jonabbey@arlut.utexas.edu
Applied Research Laboratories                 The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 205 bytes
Desc: not available
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130608/3175e9e4/attachment.sig>

On Fri, Jun 7, 2013 at 8:22 AM, Alexey Melnikov
<alexey.melnikov@isode.com>wrote:

> On 6 Jun 2013, at 21:15, Jan Kundr?t <jkt@flaska.net> wrote:
> > The option b) looks like a reasonable stopgap measure with a very
> limited potential for false positives. It is also implementable very
> quickly within my client. The option a) will require a bit more work, but
> it looks like something more "robust" than b).
>
> Option a) looks quite unreliable (and dangerous) to me.
> I hope newer versions of UW don't suffer from this.
>
>
Yes, go for (b); it's what I'd do. (In fact, Inky possible already does
this).

I might see if I can knock out an RFC 4731 bis which clarifies this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130607/dd8a8f15/attachment.html>

On Sat, Jun 8, 2013, at 04:10 PM, Jonathan Abbey wrote:
> On Thu, 06 Jun 2013 15:15:32 -0500, Jan Kundr?t wrote:
> | Hi,
> | one of my users has reported [1] that his IMAP server (UW-IMAPD,
> | reporting as "2007e.404" in the initial greetings) is using the
> | wrong [2] interpretation of the ESEARCH extension -- always
> | reporting the UIDs as "min:max", effectively removing the huge
> | benefit of that extension.
> | 
> | What is the least damaging way to work around this? Here's a couple of ideas:
> | 
> | a) Detect ESEARCH response passed as min:max. When the number of
> | messages exceeds what EXISTS reports, blacklist the ESEARCH
> | extension and try once again using plain SEARCH.
> | 
> | b) Resort to ugly user-agent-sniffing-alike and blacklist ESEARCH
> | preemptively when the initial greetings contains the "IMAP4rev1
> | 2007e.404 at" string *and* the "SCAN" capability is present.
> | 
> | The option b) looks like a reasonable stopgap measure with a very
> | limited potential for false positives. It is also implementable very
> | quickly within my client. The option a) will require a bit more
> | work, but it looks like something more "robust" than b).
> | 
> | Is there some inherent drawback in -- at least temporarily -- going
> | with b) that I might be missing? It's an obsolete, unmaintained
> | server with not that great market share, so implementing extra
> | workarounds just for it are not terribly high on my list of
> | priorities...
> 
> Take a look at Panda IMAP at
> 
>   http://github.com/jonabey/panda-imap
> 
> that repo includes the very last version of Mark Crispin's personal
> fork of UW IMAP that he did after he was laid off from UW.  I know he
> fixed a lot of non-security related stuff in his code that was never
> back-ported into UW IMAP.
> 
> If it turns out that the bug you're describing is still in this code,
> I'd be happy to work on fixing it / accepting patches to fix it.

I think in Jan's case as a client author, there's often a limited amount
that you can do to get servers fixed.  Sure this path will lead to the
bugfixes appearing sometime in the next 2-3 years once it gets accepted
upstream, packaged by distributions, and finally slipstreamed onto
everyone's servers in the next round of stable updates.

Even as a server author, you wouldn't believe the resistance to updates.
"it's not in Debian stable, we're not using it".  So plenty of my 2 year
old Cyrus bugs are still out there in the real world.  Unless it's a
security issue, good luck getting it fixed quickly.

Hence: workarounds.

Bron.

-- 
  Bron Gondwana
  brong@fastmail.fm

On Saturday, 8 June 2013 08:10:48 CEST, Jonathan Abbey wrote:
>   http://github.com/jonabey/panda-imap

Assuming your github username is with double Bs, the offending code is likely found at https://github.com/jonabbey/panda-imap/blob/master/src/imapd/imapd.c#L939 . I'm not sure whether I'm interpreting it correctly, it's a foreign code with a very different style than I'm used to.

That code appears to be broken on a fourth read or so :) -- it iterates over all messages and if a particular sequence of messages matches, it will output the result as min:max. That is, however, the exact problem here -- the code does not check whether the range of UIDs assigned to these messages has any holes within. In other words, the code produces correct results for SEARCH RETURN ..., but fails for UID SEARCH RETURN ... because the UIDs, unlike sequence numbers, are not guaranteed to be consecutive.

This is consistent with the output of `git blame` which suggests that the code in question comes from commit 199adc96 which is marked as "imap-2006i.tar.Z" in your repository. The capability advertised by the user's server says 2007e.404, and that version is still affected.

You can easily check this yourself. Create new mailbox, append three messages, remove the middle one and send `UID SEARCH RETURN (ALL) ALL`. If you get back something like "1,3", the code is OK, if you get "1:3", it's wrong.

I do not dare writing a patch for that code -- the assumption that one could get away with acting as if the client knew which UIDs exist and which are gone is a big one. The loop at line 952 looks like the obvious problem (it shall check for a consecutive range of UIDs, but only if the command is the UID variant of ESEARCH...), but more changes might be needed. Well, I've had enough C for now :).

On an unrelated topic -- I've noticed that you're using "|" as a quote indicator. This breaks clients which respect the long-standing convention (more or less also codified in RFC 3676 [1] -- yes, your MUA doesn't do format=flowed, but it's still useful for clients to safely detect a quoted text) that quotes are always done by ">". I was wondering if there was a particular reason for this setting, or whether it just happened to be a system-wide settings on your machine.

Yes, I *could* change the format=flowed decoder in Trojita so that it supports non-standard quoting options, but this is the first mail I've read since adding that feature four months ago, so I'm hoping that sticking with language of that RFC still has a chance of working.

With kind regards,
Jan

[1] http://tools.ietf.org/html/rfc3676

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

On Sat, 08 Jun 2013 01:39:00 -0500, Bron Gondwana wrote:
| 
| I think in Jan's case as a client author, there's often a limited amount
| that you can do to get servers fixed.  Sure this path will lead to the
| bugfixes appearing sometime in the next 2-3 years once it gets accepted
| upstream, packaged by distributions, and finally slipstreamed onto
| everyone's servers in the next round of stable updates.

Oh, of course.  Sorry, I didn't pick up the clue that Jan was a client
author.

| Even as a server author, you wouldn't believe the resistance to updates.
| "it's not in Debian stable, we're not using it".  So plenty of my 2 year
| old Cyrus bugs are still out there in the real world.  Unless it's a
| security issue, good luck getting it fixed quickly.
| 
| Hence: workarounds.

Right.  Nonetheless, I'd like to get it fixed in Panda if it's a
problem there.  For my local users, if nothing else.

 Jon

-- 
-------------------------------------------------------------------------------
Jonathan Abbey 				              jonabbey@arlut.utexas.edu
Applied Research Laboratories                 The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 205 bytes
Desc: not available
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130608/bfe3e9fd/attachment.sig>