mailing list archives

We're seeing a problem in the Firefox OS Gaia e-mail app where if we 
manage to send our "A1 CAPABILITY" request to the gmail IMAP server 
(initial-TLS imaps/993) before it sends its greeting, it acts like it 
never hears the request.  Our not-so-clever state machine hangs until 
the connection times out.

There's more details and pcap dumps at https://bugzil.la/977867#c21 but 
basically we sometimes manage to get that request in the same TCP packet 
as the conclusion of TLS setup.  The TCP/TLS packet with the server 
greeting ACKs that packet, so it's quite conceivable our request is 
managing to fall into a very specific edge case/race.

The spec does not seem to have a final word on this, although the state 
flow diagram in http://tools.ietf.org/html/rfc3501#section-3 does seem 
to imply that it's bad form to not wait for the server greeting.  While 
I think it's clear that for gmail we need to wait for the server 
greeting, I am wondering if this is just an unusual complication/bug of 
gmail's likely more complicated server architecture.  I'd expect a 
server intentionally being pedantic to issue a NO or BAD request...

Thoughts / has anyone seen servers that intentionally or accidentally 
get upset in cases like this?

Thanks,
Andrew

PS: We do have a lot of round-trip fat in our connection establishment, 
so it's not the end of the world to explicitly wait for the server 
greeting in all cases, but if it's just gmail (and we appropriately add 
some improved timeout logic that triggers a persistent quirk for the 
server), I could see it make sense to continue not waiting for the greeting.

Is this behavior new?

We're just finishing the rollout of a new IMAP proxy, its possible its a
bug in the new impl.

Brandon
On Mar 7, 2014 7:06 AM, "Andrew Sutherland" <asuth@mozilla.com> wrote:

> We're seeing a problem in the Firefox OS Gaia e-mail app where if we
> manage to send our "A1 CAPABILITY" request to the gmail IMAP server
> (initial-TLS imaps/993) before it sends its greeting, it acts like it never
> hears the request.  Our not-so-clever state machine hangs until the
> connection times out.
>
> There's more details and pcap dumps at https://bugzil.la/977867#c21 but
> basically we sometimes manage to get that request in the same TCP packet as
> the conclusion of TLS setup.  The TCP/TLS packet with the server greeting
> ACKs that packet, so it's quite conceivable our request is managing to fall
> into a very specific edge case/race.
>
> The spec does not seem to have a final word on this, although the state
> flow diagram in http://tools.ietf.org/html/rfc3501#section-3 does seem to
> imply that it's bad form to not wait for the server greeting.  While I
> think it's clear that for gmail we need to wait for the server greeting, I
> am wondering if this is just an unusual complication/bug of gmail's likely
> more complicated server architecture.  I'd expect a server intentionally
> being pedantic to issue a NO or BAD request...
>
> Thoughts / has anyone seen servers that intentionally or accidentally get
> upset in cases like this?
>
> Thanks,
> Andrew
>
> PS: We do have a lot of round-trip fat in our connection establishment, so
> it's not the end of the world to explicitly wait for the server greeting in
> all cases, but if it's just gmail (and we appropriately add some improved
> timeout logic that triggers a persistent quirk for the server), I could see
> it make sense to continue not waiting for the greeting.
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20140307/af72d16a/attachment.html>

On Friday, 7 March 2014 16:05:48 CEST, Andrew Sutherland wrote:
> We're seeing a problem in the Firefox OS Gaia e-mail app where 
> if we manage to send our "A1 CAPABILITY" request to the gmail 
> IMAP server (initial-TLS imaps/993) before it sends its 
> greeting, it acts like it never hears the request.  Our 
> not-so-clever state machine hangs until the connection times 
> out.

Interesting. What my client is doing is patiently waiting for the initial 
response and checking whether the capability list is passed in there as a 
response code. The same for the tagged OK for the STARTTLS.

I am wondering whether you are deliberately trading bandwidth for latency, 
and whether you have some measurements showing that sending the CAPABILITY 
speculatively is beneficial. This is something which I haven't thought 
about, and if you have already measured this, I would like to, er, "be 
isnpired" :) and do it this way as well.

With kind regards,
Jan

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

Aha!  Yes, the behavior does appear to be new and is manifesting in 
older releases that didn't experience this problem before*.  QA filed 
the bug with a log indicating this problem on 2/27 on 
https://bugzil.la/977867, but it's conceivable this was happening before 
that time.

Thanks!
Andrew

* But that doesn't necessarily guarantee things given the high rate of 
platform change and platform backports occurring for Firefox OS.  In 
this case our IMAP implementation is unchanged and our TCPSocket 
implementation on top of Necko/NSS is unchanged.  I assume NSS changes 
and gets uplifts.

On 03/07/2014 10:34 AM, Brandon Long wrote:
>
> Is this behavior new?
>
> We're just finishing the rollout of a new IMAP proxy, its possible its 
> a bug in the new impl.
>
> Brandon
>
> On Mar 7, 2014 7:06 AM, "Andrew Sutherland" <asuth@mozilla.com 
> <mailto:asuth@mozilla.com>> wrote:
>
>     We're seeing a problem in the Firefox OS Gaia e-mail app where if
>     we manage to send our "A1 CAPABILITY" request to the gmail IMAP
>     server (initial-TLS imaps/993) before it sends its greeting, it
>     acts like it never hears the request.  Our not-so-clever state
>     machine hangs until the connection times out.
>
>     There's more details and pcap dumps at
>     https://bugzil.la/977867#c21 but basically we sometimes manage to
>     get that request in the same TCP packet as the conclusion of TLS
>     setup.  The TCP/TLS packet with the server greeting ACKs that
>     packet, so it's quite conceivable our request is managing to fall
>     into a very specific edge case/race.
>
>     The spec does not seem to have a final word on this, although the
>     state flow diagram in http://tools.ietf.org/html/rfc3501#section-3
>     does seem to imply that it's bad form to not wait for the server
>     greeting.  While I think it's clear that for gmail we need to wait
>     for the server greeting, I am wondering if this is just an unusual
>     complication/bug of gmail's likely more complicated server
>     architecture.  I'd expect a server intentionally being pedantic to
>     issue a NO or BAD request...
>
>     Thoughts / has anyone seen servers that intentionally or
>     accidentally get upset in cases like this?
>
>     Thanks,
>     Andrew
>
>     PS: We do have a lot of round-trip fat in our connection
>     establishment, so it's not the end of the world to explicitly wait
>     for the server greeting in all cases, but if it's just gmail (and
>     we appropriately add some improved timeout logic that triggers a
>     persistent quirk for the server), I could see it make sense to
>     continue not waiting for the greeting.
>     _______________________________________________
>     Imap-protocol mailing list
>     Imap-protocol@u.washington.edu <mailto:Imap-protocol@u.washington.edu>
>     http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20140307/93dc5419/attachment.html>

On 7.3.2014, at 18.01, Jan Kundr?t <jkt@flaska.net> wrote:

> On Friday, 7 March 2014 16:05:48 CEST, Andrew Sutherland wrote:
>> We're seeing a problem in the Firefox OS Gaia e-mail app where if we manage to send our "A1 CAPABILITY" request to the gmail IMAP server (initial-TLS imaps/993) before it sends its greeting, it acts like it never hears the request.  Our not-so-clever state machine hangs until the connection times out.
> 
> Interesting. What my client is doing is patiently waiting for the initial response and checking whether the capability list is passed in there as a response code.

I think the ideal solution would be for everyone to start sending the CAPABILITY in the initial reply. Dovecot v2.x actually sends only a truncated CAPABILITY reply that contains only the extensions relevant to client before login. If you want to be sure not to break any clients, you could still do that and send the full capabilities list as a reply to the CAPABILITY command. The clients that understand the initial reply will also understand the updated OK [CAPABILITY ..] reply to LOGIN/AUTHENTICATE, while those that don't understand it will just request the CAPABILITY command directly.

We've identified at least one bug here and submitted a fix.  Not sure
the exact timeframe of rollout.

Brandon

On Fri, Mar 7, 2014 at 4:17 PM, Andrew Sutherland <asuth@mozilla.com> wrote:
> Aha!  Yes, the behavior does appear to be new and is manifesting in older
> releases that didn't experience this problem before*.  QA filed the bug with
> a log indicating this problem on 2/27 on https://bugzil.la/977867, but it's
> conceivable this was happening before that time.
>
> Thanks!
> Andrew
>
> * But that doesn't necessarily guarantee things given the high rate of
> platform change and platform backports occurring for Firefox OS.  In this
> case our IMAP implementation is unchanged and our TCPSocket implementation
> on top of Necko/NSS is unchanged.  I assume NSS changes and gets uplifts.
>
>
> On 03/07/2014 10:34 AM, Brandon Long wrote:
>
> Is this behavior new?
>
> We're just finishing the rollout of a new IMAP proxy, its possible its a bug
> in the new impl.
>
> Brandon
>
> On Mar 7, 2014 7:06 AM, "Andrew Sutherland" <asuth@mozilla.com> wrote:
>>
>> We're seeing a problem in the Firefox OS Gaia e-mail app where if we
>> manage to send our "A1 CAPABILITY" request to the gmail IMAP server
>> (initial-TLS imaps/993) before it sends its greeting, it acts like it never
>> hears the request.  Our not-so-clever state machine hangs until the
>> connection times out.
>>
>> There's more details and pcap dumps at https://bugzil.la/977867#c21 but
>> basically we sometimes manage to get that request in the same TCP packet as
>> the conclusion of TLS setup.  The TCP/TLS packet with the server greeting
>> ACKs that packet, so it's quite conceivable our request is managing to fall
>> into a very specific edge case/race.
>>
>> The spec does not seem to have a final word on this, although the state
>> flow diagram in http://tools.ietf.org/html/rfc3501#section-3 does seem to
>> imply that it's bad form to not wait for the server greeting.  While I think
>> it's clear that for gmail we need to wait for the server greeting, I am
>> wondering if this is just an unusual complication/bug of gmail's likely more
>> complicated server architecture.  I'd expect a server intentionally being
>> pedantic to issue a NO or BAD request...
>>
>> Thoughts / has anyone seen servers that intentionally or accidentally get
>> upset in cases like this?
>>
>> Thanks,
>> Andrew
>>
>> PS: We do have a lot of round-trip fat in our connection establishment, so
>> it's not the end of the world to explicitly wait for the server greeting in
>> all cases, but if it's just gmail (and we appropriately add some improved
>> timeout logic that triggers a persistent quirk for the server), I could see
>> it make sense to continue not waiting for the greeting.
>> _______________________________________________
>> Imap-protocol mailing list
>> Imap-protocol@u.washington.edu
>> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
>

On 03/07/2014 12:27 PM, Timo Sirainen wrote:
> On 7.3.2014, at 18.01, Jan Kundr?t <jkt@flaska.net> wrote:
>
>> Interesting. What my client is doing is patiently waiting for the initial response and checking whether the capability list is passed in there as a response code.

Our IMAP implementation has not gotten a lot of love.  We started from 
https://github.com/mscdex/node-imap/ as a base and this was simply its 
default behavior which worked, so we ran with it.  We have a backlog of 
things to do to improve our IMAP implementation, but we do want 
connection setup to be as fast as possible so it's nice to know the 
right thing to do when correcting immediate issues and longer term.


> I think the ideal solution would be for everyone to start sending the CAPABILITY in the initial reply. Dovecot v2.x actually sends only a truncated CAPABILITY reply that contains only the extensions relevant to client before login. If you want to be sure not to break any clients, you could still do that and send the full capabilities list as a reply to the CAPABILITY command. The clients that understand the initial reply will also understand the updated OK [CAPABILITY ..] reply to LOGIN/AUTHENTICATE, while those that don't understand it will just request the CAPABILITY command directly.

Yes, dovecot's behavior here is very nice.  We do understand and want to 
continue to understand LOGINDISABLED so we can treat the situation as an 
appropriate type of transient failure, so it's good for us to have that 
information.

However, if a server such as dovecot does this and we have persisted 
this knowledge and are on an imaps/993/initial-TLS connection, would it 
then be okay for us to attempt our login as our first step before having 
seen the greeting?  Assuming the server always sends the greeting 
regardless of our initial command, we can drop the connection and/or 
ignore the login failure when we see that there was a LOGINDISABLED command.

Thanks,
Andrew