mailing list archives

Another BINARY issue not specified by the current BINARY RFC: How to handle fetching BINARY[x] or BINARY.SIZE[x] when the MIME part exists but contains invalid base64/quoted-printable data? The possibilities I think would be:

a) Just ignore the invalid data and reply as best as you can. I'm not sure if this is actually useful in any case since the result will be invalid, and it might even cause security problems in case viruses try to exploit it at some point when virus scanner and IMAP server do the decoding differently. I remember reading a few years ago a lot of reports about this issue.

b) Reply NIL to BINARY[x] and 0 to BINARY.SIZE[x] and overall reply OK to the FETCH.

c) Abort and return NO to the FETCH, similarly to as for [UNKNOWNCTE] handling, except without that resp-code.

I implemented b) today for Dovecot, thinking that it would probably cause less client confusion than c) and would be more secure then a).

Hi Timo,

On 04/12/2012 11:12, Timo Sirainen wrote:
> Another BINARY issue not specified by the current BINARY RFC: How to handle fetching BINARY[x] or BINARY.SIZE[x] when the MIME part exists but contains invalid base64/quoted-printable data? The possibilities I think would be:
>
> a) Just ignore the invalid data and reply as best as you can. I'm not sure if this is actually useful in any case since the result will be invalid, and it might even cause security problems in case viruses try to exploit it at some point when virus scanner and IMAP server do the decoding differently. I remember reading a few years ago a lot of reports about this issue.
>
> b) Reply NIL to BINARY[x] and 0 to BINARY.SIZE[x] and overall reply OK to the FETCH.
>
> c) Abort and return NO to the FETCH, similarly to as for [UNKNOWNCTE] handling, except without that resp-code.

I think both b) and c) would be Ok. But b) is probably less likely to 
surprise client authors.

> I implemented b) today for Dovecot, thinking that it would probably cause less client confusion than c) and would be more secure then a).

Maybe it is worth to do b) but also return "UNKNOWNCTE"?

On 6.12.2012, at 16.17, Alexey Melnikov wrote:

> On 04/12/2012 11:12, Timo Sirainen wrote:
>> Another BINARY issue not specified by the current BINARY RFC: How to handle fetching BINARY[x] or BINARY.SIZE[x] when the MIME part exists but contains invalid base64/quoted-printable data? The possibilities I think would be:
>> 
>> a) Just ignore the invalid data and reply as best as you can. I'm not sure if this is actually useful in any case since the result will be invalid, and it might even cause security problems in case viruses try to exploit it at some point when virus scanner and IMAP server do the decoding differently. I remember reading a few years ago a lot of reports about this issue.
>> 
>> b) Reply NIL to BINARY[x] and 0 to BINARY.SIZE[x] and overall reply OK to the FETCH.
>> 
>> c) Abort and return NO to the FETCH, similarly to as for [UNKNOWNCTE] handling, except without that resp-code.
> 
> I think both b) and c) would be Ok. But b) is probably less likely to surprise client authors.
> 
>> I implemented b) today for Dovecot, thinking that it would probably cause less client confusion than c) and would be more secure then a).
> 
> Maybe it is worth to do b) but also return "UNKNOWNCTE"?

So:

a FETCH 1 BINARY[1]
* 1 FETCH (BINARY[1] NIL)
a OK [UNKNOWN-CTE] Invalid input?

I don't know how clients currently handle UNKNOWN-CTE. Maybe some would think that if it's returned then the server can never decode that content-transfer-encoding?..

On Mon, Dec 10, 2012 at 9:56 PM, Timo Sirainen <tss@iki.fi> wrote:

> On 6.12.2012, at 16.17, Alexey Melnikov wrote:
>
> > On 04/12/2012 11:12, Timo Sirainen wrote:
> >> Another BINARY issue not specified by the current BINARY RFC: How to
> handle fetching BINARY[x] or BINARY.SIZE[x] when the MIME part exists but
> contains invalid base64/quoted-printable data? The possibilities I think
> would be:
> >>
> >> a) Just ignore the invalid data and reply as best as you can. I'm not
> sure if this is actually useful in any case since the result will be
> invalid, and it might even cause security problems in case viruses try to
> exploit it at some point when virus scanner and IMAP server do the decoding
> differently. I remember reading a few years ago a lot of reports about this
> issue.
> >>
> >> b) Reply NIL to BINARY[x] and 0 to BINARY.SIZE[x] and overall reply OK
> to the FETCH.
> >>
> >> c) Abort and return NO to the FETCH, similarly to as for [UNKNOWNCTE]
> handling, except without that resp-code.
> >
> > I think both b) and c) would be Ok. But b) is probably less likely to
> surprise client authors.
> >
> >> I implemented b) today for Dovecot, thinking that it would probably
> cause less client confusion than c) and would be more secure then a).
> >
> > Maybe it is worth to do b) but also return "UNKNOWNCTE"?
>
> So:
>
> a FETCH 1 BINARY[1]
> * 1 FETCH (BINARY[1] NIL)
> a OK [UNKNOWN-CTE] Invalid input?
>
> I don't know how clients currently handle UNKNOWN-CTE. Maybe some would
> think that if it's returned then the server can never decode that
> content-transfer-encoding?..


I would think [PARSE] would be more appropriate than UNKNOWN-CTE.  Or just
stick with [ALERT].

Also, what if there is more than one message requested?  We usually just
fail to return the broken message and respond NO to the whole fetch, but
return all the non-broken messages.

Also, I wouldn't use "Invalid input", maybe its just my thinking, but I
would think the "input" is the IMAP command, but its an invalid message
data.

Brandon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20121210/accaf8bb/attachment.html>

On Tuesday, 11 December 2012 07:16:35 CEST, Brandon Long wrote:
> I would think [PARSE] would be more appropriate than 
> UNKNOWN-CTE.  Or just stick with [ALERT].

ALERT is reserved for situations which MUST be communicated to the user. If the IMAP server cannot decode a particular CTE, another response code shall be used -- this is a condition where the client shall make its decision on how to proceed, but definitely not by showing an ALERT to the user.

Please, don't use ALERT for this.

With kind regards,
Jan

Quoting Brandon Long <blong@google.com>:

> On Mon, Dec 10, 2012 at 9:56 PM, Timo Sirainen <tss@iki.fi> wrote:
>
>> a FETCH 1 BINARY[1]
>> * 1 FETCH (BINARY[1] NIL)
>> a OK [UNKNOWN-CTE] Invalid input?

As a client author, I would rather see this return with a NO tagged  
response.  This indicates that there is something wrong with the part  
and the user can be notified that the part is unavailable ("this part  
is broken! have sender re-send the message(?)") or, alternatively, the  
UI can hide the part.  In the above example, our client would simply  
disregard the UNKNOWN-CTE as irrelevant.

>> I don't know how clients currently handle UNKNOWN-CTE. Maybe some would
>> think that if it's returned then the server can never decode that
>> content-transfer-encoding?..
>
> I would think [PARSE] would be more appropriate than UNKNOWN-CTE.  Or just
> stick with [ALERT].

Agree with others that ALERT doesn't make sense.  My interpretation of  
ALERT is that its contents are independent of command interaction,  
especially since there is no temporal requirement that alerts are  
displayed when they are received.

Not sure if UNKNOWN-CTE is correct either since: in the example given,  
it's a known CTE that the server can handle. It is bad message data  
instead.

Although for that matter, PARSE doesn't seem to work either since it  
is defined as an inability to parse the (MIME) headers of a section.   
In this example, the headers are fine - it's the body that is broken.

I would throw out CORRUPTION as a possibility.

> Also, what if there is more than one message requested?  We usually just
> fail to return the broken message and respond NO to the whole fetch, but
> return all the non-broken messages.

See above.  I would agree that this is probably the best solution.

michael

On 11/12/2012 06:16, Brandon Long wrote:
>
> On Mon, Dec 10, 2012 at 9:56 PM, Timo Sirainen <tss@iki.fi 
> <mailto:tss@iki.fi>> wrote:
>
>     On 6.12.2012, at 16.17, Alexey Melnikov wrote:
>
>     > On 04/12/2012 11:12, Timo Sirainen wrote:
>     >> Another BINARY issue not specified by the current BINARY RFC:
>     How to handle fetching BINARY[x] or BINARY.SIZE[x] when the MIME
>     part exists but contains invalid base64/quoted-printable data? The
>     possibilities I think would be:
>     >>
>     >> a) Just ignore the invalid data and reply as best as you can.
>     I'm not sure if this is actually useful in any case since the
>     result will be invalid, and it might even cause security problems
>     in case viruses try to exploit it at some point when virus scanner
>     and IMAP server do the decoding differently. I remember reading a
>     few years ago a lot of reports about this issue.
>     >>
>     >> b) Reply NIL to BINARY[x] and 0 to BINARY.SIZE[x] and overall
>     reply OK to the FETCH.
>     >>
>     >> c) Abort and return NO to the FETCH, similarly to as for
>     [UNKNOWNCTE] handling, except without that resp-code.
>     >
>     > I think both b) and c) would be Ok. But b) is probably less
>     likely to surprise client authors.
>     >
>     >> I implemented b) today for Dovecot, thinking that it would
>     probably cause less client confusion than c) and would be more
>     secure then a).
>     >
>     > Maybe it is worth to do b) but also return "UNKNOWNCTE"?
>
>     So:
>
>     a FETCH 1 BINARY[1]
>     * 1 FETCH (BINARY[1] NIL)
>     a OK [UNKNOWN-CTE] Invalid input?
>
>     I don't know how clients currently handle UNKNOWN-CTE. Maybe some
>     would think that if it's returned then the server can never decode
>     that content-transfer-encoding?..
>
>
> I would think [PARSE]

PARSE seems to be Ok for this. It doesn't quite match the definition in 
RFC 3501, but it if fairly close.

> would be more appropriate than UNKNOWN-CTE.

After reviewing its definition, I tend to agree.

CORRUPTION also seems wrong: I think it is typically about mailbox 
storage, i.e. something a mail server administrator might be able to fix.

> Or just stick with [ALERT].

As per other responses, I think this one is wrong.

> Also, what if there is more than one message requested?  We usually 
> just fail to return the broken message and respond NO to the whole 
> fetch, but return all the non-broken messages.
>
> Also, I wouldn't use "Invalid input", maybe its just my thinking, but 
> I would think the "input" is the IMAP command, but its an invalid 
> message data.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20121213/b52fdbf3/attachment.html>

On 11/12/2012 11:55, Jan Kundr?t wrote:
> On Tuesday, 11 December 2012 07:16:35 CEST, Brandon Long wrote:
>> I would think [PARSE] would be more appropriate than UNKNOWN-CTE.  Or 
>> just stick with [ALERT].
>
> ALERT is reserved for situations which MUST be communicated to the 
> user. If the IMAP server cannot decode a particular CTE, another 
> response code shall be used -- this is a condition where the client 
> shall make its decision on how to proceed, but definitely not by 
> showing an ALERT to the user.
>
> Please, don't use ALERT for this.
+1.

I'm confused about the consensus against ALERT.

This is an error condition that doesn't have a correct error code defined,
how is the client supposed to know what the error is to tell the user?  The
server does know the exact error condition, however.

In general, I haven't understood the new error responses as "replacements"
for ALERT, as they're not backward compatible.  A client which gets an
error response is doesn't understand does what, nothing?  So if I change an
ALERT to a CANNOT, I'm just having clients which don't know CANNOT drop
information instead of helping the user.

I'm also not sure what the client is supposed to do in this situation
except display an error to the user.  Do you expect the client to fall back
to a non-binary fetch and hope it has better luck with decoding than the
server did?  And that's on top of the fact that the client doesn't even
know which message was broken.

Anyways, CORRUPTION to me implies errors accumulated in transport or
storage, which the user might attribute to the server, which is hardly the
case if the message was broken when the server acquired it.

Brandon


On Tue, Dec 11, 2012 at 12:36 PM, Michael M Slusarz
<slusarz@curecanti.org>wrote:

> Quoting Brandon Long <blong@google.com>:
>
>  On Mon, Dec 10, 2012 at 9:56 PM, Timo Sirainen <tss@iki.fi> wrote:
>>
>>  a FETCH 1 BINARY[1]
>>> * 1 FETCH (BINARY[1] NIL)
>>> a OK [UNKNOWN-CTE] Invalid input?
>>>
>>
> As a client author, I would rather see this return with a NO tagged
> response.  This indicates that there is something wrong with the part and
> the user can be notified that the part is unavailable ("this part is
> broken! have sender re-send the message(?)") or, alternatively, the UI can
> hide the part.  In the above example, our client would simply disregard the
> UNKNOWN-CTE as irrelevant.
>
>
>  I don't know how clients currently handle UNKNOWN-CTE. Maybe some would
>>> think that if it's returned then the server can never decode that
>>> content-transfer-encoding?..
>>>
>>
>> I would think [PARSE] would be more appropriate than UNKNOWN-CTE.  Or just
>> stick with [ALERT].
>>
>
> Agree with others that ALERT doesn't make sense.  My interpretation of
> ALERT is that its contents are independent of command interaction,
> especially since there is no temporal requirement that alerts are displayed
> when they are received.
>
> Not sure if UNKNOWN-CTE is correct either since: in the example given,
> it's a known CTE that the server can handle. It is bad message data instead.
>
> Although for that matter, PARSE doesn't seem to work either since it is
> defined as an inability to parse the (MIME) headers of a section.  In this
> example, the headers are fine - it's the body that is broken.
>
> I would throw out CORRUPTION as a possibility.
>
>
>  Also, what if there is more than one message requested?  We usually just
>> fail to return the broken message and respond NO to the whole fetch, but
>> return all the non-broken messages.
>>
>
> See above.  I would agree that this is probably the best solution.
>
> michael
>
>
> ______________________________**_________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman2.u.washington.**edu/mailman/listinfo/imap-**protocol<http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20121211/ec3c4586/attachment.html>

On 2012-12-11, at 12:47 PM, Brandon Long wrote:

> In general, I haven't understood the new error responses as "replacements" for ALERT, as they're not backward compatible.  A client which gets an error response is doesn't understand does what, nothing?  So if I change an ALERT to a CANNOT, I'm just having clients which don't know CANNOT drop information instead of helping the user.

ALERT almost always generates a blocking popup in the UI containing the raw error text string from the server.  This information is almost always useless to any end user, and just serves to annoy them.

There are only two cases where I generate an ALERT: for an administrative server shutdown (e.g. a MOTD style 'server down for maintenance until 0630' message), or when the user's auth credentials are not useable (e.g. 'your password has expired').

--lyndon

Quoting Brandon Long <blong@google.com>:

> I'm confused about the consensus against ALERT.
>
> This is an error condition that doesn't have a correct error code defined,
> how is the client supposed to know what the error is to tell the user?  The
> server does know the exact error condition, however.

The problem with ALERT is that the contents must ALWAYS be shown to  
the user.  There are all sorts of IMAP commands where error responses  
can (and should) be handled without the user knowing about it.

Imagine that a user opens a mailbox and the client begins to download  
messages in the background.  When it reaches the message with bad  
binary data, and the server throws an ALERT, this ALERT (e.g. "Bad  
message data") will be displayed while the user is likely still in  
mailbox-level view.  It is sure to elicit a "WTF?" given that the  
alert has no context to the activity the user is performing.

> I'm also not sure what the client is supposed to do in this situation
> except display an error to the user.  Do you expect the client to fall back
> to a non-binary fetch and hope it has better luck with decoding than the
> server did?

I definitely agree that, when the bad part is attempted to be  
displayed to the user, there should be some sort of error message  
displayed.  But the contents and timing of the error message should be  
the client's responsibility.  The server's job is to provide  
sufficient error code information so that the client can accurately  
and clearly explain to the user the problem.

> And that's on top of the fact that the client doesn't even
> know which message was broken.

We send body requests for each part separately for precisely this  
reason.  Slightly less efficient but necessary for correct error  
handling.  It's the same sort of error reporting vs. efficiency  
tradeoff as not using LITERAL+ when doing a MULTIAPPEND.

> Anyways, CORRUPTION to me implies errors accumulated in transport or
> storage, which the user might attribute to the server, which is hardly the
> case if the message was broken when the server acquired it.

I'll agree that as written (especially in the example), CORRUPTION  
seems to address the issue of something wrong from a server  
metadata/infrastructure POV rather than an issue with the contents of  
the message.  In the absence of an explicit BROKEN response code, we  
are probably trying to fit a square peg into a triangular hole.

michael

Quoting Lyndon Nerenberg <lyndon@orthanc.ca>:

> There are only two cases where I generate an ALERT: for an  
> administrative server shutdown (e.g. a MOTD style 'server down for  
> maintenance until 0630' message), or when the user's auth  
> credentials are not useable (e.g. 'your password has expired').

I've also seen ALERTs issued for over quota (or approaching quota) events.

michael