mailing list archives

Hello,

Given a fetch command for one message, with two attributes:

ABC FETCH 1 (FLAGS RFC822.SIZE)

There is usually one response per message, including both attributes:

* 1 FETCH (FLAGS (\Seen) RFC822.SIZE 44827)

Would it be compliant to return instead two responses, one for each attribute?

* 1 FETCH (FLAGS (\Seen))
* 1 FETCH (RFC822.SIZE 44827)

Thanks in advance.

--

On Sunday, November 13, 2011 9:00 PM, "Petite Abeille" <petite_abeille@mac.com> wrote:
> Hello,
> 
> Given a fetch command for one message, with two attributes:
> 
> ABC FETCH 1 (FLAGS RFC822.SIZE)
> 
> There is usually one response per message, including both attributes:
> 
> * 1 FETCH (FLAGS (\Seen) RFC822.SIZE 44827)
> 
> Would it be compliant to return instead two responses, one for each attribute?
> 
> * 1 FETCH (FLAGS (\Seen))
> * 1 FETCH (RFC822.SIZE 44827)

Compliant or not, I can bet you it will break a lot of clients.

Bron.
-- 
  Bron Gondwana
  brong@fastmail.fm

On 13.11.2011, at 22.00, Petite Abeille wrote:

> Given a fetch command for one message, with two attributes:
> 
> ABC FETCH 1 (FLAGS RFC822.SIZE)
> 
> There is usually one response per message, including both attributes:
> 
> * 1 FETCH (FLAGS (\Seen) RFC822.SIZE 44827)
> 
> Would it be compliant to return instead two responses, one for each attribute?
> 
> * 1 FETCH (FLAGS (\Seen))
> * 1 FETCH (RFC822.SIZE 44827)


Possibly would be compliant, but you'd most likely also break tons of clients by doing that, so it's a rather pointless thing to do.

On Sun, 13 Nov 2011, Petite Abeille wrote:
> Would it be compliant to return instead two responses, one for each
> attribute?

It is completely compliant, and any client which does not support it is
broken.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

>> Would it be compliant to return instead two responses, one for each attribute?
>> 
>> * 1 FETCH (FLAGS (\Seen))
>> * 1 FETCH (RFC822.SIZE 44827)
> 

On Nov 13, 2011, at 9:14 PM, Bron Gondwana wrote:

> Compliant or not, I can bet you it will break a lot of clients.

On Nov 13, 2011, at 9:17 PM, Timo Sirainen wrote:

> Possibly would be compliant, but you'd most likely also break tons of clients by doing that, so it's a rather pointless thing to do.

Ok, fair enough.

But, on the other hand, doesn't it happen already? For example when a BODY fetch triggers a FLAGS fetch? 

ABC 1 FETCH (BODY[HEADER] )
* 1 FETCH (BODY[HEADER] {342}
...
* 1 FETCH (FLAGS (\Seen))
ABC OK FETCH completed

On 11/13/2011 09:37 PM, Petite Abeille wrote:
> But, on the other hand, doesn't it happen already? For example when a BODY fetch triggers a FLAGS fetch?

It's safe for flags+rest. but not so safe for e.g. uid+rest:

* 42 FETCH (UID 100)
* 42 FETCH (RFC822.SIZE 12345)

Arnt

On Sun, Nov 13, 2011 at 09:37:48PM +0100, Petite Abeille wrote:
> >> * 1 FETCH (FLAGS (\Seen))
> >> * 1 FETCH (RFC822.SIZE 44827)
> On Nov 13, 2011, at 9:14 PM, Bron Gondwana wrote:
> 
> > Compliant or not, I can bet you it will break a lot of clients.
> 
> But, on the other hand, doesn't it happen already? For example when a BODY fetch triggers a FLAGS fetch? 
> 
> ABC 1 FETCH (BODY[HEADER] )
> * 1 FETCH (BODY[HEADER] {342}
> ...
> * 1 FETCH (FLAGS (\Seen))
> ABC OK FETCH completed

Not with Cyrus - we collect up all the data about which fetches will
be required first, and then iterate the mailbox, spitting them out
in order.

This is quite specific, and for a reason - we calculate what flag
changes might be needed and do them with a mailbox lock, then we
release the lock before sending the response, so that other clients
are not waiting on the lock while network traffic is going on.

But it does have the other advantage of never generating duplicate
FETCH responses too.

Bron.

On Sun, 13 Nov 2011, Arnt Gulbrandsen wrote:
> It's safe for flags+rest. but not so safe for e.g. uid+rest:
> * 42 FETCH (UID 100)
> * 42 FETCH (RFC822.SIZE 12345)

Those pair responses is completely compliant. Any client which has the
slightest problem with it is broken.

Servers can do this. There are servers that do it.

The UID value does not identify the message to which it applies. The
message sequence number identifies the message.

Now, with that said, RFC 3501 says:
       The number after the "*" in an untagged FETCH response is always a
       message sequence number, not a unique identifier, even for a UID
       command response.  However, server implementations MUST implicitly
       include the UID message data item as part of any FETCH response
       caused by a UID command, regardless of whether a UID was specified
       as a message data item to the FETCH.

The second sentence seems to say that a server can't do:
 	* 42 FETCH (UID 100)
 	* 42 FETCH (RFC822.SIZE 12345)
in response to a UID FETCH n RFC822.SIZE command. That is a reasonable
interpretation of that statement.

HOWEVER!! Suppose the server did UID FETCH n ENVELOPE, and the server
decided to send the RFC822.SIZE as well. In that case, the FETCH response
with the RFC822.SIZE is (as far as the protocol is concerned) not "caused"
by the UID FETCH; and therefore that pair is alright.

This effectively means that a client MUST NOT assume that a UID will be
carried in a FETCH response; only that, somewhere in there, a UID will be
given. It must deal with message sequence number 42, of which UID 100 is a
property.

It's a difference that makes no real difference.

In other words even though it seems UID FETCH 100 RFC822.SIZE may not be
responded with:
 	* 42 FETCH (UID 100)
 	* 42 FETCH (RFC822.SIZE 12345)
the client must be able to handle that response any way. The server
certainly can do (albeit it would be pointless):
 	* 42 FETCH (UID 100 RFC822.SIZE 12345)
 	* 42 FETCH (RFC822.SIZE 12345)

The bottom line is that ANY untagged response is valid at ANY time. Client
implementations MUST be able to coalesce multiple untagged responses for
the same message based upon message sequence number.

It saves some bits on the wire if a server coalesces response data. It is
therefore very nice to have, and most servers do it. It is also absolutely
NOT a requirement that any server do so.

It may seem to be alright to write a client that ignores message sequence
numbers and only uses UIDs. A client author that does this deserves to
have his client break, and have his customers complain at him.

Server authors can already easily make your client break by violating
IMAP. Do you want to give them the power to make your client break by
obeying IMAP better than you do?

In conclusion:

Server authors should avoid doing things that are pointless and bandwidth
consuming.

Client authors MUST handle something that seems pointless from the server,
because some times a server implementation has no choice due to
circumstances out of its control.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Sun Nov 13 21:30:49 2011, Mark Crispin wrote:
> On Sun, 13 Nov 2011, Arnt Gulbrandsen wrote:
>> It's safe for flags+rest. but not so safe for e.g. uid+rest:
>> * 42 FETCH (UID 100)
>> * 42 FETCH (RFC822.SIZE 12345)
> 
> Those pair responses is completely compliant. Any client which has  
> the
> slightest problem with it is broken.
> 
> Servers can do this. There are servers that do it.

As a note without conclusion, there are two variations of this which  
are significant when considered as the response to a UID FETCH:

1) Sending everything in one response, but not having UID first.

* 42 FETCH (RFC822.SIZE 12345 UID 100)

This is entirely legitimate, and trips up some clients.

2) Sending as Mark's example, but reversed:

* 42 FETCH (RFC822.SIZE 12345)
* 42 FETCH (UID 100 ENVELOPE (...))

This will trip up some clients; but in practise the RFC822.SIZE  
response will simply be discarded. Still, this is particularly  
pathological anyway.

In pragmatic terms, if I were doing a revision of IMAP, I'd have  
these as MUST NOT send, SHOULD accept. (That is, I'd document that  
existing deployed clients choke on such constructs, therefore these  
actively harm interoperability; on the other hand servers exist that  
send such things, and clients ought to be capable to accepting it). I  
know this is nastier than people adhering to the specification in the  
first place, but I think it'd be more practical at this stage.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 14 Nov 2011, Dave Cridland wrote:
> 1) Sending everything in one response, but not having UID first.
> * 42 FETCH (RFC822.SIZE 12345 UID 100)
> This is entirely legitimate, and trips up some clients.

Such clients deserve to be tripped up.

> 2) Sending as Mark's example, but reversed:
> * 42 FETCH (RFC822.SIZE 12345)
> * 42 FETCH (UID 100 ENVELOPE (...))
> This will trip up some clients; but in practise the RFC822.SIZE
> response will simply be discarded. Still, this is particularly
> pathological anyway.

It is pathological if RFC822.SIZE was not requested by the UID FETCH. It
is forbidden by the statement that I quoted in RFC 3501.

> In pragmatic terms, if I were doing a revision of IMAP, I'd have
> these as MUST NOT send, SHOULD accept.

I disagree.

That statement in RFC 3501 was added because certain individuals
threatened to poo their diapers otherwise. It has caused more problems
than it solved. It helped create a fantasy that message sequence numbers
can be ignored. That is not the IMAP model, and a great deal of IMAP's
functionality is lost by doing so. If you really want that, use HTTP.

UID is not an access key. It is a message property. In the examples above,
RFC822.SIZE 12345 and UID 100 are both properties of message 42.
RFC822.SIZE 12345 is not a property of UID 100.

Now, it may be useful to determine that the message with property UID 100
also has the property RFC822.SIZE 12345. But that only happens through the
common access key of message 42.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon Nov 14 16:36:34 2011, Mark Crispin wrote:
> On Mon, 14 Nov 2011, Dave Cridland wrote:
>> 1) Sending everything in one response, but not having UID first.
>> * 42 FETCH (RFC822.SIZE 12345 UID 100)
>> This is entirely legitimate, and trips up some clients.
> 
> Such clients deserve to be tripped up.
> 
> 
I don't disagree... But I'm also pragmatic enough to consider that  
insufficient motivation to avoid tripping them up if practical.


>> 2) Sending as Mark's example, but reversed:
>> * 42 FETCH (RFC822.SIZE 12345)
>> * 42 FETCH (UID 100 ENVELOPE (...))
>> This will trip up some clients; but in practise the RFC822.SIZE
>> response will simply be discarded. Still, this is particularly
>> pathological anyway.
> 
> It is pathological if RFC822.SIZE was not requested by the UID  
> FETCH. It
> is forbidden by the statement that I quoted in RFC 3501.
> 
> 
Well, it's just plain wrong if it's been requested, and it's weird if  
it hasn't been.

> UID is not an access key. It is a message property. In the examples  
> above,
> RFC822.SIZE 12345 and UID 100 are both properties of message 42.
> RFC822.SIZE 12345 is not a property of UID 100.
> 
> Now, it may be useful to determine that the message with property  
> UID 100
> also has the property RFC822.SIZE 12345. But that only happens  
> through the
> common access key of message 42.

Again, I don't disagree - even if internally in Polymer, the UID *is*  
the primary key.

But it seems a relatively simple way of improving interoperability  
without actively causing any worse problems.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

On Mon, 14 Nov 2011, Dave Cridland wrote:
>> Such clients deserve to be tripped up.
> I don't disagree... But I'm also pragmatic enough to consider that
> insufficient motivation to avoid tripping them up if practical.

If you were the Very Big Corporation Of Monopoly, you would be happy to
trip up those clients in order to drive them out of business.

You already trip up other vendors' products with your violations of the
standard. The problem is that gets you flamed and may attract unwelcome
attention by regulators. It's better, if you can arrange it, to trip up
other vendors' products with your unexpected compliance with the standard,

> Well, it's just plain wrong if it's been requested, and it's weird if
> it hasn't been.

Yes. On the other hand, a client that implemented IMAP according to the
standard would have no problem either way.

> Again, I don't disagree - even if internally in Polymer, the UID *is*
> the primary key.

Polymer is not the only implementation with that internal implementation.
However, that leads to two deadly traps for incautious implementors (and
I know that you are not one of these and certainly do it right in
Polymer):

The first trap is to confuse your internal implementation with IMAP. Time
and time again, I have seen internal implementations that were SIMILAR to
IMAP UIDs, but did not have the same precise semantics. Not good.

The second trap is failing to implement message sequence numbers as a
primary key in the client-facing part. This may be as simple as a vector,
indexed by message sequence number, to the internal key. You may, if and
ONLY if there is an exact sematic mapping between IMAP UID and internal
key, shortcut the UID->MSN->internal mapping.

Too often, I see the shortcut when the semantics aren't the same, and I
see truly bizarre (and dysfunctional!) implementations of MSN.

I won't tell you about the implementation that, for each command,
enumerated all the UIDs and then did a bubble sort in order to implement
MSNs (and UID ranges). Oh I guess I just did. Sorry if that made you puke
your lunch.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

On Mon, Nov 14, 2011 at 12:37:42PM -0800, Mark Crispin wrote:
> The second trap is failing to implement message sequence numbers as a
> primary key in the client-facing part. This may be as simple as a vector,
> indexed by message sequence number, to the internal key. You may, if and
> ONLY if there is an exact sematic mapping between IMAP UID and internal
> key, shortcut the UID->MSN->internal mapping.
> 
> Too often, I see the shortcut when the semantics aren't the same, and I
> see truly bizarre (and dysfunctional!) implementations of MSN.
> 
> I won't tell you about the implementation that, for each command,
> enumerated all the UIDs and then did a bubble sort in order to implement
> MSNs (and UID ranges). Oh I guess I just did. Sorry if that made you puke
> your lunch.

The problem is that the MID is a MVCC snapshot of a particularly un-useful
view of the messages (strictly ascending UID order), with a view lifetime
which is defined, but is very dependent on the specific commands you issue,
so if you want to do one of those actions, you need to handle the snapshot
jumping forwards.

There's no way to get a snapshot of any OTHER view of the messages, so
unless you want the view that MID gives you, you're kinda screwed.  You
wind up having to keep a MID to UID mapping even if you don't actually
care about MIDs at all, and you need to update all the MIDs any time
something gets expunged from somewhere in the middle.

In other words, it's an impedence mismatch to what client writers want, and
probably what server writers want too.

They get it wrong because they don't think the way the IMAP protocol thinks,
and they don't think that way because frankly, it's a pretty scummy data
model unless you want to do approximately one and only one thing with the
data: access it in UID order.

Bron.