mailing list archives

I thought this might be a good list to ask a simple, but admittedly
subjective question: If Mark Crispin was creating IMAP from scratch, in
the world of today, would it still be a line based protocol like it was
with RFC3501, or would he have gone with something more stateless, like
a JSON-RPC paradigm, like JMAP?

For quick reference:

https://tools.ietf.org/html/rfc3501

Or JMAP:

http://jmap.io/spec.html

Or my own bastardized protocol used for webmail to server access, which
I created a few years back:

https://github.com/lavabit/magma.classic/raw/master/docs/magma.web.api.pdf

L~

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150307/c129063a/attachment.sig>

Mark really liked Lisp and terminal servers. I think that answers the 
question, but perhaps not for reasons that interest you, right?

Arnt

If he were around, this question would have started a fun argument.
I miss that guy.

snow.


On 3/7/2015 7:14 AM, Ladar Levison wrote:
> I thought this might be a good list to ask a simple, but admittedly
> subjective question: If Mark Crispin was creating IMAP from scratch, in
> the world of today, would it still be a line based protocol like it was
> with RFC3501, or would he have gone with something more stateless, like
> a JSON-RPC paradigm, like JMAP?
>
> For quick reference:
>
> https://tools.ietf.org/html/rfc3501
>
> Or JMAP:
>
> http://jmap.io/spec.html
>
> Or my own bastardized protocol used for webmail to server access, which
> I created a few years back:
>
> https://github.com/lavabit/magma.classic/raw/master/docs/magma.web.api.pdf
>
> L~
>
>
>
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150307/fb819105/attachment.html>

On 3/7/2015 6:14 AM, Ladar Levison wrote:
> I thought this might be a good list to ask a simple, but admittedly
> subjective question: If Mark Crispin was creating IMAP from scratch, in
> the world of today, would it still be a line based protocol like it was
> with RFC3501, or would he have gone with something more stateless, like
> a JSON-RPC paradigm, like JMAP?

If Mark was redesigning IMAP today, I imagine it would end up looking 
more or less like IMAP looks today with the biggest changes being some 
IMAP extensions being mandatory and the entire protocol (except message 
literals) being UTF-8.

 From reading his messages in this mailing list, he would focus on 
supporting use cases of clients, but primarily what he thinks a "good" 
IMAP client looks like--unlike many others here, he was fully insistent 
on message sequence numbers being the only right way to do things. A 
stateful, line-based protocol would be far simpler for clients to 
implement (particularly since I also get the impression that he would 
have eschewed needing to use several layers of frameworks).

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth

> I thought this might be a good list to ask a simple, but admittedly
> subjective question: If Mark Crispin was creating IMAP from scratch, in
> the world of today, would it still be a line based protocol like it was
> with RFC3501, or would he have gone with something more stateless, like
> a JSON-RPC paradigm, like JMAP?

In addition to mail-specific aspects, something like IMAP needs to
solve the problem of (concurrent) access to remote storage (over a
network).

Looking at current solutions to that problem in its generic form may
suggest stateful is the way to go. (In particular the evolution of
NFS.)

Regards,

	- Joel

On 03/07/2015 04:14 AM, Ladar Levison wrote:
> I thought this might be a good list to ask a simple, but admittedly
> subjective question: If Mark Crispin was creating IMAP from scratch, in
> the world of today, would it still be a line based protocol like it was
> with RFC3501, or would he have gone with something more stateless, like
> a JSON-RPC paradigm, like JMAP?

He'd make changes, but knowing Mark, they would probably not be what 
you'd expect.

IMAP 2 came after the advent of RPC mechanisms, and if Mark had wanted 
to use one, he would have.  Since I've been around IMAP, CORBA, XML-RPC, 
BEEP, "REST", and others have come into and out of vogue.  Mark didn't 
use them.  He also didn't use SMTP-style dot stuffing.

Swapping in JSON in place of s-expressions is reasonable, so maybe we'd 
see less parens and more square brackets and commas  Yet IMAP has a 
strange grammar that looks like s-expressions, parses as s-expressions, 
yet is spelled out byte for byte without free whitespace, a hallmark of 
JSON and s expressions.  So it's hard to say what Mark would have done.

IMAP had counted-length strings way before HTTP, and IMAP uses them more 
pervasively.  IMAP 2 also happened after early RPC protocols had started 
to appear.  I think if Mark had intended a non-line protocol underneath 
IMAP, again, he could have.

I recall Mark commenting once about the reality of TCP connections being 
a lot less reliable than when the protocol was invented.  If he was 
starting over, he'd have to take that into account.  I don't know how 
he'd solve it.  I don't know how I'd solve it.

The LIST and LSUB commands were a compromise nobody loves.  Mark 
commented on one occasion that he'd figured out The Right Thing, an 
s-expression notation instead of OS-style paths.  This would have helped.

Mark once said he might have implemented folders purely with flags, but 
he ran out of flag bits in messages in one of the older (pre-mbx) 
mailbox formats.  If you're trying to do archeology in this, it was in 
the context of Gmail being different than other implementations, and I 
think the discussion was on this list.

Sometimes, what others viewed as warts, Mark viewed as features.  One 
reason for this is that Mark's server supported a bunch of different 
mailbox formats with varying levels of feature support.  (No reliable 
UIDs on MH mailboxes, classic Unix mbox performance is terrible, etc.) 
Mark would oppose things that couldn't be implemented in those, so UW 
was never going to get CONDSTORE or QRESYNC support.  A MOVE command was 
routinely opposed by Mark, in part because it was essentially impossible 
for correct implementation in most of the per-file mailbox formats, but 
also because he just didn't get Trash folders.

And I thought it odd that his imapd reported all your files in your home 
directory as mailboxes, each with a single message, but that's how Mark 
wanted it (as I learned the hard way).

This is a long-winded way of saying, if Mark were creating IMAP today, 
he'd still be worrying about how to implement it on TOPS-20, just in 
case.  Because that's who he was.

Tim

On Mar 7, 2015, at 4:14 AM, Ladar Levison <ladar@lavabitllc.com> wrote:

> I thought this might be a good list to ask a simple, but admittedly
> subjective question: If Mark Crispin was creating IMAP from scratch, in
> the world of today, would it still be a line based protocol like it was
> with RFC3501, or would he have gone with something more stateless, like
> a JSON-RPC paradigm, like JMAP?

[ Coming in a bit late, I realize ... ]

For MRC, state was king.  And for a remote mailbox access service, he believed state was critical to building an efficient protocol.

He would have scoffed at comments about 'line' vs 'streaming' and 'text' vs 'binary' as indicative of the spewer of such nonsense being beneath contempt ;-)  What Mark got, and what so many other self-proclaimed experts in this discussion are blissfully oblivious to, is that it's the RTTs that kill an interactive protocol, and nothing else.  He firmly believed that every IMAP client developer should be forced to work behind a 1200 bps (yes, 1200) network link.  He rightly surmised that that was the only way to ensure an IMAP client would make the most efficient use of the protocol.  And I agree completely, although I did try to convince him that 9600 bps was perhaps a reasonable compromise.

Many times we talked about what the son-of-IMAP might look like.  For quite a few years I have been bouncing around an idea for an MUA access protocol.  Conceptually it is very different from IMAP, but it retains the philosophy in many ways.  I had some wonderful nights having my skin blistered by his critiques.  Fortunately there was a lot of beer at hand to quench the flames :-) The once thing we did - almost - come to agreement on was that a purely Sexpr-based protocol syntax was the right way to do it.  He always got grumpy talking about what he considered the deficiencies and compromises in the IMAP syntax that were 'forced' on him by others.  (How much of that was actually true I can't speak to.  But I know there are people still likely reading this mailing list who mightily pissed him off over what he perceived to be political manipulation of the spec for corporate gain.) (I also can't believe that he didn't just tell them to "see figure one," but there you go.)

What I took away from our conversations was this.  He thought the explosion of IMAP extensions pointed to a fundamental flaw in the design of the protocol, one that needed fixing.  He could not pin down, to his own satisfaction, what that flaw was (or flaws were); he had his own misgivings about some of the design, but he was never forthcoming to me about what he thought they were.  (He didn't want to criticize anything until he knew he could shred it with 100% validity, and that included his own work.)  But he also thought at least some of the extensions were due to pure laziness and ineptitude on the part of client writers.  (He could peel paint with that topic of conversation!)

I miss the cranky bastard, with his guns and politics and libertarian take on life :-)  We were polar opposites on the first two of those three (maybe all of them), but he respected what you had to say, even as he let loose with how he thought you were a complete idiot and did not deserve to grace a keyboard with your fingers :-)

As April approaches, I hope you will all join me in raising a glass of something uncomfortable in his memory, while you read one of his great works: http://www.ietf.org/rfc/rfc748.txt

Especially the young turks recently gracing us with their presence.  They might learn much.  Or not.

--lyndon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 817 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150310/fb9114cd/attachment.sig>

On Sat, Mar 7, 2015 at 3:18 PM, Arnt Gulbrandsen
<arnt@gulbrandsen.priv.no> wrote:
> Mark really liked Lisp and terminal servers.

IIRC, the first version of Mark's toolkit was written in Lisp. So
probably with today's abundance in Lisp like languages, he would have
picked one and write it in Lisp.

The OP also asked whether it would have been more stateless. Well,
given the criticism that Mark exercised on how students do not learn
how to maintain state in their programs, it seems to me that he would
have gone stateless. He would have been Mark if he did.

-- 
"If technology is your thing plan to die reading manuals" --Gene Woolsey

On Sun, Mar 8, 2015, at 06:25 AM, Joshua Cranmer wrote:
> On 3/7/2015 6:14 AM, Ladar Levison wrote:
> > I thought this might be a good list to ask a simple, but admittedly
> > subjective question: If Mark Crispin was creating IMAP from scratch,
> > in the world of today, would it still be a line based protocol like
> > it was with RFC3501, or would he have gone with something more
> > stateless, like a JSON-RPC paradigm, like JMAP?
>
> If Mark was redesigning IMAP today, I imagine it would end up looking
> more or less like IMAP looks today with the biggest changes being some
> IMAP extensions being mandatory and the entire protocol (except
> message literals) being UTF-8.
>
>  From reading his messages in this mailing list, he would focus on
>  supporting use cases of clients, but primarily what he thinks a
>  "good" IMAP client looks like--unlike many others here, he was fully
>  insistent on message sequence numbers being the only right way to do
>  things. A stateful, line-based protocol would be far simpler for
>  clients to implement (particularly since I also get the impression
>  that he would have eschewed needing to use several layers of
>  frameworks).

Yeah, I would have been arguing very strongly against this - at least
with the whole "you can't tell the client about expunges" because it
changes sequence numbers problem - because it doesn't interact well with
CONDSTORE and friends, and it requires the server to keep COW state on
the entire mailbox for each connected client.

Of course, I'm not Mark Crispin :)  I don't like anything which forces
either end to do more work than it actually needs - and IMAP has a pile
of required work that many clients don't actually use (like the UNSEEN
and RECENT values which aren't used by many clients).  There's no way
for a client specify that it doesn't need the server to do the
bookkeeping on its behalf.

Bron.


-- 
  Bron Gondwana
  brong@fastmail.fm

Messages in files is itself kind of old-school.  There's still a place
for MUA/MDA separation, but perhaps the MDA should just be an SQL or
NoSQL database; dump all the messages in MySQL.  Then the "access
protocol" would simply be the appropriate set of queries for whatever
the client is trying to achieve.  The DB schema would define the protocol.

Triggers in the DB could do the thread calculation.

And let the DB guys solve the intermittent connection problems :-).
Oracle, for instance, has a "Mobile Server" product that synchonizes a
local cache on a mobile device with a remote Oracle DB, including SSL
encryption on the sync connection, and data compression.

Bill

On Wed, Mar 11, 2015, at 01:23 PM, Lyndon Nerenberg wrote:
> 
> On Mar 7, 2015, at 4:14 AM, Ladar Levison <ladar@lavabitllc.com> wrote:
> 
> > I thought this might be a good list to ask a simple, but admittedly
> > subjective question: If Mark Crispin was creating IMAP from scratch, in
> > the world of today, would it still be a line based protocol like it was
> > with RFC3501, or would he have gone with something more stateless, like
> > a JSON-RPC paradigm, like JMAP?
> 
> [ Coming in a bit late, I realize ... ]
> 
> For MRC, state was king.  And for a remote mailbox access service, he believed state was critical to building an efficient protocol.
> 
> He would have scoffed at comments about 'line' vs 'streaming' and 'text' vs 'binary' as indicative of the spewer of such nonsense being beneath contempt ;-)  What Mark got, and what so many other self-proclaimed experts in this discussion are blissfully oblivious to, is that it's the RTTs that kill an interactive protocol, and nothing else.  He firmly believed that every IMAP client developer should be forced to work behind a 1200 bps (yes, 1200) network link.  He rightly surmised that that was the only way to ensure an IMAP client would make the most efficient use of the protocol.  And I agree completely, although I did try to convince him that 9600 bps was perhaps a reasonable compromise.

This has been really good for us designing JMAP, because we are in Australia and our servers in New York, so latency is a real consideration for us.  Round trips are a big deal.

> Many times we talked about what the son-of-IMAP might look like.  For quite a few years I have been bouncing around an idea for an MUA access protocol.  Conceptually it is very different from IMAP, but it retains the philosophy in many ways.  I had some wonderful nights having my skin blistered by his critiques.  Fortunately there was a lot of beer at hand to quench the flames :-) The once thing we did - almost - come to agreement on was that a purely Sexpr-based protocol syntax was the right way to do it.  He always got grumpy talking about what he considered the deficiencies and compromises in the IMAP syntax that were 'forced' on him by others.  (How much of that was actually true I can't speak to.  But I know there are people still likely reading this mailing list who mightily pissed him off over what he perceived to be political manipulation of the spec for corporate gain.) (I also can't believe that he didn't just tell them to "see figure one," but there you go.)

Have you had a look at jmap.io?  I would have loved to see Mark's feedback on it.  I suspect sitting down and talking over beers would have been more valuable than talking over email, because describing things over email is hard.

JMAP is very much inspired by IMAP and built on the raw concepts.  The one place where Mark and I disagreed a lot was on the one canonical ordering (last arrived) for the mailbox, and all the state being easy to use if you wanted that ordering, but impossible or very hard for everything else.  Trying to get an efficient view on something OTHER than last arrived on a million message mailbox is crazy with IMAP.

Which means copying in old messages from an archive mixes things up.  Most clients seem to want to sort to either INTERNALDATE (received date) or the Date: header.

So in JMAP we actually make the server keep more state so that we can provide efficient updates to whichever that the client wants, including sorts or searches (which allows things like a view of unread messages, or pinned messages on top as a mailbox view)

> What I took away from our conversations was this.  He thought the explosion of IMAP extensions pointed to a fundamental flaw in the design of the protocol, one that needed fixing.  He could not pin down, to his own satisfaction, what that flaw was (or flaws were); he had his own misgivings about some of the design, but he was never forthcoming to me about what he thought they were.  (He didn't want to criticize anything until he knew he could shred it with 100% validity, and that included his own work.)  But he also thought at least some of the extensions were due to pure laziness and ineptitude on the part of client writers.  (He could peel paint with that topic of conversation!)

I'd say there's half and half here.  The fact that you could only do windowing by message number, and everything else was full mailbox search/sort was a big issue for clients.  And getting updates to everything you're interested in.

> I miss the cranky bastard, with his guns and politics and libertarian take on life :-)  We were polar opposites on the first two of those three (maybe all of them), but he respected what you had to say, even as he let loose with how he thought you were a complete idiot and did not deserve to grace a keyboard with your fingers :-)

Yep, I remember those!

> As April approaches, I hope you will all join me in raising a glass of something uncomfortable in his memory, while you read one of his great works: http://www.ietf.org/rfc/rfc748.txt
>
> Especially the young turks recently gracing us with their presence.  They might learn much.  Or not.

I think a glass of something uncomfortable will work well with reading that one!  I'll be on a boat between Melbourne and Tasmania on that night, and I'll definitely drink a glass of something horrible in his memory.

Bron.


-- 
  Bron Gondwana
  brong@fastmail.fm

On 3/10/15 7:23 PM, Lyndon Nerenberg wrote:

> And I agree completely, although I did try to convince him that 9600 bps was perhaps a reasonable compromise.

And I would bet you were unsuccessful in this, much to Mark's glee.

Mark had some unusual opinions, but he had pretty good reasons for them. 
  Well, except using NIL in C programs, that's just weird.

We are all worse off for the loss of him.

Tim

On Mar 10,  7:23pm, Lyndon Nerenberg wrote:
}
} I miss the cranky bastard, with his guns and politics and libertarian
} take on life :-) We were polar opposites on the first two of those
} three (maybe all of them), but he respected what you had to say, even
} as he let loose with how he thought you were a complete idiot and did
} not deserve to grace a keyboard with your fingers :-)

Back when I was still writing email client software, my colleagues and I
all had business cards printed up with "job titles" based on something
Mark had said to us on the mailing list.  I felt strangely honored to
have been referred to as a "Serious Threat" (to the integrity of the
IMAP protocol) just for some comment I made; I don't even recall what.
One of my colleagues was "Pedantic Crank" for pointing out something
inconsistent in the IMAP4 grammar.

Ah, those were the days.

(When replying with a headache you write things upside down)

On Sat, Mar 7, 2015 at 3:23 PM, Yiorgos Adamopoulos
<yiorgos.adamopoulos@gmail.com> wrote:
> The OP also asked whether it would have been more stateless. Well,
> given the criticism that Mark exercised on how students do not learn
> how to maintain state in their programs, it seems to me that he would
> have gone stateless. He would have been Mark if he did.

CORRECTION: I believe that Mark would still write it the way he did
with regards to state (i.e. not stateless). He wouldn't have been Mark
if he did not.

-- 
"If technology is your thing plan to die reading manuals" --Gene Woolsey

On Mon, Mar 9, 2015, at 03:13 AM, Bill Janssen wrote:
> Messages in files is itself kind of old-school.  There's still a place
> for MUA/MDA separation, but perhaps the MDA should just be an SQL or
> NoSQL database; dump all the messages in MySQL.  Then the "access
> protocol" would simply be the appropriate set of queries for whatever
> the client is trying to achieve.  The DB schema would define the
> protocol.

So this is pretty much what JMAP is actually, it's a data model query
and update protocol which is designed around batch operations (and batch
queries) to reduce chattyness.

> Triggers in the DB could do the thread calculation.

Roughly what I'm doing for the non-Gmail case in the JMAP proxy (which
is sqlite3 backed) where I generate an ID from the envelope
datastructure, and then do a messageid lookup on the In-Reply-To (cheap
and nasty implementation, a better one would be what we do in Cyrus)

https://github.com/jmapio/jmap-perl/blob/master/JMAP/ImapDB.pm#L279

https://github.com/brong/cyrus-imapd/blob/07381525b6dcaa57fb1f9110ffec83d3618a0b96/imap/message.c#L3036

> And let the DB guys solve the intermittent connection problems :-).
> Oracle, for instance, has a "Mobile Server" product that synchonizes a
> local cache on a mobile device with a remote Oracle DB, including SSL
> encryption on the sync connection, and data compression.

The problem is, DB protocols are pretty chatty.  And you need to solve
authencation too.

Seriously, I think we've pretty much done what you're suggesting in
JMAP, but we've solved the latency problems that chatty protocols have
when you don't have a nice short piece of solid copper between you and
the server as well.

Bron.

-- 
  Bron Gondwana
  brong@fastmail.fm

Bron Gondwana <brong@fastmail.fm> wrote:

> > And let the DB guys solve the intermittent connection problems :-).
> > Oracle, for instance, has a "Mobile Server" product that synchonizes a
> > local cache on a mobile device with a remote Oracle DB, including SSL
> > encryption on the sync connection, and data compression.
> 
> The problem is, DB protocols are pretty chatty.  And you need to solve
> authencation too.

I'm guessing that's why Oracle has a sync protocol instead of a remote
DB protocol.  The mobile client works against an on-phone copy of the
DB, using a variant of Oracle's full SQL -- the documentation says it is
SQLite compatible -- and then there's a minimal sync protocol which
keeps that in step with the full DB back on the server.  Interesting
question as to whether the on-phone copy is a subset or not.

> Seriously, I think we've pretty much done what you're suggesting in
> JMAP, but we've solved the latency problems that chatty protocols have
> when you don't have a nice short piece of solid copper between you and
> the server as well.

Sure, JMAP's great, but it's very email-specific.  Seems to me this is a
problem which lots of other domains have as well.  So: take JMAP, skip
the enumeration of email object types, generalize the getFooUpdates to
get/putTableUpdates, add some ability to specify per-table
parameterization which says things like how much of the remote table you
need locally, how frequently it needs to be updated, how precious
on-device changes are, etc., and you've got a general protocol.

That way, when you want to add a new object type, say Conference Rooms,
you don't need to change the protocol.

Bill

> 
> Bron.
> 
> -- 
>   Bron Gondwana
>   brong@fastmail.fm
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

By using a generic DB protocol to sync emails, it sounds like it would be easy to break your email database consistency.  
It would also add a big constraints with the choice of implementation of the storage of the emails on server side and client side.

There was also a mention of using a binary protocol in this thread.
I think using human readable text protocol is still a very good idea since it makes it easier to debug just by logging input/output of the protocol.
I?ve used a lot this way of debugging things: for IMAP, SMTP and (unrelated but for) HTTP requests.

--  
Hoa V. DINH


On Monday, March 9, 2015 at 8:35 AM, Bill Janssen wrote:

> Bron Gondwana <brong@fastmail.fm (mailto:brong@fastmail.fm)> wrote:
>  
> > > And let the DB guys solve the intermittent connection problems :-).
> > > Oracle, for instance, has a "Mobile Server" product that synchonizes a
> > > local cache on a mobile device with a remote Oracle DB, including SSL
> > > encryption on the sync connection, and data compression.
> > >  
> >  
> >  
> > The problem is, DB protocols are pretty chatty. And you need to solve
> > authencation too.
> >  
>  
>  
> I'm guessing that's why Oracle has a sync protocol instead of a remote
> DB protocol. The mobile client works against an on-phone copy of the
> DB, using a variant of Oracle's full SQL -- the documentation says it is
> SQLite compatible -- and then there's a minimal sync protocol which
> keeps that in step with the full DB back on the server. Interesting
> question as to whether the on-phone copy is a subset or not.
>  
> > Seriously, I think we've pretty much done what you're suggesting in
> > JMAP, but we've solved the latency problems that chatty protocols have
> > when you don't have a nice short piece of solid copper between you and
> > the server as well.
> >  
>  
>  
> Sure, JMAP's great, but it's very email-specific. Seems to me this is a
> problem which lots of other domains have as well. So: take JMAP, skip
> the enumeration of email object types, generalize the getFooUpdates to
> get/putTableUpdates, add some ability to specify per-table
> parameterization which says things like how much of the remote table you
> need locally, how frequently it needs to be updated, how precious
> on-device changes are, etc., and you've got a general protocol.
>  
> That way, when you want to add a new object type, say Conference Rooms,
> you don't need to change the protocol.
>  
> Bill
>  
> >  
> > Bron.
> >  
> > --  
> > Bron Gondwana
> > brong@fastmail.fm (mailto:brong@fastmail.fm)
> > _______________________________________________
> > Imap-protocol mailing list
> > Imap-protocol@u.washington.edu (mailto:Imap-protocol@u.washington.edu)
> > http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
> >  
>  
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu (mailto:Imap-protocol@u.washington.edu)
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/9cc04d4a/attachment.html>

Bill Janssen writes:
> Sure, JMAP's great, but it's very email-specific.

I put that differently: It leverages JSON well and does a great job of 
avoiding second-system syndrome.

What you suggest doing risks following Postgres multimaster replication 
down a long, winding, tiresome path.

Arnt

On 3/9/2015 10:35 AM, Bill Janssen wrote:
> I'm guessing that's why Oracle has a sync protocol instead of a remote
> DB protocol. The mobile client works against an on-phone copy of the
> DB, using a variant of Oracle's full SQL -- the documentation says it
> is SQLite compatible -- and then there's a minimal sync protocol which
> keeps that in step with the full DB back on the server. Interesting
> question as to whether the on-phone copy is a subset or not. 


On 3/8/2015 11:13 AM, Bill Janssen wrote:
> And let the DB guys solve the intermittent connection problems :-).
> Oracle, for instance, has a "Mobile Server" product that synchonizes a
> local cache on a mobile device with a remote Oracle DB, including SSL
> encryption on the sync connection, and data compression.
>

Bill, how much is Larry paying you?

It might be fun to submit an RFC draft that "mandates" the use of
something like the Oracle Mobile Server. Or heck, requires everyone
install Panagon on their machine to access their email - just to hear
the groans. If carrier pidgeons can make it past the rfc-editor then why
not?

:)

On a slightly more serious note, Dovecot has a protocol for syncing
message stores, but I haven't studied it closely yet.

L~


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150311/38093226/attachment.sig>

Hoa V. DINH <dinh.viet.hoa@gmail.com> wrote:

> I think using human readable text protocol is still a very good idea since it makes it easier to debug just by logging input/output of the protocol.

Yes, that's always been the argument.  On the other hand, you don't
read those logs with your bare Mark-I eyeballs.  You use a tool, even
if it's just less+xterm.  Writing a WireShark plugin is so streamlined
these days that I see no real advantage in the text formats anymore.

Bill

> human readable text protocol is still a very good idea since it makes it easier to debug just by logging input/output of the protocol.

Binary can be transformed to text easier than current "text"-flavoured
formats. It is possible to open any major encoding in good text
editors, or convert between encodings via libraries for display in
browser. Binary however is much easier to interpret in code than text.
Current standards (i.e. case insensitive commands, mixture of
encodings within a single text block, injection of CRLF which is not
part of the text), complicate interpreting a lot. It also consumes a
lot of CPU resources - visibly.

SMTP protocol would benefit from a few very clear (as in: explicit
sequence of bytes) instructions and explicit expectations about each
immediate payload: what follows (type, encoding), what byte length is
expected.

Email server does not need to guess: is it over yet? how much more to
expect? It needs to know with each command, what payload to prepare
for. It does not need to analyze / parse each line of the content as
it trickles in.

Something like this:
n1 bytes: command
n2 bytes: args (type: header, body, mime, etc)
n3 bytes: length
n4 bytes: content

both the specs and the code would be a few times shorter than what
they are with the current standard.

no guesswork.

Ladar Levison <ladar@lavabitllc.com> wrote:

> On 3/8/2015 11:13 AM, Bill Janssen wrote:
> > And let the DB guys solve the intermittent connection problems :-).
> > Oracle, for instance, has a "Mobile Server" product that synchonizes a
> > local cache on a mobile device with a remote Oracle DB, including SSL
> > encryption on the sync connection, and data compression.
> >
> 
> Bill, how much is Larry paying you?
> 
> It might be fun to submit an RFC draft that "mandates" the use of
> something like the Oracle Mobile Server. Or heck, requires everyone
> install Panagon on their machine to access their email - just to hear
> the groans. If carrier pidgeons can make it past the rfc-editor then why
> not?
> 
> :)

I did say, "for instance".  A number of different database systems
support replication, but replication for mobile is a harder problem.

I should admit: we're experimenting with new email MUA.  I wrote the
MDA to support it, and designed a custom (non-IMAP) MUA/MDA protocol to
support unique features of the MUA -- and it doesn't use database
replication.

Bill

> 
> On a slightly more serious note, Dovecot has a protocol for syncing
> message stores, but I haven't studied it closely yet.
> 
> L~
> 
> 
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol

On 9 March 2015 at 21:17, Bill Janssen <janssen@parc.com> wrote:

> Hoa V. DINH <dinh.viet.hoa@gmail.com> wrote:
>
> > I think using human readable text protocol is still a very good idea
> since it makes it easier to debug just by logging input/output of the
> protocol.
>
> Yes, that's always been the argument.  On the other hand, you don't
> read those logs with your bare Mark-I eyeballs.  You use a tool, even
> if it's just less+xterm.  Writing a WireShark plugin is so streamlined
> these days that I see no real advantage in the text formats anymore.
>

That would indeed explain the recent resurgence of ASN.1 based protocols.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/ab72d604/attachment.html>

On Mar 9, 2015, at 2:19 PM, Imants Cekusins <imantc@gmail.com> wrote:

> Current standards (i.e. case insensitive commands, mixture of
> encodings within a single text block, injection of CRLF which is not
> part of the text), complicate interpreting a lot. It also consumes a
> lot of CPU resources - visibly.

Is your measurement data and methodology for this available online anywhere?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 817 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/c6377933/attachment.sig>

On Monday, 9 March 2015 22:19:35 CET, Imants Cekusins wrote:
> It also consumes a lot of CPU resources - visibly.

I don't know which server you're basing your observations on, but my 
suggestion is to try benchmarking a recent version of one of the popular 
Linux mail servers at first (perhaps Postfix with Dovecot or Cyrus). If the 
actual throughput of this stack still bothers you, please post the numbers 
-- I'm always interested in reading a well-executed benchmark. Comparing 
the data to some other e-mail servers might be even a bit entertaining, I 
suspect.

With kind regards,
Jan

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

On 3/11/2015 11:24 AM, Bill Janssen wrote:
> I did say, "for instance".  A number of different database systems
> support replication, but replication for mobile is a harder problem.

Very true. As I recall Microsoft also has a DB replication/sync
framework that is supposed to work well over unreliable/slow network
links. I've never put their claims to the test though.

https://msdn.microsoft.com/en-us/sync/bb736753.aspx

> I should admit: we're experimenting with new email MUA.  I wrote the
> MDA to support it, and designed a custom (non-IMAP) MUA/MDA protocol to
> support unique features of the MUA -- and it doesn't use database
> replication.

Link? Or you keeping this piece of brilliance all to yourself?

I'm still looking for inspiration on my protocol design effort. I know
how to handle most of the functionality, but I'm still wrestling with
how to sync an encrypted binary file back to the server in a way that is
safe, and not overly complex... So when I hear "replication" I start to
think you may have the answer I'm looking for.

Of course before I worry about that problem I still need to decide which
scheme I'm going with. Lately I've been leaning towards sticking with a
line based protocol, aka, IMAP, but having it to send/receive JSON
objects...





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150311/cec8ca40/attachment.sig>

> Is your measurement data and methodology for this available online anywhere?

this is a subjective assessment. Email servers are way too busy for
what they deliver. I believe it is for these reasons:

Parsing takes time. So does transfer decoding and charset conversion,
unwrapping text, dot stuffing.

It can be reduced to:
no case conversion for commands; no parsing for commands even: a
simple switch case statement suffice;

no parsing for CRLF, empty line or DOT CRLF; actually, no content
parsing at all is necessary for transmission;

no transfer encoding / decoding (it is transferred as binary);

no charset decoding: a single encoding is applied to the entire (text
part of) the message.


Simply from pragmatic point of view: if these tasks need not to be
done, server is less busy and can process higher throughput.

Ok it appears I am not the only one considering current protocol slow.
I read this a while ago so I do not claim to be a discoverer of this
solution.

This is what D. J. Bernstein apparently wrote:


> The bottleneck in mailing list delivery today is SMTP latency. It typically takes more than ten seconds to transfer a message to another Internet host through SMTP. My mail transfer protocol, QMTP, is much faster, but how can qmail tell whether a remote host supports it? Answer: encode the information into the remote host's MX record. A future version of qmail-remote will support this. My target is 100 million remote deliveries per day on a 16MB machine.

http://cr.yp.to/qmail/future.html


> The Quick Mail Transfer Protocol (QMTP) is a replacement for the
   Simple Mail Transfer Protocol (SMTP). QMTP eliminates any need for
   end-of-line scanning between hosts with the same end-of-line
   convention. It features automatic pipelining and chunking, 8-bit
   transmission, prior declaration of the message size, and efficient
   batching. It is designed to be very easy to implement.


http://cr.yp.to/proto/qmtp.txt

> benchmarking a recent version of one of the popular Linux mail servers at first (perhaps Postfix with Dovecot or Cyrus)

I tried to set up and configure this exact combination for virtual
hosting. It took me a long time and in the end I gave up. Yes, I
understand someone somewhere succeeded to configure it. My point is
not that it is impossible to set it up.

My point (well, other than speed, robustness & simplicity) is that
with simplifed protocols though we could have more servers, a choice
between servers & configurations, supporting multitude of storage
backends.

Wouldn't it be nice - to choose? eh?

Ladar Levison <ladar@lavabitllc.com> wrote:

> > I should admit: we're experimenting with new email MUA.  I wrote the
> > MDA to support it, and designed a custom (non-IMAP) MUA/MDA protocol to
> > support unique features of the MUA -- and it doesn't use database
> > replication.
> 
> Link? Or you keeping this piece of brilliance all to yourself?

Sorry; Xerox proprietary for the moment.  And I wouldn't call it
brilliant; I try to focus on doing simple things efficiently instead of
clever things incorrectly.  But the concept is that the MUA/MDA split is
not necessarily where things should be divided; this system is more like
the split client you see in so many Web apps, where part of it runs in
the back-end and part in the front-end.  So the protocol is about
supporting the app, rather than about email per se.

> I'm still looking for inspiration on my protocol design effort. I know
> how to handle most of the functionality, but I'm still wrestling with
> how to sync an encrypted binary file back to the server in a way that is
> safe, and not overly complex... So when I hear "replication" I start to
> think you may have the answer I'm looking for.

There's always MH on top of AFS (http://www.openafs.org/)?  Has nmh
been ported to Android/iOS?

> Of course before I worry about that problem I still need to decide which
> scheme I'm going with. Lately I've been leaning towards sticking with a
> line based protocol, aka, IMAP, but having it to send/receive JSON
> objects...

Good luck!

Bill

Unfortunately, mail servers have to deal with the existing world of emails,
where there are charset encoding, MIME encoding, spinning disks and memory.

Lots of those operation have to happen until every users of emails move to anything else than email.
That said, spinning disks and memory might be still around.

Also, I actually think that the text protocol is neglectable in term of CPU usage (and that?s also a subjective assessment), compare to let?s say indexing of emails for search.

--  
Hoa V. Dinh


On Monday, March 9, 2015 at 4:44 PM, Imants Cekusins wrote:

> > Is your measurement data and methodology for this available online anywhere?
>  
>  
> this is a subjective assessment. Email servers are way too busy for
> what they deliver. I believe it is for these reasons:
>  
> Parsing takes time. So does transfer decoding and charset conversion,
> unwrapping text, dot stuffing.
>  
> It can be reduced to:
> no case conversion for commands; no parsing for commands even: a
> simple switch case statement suffice;
>  
> no parsing for CRLF, empty line or DOT CRLF; actually, no content
> parsing at all is necessary for transmission;
>  
> no transfer encoding / decoding (it is transferred as binary);
>  
> no charset decoding: a single encoding is applied to the entire (text
> part of) the message.
>  
>  
> Simply from pragmatic point of view: if these tasks need not to be
> done, server is less busy and can process higher throughput.
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu (mailto:Imap-protocol@u.washington.edu)
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>  
>  


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/08be8be3/attachment.html>

On Mar 9, 2015, at 4:44 PM, Imants Cekusins <imantc@gmail.com> wrote:

>> Is your measurement data and methodology for this available online anywhere?
> 
> this is a subjective assessment. Email servers are way too busy for
> what they deliver. I believe 

No, it's not.  Everything you are claiming can be measured.  No belief required.

Do you have measurements to back up your claim?  I don't discount it out of hand, but if you can't back it up with proof, I won't believe it.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 817 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/f296d2ae/attachment.sig>

On Mar 9, 2015, at 4:44 PM, Imants Cekusins <imantc@gmail.com> wrote:

> Simply from pragmatic point of view: if these tasks need not to be
> done, server is less busy and can process higher throughput.

The micro-optimization fallacy.

Before you dive in to this too far, spend a bit of time reading the papers Rob Pike and Brian Kernighan have published over the years.

Also, consider the latency of a network connection vs. the speed of the CPUs processing the data travelling to/from them.  Is the CPU you are using to parse your IMAP/SMTP stream so slow that it cannot do a 'c = tolower(getchar())' in real time?  Is it slower than your network link?  Even my ancient pre-Android phone can keep up with that.  Let alone the group of mail servers I run that manage to push through several hundred messages a second.

--lyndon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 817 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/1e28d2a4/attachment.sig>

It's off topic here.
The mailing list is about IMAP.
It's not about SMTP.


-- 
Hoa V. Dinh


On Tuesday, March 10, 2015 at 4:14 AM, Imants Cekusins wrote:

> Ok it appears I am not the only one considering current protocol slow.
> I read this a while ago so I do not claim to be a discoverer of this
> solution.
> 
> This is what D. J. Bernstein apparently wrote:
> 
> 
> > The bottleneck in mailing list delivery today is SMTP latency. It typically takes more than ten seconds to transfer a message to another Internet host through SMTP. My mail transfer protocol, QMTP, is much faster, but how can qmail tell whether a remote host supports it? Answer: encode the information into the remote host's MX record. A future version of qmail-remote will support this. My target is 100 million remote deliveries per day on a 16MB machine.
> 
> http://cr.yp.to/qmail/future.html
> 
> 
> > The Quick Mail Transfer Protocol (QMTP) is a replacement for the
> Simple Mail Transfer Protocol (SMTP). QMTP eliminates any need for
> end-of-line scanning between hosts with the same end-of-line
> convention. It features automatic pipelining and chunking, 8-bit
> transmission, prior declaration of the message size, and efficient
> batching. It is designed to be very easy to implement.
> 
> 
> http://cr.yp.to/proto/qmtp.txt
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150310/2c24d7d0/attachment.html>

On Tuesday, 10 March 2015 12:29:50 CET, Imants Cekusins wrote:
> My point (well, other than speed, robustness & simplicity) is that
> with simplifed protocols though we could have more servers, a choice
> between servers & configurations, supporting multitude of storage
> backends.
>
> Wouldn't it be nice - to choose? eh?

I see plenty of choice at e.g. [1].

What I suspect is that you might have had bad experience with performance 
of a random mail server, and are drawing conclusion from there. Sure, 
optimizing raw performance of a given protocol is nice (and being done, see 
BDAT and other examples people have mentioned), but maybe the actual 
reasons for slowness are somewhere else but in the live transformation of 
incoming bytestream?

Cheers,
Jan

[1] http://en.wikipedia.org/wiki/List_of_mail_servers

-- 
Trojit?, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/

I have a server to server replication protocol for IMAP servers mostly specced out.
It's based on the current Cyrus server to server replication protocol to a large degree.

Sadly I was in the middle of moving back from Norway to Australia when I wrote this:

http://lists.andrew.cmu.edu/pipermail/cyrus-devel/2012-December/002703.html

And got dragged down other paths.  I'm still working on generalising what is in Cyrus
though.  The most tricky bit is agreeing on the format for the checksum over the
entire mailbox contents which is used as a double-check for consistency after
applying changes.  Without that, it's still pretty good - but there are some issues
that can go undetected in the case of a split brain.

I also have notes from an email I put together after visiting David Carter (author of
the original replication support in Cyrus) which I'm just going to paste in at the end
of this message, because I don't appear to have sent them to a public list...

I have done some work since then on moving all the extended items into
namespaced per-message annotations on the wire, so that any server which
supports annotations (yeah, right - I think there are about 3) can keep information
with full fidelity even if they don't support a feature themselves.

Bron.


Sync Protocol: wire format

I spent the afternoon with David Carter and Tony Finch in Cambridge
looking over how Cyrus currently stores index records and per-mailbox
data and classifying the fields.

Glossary:

C Set at create time, immutable after
D Derived from RFC822 message, immutable
M Mutable
I Internal to Cyrus, irrelevant to sync because not exposed

cyrus.index records (per message):

C UID
C INTERNALDATE
D SENTDATE
D SIZE
D HEADERSIZE
D GMTIME
I CACHEOFFSET
I LAST_UPDATED
M SYSTEM_FLAGS
M USER_FLAGS
D CONTENT_LINES
I CACHE_VERSION
C GUID
M MODSEQ
I CACHE_CRC
I RECORD_CRC

Also per-message:

M ANNOTATIONS
C RFC822 message content

We talked a lot about GUID.  The conclusion was that for a vendor-neutral
protocol, you want GUID to be an opaque blob of somewhat arbitrary size
(perhaps the 70 bytes that POP3 UIDL gives - RFC1939).

It should not be necessary to have a GUID at all, or even MODSEQs, to use
this protocol.  Just without these things you lose abilities like incremental
updates and implicit cross-referencing.

There are two necessary formats for a record.  One is a wire format to
succinctly describe either a CREATE or UPDATE on a message, and the other
is a canonical serialisation format to calculate the SYNC_CRC.

The ordering of fields is chosen as follows:

UID
MODSEQ
FLAGS
INTERNALDATE
GUID
ANNOTATIONS
CRC32

The CRC32 buffer format is specified with upper case exact string keys,
as follows:

UID <number>
MODSEQ <number>
FLAGS (sorted: <flag>, ...)
INTERNALDATE <iso8601>
GUID <astring>
ANNOTATIONS (sorted: (/name user value user value), ...)
CRC32 <num>

NOTE: the buffer format has a single space rather than endline between each
key.

The sort for FLAGS is purely ASCII byte values.  The sort for ANNOTATIONS is
sorted by name, and within the values by user, with NIL (for shared) sorting
first.

If there is no MODSEQ, then the MODSEQ item is entirely omitted.

If there is no GUID, then the GUID item is entirely omitted.

The FLAGS () and ANNOTATIONS () items are not included in the CRC32 format if
they are an empty list.

So the most trivial case is:

UID <value> INTERNALDATE <value> CRC32 <value>

Extended values (like the CID for FastMail's conversations patch) can either
be added by extending the format, or by creating a synthetic vendor ANNOTATION
field.

CREATES:
========

The create format is precisely like the CRC32 format, except that it may
contain either a key RFC822 with the entire message, a key XREF with a
triple (mailboxname uidvalidity uid) or just the GUID and rely on automatic
linkage to other messages with the same GUID.

GUID is defined as being unique to a particular RFC822 message text.  It is
the server's reponsibility to come up with something unique.

So CREATE is just the CRC32 format with an additional either RFC822 or XREF
field to specify the message body.  In the sync protocol, it's sender's
responsibility to ensure that the server already has the XREF'ed message.

In the incremental backup case, it's the backup server's responsibility to
check that the XREF'ed message already exists in the previous backup.

UPDATES:
========

To update an existing record, a record with the same UID is created.  If
it contains any CREATION ONLY field, then it's required to match exactly
(e.g. INTERNALDATE, CRC32 or GUID).  If it doesn't, then UID promotion logic
takes over.  It also needs to have a higher MODSEQ value than the previous
record of course.

Otherwise, the format is the same as CREATE.

E.g.

UID 5 MODSEQ 100 FLAGS (\Seen $foo)

If any field is absent, it is unchanged.  If the FLAGS () or ANNOTATIONS ()
lists are present, then they are a SET - changing the replica to contain
exactly what is in them, removing anything not mentioned.

VANISHED:
=========

To deal with cases where the server has forgotten precisely which UIDs were
removed since the previous MODSEQ value, there needs to be a way to say
"everything in these ranges of UIDs is was removed".

It also needs to be as if it happened at the last known MODSEQ (DELETEDMODSEQ
in Cyrus terms).

The format is

VANISHED (AT <modseq> UIDS <range>)

e.g.

VANISHED (AT 5001 UIDS 1:20,23:25,27:201)

This will cause the receiver to expunge any messages matching that UID range,
and to do so at a MODSEQ of 5001.  NOTE: it only makes sense for this modseq
to be in the future for the receiver, otherwise you would not be sending the
vanished range, because the receiver would already have seen those changes.

---------------

That's enough to cover all the interesting cases for messages - now on to
mailboxes:

cyrus.index header

I GENERATION
I FORMAT
I MINOR_VERSION
I START_OFFSET
I RECORD_SIZE
I NUM_RECORDS
I LAST_APPENDDATE
M LAST_UID
I QUOTA_USED
M POP3_LAST_LOGIN => metadata
C UIDVALIDITY
I DELETED
I ANSWERED
I FLAGGED
M OPTIONS => metadata
I LEAKED_CACHE
M HIGHESTMODSEQ
I DELETEDMODSEQ
D EXISTS
I FIRST_EXPUNGED
I LAST_REPACK_TIME
I HEADER_FILE_CRC
D SYNC_CRC
M RECENT_UID
I RECENT_TIME

mailboxes.db:

M ACL
M SPECIAL-USE => metadata

cyrus.header:

C UNIQUEID
M QUOTAROOT ()
M FLAGS ()

annotations.db:

M METADATA

CREATE format (== CRC32 format - perhaps just XOR with the messages CRC):

NAME <name>
UIDVALIDITY <number>
HIGHESTMODSEQ <number>
LAST_UID <number>
ACL (name value name value)
QUOTAROOT (root, ...)
FLAGS (flag, ...)
METADATA (...)

ACLs are sorted by name and normalised by removing cd which got removed in RFC x

QUOTAROOTs are sorted by ASCII bytes

FLAGS are sorted by ASCII bytes

METADATA are sorted and stored just like ANNOTATION:

           C: a SETMETADATA INBOX (/private/comment "My new comment"
                               /shared/comment "This one is for you!")
           S: a OK SETMETADATA complete

Becomes:

METADATA (/comment (NIL "This one is for you!" brong "My new comment"))

Being a shared comment and a private comment for brong.

TODO:
=====

* list RFCs next to each item
* Sieve / Subs / etc - user level stuff
* Server level annotations
* Quotaroots
* define conflict resolution protocols (uid promotion, value resolution,
  rename handling)
* dates: nail down format YYYY-MM-DDThh:mm:ssZ
* Conversations extended format

-- 
  Bron Gondwana
  brong@fastmail.fm

On Mar 11, 2015, at 10:30 AM, Bill Janssen <janssen@parc.com> wrote:

> Has nmh
> been ported to Android/iOS?

Oh gawd. Really?  You would read and type into a command line interface on a phone for your email?

If you want to offer up Android and iOS buildbot slaves, I will add then to the build cluster.  But I ain't porting the code!  (I mean, eeew!)

--lyndon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 817 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150311/20642381/attachment.sig>

We witnessed quite a big portion of cpu involved in un-dot-stuffing smtp
(and correcting line endings) and saved quite a bit of that with supporting
BDAT.

OTOH, I don't think IMAP parsing has been a large CPU issue for us,
partially because with LITERAL we already skip through the larger data
pieces.

Brandon

On Mon, Mar 9, 2015 at 4:54 PM, Hoa V. Dinh <dinh.viet.hoa@gmail.com> wrote:

> Unfortunately, mail servers have to deal with the existing world of emails,
> where there are charset encoding, MIME encoding, spinning disks and memory.
>
> Lots of those operation have to happen until every users of emails move to
> anything else than email.
> That said, spinning disks and memory might be still around.
>
> Also, I actually think that the text protocol is neglectable in term of
> CPU usage (and that?s also a subjective assessment), compare to let?s say
> indexing of emails for search.
>
> --
> Hoa V. Dinh
>
> On Monday, March 9, 2015 at 4:44 PM, Imants Cekusins wrote:
>
> Is your measurement data and methodology for this available online
> anywhere?
>
>
> this is a subjective assessment. Email servers are way too busy for
> what they deliver. I believe it is for these reasons:
>
> Parsing takes time. So does transfer decoding and charset conversion,
> unwrapping text, dot stuffing.
>
> It can be reduced to:
> no case conversion for commands; no parsing for commands even: a
> simple switch case statement suffice;
>
> no parsing for CRLF, empty line or DOT CRLF; actually, no content
> parsing at all is necessary for transmission;
>
> no transfer encoding / decoding (it is transferred as binary);
>
> no charset decoding: a single encoding is applied to the entire (text
> part of) the message.
>
>
> Simply from pragmatic point of view: if these tasks need not to be
> done, server is less busy and can process higher throughput.
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
>
>
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman13.u.washington.edu/mailman/listinfo/imap-protocol
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/02939ddd/attachment.html>

> text protocol is neglectable in term of CPU usage ..., compare to let's say indexing of emails for search.

well indexing is another topic. business email servers may or may not
index email. of course, there is virus scanning, spam filtering - all
this consumes resources.

all I am saying: current protocols are wasteful from several points of
view and can be simplified.

if anything, simpler protocols would lead to
- fewer production issues,
- shorter development time (for email server developers),
- more email server implementations, (choice is good)
- better protocol compliance (simpler protocol makes it easier to comply with).
- more robust mail delivery (less confusion over protocol)

Let's take chat messages as an example. They are fast. much faster
than email, aren't they? Why? Isn't it a similar concept: text
(transmitted as bytes) flowing from one browser to another - via an
intermediate server?

Web servers are getting more efficient, databases are getting more efficient.

Why not bring email servers up to speed?

> you might have had bad experience with performance of a random mail server

Actually, I did not get this far. I could not configure one of the
better email servers currently available. It is easier to write one
from the ground up.

My performance observations are based on experience with production
IMAP servers and on writing actual SMTP & IMAP server code compliant
with the current protocols.

Yes, this code may be inefficient, can be optimized.

Yet this seems obvious: regardless of how quickly a server can do a
task, not doing this task (parsing, conversion and altering bytes in
any way) saves time.

Could we agree on this as an axiom?

On 3/11/2015 11:19 PM, Lyndon Nerenberg wrote:
> On Mar 11, 2015, at 10:30 AM, Bill Janssen <janssen@parc.com> wrote:
>
>> Has nmh
>> been ported to Android/iOS?
> Oh gawd. Really?  You would read and type into a command line interface on a phone for your email?

I routinely ssh from my phone to my mail server to read mail in mutt.  I 
don't usually send mail that way, but I do read and type into a CLI on a 
phone.  Often.

I'm a dinosaur and I admit it.

Lyndon Nerenberg <lyndon@orthanc.ca> wrote:

> 
> On Mar 11, 2015, at 10:30 AM, Bill Janssen <janssen@parc.com> wrote:
> 
> > Has nmh
> > been ported to Android/iOS?
> 
> Oh gawd. Really?  You would read and type into a command line interface on a phone for your email?

Well, no.  I'd use MH-e on top of nmh on the phone.

Bill

> with LITERAL we already skip through the larger data pieces

yes Literal with IMAP and BDAT would be good examples to follow.
However because standards allow various combinations in syntax, all
these need to be taken care of in every server implementation.

these features made perfect sense back in the day however things moved
on quite a bit since '90s.

why not make the format as simple as possible to prepare, deliver and
display - given the advances in technology?

maybe even give an extra thought to preventing spam - if this is possible..

On 3/9/2015 7:01 PM, Brandon Long wrote:
> We witnessed quite a big portion of cpu involved in un-dot-stuffing 
> smtp (and correcting line endings) and saved quite a bit of that with 
> supporting BDAT.

Yeah, dot-stuffing definitely seems to me to be the slowest and worst 
part of SMTP: if you use BDAT, you could use splice() or similar kernel 
features to just shunt things between sockets and files without needing 
to do immediate inspection. And I'm guessing that almost of the time 
spent having to work with the SMTP protocol itself (as opposed to 
necessary routing or processing logic) is likely to be worrying about 
dot-stuffing, since it's most of the datastream by size and it's not a 
unitary transformation.

> On Mon, Mar 9, 2015 at 4:54 PM, Hoa V. Dinh <dinh.viet.hoa@gmail.com 
> <mailto:dinh.viet.hoa@gmail.com>> wrote:
>
>     Unfortunately, mail servers have to deal with the existing world
>     of emails,
>     where there are charset encoding, MIME encoding, spinning disks
>     and memory.
>

Well, if you're talking about replacing the rfc 5322/MIME format, there 
is certainly a large number of historical warts and issues that could be 
fixed that are completely orthogonal to text versus binary.

-- 
Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald E. Knuth

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/25e90d05/attachment.html>

On Tue, Mar 10, 2015, at 11:15 AM, Imants Cekusins wrote:
> > text protocol is neglectable in term of CPU usage ..., compare to let's say indexing of emails for search.
> 
> well indexing is another topic. business email servers may or may not
> index email. of course, there is virus scanning, spam filtering - all
> this consumes resources.
> 
> all I am saying: current protocols are wasteful from several points of
> view and can be simplified.
> 
> if anything, simpler protocols would lead to
> - fewer production issues,
> - shorter development time (for email server developers),
> - more email server implementations, (choice is good)
> - better protocol compliance (simpler protocol makes it easier to comply with).
> - more robust mail delivery (less confusion over protocol)

So it looks like we're talking about SMTP more than IMAP here.  One thing with the client<->server protocol is that you can do a lot of pre-calculation and caching on the server, particularly if you know what fields clients are likely to request.  If you're re-creating the ENVELOPE on every read, then your server will be slow for sure.

But yes, simpler protocols are good for a lot of things.  Text vs not text is not so much of a big issue there though, if you can just load a JSON library or even (cough, splutter) an XML library and get on with business, it doesn't matter what the wire protocol is so much.  A protocol which is nicely regular and easy to parse/recreate is valuable, because then you can proxy it easily.

We use nginx as an IMAP proxy, but it all it does is the authentication step, and then provides a dumb stream proxy to the backends.  I would love if it was easy to make it a bit more protocol aware, so we could easily inject a notice to the proxy that this backend was going away, and to switch the connection to this new replica.  Of course, that sucks anyway because of selected mailbox state, and THAT would need to be moved to the replica - or uploaded to nginx and streamed back in.  Messy either way.  Having that state removed so that you don't need to serialise it to a new backend would make me very happy.

Bron ( or just stop having connections that persist for days )

-- 
  Bron Gondwana
  brong@fastmail.fm

Please go ahead.  

(Btw, text protocol doesn?t seem to be an issue with chat app (FYI they use jabber, which is a verbose protocol using XML))



--  
Hoa V. DINH


On Monday, March 9, 2015 at 9:25 PM, Imants Cekusins wrote:

> Let's take chat messages as an example. They are fast. much faster
> than email, aren't they? Why? Isn't it a similar concept: text
> (transmitted as bytes) flowing from one browser to another - via an
> intermediate server?
>  
> Web servers are getting more efficient, databases are getting more efficient.
>  
> Why not bring email servers up to speed?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20150309/b880920d/attachment.html>

> we're talking about SMTP more than IMAP here.

SMTP first of all but both really. IMAP is more complex than SMTP. it
is very slow too - maybe for different reasons.

generally, it can takes less time to send, receive and read email.


> Text vs not text is not so much of a big issue there though,

bytes are sent across the wire. any transformations we perform are an
extra cost, come at a price. do we really absolutely need all of these
transformations?

btw even today we can ship utf8 or utf16 to a modern browser in binary
format - and parse the bytes in browser to display to the user.
obviously, single encoding across the message helps in this case.

think about this: original blob message- as sent - are shipped
directly to browser - no server parsing or indeed any transformations
on the server. ok, maybe filter messages somehow.