mailing list archives

meli community discussions

⚠️ if something does not work as intended when interracting with the mailing lists,
reach out Github mirror Gitea repo @epilys:matrix.org

E-mail headers
From: "Lynn W. Taylor" <Lynn@BusCom.net>
To: imap-protocol@u.washington.edu
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4DEBB242.2090200@BusCom.net permalink / raw / eml / mbox
I need to implement IMAP as part of an existing project, and I need a 
tiny bit of practical guidance.

Simply adopting existing code isn't really an option, so I'll be 
implementing from scratch.

IMAP has a lot of features.  Which features should I implement first, 
and at what point do I have something to start testing?

Thanks in advance for any insight.

-- Lynn
Reply
E-mail headers
From: dkonigsberg@logicprobe.org
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4DEBB58D.3090101@logicprobe.org permalink / raw / eml / mbox
On 6/5/2011 9:43 AM, Lynn W. Taylor wrote:
> I need to implement IMAP as part of an existing project, and I need a 
> tiny bit of practical guidance.
>
> Simply adopting existing code isn't really an option, so I'll be 
> implementing from scratch.
>
> IMAP has a lot of features.  Which features should I implement first, 
> and at what point do I have something to start testing?
>
> Thanks in advance for any insight.
It might first be a good idea to clarify whether you're implementing a 
client or a server.

-- 
----------------------------
  Derek Konigsberg
  dkonigsberg@logicprobe.org
----------------------------
Reply
E-mail headers
From: mrc+imap@panda.com
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: alpine.OSX.2.00.1106051606130.4942@hsinghsing.panda.com permalink / raw / eml / mbox
Most of what others have posted here is good advice.  I will not duplicate
what they said.

If you enjoy beating your head against a wall incessantly, then by all
means ignore what I am about to say.  Over the years, I have learned that
there are individuals who delight in beating their heads against the wall,
and that it is useless for me to attempt to deprive them of their fun.

If you don't enjoy an aching head, then listen up.

First and foremost, the Formal Syntax section of RFC 3501 should be your
holy book.  If any part of RFC 3501 distracts you from the Formal Syntax,
ignore it in favor of the Formal Syntax.

Your jaw will drop when you first see the Formal Syntax.  Your eyes will
glaze over.  You will start saying "no, no, no."  Just work through that
stage.  It's a steep hill to climb, but once you make it to the top you
will see everything with crystal clarity.

IMAP is a very subtle protocol in places, and only the Formal Syntax
relates the full subtlty.  When a space is required, then exactly ONE
space (not ZERO, not TWO) MUST be there; and when a space is not required
then it is FORBIDDEN.

Whatever you do, DO NOT ATTEMPT TO IMPLEMENT ANY COMMAND OR RESPONSE BY
LOOKING AT THE EXAMPLES!  You are guarantee to screw up if you use the
examples as a model (that means YOU, Microsoft!).  Use only the Formal
Syntax.  The result should indeed look like the examples; and if it
doesn't then study the Formal Syntax to understand why.

Do I still have your attention?  Good.  You apparently don't want to bang
your head against the wall.  I just saved you for one of the major causes
of head-banging.

Second, mind the transition between literal and non-literal modes.  You
are either outputting a line, which is set of octets terminated by CRLF;
or you are outputting a literal, which is a precisely counted number of
octets with no termination.  However, in all cases in IMAP, there is a
line after a literal (even if it is just a CRLF to end the command or
response).

There are some deep insights in the previous paragraph that will be of
great use to you if you implement your server in a network I/O
infrastructure in which you are doing either "line" or "buffered" mode.
Go ahead.  Ask me how I know... :)

Some commands may have multiple literals.  So, if a command has two
arguments, both of which come in as literals, you must: read line, read
sized buffer, read line, read sized buffer, read line.  The SEARCH command
can have quite a few literals.  Or it may have none.  Be prepared.

Third, you are best off parsing commands into a struct, following the
Formal Syntax.  Think about extensibility.  You'll be glad you did if you
ever implement the MULTIAPPEND extension.

Something else is important: there is ABSOLUTELY NOTHING that a command
can do, as part of the parsing of the command, that changes anything in
the mailbox or the server session.  If a command responds with either NO
or BAD, then the command is guaranteed to have done nothing.  Thus,
commands are in three phases: parse, action, response; and there is an
action if and only if the response is OK.

Fourth, implement responses by a struct, again following the Formal
Syntax.  You'll have lots of enums for types to identify which union to
follow.  But it's possible to define all of IMAP response in terms of a
single struct (truly the Struct From Hell...but once you get it it will be
a delight).  Now you just need an engine to poot it out.  Guess what.  I
just saved you a HUGE amount of time in writing the IMAP response
generator.

Writing response routines is obsolete technology.  I know because I have
done it both ways.  It is so much faster to have a single response engine
that poots out a response struct tree.

Fifth, keep in mind that pretty much all of RFC 3501 is mandatory to
implement.  You CAN save yourself some time by not implementing non-INBOX
mailboxes (thus the LIST command only poots a \NoInferiors mailbox called
INBOX, and CREATE/DELETE/RENAME all respond with NO).  But you'll probably
need the non-INBOX mailboxes sooner rather than later.

When you do implement other mailboxes, don't try to be magic about
subscriptions.  Just treat it like server-based bookmarks.

Modified UTF-7 is a royal pain; and I had hoped that it would have been
extinct by now.  But you can defer doing anything about it until you
implement non-INBOX mailboxes.

Sixth, you probably will want a vector of messages in the selected
mailbox, indexed by message sequence number.  In this vector you will
want, at a minimum, the internal date, size, flags, and UID.  You'll
probably want some pointers to message contents, and perhaps preparsed
ENVELOPE and BODYSTRUCTURE.  Do UID lookup via binary search through this
vector (not linear search).

Seventh, this reminds me about something else.  Those BODYSTRUCTURE things
have lots of size counts.  These, and the RFC822.SIZE, must all be EXACT
(do you hear me, Novell?  Microsoft?).  Clients that do partial fetching
depend upon those counts being exact.  No, it is not OK to guess.  These
are NOT advisory.

Eighth, flags being booleans is a common implementation and that's quite
alright.  However, you'll need some way to record keyword names.  Unless
you are providing access to a non-owned mailbox, all flags should be
permanent.

Ninth, slash should be the One And Only True Hierarchy Delimiter.  Almost
everybody agrees that it was a mistake to allow others.

Tenth, OK, I lied.  I will repeat something that someone else said.  Mind
that COPY requires atomicity.  So does the MULTIAPPEND extension, if you
ever implement that.  If you turn out to be talented enough to implement
MULTIAPPEND, you can claim that you truly grok IMAP.

Eleventh, a server session only needs one mailbox open at a time.  If you
have a multithreaded server that supports multiple sessions, consider a
design that allows you to spawn off additional server processes when the
current process fills up.  A 32-bit address space simply is not enough for
a modern multi-session server.

Finally, pat yourself on the back.  You decided to ask for advice before
you started implementing, instead of diving in, making all the newbie
mistakes, and only then ask questions.  You're already ahead of the game.

Regards,

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
Reply
E-mail headers
From: Lynn@BusCom.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4DEBBAB3.2030301@BusCom.net permalink / raw / eml / mbox
On 6/5/2011 9:57 AM, Derek Konigsberg wrote:
> On 6/5/2011 9:43 AM, Lynn W. Taylor wrote:
>> I need to implement IMAP as part of an existing project, and I need a
>> tiny bit of practical guidance.
>>
>> Simply adopting existing code isn't really an option, so I'll be
>> implementing from scratch.
>>
>> IMAP has a lot of features.  Which features should I implement first,
>> and at what point do I have something to start testing?
>>
>> Thanks in advance for any insight.
> It might first be a good idea to clarify whether you're implementing a
> client or a server.
>
Sorry.  Server.
Reply
E-mail headers
From: rfs9999@earthlink.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4E3382E3.3010509@earthlink.net permalink / raw / eml / mbox
Is there a way to reliably figure out the mailbox hierarchy prefix when 
an IMAP server does not support the NAMESPACE command?

Thanks,
Rick
Reply
E-mail headers
From: Lynn@BusCom.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4DEC6327.60309@BusCom.net permalink / raw / eml / mbox
My first bit of code ran successfully in 1969.  It was on cards.

I've learned a few important lessons since:

"There are no stupid questions" except the ones that aren't asked, of 
course.

Whenever possible, ask someone whose been there.  The good guys will share.

Don't ignore "shared pain" -- always pay attention.

I'm reading all of comments carefully and trying to figure out how best 
to implement them.

I've read through the RFC more than once, and it's hard to find a place 
to start.

-- Lynn

On 6/5/2011 4:58 PM, Mark Crispin wrote:
> Finally, pat yourself on the back.  You decided to ask for advice before
> you started implementing, instead of diving in, making all the newbie
> mistakes, and only then ask questions.  You're already ahead of the game.
Reply
E-mail headers
From: Lynn@BusCom.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4DEBBEC1.3080600@BusCom.net permalink / raw / eml / mbox
On 6/5/2011 10:19 AM, Lynn W. Taylor wrote:
> On 6/5/2011 9:57 AM, Derek Konigsberg wrote:
>> On 6/5/2011 9:43 AM, Lynn W. Taylor wrote:
>>> I need to implement IMAP as part of an existing project, and I need a
>>> tiny bit of practical guidance.
>>>
>>> Simply adopting existing code isn't really an option, so I'll be
>>> implementing from scratch.
>>>
>>> IMAP has a lot of features.  Which features should I implement first,
>>> and at what point do I have something to start testing?
>>>
>>> Thanks in advance for any insight.
>> It might first be a good idea to clarify whether you're implementing a
>> client or a server.
>>
> Sorry.  Server.
>
P.S. I'm not looking for a complete roadmap, I can fill in most of the 
detail.

Tags and UIDs are not a big deal.

What I want to do is get enough IMAP working that I can start testing 
with a client, even if the server isn't 100% practical, instead of 
implementing some critical command last, and finding out that I made 
some critical mistake very early in the process.

Thanks in advance for any advice.

-- Lynn
Reply
E-mail headers
From: guenther+imap@sendmail.com
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: alpine.BSO.2.00.1107292118540.19142@morgaine.smi.sendmail.com permalink / raw / eml / mbox
On Fri, 29 Jul 2011, Rick Sanders wrote:
> Is there a way to reliably figure out the mailbox hierarchy prefix when an
> IMAP server does not support the NAMESPACE command?

RFC 3501, section 6.3.8, paragraph 4:

      An empty ("" string) mailbox name argument is a special request to
      return the hierarchy delimiter and the root name of the name given
      in the reference.  The value returned as the root MAY be the empty
      string if the reference is non-rooted or is an empty string.  In
      all cases, a hierarchy delimiter (or NIL if there is no hierarchy)
      is returned.  This permits a client to get the hierarchy delimiter
      (or find out that the mailbox names are flat) even when no
      mailboxes by that name currently exist.


Note:
1) That quote is the only place in the spec where "root" is used in 
   reference to mailbox names, so it's not clear what requirements are
   placed on the name that is returned other than those implied in 
   general by the historical context of roots in hierarchical namespaces.
   Those that don't believe standards exist in a historical context would 
   say that a server can return whatever it wants there and that it has
   no meaning.  Those who believe otherwise think the former are nihilists 
   that need to be kept away from computers.

2) There's no guarantee that there's only one "mailbox hierarchy prefix"
   as you call it on the server.  You asked for *the* prefix; what if
   there are many?


Philip Guenther
Reply
E-mail headers
From: imap@maclean.com
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: mailman.3.1528486486.22076.imap-protocol@mailman13.u.washington.edu permalink / raw / eml / mbox
At 01:37 PM 6/5/2011, Lynn W. Taylor wrote:
>P.S. I'm not looking for a complete roadmap, I can fill in most of the detail.
>
>Tags and UIDs are not a big deal.
>
>What I want to do is get enough IMAP working that I can start 
>testing with a client, even if the server isn't 100% practical, 
>instead of implementing some critical command last, and finding out 
>that I made some critical mistake very early in the process.

Here are 10 tips based on my own experience.  From what you say, some 
of these may not be relevant to your case but I will be inclusive.  I 
am also assuming both that you are not starting out intimately 
familiar with the protocol and that messages are stored in 
RFC2822-MIME format.  A general guideline is that, while books may 
help you get a feeling for IMAP -- they did for me, when actually 
implementing it is vital to pay full attention to the relevant RFCs.

1). You will need a fairly sophisticated message parser, notably for 
extracting MIME parts and generating BODYSTRUCTURE information.  If 
you do not start out with such a parser ready-made then you might 
well choose to begin by building one.

2). If your mailstore does not have UIDs (or something that can be 
suitably mapped into UIDs) then decide how you will create and store 
UIDs.  Incidentally IMAP specs refer to a UID command; I find it 
easier to view this as a command qualifier.

3). Decide what permanents flags to support and how.  Decide what 
hierarchy delimiter to use; I believe that most servers use slash, 
backslash or dot but I have not seen a problem using even a 
non-graphic character such as Rubout (ASCII 127).  By the way, I 
think that RFC 3501 does not make this clear but the hierarchy 
delimiter must be a 7-bit ASCII character.

4). Decide what data to keep in memory for each message in an open 
mailbox.  (By "open", I mean one for which a session is in progress, 
that is opened as the result of a SELECT or EXAMINE command.)  A 
likely minimum would be UID, IMAP message flags, message size and, of 
course, a reference to the message itself.  I am assuming that this 
information is indexed by message number and hence that that is not 
explicitly stored.  You may find reason to add more data such as a 
flag indicating if the message is matched in a current SEARCH.

5). I don't recall if I did this when I wrote my server, but I would 
probably start now by implementing the LIST command.  This involves a 
number of elements including modified UTF-7 encoding and 
understanding the distinction between the two arguments of the 
command.  I wrote unit test programs for my backend LIST code (and 
that of several other commands) and I highly recommend doing the 
same.  These were simple console programs each of which took a set of 
IMAP command arguments as input and emitted the results as 
output.  In the early stages, testing with actual IMAP connections 
(be they from a client or a test program) is unnecessarily 
awkward.  (I am assuming here that there are other means, outside of 
the IMAP server, to create mailboxes; if this does not hold then you 
may want to implement CREATE first.)

6). Next I would implement SELECT and EXAMINE.  There are intricacies 
here such as checking what rights the user in question has to the 
mailbox and ensuring coordination among threads (and, if relevant, 
with other programs) that are simultaneously accessing the mailbox.

7). Next I would implement FETCH.  This is a biggie because of the 
range of data items, macros, dealing with literals, and so on.

8). Perhaps at about this point, I would work on the command 
processor (recognising commands, checking mode, parsing arguments, 
etc.) and, unless I had a ready-made one, the listener.  This allows 
one to start testing online.  I suggest implementing the command 
processor at a fairly late stage because, for some commands, 
argument-parsing is tricky and you will likely find it easier once 
you have implemented the commands themselves and have a good 
understanding of how they work -- although the critical thing to do 
here is to follow the RFC's grammar very closely.  AUTHENTICATION can 
be done at any stage since you can fake it until such point as you 
choose to made it real.

9). Next I would take care of the "minor" commands:  CREATE, LSUB, 
NOOP, CLOSE, STORE, CAPABILITY, APPEND, etc.  But do not think of 
COPY as minor -- it can be a substantial job because of the required atomicity.

10). I would leave SEARCH for last.  When I wrote my server, I left 
it to last because I feared it would be the most difficult to do.  I 
was wrong; it proved much easier than I had expected but it is, 
nevertheless, a hefty amount of work.  It may also be fair to say 
that it is the least critical command but, all the same, it has to be 
done and done completely (unless of course your client is intended to 
work with a strictly limited set of clients that do not use it.)  And 
testing SEARCH can be hard because few clients implement it well; 
indeed, some do not offer it at all.  Of course, if your mailstore 
manager already has search capabilities then you may be able to 
leverage those and make the job a snap.
Reply
E-mail headers
From: mrc+imap@panda.com
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: alpine.OSX.2.00.1107292242400.911@hsinghsing.panda.com permalink / raw / eml / mbox
On Fri, 29 Jul 2011, Philip Guenther wrote:
>   Those that don't believe standards exist in a historical context would
>   say that a server can return whatever it wants there and that it has
>   no meaning.  Those who believe otherwise think the former are nihilists
>   that need to be kept away from computers.

Amen.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
Reply
E-mail headers
From: rfs9999@earthlink.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4E8F0D21.3040004@earthlink.net permalink / raw / eml / mbox
I'm working with a server whose response to a FETCH (rfc822) is 
different than I have seen before.

 >> 1 FETCH 1 (rfc822)
<< * 1 EXISTS
<< * 1 FETCH (RFC822[] {0}

I'm used to seeing something like * 1 FETCH (RFC822 {1522}.  In this 
case I have to send a separate 1 FETCH 1 (rfc822.size) command to get 
the message size:   << 1 FETCH (rfc822.size 5149).

Just curious, is this compliant?

-Rick
Reply
E-mail headers
From: dave@cridland.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 8189.1317998191.572045@puncture permalink / raw / eml / mbox
On Fri Oct  7 15:30:57 2011, Rick Sanders wrote:
> I'm working with a server whose response to a FETCH (rfc822) is  
> different than I have seen before.
> 
> >> 1 FETCH 1 (rfc822)
> << * 1 EXISTS
> << * 1 FETCH (RFC822[] {0}
> 
> 
Yeah, that's wrong...


> I'm used to seeing something like * 1 FETCH (RFC822 {1522}.  In  
> this case I have to send a separate 1 FETCH 1 (rfc822.size) command  
> to get the message size:   << 1 FETCH (rfc822.size 5149).
> 
> Just curious, is this compliant?

I don't think so. Try asking for BODY[] instead, though - it's the  
same thing as RFC822, but more modern syntax, so it might work better  
against that server.

Dave.
-- 
Dave Cridland - mailto:dave@cridland.net - xmpp:dwd@dave.cridland.net
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
Reply
E-mail headers
From: tss@iki.fi
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 1317998399.21919.63.camel@hurina permalink / raw / eml / mbox
On Fri, 2011-10-07 at 09:30 -0500, Rick Sanders wrote:
> I'm working with a server whose response to a FETCH (rfc822) is 
> different than I have seen before.
> 
>  >> 1 FETCH 1 (rfc822)
> << * 1 EXISTS
> << * 1 FETCH (RFC822[] {0}
> 
> I'm used to seeing something like * 1 FETCH (RFC822 {1522}.  In this 
> case I have to send a separate 1 FETCH 1 (rfc822.size) command to get 
> the message size:   << 1 FETCH (rfc822.size 5149).
> 
> Just curious, is this compliant?

No, the server is broken. But you could try to work around it with
fetching BODY.PEEK[] instead.
Reply
E-mail headers
From: rfs9999@earthlink.net
To: imap-protocol@localhost
Date: Fri, 08 Jun 2018 12:34:46 -0000
Message-ID: 4E8F1284.8000400@earthlink.net permalink / raw / eml / mbox
> Try asking for BODY[] instead, though - it's the same
> thing as RFC822, but more modern syntax, so it might work better against
> that server.

Yep, that works much better.

1 fetch 1 body[]
* 1 FETCH (BODY[] {5149}

Thanks, Dave!

-Rick
Reply