mailing list archives

Folks,

Please forgive this somewhat off-topic message.  I'm looking for more
funding to work on my UpLib IMAP server implementation.  UpLib stores
all messages in a DB, and uses tags for folders, thus storing only one
copy of messages that might occur in multiple folders, or are stored by
multiple users.  I'm interested in figuring some strawman stats about
email duplication that could be addressed by a site-wide IMAP server.
For example, how many mailing lists an "average" organization has; the
average number of users per mailing list; the average message traffic
(in both messages and bytes) per day, per week, per year.  I know that a
number of you have actual experience with this.  I'd be grateful for any
pointers.

Thanks.

Bill

I don't have stats, but I've done several MIME stores that way. Further 
improvements come from allowing some flags to be stored along with the 
message, while others along with the cross-post record 
(message-to-folder association). And further improvements still come 
from allowing a cross-post header to have some custom headers.

In a MIME store, lots of people forward the same attachment, but in each 
case, the part header as different headers. Some MUAs use a content-type 
"name" attribute field-param to store the file name, others use the 
Content-Disposition "filename" field-param, etc. But in both cases, you 
really want to store the 15 MB zip file attachment just once.

I never worked on optimizing representation of a single mime part in all 
it different varieties of content transfer encoding, versus storing 
decoded and having to encode on the outbound stream, but that would have 
been super fun.

Apple was hiring for a new custom MIME store project for their hosting 
several years back, around 2002-3 I think. That would have been fun.

As for what is profitable, I think Facebook, Twitter, et. al. are 
successfully demonstrating that, en-masse, communication is routing 
around the dinosaur e-mail protocols. Techies like us (especially the 
older and the entrenched) are usually the last to wake up to the new 
reality.

David



Bill Janssen wrote:
> Folks,
>
> Please forgive this somewhat off-topic message.  I'm looking for more
> funding to work on my UpLib IMAP server implementation.  UpLib stores
> all messages in a DB, and uses tags for folders, thus storing only one
> copy of messages that might occur in multiple folders, or are stored by
> multiple users.  I'm interested in figuring some strawman stats about
> email duplication that could be addressed by a site-wide IMAP server.
> For example, how many mailing lists an "average" organization has; the
> average number of users per mailing list; the average message traffic
> (in both messages and bytes) per day, per week, per year.  I know that a
> number of you have actual experience with this.  I'd be grateful for any
> pointers.
>
> Thanks.
>
> Bill
> _______________________________________________
> Imap-protocol mailing list
> Imap-protocol@u.washington.edu
> http://mailman2.u.washington.edu/mailman/listinfo/imap-protocol
>

Bill Janssen writes:
> I'm interested in figuring some strawman stats about email duplication 
> that could be addressed by a site-wide IMAP server. For example, how 
> many mailing lists an "average" organization has; the average number 
> of users per mailing list; the average message traffic (in both 
> messages and bytes) per day, per week, per year. I know that a number 
> of you have actual experience with this.

Yes, but unfortunately such numbers tend to be tied a little too tightly 
to private email.

I do know a couple of things, though:

1. The number of mailboxes in which a particular bodypart is stored 
varier per installation. I haven't seen an average greater than 4, I've 
seen several only a little below, and I have seen averages close to 1. 
I'd use 3 or 4 as a number if I were you.

2. Mailing lists are difficult to count. You can count owner-* 
addresses, you can count List-ID fields, but there's a lot of 
unofficial address lists stored in people's address books or even 
heads.

Arnt

On Sun, 12 Jul 2009, Bill Janssen wrote:
>
> I'm interested in figuring some strawman stats about email duplication
> that could be addressed by a site-wide IMAP server. For example, how
> many mailing lists an "average" organization has; the average number of
> users per mailing list; the average message traffic (in both messages
> and bytes) per day, per week, per year.

We have an average of two recipients per message. This underestimates the
amount of de-duplication that you could do, because large mailing lists
are split up into multiple messages, and I don't have any stats that would
allow me to say anything about duplicate forwarded attachments.

Message traffic averages 10 messages per user per day and about 1Mbyte per
user per day (in both cases ignoring multi-recipient messages). Number of
messages isn't increasing much, but message size is going up by about
12.5% per annum.

I work for Cambridge University. We have quite a skewed user base - I
expect if I had stats covering just staff the numbers would be rather
higher, plus I don't have any stats on traffic inside departmental
Exchange servers such as the one used by about 1000 administrative staff.

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

David Rauschenbach <davidr@imap.cc> wrote:

> In a MIME store, lots of people forward the same attachment, but in
> each case, the part header as different headers. Some MUAs use a
> content-type "name" attribute field-param to store the file name,
> others use the Content-Disposition "filename" field-param, etc. But in
> both cases, you really want to store the 15 MB zip file attachment
> just once.

Yes, I should really do that, too.  I'm close already; attachments are
stored as separate UpLib documents, with pointers to that document in
the message.  But the pointers are document IDs, not document
fingerprints, so duplicates are possible.

> As for what is profitable, I think Facebook, Twitter, et. al. are
> successfully demonstrating that, en-masse, communication is routing
> around the dinosaur e-mail protocols. Techies like us (especially the
> older and the entrenched) are usually the last to wake up to the new
> reality.

Perhaps.  I'm already thinking about a Google Wave server for UpLib.

Bill

Ah yes, Wave, the new e-mail protocol now being specified by the 
internet's leading marketing & spyware company. Times 'a changin'. Or, 
maybe not at all different from the D.O.J. roots of it all, considering 
the definition of corporatism, and that they're one and the same.

The Wave video demo is pretty good - I'm sure it will do well. As for my 
role in things... the legacy protocols will still be the best way to 
push content out to phones, and in that regard, IMAP has that very 
useful disconnected mode, whereas Wave looks like it's for broadband 
tethered devices only. In other words, pro-sumer Americans who never 
travel abroad (the subset of the subset, globally speaking). Or, "the 
bankrupt subset" might be the more distinct moniker for the next decade.

I suppose Wave might get a disconnected mode shoe-horned in there late, 
like an etag extension. It's easy to forget UIDPLUS was even late to the 
IMAP party.

David



Bill Janssen wrote:
> David Rauschenbach <davidr@imap.cc> wrote:
>
>   
>> In a MIME store, lots of people forward the same attachment, but in
>> each case, the part header as different headers. Some MUAs use a
>> content-type "name" attribute field-param to store the file name,
>> others use the Content-Disposition "filename" field-param, etc. But in
>> both cases, you really want to store the 15 MB zip file attachment
>> just once.
>>     
>
> Yes, I should really do that, too.  I'm close already; attachments are
> stored as separate UpLib documents, with pointers to that document in
> the message.  But the pointers are document IDs, not document
> fingerprints, so duplicates are possible.
>
>   
>> As for what is profitable, I think Facebook, Twitter, et. al. are
>> successfully demonstrating that, en-masse, communication is routing
>> around the dinosaur e-mail protocols. Techies like us (especially the
>> older and the entrenched) are usually the last to wake up to the new
>> reality.
>>     
>
> Perhaps.  I'm already thinking about a Google Wave server for UpLib.
>
> Bill
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20090712/87d5edab/attachment.html>

David Rauschenbach writes:
> As for my role in things... the legacy protocols will still be the 
> best way to push content out to phones, and in that regard, IMAP has 
> that very useful disconnected mode, whereas Wave looks like it's for 
> broadband tethered devices only.

IMAP has the very useful property that it uses ordinary email addresses. 
The kind of address which everyone has.

IPeople have predicted the death of 822 email for some time now; doesn't 
seem to happen. Anecdote alert I suppose, but the last time my dance 
club asked for its members' contact details, email had shifted _up_ to 
the top spot.

Arnt