mailing list archives

Hi,

I'm implementing RCS message storage server (which is used for Joyn
messaging). RCS uses IMAP to access/synchronize chat logs from multiple
clients.

RCS specification specifies that new mailbox should be created for every
new conversation (1-1 chat, and group chat). It implies that a user may
have hundreds of mailboxes. IMAP has several extensions that helps to
reduce both number of commands and bandwidth while synchronizing single
mailbox (for example, RFC5162). However it seems that there is no extension
to help synchronizing multiple mailboxes at once. That means that if a user
has a 100 mailboxes, it will execute at least 100 commands to server every
time it lost connection between server, which is quite frequent in mobile
environment. Our goal is a build a server which can handle million
subscribers, and it seems infeasible to build such server when each clients
send 100 commands every time it reconnects to server.

Is there any solutions or IMAP extensions that solves the problem?


Jihyun Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130614/ead97826/attachment.html>

On Fri, Jun 14, 2013, at 06:50 PM, Jihyun Yu wrote:

Hi,

I'm implementing RCS message storage server (which is used for Joyn
messaging). RCS uses IMAP to access/synchronize chat logs from multiple
clients.

RCS specification specifies that new mailbox should be created for
every new conversation (1-1 chat, and group chat). It implies that a
user may have hundreds of mailboxes. IMAP has several extensions that
helps to reduce both number of commands and bandwidth while
synchronizing single mailbox (for example, RFC5162). However it seems
that there is no extension to help synchronizing multiple mailboxes at
once. That means that if a user has a 100 mailboxes, it will execute at
least 100 commands to server every time it lost connection between
server, which is quite frequent in mobile environment. Our goal is a
build a server which can handle million subscribers, and it seems
infeasible to build such server when each clients send 100 commands
every time it reconnects to server.

Is there any solutions or IMAP extensions that solves the problem?


I suspect you're looking for RFC5819 here - LIST RETURN STATUS.  You
can then use RFC5162 for each mailbox with a changed HIGHESTMODSEQ to
get the actual changes, so it become 1 command plus another 1 command
per mailbox which has changed since disconnection.

We cheat at FastMail - we've got a patched Cyrus server which uses a
single HIGHESTMODSEQ counter per user, as well as a single
MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on
rename, create or delete (keeping a tombstone record), we only need to
keep track of two numbers - one for any changes to the folder listing,
and a second for any changes to any mailbox.  This allows us to
optimise our web interface significantly (and provide the same
advantages to any future clients that talk to our JSON API directly).

BUT - if you already control both the client and the server, you can do
whatever you want.  If you don't control the client, you probably can't
force it to use your fancy extensions anyway.

Good luck,
Bron.
-- 
  Bron Gondwana
  brong@fastmail.fm

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman13.u.washington.edu/pipermail/imap-protocol/attachments/20130614/76f6b235/attachment.html>

On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:

> We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).

Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?

On Fri, Jun 14, 2013, at 10:22 PM, Timo Sirainen wrote:
> On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:
> 
> > We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).
> 
> Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?

It has no guarantee anyway.  You can't keep the UIDVALIDITY on rename if the destination name previously existed with that same UIDVALIDITY.  Which means it either has to either trust that it's still the same folder after rename...

Anyway, rename of folders is a pretty rare case all around, so I'm not super concerned about it.  We could probably bump the MAXUIDVALIDITY without actually changing the folder's UIDVALIDITY and it will still work though.  I can do that now that we're tracking deleted folder UIDVALIDITY.

Bron.

-- 
  Bron Gondwana
  brong@fastmail.fm

On 14.6.2013, at 15.36, Bron Gondwana <brong@fastmail.fm> wrote:

> On Fri, Jun 14, 2013, at 10:22 PM, Timo Sirainen wrote:
>> On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:
>> 
>>> We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).
>> 
>> Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?
> 
> It has no guarantee anyway.  You can't keep the UIDVALIDITY on rename if the destination name previously existed with that same UIDVALIDITY.  Which means it either has to either trust that it's still the same folder after rename?

I'm increasing UIDVALIDITY every time a new mailbox is created, so no two mailboxes have the same. This avoids conflicts with RENAME.

On Sat, Jun 15, 2013, at 03:55 AM, Timo Sirainen wrote:
> On 14.6.2013, at 15.36, Bron Gondwana <brong@fastmail.fm> wrote:
> 
> > On Fri, Jun 14, 2013, at 10:22 PM, Timo Sirainen wrote:
> >> On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:
> >> 
> >>> We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).
> >> 
> >> Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?
> > 
> > It has no guarantee anyway.  You can't keep the UIDVALIDITY on rename if the destination name previously existed with that same UIDVALIDITY.  Which means it either has to either trust that it's still the same folder after rename?
> 
> I'm increasing UIDVALIDITY every time a new mailbox is created, so no two mailboxes have the same. This avoids conflicts with RENAME.

We do this as well - but only within individual users, and mailboxes can be copied between users (yay shared mailboxes), so that would become prohibitively messy on a really large Cyrus instance.

Bron.


-- 
  Bron Gondwana
  brong@fastmail.fm

On 15.6.2013, at 0.42, Bron Gondwana <brong@fastmail.fm> wrote:

> On Sat, Jun 15, 2013, at 03:55 AM, Timo Sirainen wrote:
>> On 14.6.2013, at 15.36, Bron Gondwana <brong@fastmail.fm> wrote:
>> 
>>> On Fri, Jun 14, 2013, at 10:22 PM, Timo Sirainen wrote:
>>>> On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:
>>>> 
>>>>> We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).
>>>> 
>>>> Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?
>>> 
>>> It has no guarantee anyway.  You can't keep the UIDVALIDITY on rename if the destination name previously existed with that same UIDVALIDITY.  Which means it either has to either trust that it's still the same folder after rename?
>> 
>> I'm increasing UIDVALIDITY every time a new mailbox is created, so no two mailboxes have the same. This avoids conflicts with RENAME.
> 
> We do this as well - but only within individual users, and mailboxes can be copied between users (yay shared mailboxes), so that would become prohibitively messy on a really large Cyrus instance.

I don't allow RENAME across user boundaries. Seems potentially dangerous.

On Sat, Jun 15, 2013, at 07:49 AM, Timo Sirainen wrote:
> On 15.6.2013, at 0.42, Bron Gondwana <brong@fastmail.fm> wrote:
> 
> > On Sat, Jun 15, 2013, at 03:55 AM, Timo Sirainen wrote:
> >> On 14.6.2013, at 15.36, Bron Gondwana <brong@fastmail.fm> wrote:
> >> 
> >>> On Fri, Jun 14, 2013, at 10:22 PM, Timo Sirainen wrote:
> >>>> On 14.6.2013, at 13.08, Bron Gondwana <brong@fastmail.fm> wrote:
> >>>> 
> >>>>> We cheat at FastMail - we've got a patched Cyrus server which uses a single HIGHESTMODSEQ counter per user, as well as a single MAXUIDVALIDITY.  Since we always bump the UIDVALIDITY of folders on rename, create or delete (keeping a tombstone record), we only need to keep track of two numbers - one for any changes to the folder listing, and a second for any changes to any mailbox.  This allows us to optimise our web interface significantly (and provide the same advantages to any future clients that talk to our JSON API directly).
> >>>> 
> >>>> Won't the UIDVALIDITY change on RENAME invalidate the local cache even for the client that issued the RENAME command?
> >>> 
> >>> It has no guarantee anyway.  You can't keep the UIDVALIDITY on rename if the destination name previously existed with that same UIDVALIDITY.  Which means it either has to either trust that it's still the same folder after rename?
> >> 
> >> I'm increasing UIDVALIDITY every time a new mailbox is created, so no two mailboxes have the same. This avoids conflicts with RENAME.
> > 
> > We do this as well - but only within individual users, and mailboxes can be copied between users (yay shared mailboxes), so that would become prohibitively messy on a really large Cyrus instance.
> 
> I don't allow RENAME across user boundaries. Seems potentially dangerous.

Legacy - gotta love it.  You can't take things away from people...

That said, this is probably something that could be a config switch defaulting to "don't allow it unless the user is an admin" or something.

Bron.

-- 
  Bron Gondwana
  brong@fastmail.fm