[SOGo] Problems on large mailbox for activesync on Outlook 2013

[Jacques Morrisseau]

Hello,

I use ActiveSync with Sogo for mobile clients access which is mostly
working. When I configure a test user on Outlook 2013, everything seems
to be working, but when I try to configure longtime users with large
IMAP mailboxes the folders appears but they stay empty.

It's as if the the request takes too long and something gives up
somewhere along the way. I have an Nginx server as an entry point and
the standard Apache server along with the sogo version 2.2.8 server in
the back. I tried adjusting the timeout value of both and it just seems
to take longer to fail.

* Are there any other value I should look into?
o Such as " SOGoMaximumSyncWindowSize", presently set to 50.
* Do you have any pointers of stuff to look at or any troubleshooting
tips.

Thanks!
--
*Jacques Morrisseau, *
IT manager

Innobec <http://www.innobec.com>

*T* 1-514-272-0979 *#217*
*T* 1-800-972-0979 *#217*
*C* 438-883-0390
www.innobec.com <http://www.innobec.com>

[Ludovic Marcotte]

Post by Jacques Morrisseau
I use ActiveSync with Sogo for mobile clients access which is mostly
working. When I configure a test user on Outlook 2013, everything
seems to be working, but when I try to configure longtime users with
large IMAP mailboxes the folders appears but they stay empty.

What's the value of imap_max_line_length in your Dovecot configuration?

Perhaps SOGo is generating a very long command and Dovecot rejects it.

Thanks,

[Jacques Morrisseau]

Here's what I get from "dovecot -a"

imap_max_line_length = 64 k

I think it's the default value.

*Jacques Morrisseau, *
IT manager

Innobec <http://www.innobec.com>

*T* 1-514-272-0979 *#217*
*T* 1-800-972-0979 *#217*
*C* 438-883-0390
www.innobec.com <http://www.innobec.com>

Post by Ludovic Marcotte

Post by Jacques Morrisseau
I use ActiveSync with Sogo for mobile clients access which is mostly
working. When I configure a test user on Outlook 2013, everything
seems to be working, but when I try to configure longtime users with
large IMAP mailboxes the folders appears but they stay empty.

What's the value of imap_max_line_length in your Dovecot configuration?
Perhaps SOGo is generating a very long command and Dovecot rejects it.
Thanks,
--
Ludovic Marcotte
Inverse inc. :: Leaders behind SOGo (http://sogo.nu) and PacketFence (http://packetfence.org)

[Ludovic Marcotte]

Post by Jacques Morrisseau
imap_max_line_length = 64 k

Quadruple it.

[Jacques Morrisseau]

Did that, left Outlook 2013 running for the night, still not getting my
mail.

Any other suggestions?

Thanks!

*Jacques Morrisseau, *
IT manager

Innobec <http://www.innobec.com>

*T* 1-514-272-0979 *#217*
*T* 1-800-972-0979 *#217*
*C* 438-883-0390
www.innobec.com <http://www.innobec.com>

Post by Ludovic Marcotte

Post by Jacques Morrisseau
imap_max_line_length = 64 k

Quadruple it.

[Ludovic Marcotte]

Post by Jacques Morrisseau
Any other suggestions?

How big is the mail folder? In terms of number of emails and size in MB?

[Miroslav Zubcic]

Post by Ludovic Marcotte

Post by Jacques Morrisseau
Any other suggestions?

How big is the mail folder? In terms of number of emails and size in MB?

Hi.

I have the same problem, but I went a step further. I'm testing SOGo for
a 2 weeks now (2.2.8, 2.2.9 on CentOS 6.5). Small mailboxes are synced
with Outlook 2013 and Android 4.2.1 exchange/activesync client, but when
I try this with a long living big IMAP users (3-4 GB). Things are
failing on default WOWatchDogRequestTimeout timeout ... enlarged to 60
minutes after I notice this.

I have used cyrus-imapd 2.3.16 - there is no chance to get big mailbox
useful in Outlook.

I get better results with setup of cyrus-imapd 2.4.17 on SOGo server
itself yesterday, and synced with imapsync one large mailbox from 2.3.16
(2.7 GB), and reconfigured SOGo to use new cyrus imap server. Now, I can
get last month of INBOX synced in reasonable (3-4 minutes) time, even a
whole INBOX in 10-12 minutes, but only main INBOX -- IMAP subfolders
(20-30 of them) are not synced ever - they remain empty in Android and
in Outlook too (default calendar and contacts imported via web interface
are synced in Outlook). I have waited for 2 hours, just in case.

BTW, all timeouts are set 5-6 times bigger from default (sogo.conf,
Apache proxy) in order to get enough time for slow big actions, but
after optimization on IMAP side, timeouts are not an issue anymore.

This are some interesting errors from /var/log/sogo/sogo.conf:

Oct 03 12:37:54 sogod [16001]: [ERROR]
<0x0x7f2b6ad6c038[NGImap4Connection]> could not sort contents of URL:
imaps://mzubcic%***@localhost/%28null%29/?tls=YES

Oct 03 12:41:52 sogod [16000]: <0x0x7f2b6e271648[NGImap4Client]> Note:
no key found for sorting, using 'DATE': (null)

This two lines are repeating in sogo.log 2-3 every couple of seconds
while main INBOX is opened in Outlook 2013. Notice this "%28null%29" in
place of mailbox name. List/structure of mailboxes/folders is always
synced correctly in Outlook.

If I enable ImapDebugEnabled in sogo.conf, I can observe in sogo.log big
amount of IMAP operations that seems it is in some loop and never
ending. Like this:

S[0x7f3147e42c08]: * ANNOTATION "INBOX" "/comment" ("value.priv"
"srv1_labos_test_3e80_19de9a86_20")
S[0x7f3147e42c08]: * ANNOTATION "INBOX.Drafts" "/comment" ("value.priv"
"srv1_labos_test_4f11_19dee082_653")
S[0x7f3147e42c08]: * ANNOTATION "INBOX.Sent" "/comment" ("value.priv"
"srv1_labos_test_4f11_19dee082_646")
...

and then:

C[0x7f3147e32678]: 4439 setannotation "INBOX.Sent" "/comment"
("value.priv" "srv1_labos_test_4f11_19dee082_f92")
S[0x7f3147e42c08]: 4439 OK Completed
C[0x7f3147e32678]: 4440 setannotation "INBOX.oracle" "/comment"
("value.priv" "srv1_labos_test_4f11_19dee082_f93")
S[0x7f3147e42c08]: 4440 OK Completed

after that:

C[0x7f3147e32678]: 4465 unselect
S[0x7f3147e42c08]: 4465 BAD Please select a mailbox first
C[0x7f3147e32678]: 4466 select "(null)"
S[0x7f3147e42c08]: 4466 NO Mailbox does not exist
C[0x7f3147e32678]: 4467 unselect
S[0x7f3147e42c08]: 4467 BAD Please select a mailbox first
C[0x7f3147e32678]: 4468 select "(null)"
S[0x7f3147e42c08]: 4468 NO Mailbox does not exist
C[0x7f3147e32678]: 4469 getannotation "*" "/comment" "value.priv"

... and than it restarts the whole from the beggining with ANNOTATION
"INBOX" etc ...

[Jacques Morrisseau]

The total size for my own folder is 2.5GB (with more or less 60000
email), the one for my boss is 40GB (unknown number of file at this moment).


*Jacques Morrisseau, *
IT manager

Innobec <http://www.innobec.com>

*T* 1-514-272-0979 *#217*
*T* 1-800-972-0979 *#217*
*C* 438-883-0390
www.innobec.com <http://www.innobec.com>

Post by Ludovic Marcotte

Post by Jacques Morrisseau
Any other suggestions?

How big is the mail folder? In terms of number of emails and size in MB?

[Ludovic Marcotte]

Hi,

We'll run some tests early week with a folder having 100 000 messages in
it. I'll keep you posted on the results.

Thanks,

Post by Jacques Morrisseau
The total size for my own folder is 2.5GB (with more or less 60000
email), the one for my boss is 40GB (unknown number of file at this moment).

[Jacques Morrisseau]

An update on my side,

After letting the client off for a while, I've turned it on to check on
some settings then forget about it. After maybe 30-60 minutes, I
noticed that Outlook 2013 was still on. I was about to close it when I
noticed that I could see my mails, my events and my adress book.

My boss is still not getting any luck on his side, but he his going to
let it run tonight from his home for the night.

We had some problems with Sogod today, so I reduced the number of
worker from 50 to 40 and SxVMemLimit from 512 to 384. Sogod was eating
away all of the memory and crashed the server once this morning and
almost a second time around noon. But then again it might have been the
Outlook client. We will investigate further outside business hour.

Thanks for looking it up and have a great weekend!

*Jacques Morrisseau, *
IT manager

Innobec <http://www.innobec.com>

*T* 1-514-272-0979 *#217*
*T* 1-800-972-0979 *#217*
*C* 438-883-0390
www.innobec.com <http://www.innobec.com>

Post by Ludovic Marcotte
Hi,
We'll run some tests early week with a folder having 100 000 messages
in it. I'll keep you posted on the results.
Thanks,

Post by Jacques Morrisseau
The total size for my own folder is 2.5GB (with more or less 60000
email), the one for my boss is 40GB (unknown number of file at this moment).

[Ludovic Marcotte]

Post by Jacques Morrisseau
The total size for my own folder is 2.5GB (with more or less 60000
email), the one for my boss is 40GB (unknown number of file at this moment).

To handle large folders like that, you would need to increase the word
limit on your IMAP server much more.

Say you have 60 000 messages and they each have an IMAP UID larger than
10000 - 5 bytes are used, plus each comma separating them, during a
request. So basically you would need a buffer of around 350 KB.

The parameter to adjust in Dovecot is "imap_max_line_length" while on
Cyrus, it's "maxword".

I've just tried with 100 000 messages on Cyrus and a 2MB buffer was
good. Outlook is happily syncing chunks of 1024 messages.

Ultimately, we would need to modify SOPE to split very large requests
into smaller ones but for now, we don't do it and you can adjust the
parameters from your IMAP server to better handle this. The
documentation has just been updated for that.

Ludo