View Issue Details

IDProjectCategoryView StatusLast Update
0005269SOGoWeb Mailpublic2021-03-01 17:03
ReporterRigacci.Org Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionno change required 
PlatformServerOSDebianOS Version10.8 (Buster)
Product Version4.0.7 
Summary0005269: SOGo v4.0.7 very slow performance loading webmail home page (three times slower than v3.2.6)
Description

I just installed SOGo 4.0.7 on a new Debian 10.8 server; the server is not yet in production, so the CPU load is almost zero.

Loading the home page of the webmail requires 8.8 seconds, this compares very badly with another old installation I have, where SOGo 3.2.6 requires only 2.7 seconds (and the old server has less horsepower than the old one!).
I enabled SOGoDebugRequests = YES; and discovered that there are four requests that took more than 2 seconds each, this is confirmed also in the browser console.
I enabled loggin on the Dovecot IMAP server, and I see that the four requests are answered almost immediately; so I think that the problem is related to SOGo only. Performances of CPU, RAM and disk are by far better on the new server that in the old one.
Can be the new 4.0.7 version so slower that the old 3.2.6 one?

How can I trace the SOGo opration to eventually discover bottlenecks?

Steps To Reproduce

Enable browser web console
Open your webmail home page
Start a network performance timing
Reload the page

Additional Information

SOGo and Dovecot log compared: SOGo says "request took 2.xxx seconds to execute", whereas Dovecot IMAP is almost instantaneous.

==== /var/log/sogo/sogo.log (SOGoDebugRequests = YES;) ====

Feb 26 18:21:30 sogod [5224]: 185.21.73.97 "GET /SOGo/so/username@example.org/Calendar/alarmslist?browserTime=1614360090 HTTP/1.1" 200 63/0 0.017 - - 0
Feb 26 18:21:30 sogod [5224]: |SOGo| starting method 'GET' on uri '/SOGo/so/username@example.org/Mail/0/view'
Feb 26 18:21:32 sogod [5225]: |SOGo| request took 2.364151 seconds to execute

Feb 26 18:21:32 sogod [5225]: 185.21.73.97 "POST /SOGo/so/username@example.org/Mail/0/folderDrafts/expunge HTTP/1.1" 204 0/0 2.367 - - 2M
Feb 26 18:21:33 sogod [5224]: |SOGo| request took 2.467569 seconds to execute

Feb 26 18:21:33 sogod [5224]: 185.21.73.97 "GET /SOGo/so/username@example.org/Mail/0/view HTTP/1.1" 200 1025/0 2.469 - - 0
Feb 26 18:21:33 sogod [5224]: |SOGo| starting method 'POST' on uri '/SOGo/so/username@example.org/Mail/0/folderDrafts/view'
Feb 26 18:21:35 sogod [5224]: |SOGo| request took 2.213646 seconds to execute

Feb 26 18:21:35 sogod [5224]: 185.21.73.97 "POST /SOGo/so/username@example.org/Mail/0/folderDrafts/view HTTP/1.1" 204 0/48 2.217 - - 0
Feb 26 18:21:36 sogod [5224]: |SOGo| starting method 'POST' on uri '/SOGo/so/username@example.org/Mail/unseenCount'
Feb 26 18:21:38 sogod [5224]: |SOGo| request took 2.788346 seconds to execute

==== /var/log/mail.log (mail_debug = yes) ====

Feb 26 18:21:31 server dovecot: imap-login: Login: user=<username@example.org>
Feb 26 18:21:32 server dovecot: imap(username@example.org)<5243><KSuxf0C8vL5/AAAB>:
Logged out in=110 out=1162 deleted=0 expunged=0 trashed=0 hdr_count=0
hdr_bytes=0 body_count=0 body_bytes=0

Feb 26 18:21:33 server dovecot: imap-login: Login: user=<username@example.org>
Feb 26 18:21:33 server dovecot: imap(username@example.org)<5246><i9vAf0C8wr5/AAAB>:
Logged out in=130 out=1478 deleted=0 expunged=0 trashed=0 hdr_count=0
hdr_bytes=0 body_count=0 body_bytes=0

Feb 26 18:21:35 server dovecot: imap-login: Login: user=<username@example.org>
Feb 26 18:21:35 server dovecot: imap(username@example.org)<5250><xV7nf0C8xr5/AAAB>:
Logged out in=110 out=1162 deleted=0 expunged=0 trashed=0 hdr_count=0
hdr_bytes=0 body_count=0 body_bytes=0

Feb 26 18:21:38 server dovecot: imap-login: Login: user=<username@example.org>
Feb 26 18:21:38 server dovecot: imap(username@example.org)<5255><5rUQgEC8yr5/AAAB>:
Logged out in=163 out=1622 deleted=0 expunged=0 trashed=0 hdr_count=0
hdr_bytes=0 body_count=0 body_bytes=0

Tagsperformance, slow, webmail

Activities

Rigacci.Org

Rigacci.Org

2021-03-01 11:54

reporter   ~0015113

Doing an strace of the SOGo process I can see a 2 seconds pause during the IMAP login:

10:48:09.580528 sendto(10, "1 login \"username@example"..., 54, MSG_NOSIGNAL, NULL, 0) = 54
10:48:09.580726 sendto(10, "\r\n", 2, MSG_NOSIGNAL, NULL, 0) = 2
10:48:09.580859 recvfrom(10, 0x5562fc27da30, 512, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
10:48:09.580934 poll([{fd=10, events=POLLRDNORM}], 1, 3600000) = 1 ([{fd=10, revents=POLLRDNORM}])
10:48:11.604256 recvfrom(10, "1 OK [CAPABILITY IMAP4rev1 SASL-"..., 512, 0, NULL, NULL) = 379
10:48:11.604717 sendto(10, "2 capability", 12, MSG_NOSIGNAL, NULL, 0) = 12

It is very strange, because talking directly to the IMAP server I cannot see any delay between the login command and the server response.

Rigacci.Org

Rigacci.Org

2021-03-01 16:54

reporter   ~0015115

Good news, finally! It turned out it was an IMAP issue indeed!
Our Dovecot IMAP authentication relies on two backends, where the first one (the Unix PAM subsystem) adds a 2-seconds delay on failures.
So the solution was to swap the order of authentication backends, because the PAM is used only as a fallback if the first one fails.
It was difficult to see that delay, because the IMAP daemon logs the login only once succeded, it does not log start and end timestamps. I used a send/expect script to measure precise IMAP login timing.
Sorry for bothering you all :-(

Issue History

Date Modified Username Field Change
2021-03-01 08:48 Rigacci.Org New Issue
2021-03-01 08:48 Rigacci.Org Tag Attached: performance
2021-03-01 08:48 Rigacci.Org Tag Attached: slow
2021-03-01 08:48 Rigacci.Org Tag Attached: webmail
2021-03-01 11:54 Rigacci.Org Note Added: 0015113
2021-03-01 16:54 Rigacci.Org Note Added: 0015115
2021-03-01 17:03 francis Status new => closed
2021-03-01 17:03 francis Resolution open => no change required