View Issue Details

IDProjectCategoryView StatusLast Update
0005416SOGoBackend Calendarpublic2023-01-31 13:44
Reporterlandry Assigned Tosebastien  
PrioritynormalSeveritymajorReproducibilityalways
Status closedResolutionfixed 
Product Version5.2.0 
Fixed in Version5.8.0 
Summary0005416: sogo mysteriously adds XML-encoded UTF-8 BOM to calendar-data & getetag REPORT replies
Description

using sogo with thunderbird clients to access calendars, previously running 5.0.1 on OpenBSD 6.9. frontend nginx 1.18, backend postgresql 12.6, database with UTF8 encoding and C locale.

upgraded the complete stack to OpenBSD 7.0, with SOGo 5.2.0, nginx 1.20, postgresql 12.8 - suddenly the events saved by thunderbird cant be displayed/parsed back by thunderbird, which loudly complaints (filed https://bugzilla.mozilla.org/show_bug.cgi?id=1737067 with details about thunderbird)

Thunderbird complains that it cant find the BEGIN:VCALENDAR item in the returned ICS file:

Calendar: Failed to parse item: BEGIN:VCALENDAR

Dug a bit more with tcpdump/wireshark to look at what was sent back by sogo after PUT'ing an event, and the content mysteriously contains XML-encoded UTF-8 BOM character (eg ) for the calendar-data and getetag XML tags:

...
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:response><D:href>/SOGo/dav/breuil/Calendar/personal/fc0baba2-1a35-4f51-a4b8-990e04f68b47.ics</D:href><D:propstat><D:prop><C:calendar-data>BEGIN:VCALENDAR...
...
END:VCALENDAR</C:calendar-data><D:getetag>"gcs00000000"</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>

afaict In the psql database, the event c_content field doesnt seem to contain the UTF-8 char...

sogo=> select c_name,substr(c_content,0,16) From sogo_store where c_content like '%fc0baba2-1a35-4f51-a4b8-990e04f68b47
%';
                  c_name                  |     substr      
------------------------------------------+-----------------
 fc0baba2-1a35-4f51-a4b8-990e04f68b47.ics | BEGIN:VCALENDAR

although i'm not 100% sure as printing only the first char of the field prints nothing (unsure if substr starts at pos 0 or 1..)

This is ... mystifying. The same thunderbird instance can display events saved in davical, so it's not a thunderbird issue.

Keeping OpenBSD 7.0, i've tried reverting to SOGo 5.0.1, no improvement. To nginx 1.18, no improvement.

I'm looking for help figuring out where in the sogo code the REPORT replies are generated, i've perused the sogo changelog between 5.0.1 and 5.2.0 and didnt find anything vaguely related to this.

TagsCalDAV

Activities

landry

landry

2021-10-26 07:26

reporter   ~0015581

of course the strange XML-encoded BOM is stripped by mantis HTML parseer, but i'm attaching the resulting XML showing that the opening tags calendar-data getetag start by & followed by #65279;

mget.ics (1,159 bytes)   
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:response><D:href>/SOGo/dav/breuil/Calendar/personal/fc0baba2-1a35-4f51-a4b8-990e04f68b47.ics</D:href><D:propstat><D:prop><C:calendar-data>&#65279;BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Europe/Paris
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
CREATED:20160908T082555Z
LAST-MODIFIED:20160908T082611Z
DTSTAMP:20160908T082611Z
UID:fc0baba2-1a35-4f51-a4b8-990e04f68b47
SUMMARY:Appel PIGMA georchestra
DTSTART;TZID=Europe/Paris:20160909T100000
DTEND;TZID=Europe/Paris:20160909T110000
TRANSP:OPAQUE
CLASS:PUBLIC
END:VEVENT
END:VCALENDAR</C:calendar-data><D:getetag>&#65279;&quot;gcs00000000&quot;</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>
mget.ics (1,159 bytes)   
landry

landry

2021-10-26 08:15

reporter   ~0015582

to be 100% clear, with a space added to workaround the parser 'eating' the xml-encoded char, here's the edited content i get on the wire:

...
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:response><D:href>/SOGo/dav/breuil/Calendar/personal/fc0baba2-1a35-4f51-a4b8-990e04f68b47.ics</D:href><D:propstat><D:prop><C:calendar-data>& #65279;BEGIN:VCALENDAR...
...
END:VCALENDAR</C:calendar-data><D:getetag>& #65279;"gcs00000000"</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response></D:multistatus>

landry

landry

2021-10-26 13:40

reporter   ~0015583

after hours of debugging, found out the culprit: gnustep-base had got upgraded from 1.27.0 to 1.28.0, and if i revert this gnustep-base upgrade (and build sogo/sope 5.2.0 on top) i have a working setup, no more crazy & #65279; hidden char in the caldav transaction.

now to peruse the gnustep-base changelog to figure out what broke it..

landry

landry

2021-10-26 14:06

reporter   ~0015584

filed https://github.com/gnustep/libs-base/issues/212 at gnustep-base repo.

landry

landry

2021-10-26 14:50

reporter   ~0015585

after reverting https://github.com/gnustep/libs-base/commit/bd5f2909e6edc8012a0a6e44ea1402dfbe1353a4 on top of gnustep 1.28.0, i have a working sogo 5.2.0. So that's a sogo regression from this commit, dunno if sogo/sope should adapt to this behaviour.

mcmer

mcmer

2021-10-28 13:21

reporter   ~0015591

I've been bitten by this, too, with the same update history.
I tested with DAVx5 for android (no logs), thunderbird/lightning for OpenBSD (no logs), em client for windows (client log attached), caldavsynchronizer/outlook for windows (client log attached) and cadaver (a command line DAV client, which did not expose the issue, a cat(1) of the .ics is attached).
The symptom was that you could add events but not change them.
@landry's findings and patch saved my day, thanks!

Dear team sogo, is the change in gnustep/libs-base wrong?
Or is it correct and you will adjust?

sogo.err.emclient.txt (5,326 bytes)   
11:50:00 merma@phnx [CalDAV / CardDAV] MailClient.Accounts.ConnectionException: The format of value '"gcs00000001"' is invalid.

          ---> System.FormatException: The format of value '"gcs00000001"' is invalid.

            at System.Net.Http.Headers.HttpHeaderParser.ParseValue(String value, Object storeValue, Int32& index)

            at System.Net.Http.Headers.HttpHeaders.ParseAndAddValue(HeaderDescriptor descriptor, HeaderStoreItemInfo info, String value)

            at System.Net.Http.Headers.HttpHeaders.Add(HeaderDescriptor descriptor, String value)

            at System.Net.Http.Headers.HttpHeaders.Add(String name, String value)

            at MailClient.Protocols.CalDav.ProtocolCommands.PutCommand.Prepare(HttpRequestMessage request)

            at MailClient.Protocols.CalDav.ProtocolCommands.Connector.RunCommand(CalDavAccount account, ICommand command, CancellationToken cancellationToken)

            --- End of inner exception stack trace ---

            at MailClient.Protocols.CalDav.ProtocolCommands.Connector.RunCommand(CalDavAccount account, ICommand command, CancellationToken cancellationToken)

            at MailClient.Protocols.CalDav.Synchronizer.UploadCommand`1.DoUpload(Uri selectPath, TStorageItem item, Boolean overwrite)

            at MailClient.Protocols.CalDav.Synchronizer.UploadCommand`1.Execute(WorkerStatus status)

            at MailClient.Commands.Command.Process(WorkerStatus status)

           EnqueuedStackTrace =

                at MailClient.Storage.Application.Item`2.EndUpdate()

                at MailClient.Storage.Application.Schedule.ScheduleItem`1.EndUpdate()

                at MailClient.UI.Forms.formEvent.SaveDataToEventWithDescription(String innerHtmlText, String innerDocumentHtml, Action`1 onSuccess, Boolean saveToFolder, EventItem& evnt)

                at MailClient.UI.Forms.formEvent.<>c__DisplayClass62_1.<SaveDataToEvent>b__1(String innerDocumentHtml)

                at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.<>c__DisplayClass209_0.<GetCleanedDocumentHtml>b__0(CefProcessMessage message)

                at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.<>c__DisplayClass231_0.<EvaluateJavaScript>b__0(CefProcessMessage message, CefFrame f)

                at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebBrowserEx.OnMessageReceived(CefProcessMessage message)

                at MailClient.Common.UI.Controls.CefWebBrowserEx.CefWebClientEx.OnProcessMessageReceived(CefBrowser browser, CefProcessId sourceProcess, CefProcessMessage message)

                at Xilium.CefGlue.CefClient.on_process_message_received(cef_client_t* self, cef_browser_t* browser, CefProcessId source_process, cef_process_message_t* message)

                at Interop.User32.DispatchMessageW(MSG& msg)

                at System.Windows.Forms.Application.ComponentManager.Interop.Mso.IMsoComponentManager.FPushMessageLoop(UIntPtr dwComponentID, msoloop uReason, Void* pvLoopData)

                at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(msoloop reason, ApplicationContext context)

                at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(msoloop reason, ApplicationContext context)

               at System.Windows.Forms.Application.Run(ApplicationContext context)

                at MailClient.Program.<>c.<Main>b__169_1()

                at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)

                at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

                at System.Delegate.DynamicInvokeImpl(Object[] args)

                at System.Delegate.DynamicInvoke(Object[] args)

                at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry tme)

                at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)

                at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

                at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

                at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry tme)

                at System.Windows.Forms.Control.InvokeMarshaledCallbacks()

                at System.Windows.Forms.Control.WndProc(Message& m)

                at System.Windows.Forms.ScrollableControl.WndProc(Message& m)

                at System.Windows.Forms.ContainerControl.WndProc(Message& m)

                at System.Windows.Forms.Form.WndProc(Message& m)

                at MailClient.Common.UI.Forms.BaseForm.WndProc(Message& m)

                at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

                at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

                at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, WM msg, IntPtr wparam, IntPtr lparam)

                at Xilium.CefGlue.Interop.libcef.run_message_loop()

                at MailClient.Program.Main(String[] args)

           GUIStatus_Exception_Reported = True
sogo.err.emclient.txt (5,326 bytes)   
sogo.cadaver-dav.ics.txt (996 bytes)   
$ cat 53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics

Displaying
`/SOGo/dav/merma/Calendar/personal/53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics'
:

BEGIN:VCALENDAR

PRODID:DAVx5/4.0-ose ical4j/3.1.0 (org.lineageos.etar)

VERSION:2.0

BEGIN:VTIMEZONE

TZID:Europe/Vienna

LAST-MODIFIED:20201010T011803Z

BEGIN:DAYLIGHT

TZNAME:CEST

TZOFFSETFROM:+0100

TZOFFSETTO:+0200

DTSTART:19810329T020000

RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU

END:DAYLIGHT

BEGIN:STANDARD

TZNAME:CET

TZOFFSETFROM:+0200

TZOFFSETTO:+0100

DTSTART:19961027T030000

RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU

END:STANDARD

END:VTIMEZONE

BEGIN:VEVENT

DTSTAMP:20211028T091255Z

UID:53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0

SUMMARY:Test22

LOCATION:Test22

DESCRIPTION:Test22

DTSTART;TZID=Europe/Vienna:20211030T070000

DTEND;TZID=Europe/Vienna:20211030T080000

STATUS:CONFIRMED

BEGIN:VALARM

TRIGGER:-PT1H

ACTION:DISPLAY

DESCRIPTION:Test2

END:VALARM

CLASS:PUBLIC

END:VEVENT

END:VCALENDARdav:/SOGo/dav/merma/Calendar/personal/>

sogo.cadaver-dav.ics.txt (996 bytes)   
sogo.err.caldavsynchronizer.txt (1,817 bytes)   
ERROR 2021-10-28 12:06:35,997 [        24]
onizer.Implementation.CalDavRepository`1 . TryDeserializeCalendar         -
Could not deserialize ICalData of
'/SOGo/dav/merma/Calendar/personal/53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0.ics'
.

DEBUG 2021-10-28 12:06:35,999 [        24]
onizer.Implementation.CalDavRepository`1 . TryDeserializeCalendar         -
ICalData:

?BEGIN:VCALENDAR

PRODID:DAVx5/4.0-ose ical4j/3.1.0 (org.lineageos.etar)

VERSION:2.0

BEGIN:VTIMEZONE

TZID:Europe/Vienna

LAST-MODIFIED:20201010T011803Z

BEGIN:DAYLIGHT

TZNAME:CEST

TZOFFSETFROM:+0100

TZOFFSETTO:+0200

DTSTART:19810329T020000

RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU

END:DAYLIGHT

BEGIN:STANDARD

TZNAME:CET

TZOFFSETFROM:+0200

TZOFFSETTO:+0100

DTSTART:19961027T030000

RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU

END:STANDARD

END:VTIMEZONE

BEGIN:VEVENT

DTSTAMP:20211028T091255Z

UID:53ad3f7a-ee8c-4e64-9fb0-cc02e073fff0

SUMMARY:Test22

LOCATION:Test22

DESCRIPTION:Test22

DTSTART;TZID=Europe/Vienna:20211030T070000

DTEND;TZID=Europe/Vienna:20211030T080000

STATUS:CONFIRMED

BEGIN:VALARM

TRIGGER:-PT1H

ACTION:DISPLAY

DESCRIPTION:Test2

END:VALARM

CLASS:PUBLIC

END:VEVENT

END:VCALENDAR

System.ArgumentOutOfRangeException: Der Index lag au�erhalb des Bereichs. Er
darf nicht negativ und kleiner als die Sammlung sein.

Parametername: index

   bei System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument
argument, ExceptionResource resource)

   bei
CalDavSynchronizer.Implementation.CalDavRepository`1.DeserializeCalendar(Str
ing iCalData, IStringSerializer calendarSerializer)

   bei
CalDavSynchronizer.Implementation.CalDavRepository`1.TryDeserializeCalendar(
String iCalData, IICalendar& calendar, WebResourceName
uriOfCalendarForLogging, IStringSerializer calendarSerializer,
ILoadEntityLogger logger)
lsalle@aventin.com

lsalle@aventin.com

2022-08-01 07:48

reporter   ~0016147

Last edited: 2022-08-01 11:40

I've upgraded a Ubuntu mail server runing Sogo from 20.04 to 22.04 and I can confirm that Thunderbird could not display any newly fetchted calendar events.

Ubuntu 22.04 is providing gnustep-base package 1.28.0.

--
lsalle@svmail:~$ dpkg --list|grep sogo
ii  sogo:amd64                             1:5.7.0-0ubuntu22.04+0                       amd64        a modern and scalable groupware
lsalle@svmail:~$ dpkg --list|grep gnustep
ii  gnustep-base-common                    1.28.0-4build1                               all          GNUstep Base library - common files
ii  gnustep-base-runtime                   1.28.0-4build1                               amd64        GNUstep Base library - daemons and tools
ii  gnustep-common                         2.9.0-3                                      amd64        Common files for the core GNUstep environment
ii  libgnustep-base1.28                    1.28.0-4build1                               amd64        GNUstep Base library
lsalle@svmail:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION=&quot;Ubuntu 22.04.1 LTS&quot;
lsalle@svmail:~$
the_nic

the_nic

2022-08-15 14:35

reporter   ~0016167

I have debugged this a bit and found the cause for this. here is a fix: https://github.com/Alinto/sogo/pull/324

Issue History

Date Modified Username Field Change
2021-10-26 07:12 landry New Issue
2021-10-26 07:12 landry Tag Attached: CalDAV
2021-10-26 07:26 landry Note Added: 0015581
2021-10-26 07:26 landry File Added: mget.ics
2021-10-26 08:15 landry Note Added: 0015582
2021-10-26 13:40 landry Note Added: 0015583
2021-10-26 14:06 landry Note Added: 0015584
2021-10-26 14:50 landry Note Added: 0015585
2021-10-28 13:21 mcmer Note Added: 0015591
2021-10-28 13:21 mcmer File Added: sogo.err.emclient.txt
2021-10-28 13:21 mcmer File Added: sogo.cadaver-dav.ics.txt
2021-10-28 13:21 mcmer File Added: sogo.err.caldavsynchronizer.txt
2022-08-01 07:48 lsalle@aventin.com Note Added: 0016147
2022-08-01 11:40 francis Note Edited: 0016147
2022-08-01 11:42 francis Description Updated
2022-08-15 14:35 the_nic Note Added: 0016167
2023-01-31 13:43 sebastien Assigned To => sebastien
2023-01-31 13:43 sebastien Status new => resolved
2023-01-31 13:43 sebastien Resolution open => fixed
2023-01-31 13:43 sebastien Fixed in Version => 5.8.0
2023-01-31 13:44 sebastien Status resolved => closed