View Issue Details

IDProjectCategoryView StatusLast Update
0002734SOGoActiveSyncpublic2014-06-25 15:07
Reportertfu Assigned Toludovic  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Product Version2.2.3 
Target Version2.2.6Fixed in Version2.2.6 
Summary0002734: mails are not removed from device when outside of the date range to be downloaded
Description

If "Download past mail" is set e.g. to 3 days mails older than that timeframe should be removed from the device if previously synced.
Attached patch is supposed to deal with this "softdeletes".
It might need some improvements (I'm not sure whether all possibilities are converted) and definitely detailed testing ...

TagsNo tags attached.

Activities

tfu

tfu

2014-04-24 15:29

reporter  

0001-mail-softdelete.patch (11,522 bytes)   
From 3e51b66d8ef1656567ec7ceb6e2aa4c2a96cefab Mon Sep 17 00:00:00 2001
From: root <root@example.com>
Date: Thu, 24 Apr 2014 21:21:41 +0200
Subject: [PATCH] mail-softdelete

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |   69 ++++++++++++++++++++++++----
 ActiveSync/SOGoActiveSyncDispatcher.m      |   19 ++++++++
 SoObjects/Mailer/SOGoMailFolder.h          |    3 +-
 SoObjects/Mailer/SOGoMailFolder.m          |   33 +++++++++++++
 4 files changed, 115 insertions(+), 9 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 396c266..0668cb0 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -442,12 +442,41 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   BOOL more_available;
   int i, max;
 
+  // tfu softdelete - extract the 3rd pard of the SyncKey if available (this is the date of the oldest mail synced)
+  NSArray *a;
+  NSString *oldestDate;
+
+  NSLog(@"tfu - old theSyncKey %@", theSyncKey);
+  if (theFolderType == ActiveSyncMailFolder) {
+      a = [theSyncKey componentsSeparatedByString: @"-"];
+      if ([a count] > 2) {
+         oldestDate = [a objectAtIndex: 2];
+         theSyncKey = [NSString stringWithFormat: @"%@-%@",[a objectAtIndex: 0], [a objectAtIndex: 1]];
+      } else {
+        oldestDate = nil;
+      }
+  } else {
+    oldestDate = nil;
+  }
+
+  NSLog(@"tfu - new theSyncKey %@ oldestDate %@", theSyncKey, oldestDate);
+
   //
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
   // We check this and we don't generate any commands if we don't have to.
   //
-  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]])
-    return;
+  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]]) {
+    NSLog(@"tfu no change -- checking cutof date");
+    // tfu if the oldestDate is == to the current filter (cutoff date) then there is nothing to do about softdelets
+    if (oldestDate && theFilterType) {
+         NSLog(@"tfu dayOfCommonEra Oldest %d", [[[NSCalendarDate alloc] initWithTimeIntervalSince1970:[oldestDate intValue]] dayOfCommonEra]);
+         NSLog(@"tfu dayOfCommonEra Filder %d", [theFilterType dayOfCommonEra]);
+         if ( [[[NSCalendarDate alloc] initWithTimeIntervalSince1970:[oldestDate intValue]] dayOfCommonEra] == [theFilterType dayOfCommonEra])
+            return;
+    }
+    else 
+        return;
+  }
   
   s = [NSMutableString string];
   more_available = NO;
@@ -574,10 +603,10 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
         NSArray *allMessages;
         int deleted_count;
 
-        allMessages = [theCollection syncTokenFieldsWithProperties: nil   matchingSyncToken: theSyncKey  fromDate: theFilterType];
+        allMessages = [theCollection syncTokenFieldsWithProperties: nil   matchingSyncToken: theSyncKey  fromDate: theFilterType  oldestDate: oldestDate];
         addedOrChangedMessages = [NSMutableArray array];
         deleted_count = 0;
-
+ 
         // Check for the WindowSize.
         // FIXME: we should eventually check for modseq and slice the maximum
         //        amount of messages returned to ensure we don't have the same
@@ -597,12 +626,20 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
             
             if ([command isEqualToString: @"deleted"])
               {
+		NSLog(@"tfu delete uid=%@", uid);
                 [s appendString: @"<Delete xmlns=\"AirSync:\">"];
                 [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", uid];
                 [s appendString: @"</Delete>"];
                 deleted_count++;
               }
-            else
+            else if ([command isEqualToString: @"softdelete"]) {
+		NSLog(@"tfu softdelete uid=%@", uid);
+                [s appendString: @"<SoftDelete xmlns=\"AirSync:\">"];
+                [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", uid];
+                [s appendString: @"</SoftDelete>"];
+                deleted_count++;
+            }
+             else
               {
                 [addedOrChangedMessages addObject: aMessage];
               }
@@ -633,6 +670,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                 command = @"changed";
               }
             
+            NSLog(@"tfu %@ uid %@", command, uid);
+
             if ([command isEqualToString: @"added"])
               [s appendString: @"<Add xmlns=\"AirSync:\">"];
             else
@@ -768,7 +807,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   NSMutableString *changeBuffer, *commandsBuffer;
   BOOL getChanges, first_sync;
   unsigned int windowSize, v;
-  
+
   changeBuffer = [NSMutableString string];
   commandsBuffer = [NSMutableString string];
   
@@ -783,7 +822,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   
   if (windowSize == 0 || windowSize > 512)
     windowSize = 100;
-  
+
   // We check if we must overwrite the windowSize with a system preference. This can be useful
   // if the user population has large mailboxes and slow connectivity
   if ((v = [[SOGoSystemDefaults sharedSystemDefaults] maximumSyncWindowSize]))
@@ -879,7 +918,21 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   else if (folderType == ActiveSyncTaskFolder)
     [theBuffer appendString: @"<Class>Tasks</Class>"];
   
-  [theBuffer appendFormat: @"<SyncKey>%@</SyncKey>", davCollectionTag];
+
+  //tfu soft-delete - if a date filter is set and changes have been found then there should be no older mails on device -> the "oldestDate" can be updated
+  if ([NSCalendarDate dateFromFilterType: [[(id)[theDocumentElement getElementsByTagName: @"FilterType"] lastObject] textValue]] && [changeBuffer length]) {
+    [theBuffer appendFormat: @"<SyncKey>%@-%@</SyncKey>", davCollectionTag, 
+                                  [NSString stringWithFormat: @"%d",(int)[[NSCalendarDate dateFromFilterType: 
+                                                      [[(id)[theDocumentElement getElementsByTagName: @"FilterType"] lastObject] textValue]] timeIntervalSince1970]]];
+    NSLog(@"tfu serverkey with date x %@ %@",davCollectionTag, 
+                                  [NSString stringWithFormat: @"%d",(int)[[NSCalendarDate dateFromFilterType: 
+                                                      [[(id)[theDocumentElement getElementsByTagName: @"FilterType"] lastObject] textValue]] timeIntervalSince1970]]);
+  }
+  else {
+    [theBuffer appendFormat: @"<SyncKey>%@</SyncKey>", davCollectionTag];
+  }
+
+
   [theBuffer appendFormat: @"<CollectionId>%@</CollectionId>", collectionId];
   [theBuffer appendFormat: @"<Status>%d</Status>", 1];
 
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index b1b8e4e..68925aa 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -634,6 +634,23 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                                               sortOrdering: @"REVERSE ARRIVAL"
                                                   threaded: NO];
       count = [uids count];
+
+      // tfu if nothing found lets see whether there are softdeletes which needs to be handled.
+      if (count == 0) {
+           NSString *syncKey;
+           NSArray *a;
+           NSString *oldestDate;
+
+           syncKey = [[(id)[theDocumentElement getElementsByTagName: @"SyncKey"] lastObject] textValue];
+           a = [syncKey componentsSeparatedByString: @"-"];
+           if ([a count] > 2) {
+              oldestDate = [a objectAtIndex: 2];
+              NSLog(@"tfu GetItmeEstimate dayOfCommonEra Oldest %d", [[[NSCalendarDate alloc] initWithTimeIntervalSince1970:[oldestDate intValue]] dayOfCommonEra]);
+              NSLog(@"tfu GetItmeEstimate dayOfCommonEra Filder %d", [filter dayOfCommonEra]);
+              if ( [[[NSCalendarDate alloc] initWithTimeIntervalSince1970:[oldestDate intValue]] dayOfCommonEra] != [filter dayOfCommonEra] && [filter dayOfCommonEra] > 0)
+                 count=1;
+           }
+      }
     }
   else
     {
@@ -647,6 +664,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   [s appendFormat: @"<CollectionId>%@</CollectionId>", collectionId];
   [s appendFormat: @"<Estimate>%d</Estimate>", count];
   
+  NSLog(@"tfu GetItmeEstimate count %d", count);
+
   [s appendString: @"</Collection></Response></GetItemEstimate>"];
 
   d = [[s dataUsingEncoding: NSUTF8StringEncoding] xml2wbxml];
diff --git a/SoObjects/Mailer/SOGoMailFolder.h b/SoObjects/Mailer/SOGoMailFolder.h
index eac1aec..441031b 100644
--- a/SoObjects/Mailer/SOGoMailFolder.h
+++ b/SoObjects/Mailer/SOGoMailFolder.h
@@ -99,7 +99,8 @@
 
 - (NSArray *) syncTokenFieldsWithProperties: (NSDictionary *) properties
                           matchingSyncToken: (NSString *) syncToken
-                                   fromDate: (NSCalendarDate *) theStartDate;
+                                   fromDate: (NSCalendarDate *) theStartDate
+                                 oldestDate: (NSString *) theOldestDate;
 
 /* flags */
 
diff --git a/SoObjects/Mailer/SOGoMailFolder.m b/SoObjects/Mailer/SOGoMailFolder.m
index 6408828..c42ad8f 100644
--- a/SoObjects/Mailer/SOGoMailFolder.m
+++ b/SoObjects/Mailer/SOGoMailFolder.m
@@ -2030,6 +2030,7 @@ static NSString *defaultUserID =  @"anyone";
 - (NSArray *) syncTokenFieldsWithProperties: (NSArray *) theProperties
                           matchingSyncToken: (NSString *) theSyncToken
                                    fromDate: (NSCalendarDate *) theStartDate
+                                 oldestDate: (NSString *) theOldestDate
 {
   EOQualifier *searchQualifier;
   NSMutableArray *allTokens;
@@ -2122,6 +2123,38 @@ static NSString *defaultUserID =  @"anyone";
           d = [NSDictionary dictionaryWithObject: @"deleted"  forKey: uid];
           [allTokens addObject: d];
         }
+        if (theStartDate)
+        {
+            EOQualifier *sinceDateQualifier, *beforeDateQualifier;
+            EOAndQualifier *qualifier;
+            NSArray *uids;
+
+            beforeDateQualifier = [EOQualifier qualifierWithQualifierFormat:
+                                                       @"(DATE >= %@)", [[NSCalendarDate alloc] initWithTimeIntervalSince1970:[theOldestDate intValue]] ];
+
+
+            sinceDateQualifier = [EOQualifier qualifierWithQualifierFormat:
+                                                       @"(DATE <= %@)", theStartDate ];
+
+
+            qualifier = [[EOAndQualifier alloc] initWithQualifiers: [self _nonDeletedQualifier], sinceDateQualifier, beforeDateQualifier,
+                                          nil];
+            AUTORELEASE(qualifier);
+
+            uids = [self fetchUIDsMatchingQualifier: qualifier
+                                               sortOrdering: nil
+                                                  ];
+
+            for (i = 0; i < [uids count]; i++)
+              {
+                uid = [uids objectAtIndex: i];
+                NSLog(@"tfu uid %@ OldestDate %@", uid , theOldestDate);
+                d = [NSDictionary dictionaryWithObject: @"softdelete"  forKey: uid];
+                [allTokens addObject: d];
+              }
+
+            NSLog(@"tfu softdelete %d theOldestDate %@ theStartDate %@", [uids count], theOldestDate, theStartDate);
+       }
     }
 
   return allTokens;
-- 
1.7.9.5

0001-mail-softdelete.patch (11,522 bytes)   
tfu

tfu

2014-04-29 10:02

reporter  

0002-mail-softdelete-hasPrefix.patch (1,321 bytes)   
From 5a97832e5c3d7acd532fb7352185a02eb1fdf012 Mon Sep 17 00:00:00 2001
From: root <root@example.com>
Date: Tue, 29 Apr 2014 10:22:17 +0200
Subject: [PATCH 2/2] mail-softdelete-hasPrefix

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 0668cb0..9b59b9e 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -465,8 +465,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
   // We check this and we don't generate any commands if we don't have to.
   //
-  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]]) {
-    NSLog(@"tfu no change -- checking cutof date");
+  if ([theSyncKey hasPrefix: [theCollection davCollectionTag]]) {
+    NSLog(@"tfu no change -- checking cutoff date");
     // tfu if the oldestDate is == to the current filter (cutoff date) then there is nothing to do about softdelets
     if (oldestDate && theFilterType) {
          NSLog(@"tfu dayOfCommonEra Oldest %d", [[[NSCalendarDate alloc] initWithTimeIntervalSince1970:[oldestDate intValue]] dayOfCommonEra]);
-- 
1.7.9.5

tfu

tfu

2014-04-29 10:07

reporter   ~0006973

While doing additional tests I found a small amendment to the first patch.

ludovic

ludovic

2014-05-05 14:04

administrator   ~0006990

For this one, instead of storing the date in the SyncKey, I suggest storing it inside our new cache system (coming up!).

In our new cache system, we'll store a structure like this: message-uid -> (sequence, last-seen).

Last-seen could be updated during a Change command, otherwise it'll be set to 'now' when we Add it.

When proceeding with processSyncGetChanges... we could then loop in the cache and soft-delete everything based on the filter-type.

Makes sense?

ludovic

ludovic

2014-05-15 15:12

administrator   ~0007027

The new caching code has landed. You must re-create the ActiveSync profile if you want to test it.

Could you adapt your patch to the new code?

Please have a look at SOGActiveSyncDispatcher+Sync.m and search for "DateCache". I've modified to code to ease your work.

tfu

tfu

2014-05-18 09:50

reporter  

0001-softDelete-with-dateCache.patch (5,375 bytes)   
From 4ab660a94c98924a1f30cbbbf16f43bef509c052 Mon Sep 17 00:00:00 2001
From: root <root@example.com>
Date: Sun, 18 May 2014 15:16:37 +0200
Subject: [PATCH] softDelete with dateCache

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |   40 +++++++++++++++++++++++++++-
 ActiveSync/SOGoActiveSyncDispatcher.h      |    1 +
 ActiveSync/SOGoActiveSyncDispatcher.m      |   25 +++++++++++++++++
 3 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 49cc21c..14fb5db 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -483,7 +483,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
   // We check this and we don't generate any commands if we don't have to.
   //
-  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]])
+  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]] && !([self checkSoftDeletes: [theCollection nameInContainer]]))
     return;
   
   s = [NSMutableString string];
@@ -642,6 +642,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
         else
           folderMetadata = [self _folderMetadataForKey: [theCollection nameInContainer]];
 
+
         syncCache = [folderMetadata objectForKey: @"SyncCache"];
         dateCache = [folderMetadata objectForKey: @"DateCache"];
 
@@ -711,6 +712,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                     [s appendString: @"</Delete>"];
                     
                     [syncCache removeObjectForKey: [aCacheObject uid]];
+                    [dateCache removeObjectForKey: [aCacheObject uid]];
                   }
                 else
                   {
@@ -773,6 +775,42 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                     //NSLog(@"skipping old deleted UID: %@",  [aCacheObject uid]);
                   }
               }
+
+          }
+
+          // handle soft-deletes
+          if (theFilterType) {
+              for (NSString* key in [dateCache allKeys]) {
+                  if (return_count >= theWindowSize)
+                  {
+                      more_available = YES;
+
+                      if (!(*theLastServerKey)) 
+                         *theLastServerKey =  [theCollection davCollectionTag];
+
+                         // make sure that checkSoftDelets find some work
+                         NSString *cacheKey;
+                         cacheKey = [NSString stringWithFormat: @"%@+%@+%@",
+                                  [[context activeUser] login],
+                                  [context objectForKey: @"DeviceId"],
+                                  [theCollection nameInContainer]];
+
+                         [[SOGoCache sharedCache] setValue:  @"0" forKey: cacheKey];
+
+                      break;
+                  }
+
+                  if ([[dateCache objectForKey:key] compare: theFilterType ] == NSOrderedAscending ) 
+                  {
+                     [s appendString: @"<SoftDelete xmlns=\"AirSync:\">"];
+                     [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", key];
+                     [s appendString: @"</SoftDelete>"];
+
+                     [dateCache removeObjectForKey: key];
+                     [syncCache removeObjectForKey: key];
+                     return_count++;
+                  } 
+              }
           }
         
         if (more_available)
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.h b/ActiveSync/SOGoActiveSyncDispatcher.h
index eda6a3d..2dd7acd 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.h
+++ b/ActiveSync/SOGoActiveSyncDispatcher.h
@@ -49,5 +49,6 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 - (NSURL *) folderTableURL;
 - (void) ensureFolderTableExists;
+- (BOOL) checkSoftDeletes: (NSString *) theFolder;
 
 @end
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index d8c2807..b035cab 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -165,6 +165,26 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   return [o properties];
 }
 
+- (BOOL) checkSoftDeletes: (NSString *) theFolder
+{
+
+   NSString *key, *value;
+
+   key = [NSString stringWithFormat: @"%@+%@+%@",
+                                  [[context activeUser] login],
+                                  [context objectForKey: @"DeviceId"],
+                                  theFolder];
+   value = [NSString stringWithFormat: @"%d", [[NSCalendarDate date] dayOfCommonEra]];
+
+   if ([[[SOGoCache sharedCache] valueForKey: key] isEqualToString: value])  {
+      return false;
+   }
+
+   [[SOGoCache sharedCache] setValue: value forKey: key];
+
+  return true;
+}
+
 //
 //
 //
@@ -668,6 +688,11 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                                               sortOrdering: @"REVERSE ARRIVAL"
                                                   threaded: NO];
       count = [uids count];
+
+      if (count == 0) {
+         if ( [self checkSoftDeletes: [currentCollection nameInContainer]] )
+            count = 1;
+      }
     }
   else
     {
-- 
1.7.9.5

tfu

tfu

2014-05-18 09:50

reporter   ~0007048

Please review attached patch.

ludovic

ludovic

2014-05-22 09:32

administrator   ~0007066

I've reviewed the patch. I don't really like the idea of using SOGoCache to detect if we must process soft-deletes or not.

Why not modify GetItemEstimate so that it gets the 'dateCache' and count properly the number of items that are about to be soft-deleted?

tfu

tfu

2014-05-22 16:21

reporter   ~0007080

I'm not sure whether I understand your suggestion. After counting the soft-deletes in GetItemEstimate how can following line in processSyncGetChanges be passed if nothing else has changed in the folder:
if ([theSyncKey isEqualToString: [theCollection davCollectionTag]]
return;

It would be cool if you could amend the patch with what you have in mind.

ludovic

ludovic

2014-05-22 19:49

administrator   ~0007083

That check should be moved down after we check for soft-deletes.

ludovic

ludovic

2014-05-27 15:03

administrator   ~0007133

Why -checkSoftDeletes doesn't return the item count to delete?

You know what to delete by looking at the cache.

tfu

tfu

2014-05-27 15:23

reporter  

0003-sd2.patch (4,028 bytes)   
From 96aab54fcd65bd7702ff526342b343a1d50abbd1 Mon Sep 17 00:00:00 2001
From: root <root@example.com>
Date: Tue, 27 May 2014 20:02:32 +0200
Subject: [PATCH 3/3] sd2

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |   44 ++++++++++++++++++++++++++--
 ActiveSync/SOGoActiveSyncDispatcher.m      |   21 +++++++++++++
 2 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 91f5d5d..787fe9a 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -480,7 +480,47 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   NSMutableString *s;
   
   BOOL more_available;
-  int i, max;
+  int i, max, sd_count;
+  s = [NSMutableString string];
+
+  NSMutableDictionary *folderMetadata, *dateCache, *syncCache;
+  NSMutableArray *sdUids;
+
+  if (theFolderType == ActiveSyncMailFolder && !([theSyncKey isEqualToString: @"-1"])) {
+      if (theFilterType) {
+         sdUids =[[NSMutableArray alloc] init];
+         sd_count=0;
+
+         folderMetadata = [self _folderMetadataForKey: [theCollection nameInContainer]];
+         dateCache = [folderMetadata objectForKey: @"DateCache"];
+         syncCache = [folderMetadata objectForKey: @"SyncCache"];
+
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: theFilterType ] == NSOrderedAscending ) {
+                [s appendString: @"<SoftDelete xmlns=\"AirSync:\">"];
+                [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", key];
+                [s appendString: @"</SoftDelete>"];
+NSLog(@"tfu softdelete %@ %@",key,syncCache);
+NSLog(@"tfu softdelete %@ %@",key,dateCache);
+                [syncCache removeObjectForKey: key];
+                [dateCache removeObjectForKey: key];
+
+                sd_count++;
+            }
+
+            if (sd_count >= theWindowSize) {
+               [folderMetadata setObject: [NSNumber numberWithBool: YES]  forKey: @"MoreAvailable"];
+               *theLastServerKey = theSyncKey;
+
+            [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]]; 
+
+            return;
+            }
+         }
+
+         [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]]; 
+      }
+  }
 
   //
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
@@ -489,8 +529,6 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   if ([theSyncKey isEqualToString: [theCollection davCollectionTag]])
     return;
   
-  s = [NSMutableString string];
-  
   more_available = NO;
 
   switch (theFolderType)
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index 48d5c53..fd99bab 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -845,6 +845,27 @@ break;
                                               sortOrdering: @"REVERSE ARRIVAL"
                                                   threaded: NO];
       count = [uids count];
+
+      // count number of uids due to softdelete
+      NSMutableDictionary *dateCache;
+      SOGoCacheGCSObject *o;
+      NSMutableArray *sdUids;
+
+      sdUids =[[NSMutableArray alloc] init];
+
+      if (filter) {
+         o = [SOGoCacheGCSObject objectWithName: [NSString stringWithFormat: @"%@+folder%@", [context objectForKey: @"DeviceId"], realCollectionId] inContainer: nil];
+         [o setObjectType: ActiveSyncGlobalCacheObject];
+         [o setTableUrl: [self folderTableURL]];
+         [o reloadIfNeeded];
+         dateCache = [[o properties] objectForKey: @"DateCache"];
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: filter ] == NSOrderedAscending )
+               [sdUids addObject:[dateCache objectForKey:key]];
+         }
+
+         count+= [sdUids count];
+      }
     }
   else
     {
-- 
1.7.9.5

0003-sd2.patch (4,028 bytes)   
tfu

tfu

2014-05-27 15:26

reporter   ~0007134

Due to other priorities a cannot work on it at the moment, but I just uploaded what I have at the moment: 0003-sd2.patch.

ludovic

ludovic

2014-05-27 16:02

administrator   ~0007135

Good start - I'll bump this to v2.2.5 as we want to get 2.2.4 out ASAP.

It won't be a problem to add this feature later because the cache will still be populated correctly.

tfu

tfu

2014-06-24 13:47

reporter  

0001-softDelete.patch (5,845 bytes)   
From 82cf481afec0dc1503d6d732be63f96173c7dbf6 Mon Sep 17 00:00:00 2001
From: root <root@poldi.hopto.org>
Date: Fri, 20 Jun 2014 15:56:05 +0200
Subject: [PATCH 1/2] softDelete

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |   58 ++++++++++++++++++++++++----
 ActiveSync/SOGoActiveSyncDispatcher.m      |   30 ++++++++++++++
 2 files changed, 81 insertions(+), 7 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 8c206ba..5259da6 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -477,21 +477,64 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                  lastServerKey: (NSString **) theLastServerKey
 
 {
-  NSMutableDictionary *folderMetadata;
+  //NSMutableDictionary *folderMetadata;
   NSMutableString *s;
   
   BOOL more_available;
-  int i, max;
+  int i, max, sd_count;
+  s = [NSMutableString string];
+
+  NSMutableDictionary *folderMetadata, *dateCache, *syncCache;
+
+  more_available = NO;
+
+  if (theFolderType == ActiveSyncMailFolder && !([theSyncKey isEqualToString: @"-1"])) {
+      if (theFilterType) {
+         sd_count=0;
+
+         folderMetadata = [self _folderMetadataForKey: [theCollection nameInContainer]];
+         dateCache = [folderMetadata objectForKey: @"DateCache"];
+         syncCache = [folderMetadata objectForKey: @"SyncCache"];
+
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: theFilterType ] == NSOrderedAscending ) {
+                [s appendString: @"<SoftDelete xmlns=\"AirSync:\">"];
+                [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", key];
+                [s appendString: @"</SoftDelete>"];
+
+                [syncCache removeObjectForKey: key];
+                [dateCache removeObjectForKey: key];
+
+                sd_count++;
+            }
+
+            if (sd_count >= theWindowSize) {
+               [folderMetadata setObject: [NSNumber numberWithBool: YES]  forKey: @"MoreAvailable"];
+               more_available = YES;
+               *theLastServerKey = theSyncKey;
+
+               [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]];
+
+               goto return_response;
+            }
+         }
+
+         if (more_available)
+           [folderMetadata setObject: [NSNumber numberWithBool: YES]  forKey: @"MoreAvailable"];
+         else
+           [folderMetadata removeObjectForKey: @"MoreAvailable"];
+
+         [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]];
+      }
+  }
 
   //
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
   // We check this and we don't generate any commands if we don't have to.
   //
-  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]])
+  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]] && !([s length]))
     return;
   
-  s = [NSMutableString string];
-  
   more_available = NO;
 
   switch (theFolderType)
@@ -777,7 +820,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                     [s appendString: @"</Add>"];
                     
                     [syncCache setObject: [aCacheObject sequence]  forKey: [aCacheObject uid]];
-                    [dateCache setObject: [NSCalendarDate date]  forKey: [aCacheObject uid]];
+                    [dateCache setObject: [mailObject date]  forKey: [aCacheObject uid]];
                     return_count++;
                   }
                 else
@@ -805,6 +848,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
       break;
     } // switch (folderType) ...
   
+return_response:
+
   if ([s length])
     {
       [theBuffer appendString: @"<Commands>"];
@@ -1008,7 +1053,6 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
         davCollectionTag = [collection davCollectionTag];
     }
 
-
   // Generate the response buffer
   [theBuffer appendString: @"<Collection>"];
   
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index b98ce3b..35ae9d0 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -919,6 +919,9 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                                               sortOrdering: @"REVERSE ARRIVAL"
                                                   threaded: NO];
       count = [uids count];
+      // add number of uids due to softDelete
+      count+= [[self softDeleteUids: filter collectionId: realCollectionId] count];
+
     }
   else
     {
@@ -2216,4 +2219,31 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   [cm releaseChannel: channel]; 
 }
 
+- (NSArray *) softDeleteUids: (NSCalendarDate *) theFilter
+                collectionId: (NSString *) theCollectionId
+{
+
+      // count number of uids due to softdelete
+      NSMutableDictionary *dateCache;
+      SOGoCacheGCSObject *o;
+      NSMutableArray *sdUids;
+
+      sdUids =[[NSMutableArray alloc] init];
+
+      if (theFilter) {
+         o = [SOGoCacheGCSObject objectWithName: [NSString stringWithFormat: @"%@+folder%@", [context objectForKey: @"DeviceId"], theCollectionId] inContainer: nil];
+         [o setObjectType: ActiveSyncGlobalCacheObject];
+         [o setTableUrl: [self folderTableURL]];
+         [o reloadIfNeeded];
+         dateCache = [[o properties] objectForKey: @"DateCache"];
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: theFilter ] == NSOrderedAscending ) {
+               [sdUids addObject:[dateCache objectForKey:key]];
+             }
+         }
+      }
+
+       return sdUids;
+
+}
 @end
-- 
1.7.10.4

0001-softDelete.patch (5,845 bytes)   
tfu

tfu

2014-06-24 13:52

reporter   ~0007234

Can you please review attach patch (0001-softDelete.patch) and let me know what you thing about.

ludovic

ludovic

2014-06-24 14:44

administrator   ~0007235

Last edited: 2014-06-24 14:44

It seems to me that your call to "goto return_response;" should just be a break, otherwise you don't set MoreAvailable.

ludovic

ludovic

2014-06-24 15:00

administrator   ~0007236

You do set it, but:

     if (more_available)
       [folderMetadata setObject: [NSNumber numberWithBool: YES]  forKey: @&quot;MoreAvailable&quot;];
     else
       [folderMetadata removeObjectForKey: @&quot;MoreAvailable&quot;];

should probably just be:

       [folderMetadata removeObjectForKey: @&quot;MoreAvailable&quot;];
tfu

tfu

2014-06-24 15:28

reporter  

0001-softDelete1.patch (6,251 bytes)   
From 85ad95ae7c60984d7dce9a17085a59c8cbef2349 Mon Sep 17 00:00:00 2001
From: root <root@poldi.hopto.org>
Date: Tue, 24 Jun 2014 21:05:53 +0200
Subject: [PATCH] softDelete1

---
 ActiveSync/SOGoActiveSyncDispatcher+Sync.m |   56 ++++++++++++++++++++++++----
 ActiveSync/SOGoActiveSyncDispatcher.m      |   30 +++++++++++++++
 2 files changed, 79 insertions(+), 7 deletions(-)

diff --git a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
index 8c206ba..347e6a7 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher+Sync.m
@@ -126,6 +126,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   [[o properties] removeObjectForKey: @"SyncKey"];
   [[o properties] removeObjectForKey: @"SyncCache"];
   [[o properties] removeObjectForKey: @"DateCache"];
+  [[o properties] removeObjectForKey: @"MoreAvailable"];
 
   [[o properties] addEntriesFromDictionary: theFolderMetadata];
   [o save];
@@ -477,21 +478,61 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                  lastServerKey: (NSString **) theLastServerKey
 
 {
-  NSMutableDictionary *folderMetadata;
+  //NSMutableDictionary *folderMetadata;
   NSMutableString *s;
   
   BOOL more_available;
-  int i, max;
+  int i, max, sd_count;
+  s = [NSMutableString string];
+
+  NSMutableDictionary *folderMetadata, *dateCache, *syncCache;
+
+  more_available = NO;
+
+  if (theFolderType == ActiveSyncMailFolder && !([theSyncKey isEqualToString: @"-1"])) {
+      if (theFilterType) {
+         sd_count=0;
+
+         folderMetadata = [self _folderMetadataForKey: [theCollection nameInContainer]];
+         dateCache = [folderMetadata objectForKey: @"DateCache"];
+         syncCache = [folderMetadata objectForKey: @"SyncCache"];
+
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: theFilterType ] == NSOrderedAscending ) {
+                [s appendString: @"<SoftDelete xmlns=\"AirSync:\">"];
+                [s appendFormat: @"<ServerId xmlns=\"AirSync:\">%@</ServerId>", key];
+                [s appendString: @"</SoftDelete>"];
+
+                [syncCache removeObjectForKey: key];
+                [dateCache removeObjectForKey: key];
+
+                sd_count++;
+            }
+
+            if (sd_count >= theWindowSize) {
+               // set MoreAvailable in cache
+               [folderMetadata setObject: [NSNumber numberWithBool: YES]  forKey: @"MoreAvailable"];
+               [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]];
+
+               more_available = YES;
+               *theLastServerKey = theSyncKey;
+               // since WindowSize is reached don't even try to add more to the response - jump to the end and return the response
+               goto return_response;
+            }
+         }
+
+         [folderMetadata removeObjectForKey: @"MoreAvailable"];
+         [self _setFolderMetadata: folderMetadata forKey: [theCollection nameInContainer]];
+      }
+  }
 
   //
   // No changes in the collection - 2.2.2.19.1.1 Empty Sync Request.
   // We check this and we don't generate any commands if we don't have to.
   //
-  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]])
+  if ([theSyncKey isEqualToString: [theCollection davCollectionTag]] && !([s length]))
     return;
   
-  s = [NSMutableString string];
-  
   more_available = NO;
 
   switch (theFolderType)
@@ -777,7 +818,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                     [s appendString: @"</Add>"];
                     
                     [syncCache setObject: [aCacheObject sequence]  forKey: [aCacheObject uid]];
-                    [dateCache setObject: [NSCalendarDate date]  forKey: [aCacheObject uid]];
+                    [dateCache setObject: [mailObject date]  forKey: [aCacheObject uid]];
                     return_count++;
                   }
                 else
@@ -805,6 +846,8 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
       break;
     } // switch (folderType) ...
   
+return_response:
+
   if ([s length])
     {
       [theBuffer appendString: @"<Commands>"];
@@ -1008,7 +1051,6 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
         davCollectionTag = [collection davCollectionTag];
     }
 
-
   // Generate the response buffer
   [theBuffer appendString: @"<Collection>"];
   
diff --git a/ActiveSync/SOGoActiveSyncDispatcher.m b/ActiveSync/SOGoActiveSyncDispatcher.m
index b98ce3b..35ae9d0 100644
--- a/ActiveSync/SOGoActiveSyncDispatcher.m
+++ b/ActiveSync/SOGoActiveSyncDispatcher.m
@@ -919,6 +919,9 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
                                               sortOrdering: @"REVERSE ARRIVAL"
                                                   threaded: NO];
       count = [uids count];
+      // add number of uids due to softDelete
+      count+= [[self softDeleteUids: filter collectionId: realCollectionId] count];
+
     }
   else
     {
@@ -2216,4 +2219,31 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   [cm releaseChannel: channel]; 
 }
 
+- (NSArray *) softDeleteUids: (NSCalendarDate *) theFilter
+                collectionId: (NSString *) theCollectionId
+{
+
+      // count number of uids due to softdelete
+      NSMutableDictionary *dateCache;
+      SOGoCacheGCSObject *o;
+      NSMutableArray *sdUids;
+
+      sdUids =[[NSMutableArray alloc] init];
+
+      if (theFilter) {
+         o = [SOGoCacheGCSObject objectWithName: [NSString stringWithFormat: @"%@+folder%@", [context objectForKey: @"DeviceId"], theCollectionId] inContainer: nil];
+         [o setObjectType: ActiveSyncGlobalCacheObject];
+         [o setTableUrl: [self folderTableURL]];
+         [o reloadIfNeeded];
+         dateCache = [[o properties] objectForKey: @"DateCache"];
+         for (NSString* key in [dateCache allKeys]) {
+             if ([[dateCache objectForKey:key] compare: theFilter ] == NSOrderedAscending ) {
+               [sdUids addObject:[dateCache objectForKey:key]];
+             }
+         }
+      }
+
+       return sdUids;
+
+}
 @end
-- 
1.7.10.4

0001-softDelete1.patch (6,251 bytes)   
tfu

tfu

2014-06-24 15:30

reporter   ~0007237

Yes - I slightly modified the patch and added some comments.
_setFolderMetadata: -> added [[o properties] removeObjectForKey: @"MoreAvailable"];

ludovic

ludovic

2014-06-25 12:40

administrator   ~0007238

Are you sure about that;

  • [dateCache setObject: [NSCalendarDate date] forKey: [aCacheObject uid]];
  • [dateCache setObject: [mailObject date] forKey: [aCacheObject uid]];

What I don't like is that a spammer could set a mail in the future, say 2020 and the mail would 'never' softdelete itself.

tfu

tfu

2014-06-25 14:40

reporter   ~0007239

You are right if you want to avoid such a scenario and as long the device is synced on regular basis it would make no difference for the user whether a mail is soft-deleted based on [NSCalendarDate date] (synced) or [mailObject date] (received).

ludovic

ludovic

2014-06-25 15:07

administrator   ~0007240

Modified patch pushed: https://github.com/inverse-inc/sogo/commit/5419f411e5bae58cc17ab22b3b95a88efab74252

Thanks for your contribution!

Issue History

Date Modified Username Field Change
2014-04-24 15:29 tfu New Issue
2014-04-24 15:29 tfu File Added: 0001-mail-softdelete.patch
2014-04-29 10:02 tfu File Added: 0002-mail-softdelete-hasPrefix.patch
2014-04-29 10:07 tfu Note Added: 0006973
2014-05-05 14:04 ludovic Note Added: 0006990
2014-05-05 14:24 ludovic Target Version => 2.2.4
2014-05-15 15:12 ludovic Note Added: 0007027
2014-05-18 09:50 tfu File Added: 0001-softDelete-with-dateCache.patch
2014-05-18 09:50 tfu Note Added: 0007048
2014-05-22 09:32 ludovic Note Added: 0007066
2014-05-22 16:21 tfu Note Added: 0007080
2014-05-22 19:49 ludovic Note Added: 0007083
2014-05-27 15:03 ludovic Note Added: 0007133
2014-05-27 15:23 tfu File Added: 0003-sd2.patch
2014-05-27 15:26 tfu Note Added: 0007134
2014-05-27 16:02 ludovic Note Added: 0007135
2014-05-27 16:02 ludovic Target Version 2.2.4 => 2.2.5
2014-06-24 13:47 tfu File Added: 0001-softDelete.patch
2014-06-24 13:52 tfu Note Added: 0007234
2014-06-24 14:44 ludovic Note Added: 0007235
2014-06-24 14:44 ludovic Note Edited: 0007235
2014-06-24 15:00 ludovic Note Added: 0007236
2014-06-24 15:28 tfu File Added: 0001-softDelete1.patch
2014-06-24 15:30 tfu Note Added: 0007237
2014-06-25 12:40 ludovic Note Added: 0007238
2014-06-25 14:40 tfu Note Added: 0007239
2014-06-25 15:06 ludovic Target Version 2.2.5 => 2.2.6
2014-06-25 15:07 ludovic Note Added: 0007240
2014-06-25 15:07 ludovic Status new => resolved
2014-06-25 15:07 ludovic Fixed in Version => 2.2.6
2014-06-25 15:07 ludovic Resolution open => fixed
2014-06-25 15:07 ludovic Assigned To => ludovic