File and Folder Backup retention policies
At the last step of the File and Folder Backup wizard, there are new retention policies added:
These policies provide data deletion from the cloud of files that are either old (not changed on the local machine for X days) or don't exist on the machine. The policy doesn't touch the data on the local machine. Peculiarities of the policies:
- can be switched on separately or combined;
- reflect "natural" retention policies for old and unnecessary (deleted) files and can be used when a customer wants to save only recent versions of files in the cloud;
- if a file has versions, the policies remove "all or nothing";
- the policies can be switched only when option "send email reports" is chosen. This is done for safety so customers always know that policies are switched on and what amount of data is deleted from the cloud.
How it works
1. If you check 'Not changed on the local computer for X days'.
After file uploading is completed, the next step of OBRM/SOS backup session is scanning all the metadata on backed up files in 'Protect_.db'. If ‘modifyDate' field for some file corresponds to the condition 'not changed for X days', it fills the field ‘notificationUnchangedSentOn' with the date when this condition found. The field ‘notificationUnchangedSentOn' means, that at the end of the backup job, the name of this file will be included into the special email report (but not more than 50 names totally), which will be sent to the email address, mentioned in Backup Reports section. This report alerts the customer, that these files (and all their versions) will be deleted from the cloud in 3 days (or later, if the backup job is scheduled for a later time) during the next scheduled backup job. During the next backup job, the client checks whether these 3 days are already passed, and if it is true – deletes the file (and all its versions) from the cloud. If during these 3 days (or more, if the backup job is scheduled for a later time) the file is modified, the field ‘notificationUnchangedSentOn' is cleared.
2. If you check 'Deleted from the local computer, after waiting X days'.
After file uploading is completed, the next step of OBRM/SOS backup session is scanning all the metadata on backed up files in 'Protect_.db'. If the application finds that some files are deleted from the local computer, it fills the field ‘notificationDeletedSentOn' with the date when this deletion found (not with the date when a file was in fact deleted). The field ‘notificationDeletedSentOn' means, that at the end of the backup job, the name of this file will be included in the special email report (but not more than 50 names totally), which will be send to the email address, mentioned in Backup reports section. This report alerts the customer that these files (and all their versions) will be deleted from the cloud in 3 days (or later, if the backup job is scheduled for later time) during the next scheduled backup job. During the next backup job, the client checks whether 3 days are already passed, and if this is true – deletes the file (and all its versions) from the cloud. If during these 3 days (or more, if the backup job scheduled for later time) the file is added to the local computer, the field ‘notificationDeletedSentOn' is cleared.
If the retention policy is modified, all the fields ‘notificationDeletedSentOn' and ‘notificationUnchangedSentOn' for all the files cleared, and no one file is deleted until the updated retention policy conditions are met.
The retention policy will not work for files which (for some reason) have no records in the ‘Protect_.db’. For example, the files that had been uploaded to the cloud and then deleted from the local computer. Then OBRM/SOS application was uninstalled (‘Protect_.db’ file was deleted) and the application was installed again. So, the application has no information about uploaded data to the cloud, which was deleted from the local computer.
Retention policy second option "delete files from the cloud if they are deleted from the local computer, after waiting X days (5 days for this example)
If you delete a file from the local computer which is selected in the backup set, SOS will wait for 5 days and then delete the same from the cloud.
What one would like to know if whether files deleted locally PRIOR to the tickbox being selected (ie prior to this release of software) will be deleted from online storage once 5 days has elapsed. IE:-
Monday 18th July - delete file #1
Wednesday 20th July - Upgrade software to latest version and select the tickbox to delete online files 5 days after local deletion
Thursday 21st July - delete file #2
It is clear that on Tuesday 26th July file #2 will be removed from online storage, as that is 5 days after a local deletion that took place after the latest release of software had been installed and the tickbox ticked.
The question is whether on Saturday 23rd July file #1 will be deleted from online storage, as that deletion occurred prior to the latest release being installed and the tickbox being ticked? Is the software able to detect files delete prior to this release and tickbox?
- File #1 will be deleted on July 25th (Thursday) because the first backup with the retention policy switched on will check whether files are presented on the disk. For the deleted files, it will record "Deleted date" as the current backup date (when OBRM/SOS found out that files are not presented). According to the example, it will mark it as 07/20 and will delete them in 5 days.
At the end of every backup session, OBRM/SOS checks whether files presented in the backup set have been deleted from the local machine.
Please, note that the policy is not started by itself, only at the end of the backup session.
Consider that the backups run daily from the machine, so the scenario is following:
Monday 18th July - delete file #1 - policy doesn't exist, no files are deleted or marked for deletion;
Wednesday 20th July - Upgrade software to latest version and select the tickbox to delete online files 5 days after local deletion:
- run backup;
- OBRM checks whether files from the backup set are presented on the disk;
- for files deleted before 20th July (i.e. #1 file deleted on 18th of July) OBRM sets DeletedDate as 20th July; Email report is sent http://prntscr.com/c0kywu with the notification that files will be deleted in 5 days (07/25)
Thursday 21st July - delete file #2
- run backupon on 20th July;
- file #2 is marked for deletion on 26th of July;
25th of July;
- run backup;
- file #1 is deleted from the cloud (it was marked for deletion on 20th of July when the retention policy was switched on).
Advanced Folder Backup retention policies
Two new retention policies were added:
- Move Files to cloud X days to delay delete;
- "Temporary" with several options
The policy <<Move Files to cloud X days to delay delete>> corresponds to the older "Time-Limited Backup". It deletes files at the chosen folder from the local machine if they were backed up more than X days ago (the last backup date is earlier than X days). If a file is modified within X days prior to the current backup session, it is backed up at the current backup session and the timer resets. Hence, if it is not be backed up within X days between the current backup session and the next backup session, it will be deleted from the local machine.
<<Temporary>> policy deletes files from the cloud (that are included to the chosen folder) that were not modified or not backed up within X days. A customer can choose between modification date and backup date:
Frequently asked question is: wouldn't the most recent backup date of a file be the same date as the most recent back up, in general?
- In general - not. If a file is not changed, it is not backed up so "backup date" remains older that the recent backup.
How does the file backup date differs from the file modification date?
- Modification date is a file attribute recorded in the filesystem. It is changed when a file is modified (obvious). For example, the scheduled backup runs at 7:00 PM every day and I modify a document the last time at 5:00 PM every day, then, after the backup, the cloud will store the information:
- the file modification date - 5:00 PM [date of modification];
- backup date - 7:00 PM [date of backup].
If a file is backed up once a week (i.e. on Sundays), then, obviously, backup date will differ from modified date.
When is modification date of a file more recent than the backup date??
- Modification date is changed if a file is modified (obviously), however, the contents of the file may not be changed.
If OBRM sees that the modification date of a file, it differs from the one recorded at the previous backup session, OBRM then calculates a hash of a file. If the hash is the same as before, the contents of a file is actually not changed so it is not uploaded to the cloud. So, the backup date remains an old one but modification date appears to be more recent.
How many days is the limit on retention policies for backup software? Is it 365 Max?
- The limit of days for retention policies is 3660 days.
What does this do? Isn't the retention policy looking at each individual file, rather than folders?
Advanced folder backup operates on the whole folders added:
- no options to choose a single file or several files when configuring Advanced Folder Backup;
- no options to EXCLUDE the subfolders when a large parent folder is added (then, one needs to Add the needed subfolders to the wizard separately);
Hence, "Include sub-folders" was implemented as an option for the customers who want to choose whether the retention policy should check its subfolders or operate only on files that are placed directly under the folder.
NOTE: "Include sub-folders" option determines the choice where to remove, not what to backup.
This retention policy will backup ALL files and folders from E:\downloads but it will delete from the cloud the files which are modified earlier than 10 days ago and are stored under the folder E:\downloads directly. Any files from its subfolders will NOT be deleted from the cloud.
Delete File Versions
It deletes all file versions which modification date or backup date older than X days.
However, as OBRM/SOS has cache options, it stores baselines and deltas (differentials from the last full backup of the file).
The versions' deletion rule is following:
Where B = baseline, d = delta
| = the threshold (older than X days)
B1 B2 | B3 B4 => B3 B4
B1 d11 d12 | d13 B2 d21 d22 => B1 d13 B2 d21 d22
B1 d11 d12 | B2 d21 d22 => B2 d21 d22
B1 d11 | B2 => B2
B1 d11 d12 d13 | => remove all versions of the file
Brief explanations about Baselines and Deltas can be found in: Do all file versions count against my data limit?)
The Additional settings - skip files modified more than X days ago
If this option is enabled, the files in the added folder/subfolders which were modified more than X days ago, will NOT be added to the backup set and backed up.