I manage a TFS server. Last night we upgraded from TFS 2018.3 to Azure DevOps Server 2019.1.
As a part of our testing procedures, we test for backup failure (and passing) notifications. These notifications were tested also just prior to the upgrade and both notification messages were being sent as expected.
Before the upgrade, Scheduled Backups were disabled. Our backups folder was then segregated to a safe location (renamed). A new folder was created for the upgraded backups with the proper permission settings.
After the upgrade was complete, Scheduled Backups were configured (to the same settings as before). The configuration was successful with no errors or warnings. The test database appeared briefly in the folder. BackupSettings.xml appeared in the folder and looks correct.
A full backup was initiated. This was allowed to complete and looks correct. BackupSets.xml looks correct.
Our settings call for a "Transaction Log Backup Interval" of 15 minutes. This appears to be working correctly. All of the proper .trn files for all databases appear in the folder at the proper time. (Note that we received the familiar "blocked" Transactional_*.log" result during the full backup. This is normal and expected.)
We then turned on both of the "Provide Scheduled Backups Alert Settings". After the 15 minute interval, all admins received the "Azure DevOps Database Backup Succeeded" email alert.
At this point we blocked permissions to write to the "Network Backup Path". I monitored the "Job Schedule" window on the "Azure DevOps Server Administration Console" as well as the failed job log at: https:///tfs/_oi/_jobmonitoring?_a=history
After the 15 minute window elapsed, a failed job appeared with the history as follows:
Result Message:
[1/26/2020 12:55:42 AM] [Info] Requested Backup Job: Transactional [1/26/2020 12:55:42 AM] [Info] -------------------------------------------------------------------
[1/26/2020 12:55:42 AM] [Info] Jobs Requested: Transactional. Completed: None. To Run: Transactional.
[1/26/2020 12:55:42 AM] [Info] RunTransactionalBackupJob
[1/26/2020 12:55:42 AM] [Error] Exception Message: Access to the path '\\redacted\backup\production\BackupSets.xml.temp' is denied. (type UnauthorizedAccessException)
Exception Stack Trace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy) at System.IO.FileStream..ctor(String path, FileMode mode) at Microsoft.TeamFoundation.Admin.BackupSets.Save(String folder) at Microsoft.TeamFoundation.Admin.BackupSets.Initialize() at Microsoft.TeamFoundation.Admin.BackupSets.Load(String folder) at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunTransactionalBackupJob(IVssRequestContext requestContext, BackupSettings settings, ITFLogger logger) at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunBackupJob(IVssRequestContext requestContext, BackupJobType requestedJobType, String& resultMessage)
[1/26/2020 12:55:42 AM] [Info] Backup Job Finished. Completed: None. Result = Failed
The job schedule windows updated properly and noted that the backup had failed.
No failure notification email was ever received.
Settings were checked, and were correct. Settings were disabled and reconfigured, and the test repeated. Same results.
An investigation of notification settings was made, but there do not appear to be any settings that can prohibit backup failure notifications from being sent.
We searched for a failed, or partially successful job on the _oi page entitled "Notification E-Mail Delivery", there was no such failed job.
We tested the Notification E-Mail Delivery job to see if we could cause a trivial "Partially Succeeded" result. By placing a false email address on one of our users profiles we could do this. The job result appeared, and the history showed expected results. We reverted the false email setting, and verified there were no further Partially Succeeded jobs from Notifications E-Mail Delivery.
There is no indication, however, that in a backup failure situation that any notification attempt is ever made by Azure DevOps Server 2019.1.
Can anyone help to resolve this issue?
"we blocked permissions to write to the "Network Backup Path"
If you blocked the access to backup location. backup failure email alert will not be sent out.
As discussed in this thread, backup settings config file is unavailable when backup location is unavailable, which is the reason of backup failure email alert failing to be sent out. For testing the backup failure alert you can try other ways to purposely fail the backup