Notes Database cannot be found in Backup

[felix-stvmz]

Hello there,
I'm currently encountering an issue where dominobackup can't copy the databasefile from the mounted veeam backup to the configured path on the server. And I don't understand why this is the case. The field "Source" in the relevant part of the logsection is empty but there is no obvious way to fill it in the restore job created in the Domino Backup application. Or is it normal that this field is empty?

load restore -v -V
24.09.2024 09:55:13 Remote console command issued by Felix ZZ-Hart/KDZ/Mainz: load restore -v -V
Restore-Log: CONFIG [RestoreDatabase] -> [dominobackup.nsf]
Restore-Log: Opening [dominobackup.nsf]
Restore-Log: Lookup Config: [SERVER:CN=d2-l-ibmweb-01/OU=Server/O=Mainz] --> 0x902
Restore-Log: Config: [Server] : [0x902]
Restore-Log: Found Server Config Note: [0x902] for key [CN=d2-l-ibmweb-01/OU=Server/O=Mainz]
Restore-Log: CONFIG [NodeName] -> [d2-l-ibmweb-01]
Restore-Log: CONFIG [ScriptDir] -> [/opt/hcl/domino/backup/veeam ]
Restore-Log: CONFIG [BackupLogDir] -> [/local/Dominobackup/Log]
Restore-Log: CONFIG [LogCommandOutputOptions] -> [E]
Restore-Log: Command: [RestorePreCommand] Type: 0, IsFormula: 0 -> []
Restore-Log: Command: [RestorePostCommand] Type: 2, IsFormula: 0 -> [/opt/hcl/domino/backup/veeam/restore_post.sh]
Restore-Log: Command: [RestoreDbCommand] Type: 2, IsFormula: 0 -> [/opt/hcl/domino/backup/veeam/restore_db.sh]
Restore-Log: Command: [RestoreFileCommand] Type: 0, IsFormula: 0 -> []
Restore-Log: Command: [RestoreSnapshotCommand] Type: 2, IsFormula: 0 -> [/opt/hcl/domino/backup/veeam/restore_db.sh]
Restore-Log: Command: [RestoreTranslogCommand] Type: 0, IsFormula: 0 -> []
Restore-Log: Command: [RestoreDaosCommand] Type: 0, IsFormula: 0 -> []
Restore-Log: CONFIG [BackupTargetDirDb] -> [/local/Dominobackup ]
Restore-Log: CONFIG [BackupTargetDirTranslog] -> [/local/Dominobackup/Translog ]
Restore-Log: DEFAULT [BackupTargetDirDaos] -> []
Restore-Log: DEFAULT [RestoreOkString] -> []
Restore-Log: DEFAULT [RestoreErrString] -> []
Restore-Log: DEFAULT [ThirdPartyRestoreDateFormula] -> []
Restore-Log: Running on: 1 restore documents
Restore-Log: Running on Note [0x12EA] [dominobackup.nsf]
Restore-Log: Running on Restore Note [0x12EA]
Restore-Debug: IF:FileDelete [/local/notesdata/restored/dominoblog.ntf.DELTA]: Err: 0x103
Restore-Debug: IF:FileCreate[/local/notesdata/IBM_TECHNICAL_SUPPORT/restore_error_d2-l-ibmweb-01_2024_09_24@09_55_13.log]: Err: 0x0
Restore-Debug: Running [RESTORE] CMD: [/opt/hcl/domino/backup/veeam/restore_db.sh "/local/notesdata/dominoblog.ntf" "dominoblog.ntf" "F327F936917596ABC1258AB60055F5C3" "d2-l-ibmweb-01" "" "SNAP" "20240924073100" "/local/Dominobackup " "/local/notesdata/
Restore-Debug: CMD-RET: [[2024-09-24 09:55:13] --- RESTORE DB ---
Restore-Debug: CMD-RET: [PhysicalFileName : /local/notesdata/dominoblog.ntf
Restore-Debug: CMD-RET: [FileName : dominoblog.ntf
Restore-Debug: CMD-RET: [BackupReference : F327F936917596ABC1258AB60055F5C3
Restore-Debug: CMD-RET: [BackupNode : d2-l-ibmweb-01
Restore-Debug: CMD-RET: [BackupName :
Restore-Debug: CMD-RET: [BackupMode : SNAP
Restore-Debug: CMD-RET: [BackupStartDT : 20240924073100
Restore-Debug: CMD-RET: [BackupTargetDir : /local/Dominobackup
Restore-Debug: CMD-RET: [RestoreFileName : /local/notesdata/restored/dominoblog.ntf
Restore-Debug: CMD-RET: [FileExtension : ntf
Restore-Debug: CMD-RET: [Appending missing .DAD extension
Restore-Debug: CMD-RET: [Mouting snaphot
24.09.2024 09:55:13 Restore: Translog Mode: CIRCULAR
24.09.2024 09:55:13 Restore: Domino Database Restore
24.09.2024 09:55:13 Restore: Started
24.09.2024 09:55:13 Restore: Restoring to [restored/dominoblog.ntf] [/local/notesdata/restored/dominoblog.ntf]
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [Domino Backup for Veeam Restore
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [Command : [mount]
Restore-Debug: CMD-RET: [Date : [20240924073100]
Restore-Debug: CMD-RET: [Connect : [172.24.65.105]
Restore-Debug: CMD-RET: [VM Host : [d2-l-ibmweb-01]
Restore-Debug: CMD-RET: [RestoreOS : [Linux]
Restore-Debug: CMD-RET: [Account : [veeam-mount]
Restore-Debug: CMD-RET: [MountFUSE : [True]
Restore-Debug: CMD-RET: [ClockSkewMin : [5]
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [Backup Time String : 20240924073100
Restore-Debug: CMD-RET: [Backup Time (local time) : 09/24/2024 09:31:00
Restore-Debug: CMD-RET: [Backup Time (UTC) : 09/24/2024 07:31:00
Restore-Debug: CMD-RET: [Backup Search Time UTC : 09/24/2024 07:26:00
Restore-Debug: CMD-RET: [RestoreVmHost : d2-l-ibmweb-01
Restore-Debug: CMD-RET: [Credentials used: : veeam-mount
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [Backup search: 1,1 seconds )
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [24.09.2024 09:56:17 Mounting backup restore point: 35e133a2-ef65-4fe8-b298-827aa5f14871 Backup Time Local: 24.09.2024 09:31:57 Backup Time UTC: 24.09.2024 07:31:57
24.09.2024 09:55:21 5 Transactions/Minute, 5 Notes Users
Restore-Debug: CMD-RET: [24.09.2024 09:56:30 Mount operation finished ( 13,0 seconds )
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [Backup Job Name : d2-l-ibmweb-01
Restore-Debug: CMD-RET: [Restore Point : 2024-09-24T09:31:57.0000000+02:00
Restore-Debug: CMD-RET: [VM Name : d2-l-ibmweb-01
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [Mounted Disk : /tmp/Veeam.Mount.Disks.3f46364a-b44c-4956-8ec1-61537e46c8f4/{30c4ad9d-3a5a-4209-8641-81cf2c12816f}_0 /tmp/Veeam.Mount.Disks.3f46364a-b44c-4956-8ec1-61537e46c8f4/{622b7dcf-c97c-427e-998f-f5377ece7934}_0 /tmp/Veea
Restore-Debug: CMD-RET: [Mounted At : /tmp/Veeam.Mount.FS.3f46364a-b44c-4956-8ec1-61537e46c8f4/8cd21829-e170-4899-bc10-c5ecdb62677e_0_btrfs/boot/efi /tmp/Veeam.Mount.FS.3f46364a-b44c-4956-8ec1-61537e46c8f4/8cd21829-e170-4899-bc10-c5ecdb62677e_0_btrf
Restore-Debug: CMD-RET: [Mounted As :
Restore-Debug: CMD-RET: [Available From : 172.24.65.105 (Port: )
Restore-Debug: CMD-RET: [Available Via :
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [OK: Mount operations successfully completed
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [0
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [DOMBACK_TAG_FILE : [dominobackup_20240924073100.tag]
Restore-Debug: CMD-RET: [FOUND_FILE_PATH : []
Restore-Debug: CMD-RET: [FOUND_NOTESDATA : [/tmp/Veeam.Mount.FS.8ec46805-7ac7-4def-b2fe-0cafa8c8dcc9/{12ccab92-9cc3-441d-b63b-02db62014c95}_0/notesdata]
Restore-Debug: CMD-RET: [SOURCE : []
Restore-Debug: CMD-RET: [TARGET : [/local/notesdata/restored/dominoblog.ntf.DAD]
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [[2024-09-24 09:55:31] Restore operation completed
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [Return: ERROR - Cannot copy file
Restore-Debug: Finished [RESTORE]
Restore-Debug: Running [RESTORE] CMD: [/opt/hcl/domino/backup/veeam/restore_post.sh "" "" "" "d2-l-ibmweb-01" "" "" "20240924073100" "" "" ""]
Restore-Debug: CMD-RET: [Unmounting snaphots
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [Domino Backup for Veeam Restore
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [Command : [unmount]
Restore-Debug: CMD-RET: [Date : []
Restore-Debug: CMD-RET: [Connect : [172.24.65.105]
Restore-Debug: CMD-RET: [VM Host : [d2-l-ibmweb-01]
Restore-Debug: CMD-RET: [RestoreOS : [Linux]
Restore-Debug: CMD-RET: [Account : [veeam-mount]
Restore-Debug: CMD-RET: [MountFUSE : [True]
Restore-Debug: CMD-RET: [ClockSkewMin : [5]
Restore-Debug: CMD-RET: [-------------------------------------------------------------------------------------
Restore-Debug: CMD-RET: [
Restore-Debug: CMD-RET: [24.09.2024 09:56:33 Unmounting 1 backup(s) ...
Restore-Debug: CMD-RET: [24.09.2024 09:56:42 Unmount operation fishined ( 9,2 seconds )
Restore-Debug: CMD-RET: [OK: Backup(s) unmounted
Restore-Debug: CMD-RET: [0
Restore-Debug: CMD-RET: [Return: PROCESSED ()
Restore-Debug: Finished [RESTORE]
Restore-Debug: IF:FileOpen [/local/notesdata/restored/dominoblog.ntf.DELTA]: Err: 0x103
Restore-Log: No Delta file present - Skipping [/local/notesdata/restored/dominoblog.ntf.DAD]
Restore-Log: ChangeDatabase: [/local/notesdata/restored/dominoblog.ntf.DAD] [dominoblog.ntf]
Restore-Debug: DomBackupLogFile::Close: Bytes written: 0
Restore-Debug: IF:FileExistCheck [/local/notesdata/IBM_TECHNICAL_SUPPORT/restore_error_d2-l-ibmweb-01_2024_09_24@09_55_13.log]: 0x0 IsDir: 0
Restore-Debug: IF:FileDelete [/local/notesdata/IBM_TECHNICAL_SUPPORT/restore_error_d2-l-ibmweb-01_2024_09_24@09_55_13.log]: Err: 0x0
Restore-Debug: Attaching Log File [] Bytes: 0
Restore-Log: Running on: 0 restore documents
24.09.2024 09:55:44 Restore: Restoring Database [/local/notesdata/restored/dominoblog.ntf.DAD]
24.09.2024 09:55:44 Restore: Cannot start empty apply log info for database [/local/notesdata/restored/dominoblog.ntf.DAD] : File does not exist
24.09.2024 09:55:44 Restore: Error opening Database [/local/notesdata/restored/dominoblog.ntf.DAD] : File does not exist
24.09.2024 09:55:44 Restore: Error restoring database [dominoblog.ntf] -> [restored/dominoblog.ntf] : File does not exist
24.09.2024 09:55:44 Restore: Finished

Thank you for your help!
Regards, Felix

[Daniel-Nashed]

it's difficult to say what is happening here.
can you check first that the right snapshot is mounted.
then look into the backup by mounting it to see if the delta files are present.

[felix-stvmz]

The right snapshot is mounted but there are no delta files despite the configuration that there should be a directory for delta files.

[Daniel-Nashed]

Oh wait. This is Linux which stores the Delta files locally and the files need to be backed up to a different place.
This can be any type of file storage or local on the same server backed up with another Veeam job.
The backup and restore script are designed to match. And the location for backup and restore is the same.

It's mentioned here but could be more obvious documented.

The snapshot data is on Veeam. But the delta files must be stored in a different way. The easiest way is local on a different disk or same disk backed up with a different job. The files are small and can remain on disk. they are managed by Domino Backup prune operations.

[felix-stvmz]

Thank you for your quick answer. I will take a closer look into that in a week after my vacation.

[felix-stvmz]

Do I have to enter something in the following fields in the restore_db.sh script, e.g. a variable for the mounted Veeam backup or something similar?
# Search mounted directories for backup tag file to verify the right mount and get the Domino data directory FOUND_FILE_PATH= FOUND_NOTESDATA=

[Daniel-Nashed]

no you don't need to specify anything separately. there is no need to change anything in the file.
the lines you are showing are the init for the variables.

if you look further down, you see how it is filled.

FOUND_FILE_PATH=$(find $VEEAM_RESTORE_BASE_PATH -name "$DOMBACK_TAG_FILE" | head -1 2>/dev/null)

[felix-stvmz]

I thought so, but in this case I don't see whats causing the "Return: ERROR - Cannot copy file" error in the restore process. Everything shows as successful in the restorelog except the file copy. And there is no reason for that in the log.

[Daniel-Nashed]

you can add more trace statements into the bash file. they will be logged end to end into your restore log.
the mount works as the log shows. you would need to look into the backup to see if the file is there and you can also dump the directory listing from the mounted snapshot.