Error message

Backups fails with the following error message:

bsock.c:184 Unable to connect to Storage daemon on 192.168.1.15:9103. ERR=Connection refused

Also, you may see the following notices:

primary database is inaccessible
insufficient storage space on /raid
storage space critically low /raid
storage space critically low /root full
storage space critically low /var/log/full

Note

If you see storage space critically low /raid/:catalogs: or primary database is inaccessible, and you have a lot of space on the RAID usage bar, this is a different error message involving space on the database and will require extra help. Please contact Infrascale Support.

Steps to resolve

These messages indicate that the space on the CFA RAID is full. The CFA won’t overwrite the oldest jobs when the CFA runs out of space, rather the newest jobs will fail.

Below are the steps to clear the RAID space. Try them in the order they’re described.

Delete erred and canceled jobs

Erred jobs may take up the hidden space if there are 0 bytes showing as saved.

It’s always recommended to delete erred jobs unless you are troubleshooting an issue and need them for diagnostics.

To make the CFA automatically delete erred jobs:

  1. In the CFA Management Console, go to Settings.

  2. In the Backups group, click Automatic Job Management.

  3. Select Immediately Delete Erred Jobs, and then click Apply.

Note

You’ll still see the erred jobs for the first 24–48 hours in Jobs > Recent Jobs.

To delete already existing erred and canceled jobs:

  1. In the CFA Management Console, go to Jobs > History.

  2. Click Filters on the toolbar.

  3. Click Status, clear everything except Erred and Canceled .

  4. Select the jobs you want to delete in the list.

  5. Click Delete on the toolbar and confirm the deletion of the erred and canceled jobs.

Delete unreferenced data

After you delete a job from the CFA, certain files still remain in the system. These files are called the unreferenced data. They take up the disk space, and you can safely remove them if you’re running out of space. For this:

  1. In the CFA Management Console, go to Settings.

  2. In the Backups group, click Unreferenced Data.

  3. Click Scan For Unreferenced Volumes to find all unreferenced data.

  4. Select all paths in the list.

  5. Click Delete Selected Volumes and confirm the deletion of all unreferenced data.

Delete jobs manually

If you deleted all erred and canceled jobs, and the unreferenced data files, but you still need to free the disk space, you may also delete old or not critical backup jobs.

For this, follow the same steps as for deleting the erred and canceled jobs, but using other status filters.

Important

To avoid any possible data loss, make sure to keep at least the most recent full backup, and any later backups.

Garbage collection

The garbage collection helps you clean the leftover data on the CFA. This data may include, for example, the backup jobs left after you deleted a client, or the old backup jobs removed incompletely.

By default, the garbage collection runs automatically every day at a given time.

To set time to run the garbage collection, or to start the garbage collection manually, follow these steps:

  1. In the CFA Management Console, go to the Settings tab.

  2. In the Tools group, click Deduplicated File System.

    • To start the garbage collection manually, click Start next to Garbage Collection.

    • To set time for automatic garbage collection, click Edit next to Garbage Collection Time of Day.

Garbage collection can take a long time to run, especially if you deleted a lot of data.

Try to run the garbage collection each time after you delete a backup job to make sure the system clears the leftover data completely.

The garbage collection will be deferred for up to 12 hours before it gives up, and will be retried at its regular time. One exception is if the system is running low on space, at which point the garbage collection will proceed whether there are jobs deduping or not.

If you have the block level deduplication enabled, the system reserves some space for the block store. Block level deduplication has its own garbage collection process. To run this process:

  1. In the CFA Management Console, go to the Settings tab.

  2. In the Tools group, click Deduplicated File System.

  3. In the Block Deduplication group, click Reclaim Storage.

Recycle jobs automatically

For long term management of space on the RAID, it’s a good idea to set recycling schedules. This will allow the CFA to automatically delete jobs once they reach an expiration date.

Note

All customers using byte-level replication need to use a schedule and retention policy that ensures you have sufficient time for a second full backup to completely replicate over to the secondary before purging the oldest or the original full backup. It’s recommended to have a minimum of two full backups on the CFA at a given time, preferred is three.

Best practices for schedules and recycling will depend on many things, like the size of the backups and your company’s policies for data backup. It may be a good idea to start on a weekly schedule and see how the backups run from there. If you find that you are running out of space quickly you may need to move to a monthly schedule or another custom schedule that you create (Unless you are backing up Exchange, in which case you will need to stay on a Weekly schedule).

Recycling can be set to echo the backup schedules:

  1. In Clients > Edit (after selecting a client from the list on the left), you will be able to edit various aspects of your backup.

  2. Once you’ve selected a schedule, you can set up recycling via the Job Recycling section. Make sure this fits with your chosen schedule’s details (which can be found or created in Clients > Schedules).

    • If you chose the weekly schedule, set the job recycling (full, incremental, and differential backups) to recycle weekly if you want the job immediately cleared off of the RAID, or set them to recycle after a longer period of time if you need them to stay around.

    • If you chose the monthly schedule and just wanted space cleared as soon as possible, you could have full backups recycle monthly, while incremental and differential backups recycle weekly.

  3. Go to Jobs > Settings, and make sure Enable Automatic Job Management is selected.

You can also select Enable Automatic Deletion of Erred Jobs.

Preserve single job set

It’s highly recommended to enable Preserve Single Job Set. For this, go to Jobs > Settings, and set Job Retention Policy to Preserve Single Job Set.

This will tell the CFA not to recycle a job until there is a viable replacement. This option is recommended to be sure the most recent full and any subsequent differentials/incrementals remain on the CFA and aren’t automatically deleted.

Keep in mind, however, that this setting will affect recycling/retention settings until new full is available.

For example, If you have a client with the following settings:

  • Backup Schedule: Monthly (full backups – on the first Sunday, differential backups – on the second/fifth Sunday, and incremental backups – every Monday-Saturday)

  • Recycling/Retention:

    • Full 5 weeks

    • Diff 1 week

    • Inc 5 days

  • Preserve Single Job Set ENABLED

This is what you will see:

  • The full and incremental backups will run for the first week.

    Expired incremental backups won’t be deleted because of the ‘preserve single job set’ setting.

  • The differential and incremental backups will run for the first week.

    • Expired incremental backups from the first week will delete because the differential has the needed data collected from the time the full backup was taken.

    • Expired incremental backups for the second week will not be deleted because of the Preserve Single Job Set setting.

  • Incremental backups will run for weeks three and four.

    Both the expired differential and the expired incremental backups from weeks two, three and four not be deleted because of the Preserve Single Job Set setting.

  • The full will run the first Sunday of the following month.

    At this point, all expired differential and incremental backups from the previous month will be deleted (once they hit their retention/recycling setting).

The more time between full backups on a client, the greater the impact of the Preserve Single Job Set feature.

If you find yourself running out of space because differential and incremental backups are not automatically deleted when their retention settings say they should be, check to see if you have Preserve Single Job Set selected. If this is the case, it’s likely that the CFA is working as designed and the backup schedules and retention settings need to be modified. Try a weekly schedule rather than a Monthly schedule and set retention settings accordingly. If you need one month’s worth of backups available for restoration but don’t seem to have the capacity on the RAID, remember you can also make use of the archiving feature to save the backup jobs for a month off the RAID.

Other space considerations

Delayed calculation
The storage space calculations, used in the bottom bar, are automatically performed every 12 hours and can’t be forced.

If space still doesn’t add up, the database is used extensively for the accounting calculations. If the database has errors, the accounting will likely be incorrect. Please contact Infrascale Support.

If replication has been suspended due to lack of space on the RAID and does not automatically start up again please go to Replication > Status on the primary CFA, click Actions, and then select Reconcile Secondary.

The trouble is that space usage is difficult to pin down if there are other processes running in the background (de-duping, importing, or new jobs, for instance). Even if all jobs are deleted from the raid the CFA then has to check for unreferenced data, recalculate space and also run the garbage collector in order to get an accurate reading (and all of this before the next set of jobs runs). If any jobs were in the process of de-duplicating when they were deleted it can hold up this calculation process (de-duplication of backups actually takes almost double the space of the backup while in progress, but can save you space in the long run, after it’s done). If garbage collection runs while jobs are completing the importing or de-duping those jobs aren’t part of the space recalculation and can offset the numbers. If large amounts of data is all deleted at once the CFA has to check itself to make sure that there aren’t bits and pieces of these jobs left behind (similar to, but not exactly like, when you defrag any other computer system, only the CFA is trying to do this while simultaneously receiving new data). Finally, the only part of the space usage that’s set to recalculate more frequently than once per day is the free space, but any of the above scenarios can throw that off.