Commvault DASH copy nightmare, and how to improve it.

So I like to write about my experience with Commvault DASH (aka Aux) copy.

DASH copy is used when you want to ship local backups to a remote site for off site backup / DR backup.

I have two Media Agents, MediaAgent is actually a disk storage, one in our HQ and the other one is at our DR site, two MAs are connected with 500mb pipe.

This is about my experience with 4TB SQL database.

Commvault uses a database call dedup database, aka DDB, to keep track of changes, it makes a full backup once and backup only changes (deltas), on DDB they store/record so called "signature", using the signature it will only backup what is changed. they keep DDB on both Local and DR MA, It reduces backup time and storage space. Great!

For DASH copy, there are two options, disk optimized and network optimized, disk optimized is the default option. There is a sub option called "source side cache".

According to their tech document, disk optimized dash copy ships only the deltas/changes, it will use signatures to determine what it already have. Network optimized will re-open the local backup create signature on the fly and ship deltas to DR MA (MediaAgent).

My experience is, with disk optimized dash copy, it ships the entire full backup to the DR MA, every time it runs the DASH copy, and DR MA calculate signature and save only the difference. When I ran it, it took more than 24 hours, and daily backups accumulated over and over again and never finished. I called their tech support, they said stay put because Dr MA is generating signatures and it take time. once it generate the signature it will be faster. It finally shipped the first copy, I was expected to see improvements, NOT! No there is no DDB involved. Yep it shipped 4TB over 500mb, over and over again. it would never finish, I had more than 15 backups stacked up! 4TB x 15.
I called them and they insisted that disk optimized dash copy ships only deltas. I showed them what I found on network utilized report, they said change it to network optimized.

With Network optimized, it actually read the whole backup, with expense, the local MA's disk queue length went up beyond 50. my local MA has 18 disks, but it sent deltas only wow!. But it took a while to read the 4TB data, it took 11 to 13 hours. it also had to reseed, which end up eating up licenses.

I called again and they said enable the Local cache with disk optimized, it will create a local signature database on the "client computer", and it will make the process faster. hmm...what about the DDB?
It turned out, the local cache database is created not on the SQL database server, it got created on the local MA. It reseeded again, and eat up the license of course. But after it created local signature database, it is now taking 4 to 5 hours to finish the dash copy.

Conclusion,

1. "disk optimized + local cache" is the best option for the DASH copy, disk optimized only option will ship full backup to the DR MA and recalculate signature, there is no DDB involved.
2. On DASH copy client computer = Local MA, if you enable the Local cache.
3. Network optimized option does not optimize network traffic, I don't even know why the option even exist.
4. If you create DASH copy, "disk optimized - local cache" is the default option. Enable the local cache and optimize the DASH copy performance and network utilization. Yes, consider when you do that it might reseed!

SCCM updatesdeplyment log's assignment ID is deployment ID, hunting for job error 0x8024000f

One day I found

Job error (0x8024000f) received for assignment ({bdd02889-257b-431c-98b3-965a16ee51d7}) action

I was wondering what the heck the assignment ID is, after googling I found assignment ID is actually deployment ID. BAH!

To find out what that is, there are two ways to do it.

1. From the SCCM console -> monitoring -> Deployments -> and search for the ID.
2. Open SCCM powershell, Get-CMDeployment -DeploymentID 



Besides, a fantastic blog about the 0x8024000f can be found here:
https://blogs.technet.microsoft.com/ken_brumfield/2014/08/24/whoa-wuau-what-the-heck-is-with-the-circular-references-0x8024000f/

Also reference for the error code: https://social.technet.microsoft.com/wiki/contents/articles/15260.windows-update-agent-error-codes.aspx