Object already exists in RSACryptoServiceProvider

I copied the source code from one application to another, both running on the same machine. I am also using the same string for containerName below in both applications.

What is preventing my new application from reading the key that was saved in the other application? All other things are equal, logged in user account etc.

     CspParameters cspParams = new CspParameters();
     cspParams.KeyContainerName = containerName;
     cspParams.Flags = CspProviderFlags.UseMachineKeyStore;

     // Get error "object already exists" below.
     RSACryptoServiceProvider  rsaKey = new RSACryptoServiceProvider(cspParams);