PeoplePicker: There was an error in the callback

Working with an environment where the QA and development machines are in a Test domain, while the production farm and all the primary users are in another domain. There is a one way domain level trust where the test domain trusts the production domain. In SharePoint 2007, we ran SetAppPassword and PeoplePicker-SearchADForests and everything worked just fine. However, in SharePoint 2010, we ran the exact same commands, and we started getting the following error when searching for users…FAIL :
There was an error in the callback
Thinking that the HTTP response would have some clues as to the cause of the error, we started out with a netmon trace. Unfortunately, there are no extra details in the actual response. I would see a POST to Picker.aspx with the search string, but the only thing in the response is the error message. No HTML, only the error. SharePoint trace logs are not showing anything either. We cleared out the PeoplePicker-SearchADForests property, and the People Picker would find users from the QA domain, and it would find users that were already added to the site. The content database was an upgrade from SharePoint 2007, so there were already Production user accounts in the site collections.
I started doing some research, and everything I was finding with regards to the error message was related to the SetAppPassword command not being ran on all the boxes. When you run SetAppPassword, it creates the following registry value:
HKEY_Local_Machine\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\Secure
AppCredentialKey : REG_BINARY
AppCredentialKey is the information used to encrypt/decrypt the information you pass into PeoplePicker-SearchADForests. This information has to be the same across all the servers, so you have to run SetAppPassword on every machine in the farm using the same password. In our case, we exported the registry key, and used Beyond Compare to validate that the entry was identical on all the servers.
Knowing that the registry key was a core piece of this error message, we did the following :

1. Updated our Hosts file on our client machine to point to a specific server
2. Ran Process Monitor on the server
3. Ran IISReset
4. Reproduced the error

We then searched the log file for the Secure registry key and found a single entry with a status of “Access Denied”. Looking at the details revealed that the Application Pool account was trying to read the registry and didn’t have access. We then checked permissions on the key to confirm that the WSS_WPG group did not have any access. In SharePoint 2007, this group has read access by default. Since our app pool needs to read the registry key to get the value for decrypting the People Picker property, we granted WSS_WPG read access. Once this change was made, all was right in the world, and the People Picker was able to return users from both domains.
You won’t see this issue if you run all your app pools as the Database Access account [aka farm account], but that’s not really a good thing to do.
To set this up in SharePoint 2010, we came up with the following steps:

• Contoso domain has a one way trust with TailSpinToys
• SharePoint 2010 is installed on machines joined to Contoso and using least privilege configuration with Contoso service accounts.
• The People Picker needs to return users from both Contoso and TailSpinToys
  

1. Run SetAppPassword on EVERY machine in the farm. This will setup your AppCredentialKey for you.
   
   StsAdm.exe –o SetAppPassword –password MyPassword
   
2. Run SetProperty for PeoplePicker-SearchADForests on ONE server in the farm. This information is stored in the configuration database, no need to run it more than once for each URL where you want the people picker
   
   StsAdm.exe –o SetProperty –pn PeoplePicker-SearchADForests –pv “domain:TailSpinToys,TailSpinToys\AccountName,AccountPassword” –url http://TeamSites
   
3. Set permissions on the Secure registry key on EVERY machine in the farm, granting the local WSS_WPG read access
   1. Open Registry Editor
   2. Navigate to : HKEY_Local_Machine\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0\Secure
   3. Right-click Secure, select Permissions
   4. Click Add
   5. Find the local WSS_WPG group
   6. Select Read access
   7. Click OK out of the dialogs