I have investigated this a bit, and here's what seems to be happening.
During both a normal GlassFish start-up and the start-up during upgrade, specifying "--passwordfile -" on the command line works to collect (among other things) AS_ADMIN_MASTERPASSWORD=the-pw from stdin.
During a normal start-up, the IdmService.postConstruct method reads the setting from stdin and sets the master password in the MasterPassword object (injected into IdmService and also injected elsewhere).
But during the upgrade-triggered start-up, I did not see the IdmService.postConstruct ever invoked, probably because no class that's run during upgrade injects it. So, the user-entered master password is never read from stdin and is never set in MasterPassword. Later during the upgrade-triggered start-up, SecuritySupportImpl looks for (and does not find) a MasterPassword in hk2, so it uses the default master password.
If the old domain's master password is not the default, then SecuritySupportImpl cannot access the keystore and truststore using the default, and that triggers the failure the user reported.
This problem persists in the trunk (4.0), because the problem occurs during the start-domain --upgrade execution, not while the upgrade tool (which is now retired from the product) runs.
A near-fix would be to @Inject IdentityManager (for which IdmService is the implementation) into SecureAdminUpgradeHelper, to make sure that the master password is read from stdin before it's needed. But that is not the complete solution because of the way SecuritySupportImpl works. It does its initialization from its constructor, not from a postConstruct method. (There are some places in the code that directly instantiate SecuritySupportImpl without using hk2). So the habitat is not injected yet. Also, during the upgrade processing, the habitat in Globals is not yet set, so the code path in SecuritySupportImpl that tries to use that does not find a habitat which it would use to look up the MasterPassword. So it just falls back and uses the default password, which fails.
Because a complete fix lies outside just the secure admin upgrade classes, I need to let the security team get involved in this now.