Failed to initialize the application bea_wls_internal – Weblogic Cluster

Background:

2 node cluster setup with 2 managed servers (Admin + Managed;  Managed).

Weblogic Server 10.3.2

Weblogic Portal 10.3.2

Admin Server and Managed server directories are different (High Availability setup, Admin Server is on shared location, managed servers on local disk)

On starting the managed servers, the server fails to start. The output log file shows the below errors:

Error Security BEA-000000 [Security:090836]The Keystore provider configured for PKICredential Mapper does not exist at location wsrpKeystore.jks.

Error Deployer BEA-149205 Failed to initialize the application 'bea_wls_internal' due to error weblogic.application.ModuleException: Failed to load webapp: 'bea_wls_internal.war'.

weblogic.application.ModuleException: Failed to load webapp: 'bea_wls_internal.war'

        at weblogic.servlet.internal.WebAppModule.prepare(WebAppModule.java:387)

        at weblogic.application.internal.flow.ScopedModuleDriver.prepare(ScopedModuleDriver.java:180)

        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:93)

        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:388)

        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)

        Truncated. see log file for complete stacktrace

javax.servlet.ServletException: [Security:090820]The internal variable ServletInfoSpi is null and it should not be.

        at weblogic.security.providers.saml.SAMLServletAuthenticationFilter.init(SAMLServletAuthenticationFilter.java:52)

        at weblogic.security.service.internal.ServletAuthenticationFilterServiceImpl$ServiceImpl.getServletAuthenticationFilters(Unknown Source)

        at weblogic.security.service.PrincipalAuthenticator.getServletAuthenticationFilters(Unknown Source)

        at weblogic.servlet.security.internal.WebAppSecurity.init(WebAppSecurity.java:80)

        at weblogic.servlet.security.internal.WebAppSecurityWLS.init(WebAppSecurityWLS.java:66)

        Truncated. see log file for complete stacktrace

Cause.

The Managed server fails to load security profiles. This is because the security files are not present in the local directory structure. This is because we changed the managed server location from the shared storage to local directory using pack/unpack utility as per standard guidelines to create high availability setup. This utility did not copy the security files.

Fix.

Copy security files from shared location to local disks.

bash-3.00$ cd /admin_server/admin/domains/portal_domain/

bash-3.00$ cp wsrpKeystore.jks DemoTrust.jks DemoIdentity.jks /opt/oracle/product/middleware/user_projects/domains/portal_domain