Failed to initialize the application bea_wls_internal – Weblogic Cluster

Background:

2 node cluster setup with 2 managed servers (Admin + Managed;  Managed).

Weblogic Server 10.3.2

Weblogic Portal 10.3.2

Admin Server and Managed server directories are different (High Availability setup, Admin Server is on shared location, managed servers on local disk)

On starting the managed servers, the server fails to start. The output log file shows the below errors:

Error Security BEA-000000 [Security:090836]The Keystore provider configured for PKICredential Mapper does not exist at location wsrpKeystore.jks.

Error Deployer BEA-149205 Failed to initialize the application 'bea_wls_internal' due to error weblogic.application.ModuleException: Failed to load webapp: 'bea_wls_internal.war'.

weblogic.application.ModuleException: Failed to load webapp: 'bea_wls_internal.war'

        at weblogic.servlet.internal.WebAppModule.prepare(WebAppModule.java:387)

        at weblogic.application.internal.flow.ScopedModuleDriver.prepare(ScopedModuleDriver.java:180)

        at weblogic.application.internal.flow.ModuleListenerInvoker.prepare(ModuleListenerInvoker.java:93)

        at weblogic.application.internal.flow.DeploymentCallbackFlow$1.next(DeploymentCallbackFlow.java:388)

        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:37)

        Truncated. see log file for complete stacktrace

javax.servlet.ServletException: [Security:090820]The internal variable ServletInfoSpi is null and it should not be.

        at weblogic.security.providers.saml.SAMLServletAuthenticationFilter.init(SAMLServletAuthenticationFilter.java:52)

        at weblogic.security.service.internal.ServletAuthenticationFilterServiceImpl$ServiceImpl.getServletAuthenticationFilters(Unknown Source)

        at weblogic.security.service.PrincipalAuthenticator.getServletAuthenticationFilters(Unknown Source)

        at weblogic.servlet.security.internal.WebAppSecurity.init(WebAppSecurity.java:80)

        at weblogic.servlet.security.internal.WebAppSecurityWLS.init(WebAppSecurityWLS.java:66)

        Truncated. see log file for complete stacktrace

Cause.

The Managed server fails to load security profiles. This is because the security files are not present in the local directory structure. This is because we changed the managed server location from the shared storage to local directory using pack/unpack utility as per standard guidelines to create high availability setup. This utility did not copy the security files.

Fix.

Copy security files from shared location to local disks.

bash-3.00$ cd /admin_server/admin/domains/portal_domain/

bash-3.00$ cp wsrpKeystore.jks DemoTrust.jks DemoIdentity.jks /opt/oracle/product/middleware/user_projects/domains/portal_domain

Invalid xsi:type qname: wsrp:wsrp-identity-asserterType – Weblogic Portal

While starting Admin server using WLST in a domain which supports Weblogic Portal, the admin server does not starts and errors out with the following errors in the WLST console:

Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server 'AdminServer': java.io.IOException: Server failed to start up. See server output log for more details.

The output log file says this:

BEA-141244 Schema validation errors while parsing /admin_server/admin/domains/portal_domain/config/config.xml - Invalid xsi:type qname: 'wsrp:wsrp-identity-asserterType' in element realm@http://www.bea.com/ns/weblogic/920/domain

Error Management BEA-141244 Schema validation errors while parsing /admin_server/admin/domains/portal_domain/config/config.xml - /admin_server/admin/domains/portal_domain/unknown:13:9: error: failed to load java type corresponding to t=wsrp-identity-asserterType@http://www.bea.com/ns/wlp/90/security/wsrp

Critical WebLogicServer BEA-000362 Server failed. Reason: [Management:141245]Schema Validation Error in /admin_server/admin/domains/portal_domain/config/config.xml see log for details. Schema validation can be disabled by starting the server with the command line option: -Dweblogic.configuration.schemaValidationEnabled=false

Cause.

The domain which we created requires portal libraries to be loaded when any of the servers starts. This can be verified from the config.xml file in which all libraries are targeted to the Admin server and the cluster. When any server is started using WLST the node manager does not reads the setDomain.env file associated with that domain and hence the server is unaware of the path of the library files.

Fix.

Start the Admin server using the startWeblogic.sh script. Navigate to “Servers è Admin Server è Server Start” tab and add the following parameters for “Arguments” section.

-d64 -Dweblogic.ext.dirs=/opt/oracle/product/middleware/patch_wlw1030/profiles/default/sysext_manifest_classpath:/opt/oracle/product/middleware/patch_wls1030/profiles/default/sysext_manifest_classpath:/opt/oracle/product/middleware/patch_wlp1030/profiles/default/sysext_manifest_classpath:/opt/oracle/product/middleware/patch_cie670/profiles/default/sysext_manifest_classpath:/opt/oracle/product/middleware/wlportal_10.3/p13n/lib/system:/opt/oracle/product/middleware/wlportal_10.3/light-portal/lib/system:/opt/oracle/product/middleware/wlportal_10.3/portal/lib/system:/opt/oracle/product/middleware/wlportal_10.3/info-mgmt/lib/system:/opt/oracle/product/middleware/wlportal_10.3/analytics/lib/system:/opt/oracle/product/middleware/wlportal_10.3/apps/lib/system:/opt/oracle/product/middleware/wlportal_10.3/info-mgmt/deprecated/lib/system:/opt/oracle/product/middleware/wlportal_10.3/content-mgmt/lib/system -Dweblogic.alternateTypesDirectory=/opt/oracle/product/middleware/wlportal_10.3/portal/lib/security

*considering that the WL_HOME is /opt/oracle/product/middleware

Native version is enabled but node manager native library could not be loaded

The biggest challenge when working with 64 bit installations is to ensure that all components pick up the correct libraries. While doing a 64 bit installation of Weblogic Portal recently, my nodemanager refused to start.

While starting the node manager, following error message is thrown:

weblogic.nodemanager.common.ConfigException: Native version is enabled but node manager native library could not be loaded

        at weblogic.nodemanager.server.NMServerConfig.initProcessControl(NMServerConfig.java:239)

        at weblogic.nodemanager.server.NMServerConfig.(NMServerConfig.java:179)

        at weblogic.nodemanager.server.NMServer.init(NMServer.java:176)

        at weblogic.nodemanager.server.NMServer.(NMServer.java:141)

        at weblogic.nodemanager.server.NMServer.main(NMServer.java:337)

        at weblogic.NodeManager.main(NodeManager.java:31)

Caused by: java.lang.UnsatisfiedLinkError: /opt/oracle/product/middleware/wlserver_10.3/server/native/solaris/sparc64/libnodemanager.so: ld.so.1: java: fatal: /opt/oracle/product/middleware/wlserver_10.3/server/native/solaris/sparc64/libnodemanager.so: wrong ELF class: ELFCLASS64 (Possible cause: architecture word width mismatch)

        at java.lang.ClassLoader$NativeLibrary.load(Native Method)

        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1728)

        at java.lang.Runtime.loadLibrary0(Runtime.java:823)

        at java.lang.System.loadLibrary(System.java:1028)

        at weblogic.nodemanager.util.UnixProcessControl.(UnixProcessControl.java:24)

        at weblogic.nodemanager.util.Platform.getProcessControl(Platform.java:114)

        at weblogic.nodemanager.server.NMServerConfig.initProcessControl(NMServerConfig.java:237)

Cause.

This error occurs because of mismatch of libraries being loaded at runtime. The OS is 64 bit; therefore the client (node manager) should be started as a 64 bit client. For some reason, the startNodeManager.sh script did not had the “-d64” flag in the java options, hence this error.

               

Fix.

Edit the startNodeManager.sh script and add the “-d64” option to JAVA_VM variable.

JAVA_VM="-d64 ${JAVA_VM}

Restart the Node manager and it should start normally.