Archive for the ‘General’ Category

This morning, I have tried to install Weblogic 10.3.4, using the wls1034_generic.jar provided by Oracle.
I got this error:
[java]ClassNotFoundException: com.bea.cie.gpr.internal.model.DelegateHomeListHelper[/java]

Indeed, this error is known: I had tried to install on Windows from a network drive, cf http://support.sas.com/kb/38/566.html.

Quick fix: copy wls1034_generic.jar on a local drive and then run it via %JAVA_HOME%\bin\java -jar wls1034_generic.jar.

Thanks to Nicolas Griso of Xebia for his support ;-).

Case

I am migrating a project from Maven 2 to Maven 3. Fortunately, Maven 3 is said to be mostly back-compatible with Maven 2.
Anyway, on first build, I got this message:

[java][INFO] Scanning for projects…
[ERROR] The build could not read 3 projects -> [Help 1]
[ERROR]
[ERROR] The project lalou.jonathan.rmi-server:1.10.0-SNAPSHOT (C:\workarea\…\pom.xml) has 2 errors
[ERROR] ‘build.plugins.plugin[org.codehaus.mojo:weblogic-maven-plugin].dependencies.dependency.scope’ for org.slf4j:slf4j-log4j12:jar must be one of [compile, runtime, system]
but is ‘test’. lalou.jonathan:rmi-server:${project.parent.version}, C:\workarea\…\pom.xml, line 103,
column 32[/java]

Quick fix

Introduce a tag pluginManagement between build and plugins, ie replace:

[xml]
<build>
(…)
<plugins>
(…)
[/xml]

with:

[xml]
<build>
(…)
<pluginManagement>
<plugins>
(…)
[/xml]

The issue looks simple, but I needed a little search.

Case

I have a log4j config file, with three appenders: INFO, console and trace. I have to add a brand new appender (let’s say: JonathanNewAppender) that will log the events of only one class (let’s say: JonathanComponent). How to configure Log4J to perform that?

Solution

Edit the log4j.properties file.
Do not change the property log4j.rootCategory, do not mention JonathanNewAppender.

[java]log4j.rootCategory=INFO, console, trace[/java]

Add the properties of the appender, for instance: Donner les proprietes de l’appender:

[java]log4j.appender.JonathanNewAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.JonathanNewAppender.Append=true
log4j.appender.JonathanNewAppender.File=logs/prsl-sent-and-received.csv
log4j.appender.JonathanNewAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.JonathanNewAppender.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss}%m%n
log4j.appender.JonathanNewAppender.threshold=INFO[/java]

And here is the trick: add the log level and the actual appender for the new class.

[java]log4j.logger.my.package.name.JonathanComponent=INFO, JonathanNewAppender[/java]

Indeed, the only field of a logger to be mandatory is the log level. When the appender is specified, it will be taked in account. Otherwise, the logger will be attached all the appenders available in log4j.rootCategory property.

Case

I have had to access a BLOB and read its content. By principle, I dislike using binary objects, which do not suit easy tracing and auditing. Anyway, in my case, floats are stored in a BLOB, and I need read them in order to validate my current development.

You have many ways to read the content of the BLOB. I used two: SQL and Java

SQL

Start your TOAD for Oracle ; you can launch queries similar to this:

[sql]SELECT UTL_RAW.cast_to_binary_float
(DBMS_LOB.submyrecord (myrecord.myrecordess,
4,
1 + (myrecordessnameid * 4)
)
) AS myrecordessvalue
FROM mytable myrecord
WHERE myrecordessid = 123456; [/sql]
You can also run a stored procedure, similar to this:
[sql]
DECLARE
blobAsVariable BLOB;
my_vr RAW (4);
blobValue FLOAT;
bytelen NUMBER := 4;
v_index NUMBER := 5;
jonathan RAW (4);
loopLength INT;
BEGIN
SELECT myField
INTO blobAsVariable
FROM myTable
WHERE tableid = (5646546846);

DBMS_LOB.READ (blobAsVariable, bytelen, 1, jonathan);
loopLength := UTL_RAW.cast_to_binary_integer (jonathan);

FOR rec IN 1 .. loopLength
LOOP
DBMS_LOB.READ (blobAsVariable, bytelen, v_index, my_vr);
blobValue := UTL_RAW.cast_to_binary_float (my_vr);
v_index := v_index + 4;
DBMS_OUTPUT.put_line (TO_CHAR (blobValue));
END LOOP;
END;[/sql]

Java

I am still not sure to be DBA expert. Indeed I am convinced I am more fluent in Java than in PL/SQL 😉

Create a Spring configuration file, let’s say BlobRuntimeTest-applicationContext.xml:
[xml]<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

<!– $Id: BlobRuntimeTest-applicationContext.xml $ –>
<bean id="dataSource" destroy-method="close" class="org.apache.commons.dbcp.BasicDataSource">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:@myDBserver:1234:MY_SCHEMA"/>
<property name="username" value="jonathan"/>
<property name="password" value="lalou"/>
<property name="initialSize" value="2"/>
<property name="minIdle" value="2"/>
</bean>

<bean id="jdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">
<property name="dataSource" ref="dataSource"/>
</bean>

</beans>[/xml]

Now create a runtime test:
[java]/**
* User: Jonathan Lalou
* Date: Aug 7, 2011
* Time: 5:22:33 PM
* $Id: BlobRuntimeTest.java $
*/
public class BlobRuntimeTest extends TestCase {
private static final Logger LOGGER = Logger.getLogger(BlobRuntimeTest.class);

private static final String TABLE = "jonathanTable";
private static final String PK_FIELD = "jonathanTablePK";
private static final String BLOB_FIELD = "myBlobField";
private static final int[] PK_VALUES = {123, 456, 789};

private ApplicationContext applicationContext;
private JdbcTemplate jdbcTemplate;

@Before
public void setUp() throws Exception {
applicationContext = new ClassPathXmlApplicationContext(
"lalou/jonathan/the/cownboy/BlobRuntimeTest-applicationContext.xml");
assertNotNull(applicationContext);
jdbcTemplate = (JdbcTemplate) applicationContext.getBean("jdbcTemplate");
assertNotNull(jdbcTemplate);
}

@After
public void tearDown() throws Exception {
}

@Test
public void testGetArray() throws Exception {
for (int pk_value : PK_VALUES) {
final Blob blob;
final byte[] bytes;
final float[] floats;

blob = (Blob) jdbcTemplate.queryForObject("select " + BLOB_FIELD + " from " + TABLE + " where " + PK_FIELD + " = " + pk_value, Blob.class);
assertNotNull(blob);
bytes = blob.getBytes(1, (int) blob.length());
// process your blob: unzip, read, concat, add, etc..
// floats = ….

LOGGER.info("Blob size: " + floats.length);
LOGGER.info(ToStringBuilder.reflectionToString(floats));
}
}
}
[/java]

Case

I have a WAR which exposes RMI services via a servlet. It has perfectly worked for months. This afternoon, while redeploying it I got this error:

[java]weblogic.management.DeploymentException: Application Jonathan-rmi-server is a WAR file, but it contains > 1 component[/java]

Complete Stacktrace

[java]weblogic.management.DeploymentException: Application Jonathan-rmi-server is a
WAR file, but it contains > 1 component.
at weblogic.servlet.internal.WarDeployment.createModule(WarDeployment.java:40)
at weblogic.servlet.internal.WarDeployment.<init>(WarDeployment.java:27)
at weblogic.servlet.internal.WarDeploymentFactory.createDeployment(WarDeploymentFactory.java:36)
at weblogic.application.internal.DeploymentManagerImpl.createDeployment(DeploymentManagerImpl.java:84)
at weblogic.deploy.internal.targetserver.BasicDeployment.createDeployment(BasicDeployment.java:149)
at weblogic.deploy.internal.targetserver.operations.ActivateOperation.createAndPrepareContainer(ActivateOperation.java:202)
at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doPrepare(ActivateOperation.java:98)
at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)[/java]

Quick fix

Google does not give much information on this error. Anyway, I found this link: WebLogic9 Avoid new lines for attributes in the weblogic.xml . In my case it was not relevant, since my WAR was a classic one, deployable under any application server, and not specifically WebLogic.

Yet, to fix the issue, I had to split the redeploy operation into two phases undeploy and deploy. I could do that either using WebLogic console, or with Maven, ie replacing

[java]mvn package weblogic:deploy[/java]

with

[java]mvn package weblogic:undeploy weblogic:deploy[/java]

Case

My project is deployed via Maven on a Jetty 6 server. I have to upgrade the server to Jetty 7.
(BTW: notice that the plugin maven-jetty-plugin was renamed as jetty-maven-plugin)

I get this error:

[java]java.lang.NoClassDefFoundError: javax/servlet/ServletRequestListener[/java]

Quick Fix

In Jetty plugin, add a dependency to servlet-api, eg:
[xml] <plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-maven-plugin</artifactId>
<version>7.4.4.v20110707</version>
<dependencies>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.4</version>
</dependency>
</dependencies>[/xml]

Case

My project is deployed via Maven on a Jetty 6 server. I have to upgrade the server to Jetty 7.
(BTW: notice that the plugin maven-jetty-plugin was renamed as jetty-maven-plugin)

I get this error:

[java] Cause: Class name which was explicitly given in configuration using ‘implementation’ attribute: ‘org.mortbay.jetty.bio.SocketConnector’ cannot be loaded[/java]

Quick fix

org.mortbay.jetty.bio.SocketConnector was removed from Jetty with the release 7. To fix the issue, use a more recent implementation. For instance, replace this block:
[xml]<connector implementation="org.mortbay.jetty.bio.SocketConnector">[/xml]
with that one:
[xml]<connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector">[/xml]

Case

My project is deployed via Maven on a Jetty 6 server. I have to upgrade the server to Jetty 7.
(BTW: notice that the plugin maven-jetty-plugin was renamed as jetty-maven-plugin)

I get this error:

[java] Error configuring: org.mortbay.jetty:jetty-maven-plugin. Reason: ERROR: Cannot override read-only parameter: contextPath in goal: jetty:run-exploded[/java]

Quick fix

Insert a tag <webAppConfig> between <configuration> and <contextPath>. In other terms, replace this block:

[xml]<configuration>
<contextPath>/I/love/USA</contextPath>
</configuration>[/xml]

with that one:

[xml]<configuration>
<webAppConfig>
<contextPath>/I/love/USA</contextPath>
</webAppConfig>
</configuration>[/xml]

Case

I have a WAR containing Axis2-built on webservices. I must deploy it under Jetty 6. The version of Axis2 is 1.5.X.
I get this exception:
[java]java.lang.ClassNotFoundException: org.apache.axis2.transport.http.AxisAdminServlet[/java]

I assume the same case may occur with Tomcat.

Explanation

The class AxisAdminServlet was removed from Axis2 between the releases 1.4 and 1.5.

Quick fix

Add the following dependency in Maven’s Jetty plugin:
[xml]<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty-plugin</artifactId>
<version>6.1.26</version>
<dependencies>
<dependency>
<groupId>org.apache.axis2</groupId>
<artifactId>axis2-kernel</artifactId>
<version>1.4.1</version>
</dependency>
</dependencies>
(…)[/xml]

Abstract

The application I work on packages Mule ESB 2.2.1 in a WAR and deploys it under a WebLogic 10.3 server. My team mates and I noticed that, on multiple deploy/undeploy cycles, the PermGen size dramatically decreased. The cause of this was the number of threads, which hardly decreased on undeployment phases, unlike the expected behaviour.
Indeed, Mule is seldom deployed as a WebApp. Rather, it is designed to be run as a standalone application, within a Tanuki wrapper. When the JVM is killed, all the threads are killed, too, and therefore no thread survives ; hence, the memory is freed and there is no reason to fear a thread leak.

Moreover, when the application is redeployed, new threads -with the same names as the “old” threads- are created. The risk is that, for any reason, a thread-name-based communication between threads may fail, because the communication pipe may be read by the wrong thread.

In my case: on WebLogic startup, there are 31 threads ; when the application is deployed, there are 150 ; when the application works (receives and handles messages), the number of threads climbs to 800 ; when the application is undeployed, only 12 threads are killed, the other remaining alive.

The question is: how to kill Mule-created threads, in order to avoid a Thread leak?

WebLogic Threads

I performed a thread dump at WebLogic startup. Here are WebLogic threads, created before any deployment occurs:

[java]Attach Listener
DoSManager
DynamicListenThread[Default[1]]
DynamicListenThread[Default]
ExecuteThread: ‘0’ for queue: ‘weblogic.socket.Muxer’
ExecuteThread: ‘1’ for queue: ‘weblogic.socket.Muxer’
ExecuteThread: ‘2’ for queue: ‘weblogic.socket.Muxer’
Finalizer
JMX server connection timeout 42
RMI Scheduler(0)
RMI TCP Accept-0
RMI TCP Connection(1)-127.0.0.1
RMI TCP Connection(2)-127.0.0.1
Reference Handler
Signal Dispatcher
Thread-10
Thread-11
Timer-0
Timer-1
VDE Transaction Processor Thread
[ACTIVE] ExecuteThread: ‘0’ for queue: ‘weblogic.kernel.Default (self-tuning)’
[ACTIVE] ExecuteThread: ‘2’ for queue: ‘weblogic.kernel.Default (self-tuning)’
[STANDBY] ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default (self-tuning)’
[STANDBY] ExecuteThread: ‘3’ for queue: ‘weblogic.kernel.Default (self-tuning)’
[STANDBY] ExecuteThread: ‘4’ for queue: ‘weblogic.kernel.Default (self-tuning)’
[STANDBY] ExecuteThread: ‘5’ for queue: ‘weblogic.kernel.Default (self-tuning)’
main
weblogic.GCMonitor
weblogic.cluster.MessageReceiver
weblogic.time.TimeEventGenerator
weblogic.timers.TimerThread
[/java]

Dispose Disposables, Stop Stoppables…

The application being deployed in a WAR, I created a servlet implementing ServletContextListener. In the method contextDestroyed(), I destroy Mule objects (Disposable, Stoppable, Model, Service, etc.) one per one.
Eg#1:

[java] final Collection<Model> allModels;
try {
allModels = MuleServer.getMuleContext().getRegistry().lookupObjects(Model.class);
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Disposing models " + allModels.size());
}
for (Model model : allModels) {
model.dispose();
}
allModels.clear();
} catch (Exception e) {
LOGGER.error(e);
}[/java]

Eg#2:

[java] private void stopStoppables() {
final Collection<Stoppable> allStoppables;
try {
allStoppables = MuleServer.getMuleContext().getRegistry().lookupObjects(Stoppable.class);
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Stopping stoppables " + allStoppables.size());
}
for (Stoppable stoppable : allStoppables) {
stoppable.stop();
}
allStoppables.clear();
} catch (MuleException e) {
LOGGER.error(e);
}
}[/java]

This first step is needed because default mechanism is flawed: Mule re-creates objects that were destroyed.

Kill Threads

The general idea to kill Mule threads is the following: perform a Unix-style “diff” between WebLogic native threads, and the threads still alive once all Mule objects have been stopped and disposed.

On Application Startup

In the ServletContextListener, I add a field that will be set in a method called in the constructor:
[java] private List<String> threadsAtStartup;
(…)
/**
* This method retrieves the Threads present at startup: mainly speaking, they are Threads related to WebLogic.
*/
private void retrieveThreadsOnStartup() {
final Thread[] threads;
final ThreadGroup threadGroup;
threadGroup = Thread.currentThread().getThreadGroup();
try {
threads = retrieveCurrentActiveThreads(threadGroup);
} catch (NoSuchFieldException e) {
LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
threadsAtStartup = new ArrayList<String>();
return;
} catch (IllegalAccessException e) {
LOGGER.error("Could not retrieve initial Threads list. The application may be unstable on shutting down ", e);
threadsAtStartup = new ArrayList<String>();
return;
}

threadsAtStartup = new ArrayList<String>(threads.length);
for (int i = 0; i < threads.length; i++) {
final Thread thread;
try {
thread = threads[i];
if (null != thread) {
threadsAtStartup.add(thread.getName());
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("This Thread was available at startup: " + thread.getName());
}
}
} catch (RuntimeException e) {
LOGGER.error("An error occured on initial Thread statement: ", e);
}
}
}
/**
* Hack to retrieve the field ThreadGroup.threads, which is package-protected and therefore not accessible
*
* @param threadGroup
* @return
* @throws NoSuchFieldException
* @throws IllegalAccessException
*/
private Thread[] retrieveCurrentActiveThreads(ThreadGroup threadGroup) throws NoSuchFieldException, IllegalAccessException {
final Thread[] threads;
final Field privateThreadsField;
privateThreadsField = ThreadGroup.class.getDeclaredField("threads");
privateThreadsField.setAccessible(true);

threads = (Thread[]) privateThreadsField.get(threadGroup);
return threads;
}
[/java]

On application shutdown

In the method ServletContextListener.contextDestroyed(), let’s call this method:
[java] /**
* Cleanses the Threads on shutdown: theorically, when the WebApp is undeployed, should remain only the threads
* that were present before the WAR was deployed. Unfornately, Mule leaves alive many threads on shutdown, reducing
* PermGen size and recreating new threads with the same names as the old ones, inducing a kind of instability.
*/
private void cleanseThreadsOnShutdown() {
final Thread[] threads;
final ThreadGroup threadGroup;
final String currentThreadName;

currentThreadName = Thread.currentThread().getName();

if (LOGGER.isDebugEnabled()) {
LOGGER.debug("On shutdown, currentThreadName is: " + currentThreadName);
}

threadGroup = Thread.currentThread().getThreadGroup();
try {
threads = retrieveCurrentActiveThreads(threadGroup);
} catch (NoSuchFieldException e) {
LOGGER.error("An error occured on Threads cleaning at shutdown", e);
return;
} catch (IllegalAccessException e) {
LOGGER.error("An error occured on Threads cleaning at shutdown", e);
return;
}

for (Thread thread : threads) {
final String threadName = thread.getName();
final Boolean shouldThisThreadBeKilled;

shouldThisThreadBeKilled = isThisThreadToBeKilled(currentThreadName, threadName);
if (LOGGER.isDebugEnabled()) {
LOGGER.info("should the thread named " + threadName + " be killed? " + shouldThisThreadBeKilled);
}
if (shouldThisThreadBeKilled) {
thread.interrupt();
thread = null;
}
}

}

/**
* Says whether a thread is to be killed<br/>
* Rules:
* <ul><li>a Thread must NOT be killed if:</li>
* <ol>
* <li>it was among the threads available at startup</li>
* <li>it is a Thread belonging to WebLogic (normally, WebLogic threads are among the list in the previous case</li>
* <li>it is the current Thread (simple protection against unlikely situation)</li>
* </ol>
* <li>a Thread must be killed: in all other cases</li>
* </ul>
*
* @param currentThreadName
* @param threadName
* @return
*/
private Boolean isThisThreadToBeKilled(String currentThreadName, String threadName) {
final Boolean toBeKilled;
toBeKilled = !threadsAtStartup.contains(threadName)
&amp;&amp; !StringUtils.contains(threadName, "weblogic")
&amp;&amp; !threadName.equalsIgnoreCase(currentThreadName);
return toBeKilled;
}
[/java]

EhCache

My application uses an EhCache. Its threads names usually end with “.data”. They are not killed by the previous actions. To get rid of them, the most elegant way is to add this block in the web.xml:
[xml] <listener>
<listener-class>net.sf.ehcache.constructs.web.ShutdownListener</listener-class>
</listener>
[/xml]
cf EhCache documentation

With all these operations, almost all threads are killed. But Java VisualVM still displays 34, vs. 31 at startup.

Tough Threads

A thread dump confirms that, at this point, 3 rebellious threads still refuse to be kill:
[java]MuleServer.1
SocketTimeoutMonitor-Monitor.1
SocketTimeoutMonitor-Monitor.1
[/java]
Let’s examine them:

• MuleServer.1: This thread is an instance of the inner class MuleServer.ShutdownThread. Indeed, this is the first thread created by Mule, and therefore appears among the threads available at startup, before the ServletContextListener is called… I did not succeed in killing it, even why trying to kill it namely, which makes sense: killing the father thread looks like suiciding the ServletContextListener.
• SocketTimeoutMonitor-Monitor.1: This thread is created by Mule’s TcpConnector and its daughter classes: HttpConnector, SslConnector, etc. Again, I could not kill them.

Conclusion

We have seen Mule suffers of major thread leaks when deployed as a WAR. Anyway, most of these leaks may be sealed.
I assume MuleSoft was aware of this issue: in the version 3 of Mule, the deployment of webapps was refactored.