Help: Tomcat crashing with "org.apache.jk.common.ChannelSocket processConnection"

Hi,

I need help with debugging. I am using Tomcat 6.0.24 and using mod_proxy to connect to httpd.

Seems like roughly 48 hours our server is crashing. I noticed that there were these messages that were in the log file before the crash occurs:

Aug 23, 2012 11:30:14 PM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2

Then about 20 minutes later these messages appear:

Aug 24, 2012 12:05:27 AM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space) executing org.apache.jk.common.ChannelSocket$SocketConnection@dd8862d, terminating thread
Aug 24, 2012 12:29:24 AM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: PermGen space

then a flood of these messages again:
Aug 24, 2012 12:50:41 AM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2

Which brings everything down forcing a restart.

I'm thinking this has to do with Apache Httpd fronting Tomcat with mod_proxy, but I'm not certain.

Any suggestions? Or anything else I can provide that could help me troubleshoot this issue?

Thank you

Also, in the Apache Httpd logs I found these messages if this helps:


[Fri Aug 24 00:11:39 2012] [error] (70014)End of file found: ajp_ilink_receive() can't receive header
[Fri

[Fri Aug 24 00:20:10 2012] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header

[Fri Aug 24 00:21:10 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

I'm not an expert on tomcat but this is looks like a big clue: java.lang.OutOfMemoryError: PermGen space

The solution probably involves looking for memory leaks and possibly increasing your heap size.

Hope this helps.

Yes, this was our first thought and we've been bumping up PermSize/MaxPermSize considerably . We're thinking if this is relative to maybe mod_proxy is not receiving responses or it's timing out.

Here is our JVM settings:


JAVA_OPTS="-server -d64"
JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m"
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=300m -XX:MaxPermSize=300m"
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=1024m -XX:MaxNewSize=1024m"
JAVA_OPTS="$JAVA_OPTS -XX:TargetSurvivorRatio=90 -XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=12"
JAVA_OPTS="$JAVA_OPTS -XX:+CMSClassUnloadingEnabled"

Our physical memory is set to 10G and only half is ever consumed. However, then the cpu goes up to 100% and stalls and we start seeing these errors/warnings.

Thanks

i am seeing similar errors with Apache/2.2.14, Tomcat/6.0.24, and ajp...i'd be very curious to hear if you found a way to fix this.