#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2005
    Posts
    390
    Rep Power
    11

    Help: Tomcat crashing with "org.apache.jk.common.ChannelSocket processConnection"


    Hi,

    I need help with debugging. I am using Tomcat 6.0.24 and using mod_proxy to connect to httpd.

    Seems like roughly 48 hours our server is crashing. I noticed that there were these messages that were in the log file before the crash occurs:

    Aug 23, 2012 11:30:14 PM org.apache.jk.common.ChannelSocket processConnection
    WARNING: processCallbacks status 2

    Then about 20 minutes later these messages appear:

    Aug 24, 2012 12:05:27 AM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
    SEVERE: Caught exception (java.lang.OutOfMemoryError: PermGen space) executing org.apache.jk.common.ChannelSocket$SocketConnection@dd8862d, terminating thread
    Aug 24, 2012 12:29:24 AM org.apache.catalina.connector.CoyoteAdapter service
    SEVERE: An exception or error occurred in the container during the request processing
    java.lang.OutOfMemoryError: PermGen space

    then a flood of these messages again:
    Aug 24, 2012 12:50:41 AM org.apache.jk.common.ChannelSocket processConnection
    WARNING: processCallbacks status 2

    Which brings everything down forcing a restart.

    I'm thinking this has to do with Apache Httpd fronting Tomcat with mod_proxy, but I'm not certain.

    Any suggestions? Or anything else I can provide that could help me troubleshoot this issue?

    Thank you
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2005
    Posts
    390
    Rep Power
    11
    Also, in the Apache Httpd logs I found these messages if this helps:


    [Fri Aug 24 00:11:39 2012] [error] (70014)End of file found: ajp_ilink_receive() can't receive header
    [Fri

    [Fri Aug 24 00:20:10 2012] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header

    [Fri Aug 24 00:21:10 2012] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
  4. #3
  5. No Profile Picture
    rebel with a cause
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    May 2004
    Location
    The Batsh!t Crazy State.
    Posts
    5,817
    Rep Power
    3462
    I'm not an expert on tomcat but this is looks like a big clue: java.lang.OutOfMemoryError: PermGen space

    The solution probably involves looking for memory leaks and possibly increasing your heap size.

    Hope this helps.
    Dear God. What is it like in your funny little brains? It must be so boring.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2005
    Posts
    390
    Rep Power
    11
    Yes, this was our first thought and we've been bumping up PermSize/MaxPermSize considerably . We're thinking if this is relative to maybe mod_proxy is not receiving responses or it's timing out.

    Here is our JVM settings:


    JAVA_OPTS="-server -d64"
    JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m"
    JAVA_OPTS="$JAVA_OPTS -XX:PermSize=300m -XX:MaxPermSize=300m"
    JAVA_OPTS="$JAVA_OPTS -XX:NewSize=1024m -XX:MaxNewSize=1024m"
    JAVA_OPTS="$JAVA_OPTS -XX:TargetSurvivorRatio=90 -XX:SurvivorRatio=5 -XX:MaxTenuringThreshold=12"
    JAVA_OPTS="$JAVA_OPTS -XX:+CMSClassUnloadingEnabled"

    Our physical memory is set to 10G and only half is ever consumed. However, then the cpu goes up to 100% and stalls and we start seeing these errors/warnings.

    Thanks
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    1
    Rep Power
    0

    did you ever track down a resolution to this?


    i am seeing similar errors with Apache/2.2.14, Tomcat/6.0.24, and ajp...i'd be very curious to hear if you found a way to fix this.

IMN logo majestic logo threadwatch logo seochat tools logo