2.4 AIX

Recommended Data Collection for Crashes and Hangs in Application Server ( AIX )

Complete Application server version

<appserver_install-directory>/bin/asadmin version --verbose

• Tar of the config files

<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*

Application Server Process Hangs/High CPU

Self-help

• For 9.x
  • Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
    eg
    Change
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
    to
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/>
  • Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html

Data Collection

<pid> is the instance PID of the appserver

• To locate the pid
  ps auxwww | grep "com.sun.aas.instanceName=<name>"

• procstack <pid>
• procmap <pid>
• procfiles <pid>
• proctree <pid>
• ps -eaf
• netstat -an
• netstat -rn
• lsof -p <pid>
• ps -mp <pid> -o THREAD
• topas > topas.out
• truss -fealid -o truss.txt -p <pid>
• tprof -skex sleep 60

  This need root user to run. It takes 60 seconds to do a profiling of the appserver server to see how the application behaves so as to detect those high cpu threads

• Issue kill -3 command for 3 times successively with an interval of a minute. This will create the java thread dump in the file as described below.

  According to http://download.boulder.ibm.com/ibmdl/pub/software/dw/jdk/diagnosis/diag142.pdf. The dumped java dump is located under $INSTANCEDIR/config/javacore*

Application Server Process Crash

• Collect the following information from IBM 's website and engage IBM Tech support
• See http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21055387

Additional data collection

• IBM 's Must-Gather notes in http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21052641 for core files

Even though IBM JDK 6 has the following feature : jps and jstack. IBM JDK 6 is not supported in SGES21/AIX yet