Recommended Data Collection for Crashes and Hangs in the Application Server ( Solaris )
Complete Application server version
<appserver_install-directory>/bin/asadmin version --verbose
- Tar of the config files
<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<config>/*
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*
Application Server Process Hangs/High CPU
Self-help
- For 9.x
- Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
eg
Change
<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
to
<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/> - Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html
- Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
Data Collection
 | In a multi-zone solaris 10 environment, please perform this data collection at the respective local zones |
- prstat -L >> output.txt
- pstack pid > pstack.out same for pmap, pldd, pfiles, pflags
- Time stamp of the issue.
- server.logs that capture the problem occurrences
- Detail the load/number of concurrent users at that time of the problem.
- Does restarting the server resolve the issue?
- After a fresh restart of the server, for how long does it run without problems?
- For 7.x and 8.x
- Issue kill -3 command for 3 times successively with an interval of a minute. This will create the java thread dump in the server.log file.
- Please issue kill -3 pid command ONLY during the time of issue.
- For 9.x
- Please use the command : asadmin generate-jvm-report --type=thread | tee threaddump.txt
- Issue the command a few times over a period of time.
- Please provide us the threaddump.txt containing the thread dump.
<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log
Or <appserver_9.1.x_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/jvm.log
- Run gcore on the pid. It will dump a core file of the process. Run the SUN Internal pkgcore script on this core file from the same system where core was generated and provide the generated files as listed below.
./pkgcore.sh corefile
casenumber_libraries.tar.gz
casenumber_corefile.tar.gz
Please upload the above files in https://supportuploads.sun.com/upload
under the cores directory and provide us the checksum details of the uploaded files. - To track the memory usage provide the gclogs using the following Java Settings : -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:gclog.txt (JDK1.4.2.x and above)
- Alternatively, you can run the following data collection script from http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd_appserv_han when the application server goes into an unresponsive situation.
Customers are advised to familiarise themselves with the script in a staging environment first before attempting in a production environment. For more information on the Sun Gathering Debug Data ( GDD ), please refer to the following website http://www.sun.com/service/gdd/index.xml.
Application Server Process Crash
- Check for core file, if there is no core file generated check the coreadm section as mentioned above.
- Check for any hs_err_pid.log. By default, the file is created in the working directory of the process
- When was the issue noticed first?(Please provide the exact time stamp of the issue). Does this issue happen during high load or any particular activity? Explain.
- How often it is occurring?
- Provide the server.log at the time of crash and the access log around the time of the issue
<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log
- Run the SUN Internal pkgcore script on this core file from the same system where core was generated and provide the generated files as listed below
./pkgcore.sh corefile
casenumber_libraries.tar.gz
casenumber_corefile.tar.gz
Please upload the above files in https://supportuploads.sun.com/upload under the cores directory and provide us the checksum details of the uploaded files.