Sun Microsystems Software Support

Tips for Troubleshooting

Troubleshooting Guide

• Application Server
  • Sun Java System Application Server 9.1 Troubleshooting Guide - http://docs.sun.com/app/docs/doc/819-3677
  • Sun Java System Application Server Enterprise Edition 8.2 Troubleshooting Guide - http://docs.sun.com/app/docs/doc/819-4738
  • Sun ONE Application Server 7, Enterprise Edition Troubleshooting Guide - http://docs.sun.com/source/817-2161-10/index.html
• Web Server
  • Sun Gathering Debug Data for Sun Java System Web Server - http://docs.sun.com/app/docs/doc/820-2483
  • Sun Java System Web Server 7.0 Troubleshooting Guide - http://docs.sun.com/app/docs/doc/820-2209
  • SUN Internal - 3 common basic types of problem encountered with Web Server - http://software-tsc.central.sun.com/twiki/bin/view/Products/WebServerTroubleshooting

Common Problems for Data Collection

• No core file produced ( Solaris )
  • Self-help
    Verify write permission in current working directory and core file
    Verify enough space for core file
    Verify ulimit -a shows "unlimited" for coredump
    Verify coreadm has per-process core dumps enabled
  • The coreadm output is incorrect
    Run coreadm(1M) and make the following recommended settings:
    #mkdir -p /var/cores
    #coreadm -g /var/cores/%f.%n.%p.core -e global -e process -e global-setid -e proc-setid -e log
    #coreadm
    global core file pattern: /var/cores/%f.%n.%p.core
    init core file pattern: core
    global core dumps: enabled
    per-process core dumps: enabled
    global setid core dumps: enabled
    per-process setid core dumps: enabled
    global core dump logging: enabled
  • Java specific 1 :
    The process received SIGSEGV or SIGILL but no core dump produced. May be the process handled it. For example, HotSpot VM uses the SIGSEGV signal for legitimate purposes such as throwing NullPointerException, deoptimization etc.
    Not all SIGSEGVs are bad! Only if the current instruction (PC) falls outside JVM generated code, the signal is unhandled by the JVM. Only in such cases, HotSpot dump core.
  • Java specific 2:
    The JNI Invocation API was used to create the VM. The standard Java launcher was not used. The custom Java launcher program handled the signal by just consuming it and produced the log entry silently.
    This has been seen with certain AppServers, WebServers. These JVM embedding programs transparently attempt to re-start (fail over) the system after abnormal termination.
    Not producing core is a feature and not a bug The above two are taken from section 3.2.3 in http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf
  • The coreadm output is correct but there is no core file generated when the application server or web server running in SSL mode crashed.
    For application server
    put the following statement "SSL_DUMP=1; export SSL_DUMP" in the startserv script of ( DAS or instance )
    For web server issue documentation http://sunsolve.central.sun.com/search/printfriendly.do?assetkey=1-9-63420-1 http://sunsolve.central.sun.com/search/printfriendly.do?assetkey=1-25-72079-1
  • Java thread dumps are not generated while performing "Ctrl and \ keys" are pressed, "kill -3" or "kill -QUIT" command.
    • Ensure the -Xrs option is not enabled in the Java flags/switch
    • For application server 9.x, please use the following command : asadmin generate-jvm-report
    • On windows the equivalent key sequence is the Ctrl and Break keys. If you can't get the console, please ensure to install
      • http://www.latenighthacking.com/projects/2003/sendSignal
      • http://www.adaptj.com/webstart/stacktrace/app/launch.jnlp

Checklist for Opening New Support Cases

• Please ensure that the following information is available when you submit your case to SUN support.

  Submitting this information when opening a support case will save time and potentially lead to a faster resolution to your problem

Check-List

• List SUN Software Product involved ( eg Application Server etc ):
• List NON-SUN Third Party Software Product involved ( eg Oracle RDMS ) :
• Hardware Platform: OS and Kernel Version:
  • Solaris : uname -a
  • HP-UX : uname -r
  • Linux : uname -a and more /etc/*-release or http://software-support.central.sun.com/twiki/pub/Products/JavaInformationGatheringMore/PlatformInfo (This is a script to gather detailled information about CPUs, OS and Framebuffers.)
  • Windows : C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt
• Patch Level:
  • Solaris : showrev -p
  • HP-UX :swlist
  • Linux : rpm -qa
  • Windows : Already provided in the C:\report.txt file above.
• Description of issue:
• Frequency of issue occurs:
• When was the problem first noticed:
• Recently Changed Variables:
• Copy and paste error messages from server log outputs, if Any :
• Steps to Reproduce Issue:

• If it is a crash and hang incident, see the Section for "Recommended Data Collection"

• Scripts for collection of core file
• Solaris
  • PkgApp
  • SUN Internal PkgCore
• Linux
  • PkgApp

Sample Support Case Entries

Sample Case 1 Sun Application Server problem on Hardware Platform Solaris-SPARC

List SUN Software Product involved ( eg Application Server etc ): Sun Java System Application Server 9.1_01 (build b09d-fcs)
List NON-SUN Third Party Software Product involved ( eg Oracle RDMS ) : Oracle11g
Hardware Platform: Sun Fire V890
OS and Kernel Version: Solaris : uname -a <br>
Sinus tapptx01 5.10 Generic_118833-03 sun4u spark SUNW,Sun-Fire-V890
Patch Level:
Solaris : showrev -p (collected and will be sent thru' E-mail after receiving the mail from the case/ Service Request owner)
Description of issue: Problems in redeployment of new version of Toplink Essential WAR File.
Frequency of issue occurs: Whenever we try to redeploy new version of Toplink Essential WAR File.
When was the problem first noticed: on 1st June 2008 at 15.55 Local Time.
Recently Changed Variables: OS Kernel Patch upgrade. But the problem was observed even before OS Kernel Patch upgrade.
Copy and paste error messages from server log outputs, if Any :
Error: The attempt to save your application for admission information has failed. Please retry. It it fails a second time, please contact the Help Desk.

Steps to Reproduce Issue:

For domain1 (problem domain):

1). Deploy the test application to root-context / on virtual server test using GUI
2). Redeploy the app using GUI
3). Fails

4). Un-deploy using GUI
5). Deploy app again to root-context / on virtual server test using GUI
6). Works

7). Un-deploy using GUI.
8). Deploy to root-context / without selecting a virtual server using GUI
9). Fails
***************
For dev (newly created domain for the same SJSAS install):-

1). Deploy the test application to root-context / no virtual server selected using GUI
2). Redeploy the app using GUI
3). Pass

4). Un-deploy the app using GUI
5). Deploy app again to root-context / on virtual server selected using GUI
6). Works

7). Un-deploy the app using GUI
8). Deploy to root-context / selecting default "server" virtual server using GUI
9). Pass

Attached the word document with screen-shots explaining about the steps described above.

The result is that it works for the domain that has no virtual servers configured and not for the domain that does have virtual servers.
The only real difference in the domain.xml files for the two domains is the configuration of a virtual server in domain1. Also domain1 makes use of an IP address for the http and iiop listeners, e.g. address="x.x.x.x" instead of address="0.0.0.0" for the dev domain.

<virtual-server hosts="applyonline-dev "
http-listeners="http-listener-1,http-listener-2" id="applyonline-dev"
log-file="${com.sun.aas.instanceRoot}/logs/server.log" state="on">
<property name="sso-enabled" value="false"/>
<property name="docroot"
value="${com.sun.aas.instanceRoot}/docroot"/>
<property name="accesslog"
value="${com.sun.aas.instanceRoot}/logs/access"/>
</virtual-server>

<virtual-server hosts="onlinehelp-dev "
http-listeners="http-listener-1,http-listener-2" id="onlinehelp-dev"
log-file="${com.sun.aas.instanceRoot}/logs/server.log" state="on">
<property name="sso-enabled" value="false"/>
<property name="docroot"
value="${com.sun.aas.instanceRoot}/docroot"/>
<property name="accesslog"
value="${com.sun.aas.instanceRoot}/logs/access"/>
</virtual-server>

Basically, both of these log files have the logging level set to FINE for the PERSISTENCE and UTILS. The two test cases executed include the following steps:

gls_server.log_2008-06-17_dev - Log file for a newly created domain in the same SJSAS instance.
gls_server.log_2008-06-17_domain1 - Log file for the domain in the instance that is causing the problem.

However, this works fine if I use asadmin CLI:
Initial deploy (contextroot provided):

asadmin deploy --host applyonline-dev --port 4848 --user admin --passwordfile pw.txt --virtualservers applyonline-dev --contextroot / ProgEnrol-war-2.0.3-SNAPSHOT.war

Redeploy:
asadmin deploy --host applyonline-dev --port 4848 --user admin --passwordfile pw.txt --virtualservers applyonline-dev --force=true ProgEnrol-war-2.0.3-SNAPSHOT.war
=========
As of now, I believe we have two work-arounds available for 9.1 U1:

1. Use the CLI with an explicit virtual server argument.
2. Use the GUI to first do an explicit "Undeploy" and then a new "Deploy". As part of the "Deploy" operation ensure that a single virtual server is explicitly specified.

I hope that this information helps to progress the case further.

Sample Case 2 Sun Application Server problem on Hardware Platform Windows-X86

List SUN Software Product involved ( eg Application Server etc ): Sun Java System Application Server PE 8.1 2005Q2 UR2
List NON-SUN Third Party Software Product involved ( eg Oracle RDMS ) : Oracle 9.2.0
Hardware Platform: Windows-X86
OS and Kernel Version:
Windows : C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt
Attachment (report.txt) will be sent thru' E-mail for the Case/ Service Request owner after receiving the mail.

However, here is the brief OS version:
Windows Server 2003 Enterprise Edition
Microsoft Windows
Version 5.2 (build 3790.srv03_sp1_gdr.070304-2232: Service Pack 1)

Patch Level:
Windows : Already provided in the C:\report.txt file above.
Description of issue: Appserver shutsdown itself, when the user logs-ff from the Machine
Frequency of issue occurs: Whenever the user logs-ff
When was the problem first noticed: 2th April '08 18:00 Hours Local Time
Recently Changed Variables: None
Copy and paste error messages from server log outputs, if Any :
Steps to Reproduce Issue:

This can be easily reproduced as below:-

1. Install Sun Java System Application Server PE 8.1 2005Q2 UR2 on Windows 2003 Server.
2. The Application Server needs to be maintained by different team members (at least 3)
3. User-1 logs-off from Windows Machine after Shift change to allow user-2 to login
4. User-2 logs into the system
5. Notice that Application Server is down itself
6. Need to start Application Server manually by user-2 again.

Sample Case 3 Java problem on Hardware Platform Linux

List SUN Software Product involved ( eg Application Server etc ): java version "1.5.0_12"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
Java HotSpot(TM) Server VM (build 1.5.0_12-b04, mixed mode)
List NON-SUN Third Party Software Product involved ( eg Oracle RDMS ) : Oracle 9.2.0
Hardware Platform:
OS and Kernel Version:
Linux : more /etc/redhat-release
Red Hat Entriprise Linux AS Release 3 ( Taroon Update 8)
Patch Level:
Linux : rpm -qa
Attachment (rmp_qa.txt) will be sent thru' E-mail to the case/ Service Request owner after receiving the mail.
Description of issue: JDK: 1.5.0_12 crashing with core file-need RCA.
Frequency of issue occurs: Random. Crashes 3/4 times a week.
When was the problem first noticed: 15th November 2007 after upgrade of JDK, HW & OS.
Recently Changed Variables: Just make sure that the system has been updated with the latest OS & Kernel patches.
Copy and paste error messages from server log outputs, if Any :
Steps to Reproduce Issue: Since this happens randomly. Test case will be provided along with sample application to reproduce this problem.

However the data below will help you to analyze the root cause of the crash:-

1). Core file
2). Library files associated with core
3). pstack, pmap & ldd outputs
ALL the files/ data Uploaded to http://supportuploads.sun.com/upload.
File name, cksum and the location of the file will be shared with the case/ Service Request owner thru' E-mail.

Sample Case 4 Web Server problem on Hardware Platform HP-UX

List SUN Software Product involved ( eg Application Server etc ): Sun Java System Web Server 6.1 SP4
List NON-SUN Third Party Software Product involved ( eg Oracle RDMS ) : Oracle 11i
Hardware Platform:
OS and Kernel Version:
HP-UX : uname -r (HPUX-11i)
HP-UX m_proxy B.11.11 U 9000/800 668369313 unlimited-user license
Patch Level:

• • HP-UX :swlist

swlist.txt will be sent thru' E-mail after receiving the mail from the case/ Service Request owner.

Description of issue: WebServer reaches hang state/ no response during load balancing.

We have two Application Servers at the back-end which are load-balanced through one Web Server at the front-end. We have setup the load-balancing successfully. Application Server A hosts two instances, whereas Application Server B hosts only one instance. Currently, the Application Server B is disabled from serving any request. Thus, the load is distributed to the two instances in Application Server A evenly now.

Frequency of issue occurs: Happens whenever the application is configured to use load-balancing feature.
When was the problem first noticed: 20th December 2007
Recently Changed Variables: None
Copy and paste error messages from server log outputs, if Any :
Steps to Reproduce Issue:

1). Install WebServer: 6.1 SP4
2). Install Application Server: SJSAS: 8.1-2005Q1
3). Configure load-balancing
4). Experience the problem

Recommended Data Collection for Crashes and Hangs in the Application Server ( Solaris )

Complete Application server version

<appserver_install-directory>/bin/asadmin version --verbose

• Tar of the config files

<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<config>/*
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*

Application Server Process Hangs/High CPU

Self-help

• For 9.x
  • Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
    eg
    Change
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
    to
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/>
  • Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html

Data Collection

In a multi-zone solaris 10 environment, please perform this data collection at the respective local zones

• prstat -L >> output.txt
• pstack pid > pstack.out same for pmap, pldd, pfiles, pflags
• Time stamp of the issue.
• server.logs that capture the problem occurrences
• Detail the load/number of concurrent users at that time of the problem.
• Does restarting the server resolve the issue?
• After a fresh restart of the server, for how long does it run without problems?
• For 7.x and 8.x
  • Issue kill -3 command for 3 times successively with an interval of a minute. This will create the java thread dump in the server.log file.
  • Please issue kill -3 pid command ONLY during the time of issue.
• For 9.x
  • Please use the command : asadmin generate-jvm-report --type=thread | tee threaddump.txt
  • Issue the command a few times over a period of time.
  • Please provide us the threaddump.txt containing the thread dump.

<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log
Or <appserver_9.1.x_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/jvm.log

• Run gcore on the pid. It will dump a core file of the process. Run the SUN Internal pkgcore script on this core file from the same system where core was generated and provide the generated files as listed below.
  ./pkgcore.sh corefile
  casenumber_libraries.tar.gz
  casenumber_corefile.tar.gz
  Please upload the above files in https://supportuploads.sun.com/upload
  under the cores directory and provide us the checksum details of the uploaded files.
• To track the memory usage provide the gclogs using the following Java Settings : -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:gclog.txt (JDK1.4.2.x and above)

• Alternatively, you can run the following data collection script from http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd_appserv_han when the application server goes into an unresponsive situation.

  Customers are advised to familiarise themselves with the script in a staging environment first before attempting in a production environment. For more information on the Sun Gathering Debug Data ( GDD ), please refer to the following website http://www.sun.com/service/gdd/index.xml.

Application Server Process Crash

• Check for core file, if there is no core file generated check the coreadm section as mentioned above.
• Check for any hs_err_pid.log. By default, the file is created in the working directory of the process
• When was the issue noticed first?(Please provide the exact time stamp of the issue). Does this issue happen during high load or any particular activity? Explain.
• How often it is occurring?
• Provide the server.log at the time of crash and the access log around the time of the issue

<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log

• Run the SUN Internal pkgcore script on this core file from the same system where core was generated and provide the generated files as listed below
  ./pkgcore.sh corefile
  casenumber_libraries.tar.gz
  casenumber_corefile.tar.gz
  Please upload the above files in https://supportuploads.sun.com/upload under the cores directory and provide us the checksum details of the uploaded files.

Recommended Data Collection for Crashes and Hangs in the Application Server ( Windows )

• Download Sysinternals Process Utilities - http://technet.microsoft.com/en-us/sysinternals/cb56073f-62a3-4ed8-9dd6-40c84cb9e2f5.aspx
  • Process Explorer
    • Enable Control-L ( View -> Lower Pane Panel )
    • Enable View -> Select Columns
      • Process Memory
        • Select All Except Private Bytes History
      • Handle
        • Select All
        • File -> Save as Handle.txt
      • DLL
        • Select All
        • File -> Save as Dll.txt

Complete Application server version

<appserver_install-directory>/bin/asadmin version --verbose

• Tar of the config files

<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*

• High CPU Application Server Process
  • Launch Process Explorer and locate the application process that indicates high cpu ( java.exe and appserv.exe )

• Right-click properties
• locate the tid of the offending process

• Click on tid ( in this case 3512 ) and click on button "Stack"

• Copy the content of the stack and paste it into a file
• Collect the Java Thread Stack dump using the instructions below steps.pdf

Application Server Process Hangs

Self-help

• For 9.x
  • Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
    eg
    Change
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
    to
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/>
  • Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html

Data Collection

• Collection of Java Thread Stack Dump ( a few sets of data over a period of time )
  • 7.x and 8.x

    This is a third-party tool : Run the following tool from adaptj - http://www.adaptj.com/main/download

    • Suggested instructions to run Adaptj 's stacktrace : steps.pdf

      Made sure Java option -Xrs is not enabled

  • 9.x
    • Run the command : asadmin generate-jvm-report --type=thread

Application Server Process Crash

• Check and sent any window dump file ( as a result of the application server crash )
• Check for any hs_err_pid.log. By default, the file is created in the working directory of the process
• When was the issue noticed first?(Please provide the exact time stamp of the issue). Does this issue happen during high load or any particular activity? Explain.
• How often it is occurring?
• Provide the server.log at the time of crash and the access log around the time of the issue

<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log

Recommended Data Collection for Crashes and Hangs in Application Server ( Linux )

Complete Application server version

<appserver_install-directory>/bin/asadmin version --verbose

• Tar of the config files

<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*

Application Server Process Hangs/High CPU

Self-help

• For 9.x
  • Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
    eg
    Change
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
    to
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/>
  • Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html

Data Collection

• top -bc > top.log or top -bc -p pid > top.log
• ps -L -o pid,tid,pcpu,state -p pid > ps.log [ locate those high readings under the %CPU column ]

  Have to perform together with a kill -3 pid. See below command

• Time stamp of the issue.
• server.logs that capture the problem occurrences
• Detail the load/number of concurrent users at that time of the problem.
• Does restarting the server resolve the issue?
• After a fresh restart of the server, for how long does it run without problems?
• For 7.x and 8.x
  • Issue kill -3 command for 3 times successively with an interval of a minute. This will create the java thread dump in the server.log file.
    Please issue kill -3 pid command ONLY during the time of issue.
    • For 9.x
      • Please use the command : asadmin generate-jvm-report --type=thread
      • Issue the command a few times over a period of time.
    • Please provide us the server.log containing the thread dump.

<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log
Or <appserver_9.1.x_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/jvm.log

Application Server Process Crash

• Check and sent any core file ( as a result of the application server crash )
• Check for any hs_err_pid.log. By default, the file is created in the working directory of the process
• When was the issue noticed first?(Please provide the exact time stamp of the issue). Does this issue happen during high load or any particular activity? Explain more.
• How often it is occurring?
• Provide the server.log at the time of crash and the access log around the time of the issue

<appserver_install-directory>/domains/<domain_name>/logs/server.log
<appserver_install-directory>/nodeagents/<nodeagent_name>/<instance_name>/logs/server.log

Additional data collection

• See disclaimer

Recommended Data Collection for Crashes and Hangs in Application Server ( AIX )

Complete Application server version

<appserver_install-directory>/bin/asadmin version --verbose

• Tar of the config files

<sjsas-install>/config/asenv.conf
<sjsas-install>/domains/<domain_name>/<instance-config>/*
For 8.x and 9.x
<sjas-install>/nodeagents/<nodeagent_name>/<instance_name>/config/*

Application Server Process Hangs/High CPU

Self-help

• For 9.x
  • Please ensure that the worker threads in the domain.xml is tuned. Default is 5.
    eg
    Change
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/>
    to
    <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="128" thread-increment="1"/>
  • Please refer to this Sun blog for more information http://weblogs.java.net/blog/jfarcand/archive/2007/03/configuring_gri_2.html

Data Collection

<pid> is the instance PID of the appserver

• To locate the pid
  ps auxwww | grep "com.sun.aas.instanceName=<name>"

• procstack <pid>
• procmap <pid>
• procfiles <pid>
• proctree <pid>
• ps -eaf
• netstat -an
• netstat -rn
• lsof -p <pid>
• ps -mp <pid> -o THREAD
• topas > topas.out
• truss -fealid -o truss.txt -p <pid>
• tprof -skex sleep 60

  This need root user to run. It takes 60 seconds to do a profiling of the appserver server to see how the application behaves so as to detect those high cpu threads

• Issue kill -3 command for 3 times successively with an interval of a minute. This will create the java thread dump in the file as described below.

  According to http://download.boulder.ibm.com/ibmdl/pub/software/dw/jdk/diagnosis/diag142.pdf. The dumped java dump is located under $INSTANCEDIR/config/javacore*

Application Server Process Crash

• Collect the following information from IBM 's website and engage IBM Tech support
• See http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21055387

Additional data collection

• IBM 's Must-Gather notes in http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21052641 for core files

Even though IBM JDK 6 has the following feature : jps and jstack. IBM JDK 6 is not supported in SGES21/AIX yet

Recommended Data Collection for HADB issue

• hadbm -V
• habdm get --all [hadb_name]
• habdm status [habd_name]
• habdm status --nodes [habd_name]
• hadbm deviceinfo --details [hadb_name]
• hadbm resourceinfo --databuf [habd_name]
• HADB history files from directory
• /etc/system

Recommended Data Collection for Crashes and Hangs in Web Server

Provide complete Web Server version

• WS6.0

<webserver_install-directory>/<instance directory>/start -version
# ./start -version
Sun Netscape Alliance
iPlanet-WebServer-Enterprise/6.0SP11 B12/15/2006 03:03

• WS6.1

<webserver_install-directory>/<instance directory>/start -version
# ./start -version
Sun Microsystems, Inc.
Sun ONE Web Server 6.1SP9 B01/11/2008 14:24

• WS7.0

<webserver_install-directory>/<instance directory>/bin/startserv --version
# ./startserv --version
Sun Microsystems, Inc.
Sun Java System Web Server 7.0U3 B06/16/2008 12:00

Note: on the Windows platform check the error logs after Web Server startup to find the specific version (also applicable on UNIX platforms).

• Follow the advice in the Sun Gathering Debug Data for Sun Java System Web Server Documentation
  • WS 6.0, 6.1 and 7.0
    • Web Server basic information
      • To Gather General Debug Data for Any Web Server Problem - http://docs.sun.com/app/docs/doc/820-2483/geaaj?a=view
    • Web Server fails to install
      • To Gather Debug Data on Web Server Installation Problems - http://docs.sun.com/app/docs/doc/820-2483/geaav?a=view
    • Web Server fails to startup
      • To Gather Debug Data on Web Server Startup Problem - http://docs.sun.com/app/docs/doc/820-2483/geabd?a=view
    • Web Server hangs or unresponsive
      • To Gather Debug Data on a Hung or Unresponsive Web Server Process - http://docs.sun.com/app/docs/doc/820-2483/gebbt?a=view
    • Web Server crashes
      • To Gather Debug Data on Web Server Crashed Process - http://docs.sun.com/app/docs/doc/820-2483/geaai?a=view

Studio Compilers and Tools Self-help

To obtain the version of the compiler or tool, type this command

$ /opt/SUNWspro/bin/version /opt/SUNWspro/bin/<product>

<product> being any one of the following

• cc - C compiler
• CC - C++ compiler
• f95 - Fortran compiler
• dbx - Source level debugger
• bcheck - Runtime checker
• collect - Performance data collector

Review the Support Matrix to ensure that your compiler or tool version conforms to its operating environment.

• Support Matrix - http://developers.sun.com/sunstudio/support/support_matrix.jsp

To obtain the current Java or operating environment versions, type these commands

$ java -version
$ uname -a

• Troubleshooting Studio Compilers and Tools
  • Installer Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-Installer
  • General Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-GeneralIssues
  • Studio IDE Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-ToolsIDE
  • Debugger Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-ToolsDbx
  • Compiler Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-Compilers
  • Linux Issues - http://wikis.sun.com/display/SunStudio/Sun+Studio+FAQ#SunStudioFAQ-PlatformSpecific

• Studio Compilers and Tools Resources
  • Developer Portal - http://developers.sun.com/sunstudio/
  • Documentation (docs.sun.com) - http://docs.sun.com/app/docs/prod/sun.stud
  • Technical Articles - http://wikis.sun.com/display/SunStudio/Sun+Studio+Technical+Articles+Collection
  • Studio Tools Forum - http://forums.sun.com/forum.jspa?forumID=852
  • C Compiler Forum - http://forums.sun.com/forum.jspa?forumID=849
  • C++ Compiler Forum - http://forums.sun.com/forum.jspa?forumID=850
  • Fortran Compiler Forum - http://forums.sun.com/forum.jspa?forumID=848
  • Studio Linux Forum - http://forums.sun.com/forum.jspa?forumID=855
  • Studio Blogs - http://developers.sun.com/sunstudio/community/blogs/index.jsp

Java Self-help

To obtain the java version, type this command:

# java -version

java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
Java HotSpot(TM) Server VM (build 1.5.0_09-b03, mixed mode)

• Troubleshooting Java SE 6
  • Troubleshooting Guides - http://java.sun.com/javase/6/webnotes/trouble/#guides
  • Quick Tips! - http://java.sun.com/javase/6/webnotes/trouble/other/matrix6-Unix.html
  • Troubleshooting Tools - http://java.sun.com/javase/6/webnotes/trouble/#tools
  • Pertinent Articles - http://java.sun.com/javase/6/webnotes/trouble/#articles
  • Pertinent Blogs - http://java.sun.com/javase/6/webnotes/trouble/#blogs
  • Pertinent Forum Sites - http://java.sun.com/javase/6/webnotes/trouble/#forums

Recommended Data Collection for Crashes and Hangs in Java (Solaris)

VM crashes

• collect Hotspot Error Logfile. It is written to the current working directory. It's name is e.g. hs_err_pid<pid>.log (where <pid> is the process id of process). In order to find out the current working directory you can use the /usr/bin/pwdx command on Solaris 8 onwards.
• To decode the hs_err log, please use http://blogs.sun.com/dave/resource/hs_err6.pl or use the web interface - http://openjdk.neojava.org/cgi-bin/hs_err.
  ( Please note: We cannot guarantee the availability or the support of these tools as they are not Sun products )
• Get the core file if any and run pkgcore - http://wikis.sun.com/download/attachments/47886215/pkgcore.sh

Recommended Data Collection for Crashes and Hangs in Java (Linux)

VM crashes

• collect Hotspot Error Logfile. It is written to the current working directory. It's name is e.g. hs_err_pid<pid>.log (where <pid> is the process id of process)
• To decode the hs_err log, please use http://blogs.sun.com/dave/resource/hs_err6.pl or use the web interface - http://openjdk.neojava.org/cgi-bin/hs_err.
  ( Please note: We cannot guarantee the availability or the support of these tools as they are not Sun products )
• Get the core file if any and run Pkg_App - http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/sun_gdd__all__pkg_a

Recommended Data Collection for Crashes and Hangs in Java (Windows)

VM crashes

• collect Hotspot Error Logfile. It is written to the current working directory. It's name is e.g. hs_err_pid<pid>.log (where <pid> is the process id of process).
• To decode the hs_err log, please use http://blogs.sun.com/dave/resource/hs_err6.pl or use the web interface - http://openjdk.neojava.org/cgi-bin/hs_err.
  ( Please note: We cannot guarantee the availability or the support of these tools as they are not Sun products )
• Get the windows logfile (Dr. Watson log or ADplus log) and a dump file (Dr. Watson user.dmp or an ADplus dump).
• Always use full crash dump type for Dr. Watson. See the following sample screenshot:

For details about Dr. Watson, please refer to Description of the Dr. Watson for Windows (Drwtsn32.exe) Tool - http://support.microsoft.com/kb/308538

Please also refer to our troubleshooting document 216912 on sunsolve for details - http://sunsolve.sun.com/search/document.do?assetkey=1-37-216912

• alternatively, you can use ADplus. Please refer to How to use ADPlus to troubleshoot "hangs" and "crashes" - http://support.microsoft.com/default.aspx?scid=kb;en-us;286350

In particular if you run the JVM as a "service", which is similar to a daemon, then ADplus is the only way. Choose these options:

adplus -crash -o <outputdir> -p <pid> -quiet -NoDumpOnFirst

Note: The option -NoDumpOnFirst is documented (run adplus -?). If the option is omitted, ADplus will create a minidump every minute, which will quickly fill up diskspace.

• Windows crash dumps for Java Processes - http://blogs.sun.com/poonam/date/20081019

• More on Windows Crash Dumps for Java Processes - http://blogs.sun.com/poonam/date/20081227

Recommended Data Collection for Java CAPS

When you open an incident with Java CAPS support through the web site, you get a template that you can fill in. If you send us an email, there isn't a template, but it's helpful to us if you can collect and provide this information. This section describes the information that we ask you to collect, and how to find it.

Java CAPS version

If you're using Java CAPS 5.x, this information is available in Enterprise Designer. Open the Help - About Sun Seebeyond Enterprise Designer dialog box.

If you're using Java CAPS 6 or higher, then you'll need to connect to the CAPS repository URL in a Browser. If you don't know the URL to use in the browser Address textbox you can find it in 5.x at the bottom right hand corner of your Enterprise Designer. For CAPS 6.x, navigate to Tools - CAPS Repository - Properties in the menus to open the CAPS Repository Properties dialog box, and note the values of the Repository host and repository port settings. Use these values to construct the URL:

http://repositoryhost:port

The default user name is Administrator, and the default password is STC.

Once you're logged in, you'll see a table with all the components that have been uploaded to the repository.

eWay Version (for the item selected above) if it is different from the above)

This may not apply to your issue. If you need it, you can find the version numbers of the components loaded in the repository by logging in to the Java Composite Application Platform Suite Installer using a browser.

If you don't know the URL to use in the browser Address textbox you can find it in 5.x at the bottom right hand corner of your Enterprise Designer. For CAPS 6.x, navigate to Tools - CAPS Repository - Properties in the menus to open the CAPS Repository Properties dialog box, and note the values of the Repository host and repository port settings. Use these values to construct the URL:

http://repositoryhost:port

The default user name is Administrator, and the default password is STC.

Once you're logged in, you'll see a table with all the components that have been uploaded to the repository.

Hardware platform and operating system with version information

Please make sure you know what Operating System the affected component is running on.

• For Windows, you can get version information from the System Properties dialog box. Right click on My Computer, and choose Properties. Note the Version and Service Pack (if any).
• For Unix, the flavor (Solaris, HP-UX, AIX) and the major version and minor version numbers, if applicable. The command

  uname -a
  

  will usually tell you what you need to know if you don't already know it.

• For Linux, please collect the distribution information. If you don't already know this, the most reliable and accurate source of the distribution can be got by executing the command

  cat /etc/*release
  

  from the command line.

Problem Description

Please provide as much information as you think is relevant: by all means be concise, but don't do so at the expense of facts.

Is this impacting a production system?

If you're losing the ability to process data, let us know. This helps us prioritize your issue.

What has recently changed in your environment?

If you are aware that there has been an external system configuration change recently, a migration, or a change of vendor, please let us know.

List of all ESRs and Hotfixes that have been applied to the system

It can be difficult to keep track of these. We recommend that you maintain a list of the ESRs and Hotfixes that you've applied.

Frequency that the issue occurs

If you're having a problem that occurs repeatably and predictably, then that's useful information.

Reproduction steps (if applicable)

Support may need to duplicate your issue in-house, particularly if there is going to be a fix. Quality Engineering will verify that fix does what we expect it to do, and can only do this if they have a procedure for recreating the issue and then verifying that the fix changes the behavior so that the issue no longer occurs.

Please try to keep the procedure as simple as possible. The more complex the steps are, the more likely it is that they will be misunderstood or somehow followed slightly wrong, and we will not be able to duplicate your results without repeating the process and refining it. This can delay resolution significantly.

Business impact

Again, this helps us to set the priority of the issue. It can also help justify the urgency of a fix.

Additional information

If available, please provide screenshots, log files, or any other information you think would be helpful to Support to expedite the troubleshooting process.