9.1 MQ

MQ General Data Collection

  • When problems occur it is useful to check the version of the
    installed Message Queue and OS Version by issuing
    #./imqcmd -version and uname-a
  • Whether SJSMQ is a stand-alone MQ (shipped with OS) or part of Appserver infrastructure
    • MQ is included with
    • GlassFish / Sun Java System Application Server
    • With Java Enterprise System
    • Sun Solaris
    • Available with Java Composite Application Platform Suite (Java CAPS)
  • Timestamp of the issue
  • Has the problem occurred before? If so, when and how frequent the problem happens?
  • Is it a MQ cluster ? If so, get the required configuration and logs from all the machines
  • Please get the states of all the brokers on the cluster
    • imqcmd query bkr –u admin –b $host:$port
    • imqcmd list dst –u admin –b $host:$port
    • imqcmd list txn –u admin –b $host:$port
  • Common Files and locations
    • instance_name - name of the running broker (passed in on the command line), default is imqbrokerd
    • IMQ_ROOT - where MQ is installed
    • bin location - where the executables are located:
      • Solaris (package based) - /usr/bin
      • Linux (rpm based) - /opt/sun/mq/bin
      • Windows - <IMQ_ROOT>/bin
      • Application Server zip based install: <AS_install_location>/imq/bin
    • instance_home - based directory where logs, messages and configuration is stored:
      • Solaris (package based) - /var/imq/instances/<instance_name>
      • Linux (rpm based) - /var/opt/sun/mq/instances/<instance_name>
      • Win32 - <IMQ_ROOT>/var
      • Application Server zip based install: <AS_install_location>/imq/var
  • Broker dump
    • imqcmd dump bkr -debug -o file=/remotedir/bkr-debug.txt
      Note that this will dump the info on the broker directory
      under /remotedir (on the broker host) You need to get them physically
This command may have some performance impact. So only do it when necessary (when the broker have issue).
Note that you may need the OS tuned by file limit otherwise it may cause broker issues.
(eg: ulimit's)

Platform specific location of Message Queue data (broker config and log file)

Solaris

/var/imq/instances/instanceName/props/config.properties (Broker instance configuration properties)
/var/imq/instances/instanceName/log/ (Broker instance log file directory default location)

Linux

/var/opt/sun/mq/instances/instanceName/props/config.properties (Broker instance configuration properties)
/var/opt/sun/mq/instances/instanceName/log/ (Broker instance log file directory -default location)

Windows

IMQ_VARHOME\instances\instanceName\props\config.properties Broker instance configuration properties
IMQ_VARHOME\instances\instanceName\log\ Broker instance log file directory (default location)

Enabling MQ broker in DEBUG mode,Change in config.properties

  • imq.log.level=DEBUGHIGH
  • Or start mq with –loglevel DEBUGHIGH
  • This enables a whole lot of information to the MQlogs (log.txt)

Starting MQ

  • imqbrokerd –tty –loglevel DEBUG -name $name –port $port
    (or check /etc/init.d/imq start) using /etc/imq/imqbrokerd.conf

Stopping MQ

  • imqcmd shutdown bkr –b $host:$port –u admin

Restarting MQ (reread config)

  • imqcmd restart bkr –b $host:$port –u admin

Pause/Resume MQ

  • imqcmd pause bkr –b $host:$port –u admin
  • imqcmd resume bkr –b $host:$port –u admin

Querying broker state

  • imqcmd query bkr –u admin –b $host:$port
  • imqcmd list dst –u admin –b $host:$port
  • imqcmd list txn –u admin –b $host:$port

Problem category

  • MQ broker Unresponsive.
  • MQ broker run out of memory.
  • MQ broker High CPU.
  • MQ Broker cluster hang

Recommeded data collection Unresponsive Broker

  • Message Queue version ./imqcmd -version and OS Version uname-a
  • At the time of the problem, is the faulty broker running or not running?
  • Has the problem occurred before? If so, when?
  • Provide brief info on the setup like using the MQ for (Queue, topics, Durable consumer etc etc
    and how many typical clients),What are the clients to the broker? Are they code written by the customer?
  • What is the exact nature of the non-responsiveness of the broker? Can the client establish a network connection? Can
    the client send a message? Can the client receive a message?
  • How many clients are active at the time the problem occurs?
  • Does the failure affect both the brokers in the cluster, or just one? If only one, is it always the same one?
  • When the broker stops responding, Collect multiple threaddump (do this by kill -3 on the broker's JVM – the results
    should appear in the broker logfile).
    kill -3 $java_mq_process
    kill -3 $java_mq_process
    kill -3 $java_mq_process
    NOTE: Threaddump should be taken during the period of non-responsiveness
  • When the broker is `non responsive', does it respond to `imqcmd' requests(to query the broker.)?
    • imqcmd query bkr –b brokerhost:brokerport
telnet localhost 17676
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
101 testbroker 3.6
portmapper tcp PORTMAPPER 17676
admin tcp ADMIN 50872
jms tcp NORMAL 50871
cluster tcp CLUSTER 50873
  • Provide broker instance configuration properties and Broker instance log file specific to the platform.
  • When the broker is `non responsive', does it respond to `imqcmd' requests(to query the broker.)?

MQ broker run out of memory

  • Message Queue version ./imqcmd -version and OS Version uname-a
  • Timestamp when the issue was noticed
    date > timeoccur.txt
  • Provide broker instance configuration properties and Broker instance log file specific to the platform.
  • Enable and track memory usage (JVM options, jstat,jinfo, prstat, pmap)
    ($java_mq_process ==The java process that started the MQ)
    pstack $java_mq_process) > pstack_1.txt
    pstack $java_mq_process) > pstack_2.txt
    pstack $java_mq_process) > pstack_3.txt
    pmap $java_mq_process > pmap.txt
    prstat -L -v -p $java_mq_process > prstat.txt
    On Java 5, use jstack from Java 5 JAVA_HOME/bin/jstack
    jstack $java_mq_process > jstack.txt
    jmap -heap $java_mq_process > jmapheap.txt
  • To change the minimum and maximum heap size for the broker, use the -vmargs
    command line option when starting the broker.
    /usr/bin/imqbrokerd -vmargs "-Xms256m -Xmx1024m"
    will set the starting Java heap size to 256MB and the maximum Java heap size to 1GB.
    stop the imqbrokerd and start it again.
  • On Solaris or Linux, if starting the broker via /etc/rc* (that is, /etc/init.d
    /imq), specify broker command line arguments in the file
    • Solaris
      /etc/imq/imqbrokerd.conf
    • Linux
      /etc/opt/sun/mq/imqbrokerd.conf

Display broker metrics( JVM Heap Bytes )

  • /opt/sun/mq/bin/imqcmd metrics bkr -m cxn -b host:port

MQ broker High CPU

  • Message Queue version ./imqcmd -version and OS Version uname-a
  • Timestamp when the issue was noticed and let us know if any config changes made.
    date > timeoccur.txt
  • Collect prstat -L -v -p $java_mq_process > prstat.txt ($java_mq_process ==The java process that started the MQ)
  • When the broker consumes HIGH CPU,Collect multiple threaddump (do this by kill -3 on the broker's JVM – the results
    should appear in the broker logfile).
    kill -3 $java_mq_process
    kill -3 $java_mq_process
    kill -3 $java_mq_process
    NOTE: Threaddump should be taken during the period of HIGH-CPU
    pstack $java_mq_process) > pstack_1.txt
    pstack $java_mq_process) > pstack_2.txt
    pstack $java_mq_process) > pstack_3.txt
  • Provide broker instance configuration properties and Broker instance log file specific to the platform.

MQ Broker hang in a Cluster(multi-broker)

  • Message Queue version ./imqcmd -version and OS Version uname-a
  • Timestamp when the issue was noticed and let us know if any config changes made.
    date > timeoccur.txt
  • Provide brief info on the setup like using the MQ for (Queue, topics, Durable consumer etc etc
    and how many typical clients),What are the clients to the broker?
  • Since there is 2 MQ,Does the broker respond to `imqcmd' requests on both ports
    imqcmd query bkr –b brokerhost:brokerport
  • Enable -loglevel debug on MQ
    imqbrokerd –tty –loglevel DEBUG -name $name –port $port
    and when the problem happens get also the kill -3 of broker's JVM –
    the results should appear in the broker logfile.
    kill -3 $java_mq_process
    kill -3 $java_mq_process
    kill -3 $java_mq_process
  • Provide broker instance configuration properties and Broker instance log file specific to the platform.
  • If possible provide testcase description to test in-house ,like what needs to be setup and the Queue/Topics that is needed and how this is started -

Labels

kmessagequeue kmessagequeue Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Sun Microsystems community and they might not be employed or in any way formally affiliated with Sun Microsystems. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Sun nor any other party necessarily agrees with them.

Copyright 1994-2009 Sun Microsystems, Inc.
Powered by Atlassian Confluence Sun Guidelines on Public Discourse Privacy Policy Terms of Use Trademarks Site Map Employment Investor Relations Contact