Sun Identity Manager

 No account was found on a resource assigned to the user (but
 Lighthouse did not really expect the account to exist).

 Did NOT find an account we expected to find on the resource. (and
 Lighthouse was sure that the account existed).

1. Manufacturer and EXACT version information about the JDK in use.  eg.

   # java --version
   java version "1.4.2"
   gij (GNU libgcj) version 4.1.0 (Ubuntu 4.1.0-1ubuntu8)

2. The bit mode of the operating system, JDK, and app server software (eg. have them verify with no uncertainty
   whether they are running in 32-bit or 64-bit mode for each piece of the puzzle).

3. OS Platform (eg. Windows 2003, AIX, Linux, Solaris, etc.)

4. JVM Environmental settings: Xms (minimum java heap), Xmx (max heap), gc (garbage collection method/mode and
   intervals - different JDKs have different command-line options for configuring the garbage collector and different
   options as to how and when the GC operates, it is critical to understand how they have configured this...their app
   server team should know)

5. Heap dumps and thread dumps of the running IDM jvm while an issue is being encountered:
    - You can have the customer execute a "kill -3" on the running java process to capture a thread dump.
    - Websphere has settings where you can configure the JVM to dump a copy of the java heap whenever an OOM condition
      is encountered, have them turn this on.  You will need to download the HP HeapAnalyzer tool to view the PHD
      (portable heap dump) files, but they can be helpful in certain OOM conditions.
    - Some JDKs have options to grab memory histograms/heap dumps from the running process while it remains alive.
      Check to see if you can gather one of those.

6. All the usual IDM tracing and logs.

1. Find the JDK's diagnostic guide on the internet so that you can read up on the memory management model used by
   that JDK version on that platform.  IBM's Diagnostic Guides for all versions can be found here:

   http://www.ibm.com/developerworks/java/jdk/diagnosis

   If you are looking for Jrockit's guides, look here:

   http://edocs.bea.com/jrockit/geninfo/diagnos/index.html

2. Use those guides to recommend some debugging steps to the customer so they can turn on extra tracing or logging
   from the JVM side, or if there is an actual error being thrown, you might find an explanation of the error in
   those guides.

3. See if their max heap is set high enough to handle the job, a decent minimum for our product is 1Gig.  Also, some
   customers like to set the minimum heap to the same size as the max....be careful in doing this.  This can negatively
   impact garbage collection in some JDKs.  I have particularly seen Jrockit do weird things if the max and min heap
   are set to the same number.  A safe minimum heap is 512M.

4. Don't be afraid to ask them to bump their max heap up to 2Gigs if you are seeing OOMs (Unless you are on the
   IBM 32-bit 1.4.2 JDK for AIX, see below).

5. Look through the logs and traces for app server/JDK error codes that might be getting thrown. Each JDK has it's
   own unique set of error codes that it will throw upon certain conditions and the diagnostic guides and/or Google
   searches will reveal lots of good information about the root cause if you can find these error codes.  Several
   performance related issues I've debugged have come down to IBM or BEA patches that needed to be installed.
   Don't be afraid to direct the customer to their other software vendors if the issue resides in that 3rd party product.

1. Every JDK handles memory allocation and processes differently.

2. Most JDKs (besides the Sun JDK) require contiguous memory space to be available before they will store objects
   in memory.

3. If a JDK that requires contiguous memory space encounters a situation where it needs to store a java object of
   the size 50M, but it can only find a slot that is 45M in size, it will throw an OOM exception and IDM will
   crash regardless of the fact that only 80 or 90% of the total heap space is in use.

4. Garbage collection is critical for these kinds of JDKs because space needs to be reclaimed and consolidated so
   that new objects have sufficient contiguous blocks of memory to operate in.

5. Each version of JDK, on each platform, in each bit mode has different nuances to be aware of, so no blanket memory
   advice is ever accurate.  Be aware of the specific details of the JDK you are dealing with (as found in the diagnostic
   guides) before you make recommendations.  For example, the IBM JDK on AIX in 32-bit mode and version 1.4.2 can only
   address 2 Gigs of memory and separates that memory into java and native spaces.  If you set the max heap (-Xmx)
   setting to 1024M, then native operations (threads, sockets, etc) only have 1 gig to operate in.  If you bump up the
   max heap to 2 Gig, then native heap has zero space to work with and has to contend with other applications to get
   space in the 500 meg shared memory pool. The IBM 1.4.2 32-bit JDK on the Windows platform has no such limitations.
   Neither does the 64-bit AIX version.  Each JDK is different....know the differences.

6. Educate yourself about the way our product uses memory (and understand that it was written for and optimized in the
   Sun JDK environment), and don't be afraid to insist that the customer try different settings until the performance
   issue is tweaked away or there is at least a satisfactory workaround found.

1. Prior to running the Reconciliation, navigate to "IDM_INSTALL/idm/debug/Show_Memory.jsp"
2. Click on the "Garbage Collect" tab
3. From the browser menu, navigate to:
   a. File\Save Page As
   b. Save the page as "BeforeReconShowMem.htm"
4. Navigate to "IDM_INSTALL/idm/debug/Show_Threads.jsp"
5. Perform the same steps in step 3.), only this time name the saved page as "BeforeReconShowThreads.htm"
6. Initiate the Reconciliation
7. When the Reconciliation "hangs", perform steps 1.) - 5.) only this time name both the files with the prefix 
   "After" in place of the "Before"
8. Obtain the 4 files from the Cu

1. From the command line cd to [IDMBASEINSTALL]/
2. Execute the following:

     lh console -c "export -v default.xml default"

1) cd $WSHOME/bin
lh console -u <user>  
lh console -u <user> -p <password file>

e.g. lh console -u myconfig -p passwd.txt 

myconfig> version
    Sun Java System Identity Manager 7.1 (20080926 Update 1 Patch 7)

2) Login to IDM. Draw mouse pointer on "Help" in the top right corner.

(a) Navigate to the app server console (JVM term/cmd window used to launch IDM)
(b) If the this is Windows environment, perform a  <ctrl>-<break>; or if it is a linux/UNIX platform perform a 
    "kill -3" to the correct PID (Request assistance from your OS administrator[s] if necessary)
(c) Repeat steps (a) and (b) every 30 seconds to a minute,  for about 5 min straight
(d) Copy the output resulted from the procedure and paste it in a .txt or .doc