Share

HUNG PROCESSES In Siebel Servers

Health of these siebel server processess play major role to keep the application and other operational functionalities smooth and fine. “Process” is a collection of many “Tasks” or “Threads“. All these threads are defined within a process, under a specified parameter within the siebel server for different functionalities.



Example: MaxSharedDbConns , MinSharedDbConns
, MinMTServers ,MaxMTServers parameters for AOM component “eCommunicationsObjMgr_enu”.

 Thus in case if these Threads/Tasks within a Process gets hanged or show the error such as:

 

  • Handling logon
  • Excited with Error
  • Running status but for more than 1 to 2 hours or greater.  

 

Hint-1: (command to check the threads status. Example: “list tasks for comp eCommunicationsObjMgr_enu” or “list active task for comp eCommunicationsObjMgr_enu “).

Hint-2: All Threads within a process might get hanged : one by one-gradually : or all of a sudden. Example: out of 100 threads defined under one process of “siebmtshmw” of AOM component, 60 might get hanged ,but application still keeps working with the remaining 40 threads but once all 100 threads gets hanged then, the application will stop opening for that process.

Hence, many times it has been observed that, these processess are getting Hanged, due to various reasons like:

·                     DB Blockings.

·                     Table Locking.

·                     Constant Concurrency in the Database.

·                     Network spikes due to packets drop.

·                     Web server to Siebel server: SISNAPI connection failure.

·                     Siebel server to Database server internal connection failure.

·                     ODBC connection failure.

·                     Gateway server performance issues. Thereby hitting “siebns.dat” file which contains all siebel server component information.

·                     Adhoc Queries hitting the DB nodes.

·                     Unix/solaris /windows OS crash.

·                     Index Rebuilding activities in Database tables during live application status.

·                     Stats gathering of Database Tables during live application status.

·                     Different EAI integrations within the siebel applications and their scripting change.

·                     EAI XML errors.

·                     High CPU and Memory Utilization of the siebel servers/spikes.

·                     High CPU and Memory Utilization of the Database servers/spikes.

 Therefore, whenever there is a degradation of the application performance happening, then very first thing which can be checked and identified are the “Hanged processess”, and once it is found that many processess have hanged together, then the best way to avoid degradation and abnormalcy in server is to take bounce of the services of the server, in order to achieve normalcy, because bounce will regenerate these processess again by killing all the hanged threads.

Note: In Multithreaded Servers/components, the hanged threads/tasks cannot be killed,directly using the kill command.(the only solution is to take the bounce of the services of server.)

Thus, all the hanged processess of the server can be achieved through a script given below:

Sample Script :

for i in `grep eCommunicationsObjMgr_enu /siebapp/siebel/siebsrvr/enterprises/APPLICATION_PROD/*/log/ APPLICATION _PROD.*.log | grep ‘Created server process’ |  awk ‘{print $13}’`;

do

DT=`date ‘+%Y/%m/%d %H:%M:%S’`;

H=`hostname`;

PS=`ps -eafo pid,vsz,rss,pcpu,nlwp,user,args | grep $i | grep -v grep`;

echo $H $DT $i $PS ;

done | grep siebmtshmw | more

   Here APPLICATION_PROD is Enterprise Name. 

Thus the above script can be configured within the siebel server and can be executed in any path within the siebel server, in order to get “All the Siebel Application Processes running within the server”.  

Now there are some steps which is required to be followed further, in order to analyze which process has got hanged,out of many process listed ,after executing the above script :

Example:

The Red marked box in the above screenshot contains ,all the Threads running within each Process

And each process has a Thread Limit defined in the server parameter: Suppose if the Thread limit defined for a process is 100 then we will analyze the hanging threads as per the 100 count declared.

In case if the Threads count is showing greater than 100 like (120/140/190/200) for a particular siebmtshmw process, then it means that Process might have hanged. 


 If the Thread count is showing single digit value, then also it might be concluded that the Process has hanged.

But still there are some more analyzation required to conclude that a process has completely hanged:   



  

If the thread count seen is 80 to 120+ ,then it is always not mandatory that the process has hanged.

1. We need to check the status of the same thread count, within 5 to 10 minutes time gap ,if the status of the thread or the count remains the same even after consecutive time periods:

2. Then it can be concluded that the respective process has hanged because after consecutive time periods the thread counts might show deviation in the count thereby releasing memory after successful execution.

We also need to match the given :process ids greped in the server(OS)  ,with that of the enterprise log,as per the given script.

1. Both the process ids should remain the same in both the cases.

2. In case if the process id generated in server level (prstsat) is not matching the process id generated in enterprise log(APPLICATION_PROD) or if  a particular process id is present in server but not present in enterprise log(APPLICATION_PROD) ,then it implies that there is some issue with that process.

 

 

 

   

7 comments on “HUNG PROCESSES In Siebel Servers”

  1. abhishek kumar Reply

     
    Excited with Error
    Excited with Error  –> It means the task is not hanged and has got finished with errors and it does not consumes any memory and hence won't play much role in performance degradation.

  2. sourav mukherjee sourav mukherjee Reply

    exactly, but sometimes many no. of tasks/threads, showing the same error "excited with error",
    for a longer time , or if any new task started ,goes to the status of excited with error,without processing the request,in a very short span of time,definitely brings to a conclusion of performance  degradation.

  3. anushka sharma Reply

    superb document…
    please keep publishing ,such tricks and techniques of improving siebel application performance

  4. Nitin Reply

    Must Say "Really a superb article"
    But is this command okay ? I mean I got error when I tried to run this :
    list active task for comp <comp_name>
    however below command was succesful
    list active session for login 60030303
    Also please let me know the command to list the hung session or task in srvrmgr prompt ?

  5. Rohit verma Reply

    @nitin-  are you a complete fresher in siebel!!!!!
    your questions doesnt make any sense

  6. abhishek kumar Reply

    Hi Nitin,
    What error are you getting when you use below command:
    list active task for comp <comp_name>
    list hung session for comp <comp_name> will give you the hung session list.

  7. blaster Reply

    Hi, i found your article informational. however, i tried to run the script for hung process but it's not working giving error which i dont understand since i am new in unix. can you send me in my email instead the script so i will run it exectly. thank you very much in advance. hope to received your response asap.

Leave A Reply

Your email address will not be published. Required fields are marked *

error: Content is protected !!