work with solaris contract subsystem

javasee

A process contract is the formal definition of the relationship thatexists between a Process A and its monitoring rocess. In case process Aterminates abnormally, the monitoring process will be able to restartit.
This is a small exercise that is using the contract subsystem and its associated commmands.
0. Get some information
# man contract
1. SMF and contracts
Check the state of the “syslog” service :
# svcs system-log
STATE          STIME    FMRI
online         Dec_10   svc:/system/system-log:default
#
# pgrep -fl syslog
  658 /usr/sbin/syslogd
# ps -o pid,comm,ctid | grep syslog
  658 /usr/sbin/syslogd       35                                                   
# svcs -pv system-log
STATE          NSTATE        STIME    CTID   FMRI
online         -             Dec_10       35 svc:/system/system-log:default
               Dec_10        658 syslogd
Allthese commands show that the service is enabled ( syslogd is startedwith pid 658 ) and is monitored by the contract subsystem usingContractID 35.
Another command shows more information about the contract :
#ctstat -i 35
CTID    ZONEID  TYPE    STATE   HOLDER  EVENTS  QTIME   NTIME
35      0       process owned   7       0       -       -      
Or in verbose mode :
#ctstat -vi 35
CTID    ZONEID  TYPE    STATE   HOLDER  EVENTS  QTIME   NTIME
35      0       process owned   7       0       -       -
        cookie:                0×20
        informative event set: none
        critical event set:    core signal hwerr empty
        fatal event set:       none
        parameter set:         inherit regent
        member processes:      658
        inherited contracts:   none
The HOLDER field indicates that the process holding the contract is having pid 7 :
# ps -fp 7
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root     7     1   0   Dec 10 ?           0:41 /lib/svc/bin/svc.startd
which is the main SMF daemon.  Let’s find out more about the contract held by this process. The /proc file system
can help there :
# cd /proc/7/contracts/
# ls
17  18  19  22  27  30  32  34  35  36  37  38  39  40  42  43  46  50 83
where we find out why ’svc.startd’, also knows as svc://system/svc/restarter:default is called the master
restarter in the SMF framework : it is holding contracts for many many services.
Comingback to the verbose output of the ctstat command, we see that someevents received by the holder of contract 35 are considered critical.Among them, “signal” is an event that indicates the reception of afatal signal from another process. The restarter has the job ofrestarting syslogd if one ‘critical’ event is received.
Let’s check that…
2. The master restarter
Before helping syslogd to die, let’s open another terminal and type
#ctwatch -rv 35
CTID    EVID    CRIT ACK CTTYPE   SUMMARY
which can be used to see all the events related to contract 35.
We can now use svcadm :
#svcadm refresh system-log
Nothing visible, a quick look at the Pid of syslogd tells us that it only reread its config file. We did the
famous “pkill -HUP syslogd” in the SMF way. Let’s try again :
#svcadm restart system-log

#svcs -pv system-log
STATE          NSTATE        STIME    CTID   FMRI
online         -             10:24:37     89 svc:/system/system-log:default
               10:24:37     2173 syslogd
while the other “ctwatch” terminal shows :
#ctwatch -rv 35
CTID    EVID    CRIT ACK CTTYPE   SUMMARY
35      33      crit no  process  contract empty
Whathappend ? The syslogd process was terminated by svcadm. The contractbeing linked to the process is then also terminated and is ‘empty’.While starting another syslogd, another contract, number 89, wascreated.
Now in the “ctwatch” terminal, let’s type
#ctwatch -rv 89
CTID    EVID    CRIT ACK CTTYPE   SUMMARY
while in the other window, we kill the syslog daemon (only in the global zone) :
#pkill -9 -z 0 syslogd
We observe in the “ctwatch terminal” :
#ctwatch -rv 89
CTID    EVID    CRIT ACK CTTYPE   SUMMARY
89      34      crit no  process  process 2173 received a fatal signal
        signal: 9 (SIGKILL)
        sender pid: 2187
        sender ctid: 86
89      35      crit no  process  contract empty
Whichshows that the contract subsystem was notified that the syslogd processreceived a signal. We even may know who sent it. In our case, it is the“kill” command, which terminated already. The result of the signal isthat
contract is now empty, ended.
But :
# svcs -pv system-log
STATE          NSTATE        STIME    CTID   FMRI
online         -             10:27:04     90 svc:/system/system-log:default
               10:27:04     2193 syslogd
shows that the master restarter has done its job. The system-log service is still online because “svc.startd” has
instructed “init” to fork and exec a new version of syslogd to keep the service running.
3. The contract file system
Everyoneknows the /proc filesystem, used to provide information to systemadministrators about the running processes in a nice well-knownfile-based manner. Commands like “ps”, “pfile”, “pgrep”,… get theirinformation by opening and reading files in /proc which are actually aninterface to the process structures maintained by the kernel.
Thesame is true for ctfs, the contract file system. All the contractcommands get their input from the kernel through another pseudofilesystem mounted on /system/contract.
Example :
# truss -t open ctstat -i 35
open(”/var/ld/ld.config”, O_RDONLY)             Err#2 ENOENT
open(”/lib/libcontract.so.1″, O_RDONLY)         = 3
open(”/lib/libuutil.so.1″, O_RDONLY)            = 3
open(”/lib/libc.so.1″, O_RDONLY)                = 3
open(”/lib/libnvpair.so.1″, O_RDONLY)           = 3
open(”/lib/libnsl.so.1″, O_RDONLY)              = 3
open(”/platform/SUNW,Sun-Blade-100/lib/libc_psr.so.1″, O_RDONLY) = 3
open(”/usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.3″, O_RDONLY) = 3
CTID    ZONEID  TYPE    STATE   HOLDER  EVENTS  QTIME   NTIME
open64(”/system/contract/all/35/status”, O_RDONLY) = 3
35      0       process owned   7       0       -       -      
4. Monitor and restart ANY application
Thecommand ‘ctrun’ can be used to create a contract for any application.It will then monitor the application process for all the events thatyou specify and restart the application if some fatal event occurs.Check it out!
#ctrun -r 0 -o noorphan -f signal /usr/openwin/bin/xclock &
Myxclock will be restarted any number of times ( -r 0 ), ctrun will makesure that all processes get killed before restarting ( -o noorphan )and we monitor the ’signal’ type of fatal event.
You get a contractid and so on.
Then just kill the clock and see it nicely reappear…
1  What is the contract file system?

From contract(4): " The /system/contract file system acts as the primary  interface  to  the contract subsystem."
From learning solaris: "A process contract is the formal definition of the relationship that exists between a Process A and its monitoring process."
In my own words : "The contract file system sets up a framework in the kernel that enables real time monitoring of processes and enables us to act on changes in process states immediately."
2  How to use the contract file system>

The contract file system is primarily used by the SMF(5) framework when services are started. The service restarter uses contracts to keep track of the processes in a contract.
It is also possible for applications and users to create contracts on the fly. The following commands are used by users to work with and manage contracts:

• ctstat
  
• ctrun
  
• ctwatch
  
• /system/contract
  

3  Examine your system

The contract framework is implemented as a pseudo file system.At system boot the contract file system is mounted at/system/boot.
www2.petervg.nl # df -n /system/contract/system/contract   : ctfs    www2.petervg.nl # grep ctfs /etc/vfstabctfs            -       /system/contract        ctfs    -       no      -www2.petervg.nl # ls /system/contract//all22260  22362  6063   6064   6065   6066   6073   6078   6079   6083   6090   6091   6093   6094   6095   6096   6097   6098
4  Create our own contract

With the ctrun command we can run a command and place it in a new contract.
4.1  New process without a new contract

www2.petervg.nl # ptree -c $$14536 zsched  [process contract 22362]    3168  /usr/lib/ssh/sshd      3171  /usr/lib/ssh/sshd        3173  -bash          3265  sleep 555          3279  ptree -c 3173www2.petervg.nl # sleep 555 &[2] 3280www2.petervg.nl # ptree -c $$14536 zsched  [process contract 22362]    3168  /usr/lib/ssh/sshd      3171  /usr/lib/ssh/sshd        3173  -bash          3280  sleep 555          3281  ptree -c 3173
As we can see, the sleep 555 commandis running in the same contract (22362) as the parent bash shell.
4.2  New command with a new contract

www2.petervg.nl # ptree -c $$14536 zsched  [process contract 22362]    3168  /usr/lib/ssh/sshd      3171  /usr/lib/ssh/sshd        3173  -bash          3300  ptree -c 3173www2.petervg.nl # ctrun sleep 666 &[1] 3301www2.petervg.nl # ptree -c $$14536 zsched  [process contract 22362]    3168  /usr/lib/ssh/sshd      3171  /usr/lib/ssh/sshd        3173  -bash          3301  ctrun sleep 666            [process contract 22366]              3302  sleep 666          3303  ptree -c 3173
And in this example we see that the sleep 666command is actually run under a new contract (22366).
But does this contract automagically restart the sleep process whenit is killed? The answer is no. By a contract will not restart theprocesses in a contract.
5  Checking the status of a contract

When a contract is started we can use the ctstat command to get informationabout the contract. The ctwatch command can then be usedto monitor the use of the contract over time.
www2.petervg.nl # ps -o ctid,pid,ppid,args | grep sleep22362  3577  3173 ctrun sleep 55522385  3578  3577 sleep 55522362  3599  3591 grep sleep
With the ps -o ctid command it ispossible to identify the contract a particular command is runningin.
www2.petervg.nl # ptree -c $$14536 zsched  [process contract 22362]    3168  /usr/lib/ssh/sshd      3171  /usr/lib/ssh/sshd        3173  -bash          3591  ksh            4323  ctrun sleep 666              [process contract 22449]                4324  sleep 666            4336  ptree -c 3591www2.petervg.nl # ctstat -i 22449CTID    ZONEID  TYPE    STATE   HOLDER  EVENTS  QTIME   NTIME   22449   12      process owned   4323    0       -       -      
In the previous output we see that the contract under investigationis contract 22449 which is running in zone 12. The current contractis a process contract which is held by process 4323. This meansthat process 4323 created the contract and started the process inthis contract.
With the -v option we can obtain more verbose information abouta specific contract.
www2.petervg.nl # ctstat -v -i 22452CTID    ZONEID  TYPE    STATE   HOLDER  EVENTS  QTIME   NTIME   22452   12      process owned   4408    0       -       -               cookie:                0        informative event set: core        critical event set:    hwerr empty        fatal event set:       hwerr        parameter set:         none        member processes:      4409        inherited contracts:   none