yum -y install strace

[root@node01 ~]# strace  strace: must have PROG [ARGS] or -p PID  Try 'strace -h' for more information.

In  the  simplest  case  strace  runs  the  specified command until it exits.  It intercepts and records the system calls which are called by a process and the signals which are  received by a process.  The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.    在最簡單的情況下，strace運行指定的命令，直到它退出。它攔截並記錄由進程調用的系統調用和由進程接收的信號。每個系統調用的名稱、參數及其返回值都打印在標準錯誤上，或者打印到使用-o選項指定的文件中。  strace is a useful diagnostic, instructional, and debugging tool.  System administrators, diagnosticians and trouble-shooters will find it invaluable for solving  problems  with  programs for which the source is not readily available since they do not need to be recompiled in order to trace them.  Students, hackers and the overly-curious will find that a  great deal can be learned about a system and its system calls by tracing even ordinary programs.  And programmers will find that since system calls and signals are  events  that  happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.    strace是一個有用的診斷、指導和調試工具。系統管理員、診斷人員和故障排除人員將發現它對於解決程序的問題非常有價值，而這些程序的源代碼並不容易獲得，因為不需要重新編譯來跟蹤它們。  學生、黑客和過分好奇的人會發現，通過跟蹤一個系統及其系統調用，甚至是普通的程序，都可以了解到很多東西。程序員會發現，由於系統調用和信號是發生在用戶/內核接口上的事件，所以應該仔細檢查一下。  Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value.  An example from stracing the command "cat /dev/null" is:  跟蹤中的每一行都包含系統調用名，後面是括號中的參數和返回值。例如，命令“cat /dev/null”為:  open("/dev/null", O_RDONLY) = 3  也就是說， open是系統調用名，"/dev/null", O_RDONLY是傳入的參數，返回值是3， 查看open的man手冊(http://man7.org/linux/man-pages/man2/open.2.html)，在RETURN VALUES描述中，說道返回 一個文件描述符或者是錯誤碼 -1  Errors (typically a return value of -1) have the errno symbol and error string appended.    一個文件不存在的一個栗子：    open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)        Signals are printed as signal symbol and decoded siginfo structure. An excerpt from stracing and interrupting the command "sleep 666" is:    信號以信號符號的形式輸出，並解碼siginfo結構。以下是“sleep 666”命令的一段摘錄:    sigsuspend([] <unfinished ...>  --- SIGINT {si_signo=SIGINT, si_code=SI_USER, si_pid=...} ---  +++ killed by SIGINT +++    If a system call is being executed and meanwhile another one is being called from a different thread/process then strace will try to preserve the order of those events and mark  the ongoing call as being unfinished. When the call returns it will be marked as resumed.  如果正在執行一個系統調用，同時從另一個線程/進程調用另一個系統調用，那麼strace將嘗試保留這些事件的順序，並將正在進行的調用標記為未完成。當調用返回時，它將被標記為已恢復。  [pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>  [pid 28779] clock_gettime(CLOCK_REALTIME, {1130322148, 939977000}) = 0  [pid 28772] <... select resumed> ) = 1 (in [3])    Interruption of a (restartable) system call by a signal delivery is processed differently as kernel terminates the system call and also arranges its immediate reexecution after  the signal handler completes.  當內核終止系統調用並在信號處理程序完成後安排它的立即重新執行時，信號傳遞對(可重新啟動的)系統調用的中斷以不同的方式進行處理。    read(0, 0x7ffff72cf5cf, 1) = ? ERESTARTSYS (To be restarted)  --- SIGALRM ... ---  rt_sigreturn(0xe) = 0  read(0, "", 1) = 0    Arguments are printed in symbolic form with a passion. This example shows the shell performing ">>xyzzy" output redirection:  參數是以象徵性的形式印出來的。這個例子顯示了執行“>>xyzzy”輸出重定向的shell:  open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3    Here the third argument of open is decoded by breaking down the flag argument into its three bitwise-OR constituents and printing the mode value in octal by tradition. Where  traditional or native usage differs from ANSI or POSIX, the latter forms are preferred. In some cases, strace output has proven to be more readable than the source.  在這裡，open的第三個參數通過將flag參數分解成它的三個bit - or組成部分並按傳統打印八進制的模式值來解碼。當傳統的或本地的用法與ANSI或POSIX不同時，最好使用後者。在某些情況下，strace輸出被證明比源代碼更具可讀性。  Structure pointers are dereferenced and the members are displayed as appropriate. In all cases arguments are formatted in the most C-like fashion possible. For example, the  essence of the command "ls -l /dev/null" is captured as:  結構指針被取消引用，成員被適當地顯示。在所有情況下，參數的格式都儘可能類似於c。例如，“ls -l /dev/null”命令被捕獲為:  lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0    Notice how the 'struct stat' argument is dereferenced and how each member is displayed symbolically. In particular, observe how the st_mode member is carefully decoded into a  bitwise-OR of symbolic and numeric values. Also notice in this example that the first argument to lstat is an input to the system call and the second argument is an output.  Since output arguments are not modified if the system call fails, arguments may not always be dereferenced. For example, retrying the "ls -l" example with a non-existent file  produces the following line:  請注意“struct stat”參數是如何取消引用的，以及每個成員是如何象徵性地顯示的。  特別要注意的是，st_mode成員是如何被仔細地解碼成位或符號和數字值的。  在本例中還要注意，lstat的第一個參數是系統調用的輸入，第二個參數是輸出。  因為如果系統調用失敗，輸出參數不會被修改，所以參數可能不會總是被取消引用。  例如，用一個不存在的文件重試“ls -l”的例子，結果如下:  lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)    In this case the porch light is on but nobody is home.  在這種情況下，門廊的燈是亮的，但沒有人在家。（哈哈，文檔說的很含蓄）  Character pointers are dereferenced and printed as C strings. Non-printing characters in strings are normally represented by ordinary C escape codes. Only the first strsize  (32 by default) bytes of strings are printed; longer strings have an ellipsis appended following the closing quote.  字符指針被取消引用並打印為C字符串。字符串中的非打印字符通常由普通的C轉義碼錶示。只有第一個strsize  (默認為32)打印字符串位元組;較長的字符串在結束引用後附加省略號。  Here is a line from "ls -l" where the getpwuid library routine is reading the password file:  read(3, "root::0:0:System Administrator:/"..., 1024) = 422    While structures are annotated using curly braces, simple pointers and arrays are printed using square brackets with commas separating elements.  結構使用花括號進行注釋，而簡單指針和數組使用方括號和逗號分隔元素進行打印。  Here is an example from the  command "id" on a system with supplementary group ids:    getgroups(32, [100, 0]) = 2    On the other hand, bit-sets are also shown using square brackets but set elements are separated only by a space.  另一方面，位集也使用方括號顯示，但集元素僅用空格分隔。  Here is the shell preparing to execute an external command:    sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0    Here the second argument is a bit-set of two signals, SIGCHLD and SIGTTOU. In some cases the bit-set is so full that printing out the unset elements is more valuable. In that  case, the bit-set is prefixed by a tilde like this:  這裡的第二個參數是兩個信號的位集，SIGCHLD和SIGTTOU。在某些情況下，位集是如此的滿，以至於打印出未設置的元素更有價值。在這種情況下，位集前面加一個波浪號，就像這樣  sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0    Here the second argument represents the full set of all signals.  這裡，第二個參數表示所有信號的完整集合。

[root@node01 ~]# strace -h  usage: strace [-CdffhiqrtttTvVwxxy] [-I n] [-e expr]...                [-a column] [-o file] [-s strsize] [-P path]...                -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]     or: strace -c[dfw] [-I n] [-e expr]... [-O overhead] [-S sortby]                -p pid... / [-D] [-E var=val]... [-u username] PROG [ARGS]    Output format:    -a column      alignment COLUMN for printing syscall results (default 40)    -i             print instruction pointer at time of syscall    -o file        send trace output to FILE instead of stderr    -q             suppress messages about attaching, detaching, etc.    -r             print relative timestamp    -s strsize     limit length of print strings to STRSIZE chars (default 32)    -t             print absolute timestamp    -tt            print absolute timestamp with usecs    -T             print time spent in each syscall    -x             print non-ascii strings in hex    -xx            print all strings in hex    -y             print paths associated with file descriptor arguments    -yy            print protocol specific information associated with socket file descriptors    Statistics:    -c             count time, calls, and errors for each syscall and report summary    -C             like -c but also print regular output    -O overhead    set overhead for tracing syscalls to OVERHEAD usecs    -S sortby      sort syscall counts by: time, calls, name, nothing (default time)    -w             summarise syscall latency (default is system time)    Filtering:    -e expr        a qualifying expression: option=[!]all or option=[!]val1[,val2]...       options:    trace, abbrev, verbose, raw, signal, read, write    -P path        trace accesses to path    Tracing:    -b execve      detach on execve syscall    -D             run tracer process as a detached grandchild, not as parent    -f             follow forks    -ff            follow forks with output into separate files    -I interruptible       1:          no signals are blocked       2:          fatal signals are blocked while decoding syscall (default)       3:          fatal signals are always blocked (default if '-o FILE PROG')       4:          fatal signals and SIGTSTP (^Z) are always blocked                   (useful to make 'strace -o FILE PROG' not stop on ^Z)    Startup:    -E var         remove var from the environment for command    -E var=val     put var=val in the environment for command    -p pid         trace process with process id PID, may be repeated    -u username    run command as username handling setuid and/or setgid    Miscellaneous:    -d             enable debug output to stderr    -v             verbose mode: print unabbreviated argv, stat, termios, etc. args    -h             print help message    -V             print version  

[root@hadoop ~]# man strace | col -b >> aa.txt

OPTIONS         -c       Count time, calls, and errors for each system call and report a summary on program exit.  On Linux, this attempts to show system time (CPU time spent  running             in the kernel) independent of wall clock time.  If -c is used with -f or -F (below), only aggregate totals for all traced processes are kept.           -C       Like -c but also print regular output while processes are running.           -D       Run    tracer    process as a detached grandchild, not as parent of the tracee.    This reduces the visible effect of strace by keeping the tracee a direct child of             the calling process.           -d       Show some debugging output of strace itself on the standard error.           -f       Trace child processes as they are created by currently traced processes as a result of the fork(2), vfork(2) and clone(2) system calls.  Note that -p  PID  -f             will attach all threads of process PID if it is multi-threaded, not only thread with thread_id = PID.           -ff       If  the  -o filename option is in effect, each processes trace is written to filename.pid where pid is the numeric process id of each process.  This is incom‐             patible with -c, since no per-process counts are kept.           -F       This option is now obsolete and it has the same functionality as -f.           -h       Print the help summary.           -i       Print the instruction pointer at the time of the system call.           -k       Print the execution stack trace of the traced processes after each system call (experimental).  This option is available only if strace is built  with  libun‐             wind.           -q       Suppress  messages  about attaching, detaching etc.    This happens automatically when output is redirected to a file and the command is run directly instead of             attaching.           -qq       If given twice, suppress messages about process exit status.           -r       Print a relative timestamp upon entry to each system call.  This records the time difference between the beginning of successive system calls.           -t       Prefix each line of the trace with the time of day.           -tt       If given twice, the time printed will include the microseconds.           -ttt       If given thrice, the time printed will include the microseconds and the leading portion will be printed as the number of seconds since the epoch.           -T       Show the time spent in system calls.     This records the time difference between the beginning and the end of each system call.           -w       Summarise the time difference between the beginning and end of each system call.  The default is to summarise the system time.           -v       Print unabbreviated versions of environment, stat, termios, etc.  calls.  These structures are very common in calls and so the  default  behavior  displays    a             reasonable subset of structure members.  Use this option to get all of the gory details.           -V       Print the version number of strace.           -x       Print all non-ASCII strings in hexadecimal string format.           -xx       Print all strings in hexadecimal string format.           -y       Print paths associated with file descriptor arguments.           -yy       Print protocol specific information associated with socket file descriptors.           -a column   Align return values in a specific column (default column 40).           -b syscall  If  specified syscall is reached, detach from traced process.  Currently, only execve syscall is supported.    This option is useful if you want to trace multi-             threaded process and therefore require -f, but don't want to trace its (potentially very complex) children.           -e expr       A qualifying expression which modifies which events to trace or how to trace them.  The format of the expression is:                     [qualifier=][!]value1[,value2]...               where qualifier is one of trace, abbrev, verbose, raw, signal, read, or write and value is a qualifier-dependent symbol or number.  The default  qualifier  is             trace.   Using an exclamation mark negates the set of values.  For example, -e open means literally -e trace=open which in turn means trace only the open sys‐             tem call.  By contrast, -e trace=!open means to trace every system call except open.     In addition, the special values all and none have the obvious meanings.               Note that some shells use the exclamation point for history expansion even inside quoted arguments.    If so, you must escape the exclamation point with a back‐             slash.           -e trace=set             Trace  only    the  specified    set  of     system     calls.      The  -c  option  is  useful  for determining which system calls might be useful to trace.  For example,             trace=open,close,read,write means to only trace those four system calls.  Be careful when making inferences about the user/kernel boundary if only a subset of             system calls are being monitored.  The default is trace=all.           -e trace=file             Trace all system calls which take a file name as an argument.  You can think of this as an abbreviation for -e trace=open,stat,chmod,unlink,...  which is use‐             ful to seeing what files the process is referencing.     Furthermore, using the abbreviation will ensure that you don't accidentally forget  to     include  a  call             like lstat in the list.  Betchya woulda forgot that one.           -e trace=process             Trace all system calls which involve process management.  This is useful for watching the fork, wait, and exec steps of a process.           -e trace=network             Trace all the network related system calls.           -e trace=signal             Trace all signal related system calls.           -e trace=ipc             Trace all IPC related system calls.           -e trace=desc             Trace all file descriptor related system calls.           -e trace=memory             Trace all memory mapping related system calls.           -e abbrev=set             Abbreviate the output from printing each member of large structures.     The default is abbrev=all.  The -v option has the effect of abbrev=none.           -e verbose=set             Dereference structures for the specified set of system calls.  The default is verbose=all.           -e raw=set  Print raw, undecoded arguments for the specified set of system calls.  This option has the effect of causing all arguments to be printed in hexadecimal.  This             is mostly useful if you don't trust the decoding or you need to know the actual numeric value of an argument.           -e signal=set             Trace only the specified subset of signals.    The default is signal=all.  For example, signal =! SIGIO (or signal=!io) causes SIGIO signals not to be traced.           -e read=set Perform a full hexadecimal and ASCII dump of all the data read from file descriptors listed in the specified set.  For example, to see all input  activity  on             file     descriptors 3 and 5 use -e read=3,5.  Note that this is independent from the normal tracing of the read(2) system call which is controlled by the option             -e trace=read.           -e write=set             Perform a full hexadecimal and ASCII dump of all the data written to file descriptors listed in the specified set.  For example, to see all output activity on             file     descriptors  3     and  5     use  -e write=3,5.  Note that this is independent from the normal tracing of the write(2) system call which is controlled by the             option -e trace=write.           -I interruptible             When strace can be interrupted by signals (such as pressing ^C).  1: no signals are blocked; 2: fatal signals are blocked while decoding syscall (default); 3:             fatal  signals  are    always    blocked (default if '-o FILE PROG'); 4: fatal signals and SIGTSTP (^Z) are always blocked (useful to make strace -o FILE PROG not             stop on ^Z).           -o filename Write the trace output to the file filename rather than to stderr.  Use filename.pid if -ff is used.     If the argument begins with '|' or  with  '!'    then  the             rest of the argument is treated as a command and all output is piped to it.    This is convenient for piping the debugging output to a program without affecting             the redirections of executed programs.           -O overhead Set the overhead for tracing system calls to overhead microseconds.    This is useful for overriding the default heuristic for guessing how much time    is  spent             in  mere  measuring    when  timing system calls using the -c option.    The accuracy of the heuristic can be gauged by timing a given program run without tracing             (using time(1)) and comparing the accumulated system call time to the total produced using -c.           -p pid       Attach to the process with the process ID pid and begin tracing.  The trace may be terminated at any time by a keyboard  interrupt  signal  (CTRL-C).   strace             will     respond  by  detaching itself from the traced process(es) leaving it (them) to continue running.  Multiple -p options can be used to attach to many pro‐             cesses in addition to command (which is optional if at least one -p option is given).  -p "`pidof PROG`" syntax is supported.           -P path       Trace only system calls accessing path.  Multiple -P options can be used to specify several paths.           -s strsize  Specify the maximum string size to print (the default is 32).  Note that filenames are not considered strings and are always printed in full.           -S sortby   Sort the output of the histogram printed by the -c option by the specified criterion.  Legal values are time, calls, name, and nothing (default is time).           -u username Run command with the user ID, group ID, and supplementary groups of username.  This option is only useful when running as root and enables the correct  execu‐             tion of setuid and/or setgid binaries.  Unless this option is used setuid and setgid programs are executed without effective privileges.           -E var=val  Run command with var=val in its list of environment variables.           -E var       Remove var from the inherited list of environment variables before passing it on to the command.