Mastering the art of troubleshooting book download

Preface
Chapter 1: Best Practices and Initial Investigation
Chapter 2: strace and System Call Tracing Explained
Chapter 3: The /proc Filesystem
Chapter 4: Compiling
Chapter 5: The Stack
Chapter 6: The GNU Debugger (GDB)
Chapter 7: Linux System Crashes and Hangs
Chapter 8: Kernel Debugging with KDB
Chapter 9: ELF: Executable and Linking Format
A: The Toolbox
B: Data Collection Script
Index
Contents
Preface xvii
1 Best Practices and Initial Investigation 1
1.1 Introduction 1
1.2 Getting Your System(s) Ready for Effective Problem
Determination 2
1.3 The Four Phases of Investigation 3
1.3.1 Phase #1: Initial Investigation Using Your Own Skills 5
1.3.2 Phase #2: Searching the Internet Effectively 9
1.3.3 Phase #3: Begin Deeper Investigation (Good Problem
Investigation Practices) 12
1.3.4 Phase #4: Getting Help or New Ideas 21
1.4 Technical Investigation 28
1.4.1 Symptom Versus Cause 28
1.5 Troubleshooting Commercial Products 38
1.6 Conclusion 39
2 strace and System Call Tracing Explained 41
2.1 Introduction 41
2.2 What Is strace? 41
2.2.1 More Information from the Kernel Side 45
2.2.2 When to Use It 48
2.2.3 Simple Example 49
2.2.4 Same Program Built Statically 53
2.3 Important strace Options 54
2.3.1 Following Child Processes 54
2.3.2 Timing System Call Activity 55
2.3.3 Verbose Mode 57
2.3.4 Tracing a Running Process 59
2.4 Effects and Issues of Using strace 60
2.4.1 strace and EINTR 61
2.5 Real Debugging Examples 62
2.5.1 Reducing Start Up Time by Fixing
LD_LIBRARY_PATH 62
2.5.2 The PATH Environment Variable 65
2.5.3 stracing inetd or xinetd (the Super Server) 66
2.5.4 Communication Errors 68
2.5.5 Investigating a Hang Using strace 69
2.5.6 Reverse Engineering (How the strace Tool Itself Works) 71
2.6 System Call Tracing Examples 74
2.6.1 Sample Code 75
2.6.2 The System Call Tracing Code Explained 87
2.7 Conclusion 88
3 The /proc Filesystem 89
3.1 Introduction 89
3.2 Process Information 90
3.2.1 /proc/self 90
3.2.2 /proc/ in More Detail 91
3.2.3 /proc//cmdline 107
3.2.4 /proc//environ 107
3.2.5 /proc//mem 107
3.2.6 /proc//fd 108
3.2.7 /proc//mapped base 108
3.3 Kernel Information and Manipulation 109
3.3.1 /proc/cmdline 109
3.3.2 /proc/config.gz or /proc/sys/config.gz 109
3.3.3 /proc/cpufreq 109
3.3.4 /proc/cpuinfo 110
3.3.5 /proc/devices 110
3.3.6 /proc/kcore 111
3.3.7 /proc/locks 111
3.3.8 /proc/meminfo 111
3.3.9 /proc/mm 111
3.3.10 /proc/modules 112
3.3.11 /proc/net 112
3.3.12 /proc/partitions 112
3.3.13 /proc/pci 113
3.3.14 /proc/slabinfo 113
x Contents
3.4 System Information and Manipulation 113
3.4.1 /proc/sys/fs 113
3.4.2 /proc/sys/kernel 115
3.4.3 /proc/sys/vm 120
3.5 Conclusion 120
4 Compiling 121
4.1 Introduction 121
4.2 The GNU Compiler Collection 121
4.2.1 A Brief History of GCC 121
4.2.2 GCC Version Compatibility 122
4.3 Other Compilers 122
4.4 Compiling the Linux Kernel 123
4.4.1 Obtaining the Kernel Source 123
4.4.2 Architecture Specific Source 124
4.4.3 Working with Kernel Source Compile Errors 124
4.4.4 General Compilation Problems 128
4.5 Assembly Listings 133
4.5.1 Purpose of Assembly Listings 134
4.5.2 Generating Assembly Listings 135
4.5.3 Reading and Understanding an Assembly Listing 136
4.6 Compiler Optimizations 140
4.7 Conclusion 149
5 The Stack 151
5.1 Introduction 151
5.2 A Real-World Analogy 152
5.3 Stacks in x86 and x86-64 Architectures 153
5.4 What Is a Stack Frame? 157
5.5 How Does the Stack Work? 159
5.5.1 The BP and SP Registers 159
5.5.2 Function Calling Conventions 162
5.6 Referencing and Modifying Data on the Stack 171
5.7 Viewing the Raw Stack in a Debugger 173
5.8 Examining the Raw Stack in Detail 176
5.8.1 Homegrown Stack Traceback Function 180
5.9 Conclusion 191
6 The GNU Debugger (GDB) 193
6.1 Introduction 193
6.2 When to Use a Debugger 194
6.3 Command Line Editing 195
Contents xi
6.4 Controlling a Process with GDB 196
6.4.1 Running a Program Off the Command Line with GDB 197
6.4.2 Attaching to a Running Process 199
6.4.3 Use a Core File 200
6.5 Examining Data, Memory, and Registers 204
6.5.1 Memory Map 204
6.5.2 Stack 206
6.5.3 Examining Memory and Variables 210
6.5.4 Register Dump 217
6.6 Execution 220
6.6.1 The Basic Commands 221
6.6.2 Settings for Execution Control Commands 223
6.6.3 Breakpoints 228
6.6.4 Watchpoints 230
6.6.5 Display Expression on Stop 234
6.6.6 Working with Shared Libraries 235
6.7 Source Code 238
6.8 Assembly Language 240
6.9 Tips and Tricks 241
6.9.1 Attaching to a Process.Revisited 241
6.9.2 Finding the Address of Variables and Functions 244
6.9.3 Viewing Structures in Executables without Debug
Symbols 246
6.9.4 Understanding and Dealing with Endian-ness 250
6.10 Working with C++ 252
6.10.1 Global Constructors and Destructors 252
6.10.2 Inline Functions 256
6.10.3 Exceptions 257
6.11 Threads 260
6.11.1 Running Out of Stack Space 265
6.12 Data Display Debugger (DDD) 266
6.12.1 The Data Display Window 268
6.12.2 Source Code Window 272
6.12.3 Machine Language Window 273
6.12.4 GDB Console Window 274
6.13 Conclusion 274
7 Linux System Crashes and Hangs 275
7.1 Introduction 275
7.2 Gathering Information 275
7.2.1 Syslog Explained 276
7.2.2 Setting up a Serial Console 277
xii Contents
7.2.3 Connecting the Serial Null-Modem Cable 278
7.2.4 Enabling the Serial Console at Startup 279
7.2.5 Using SysRq Kernel Magic 281
7.2.6 Oops Reports 281
7.2.7 Adding a Manual Kernel Trap 281
7.2.8 Examining an Oops Report 284
7.2.9 Determining the Failing Line of Code 289
7.2.10 Kernel Oopses and Hardware 293
7.2.11 Setting up cscope to Index Kernel Sources 294
7.3 Conclusion 295
8 Kernel Debugging with KDB 297
8.1 Introduction 297
8.2 Enabling KDB 297
8.3 Using KDB 299
8.3.1 Activating KDB 299
8.3.2 Resuming Normal Execution 300
8.3.3 Basic Commands 300
8.4 Conclusion 305
9 ELF: Executable and Linking Format 307
9.1 Introduction 307
9.2 Concepts and Definitions 309
9.2.1 Symbol 309
9.2.2 Object Files, Shared Libraries, Executables, and Core
Files 311
9.2.3 Linking 314
9.2.4 Run Time Linking 318
9.2.5 Program Interpreter / Run Time Linker 318
9.3 ELF Header 318
9.4 Overview of Segments and Sections 324
9.5 Segments and the Program Header Table 325
9.5.1 Text and Data Segments 329
9.6 Sections and the Section Header Table 331
9.6.1 String Table Format 335
9.6.2 Symbol Table Format 335
9.6.3 Section Names and Types 338
9.7 Relocation and Position Independent Code (PIC) 362
9.7.1 PIC vs. non-PIC 363
9.7.2 Relocation and Position Independent Code 366
9.7.3 Relocation and Linking 367
9.8 Stripping an ELF Object 371
Contents xiii
9.9 Program Interpreter 372
9.9.1 Link Map 376
9.10 Symbol Resolution 377
9.11 Use of Weak Symbols for Problem Investigations 382
9.12 Advanced Interception Using Global Offset Table 386
9.13 Source Files 390
9.14 ELF APIs 392
9.15 Other Information 392
9.16 Conclusion 392
A The Toolbox 393
A.1 Introduction 393
A.2 Process Information and Debugging 393
A.2.1 Tool: GDB 393
A.2.2 Tool: ps 393
A.2.3 Tool: strace (system call tracer) 394
A.2.4 Tool: /proc filesystem 394
A.2.5 Tool: DDD (Data Display Debugger) 394
A.2.6 Tool: lsof (List Open Files) 394
A.2.7 Tool: ltrace (library call tracer) 395
A.2.8 Tool: time 395
A.2.9 Tool: top 395
A.2.10 Tool: pstree 396
A.3 Network 396
A.3.1 Tool: traceroute 396
A.3.2 File: /etc/hosts 396
A.3.3 File: /etc/services 396
A.3.4 Tool: netstat 397
A.3.5 Tool: ping 397
A.3.6 Tool: telnet 397
A.3.7 Tool: host/nslookup 397
A.3.8 Tool: ethtool 398
A.3.9 Tool: ethereal 398
A.3.10 File: /etc/nsswitch.conf 398
A.3.11 File: /etc/resolv.conf 398
A.4 System Information 399
A.4.1 Tool: vmstat 399
A.4.2 Tool: iostat 399
A.4.3 Tool: nfsstat 399
A.4.4 Tool: sar 400
A.4.5 Tool: syslogd 400
A.4.6 Tool: dmesg 400
xiv Contents
A.4.7 Tool: mpstat 400
A.4.8 Tool: procinfo 401
A.4.9 Tool: xosview 401
A.5 Files and Object Files 401
A.5.1 Tool: file 401
A.5.2 Tool: ldd 401
A.5.3 Tool: nm 402
A.5.4 Tool: objdump 402
A.5.5 Tool: od 402
A.5.6 Tool: stat 402
A.5.7 Tool: readelf 403
A.5.8 Tool: strings 403
A.6 Kernel 403
A.6.1 Tool: KDB 403
A.6.2 Tool: KGDB 403
A.6.3 Tool: ksymoops 404
A.7 Miscellaneous 404
A.7.1 Tool: VMWare Workstation 404
A.7.2 Tool: VNC Server 405
A.7.3 Tool: VNC Viewer 405
B Data Collection Script 407
B.1 Overview 407
B.1.1 -thorough 409
B.1.2 -perf, -hang , -trap, -error 409
B.2 Running the Script 410
B.3 The Script Source 410
B.4 Disclaimer 419

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s