【调试】sysRq键使用方法

SysRq键简介

SysRq键是一个魔术案件，只要在内核没有完全卡死的情况下，内核都会相应SysRq 键的输入，使用这些组合键都可以搜集包括系统内存使用、CPU任务处理、进程运行状态等系统运行信息。

配置

内核配置选项中要使能CONFIG_MAGIC_SYSRQ选项，这样系统启动之后，会生成/proc/sysrq-trigger节点用于调试。

修改/etc/sysctl.conf启动SysRq。

kernel.sysrq=1

执行下列命令，使修改生效。

sysctl?-p

也可以通过写echo "number" >/proc/sys/kernel/sysrq节点临时使能sysrq功能。

写入不同的值使能不同的功能：

值	描述/说明
0	完全禁用 sysrq
1	启用 sysrq 的所有功能
2	允许控制终端日志级别
4	允许控制终端日志级别
8	允许调试进程dump
16	允许执行sync命令
32	允许重新挂载文件系统为之读
64	允许发送信号给进程(term,kill,oom-kill)
128	允许重启/关机
256	允许调整实时任务的优先级

如何使用SysRq

echo??<command?key>??>?/proc/sysrq-trigger

命令键 <command key> 是区分大小写的。

<command key> ?具体含义如下。

命令键	功能
`b`	将立即重启系统，不会同步或者卸载磁盘。
`c`	将执行系统 crash，如果配置了系统 crashdump，将执行 crashdump。
`d`	显示所有持有的锁。
`e`	发送 SIGTERM 信号给所有进程，除了 init 进程。
`f`	将调用 oom killer 杀掉一个过度占用内存的进程，如果什么任务都没杀，也不会 panic。
`g`	kgdb 使用（内核调试器）。
`h`	将会显示帮助。（实际上除了这里列举的键，其他的都将显示帮助，但是 `h` 容易记住）:-)
`i`	发送 SIGKILL 给所有进程，除了 init 进程。
`j`	强制性的 “解冻它” - 用于被 FIFREEZE ioctl 操作冻住的文件系统。
`k`	安全访问秘钥(SAK)杀掉在当前虚拟控制台的所有程序，注意：参考下面 SAK 节重要论述。
`l`	显示所有活动 cpu 的栈回溯。
`m`	将导出当前内存信息到你的控制台。
`n`	用于使所有实时任务变成普通任务。
`o`	将关闭系统（如果配置和支持的话）。
`p`	将导出当前寄存器和标志位到控制台。
`q`	将导出每个 cpu 上所有已装备的高精度定时器（不是完整的 time_list 文件显示的 timers）和所有时钟事件设备的详细信息。
`r`	关闭键盘的原始模式，设置为转换模式。
`s`	将尝试同步所有的已挂载文件系统。
`t`	将导出当前所有任务列表和它们的信息到控制台。
`u`	将尝试重新挂载已挂载文件系统为只读。
`v`	强制恢复帧缓存控制台。
`v`	触发 ETM 缓存导出 [ARM 架构特有]
`w`	导出处于不可中断状态（阻塞）的任务。
`x`	在 ppc/powerpc 架构上用于 xmon 接口。在 sparc64 架构上用于显示全局的 PMU（性能监控单元）寄存器。在 MIPS 架构上导出所有的 tlb 条目。
`y`	显示全局 cpu 寄存器 [SPARC-64 架构特有]
`z`	导出 ftrace 缓存信息
`0`-`9`	设置控制台日志级别，该级别控制什么样的内核信息将被打印到你的控制台。（比如 `0` ，将使得只有紧急信息，像 PANICs or OOPSes 才能到你的控制台。）

打印内存使用信息

root@firefly:~#?echo??m?>?/proc/sysrq-trigger
[?3100.196187]?sysrq:?SysRq?:?Show?Memory
[?3100.196675]?Mem-Info:
[?3100.196931]?active_anon:48062?inactive_anon:2987?isolated_anon:0
[?3100.196931]??active_file:40449?inactive_file:53540?isolated_file:0
[?3100.196931]??unevictable:0?dirty:22?writeback:0?unstable:0
[?3100.196931]??slab_reclaimable:10518?slab_unreclaimable:6599
[?3100.196931]??mapped:44843?shmem:3710?pagetables:1568?bounce:0
[?3100.196931]??free:810482?free_pcp:497?free_cma:0
[?3100.199998]?DMA?free:3241928kB?min:7920kB?low:9900kB?high:11880kB?active_anon:192248kB?inactive_anon:11948kB?active_file:161796kB?inactive_file:214160kB?unevictable:0kB?isolated(anon):0kB?isolated(file):0kB?present:4030464kB?managed:3933592kB?mlocked:0kB?dirty:88kB?writeback:0kB?mapped:179372kB?shmem:14840kB?slab_reclaimable:42072kB?slab_unreclaimable:26396kB?kernel_stack:5792kB?pagetables:6272kB?unstable:0kB?bounce:0kB?free_pcp:1988kB?local_pcp:224kB?free_cma:0kB?writeback_tmp:0kB?pages_scanned:0?all_unreclaimable??no
[?3100.204031]?lowmem_reserve[]:?0?0?0
[?3100.204414]?DMA:?272*4kB?(U)?95*8kB?(UM)?177*16kB?(UME)?88*32kB?(UME)?50*64kB?(UM)?32*128kB?(UM)?6*256kB?(UM)?4*512kB?(UM)?2*1024kB?(M)?3*2048kB?(UME)?785*4096kB?(M)?=?3241928kB
[?3100.206208]?Node?0?hugepages_total=0?hugepages_free=0?hugepages_surp=0?hugepages_size=2048kB
[?3100.206990]?97710?total?pagecache?pages
[?3100.207345]?0?pages?in?swap?cache
[?3100.207679]?Swap?cache?stats:?add?0,?delete?0,?find?0/0
[?3100.208156]?Free?swap??=?0kB
[?3100.208425]?Total?swap?=?0kB
[?3100.208721]?1007616?pages?RAM
[?3100.209003]?0?pages?HighMem/MovableOnly
[?3100.209352]?24218?pages?reserved
root@firefly:~#?

打印进程列表

root@firefly:~#?echo??t?>?/proc/sysrq-trigger
[?3357.426399]?sysrq:?SysRq?:?Show?State
[?3357.426902]?Call?trace:
[?3357.427155]?[<ffffff8008085048>]?__switch_to+0x94/0xbc
[?3357.427759]?[<ffffff8008eee8cc>]?__schedule+0x334/0x65c
[?3357.428249]?[<ffffff8008eeec64>]?schedule+0x70/0x84
[?3357.428762]?[<ffffff8008ef175c>]?schedule_hrtimeout_range_clock+0x54/0xd8
[?3357.429390]?[<ffffff8008ef1818>]?schedule_hrtimeout_range+0x38/0x48
[?3357.430004]?[<ffffff80082286e8>]?SyS_epoll_wait+0x2c0/0x334
[?3357.430515]?[<ffffff8008228810>]?SyS_epoll_pwait+0xb4/0x104
[?3357.431057]?[<ffffff8008082f70>]?el0_svc_naked+0x24/0x28
[?3357.431576]?Call?trace:
[?3359.224688]?runnable?tasks:
[?3359.224688]?????????????task???PID?????????tree-key??switches??prio?????wait-time?????????????sum-exec????????sum-sleep
[?3359.224688]?----------------------------------------------------------------------------------------------------------
[?3359.226942]???????watchdog/5????31????????-8.990375???????842?????0?????????0.010418????????33.639974?????????0.011083?/
[?3359.227935]??????migration/5????32?????????0.000000????????11?????0?????????0.006417?????????0.703626?????????0.011083?/
...............................
[?3359.280491]?????????rsyslogd???620?????12549.402602????????31???120?????????3.959710????????13.067249???3000146.370846?/
[?3359.281483]????????????gmain???649??????2780.837080?????????4???120?????????1.497875?????????0.234791????????42.333083?/
[?3359.282473]????????????gdbus???653??????6084.558463???????235???120????????11.845505????????88.143377??????5894.875470?/
[?3359.283464]???probing-thread???657??????2798.851658?????????1???120?????????0.171209?????????0.046375?????????0.000000?/
[?3359.284458]????????????gmain???666?????13129.747017???????842???120?????????1.635792???????160.937875???...............................??
[?3359.347891]??????kbase_event??1087??????5737.991190?????????2???100?????????0.000000?????????0.015416?????????0.022167?/
[?3359.348888]??????kworker/5:0??1415?????12729.712418??????6352???120????????94.661328???????379.266952????300531.917073?/
[?3359.349880]??????kworker/5:1??1421?????13139.623478??????4592???120????????40.804756???????269.058278????218093.165092?/
[?3359.350919]?

打印 CPU 信息

root@firefly:~#?echo??w?>?/proc/sysrq-trigger
[?3461.098259]?sysrq:?SysRq?:?Show?Blocked?State
[?3461.099495]?Sched?Debug?Version:?v0.11,?4.4.194+?#15
[?3461.099987]?ktime???????????????????????????????????:?3460974.551474
[?3461.100559]?sched_clk???????????????????????????????:?3461099.486338
[?3461.101163]?cpu_clk?????????????????????????????????:?3461099.486629
[?3461.101733]?jiffies?????????????????????????????????:?4298128273
[?3461.102303]?
[?3461.102457]?sysctl_sched
...............................
[?3461.106084]?
[?3461.476753]?????????????task???PID?????????tree-key??switches??prio?????wait-time?????????????sum-exec????????sum-sleep
[?3461.476753]?----------------------------------------------------------------------------------------------------------
[?3461.479007]???????watchdog/5????31????????-8.990375???????868?????0?????????0.010418????????34.639809?????????0.011083?/
[?3461.479999]??????migration/5????32?????????0.000000????????11?????0?????????0.006417?????????0.703626?????????0.011083?/
[?3461.480990]??????ksoftirqd/5????33?????12976.655002???????282???120????????31.262910?????????9.577926???3226896.156313?/
....................
[?3461.598951]????????????gdbus???981??????5621.043245????????63???120????????10.429415?????????7.925751???????240.024166?/
[?3461.599945]??????kbase_event??1087??????5737.991190?????????2???100?????????0.000000?????????0.015416?????????0.022167?/
[?3461.600943]??????kworker/5:1??1421?????13242.484431??????6324???120????????56.331917???????371.919231????300578.331580?/
[?3461.601934]??????kworker/5:0??1424?????13267.128801???????416???120?????????3.932261????????25.079662?????19671.560460?/
[?3461.602964]?

显示所有活动 cpu 的栈回溯

root@firefly:~#?echo??l?>?/proc/sysrq-trigger
[?3602.003162]?sysrq:?SysRq?:?Show?backtrace?of?all?active?CPUs
[?3602.003919]?Call?trace:
[?3602.004175]?[<ffffff8008088300>]?dump_backtrace+0x0/0x220
root@firefly:~#?[?3602.004675]?[<ffffff8008088544>]?show_stack+0x24/0x30
[?3602.005233]?[<ffffff80085bbdac>]?showacpu+0x68/0x84
[?3602.005688]?[<ffffff8008134b68>]?flush_smp_call_function_queue+0x84/0x134
[?3602.006303]?[<ffffff8008135584>]?generic_smp_call_function_single_interrupt+0x18/0x20
[?3602.007007]?[<ffffff800808cba4>]?handle_IPI+0x19c/0x260
[?3602.007490]?[<ffffff8008080eec>]?gic_handle_irq+0x12c/0x184
[?3602.007995]?Exception?stack(0xffffffc0f2f1fd90?to?0xffffffc0f2f1fec0)
[?3602.008580]?fd80:???????????????????????????????????0000000000000000?0000000000000003
[?3602.009284]?fda0:?00000040ee976000?0000000000000020?00ffffffffffffff?0000001425865f85
[?3602.009988]?fdc0:?000000000004b382?0000000000000003?00000032b5593519?ffffff8008081800
[?3602.010691]?fde0:?0000000000001000?0000000000000000?0000000034d5d91d?00000040ee976000
[?3602.011394]?fe00:?ffffffc0f2f0a880?0000000000000000?0000000000000000?0000000000000000
[?3602.012098]?fe20:?0000000030d00800?00000346a09ab464?0000000000000002?ffffffc00a4bac00
[?3602.012801]?fe40:?ffffff80097ace18?0000000000000001?0000034697fd8ddd?ffffff800974e488
[?3602.013504]?fe60:?0000000001ab4000?00000000010f8244?0000000000000000?ffffffc0f2f1fec0
[?3602.014209]?fe80:?ffffff8008b9eb10?ffffffc0f2f1fec0?ffffff8008b9eb14?0000000060000145
[?3602.014912]?fea0:?ffffffc0f2f1fec0?ffffff8008b9eb10?ffffffffffffffff?0000000000000002
[?3602.015615]?[<ffffff80080827b4>]?el1_irq+0xb4/0x140
[?3602.016070]?[<ffffff8008b9eb14>]?cpuidle_enter_state+0x1cc/0x25c
[?3602.016624]?[<ffffff8008b9ec18>]?cpuidle_enter+0x34/0x44
[?3602.017112]?[<ffffff80081004e4>]?call_cpuidle+0x6c/0x74
[?3602.017598]?[<ffffff8008100798>]?cpu_startup_entry+0x2ac/0x2bc
[?3602.018128]?[<ffffff800808c5b0>]?secondary_start_kernel+0x190/0x1bc
[?3602.018700]?[<00000000010f8188>]?0x10f8188

触发 Crashdump

root@firefly:~#?echo??c?>?/proc/sysrq-trigger
[?3635.905960]?sysrq:?SysRq?:?Trigger?a?crash
[?3635.906476]?Unable?to?handle?kernel?NULL?pointer?dereference?at?virtual?address?00000000
[?3635.907201]?pgd?=?ffffffc0d4a3e000
[?3635.907640]?[00000000]?*pgd=0000000000000000,?*pud=0000000000000000
[?3635.908305]?Internal?error:?Oops:?96000045?[#1]?SMP
..............................................................
[?3648.331820]?Exception?stack(0xffffffc0cd8cbb30?to?0xffffffc0cd8cbc60)
[?3648.332405]?bb20:???????????????????????????????????ffffff800967ff68?0000008000000000
[?3648.333108]?bb40:?ffffffc0cd8cbd00?ffffff80085bb538?000000000000000f?0000000000000000
[?3648.333811]?bb60:?ffffff80097f72f8?0000000000000002?ffffffc0cd8cbb90?000000030003c018
[?3648.334514]?bb80:?00000000000000c3?0000000100000000?ffffffc0cd8cbc30?ffffff800810d674
[?3648.335218]?bba0:?ffffffc0cd8cbc90?ffffff80092b55d1?ffffff80095fa000?0000000000000006
[?3648.335921]?bbc0:?0000000000000063?0000000000000000?0000000000000001?0000000000000000
[?3648.336623]?bbe0:?ffffffc0f7ee8320?0000000000000000?0000000000000000?0000000000000000
[?3648.337327]?bc00:?0000000000000010?ffffff800965ead8?ffffff8008478398?7f7f7f7f7f7f7f7f
[?3648.338029]?bc20:?71277660716d73ff?7f7f7f7f7f7f7f7f?0101010101010101?0000000000000000
[?3648.338733]?bc40:?0ffffffffffffffe?0000000000000000?ffffff80081e5cb8?0000007f8147b8c8
[?3648.339434]?[<ffffff80080825cc>]?el1_da+0x24/0x8c
[?3648.339873]?[<ffffff80085bc008>]?__handle_sysrq+0xa0/0x14c
[?3648.340379]?[<ffffff80085bc434>]?write_sysrq_trigger+0x5c/0x74
[?3648.340911]?[<ffffff8008240cf4>]?proc_reg_write+0xa8/0xcc
[?3648.341406]?[<ffffff80081e4a88>]?__vfs_write+0x48/0xe8
[?3648.341877]?[<ffffff80081e5374>]?vfs_write+0xa8/0x15c
[?3648.342338]?[<ffffff80081e5d14>]?SyS_write+0x5c/0xb0
[?3648.342798]?[<ffffff8008082f70>]?el0_svc_naked+0x24/0x28

总结

SysRq 在处理系统挂起时安全重启方面已经比较完善了。