Quick reference of this noble and old school command.
Very useful for troubleshooting and server performance.
Checking CPU Activity
By default, sar command will report the CPU activity of the server. The option -u can be used to get the CPU utilization report.
To get the CPU activity report in every 2 seconds for 3 times:
$ sar -u 2 3 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 09/06/2020 _x86_64_ (2 CPU) 22:26:54 CPU %user %nice %system %iowait %steal %idle 22:26:56 all 4,03 0,00 0,50 0,00 0,00 95,47 22:26:58 all 8,08 0,00 0,51 0,00 0,00 91,41 22:27:00 all 12,50 0,00 1,00 0,00 0,00 86,50 Average: all 8,21 0,00 0,67 0,00 0,00 91,11
CPU usage for each processor
To find the CPU activity on all processors separately, you need to use the -P option.
$ sar -P ALL 1 1 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 05:38:18 CPU %user %nice %system %iowait %steal %idle 05:38:19 all 3,03 0,00 0,00 0,00 0,00 96,97 05:38:19 0 3,96 0,00 0,99 0,00 0,00 95,05 05:38:19 1 3,00 0,00 0,00 0,00 0,00 97,00 Average: CPU %user %nice %system %iowait %steal %idle Average: all 3,03 0,00 0,00 0,00 0,00 96,97 Average: 0 3,96 0,00 0,99 0,00 0,00 95,05 Average: 1 3,00 0,00 0,00 0,00 0,00 97,00
Check Memory usage
To find the memory usage (used and free memory of the server) over time using the -r switch.
$ sar -r 1 3 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 05:41:04 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 05:41:05 855300 3025220 77,96 0 1495276 2426864 30,55 1265008 1158172 100 05:41:06 854800 3025720 77,97 0 1495276 2426864 30,55 1265132 1158172 100 05:41:07 855032 3025488 77,97 0 1495276 2426864 30,55 1265012 1158172 100 Average: 855044 3025476 77,97 0 1495276 2426864 30,55 1265051 1158172 100
Check Swap Activity
Check the swap usage of the machine using the -W option:
$ sar -W 1 3 Linux 2.6.18-274.18.1.el5 (myserver) 09/06/2012 03:31:12 PM pswpin/s pswpout/s 03:31:13 PM 16.16 0.00 03:31:14 PM 16.00 0.00 03:31:15 PM 15.84 0.00 Average: 16.00 0.00
Find load averages over time
You can find the load averages overtime using the -q option:
$ sar -q 1 3 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 06:16:13 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 06:16:14 0 329 0,00 0,04 0,09 0 06:16:15 0 329 0,00 0,04 0,09 0 06:16:16 0 329 0,00 0,04 0,09 0 Average: 0 329 0,00 0,04 0,09 0
Statistics for currently mounted filesystems
$ sar -F 2 3 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 06:30:18 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 06:30:20 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root 06:30:20 796 218 21,52 21,52 523947 341 0,07 /dev/sda1 06:30:20 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 06:30:22 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root 06:30:22 796 218 21,52 21,52 523947 341 0,07 /dev/sda1 06:30:22 MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM 06:30:24 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root 06:30:24 796 218 21,52 21,52 523947 341 0,07 /dev/sda1 Summary: MBfsfree MBfsused %fsused %ufsused Ifree Iused %Iused FILESYSTEM Summary 30410 5532 15,39 15,39 18245113 166407 0,90 /dev/mapper/centos-root Summary 796 218 21,52 21,52 523947 341 0,07 /dev/sda1
Details of inode, kernel tables and file tables
$ sar -v 2 3 Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 06:57:23 dentunusd file-nr inode-nr pty-nr 06:57:25 160833 2400 132081 3 06:57:27 160833 2400 132081 3 06:57:29 160833 2400 132081 3 Average: 160833 2400 132081 3
Network statistics
$ sar -n DEV Linux 3.10.0-1127.10.1.el7.x86_64 (localhost.localdomain) 10/06/2020 _x86_64_ (2 CPU) 19:20:23 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 19:20:24 ens192 11,00 4,00 1,80 2,75 0,00 0,00 0,00 19:20:24 lo 79,00 79,00 43,62 43,62 0,00 0,00 0,00 Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s Average: ens192 11,00 4,00 1,80 2,75 0,00 0,00 0,00 Average: lo 79,00 79,00 43,62 43,62 0,00 0,00
Which interrupt number could be causing a CPU bottleneck?
$ sar -I XALL 2 10 02:07:10 AM INTR intr/s 02:07:12 AM 0 992.57 02:07:12 AM 1 0.00 02:07:12 AM 2 0.00 02:07:12 AM 3 0.00 02:07:12 AM 4 0.00 02:07:12 AM 5 0.00 02:07:12 AM 6 0.00 02:07:12 AM 7 0.00 02:07:12 AM 8 0.00 02:07:12 AM 9 350.50
‘/proc/interrupts’ file will also provide helpful information. Interrupt halts CPU processing so that I/O or other operations can occur. Processing resumes after the specific operation takes place. It is very important that each device installed in machine is provided with an interrupt setting that does not conflict with the settings used by the hardware and other devices.
$ sudo cat /proc/interrupts CPU0 CPU1 0: 48 0 IO-APIC-edge timer 1: 54 0 IO-APIC-edge i8042 8: 1 0 IO-APIC-edge rtc0 9: 0 0 IO-APIC-fasteoi acpi 12: 35 116 IO-APIC-edge i8042 14: 0 0 IO-APIC-edge ata_piix 15: 0 0 IO-APIC-edge ata_piix 16: 118 2513 IO-APIC-fasteoi vmwgfx 24: 0 0 PCI-MSI-edge PCIe PME, pciehp 25: 0 0 PCI-MSI-edge PCIe PME, pciehp 26: 0 0 PCI-MSI-edge PCIe PME, pciehp 27: 0 0 PCI-MSI-edge PCIe PME, pciehp NMI: 0 0 Non-maskable interrupts LOC: 35392807 14792833 Local timer interrupts SPU: 0 0 Spurious interrupts PMI: 0 0 Performance monitoring interrupts IWI: 2677624 215297 IRQ work interrupts
Hernan Vivani's Blog 