NUMA Off

LinuxKI Warning

Oracle May Disable NUMA Support in Linux Kernel
Date: 05/22/2015

Problem

Performance is not as expected on multi-socket systems running Oracle Database 12c or 11g on Linux.

Investigation

Each processor socket on a system should be represented as a “node” in Linux, yet numactl shows only a single node:

$ numactl --hardware

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 ... 471 472 473 474 475 476 477 478 479
node 0 size: 12582795 MB
node 0 free: 7074765 MB
node distances:
node   0 
  0:  10

The kernel logs will also show that NUMA features have been turned off

May 22 13:32:04 tux kernel: NUMA turned off
May 22 13:32:04 tux kernel: Faking a node at ...

Root Cause

Oracle provides RPMs to simplify Oracle 12c and 11g installations named oracle-rdbms-server-12cR1-preinstall and oracle-rdbms-server-11gR2-preinstall respectively. These rpms create the oracle user and associated groups (oinstall and dba), modifies kernel parameters for shared memory and the like, and sets shell resource limits. They may also add “numa=off” to the kernel boot string which disables all NUMA features in kernel. This will result in sub-optimal memory allocations and task scheduling decisions resulting poor performance, especially on scale-up platforms such as the DL580 and SuperdomeX.

Solution

Examine the kernel boot string and remove any occurrences of "numa=off":

$ cat /proc/cmdline
ro root=/dev/mapper/vg_tux-lv_root rd_NO_LUKS rd_LVM_LV=vg_tux/lv_root LANG=en_US.UTF-8 rd_NO_MD
rd_LVM_LV=vg_tux/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=512M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM numa=off
transparent_hugepage=never