Linux Kernel Live Patching with Kpatch

Introduction

Recently at work we’ve been talking about live patching solutions for the Linux kernel, in order to reduce the need for server reboots, specially when critical kernel patches need to be deployed ASAP, outside the normal patching cycles.

The Linux kernel provides this functionality natively, through a series of mechanisms such as Kprobes, Ftrace and Livepatch [1], but the implementation of userspace solutions using them depends upon the vendor: Red Hat Enterprise Linux (RHEL), which will be the focus of this article, for instance, provides Kpatch for that purpose.

In general terms, the kernel live patching works by redirecting function calls inside the kernel (using Ftrace and Livepatch), from the original functions to replacements functions – so, for instance, a function that contains a vulnerability can, in practice, be “replaced” by another (patched) one (refer to the image opening this article).

Advantages

Can immediately apply critical security patches to the kernel. [2]
Do not have to wait for long-running tasks to complete, for users to log off, or for scheduled downtime. [2]
Control the system’s uptime more and do not sacrifice security or stability. [2]

Disadvantages and limitations

It’s not a general-purpose kernel upgrade mechanism. It is used for applying simple security and bug fix updates when rebooting the system is not immediately possible. [2]
Not all patches are available through Kpatch. [2]
Can’t use SystemTap or Kprobe tools during or after loading a patch (affecting kernel debugging and development). [2]
Introduces a small amount of latency. [3]
It’s not as straightforward as updating the whole kernel and rebooting.

Overview of Kpatch in RHEL

Before we begin to explore Kpatch, it’s worth mentioning, in case you want to try it yourself, that it only works in Red Hat Enterprise Linux (CentOS doesn’t support it, if you’re asking).

The kernel patches are delivered as kernel modules, inside RPM packages, and everybody with a standard RHEL license have access to them. The patches are specific to a particular kernel version and release. When a kernel live patch RPM is installed, the module is placed in /var/lib/kpatch; it is then loaded into the running kernel, and the replacement functions it contains are targeted for redirection through the Ftrace mechanism, bypassing the original ones.

A Systemd unit is responsible for reloading the live patch modules when the system reboots, if for some reason you don’t want to use this opportunity to upgrade the whole kernel.

Updating a live patch module is simply a matter of updating the respective RPM package. Removing a module, in turn, can be done by uninstalling the RPM package, or by using the kpatch userspace utility. The kpatch Systemd unit can also be disabled, so upon reboot no live patch modules will be loaded.

Hands-on

To show the essentials of Kpatch in a real-world scenario, I’ve deployed a virtual machine with RHEL 8.2, which comes by default with kernel version 4.18.0-193. The steps described here should work about the same for RHEL 7:

1
2
3
4
5
[root@spike-vm-rhel01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 (Ootpa)

[root@spike-vm-rhel01 ~]# uname -a
Linux spike-vm-rhel01 4.18.0-193.el8.x86_64 #1 SMP Fri Mar 27 14:35:58 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

After registering the system with Red Hat (important, otherwise it won’t get any packages), we can search for available kernel live patches:

1
2
3
[root@spike-vm-rhel01 ~]# yum search $(uname -r) | grep ^kpatch
Last metadata expiration check: 3:24:21 ago on Mon 10 Aug 2020 02:51:26 PM -03.
kpatch-patch-4_18_0-193.x86_64 : Live kernel patching module for kernel-4.18.0-193.el8.x86_64

Now it’s just a matter of installing the package:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[root@spike-vm-rhel01 ~]# yum install kpatch-patch-4_18_0-193.x86_64
...
Installing: kpatch-patch-4_18_0-193-1-5.el8.x86_64   1/1
Running scriptlet: kpatch-patch-4_18_0-193-1-5.el8.x86_64   1/1
 
installing /usr/lib/kpatch/4.18.0-193.el8.x86_64/kpatch-4_18_0-193-1-5.ko (4.18.0-193.el8.x86_64)
Created symlink /etc/systemd/system/multi-user.target.wants/kpatch.service → /usr/lib/systemd/system/kpatch.service.
loading patch module: /var/lib/kpatch/4.18.0-193.el8.x86_64/kpatch-4_18_0-193-1-5.ko
...
Complete!

Notice that the running kernel version didn’t change, as reported by uname:

1
2
[root@spike-vm-rhel01 ~]# uname -r
4.18.0-193.el8.x86_64

However, we can confirm that the live patch is loaded with the kpatch userspace utility:

1
2
3
4
5
6
[root@spike-vm-rhel01 ~]# kpatch list
Loaded patch modules:
kpatch_4_18_0_193_1_5 [enabled]

Installed patch modules:
kpatch_4_18_0_193_1_5 (4.18.0-193.el8.x86_64)

Checking for fixed vulnerabilities (CVEs)

Before applying any patches, though, we should check which CVEs they fix, and compare them with the CVEs fixed by a full kernel upgrade, to decide if the live patch will suffice. Let’s check the live patch changelog:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
[root@spike-vm-rhel01 ~]# yum changelog kpatch-patch-4_18_0-193-1-5.el8.x86_64
...
Changelogs for kpatch-patch-4_18_0-193-1-5.el8.x86_64
* Fri Jun 26 12:00:00 AM 2020 Artem Savkov <[email protected]> [1-5.el8]
- Indirect Branch Prediction Barrier is force-disabled when STIBP is unavailable or enhanced IBRS is available. Indirect branch speculation can be enabled after it was force-disabled by the PR_SPEC_FORCE_DISABLE prctl command. [1847404] [1847402] {CVE-2020-10767} {CVE-2020-10768}

* Thu Jun 25 12:00:00 AM 2020 Julien Thierry <[email protected]> [1-4.el8]
- Prevent rogue cross-process SSBD shutdown [1847371] {CVE-2020-10766}

* Mon May 25 12:00:00 AM 2020 Artem Savkov <[email protected]> [1-3.el8]
- use-after-free in block/bfq-iosched.c related to bfq_idle_slice_timer_body [1835531] {CVE-2020-12657}

* Mon May 11 12:00:00 AM 2020 Joe Lawrence <[email protected]> [1-2.el8]
- Workaround kpatch ppc64le leaf functions in previous patch [1827332] {CVE-2020-10711}

* Fri May 08 12:00:00 AM 2020 Joe Lawrence <[email protected]> [1-1.el8]
- netlabel: cope with NULL catmap [1827332] {CVE-2020-10711}

* Fri Apr 10 12:00:00 AM 2020 Joe Lawrence <[email protected]> [0-0.el8]
- An empty patch to subscribe to kpatch stream for kernel-4.18.0-193.el8 [1822309]

Now let’s check the changelog of the full kernel update available:

1
2
3
4
5
6
7
8
9
[root@spike-vm-rhel01 ~]# yum changelog kernel-4.18.0-193.14.3.el8_2|grep CVE
- Reverse keys order for dual-signing (Frantisek Hrbata) [1837433 1837434] {CVE-2020-10713}
- [kernel] Move to dual-signing to split signing keys up better (pjones) [1837433 1837434] {CVE-2020-10713}
- [crypto] pefile: Tolerate other pefile signatures after first (Lenny Szubowicz) [1837433 1837434] {CVE-2020-10713}
- [acpi] ACPI: configfs: Disallow loading ACPI tables when locked down (Lenny Szubowicz) [1852968 1852969] {CVE-2020-15780}
- [firmware] efi: Restrict efivar_ssdt_load when the kernel is locked down (Lenny Szubowicz) [1852948 1852949] {CVE-2019-20908}
- [vfio] vfio/pci: Fix SR-IOV VF handling with MMIO blocking (Alex Williamson) [1837309 1837310] {CVE-2020-12888}
...
[many more CVEs listed]

As you can see, the full kernel update fixes a lot more CVEs, so if Kpatch is worth it depends on a case-by-case analysis.

Uninstallation

Before experimenting with removing a kernel live patch, let’s confirm if they are reapplied after a reboot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
[root@spike-vm-rhel01 ~]# reboot
Connection to spike-vm-rhel01 closed by remote host.
Connection to spike-vm-rhel01 closed.
...
$ ssh root@spike-vm-rhel01

[root@spike-vm-rhel01 ~]# kpatch list
Loaded patch modules:
kpatch_4_18_0_193_1_5 [enabled]

Installed patch modules:
kpatch_4_18_0_193_1_5 (4.18.0-193.el8.x86_64)

Nice! So now let’s try the removal steps:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
[root@spike-vm-rhel01 ~]# yum remove kpatch-patch-4_18_0-193.x86_64
...
Running transaction
  Preparing:    1/1
  Erasing:   kpatch-patch-4_18_0-193-1-5.el8.x86_64    1/1
  Running scriptlet:   kpatch-patch-4_18_0-193-1-5.el8.x86_64    1/1
uninstalling kpatch-4_18_0-193-1-5.ko (4.18.0-193.el8.x86_64)

  Verifying:    kpatch-patch-4_18_0-193-1-5.el8.x86_64    1/1
Installed products updated.

Removed:
  kpatch-patch-4_18_0-193-1-5.el8.x86_64

Complete!

[root@spike-vm-rhel01 ~]# kpatch list
Loaded patch modules:
kpatch_4_18_0_193_1_5 [enabled]

Installed patch modules:

Piece of cake! Another option would be to run:

1
2
[root@spike-vm-rhel01 ~]# kpatch uninstall kpatch_4_18_0_193_1_5
uninstalling kpatch_4_18_0_193_1_5 (4.18.0-193.el8.x86_64)

Or:

1
2
[root@spike-vm-rhel01 ~]# systemctl disable kpatch.service
Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

A final note: the kernel will remain patched until the next reboot, so as opposed to applying the patch, which doesn’t need a reboot, removing the patch does need so. You can confirm that by running kpatch list: even though there won’t be any installed patch modules, it will still appear as loaded, until the next reboot.

Conclusion

Kernel live patching, as provided by Red Hat in its Enterprise Linux (RHEL), may be a convenient way to apply critical patches in situations where servers can’t be rebooted. However, not all kernel patches are available through live patches, so a careful analysis of the vulnerable CVEs vs. the fixed CVEs in the live patches must be performed beforehand.

If, after applying live patches, a window of opportunity for a server reboot appears, I recommend using it to update the whole kernel. The old patched kernel can be left as-is, or uninstalled afterwards, together with any live patches applied to it.