Avoid Death, Taxes and Linux Server Reboots – Linux Kernel Updates, 3 Different Ways
Linux kernel updates are a fact of life—as dull as taxes and only slightly less inconvenient than death. Newly discovered security vulnerabilities in the Linux kernel seem to appear with monotonous regularity. In most but not all cases, the patches needed to fix them follow swiftly after. There is work involved in installing the latest Linux kernel security patches, and danger if you delay—leave it too long and threat actors might take advantage of the period of vulnerability.
Linux’s popularity as a platform for web hosting services, standalone web servers, and web applications has made it a prime target for hackers using techniques such as remote code execution (RCE), cross-site scripting (XSS), and denial of service (DoS) attacks. Keeping a system up to date with the latest operating system and application software patches is one of the most effective ways to strengthen system security and protect against these kinds of cyber threats. For Linux, the OS is difficult to keep safe, because most kernel upgrades and security patches require a system reboot.
This article explains how to update Linux kernels, without rebooting. I cover three different methods for some of the most popular Linux kernels. They are:
1) on the command line;
3) with rebootless live kernel patching tools: Oracle Ksplice Uptrack, Canonical Livepatch, Red Hat’s Kpatch, SUSE Kgraft (SLE Live Patching), and CloudLinux’s KernelCare.
1. Command Line
This is the standard way to do an update from the Linux distribution vendor’s repository, and the one most likely to be found in the documentation.
On Ubuntu, you can use these commands in a terminal.
sudo apt-get upgrade linux-image-generic sudo reboot
On Debian, it would be this.
sudo apt-get upgrade kernel sudo reboot
If you want to do a CentOS kernel update, one for Red Hat Enterprise Linux (RHEL), or for any other RPM-based distribution, use this.
sudo yum update kernel sudo reboot
So far, so easy. But the kernel patch won’t take effect until you reboot.
Reboot? Yes. You have to kick off your users, save your files, close down your processes and possibly make a lot of people very unhappy (for example, anyone in the middle of a purchase). And then you have to wait for your Linux server to come up again and recover its state. How long does yours take to bounce? Will customers notice? Even if they won’t, you have to notify them first.
This is one reason why many system administrators defer patch installation, avoiding downtime but compromising system security.
+ Pros: No installation.
– Cons: Not automated. Reboot required.
2. kexec: Quicker reboots
You can make the rebooting step quicker by using
kexec. This Linux kernel system call lets you boot into a new kernel, skipping the boot loader and hardware initialization phases, and significantly shortening your reboot time.
To use it, you first need to install
sudo apt-get install kexec-tools
You’ll see a configuration window, something like this:
sudo yum install kexec-tools
Next, you install a new kernel. List them, then choose the one you want.
sudo yum update kernel sudo rpm -qa kernel
The output should be something like this.
Now boot into your chosen version.
sudo kexec -l /boot/vmlinuz-3.10.0-862.3.2.el7.x86_64 \ --initrd=/boot/initramfs-3.10.0-862.3.2.el7.x86_64.img \ --reuse-cmdline sudo sync; sudo umount -a; sudo kexec -e
You can use the next command if you have no patience (but see the warning below before you do so).
sudo kexec -e
WARNING! This is like power-cycling your server without giving the reboot command time to properly kill your processes, synchronize your file caches and unmount your file systems. It can cause data loss or corruption.
+ Pros: Faster boot.
– Cons: One-time install. More finger-work (and more potential for error unless you script it well).
3. Update your kernel without rebooting
Yes, you read that correctly. There is a way to do it.
There are times when security patching is super-critical, but so are the processes that stop when you reboot. If you’re running an ‘always-on’ or ‘high-availability’ system, you’ll already be familiar with this dilemma.
Rebootless kernel updating lets you ‘have your cake and eat it (too)’. It is not a replacement for full kernel upgrades, as it only applies patches for security vulnerabilities or critical bug fixes. But, in many cases, this is all you need, and it is possible to keep a server safe and running for years between reboots using these methods.
A number of leading Linux vendors offer rebootless kernel updates. The one you choose depends on the distribution you run. In the remainder of this article we’ll talk about the following products:
Ksplice by Oracle (for Oracle Linux updates, Ksplice Uptrack for enterprise)
Kpatch by Red Hat (for RHEL kernel updates and CentOS updates)
Livepatch by Canonical (for Ubuntu kernel updates)
Kgraft by SUSE (for SUSE updates only)
CloudLinux KernelCare (for multiple OSes)
Ksplice was the first commercially-available implementation of rebootless kernel updating. Ksplice Inc. was eventually acquired by Oracle so that now it is only available (unsurprisingly) on Oracle Linux and RedHat Enterprise Linux distributions, and the deployment needs a license from Oracle.
To deploy it, run:
sudo wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc sudo sh install-uptrack-oc --autoinstall
Note, there is no reboot command, and you only need to run the install script once in the lifetime of the server. After that, the Uptrack service will automatically detect new kernel updates and deploy them for you. There’s no scheduling, no downtime, and nothing more to do.
+ Pros: No reboot required. Automatic updates.
– Cons: Only for Oracle distributions. Requires a support license.
This is Canonical’s technology for (guess what?) live-patching kernels. (Canonical is the company behind the popular Ubuntu Linux distribution.) You can even create your own patches, although it can be difficult, time-consuming work. (Some vendors will create an Ubuntu upgrade kernel for you, for a fee.)
The service is available for Ubuntu 16.04 and later, and RHEL 7.x (beta).
It’s deployed like this.
sudo snap install canonical-livepatch sudo canonical-livepatch enable [TOKEN]
Note: The Canonical Livepatch service is free for up to 3 machines for Ubuntu Community members. You can sign up for a token here.
+ Pros: No reboot required. Automatic kernel updates.
– Cons: Non-trivial custom kernel patches. Limit to the number of updatable hosts (additional hosts for a fee).
Red Hat Kpatch
This is Red Hat’s own kernel patching tool. It was announced in 2014 and has been ported to work on others in the same family (Fedora, CentOS) as well as for some Debian-based systems (Ubuntu, Gentoo).
Here’s an example of deploying it on RHEL 7.
sudo yum install kpatch sudo yum install kpatch-patch-X.X.X.el7.x86_64.rpm
Unlike Ubuntu’s Livepatch service or Oracle’s Ksplice, it’s not automatic, and you must manually check for and install each kernel patch as it becomes available.
+ Pros: No reboot required.
– Cons: Not automated. Limited distributions.
Developed and announced at almost the same time as Red Hat’s solution, Kgraft is SUSE’s live patching offering (known as SUSE Linux Enterprise Live Patching). It’s only for SUSE’s own Linux Enterprise Server 12, and comes preinstalled, so there’s really nothing to do (except pay for it). It works on a different principle to most other approaches but has a feature-set comparable with Kpatch.
+ Pros: No installation needed. No reboot required.
– Cons: Single platform support. Commercial (but there is a generous 60-day free trial).
Also launched in 2014, KernelCare’s Linux kernel live patching service stands out among the kernel patching solutions in its OS coverage, which includes CentOS, RHEL, Oracle Linux, Debian, Ubuntu and others. And like Oracle’s solution, KernelCare supports the older 2.6.32 kernels from RHEL 6.
Here’s how to install KernelCare:
sudo wget -qq -O - \ https://repo.cloudlinux.com/kernelcare/kernelcare_install.sh | bash sudo /usr/bin/kcarectl --register <your key>
KernelCare is an ‘install and forget’ solution. Once installed, KernelCare automatically downloads and applies new kernel security patches, without rebooting the server.
But in contrast to its closest competitors, KernelCare can handle some of the more complex patches for vulnerabilities such as Meltdown (CVE-2017-5754), Spectre (CVE-2017-5753 & CVE-2017-5715), and more recently, the Linux kernel buffer overflow flaw known, romantically, as Mutagen Astronomy (CVE-2018-14634). KernelCare supports custom patch configurations, fixed-date patches, delayed patches, and rebootless rollbacks, i.e. patch removals.
Like the other vendors considered here, KernelCare also springs from a good blood line—its creator is CloudLinux, the leading web hosting Linux-based OS vendor.
+ Pros: Easy install. No reboot required. Wide OS coverage (including one of the most popular Linux flavors, Ubuntu). Supports custom and fixed-date patching. Good support and industry know-how from CloudLinux.
– Cons: Commercial (but there is a free, 30-day trial). There is also a free KernelCare license for non-profit organizations.
If your server is non-critical and can endure a period offline, updating the kernel is relatively painless using the standard tools on the command line.
If you’re running an always-on system, (i.e. you can’t or won’t reboot), take a look at live kernel patching solutions. Of these, there are three kinds: