A Dive Deep into Kernel Parameters — Part 1: Kernel Boot Parameters

JustAboutCloud on 2020-06-27

A Dive Deep into Kernel Parameters — Part 1: Kernel Boot Parameters

Scope of the Article

EC2 (Elastic Compute Cloud) is the most used service by AWS because it’s reliable, scalable and flexible. EC2 is also a back-bone of the AWS as it is being used directly or indirectly in many other AWS Services. For most of the part, the public AMI’s provided by Amazon and other large vendors are already vetted and rigorously tested for production use. However sometimes here come a need when we may have to tweak the behavior of the EC2 Instances by adjusting their kernel parameters.

In this series, I will discuss about what are the kernel parameters (including kernel boot parameters) and how to change the kernel parameters for Amazon Linux 1/2, RHEL 5/6/7/8, CentOS 6/7, SLES 12/15 and Ubuntu 14.04/16.04/18.04/20.04 on AWS EC2 Instances. It is the first part of this dive deep series about kernel parameters. Let’s get started.

Introduction to Kernel Parameters

A Linux kernel is a complex piece of software and like any other software in the world, it exhibit a specific behavior based on the default parameters specified in the kernel code. A few of such examples are how kernel manages the disk, memory or how should the system boot. These parameters controls how the Linux system behave and can be edited during kernel run-time as well as on boot to tweak the system behavior.

There are 3 ways you can add or edit the kernel parameters, depending on your case, to control the kernel’s behavior:

When booting the kernel (kernel boot parameters), these are invoked through boot loader.
Through /proc and /sys pseudo filesystems using “sysctl” during kernel run-time.
When compiling (or re-compiling) the kernel and it’s subsystems (such as initrd)

Kernel Boot Parameters

During a Linux system boots it need a few boot time parameters to supply the kernel information about hardware. Which is why we know it as kernel boot parameters. Specifically we deal with boot parameters when the kernel wouldn’t be able to determine a specific parameter on it’s own, so we add it. Sometimes we edit the parameters to override the values kernel detect for a specific parameters.

During the Linux kernel boot sequence a boot loader holds the responsibility to load and kick off the kernel execution. So in order to be able to pass these boot parameters, we use boot loader, such as GRUB.

The kernel boot parameters are specified in a list of string, separated by spaces, with the form as: name[=value_1][,value_2]…[,value_10]

At present, kernel code can handle as maximum of 10 comma separated values for a boot parameter. However you can re-use the same keyword.

For example, I have shared the Amazon Linux 1 instance grub configuration. In this, I have modified the “console” parameter to have 2 values (Can have up to 10 values)

kernel /boot/vmlinuz-4.14.77-70.59.amzn1.x86_64 root=LABEL=/ console=ttyS0,tty1 selinux=0

We can provide the same boot parameters to kernel like below example as well. Where rather than specifying the “console” values as one parameter, we re-used the console to specify another value. Here each parameter “console” can have 10 values, if needed in complicated situations to overcome previous limitation.

kernel /boot/vmlinuz-4.14.77-70.59.amzn1.x86_64 root=LABEL=/ console=ttyS0 console=tty1 selinux=0

Now there are more than 1 way to modify the kernel boot parameters. Also all Linux distributions mentioned earlier have somewhat similar steps to modify the boot parameters expect Ubuntu.

1. The Universal Method — Update kernel boot parameters through grubby Utility

grubby is a command line tool for updating and displaying information about the boot loader configuration files. grubby works well with all Linux Distributions including Amazon Linux 1/2, RHEL 5/6/7/8, CentOS 6/7 and SLES 12/15. However grubby doesn’t work with Ubuntu Systems.

PS: grubby is installed on all Linux distributions mentioned above by default except SLES. You can find the instructions to install grubby under “Add repository and install manually” section here.

Advantages of grubby

Definitely working with utility tool as grubby has many advantages including simplicity, consistent way across many distributions. Using grubby also means that you don’t require you to have deep knowledge of grub configuration files. For example, you can get the effective kernel with just a command as:

# grubby --default-kernel
/boot/vmlinuz-4.14.181-108.257.amzn1.x86_64

Notice that this is the current kernel version. You will get same from “uname -r” (current kernel version) or from the default menu entry of the grub configuration. You can get more information (as you would get from the grub configuration file) using -info option as:

# grubby --info /boot/vmlinuz-$(uname -r) 
index=0
kernel=/boot/vmlinuz-4.14.181-108.257.amzn1.x86_64
args="console=tty1,ttyS0 selinux=0 nvme_core.io_timeout=4294967295" 
root=LABEL=/
initrd=/boot/initramfs-4.14.181-108.257.amzn1.x86_64.img

Try “grubby -info ALL” as well. Make yourself more familiar with the other options using “grubby -help”

During booting process, the boot loader pass boot parameters to Linux kernel in memory buffer called the kernel command line. A file /proc/cmdline in pseudo filesystem /proc also hold a copy of kernel boot parameters. We can verify it with the following command:

# cat /proc/cmdline 
root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295

Working with grubby

In order to add/edit the kernel parameters using the grubby, we need to specify parameters using -args=args along with the -update-kernel=default-kernel-path option. For example, I have added the “clocksource=tsc tsc=reliable” parameters. “clocksource=tsc” parameter would mean to set clocksource to tsc (clocksource=tsc) if not already set. “ tsc=reliable” parameter skip all the TSC stability checks during boot time.

# grubby --args="clocksource=tsc tsc=reliable" --update-kernel /boot/vmlinuz-$(uname -r)

PS: Use -remove-arguments to delete the existing parameters.

Reboot the system, confirm that parameters “clocksource=tsc tsc=reliable” are added:

# reboot 
# cat /proc/cmdline 
root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 clocksource=tsc tsc=reliable

Or we can verify it the grubby way as well:

# grubby --info /boot/vmlinuz-$(uname -r)

You can further verify that the clocksource “tsc” is applied with the following command:

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource 
tsc

That’s all from the grubby side of things.

2. Linux Distributions with Legacy GRUB as boot loader

Amazon Linux 1 along with RHEL6.X, RHEL5.X and CentOS6 uses GRUB (also known as Legacy GRUB) as a boot loader. You can find about the version in such distributions as:

# grub-install -v 
grub-install (GNU GRUB 0.97)

The grub configuration file reside in (/boot/grub/grub.conf) and have 2 symbolic links (/etc/grub.conf & /boot/grub/menu.lst) to it. We need to edit this grub configuration to add additional kernel boot parameters. However to do this effectively, let’s understand the basic of grub configuration first. Below is the grub configuration from Amazon Linux 1 EC2 instance:

Understanding GRUB Configuration

# cat /etc/grub.conf 
# created by imagebuilder 
default=0 
timeout=0 
hiddenmenu

title Amazon Linux 2018.03 (4.14.181-108.257.amzn1.x86_64) 
root (hd0,0) 
kernel /boot/vmlinuz-4.14.181-108.257.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 
initrd /boot/initramfs-4.14.181-108.257.amzn1.x86_64.img

title Amazon Linux 2018.03 (4.14.177-107.254.amzn1.x86_64) 
root (hd0,0) 
kernel /boot/vmlinuz-4.14.177-107.254.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 
initrd /boot/initramfs-4.14.177-107.254.amzn1.x86_64.img

Now as you can see there are more than 1 menu entries in the grub configuration. Menu entry holds information kernel and initrd image locations along with the parameters. Each entry start with “title..” and have the numeric order starting from 0. The value of “default” in the configuration determine the current kernel image to use for booting the serve=. In the above config, the value is “0” (default=0), which means the first menu entry (4.14.181–108.257.amzn1.x86_64) is effective. Let’s look at the kernel line from the configuration for a moment. All the name[=value_1] after kernel image location (in bold & italic — separated by space) are boot parameters.

kernel /boot/vmlinuz-4.14.181–108.257.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295

It is important to know it so that you add the parameter to correct menu entry.

Working with GRUB Configuration

Now edit the configuration file (/etc/grub.conf) with a choice of your editor and add/edit the parameter you want. Like earlier, I have added the same “clocksource=tsc tsc=reliable” parameters.

kernel /boot/vmlinuz-4.14.181–108.257.amzn1.x86_64 root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 clocksource=tsc tsc=reliable

Reboot the system for it to take effect. We can validate it the same way we did it in the Option 1:

# reboot 
# cat /proc/cmdline 
root=LABEL=/ console=tty1 console=ttyS0 selinux=0 nvme_core.io_timeout=4294967295 clocksource=tsc tsc=reliable

Though this option works perfectly fine and is still widely used. The disadvantage of this option is that it requires you to have the basic understanding of the grub configuration. Also you should know what menu entry is effective and where to change the parameters.

3. Linux Distributions with GRUB2 as boot loader

Amazon Linux 2 along with RHEL7.X, CentOS7, SLES15 and SLES12 uses GRUB2 as a boot loader. You can find about the version in such distributions as:

# grub2-install -V 
grub2-install (GRUB) 2.03

For systems with GRUB2 as their default boot loader, the grub2 configuration resides in the “/boot/grub2/grub.cfg”. Only and only grub should maintain “/boot/grub2/grub.cfg”. This file is automatically generated by “grub2-mkconfig” using templates from /etc/grub.d folders and /etc/default/grub file. We’ll talk about it in a minute.

Understanding GRUB2 Configuration

PS: If you’re running instances from ARM family mentioned here and here, The grub configuration must be somewhere under “/boot/efi/EFI/” folder. For example “/boot/efi/EFI/redhat/grub.cfg” is for RHEL7 running ARM.

The kernel boot parameters are defined in /etc/default/grub in Linux distributions with GRUB2. Let’s have a look from Amazon Linux 2 Instance:

# cat /etc/default/grub 
GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0" 
GRUB_TIMEOUT=0 
GRUB_DISABLE_RECOVERY="true"

“GRUB_CMDLINE_LINUX_DEFAULT” holds the kernel boot (command line) parameters. It is the line where we need to edit grub configuration to play with boot parameters.

GRUB_CMDLINE_LINUX_DEFAULT=”console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295

Working with GRUB2 Configuration

Edit the default grub configuration file (/etc/default/grub.conf) with a choice of your editor. Add/Edit the parameter you want for GRUB_CMDLINE_LINUX_DEFAULT line. For example, I have added the “ systemd.log_level=debug systemd.log_target=console “ parameters. These additional parameter would enable systemd debug mode and redirect them to console. So after update, my entry would be something like this in /etc/default/grub file:

GRUB_CMDLINE_LINUX_DEFAULT=”console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 nvme_core.io_timeout=4294967295 rd.emergency=poweroff rd.shell=0 systemd.log_level=debug systemd.log_target=console”

Now, in order to auto-generate the grub2 configuration based on the parameters we’ve added. We need to run “grub2-mkconfig” with the path of grub2 configuration as:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the system and validate the changes from /proc/cmdline as we did earlier.

4. For Ubuntu 14.04 / 16.04 / 18.04 / 20.04 EC2 Instances

GRUB 2 is the default boot loader and manager for Ubuntu since version 9.10. This is why the process of changing the kernel boot parameters is almost similar as other GRUB 2 Linux Distributions as we discussed above.

Now the location of grub configuration file for Ubuntu is /etc/default/grub.d/50-cloudimg-settings.cfg.

In order to add the parameters, edit /etc/default/grub.d/50-cloudimg-settings.cfg and modify the parameter needed on GRUB_CMDLINE_LINUX_DEFAULT same as we did above.

Once done, run update-grub command to re-generate a appropriate grub configuration:

# update-grub

PS: update-grub is the alternative or say the truncated version of grub-mkconfig -o /boot/grub/grub.cfg

Wrap Up

That’s all from my side. I hope you find this article valuable. I will soon come back with the second article, where we’ll talk about the kernel run-time parameters and many more. Any suggestions on this article are welcome.

Most importantly If you have faced an issue following the article, please comment or contact us

Originally published at https://justabout.cloud on June 16, 2020.