Run one command, get a QEMU or gem5 Buildroot BusyBox virtual machine built from source with several minimal Linux kernel 4.15 module development example tutorials with GDB and KGDB step debugging and minimal educational hardware models. "Tested" in x86, ARM and MIPS guests, Ubuntu 17.10 host.
- 1. Getting started
- 2. GDB step debugging
- 3. KGDB
- 4. gdbserver
- 5. Architecture
- 6. init
- 7. modprobe
- 8. X11
- 9. Count boot instructions
- 10. initrd
- 11. ftrace
- 12. QEMU user mode
- 13. Snapshot
- 14. Educational hardware models
- 15. gem5
- 16. Failed action
- 17. Insane action
- 18. Conversation
Reserve 12Gb of disk and run:
git clone https://github.com/cirosantilli/linux-kernel-module-cheat cd linux-kernel-module-cheat ./configure && ./build && ./run
The first build will take a while (GCC, Linux kernel), e.g.:
-
2 hours on a mid end 2012 laptop
-
30 minutes on a high end 2017 desktop
If you don’t want to wait, you could also try to compile the examples and run them on your host computer as explained on at Run on host, but as explained on that section, that is dangerous, limited, and will likely not work.
After QEMU opens up, you can start playing with the kernel modules:
root insmod /hello.ko insmod /hello2.ko rmmod hello rmmod hello2
This should print to the screen:
hello init hello2 init hello cleanup hello2 cleanup
which are printk messages from init and cleanup methods of those modules.
All available modules can be found in the kernel_module directory.
head kernel_module/modulename.c
Many of the modules have userland test scripts / executables with the same name as the module, e.g. form inside the guest:
/modulename.sh /modulename.out
The sources of those tests will further clarify what the corresponding kernel modules does. To find them on the host, do a quick:
git ls-files | grep modulename
If you make changes to the kernel modules or most configurations tracked on this repository, you can just use again:
./build ./run
and the modified files will be rebuilt.
If you change any package besides kernel_module, you must also request those packages to be reconfigured or rebuilt with extra targets, e.g.:
./build -t linux-reconfigure -t host-qemu-reconfigure
Those aren’t turned on by default because they take quite a few seconds.
Linux and QEMU rebuilds are so common that we have dedicated shortcut flags for them:
./build -l -q
You did something crazy, and nothing seems to work anymore?
All builds are stored under buildroot/,
The most coarse thing you can do is:
cd buildroot git checkout -- . git clean -xdf .
To only nuke one architecture, do:
rm -rf buildroot/output.x86_64~
Only nuke one one package:
rm -rf buildroot/output.x86_64~/build/host-qemu-custom ./build -q
This is sometimes necessary when changing the version of the submodules, and then builds fail. We should try to understand why and report bugs.
The root filesystem is persistent across:
./run date >f sync poweroff
then:
./run cat f
This is particularly useful to re-run shell commands from the history of a previous session with Ctrl + R.
However, when you do:
./build
the disk image gets overwritten by a fresh filesystem and you lose all changes.
Remember that if you forcibly turn QEMU off without sync or poweroff from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved.
We use printk a lot, and it shows on the QEMU terminal by default. If that annoys you (e.g. you want to see stdout separately), do:
dmesg -n 1
See also: https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console
You can scroll up a bit on the default TTY with:
Shift + PgUp
but I never managed to increase that buffer:
The superior alternative is to use text mode or a telnet connection.
Show serial console directly on the current terminal, without opening a QEMU window:
./run -n
To quit QEMU forcefully, just use Ctrl + C as usual.
This mode is very useful to:
-
get full panic traces when you start making the kernel crash :-) See also: https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic
-
copy and paste commands and stdout output to / from host
-
have a large scroll buffer, and be able to search it, e.g. by using GNU
screenon host
Limitations:
-
TODO: Ctrl + C kills the emulator, and not sent to guest processes. See:
It is also hard to enter the monitor for the same reason:
+ Our workaround is:
+
./qemumonitor
+ I think the problem was reversed in older QEMU versions: https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui/1211516#1211516
+
This is however fortunate when running QEMU with GDB, as the Ctrl + C reaches GDB and breaks.
* Very early kernel messages such as early console in extract_kernel only show on the GUI, since at such early stages, not even the serial has been setup.
When debugging a module, it becomes tedious to wait for build and re-type:
root /modulename.sh
every time.
Instead, you can either run them from a minimal init:
./run -e 'init=/eval.sh - lkmc_eval="insmod /hello.ko;/poweroff.out"' -n
or run them at the end of the BusyBox init, which does things like setting up networking:
./run -e '- lkmc_eval="insmod /hello.ko;wget -S google.com;poweroff.out;"'
or add them to a new init.d entry:
cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99 vim S99 ./build ./run
and they will be run automatically before the login prompt.
S99 is a git tracked convenience symlink to the gitignored rootfs_overlay/etc/init.d/S99
Scripts under /etc/init.d are run by /etc/init.d/rcS, which gets called by the line ::sysinit:/etc/init.d/rcS in /etc/inittab.
We try to use the latest possible kernel major release version.
In QEMU:
cat /proc/version
or in the source:
cd linux git log | grep -E ' Linux [0-9]+\.' | head
Build configuration can be observed in guest with:
zcat /proc/config.gz
or on host:
cat buildroot/output.*~/build/linux-custom/.config
Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.
This allows us to control the behaviour of the kernel without rebuilding anything.
With QEMU, QEMU itself acts as the bootloader, and provides the -append option and we expose it through ./run -e, e.g.:
./run -e 'foo bar'
Then inside the host, you can check which options were given with:
cat /proc/cmdline
They are also printed at the beginning of the boot message:
dmesg | grep "Command line"
See also:
The arguments are documented in the kernel documentation: https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html
When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:
-
GRUB configuration files: https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter
-
Raspberry pi
/boot/cmdline.txton a magic partition: https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly
Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.
We have not managed to track this problem down yet, but the following workaround always works:
Ctrl + Shift + U Ctrl + C root
This started happening when we switched to building QEMU through Buildroot, and has not been observed on later Ubuntu.
Using text mode is another workaround if you don’t need GUI features.
When you start interacting with QEMU hardware, it is useful to see what is going on inside of QEMU itself.
This is of course trivial since QEMU is just an userland program on the host, but we make it a bit easier with:
./run -D
Then you could:
b edu_mmio_read c
And in QEMU:
/pci.sh
Just make sure that you never click inside the QEMU window when doing that, otherwise you mouse gets captured forever, and the only solution I can find is to go to a TTY with Ctrl + Alt + F1 and kill QEMU.
You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.
To GDB step debug the Linux kernel, first run:
./run -d
If you want to break immediately at a symbol, e.g. start_kernel of the boot sequence, run on another shell:
./rungdb start_kernel
Now QEMU will stop there, and you can use the normal GDB commands:
l n c
To skip the boot, run just:
./rungdb
and when you want to break, do Ctrl + C from GDB.
To have some fun, you can first run inside QEMU:
/count.sh
which counts to infinity to stdout, and then in GDB:
Ctrl + C break sys_write continue continue continue
And you now control the counting from GDB.
See also:
O=0 is an impossible dream, O=2 being the default: https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0 So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.
Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.
So we cannot set the breakpoints before insmod.
However, the Linux kernel GDB scripts offer the lx-symbols command, which takes care of that beautifully for us:
./run -d ./rungdb
In QEMU:
insmod /timer.ko
In GDB, hit Ctrl + C, and note how it says:
scanning for modules in /home/ciro/bak/git/linux-kernel-module-cheat/buildroot/output.x86_64~/build/linux-custom loading @0xffffffffc0000000: ../kernel_module-1.0//timer.ko
That’s lx-symbols working! Now simply:
b lkmc_timer_callback c c c
and we now control the callback from GDB!
Just don’t forget to remove your breakpoints after rmmod, or they will point to stale memory locations.
TODO: why does break work_func for insmod kthread.ko not break the first time I insmod, but breaks the second time?
Useless, but a good way to show how hardcore you are. From inside QEMU:
insmod /fops.ko cat /proc/modules
This will give a line of form:
fops 2327 0 - Live 0xfffffffa00000000
And then tell GDB where the module was loaded with:
Ctrl + C add-symbol-file ../kernel_module-1.0/fops.ko 0xfffffffa00000000
TODO: why can’t we break at early startup stuff such as:
./rungdb extract_kernel ./rungdb main
GDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb
However this is failing for us:
-
some symbols are not visible to
calleven thoughbsees them -
for those that are,
callfails with an E14 error
E.g.: if we break on sys_write on /count.sh:
>>> call printk(0, "asdf") Could not fetch register "orig_rax"; remote failure reply 'E14' >>> b printk Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824. >>> call fdget_pos(fd) No symbol "fdget_pos" in current context. >>> b fdget_pos Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations) >>>
even though fdget_pos is the first thing sys_write does:
581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582 size_t, count)
583 {
584 struct fd f = fdget_pos(fd);
See also: cirosantilli#19
KGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support.
It is useless with QEMU since we already have full system visibility with -gdb, but this is a good way to learn it.
Cheaper than JTAG (free) and easier to setup (all you need is serial), but with less visibility as it depends on the kernel working, so e.g.: dies on panic, does not see boot sequence.
Usage:
./run -k ./rungdb -k
In GDB:
c
In QEMU:
/count.sh & /kgdb.sh
In GDB:
b sys_write c c c c
And now you can count from GDB!
If you do: b sys_write immediately after ./rungdb -k, it fails with KGDB: BP remove failed: <address>. I think this is because it would break too early on the boot sequence, and KGDB is not yet ready.
See also:
In QEMU:
/kgdb-mod.sh
In GDB:
lx-symbols ../kernel_module-1.0/ b fop_write c c c
and you now control the count.
TODO: if I -ex lx-symbols to the gdb command, just like done for QEMU -gdb, the kernel oops. How to automate this step?
If you modify runqemu to use:
-append kgdboc=kbd
instead of kgdboc=ttyS0,115200, you enter a different debugging mode called KDB.
Usage: in QEMU:
[0]kdb> go
Boot finishes, then:
/kgdb.sh
And you are back in KDB. Now you can:
[0]kdb> help [0]kdb> bp sys_write [0]kdb> go
And you will break whenever sys_write is hit.
The other KDB commands allow you to instruction steps, view memory, registers and some higher level kernel runtime data.
But TODO I don’t think you can see where you are in the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source).
Step debug userland processes to understand how they are talking to the kernel.
In guest:
/gdbserver.sh /myinsmod.out /hello.ko
In host:
./rungdbserver kernel_module-1.0/user/myinsmod.out
You can find the executable with:
find buildroot/output.x86_64~/build -name myinsmod.out
TODO: automate the path finding:
-
using the executable from under
buildroot/output.x86_64~/targetwould be easier as the path is the same as in guest, but unfortunately those executables are stripped to make the guest smaller.BR2_STRIP_none=yshould disable stripping, but make the image way larger. -
outputx86_64~/staging/would be even better thantarget/as the docs say that this directory contains binaries before they were stripped. However, only a few binaries are pre-installed there by default, and it seems to be a manual per package thing.E.g.
pciutilshas forlspci:define PCIUTILS_INSTALL_STAGING_CMDS $(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \ PREFIX=$(STAGING_DIR)/usr SBINDIR=$(STAGING_DIR)/usr/bin \ install install-lib install-pcilib endefand the docs describe the
*_INSTALL_STAGINGper package config, which is normally set for shared library packages.Feature request: https://bugs.busybox.net/show_bug.cgi?id=10386
An implementation overview can be found at: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-mips-elf-with-qemu-toolchain/16214#16214
As usual, different archs work with:
./rungdbserver -a arm kernel_module-1.0/user/myinsmod.out
BusyBox executables are all symlinks, so if you do on guest:
/gdbserver.sh ls
on host you need:
./rungdbserver busybox-1.26.2/busybox
Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with:
b open c
Or simply by stepping into calls:
s
This is made possible by the GDB command:
set sysroot ${buildroot_out_dir}/staging
which automatically finds unstripped shared libraries on the host for us.
QEMU -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.
The only use case I can see for this is to debug the init process (and have fun), otherwise, why wouldn’t you just use gdbserver? Known limitations of direct userland debugging:
-
the kernel might switch context to another process, and you would enter "garbage"
-
TODO step into shared libraries. If I attempt to load them explicitly:
(gdb) sharedlibrary ../../staging/lib/libc.so.0 No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.
since GDB does not know that libc is loaded.
Custom init process:
-
Shell 1:
./run -d -e 'init=/sleep_forever.out' -n
-
Shell 2:
./rungdb-user kernel_module-1.0/user/sleep_forever.out main
BusyBox custom init process:
-
Shell 1:
./run -d -e 'init=/bin/ls' -n
-
Shell 2:
./rungdb-user -h busybox-1.26.2/busybox ls_main
This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".
BusyBox default init process:
-
Shell 1:
./run -d -n
-
Shell 2:
./rungdb-user -h busybox-1.26.2/busybox init_main
This cannot be debugged in another way without modifying the source, or /sbin/init exits early with:
"must be run as PID 1"
Non-init process:
-
Shell 1
./run -d -n
-
Shell 2
./rungdb-user kernel_module-1.0/user/sleep_forever.out Ctrl + C b main continue
-
Shell 1
/sleep_forever.out
This is of least reliable setup as there might be other processes that use the given virtual address.
The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware.
To use arm instead of x86 for example:
./build -a arm ./run -a arm
Debug:
./run -a arm -d # On another terminal. ./rungdb -a arm
Known quirks of the supported architectures are documented in this section.
TODOs:
-
only managed to run in the terminal interface (but weirdly a blank QEMU window is still opened)
-
GDB not connecting to KGDB. Possibly linked to
-serial stdio. See also: https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm -
/poweroff.outdoes not exit QEMU nor gem5, the terminal just hangs: https://stackoverflow.com/questions/31990487/how-to-cleanly-exit-qemu-after-executing-bare-metal-program-without-user-interveA blunt resolution for QEMU is on host:
pkill qemu
On gem5, it is possible to use the
m5instrumentation from guest as a good workaround:m5 exit
-
GDB step debugging of kernel modules broke at some point. This happens at 6420c31986e064c81561da8f2be0bd33483af598 on kernel v4.15, 6b0f89a8b43e8d33d3a3a436ed827f962da3008a v4.14 and 5ad68edd000685c016c45e344470f2c1867b8e39 v4.12 and also if kernel 4.9.6 is checked out. So maybe it was never working in the first place, but we never noticed?
Just afte GDB connects, we get the following message from the kernel GDB Python scripts:
loading vmlinux
Traceback (most recent call last):
File "/home/ciro/bak/git/linux-kernel-module-cheat/buildroot/output.arm~/build/linux-custom/scripts/gdb/linux/symbols.py", line 163, in invoke
self.load_all_symbols()
File "/home/ciro/bak/git/linux-kernel-module-cheat/buildroot/output.arm~/build/linux-custom/scripts/gdb/linux/symbols.py", line 150, in load_all_symbols
[self.load_module_symbols(module) for module in module_list]
File "/home/ciro/bak/git/linux-kernel-module-cheat/buildroot/output.arm~/build/linux-custom/scripts/gdb/linux/symbols.py", line 110, in load_module_symbols
module_name = module['name'].string()
File "/home/ciro/bak/git/linux-kernel-module-cheat/buildroot/output.arm~/host/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9f in position 2: invalid start byte
Error occurred in Python command: 'utf8' codec can't decode byte 0x9f in position 2: invalid start byte
+
and then after inserting the module, symbols are not found, presumably because lx-symbols never ran.
As usual, we use Buildroot’s recommended QEMU setup QEMU aarch64 setup:
This makes aarch64 a bit different from arm:
-
uses
-M virt. https://wiki.qemu.org/Documentation/Platforms/ARM explains:Most of the machines QEMU supports have annoying limitations (small amount of RAM, no PCI or other hard disk, etc) which are there because that’s what the real hardware is like. If you don’t care about reproducing the idiosyncrasies of a particular bit of hardware, the best choice today is the "virt" machine.
-M virthas some limitations, e.g. I could not pass-drive if=scsias forarm, and so Snapshot fails. -
uses initramfs, so thre is no filesystem persistency.
So, as long as you keep those points in mind, our -a aarch64 offers an interesting different setup to play with.
TODOs:
-
GDB step debugging appears to be stuck on an infinite loop:
no module object found for ''
Keep in mind that MIPS has the worst support compared to our other architectures due to the smaller community. Patches welcome as usual.
TODOs:
-
networking is not working. See also:
-
GDB step debugging does not work properly, does not find
start_kernel
When the Linux kernel finishes booting, it runs an executable as the first and only userland process.
The default path is /init, but we an set a custom one with the init= kernel command line parameter.
This process is then responsible for setting up the entire userland (or destroying everything when you want to have fun).
This typically means reading some configuration files (e.g. /etc/initrc) and forking a bunch of userland executables based on those files.
systemd is a "popular" /init implementation for desktop distros as of 2017.
BusyBox provides its own minimalistic init implementation which Buildroot uses by default.
Is the default BusyBox /init too bloated for you, minimalism freak?
No problem, just use the init kernel boot parameter:
./run -e 'init=/sleep_forever.out'
Remember that shell scripts can also be used for init https://unix.stackexchange.com/questions/174062/init-as-a-shell-script/395375#395375:
./run -e 'init=/count.sh'
Also remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:
-
run forever in a loop or long sleep
-
poweroffthe machine
The default BusyBox init scripts enable networking, and there is a 15 second timeout in case your network is down or if your kernel / emulator setup does not support it.
To disable networking, use:
./build -p -n
To restore it, run:
./build -t initscripts-reconfigure
The docs make it clear https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html
The kernel parses parameters from the kernel command line up to “–”; if it doesn’t recognize a parameter and it doesn’t contain a ‘.’, the parameter gets passed to init: parameters with ‘=’ go into init’s environment, others are passed as command line arguments to init. Everything after “–” is passed as an argument to init.
And you can try it out with:
./run -e 'init=/init_env_poweroff.sh - asdf=qwer zxcv' -n
If you are feeling fancy, you can also insert modules with:
modprobe dep2 lsmod # dep and dep2
This method also deals with module dependencies, which we almost don’t use to make examples simpler:
Removal also removes required modules that have zero usage count:
modprobe -r dep2 lsmod # Nothing.
but it can’t know if you actually insmodded them separately or not:
modprobe dep modprobe dep2 modprobe -r dep2 # Nothing.
so it is a bit risky.
modprobe searches for modules under:
ls /lib/modules/*/extra/
Kernel modules built from the Linux mainline tree with CONFIG_SOME_MOD=m, are automatically available with modprobe, e.g.:
modprobe dummy-irq
Only tested successfully in x86_64.
Build:
./build -x ./run
We don’t build X11 by default because it takes a considerable amount of time (~20%), and is not expected to be used by most users: you need to pass the -x flag to enable it.
Inside QEMU:
startx
More details: https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116
Not sure how well that graphics stack represents real systems, but if it does it would be a good way to understand how it works.
On ARM, startx hangs at a message:
vgaarb: this pci device is not a vga device
and nothing shows on the screen, and:
grep EE /var/log/Xorg.0.log
says:
(EE) Failed to load module "modesetting" (module does not exist, 0)
A friend told me this but I haven’t tried it yet:
-
xf86-video-modesettingis likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things. -
xf86-video-fbdevshould work as well, but we need to make sure fbdev is enabled, and maybe add some line to theXorg.conf
Results:
| Commit | Arch | Simulator | Instruction count |
|---|---|---|---|
7228f75ac74c896417fb8c5ba3d375a14ed4d36b |
arm |
QEMU |
680k |
7228f75ac74c896417fb8c5ba3d375a14ed4d36b |
arm |
gem5 AtomicSimpleCPU |
160M |
7228f75ac74c896417fb8c5ba3d375a14ed4d36b |
arm |
gem5 HPI |
155M |
7228f75ac74c896417fb8c5ba3d375a14ed4d36b |
x86_64 |
QEMU |
3M |
7228f75ac74c896417fb8c5ba3d375a14ed4d36b |
x86_64 |
gem5 AtomicSimpleCPU |
528M |
QEMU:
time ./run -n -e 'init=/poweroff.out' -- -trace exec_tb,file=trace time ./qemu/scripts/simpletrace.py buildroot/output.x86_64~/build/host-qemu-custom/trace-events-all trace >trace.txt wc -l trace.txt sed '/0x1000000/q' trace.txt >trace-boot.txt wc -l trace-boot.txt
gem5:
./run -a arm -g -e 'init=/eval.sh - lkmc_eval="m5 exit"' -n # Or: # ./run -a arm -g -e 'init=/eval.sh - lkmc_eval="m5 exit"' -n -- --cpu-type=HPI --caches grep sim_insts m5out/stats.txt
Notes:
-
-nis a good idea to reduce the chances that you send unwanted non-deterministic mouse or keyboard clicks to the VM. -
-e 'init=/poweroff.out'is crucial as it reduces the instruction count from 40 million to 20 million, so half of the instructions were due to userland programs instead of the boot sequence.Without it, the bulk of the time seems to be spent in setting up the network with
ifupthat gets called from/etc/init.d/S40networkfrom the default Buildroot BusyBox setup.And it becomes even worse if you try to
-net noneas recommended in the 2.7replay.txtdocs, because thenifupwaits for 15 seconds before giving up as per/etc/network/interfaceslinewait-delay 15. -
0x1000000is the address where QEMU puts the Linux kernel at with-kernelin x86.It can be found from:
readelf -e buildroot/output.x86_64~/build/linux-*/vmlinux | grep Entry
TODO confirm further. If I try to break there with:
./rungdb *0x1000000
but I have no corresponding source line. Also note that this line is not actually the first line, since the kernel messages such as
early console in extract_kernelhave already shown on screen at that point. This does not break at all:./rungdb extract_kernel
It only appears once on every log I’ve seen so far, checked with
grep 0x1000000 trace.txtThen when we count the instructions that run before the kernel entry point, there is only about 100k instructions, which is insignificant compared to the kernel boot itself.
-
We can also discount the instructions after
initruns by usingreadelfto get the initial address ofinit. One easy way to do that now is to just run:./rungdb-user kernel_module-1.0/user/poweroff.out main
And get that from the traces, e.g. if the address is
4003a0, then we search:grep -n 4003a0 trace.txt
I have observed a single match for that instruction, so it must be the init, and there were only 20k instructions after it, so the impact is negligible.
-
gem5 simulates memory latencies. So I think that the CPU loops idle while waiting for memory, and counts will be higher.
This works because we have already done the following with QEMU:
-
./configure --enable-trace-backends=simple. This logs in a binary format to the trace file.It makes 3x execution faster than the default trace backend which logs human readable data to stdout.
This also alters the actual execution, and reduces the instruction count by 10M TODO understand exactly why, possibly due to the
All QSes seenthing. -
patch QEMU source to remove the
disablefromexec_tbin thetrace-events. See also: https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/
Possible improvements:
-
to disable networking. Is replacing
initenough?-
https://superuser.com/questions/181254/how-do-you-boot-linux-with-networking-disabled
-
CONFIG_NET=ndid not significantly reduce instruction, so maybe replacinginitis enough.
-
-
logging with the default backend
loggreatly slows down the CPU, and in particular leads to this during kernel boot:All QSes seen, last rcu_sched kthread activity 5252 (4294901421-4294896169), jiffies_till_next_fqs=1, root ->qsmask 0x0 swapper/0 R running task 0 1 0 0x00000008 ffff880007c03ef8 ffffffff8107aa5d ffff880007c16b40 ffffffff81a3b100 ffff880007c03f60 ffffffff810a41d1 0000000000000000 0000000007c03f20 fffffffffffffedc 0000000000000004 fffffffffffffedc ffffffff00000000 Call Trace: <IRQ> [<ffffffff8107aa5d>] sched_show_task+0xcd/0x130 [<ffffffff810a41d1>] rcu_check_callbacks+0x871/0x880 [<ffffffff810a799f>] update_process_times+0x2f/0x60
in which the boot appears to hang for a considerable time.
-
Confirm that the kernel enters at
0x1000000, or where it enters. Once we have this, we can exclude what comes before in the BIOS.
The kernel can boot from an CPIO file, which is a directory serialization format much like tar: https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference
The bootloader, which for us is QEMU itself, is then configured to put that CPIO into memory, and tell the kernel that it is there.
With this setup, you don’t even need to give a root filesystem to the kernel, it just does everything in memory in a ramfs.
Try it out with:
./run -i
Notice how it boots fine, even though -drive is not given.
Also as expected, there is no filesystem persistency, since we are doing everything in memory:
date >f poweroff cat f # can't open 'f': No such file or directory
This can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.
The main ingredients to get this working are:
-
BR2_TARGET_ROOTFS_CPIO=y: make Buildroot generateoutput/images/rootfs.cpioin addition to the other images.It is also possible to compress that image with other options.
-
qemu -initrd: make QEMU put the image into memory and tell the kernel about it. -
CONFIG_BLK_DEV_INITRD=y: Compile the kernel with initrd support, see also: https://unix.stackexchange.com/questions/67462/linux-kernel-is-not-finding-the-initrd-correctly/424496#424496Buildroot forces that option when
BR2_TARGET_ROOTFS_CPIO=yis given
https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image asks how the mechanism works in more detail.
Most modern desktop distributions have an initrd in their root disk to do early setup.
The rationale for this is described at: https://en.wikipedia.org/wiki/Initial_ramdisk
One obvious use case is having an encrypted root filesystem: you keep the initrd in an unencrypted partition, and then setup decryption from there.
I think GRUB then knows read common disk formats, and then loads that initrd to memory with a /boot/grub/grub.cfg directive of type:
initrd /initrd.img-4.4.0-108-generic
initramfs is just like initrd, but you also glue the image directly to the kernel image itself.
So the only argument that QEMU needs is the -kernel, no -drive not even -initrd! Pretty cool.
Try it out with:
./run -a aarch64
since our aarch64 setup uses it by default.
In the background, it uses BR2_TARGET_ROOTFS_INITRAMFS, and this makes the kernel config option CONFIG_INITRAMFS_SOURCE point to the CPIO that will be embedded in the kernel image.
http://nairobi-embedded.org/initramfs_tutorial.html shows a full manual setup.
Trace a single function:
cd /sys/kernel/debug/tracing/ # Stop tracing. echo 0 > tracing_on # Clear previous trace. echo '' > trace # List the available tracers, and pick one. cat available_tracers echo function > current_tracer # List all functions that can be traced # cat available_filter_functions # Choose one. echo __kmalloc >set_ftrace_filter # Confirm that only __kmalloc is enabled. cat enabled_functions echo 1 > tracing_on # Latest events. head trace # Observe trace continously, and drain seen events out. cat trace_pipe &
Sample output:
# tracer: function
#
# entries-in-buffer/entries-written: 97/97 #P:1
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
head-228 [000] .... 825.534637: __kmalloc <-load_elf_phdrs
head-228 [000] .... 825.534692: __kmalloc <-load_elf_binary
head-228 [000] .... 825.534815: __kmalloc <-load_elf_phdrs
head-228 [000] .... 825.550917: __kmalloc <-__seq_open_private
head-228 [000] .... 825.550953: __kmalloc <-tracing_open
head-229 [000] .... 826.756585: __kmalloc <-load_elf_phdrs
head-229 [000] .... 826.756627: __kmalloc <-load_elf_binary
head-229 [000] .... 826.756719: __kmalloc <-load_elf_phdrs
head-229 [000] .... 826.773796: __kmalloc <-__seq_open_private
head-229 [000] .... 826.773835: __kmalloc <-tracing_open
head-230 [000] .... 827.174988: __kmalloc <-load_elf_phdrs
head-230 [000] .... 827.175046: __kmalloc <-load_elf_binary
head-230 [000] .... 827.175171: __kmalloc <-load_elf_phdrs
Trace all possible functions, and draw a call graph:
echo 1 > max_graph_depth echo 1 > events/enable echo function_graph > current_tracer
Sample output:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) 2.173 us | } /* ntp_tick_length */
0) | timekeeping_update() {
0) 4.176 us | ntp_get_next_leap();
0) 5.016 us | update_vsyscall();
0) | raw_notifier_call_chain() {
0) 2.241 us | notifier_call_chain();
0) + 19.879 us | }
0) 3.144 us | update_fast_timekeeper();
0) 2.738 us | update_fast_timekeeper();
0) ! 117.147 us | }
0) | _raw_spin_unlock_irqrestore() {
0) 4.045 us | _raw_write_unlock_irqrestore();
0) + 22.066 us | }
0) ! 265.278 us | } /* update_wall_time */
TODO: what do + and ! mean?
Each enable under the events/ tree enables a certain set of functions, the higher the enable more functions are enabled.
This has nothing to do with the Linux kernel, but it is cool:
sudo apt-get install qemu-user ./build -a arm cd buildroot/output.arm~/target qemu-arm -L . bin/ls
This uses QEMU’s user-mode emulation mode that allows us to run cross-compiled userland programs directly on the host.
The reason this is cool, is that ls is not statically compiled, but since we have the Buildroot image, we are still able to find the shared linker and the shared library at the given path.
In other words, much cooler than:
arm-linux-gnueabi-gcc -o hello -static hello.c qemu-arm hello
It is also possible to compile QEMU user mode from source with BR2_PACKAGE_HOST_QEMU_LINUX_USER_MODE=y, but then your compilation will likely fail with:
package/qemu/qemu.mk:110: *** "Refusing to build qemu-user: target Linux version newer than host's.". Stop.
since we are using a bleeding edge kernel, which is a sanity check in the Buildroot QEMU package.
Anyways, this warns us that the userland emulation will likely not be reliable, which is good to know. TODO: where is it documented the host kernel must be as new as the target one?
GDB step debugging is also possible with:
qemu-arm -g 1234 -L . bin/ls ../host/usr/bin/arm-buildroot-linux-uclibcgnueabi-gdb -ex 'target remote localhost:1234'
TODO: find source. Lazy now.
QEMU allows us to take snapshots at any time through the monitor.
You can then restore CPU, memory and disk state back at any time.
qcow2 filesystems must be used for that to work.
To test it out, login into the VM with and run:
/count.sh
On another shell, take a snapshot:
echo 'savevm my_snap_id' | ./qemumonitor
The counting continues.
Restore the snapshot:
echo 'loadvm my_snap_id' | ./qemumonitor
and the counting goes back to where we saved. This shows that CPU and memory states were reverted.
We can also verify that the disk state is also reversed. Guest:
echo 0 >f
Monitor:
echo 'savevm my_snap_id' | ./qemumonitor
Guest:
echo 1 >f
Monitor:
echo 'loadvm my_snap_id' | ./qemumonitor
Guest:
cat f
And the output is 0.
Our setup does not allow for snapshotting while using initrd.
We have added and interacted with a few educational hardware models in QEMU.
This is useful to learn:
-
how to create new hardware models for QEMU. Overview: https://stackoverflow.com/questions/28315265/how-to-add-a-new-device-in-qemu-source-code
-
how the Linux kernel interacts with hardware
To get started, have a look at the "Hardware device drivers" secion under kernel_module/README.adoc, and try to run those modules, and then grep the QEMU source code.
gem5 is a system simulator, much like QEMU: http://gem5.org/
For the most part, just add the -g option to the QEMU commands and everything should magically work:
./configure && ./build -a arm -g ./run -a arm -g
On another shell:
./gem5-shell
A full rebuild is currently needed even if you already have QEMU working unfortunately, see: gem5 and QEMU with the same kernel configuration
Tested architectures:
-
arm -
aarch64 -
x86_64
Like QEMU, gem5 also has a syscall emulation mode (SE), but in this tutorial we focus on the full system emulation mode (FS). For a comparision see: https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program
-
advantages of gem5:
-
simulates a generic more realistic pipelined and optionally out of order CPU cycle by cycle, including a realistic DRAM memory access model with latencies, caches and page table manipulations. This allows us to:
-
do much more realistic performance benchmarking with it, which makes absolutely no sense in QEMU, which is purely functional
-
make certain functional cache observations that are not possible in QEMU, e.g.:
-
use Linux kernel APIs that flush memory like DMA, which are crucial for driver development. In QEMU, the driver would still work even if we forget to flush caches.
-
TODO spectre / meltdown
It is not of course truly cycle accurate, as that
-
-
-
would require exposing proprietary information of the CPU designs: https://stackoverflow.com/questions/17454955/can-you-check-performance-of-a-program-running-with-qemu-simulator/33580850#33580850
-
would make the simulation even slower TODO confirm, by how much
but the approximation is reasonable.
It is used mostly for microarchitecture research purposes: when you are making a new chip technology, you don’t really need to specialize enormously to an existing microarchitecture, but rather develop something that will work with a wide range of future architectures.
-
runs are deterministic by default, unlike QEMU which has a special Record and replay mode, that requires first playing the content once and then replaying
-
-
disadvantage of gem5: slower than QEMU, see: gem5 vs QEMU performance
This implies that the user base is much smaller, since no Android devs.
Instead, we have only chip makers, who keep everything that really works closed, and researchers, who can’t version track or document code properly >:-) And this implies that:
-
the documentation is more scarce
-
it takes longer to support new hardware features
Well, not that AOSP is that much better anyways.
-
-
not sure: gem5 has BSD license while QEMU has GPL
This suits chip makers that want to distribute forks with secret IP to their customers.
On the other hand, the chip makers tend to upstream less, and the project becomes more crappy in average :-)
We have benchmarked a Linux kernel boot at commit da79d6c6cde0fbe5473ce868c9be4771160a003b with the commands:
# Try to manually hit Ctrl + C as soon as system shutdown message appears. time ./run -a arm -e 'init=/poweroff.out' time ./run -a arm -e 'm5 exit' -g time ./run -a arm -e 'm5 exit' -g -- --caches --cpu-type=HPI time ./run -a x86_64 -e 'init=/poweroff.out' time ./run -a x86_64 -e 'init=/poweroff.out' - -enable-kvm time ./run -a x86_64 -e 'init=/poweroff.out' -g
and the results were:
| Emulator | Time | N times slower than QEMU |
|---|---|---|
QEMU ARM |
6 seconds |
1 |
gem5 ARM AtomicSimpleCPU |
1 minute 40 seconds |
17 |
gem5 ARM HPI |
10 minutes |
100 |
QEMU X86_64 |
4 seconds |
1 |
QEMU X86_64 KVM |
2 seconds |
0.5 |
gem5 X86_64 |
5 minutes 30 seconds |
82 |
on a Lenovo P51 laptop with:
-
Intel Core i7-7820HQ Processor (8MB Cache, up to 3.90GHz) (4 cores 8 threads)
-
32GB(16+16) DDR4 2400MHz SODIMM
-
512GB SSD PCIe TLC OPAL2
-
Ubuntu 17.10
OK, this is why we used gem5 in the first place, performance measurements!
Let’s benchmark Dhrystone which Buildroot provides:
./gem5-bench dhrystone 1000 ./gem5-bench -r dhrystone 1000
These commands output the approximate number of CPU cycles it took Dhrystone to run.
It works like this:
-
the first commond boots linux with the default simplified
AtomicSimpleCPU, and generates a checkpoint after the kernel boots and before running the benchmark -
the second command restores the checkpoint with the more detailed
HPICPU model, and runs the benchmark. We don’t boot with it because that is much slower.
A few imperfections of our benchmarking method are:
-
when we do
m5 resetstatsandm5 exit, there is some time passed before theexecsystem call returns and the actual benchmark starts and ends -
the benchmark outputs to stdout, which means so extra cycles in addition to the actual computation. But TODO: how to get the output to check that it is correct without such IO cycles?
Those problems should be insignificant if the benchmark runs for long enough however.
TODO: even if we don’t switch to the detailed CPU model, the cycle counts on the original run and the one with checkpoint restore differ slightly. Why? Multiple checkpoint restores give the same results as expected however:
./run -a arm -e 'init=/eval.sh - lkmc_eval="m5 checkpoint;m5 resetstats;dhrystone 1000;m5 exit"' -g ./run -a arm -g -- -r 1
Now you can play a fun little game with your friends:
-
pick a computational problem
-
make a program that solves the computation problem, and outputs output to stdout
-
write the code that runs the correct computation in the smallest number of cycles possible
To find out why your program is slow, a good first step is to have a look at the statistics for the run:
cat m5out/stats.txt
Whenever we run m5 dumpstats or m5 exit, a section with the following format is added to that file:
---------- Begin Simulation Statistics ---------- [the stats] ---------- End Simulation Statistics ----------
Besides optimizing a program for a given CPU setup, chip developers can also do the inverse, and optimize the chip for a given benchmark!
The rabbit hole is likely deep, but let’s scratch a bit of the surface.
./run -a arm -c 2 -g
Check with:
cat /proc/cpuinfo getconf _NPROCESSORS_CONF
A quick ./run -g -- -h leads us to the options:
--caches --l1d_size=1024 --l1i_size=1024 --l2cache --l2_size=1024 --l3_size=1024
But keep in mind that it only affects benchmark performance of the most detailed CPU types:
*: couldn’t test because of:
This has been verified with:
m5 resetstats && dhrystone 10000 && m5 dumpstats
at commit da79d6c6cde0fbe5473ce868c9be4771160a003b with the following gem5 commands cycle counts:
# 11M ./run -a arm -g ./run -a arm -g -- --caches --l2cache # 175M ./run -a arm -g -- --caches --l1d_size=1024 --l1i_size=1024 --l2cache --l2_size=1024 --l3_size=1024 --cpu-type=HPI # 16M ./run -a arm -g -- --caches --l1d_size=1024MB --l1i_size=1024MB --l2cache --l2_size=1024MB --l3_size=1024MB --cpu-type=HPI # 20M ./run -a x86_64 -g -- --caches --l1d_size=1024 --l2cache --l2_size=1024 --l3_size=1024 ./run -a x86_64 -g -- --caches --l1d_size=1024MB --l2cache --l2_size=1024MB --l3_size=1024MB
Cache sizes can in theory be checked with the methods described at: https://superuser.com/questions/55776/finding-l2-cache-size-in-linux:
getconf -a | grep CACHE lscpu cat /sys/devices/system/cpu/cpu0/cache/index2/size
but for some reason the Linux kernel is not seeing the cache sizes:
Behaviour breakdown:
-
arm QEMU and gem5 (both
AtomicSimpleCPUorHPI), x86 gem5:/sysfiles don’t exist, andgetconfandlscpuvalue empty -
x86 QEMU:
/sysfiles exist, butgetconfandlscpuvalues still empty
TODO These look promising:
--list-mem-types --mem-type=MEM_TYPE --mem-channels=MEM_CHANNELS --mem-ranks=MEM_RANKS --mem-size=MEM_SIZE
TODO: now to verify this with the Linux kernel? Besides raw performance benchmarks.
TODO These look promising:
--ethernet-linkspeed --ethernet-linkdelay
and also: gem5-dist: https://publish.illinois.edu/icsl-pdgem5/
Clock frequency: TODO how does it affect performance in benchmarks?
./run -a arm -g -- --cpu-clock 10000000
Check with:
m5 resetstats && sleep 10 && m5 dumpstats
and then:
grep numCycles m5out/stats.txt
TODO: why doesn’t this exist:
ls /sys/devices/system/cpu/cpu0/cpufreq
If you are benchmarking compiled programs instead of hand written assembly, remember that we configure Buildroot to disable optimizations by default with:
BR2_OPTIMIZE_0=y
to improve the debugging experience.
You will likely want to change that to:
BR2_OPTIMIZE_3=y
and do a full rebuild.
TODO is it possible to compile a single package with optimizations enabled? In any case, this wouldn’t be very representative, since calls to an unoptimized libc will also have an impact on performance. Kernel-wise it should be fine though, since the kernel requires O=2.
Buildroot built-in libraries, mostly under Libraries > Other:
-
Armadillo
C++: linear algebra -
CBLAS / CLAPACK: linear algebra
-
fftw: Fourier transform
-
Eigen: linear algebra
-
Flann
-
GSL: various
-
liblinear
-
libspacialindex
-
libtommath
-
qhull
There are not yet enabled, but it should be easy to so:
-
enable them in buildroot_config_fragment and rebuild
-
create a test program that uses each library under kernel_module/user
External open source benchmarks. We will try to create Buildroot packages for them, add them to this repo, and potentially upstream:
Analogous to QEMU:
./run -a arm -e 'init=/poweroff.out' -g
Internals: when we give --command-line= to gem5, it overrides default command lines, including some mandatory ones which are required to boot properly.
Our run script hardcodes the require options in the default --command-line and appends extra options given by -e.
To find the default options in the first place, we removed --command-line and ran:
./run -a arm -g
and then looked at the line of the Linux kernel that starts with:
Kernel command line:
Analogous to QEMU, on the first shell:
./run -a arm -d -g
On the second shell:
./rungdb -a arm -g
This makes the VM stop, so from inside GDB:
continue
On a third shell:
./gem5-shell
And we now see the boot messages, and then get a shell. Now try the /continue.sh procedure described for QEMU.
TODO: how to stop at start_kernel? gem5 listens for GDB by default, and therefore does not wait for a GDB connection to start like QEMU does. So when GDB connects we might have already passed start_kernel. Maybe --debug-break=0 can be used?
Analogous to QEMU’s Snapshot, but better since it can be started from inside the guest, so we can easily checkpoint after a specific guest event, e.g. just before init is done.
Documentation: http://gem5.org/Checkpoints
./rung -a arm -g
In guest, wait for the boot to end and run:
m5 checkpoint
To restore the checkpoint, kill the VM and run:
./rung -a arm -g -- -r 1
Let’s create a second checkpoint to see how it works, in guest:
date >f m5 checkpoint
Kill the VM, and try it out:
./run -a arm -g -- -r 2
and now in the guest:
cat f
contains the date. The file f wouldn’t exist had we used the first checkpoint with -r 1.
Internals:
-
the checkpoints are stored under
m5out/cpt.* -
m5is a guest utility present inside the gem5 tree which we cross-compiled and installed into the guest
If you automate things with Kernel command line parameters as in:
./run -a arm -e 'init=/eval.sh - lkmc_eval="m5 checkpoint;m5 resetstats;dhrystone 1000;m5 exit"' -g
Then there is no need to pass the kernel command line again to gem5 for replay:
./run -a arm -g -- -r 1
since boot has already happened, and the parameters are already in the RAM of the snapshot.
gem5 can switch to a different CPU model when restoring a checkpoint.
A common combo is to boot Linux with a fast CPU, make a checkpoint and then replay the benchmark of interest with a slower CPU.
An illustrative interactive run:
./run -a arm -g
In guest:
m5 checkpoint
And then restore the checkpoint with a different CPU:
./run -a arm -g -- --caches -r 1 --restore-with-cpu=HPI
Pass options to the fs.py script:
-
get help:
./run -g -- -h
-
boot with the more detailed and slow
HPICPU model:./run -a arm -g -- --caches --cpu-type=HPI
Pass options to the gem5 executable itself:
-
get help:
./run -G '-h' -g
gem5 just assigns new ports if some ports are occupied, so we can do:
./run -g # Same as ./gem5-shell 0 ./gem5-shell
And a second instance:
./run -g ./gem5-shell 1
TODO Now we just need to network them up to have some more fun! See dist-gem5: http://publish.illinois.edu/icsl-pdgem5/
We would like to be able to run both gem5 and QEMU with the same kernel build to avoid duplication, but TODO we haven’t been able to get that working yet.
This documents our failed attempts so far.
As a result, we currently have to create two full buildroot/output* directories, which means two full GCC builds.
To test this, hack up run to use the buildroot/output.arm-gem5~ directory, and then run:
./run -a arm
Now QEMU hangs at:
audio: Could not init `oss' audio driver
and the display shows:
Guest has not initialized the display (yet).
Test it out with:
./run -a arm -g
TODO hangs at:
**** REAL SIMULATION **** warn: Existing EnergyCtrl, but no enabled DVFSHandler found. info: Entering event queue @ 0. Starting simulation... 1614868500: system.terminal: attach terminal 0
and the telnet at:
2017-12-28-11-59-51@ciro@ciro-p51$ ./gem5-shell Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ==== m5 slave terminal: Terminal 0 ====
I have also tried to copy the exact same kernel command line options used by QEMU, but nothing changed.
-
networking not working. We currently just disable it from
inittabby default to prevent waiting at startup -
gdbserver: https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5
-
gdbdebugging not working. First we need to remove the initialset arch i386:x86-64:intelrequired for QEMU. Then it cannot find the symbols.
QEMU supports deterministic record and replay by saving external inputs, which would be awesome to understand the kernel, as you would be able to examine a single run as many times as you would like.
Unfortunately it is not working in the current QEMU: https://stackoverflow.com/questions/46970215/how-to-use-qemus-deterministic-record-and-replay-feature-for-a-linux-kernel-boo
Alternatively, mozilla/rr claims it is able to run QEMU: but using it would require you to step through QEMU code itself. Likely doable, but do you really want to?
This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
-
can’t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since the Linux kernel does not have a stable kernel module API.
-
bugs can easily break you system. E.g.:
-
segfaults can trivially lead to a kernel crash, and require a reboot
-
your disk could get erased. Yes, this can also happen with
sudofrom userland. But you should not usesudowhen developing newbie programs. And for the kernel you don’t have the choice not to usesudo -
even more subtle system corruption such as not being able to rmmod
-
-
can’t control which hardware is used, notably the CPU architecture
-
can’t step debug it with GDB easily
Still interested?
cd kernel_module ./make-host.sh
If the compilation of any of the C files fails because of kernel or toolchain differences that we don’t control on the host, just rename it to remove the .c extension and try again:
mv broken.c broken.c~ ./build_host
Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:
sudo insmod hello.ko # Our module is there. sudo lsmod | grep hello # Last message should be: hello init dmest -T sudo rmmod hello # Last message should be: hello exit dmesg -T # Not present anymore sudo lsmod | grep hello
Minimal host build system sanity check example.
cd hello_host make insmod hello.ko dmesg rmmod hello.ko dmesg
Multi-call executable that implements: lsmod, insmod, rmmod, and other tools on desktop distros such as Ubuntu 16.04, where e.g.:
ls -l /bin/lsmod
gives:
lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod
and:
dpkg -l | grep -Ei
contains:
ii kmod 22-1ubuntu5 amd64 tools for managing Linux kernel modules
BusyBox also implements its own version of those executables. There are some differences.
Buildroot also has a kmod package, but we are not using it since BusyBox' version is good enough so far.
This page will only describe features that differ from kmod to the BusyBox implementation.
Name of a predecessor set of tools.
Load module under different name to avoid conflicts:
sudo modprobe vmhgfs -o vm_hgfs
platform_device.c together with its kernel and QEMU forks contains a minimal runnable example.
Good format descriptions:
Minimal example
/dts-v1/;
/ {
a;
};
Check correctness with:
dtc a.dts
Separate nodes are simply merged by node path, e.g.:
/dts-v1/;
/ {
a;
};
/ {
b;
};
then dtc a.dts gives:
/dts-v1/;
/ {
a;
b;
};
# Last point before out patches.
last_mainline_revision=v4.14
next_mainline_revision=v4.15
cd linux
# Create a branch before the rebase.
git branch "lkmc-${last_mainline_revision}"
git remote set-url origin git@github.com:cirosantilli/linux.git
git push
git remote add up git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
git fetch up
git rebase --onto "$next_mainline_revision" "$last_mainline_revision"
./build -l
# Manually fix our kernel modules if necessary.
cd ..
git branch "buildroot-2017.08-linux-${last_mainline_revision}"
git add .
git commit -m "Linux ${next_mainline_revision}"
git push
and update the README!
During update all you kernel modules may break since the kernel API is not stable.
They are usually trivial breaks of things moving around headers or to sub-structs.
The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.
This backwards compatibility is just awesome, it makes getting and running the latest master painless.
This also makes this repo the perfect setup to develop the Linux kernel.
The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.
The default Linux kernel version is bumped in Buildroot with commit messages of type:
linux: bump default to version 4.9.6
So you can try:
git log --grep 'linux: bump default to version'
Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.
You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.
cd buildroot/output.x86_64~ make menuconfig
Hit / and search for the settings.
Save and quit.
diff .config.olg .config
Copy and paste the diff additions to buildroot_config_fragment.
cd buildroot/output.x86_64~ make graph-build xdg-open graphs/build.pie-packages.pdf
Our phylosophy is:
-
if something adds little to the build time, build it in by default
-
otherwise, make it optional
This project is for people who want to learn and modify low level system components:
-
Linux kernel and Linux kernel modules
-
full systems emulators like QEMU and gem5
-
C standard libraries. This could also be put on a submodule if people show interest.
-
Buildroot. We use and therefore a large feature set of it.
Phylosophy:
-
automate as much as possible to make things more reproducible
-
do everything from source to make things understandable and hackable
Runnable stuff:
-
https://lwn.net/Kernel/LDD3/ the best book, but outdated. Updated source: https://github.com/martinezjavier/ldd3 But examples non-minimal and take too much brain power to understand.
-
https://github.com/satoru-takeuchi/elkdat manual build process without Buildroot, very few and simple kernel modules
-
https://github.com/tinyclub/linux-lab Buildroot based, no kernel modules?
-
https://github.com/linux-kernel-labs Yocto based, source inside a kernel fork subdir: https://github.com/linux-kernel-labs/linux/tree/f08b9e4238dfc612a9d019e3705bd906930057fc/tools/labs which the author would like to upstream https://www.reddit.com/r/programming/comments/79w2q9/linux_device_driver_labs_the_linux_kernel/dp6of43/
-
Android AOSP: https://stackoverflow.com/questions/1809774/how-to-compile-the-android-aosp-kernel-and-test-it-with-the-android-emulator/48310014#48310014 AOSP is basically a uber bloated Buildroot (2 hours build vs 30 minutes), Android is Linux based, and QEMU is the emulator backend. These instructions might work for debugging the kernel: https://github.com/Fuzion24/AndroidKernelExploitationPlayground
Theory:
-
http://nairobi-embedded.org you will fall here a lot when the hard Google queries start popping. They have covered everything we do here basically, but with a more manual approach, while this repo automates everything.
-
https://balau82.wordpress.com awesome low level resource