Add a custom syscall to the linux kernel

July 17, 2011 | 4 Minute Read

Hi all,

back from oblivion for a new post.

This is mostly a note for me than something new. I found some really good guides on how to add a new syscall to the linux kernel, but they’re all outdated in a way or another, so I decided to write a post to remind me how to do it.

Prerequisites:

• a 32bit linux installation of your favorite distro
• gcc, make, ncurses: the usual stuff to build your custom kernel

First: we need kernel sources.

This guide is based on the latest kernel I have installed on my archlinux box, that is linux 2.6.39.

You can wget it like this:

wget -O http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.1.tar.bz2

So now we can just expand it with the good old

tar xzf linux-2.6.39.1.tar.bz2

Ok now we can start editing files:

We can insert our syscall in an existing file or creating a new one. I decided to create a new file with my custom syscalls so this guide will take this approach.

So we start by creating our syscall file:

cd linux-2.6.39

vim kernel/omg.c

In our newly created source file we’ll insert this code (note that this is a skeleton for a working syscall):

/* adding omg syscall */
#include 
#include 

asmlinkage int sys_omg(){
	printk(KERN_EMERG "OMGWTFBBQ!!11\n");
	return 0;
}

So we need two includes: linux/kernel to use the printk and linux/linkage to use the asmlinkage tag,
used basically to tell the compiler that we want all the parameters to be passed on the stack.
Now we have to link it to the rest of the kernel
Step 1- let’s add it to the Makefile

vim kernel/Makefile

In the first lines we find a list of .o, one for each source file located inside the current folder, this is the list of the object files to be compiled.
We have to add our file to this list by adding omg.o to the end of the list, just after “jump_label.o”
Step 2- let’s add it to the syscall lists.

vim arch/x86/include/asm/unistd_32.h

Somewhere around line 352 we see the end of the syscall list.
We can add our syscall to the list like this:

#define __NR_omg		345

and then we can increment the syscall number 4 lines below to 346.
Now we have to edit

vim arch/x86/include/asm/syscalls.h

Around line 45 we can find the X86_32 only section of the file. We can insert our syscall prototype on the line below the #ifdef CONFIG_X86_32 line by adding

asmlinkage int sys_omg();

Last but not least we have to add it to the syscall_table_32.S file:

vim arch/x86/kernel/syscall_table_32.S

We move to the end of the file and we add

 .long sys_omg /* 345 */

Ok. Now we can build again our kernel and we should have our new syscall waiting to be used
How to test it? Let’s build a small C program to invoke it!

#include 
#include 
#include 

#define __NR_omg        345 

int omg() {
   return (int) syscall(__NR_omg);
};

main () {
   printf("The return code from the omg system call is %d\n", omg());
}

Build it and launch it. After that if you launch

dmesg | tail

You should see our custom message.
P.s. This guide is almost copied from this emuboy post, with only some modifications, kudos to him