Getting Rid of Stdlibc
tags: assembling, compiling, linking
To get started writing your own implementation of the standard C library you'll need to stop linking against whatever implementation that your compiler comes with. I'm working on a CentOS 7 machine and use glibc normally; now I want to use welibc (even though it doesn't really exist yet).
Normal Compilation
First, we're going to look at a "normal" compile. The beginning of the test suite for welibc is in a file called main_test.c and looks like this (comment blocks excluded)
int
main(int argc, char *argv[])
{
int ret = 0;
return ret;
}
Easy enough. This should just run and return 0 without doing anything else.
I'm using gcc to compile this project, so to compile this code we can use the
following command
$ gcc -o test_welibc main_test.c
If things went properly then this compiles with no errors and gcc doesn't
print anything. I'm also using bash for my shell and we can test that it ran
properly like so
$ ./test_welibc
$ echo $?
0
The echo $? line will print the exit status of the most recently executed
foreground pipeline, e.g. the return value of the program we just ran. We got 0
which means things are working properly. You can be sure by using a much
different return value
int
main(int argc, char *argv[])
{
int ret = 42;
return ret;
}
Then rerun it
$ gcc -o test_welibc main_test.c
$ ./test_welibc
$ echo $?
42
So, now you know how a normal compilation would run.
Compiling without stdlibc
We never explicitly told gcc to use glibc but it does this automatically
unless you instruct it not to. You would do that with the -nostdlib
-ffreestanding flags, which prevent linking against stdlib and instructs gcc
that our final product will be running in a freestanding environment which means
the standard library might not exist and our program might not necessarily start
in the main function. So, we'll add those to our compile line and see what
happens.
$ gcc -nostdlib -ffreestanding -o test_welibc main_test.c
/usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000400144
Uh... what? /usr/bin/ld? entry symbol _start? First, we didn't even tell you to
run ld and second, my code doesn't even mention _start, so why do you need that
symbol?
Compilation process
When you compile a C program, a couple things happen:
- The preprocessor performs text replacements
- The compiler converts C code into assembly instructions
- The assembler converts assembly instructions into into object code
- The linker combines all of the object files together into an executable
With the GNU compiler collection your preprocessor and compiler will be gcc,
the assembler is as, and the linker is ld. gcc will automatically run each
of these as instructed so you don't have to worry about doing each of them
individually. However, when you need the control, you can absolutely run each
step individually.
The output of each of these steps can be seen by using the -save-temps flag to
gcc, so we'll compile with that and look at each step
$ gcc -save-temps -nostdlib -ffreestanding -o test_welibc main_test.c
/usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000400144
$ ls
main_test.c main_test.i main_test.o main_test.s test_welibc
Preprocessor
The preprocessor, gcc in this case, moves through your C code and performs all
of the actions specified by the preprocessor directives, like:
#include#define#endif
This mostly performs text replacements within your code. For example, #include
actually inserts the text from whatever file you're including, and #define
replaces every instance of your macro with the value you assigned to it. It also
replaces every comment with a space.
The temporary file corresponding to this step is main_test.i, let's look at
it:
# 1 "main_test.c"
# 1 "<command-line>"
# 1 "main_test.c"
# 47 "main_test.c"
int
main(int argc, char *argv[])
{
int ret = 42;
return ret;
}
There isn't really anything too interesting here, our code is mostly unchanged.
The .i file type indicates to gcc that the code should not be preprocessed;
this makes sense because this is the output of the preprocessor. The first four
lines are all specific to the gcc C preprocessor and contain a line number
followed by a filename in quotes. These aren't really important for now, but you
can read more about them here.
Compiler
The compiler, gcc, will convert human readable code into assembly instructions
which correspond directly to a specific action that your processor needs to
carry out. For example, the processor doesn't know what return ret; means, so
we need to translate that C code into the instructions that make the processor
"return the value of ret to the calling function." To look at the assembly,
check out the test_main.s file
.file "main_test.c"
.text
.globl main
.type main, @function
main:
.LFB0:
.cfi_startproc
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset 6, -16
movq %rsp, %rbp
.cfi_def_cfa_register 6
movl %edi, -20(%rbp)
movq %rsi, -32(%rbp)
movl $42, -4(%rbp)
movl -4(%rbp), %eax
popq %rbp
.cfi_def_cfa 7, 8
ret
.cfi_endproc
.LFE0:
.size main, .-main
.ident "GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-16)"
.section .note.GNU-stack,"",@progbits
This includes some things that are unique to the DWARF-2 debugging format and are too specific to get into right now. More information about these can be found here and in the DWARF-2 specification. The .ident and .section lines can also be ignored for the purposes of this article, but more info is here.
I'll add some comments to the assembly to indicate what it's doing
.file "main_test.c" # The file that these instruction were generated from
.text # The following instructions should be executable
.globl main # Declare a global symbol named main
.type main, @function # Set the "main" symbol to be of function type
main: # Define a label named main
.LFB0: # Define a label named .LFB0
.cfi_startproc
pushq %rbp # Push the value in the rbp register on to the stack
.cfi_def_cfa_offset 16
.cfi_offset 6, -16
movq %rsp, %rbp # Copy the contents of the rsp register into the rbp register
.cfi_def_cfa_register 6
movl %edi, -20(%rbp) # Move the contents of edi into memory
movq %rsi, -32(%rbp) # Move the contents of rsi into memory
movl $42, -4(%rbp) # Place the value 42 in memory
movl -4(%rbp), %eax # Move the value we stored in memory into the eax register
popq %rbp # Pop the value on top of the stack into the rbp register
.cfi_def_cfa 7, 8
ret # Return to the calling function
.cfi_endproc
.LFE0: # Define a label named .LFE0
.size main, .-main # Calculate the size of the main function
.ident "GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-16)"
.section .note.GNU-stack,"",@progbits
There is a lot going on here but the point is that you can see the 42 getting placed into memory, getting retrieved from memory and placed into a register, and finally the function returns to the caller. The eax register is the place where calling functions check for the return value.
Assembler
The assembler, /usr/bin/as in this case, takes the assembly instructions in
the .s file and translates them into an object file which contains code that is
relocatable. This means that is will use offsets to reference things, rather
than hardcoded addresses, and will use a special address (like 0x00000000) to
indicate when something is unknown and needs to be fixed. This is done because
the assembler isn't aware of where this code will be placed in memory. The
special addresses will be fixed in the next stage. The object file also contains
the symbol table which correlates named objects to specific addresses, like
where the main function is. There are several object file formats, the one we
are looking at is called ELF, executable and linkable format.
To view the information about the object file, use readelf
$ readelf -a main_test.o
This is too much information to look at, but I'll mention the symbol table specifically, which can be produced by itself with
$ readelf -s main_test.o
Symbol table '.symtab' contains 9 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS main_test.c
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 2
4: 0000000000000000 0 SECTION LOCAL DEFAULT 3
5: 0000000000000000 0 SECTION LOCAL DEFAULT 5
6: 0000000000000000 0 SECTION LOCAL DEFAULT 6
7: 0000000000000000 0 SECTION LOCAL DEFAULT 4
8: 0000000000000000 23 FUNC GLOBAL DEFAULT 1 main
Entry 8 is the entry for our main function, you can see the size of the function, the fact that it's a global function, and that it's value is 0 which means it hasn't been fixed up yet.
Linker
The linker, /usr/bin/ld in this case, will take one or more object files and
combine them together by looking at their symbol tables to resolve unknown
symbols so that it knows exactly where to go when it needs to execute specific
parts of your code. The output of this stage is (hopefully) a fully working and
executable binary. We instructed gcc to output a file named test_welibc which
is the file that the linker produces. We can look at the final symbol table
associated with the executable to see how our symbols were resolved
$ readelf -s test_welibc
Symbol table '.symtab' contains 13 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000400120 0 SECTION LOCAL DEFAULT 1
2: 0000000000400144 0 SECTION LOCAL DEFAULT 2
3: 000000000040015c 0 SECTION LOCAL DEFAULT 3
4: 0000000000400170 0 SECTION LOCAL DEFAULT 4
5: 0000000000000000 0 SECTION LOCAL DEFAULT 5
6: 0000000000000000 0 FILE LOCAL DEFAULT ABS main_test.c
7: 0000000000000000 0 FILE LOCAL DEFAULT ABS
8: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _start
9: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 __bss_start
10: 0000000000400144 23 FUNC GLOBAL DEFAULT 2 main
11: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 _edata
12: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 _end
Entry 10 is the entry for our main function and you can see that its value is no
longer 0. However, we also have 5 more symbols than before, including the _start
symbol that was said to be missing; where did they come from? These extra
symbols are needed by the ELF file format and gcc creates them as part of the
linking process. However, the _start symbol is special because that is the
location where an ELF file will start executing. The standard C library provides
its own _start symbol and since we aren't linking against it anymore (because we
specified -nostdlib) then we get the error about not finding a symbol. You'll
also see that the "Ndx" column lists the _start symbol as being UND, or
undefined.
Bootstrapping your library
Now that we know a little more about the compilation process, ELF, and why we ended up with an error, we can look at fixing it.
First, let's just run our executable and see what happens
$ ./test_welibc
Segmentation fault
A segmentation fault generally means that we accessed a region of memory that was protected in some way. It makes sense that we get this error because the ELF file format begins execution at the address associated with the _start symbol, and our _start symbol's address (the value from the symbol table) is 0. Most systems will not allow you to reference data at, or execute code at, the address 0; so, we end up with a seg fault.
This means we need to provide a _start symbol and somehow get it to call our main function. This will require us to do some assembly programming.
Defining the _start symbol
The first file in our library will be _start.S, where we will define our start
symbol, call our main function, and finally exit from the program.
To define a global symbol, we use the .globl assembly directive, like so
.globl _start
This directive is specific to the GNU assembler (gas) so if you're using something else, be sure to look at its directive syntax (for example, nasm uses "global").
In assembly, to call a function we use the call instruction, like so
call main
And to exit from the program we will make use of the exit(2) syscall through use
of the syscall instruction
syscall
However, we must determine how to pass arguments to main, receives its return value, and pass an argument to the exit syscall. These calling conventions are established by the ABI for your architecture and operating system. In the case of Linux on x86_64 we can turn to the System V ABI which has this to say
Figure 3.4: Register Usage
%rax temporary register; with variable arguments
passes information about the number of vector
registers used; 1st return register
A.2.1 Calling Convention
1. User-level applications use as integer registers for passing the sequence
%rdi, %rsi, %rdx, %rcx, %r8 and %r9. The kernel interface uses %rdi,
%rsi, %rdx, %r10, %r8 and %r9.
2. A system-call is done via the syscall instruction. The kernel destroys
registers %rcx and %r11.
3. The number of the syscall has to be passed in register %rax.
Now that we know how to pass arguments, we will want to call main and exit like so
exit(main(argc, argv));
Where argc and argv are passed to main, whose return value is passed to
exit. Before calling main we'll need to place argc into %rdi and argv
into %rsi, main will give its return value to us in %rax which needs to be
placed into %rdi, and finally we need to place the syscall number for exit
into %rax before issuing the syscall instruction. This leaves us with two
questions: where do argc and argv come from, and what is the syscall number
for exit?
argc and argv
The ABI tells us where these come from in Figure 3.11: Initial Process Stack
Argument pointers 8 + %rsp
Argument count %rsp
And additionally makes this note about %rsp's initial value
%rsp The stack pointer holds the address of the byte with lowest address
which is part of the stack. It is guaranteed to be 16-byte aligned
at process entry
Alignment will be important in just a moment. Additionally, the pointers in the
argv array are located directly on the stack at the location specified above.
At the beginning of _start, %rsp will point to argc and argv will be 8
bytes higher on the stack. We can use pop to grab the value at the location to
which %rsp points and the resultant value of %rsp will be the value we need
for argv (remember, argv is an array of pointers and to pass that as an
argument in C we pass the location of that array). In assembler this is
pop %rdi
mov %rsp, %rsi
Our stack started off being 16-byte aligned but is now 8-byte aligned due to the
pop instruction. This means we need to re-align the stack on a 16-byte
boundary because that's what is expected upon entry to main. This can be
achieved in a few ways, but we'll be explicit about what we're doing by AND'ing
%rsp with a bit mask that will preserve all but the last nibble (-16 in
two's complement), guaranteeing 16-byte alignment. Once done, we're ready to
call main.
pop %rdi
mov %rsp, %rsi
and $-16, %rsp
call main
exit syscall number
Our last item is to figure out what the syscall number for exit is. 64-bit
syscalls on Linux are defined in the header /usr/include/asm/unistd_64.h,
which has the following line
#define __NR_exit 60
Now we save off the return value from main and call the exit syscall like so
mov %rax, %rdi
movq $60, %rax
syscall
I was given a protip to specify the size when moving immediate values rather
than relying on the default size. This is why movq is used on the immediate
value of 60.
_start.S will look something like this
.globl _start # Declare global _start symbol
_start:
pop %rdi # Place argc as first arg to main
mov %rsp, %rsi # Place location of argv[] as second arg to main
and $-16, %rsp # Realign stack on a 16-byte boundary
call main # Call main function
mov %rax, %rdi # Copy return value from main as argument to exit(2)
movq $60, %rax # Set rax to syscall number for exit
syscall # Execute the syscall specified in rax
Assembling assembly
We now have an assembly file that we need to assemble. Let's try assembling it by itself to see what happens
$ gcc -o _start _start.S
/tmp/ccZH6Hhm.o: In function `_start':
(.text+0x0): multiple definition of `_start'
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../lib64/crt1.o:(.text+0x0): first defined here
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
/tmp/ccZH6Hhm.o: In function `_start':
(.text+0x1): undefined reference to `main'
collect2: error: ld returned 1 exit status
Here we compiled with glibc (because we left out -nostdlib) and we can see that
_start is in fact defined by the normal standard library because gcc tells us
that the _start symbol was first defined in the crt1.0 file. Hopefully things
are starting to make sense as to what glibc is providing and why we get errors
when we don't link against it. You can also see there is an undefined reference
to main which should be expected since we call the main function but we haven't
provided gcc with the code for it yet.
Now, let's compile without glibc and see what output we get
$ gcc -nostdlib -ffreestanding -o _start _start.S
/tmp/ccDUeM2Z.o: In function `_start':
(.text+0x1): undefined reference to `main'
collect2: error: ld returned 1 exit status
Awesome, we don't get anymore errors about multiple definitions of start and we
still get the expected error about main. However, we really want an object file
so that we can then link it to the code where main is defined. You can instruct
gcc to stop before the linking phase which will give you an object file.
$ gcc -nostdlib -ffreestanding -c -o _start.o _start.S
Now we don't get any errors since gcc didn't try to resolve any symbols.
We'll do the same thing with our main_test.c file to get an object file with main defined inside of it
$ gcc -nostdlib -ffreestanding -c -o main_test.o main_test.c
Linking objects together
So we're now at the point where we have one file defining the _start symbol and referencing the main symbol, and one file which defines the main symbol. We need to link them together so that the unresolved symbols will be fixed up so that we can execute our code with no seg faults.
First, let's verify what we have. If we dump the symbol table for _start.o we should see the _start symbol and an undefined main symbol
$ readelf -s _start.o
Symbol table '.symtab' contains 6 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 SECTION LOCAL DEFAULT 1
2: 0000000000000000 0 SECTION LOCAL DEFAULT 3
3: 0000000000000000 0 SECTION LOCAL DEFAULT 4
4: 0000000000000000 0 NOTYPE GLOBAL DEFAULT 1 _start
5: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND main
The "Ndx" column indicates whether the symbol is undefined or not, and as we expected, the main symbol is undefined. Now, let's look at main_test.o
$ readelf -s main_test.o
Symbol table '.symtab' contains 9 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS main_test.c
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 2
4: 0000000000000000 0 SECTION LOCAL DEFAULT 3
5: 0000000000000000 0 SECTION LOCAL DEFAULT 5
6: 0000000000000000 0 SECTION LOCAL DEFAULT 6
7: 0000000000000000 0 SECTION LOCAL DEFAULT 4
8: 0000000000000000 23 FUNC GLOBAL DEFAULT 1 main
Likewise we now have a defined main symbol. Now, let's link them together to get a working binary!
We can pass object files to gcc and it knows not to preprocess, compile, or
assemble them. We still need to specify not to use glibc because gcc will
automatically link against it otherwise, and in our case we want to link only
main_test.o and _start.o:
$ gcc -nostdlib -ffreestanding -o test_welibc _start.o main_test.o
And now let's look at the final symbol table to be sure everything is defined
$ readelf -s test_welibc
Symbol table '.symtab' contains 13 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000400120 0 SECTION LOCAL DEFAULT 1
2: 0000000000400144 0 SECTION LOCAL DEFAULT 2
3: 0000000000400168 0 SECTION LOCAL DEFAULT 3
4: 0000000000400180 0 SECTION LOCAL DEFAULT 4
5: 0000000000000000 0 SECTION LOCAL DEFAULT 5
6: 0000000000000000 0 FILE LOCAL DEFAULT ABS main_test.c
7: 0000000000000000 0 FILE LOCAL DEFAULT ABS
8: 000000000040015c 0 NOTYPE GLOBAL DEFAULT 2 _start
9: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 __bss_start
10: 0000000000400144 23 FUNC GLOBAL DEFAULT 2 main
11: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 _edata
12: 0000000000601000 0 NOTYPE GLOBAL DEFAULT 4 _end
Removing startup files and standard libraries
We can add two more compiler flags to get rid of a few more system defaults.
First, we'll direct gcc not to use any of the standard system startup files
with -nostartfiles. Second, we don't want to use any of the default system
libraries since we are writing our own standard C library; this is done with the
-nodefaultlibs flag. From the man pages description of the -nostdlib flag
we're already using, it seems to be a combination of -nostartfiles and
-nodefaultlibs, but we'll include them just to be sure.
Our final compilation looks like
$ gcc -nostdlib -ffreestanding -nostartfiles -nodefaultlibs -o test_welibc _start.o main_test.o
Running glibc free
So, we haven't gotten any errors during preprocessing, compiling, assembling, or linking. It's time to test our binary that doesn't use glibc to see if it runs properly
$ ./test_welibc
$ echo $?
42
Sweet! Now we can continue on with the development of the rest of the library since we can successfully run the most basic code when linking against it.
comments powered by Disqus