The ptrace syscall

ptrace (“process trace”) is a system call in Unix and Unix-like operating systems that intercepts system calls. It’s a powerful tool that enables tools like debuggers (e.g., gdb), reverse engineering tools, tracing, code injection, and even simple sandboxing. (see proot for an example of a ptrace sandbox). The most interesting part of ptrace is that you can do all of these things completely in user space (even sandboxing!).

Sandboxing. Roughly how a ptrace sandbox works:

• Fork a child process to run the untrusted code

• Set the ptrace syscall to trace the child process and intercept and monitor the child’s syscalls

• Inspect the child process syscalls and arguments. Maybe enact some sort of security process, logging, modification, or something else.

Google’s gVisor can use ptrace as a backend for its sandboxing. It’s a lot more complicated than a vanilla ptrace sandbox and works much more like User-mode Linux (UML). 

Monitoring: ptrace powers strace (Linux) and ktrace (macOS), which are userspace monitoring utilities that expose ptrace functions as a binary. You can monitor or modify system calls for a running program. Solaris had DTrace (by Bryan Cantrill). 

Some other cool ideas you could do with ptrace

• Time-travel debugging (record the whole state with ptrace)

• Process-aware firewalling

• Transparent network traffic encryption (intercept send/recv sendto/recvfrom)

• Resource limits and throttling (intercept mmap, read, write, etc.)

• Live process migration or checkpointing (capture the entire state and copy somewhere else) 

• Custom process scheduling (intercept fork, clone, exit, wait, waitpid, etc.)

• Testing (fault injection, fuzzing, etc.)