But there *are* other ways to run a program that execve, plenty of them :)

And after understanding what went wrong I guess we should consider which additional log lines we need because in production we don't always have the luxury of debugging with strace (heavily loaded servers permissions...) What do you think?

Would also recommend checking out www.frida.re for a cross-platform solution. It's a toolkit that lets you build custom instrumentation tools – no source code or debug symbols needed. frida-trace is an example tool based on intercepting calls to exported functions (although the toolkit supports hooking any function as long as you know its address in memory). For example:
frida-trace -i open <pid>
Note that this attaches to a running ssh process. Frida also supports spawning processes, but that's not yet implemented in the frida-trace tool (hope someone interested will submit a PR for this at some point). :)

Wonderful article. Thanks Julia!

I just came across a very practical use of proc (via reddit), that reminded me of this post. Apparently, a new version of GNU screen just came out, and breaks compatibility with existing instances, so if you have any detached screens, you won't be able to reattach with the new version. With proc you can do:

$ /proc/`pidof screen`/exe -x

And it will run the old version of screen to let you reattach. Same for any other upgraded software where you still need access to the old version.

Source: http://www.reddit.com/r/lin...

You should check out https://github.com/draios/s...

"Sysdig is open source, system-level exploration: capture system state and activity from a running Linux instance, then save, filter and analyze.Think of it as strace + tcpdump + lsof + awesome sauce. With a little Lua cherry on top."

You may already be familiar with this trick, but if not you'll love LD_PRELOAD if you love strace. You can use it to hook library function calls from a closed source application.

The only way to run a program is with the execve system call. There aren’t other ways.

Nitpick: not the only way. You could mmap an ELF as executable and jump into it, or copy a file into memory then mprotect it as executable and do the same. But surely no one would try such a thing... ;)

Amazing! I love nitpicks like that :)

I have not yet played with LD_PRELOAD! So excited to try that.