Thank you, this is an excellent guide! It is well researched and clearly written. I hope you'll write about things from your to do list in the future.

You mentioned "dpkg -S". One thing I like about FreeBSD is how easy it is to find sources of built-in utilities ('w' is a variant of 'uptime'):

$ ls `whereis  -q -s w`

extern.h        Makefile        Makefile.depend pr_time.c       proc_compare.c  uptime.1        w.1             w.c

FreeBSD is a great operating system in every way.

In your zombie demonstration program, the output is backwards: the part that claims to be the child process is actually the parent and vice versa.

Fantastic article. I learned a ton about htop and the underpinnings of Linux in general. Thanks!

"These commands do the same thing: kill 10089". This is actually incorrect. If you look in the man page for kill it says "kill 123 543 2341 3453 Send the default signal, SIGTERM, to all those processes." So the default use of kill is SIGTERM, not SIGINT. Thanks for your website, though. I learned a lot so far.

Many thanks, and a minor suggestion:
strace uptime 2>&1 | grep open
can be simplified to
strace -e open uptime

EDIT: just noticed that you had suggested it already :-|

This is a great article with lots of useful info. You might want to add a section on the lsof command. It's always been very useful for me for viewing open files, descriptors and lsof -i is easier to remember than a few netstat flags.

Tiny fix
"Right, but we already have systemd-journald running. Do we need rsyncd as well?"

should be rsyslogd.

Thank you. Very helpful.

Hello, I know this post is 5 years old, but I have question.. I'm so confused about cpu utilization and cpu usage(CPU%) of process. For example, I have a system with 2 core, a process with 30 threads.
When I see htop, cpu utilization of core0 is 12%, core1 is 40%. And CPU% of my process(with 30 threads) is 80%. How can this possible??? A process' CPU% is higher than sum of utilization of every cores.

This is great! Very informative to learn the basics. Thank you sir.

Hi,
I tried to use htop after permanent high cpu processing.
One of the 4 cpu goes all the time over 90%.
I dont find a solution.
Is here anybody who can help me?
Please contact : koffie@gmx.net

Man, kudos for you. This is quite in-depth and I quite enjoy reading it

Wonderful article. This post explains some things that so far I had only skimmed over in years of using Ubuntu & other Linux distros.

great post, very informative. I have two questions:

1. You wrote, "If you replaced `sleep(20)` with `while (true) ;` then the zombie would be gone right away." - why is this the case?

2. In the "SHR - Shared Mem size" code section, you have `printf("Child used extra 2M\n");` why is this "child", rather than "adult"? Is it because `pid` is the _return value_ of fork? If you were to replace `if (pid != 0)` with `if (getpid()!=0))`, would you replace this line with `printf("Parent used extra 2M\n");`

A suggested minor fix/addition.

Following along, and regarding the "uptime" section...

On my system (archlinux) the "openat" system call is used, rather than "open":

```bash```
$ uname -r
4.13.3-1-ARCH
```

```bash```
$ strace -e openat uptime
...
openat(AT_FDCWD, "/proc/uptime", O_RDONLY) = 3
...
```

Great post! Sometimes I see the CPUs (e.g. 128 CPUs) statuses are largely red, meaning kernel usage. My experience is that this is often due to heavy IO (e.g. multiple processes reading the same file). Can you please comment on the relationship between kernel CPU usage and IO?

Outstanding work

This is pure gold.

Thanks for this great article. A minor suggestion, strace has "-o" switch that redirects output from strace to a file. From the strace manual page:

-o filename Write the trace output to the file filename rather than
to stderr. Use filename.pid if -ff is used. If the
argument begins with '|' or with '!' then the rest of the
argument is treated as a command and all output is piped
to it. This is convenient for piping the debugging
output to a program without affecting the redirections of
executed programs.

Nice write up. However, be really careful when making your minimal install of Linux distro even slimmer, as it may break the system on the next boot - you never know ;-) Personally I'd rather build my own golden image using Arch Linux, Gentoo or Debian/Ubuntu netinstall to achieve that slimmest possible install goal.

At least I won't recommend removing rsyslog as my understanding is that it is required to receive systemd journal forwarded messages and save to traditional log files under /var/log/*, don't disable cron either as it is so simple and effective ;-)

sudo echo won't work because echo is a shell builtin (same applies to shell redirect > >> etc.), it only works with executables.

A few tips:

Purge a package (traditional way) apt-get purge package, you can use apt purge package to save on typing the extra.

Do not purge open-iscsi if you want to connect to iSCSI (IP-based) storage or expose some block device on the server as iSCSI target.

Hi,
A little precision:
You say that load average counts the running processes but it includes the uninterruptible ones as well.

echo is both a shell builtin AND a executable. :)

# bash
# type echo
echo is a shell builtin
# which echo
/bin/echo

#sh
# type echo
echo is a shell builtin
# which echo
/bin/echo

great review.

thanks for the great article - I have a quickie guide on doing this in a vagrant vm instead of a droplet: https://alexlo03.wordpress....

Thanks for the article. It's really interesting.

Great article. This revealed many things that I've been wondering about procfs. I think the "Processes" section, which was about your distro and your cleanup task, seemed a bit out of the place. Perhaps it could have been better if you went into explaining some more htop/procfs features. Thanks anyway, it was a fun article.

Cool! Thanks for sharing this post!

nice, thanks for sharing; good opportunity i had understanding/reviewing the basics
so many awesome stuffs amassed and well versed... :)

Regarding your section on 'nice' and plan to look more at scheduling, you may also want to investigate 'ionice' and 'schedtool'.

Not a great idea to remove lvm

Why?

Minor typo: "Why is there a bash at the beginning?" Should say "dash" not "bash".