This note is a foundations-first map of systemd. It is grounded in the official project site systemd.io (project definition, admin docs such as FAQ, Diagnosing Boot Problems, Network online targets, API file systems) and the overview on Grokipedia’s systemd page.

Reading order for this article: Part A (must know) → Part B (daily ops) → Part C (common gotchas) → Part D (advanced doors only).

---

Part A — Foundations (learn these first)

A0. Official one-breath definition

systemd.io opens with something close to this (paraphrased carefully):

1. systemd is a suite of basic building blocks for a Linux system.
2. It provides a system and service manager that runs as PID 1 and starts the rest of the system.
3. It emphasizes parallelization, socket and D-Bus activation, on-demand daemon start, cgroup process tracking, mount/automount maintenance, and dependency-based service control.
4. “Other parts” of the suite include logging, basic system configuration helpers (hostname, date, locale), login/session tracking, and optional daemons for network, time sync, name resolution, etc.

systemd is not only “an init script replacement”—it is PID 1 plus a coordinated suite; for foundations, master PID 1 + units + systemctl + journal first.

---

A1. What problem PID 1 is solving

1. The kernel starts one first userspace process—historically “init.”
2. That process must bring up mounts, devices, services, and keep the machine coherent.
3. Old SysV-style boots often ran serial shell scripts and guessed life via PID files.
4. systemd’s foundational bet (also summarized on Grokipedia): declare services as units, build a dependency graph, start independent work in parallel, track processes with cgroups, activate some work on demand.

PID 1 is the system’s process and dependency manager, not “just the first shell.”

---

A2. The three questions that unlock systemd

Ask these before any advanced feature:

1. What is the unit? (name + type: sshd.service, multi-user.target, …)
2. What state is it in? (active, failed, inactive, … via systemctl status)
3. What did it say? (journal via journalctl -u …)

If you can answer those three under stress, you already outrank “I only know how to reboot.”

unit → status → journal is the foundation loop.

---

A3. Units — the central beginner object

A3.1 Definition

1. A unit is something systemd knows how to start, stop, or track.
2. Most units are described by a unit file (INI-style: sections and Key=Value).
3. The filename suffix is the type: .service, .timer, .socket, .target, …
4. Units form a graph: “start me after X,” “I want Y,” “I conflict with Z.”

A3.2 Unit types you should recognize early

Live in .service + .target first; meet .timer/.socket second; leave slices/scopes until you need cgroup policy.

A3.3 Where unit files live (precedence)

From admin practice also reflected in the systemd FAQ:

1. Vendor defaults: /usr/lib/systemd/system/ (or /lib/systemd/system/) — packages own these.
2. Administrator overrides: /etc/systemd/system/ — wins over vendor; packages should not stomp your files here.
3. Runtime: /run/systemd/system/ — temporary.
4. Drop-ins: /etc/systemd/system/foo.service.d/*.conf — change one setting without copying the whole unit.

How to customize safely:

1. Prefer a drop-in for small changes.
2. Or copy vendor unit to /etc/systemd/system/ and edit the copy (FAQ guidance).
3. Always run systemctl daemon-reload after unit file edits.
4. Then restart/try-reload the unit as appropriate.

Never “vim the file under /usr/lib and hope”—use /etc or drop-ins.

A3.4 Three sections of a simple .service

[Unit]
Description=Example demo service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/demo
Restart=on-failure

[Install]
WantedBy=multi-user.target

1. [Unit] — human description + relationships to other units.
2. [Service] — how the process runs (ExecStart, type, restart policy).
3. [Install] — what enable hooks into (usually a target’s .wants/).

Relationships / recipe / enablement — three jobs, three sections.

A3.5 Requires vs Wants vs After (say it out loud)

1. After=B — if both start, order me after B. Does not by itself start B.
2. Wants=B — please also start B; if B fails, I may still continue.
3. Requires=B — stronger coupling; B’s failure can fail me.
4. Wants= + After= together is a common “soft dependency with order” pattern.
5. network-online: to delay until “online,” use both After=network-online.target and Wants=network-online.target.

Order is not requirement; requirement is not “network is happy.”

---

A4. Targets — the replacement for “runlevels”

1. Classic runlevels are obsolete as the main mental model.
2. Targets carry similar goals (e.g. graphical multi-user).
3. Default boot goal is controlled by default.target (often a symlink under /etc/systemd/system/).
4. Common goals:
   • multi-user.target — multi-user, typically non-graphical full system
   • graphical.target — pulls multi-user + display stack
   • rescue.target / emergency.target — recovery modes
5. Switch now (not next boot): systemctl isolate some.target.
6. See active targets: systemctl list-units --type=target.

Targets are named milestones; default.target is “where boot aims.”

---

A5. Boot story you can narrate

You do not need every early-boot unit. Narrate this sequence:

1. Firmware/bootloader loads kernel (+ initrd as configured).
2. Kernel starts userspace; systemd becomes PID 1.
3. systemd mounts API filesystems early (proc, sys, dev, run, …)—see A6.
4. systemd reaches early targets (local FS, sockets, basic system…).
5. Services start as dependencies allow, often in parallel.
6. Machine approaches multi-user.target or graphical.target.
7. Some units stay inactive until activation (socket/timer/path…).

The illustrative boot log on systemd.io (“Reached target …”, “Started …”) is exactly this story in console form.

Boot = activate a graph until a target is satisfied.

---

A6. API file systems systemd mounts early

These are not “mystery disks.” They are kernel/userspace APIs that look like mounts:

1. /sys — devices, drivers, kernel info
2. /proc — processes, kernel settings
3. /dev — device nodes
4. /run — early runtime sockets/files (userspace)
5. /tmp — often volatile temp (may be tmpfs)
6. /sys/fs/cgroup — control group hierarchy
7. plus other special mounts (/dev/shm, /dev/pts, debug/securityfs variants, …)

Facts that save confusion:

1. They generally do not appear in /etc/fstab by default.
2. systemd mounts them early because software expects them.
3. You may list them in fstab to change options (e.g. /tmp size).
4. Some can be masked to disable auto-mount (not all—see the doc’s “(X)” set).
5. “Weird mounts in mount output” are often these APIs, not a broken fstab.

API mounts are interfaces; don’t “clean them up” like junk folders.

---

A7. cgroups in one foundation sentence

1. Linux cgroups group processes for accounting and limits.
2. systemd puts services in cgroups so “the service” is a tree, not a lonely PID file.
3. That is how systemd knows which processes belong to a unit (also why ps can show cgroup columns).
4. Resource limits (MemoryMax=, CPUQuota=, …) attach here—learn limits after start/stop/status.

cgroups answer: “which processes are this service?”

---

Part B — Daily operations (still foundations)

B1. systemctl

Learn in this order:

1. systemctl status UNIT — loaded? active? failed? recent log lines
2. systemctl start|stop|restart UNIT — runtime
3. systemctl reload UNIT — reload config if the unit supports it
4. systemctl enable|disable UNIT — boot persistence
5. systemctl is-enabled UNIT / is-active UNIT — scripting-friendly answers
6. systemctl daemon-reload — after unit file changes
7. systemctl list-units --failed — what is on fire
8. systemctl reset-failed — clear failed markers after fixing
9. systemctl cat UNIT — show the unit text systemd is using
10. systemctl show UNIT -p Fragments -p DropInPaths — where config came from (when debugging overrides)

status → logs → fix unit/drop-in → daemon-reload → restart → status again.

---

B2. Three levels of “off” (concept you will meet)

1. stop — not running now
2. disable — not started as part of normal boot wants
3. mask — symlink to /dev/null; activation is blocked hard

stop ≠ disable ≠ mask; use the weakest tool that solves the problem.

---

B3. Journal foundations

1. Services’ stdout/stderr commonly go to the journal, not your SSH terminal.
2. That is why systemctl start failures say “see system journal and systemctl status.”
3. journalctl -u UNIT — unit-centric reading.
4. journalctl -b — this boot; -b -1 previous boot.
5. journalctl -f — follow.
6. journalctl -p err — priority filter.
7. journalctl --since "10 min ago" — time window.
8. journalctl --full — avoid over-truncation in some views.
9. Persistence depends on setup (/var/log/journal vs runtime-only under /run).
10. Text log stacks (rsyslog, etc.) may still exist; journal can forward—know which path your distro emphasizes.

No log on your TTY does not mean silence—check the journal.

---

B4. systemd-analyze — boot performance basics

1. systemd-analyze / time — coarse timing of boot phases.
2. systemd-analyze blame — which units took long.
3. systemd-analyze critical-chain — what sat on the critical path.

Use these when the question is “why is boot slow?”, not when a single service misbehaves (then status+journal first).

blame lists slow units; critical-chain explains the bottleneck path.

---

B5. Network targets

Facts:

1. network-pre.target — order firewall-like work before network configuration begins; passive; pull it in only if needed.
2. network.target — “network management stack has been started”; does not mean “interfaces have IPs.” Mainly useful so shutdown order is clean (After=network.target ⇒ stop before network tears down).
3. network-online.target — actively means “network is up” as defined by the network manager’s wait service; can delay boot (default wait timeout discussed in the docs—often on the order of ~90s).
4. Client-ish things that need connectivity may Wants= + After= network-online.target.
5. Server software often should not liberally require online—local listen can work earlier; dynamic network is normal.
6. “Up” is implemented by units like NetworkManager-wait-online.service or systemd-networkd-wait-online.service—whichever stack you use.
7. Check: systemctl is-enabled NetworkManager-wait-online.service systemd-networkd-wait-online.service

network.target ≠ online; network-online.target is a deliberate, sometimes expensive, sync point.

---

B6. Per-user systemd

1. System instance: machine-wide services.
2. User instance: systemctl --user for user units.
3. User manager may not exist on minimal SSH-only sessions.
4. loginctl enable-linger USER keeps a user manager without interactive login (headless pattern).
5. Needs runtime dir / user bus pieces; “Failed to connect to bus” is often session setup, not your unit syntax.

Learn system services first; add --user when you need user-level timers/services.

---

Part C — Foundation troubleshooting

C1. Stuck boot — numbered triage

1. Remove quiet boot flags (quiet, rhgb) to see messages.
2. Decide: hang before systemd messages, or after “Welcome… / Starting… / [ OK ] Started…”
3. Wait up to several minutes—some units time out before boot continues.
4. Try other VTs (Ctrl+Alt+F#) if display manager is the broken piece.
5. If you get a shell: journalctl -b, systemctl list-jobs, systemctl --failed.
6. Rescue: kernel cmdline systemd.unit=rescue.target (or 1).
7. Emergency: systemd.unit=emergency.target or emergency; remount root RW if needed: mount -o remount,rw /.
8. Nuclear userspace: init=/bin/sh (may need to remount RW; libraries must be intact).
9. Debug shell early: systemctl enable debug-shell.service or cmdline systemd.debug_shell=1 → often tty9; disable after debugging (security).
10. Kernel requirements note from the doc: populated /dev (devtmpfs), cgroups support—embedded builders hit this.

See messages → get a shell → journal + list-jobs → rescue/emergency ladder.

C2. Failed service

1. systemctl start foo.service fails with a generic job error—expected; output is not your SSH tty.
2. systemctl status foo.service — Active state, Result, Exec exit code, snippet of logs.
3. journalctl -u foo.service -b --no-pager — fuller story.
4. Fix config/binary; daemon-reload if unit files changed; restart; re-check status.
5. If “ordered after network but still too early,” re-read network targets (B5)—classic FAQ.

status + journal beat guessing; exit codes live on the Exec line.

C3. Which service owns a process?

1. ps with cgroup column, or inspect /proc/PID/cgroup.
2. This is how “who spawned this?” becomes answerable without PID file archaeology.

Ownership is cgroup identity under systemd.

---

Part D — Pocket distillations

---

Part E — Minimal lab

# 1) PID 1 identity
ps -p 1 -o pid,comm

# 2) Default goal
systemctl get-default

# 3) Active targets
systemctl list-units --type=target --no-pager | head

# 4) A real unit (name varies by distro)
systemctl status ssh.service || systemctl status sshd.service

# 5) Its journal
journalctl -u ssh.service -n 30 --no-pager 2>/dev/null \
  || journalctl -u sshd.service -n 30 --no-pager

# 6) Failed units
systemctl --failed

# 7) Boot slow units (if curious)
systemd-analyze blame 2>/dev/null | head

# 8) Network online waiters (which stack?)
systemctl is-enabled NetworkManager-wait-online.service \
  systemd-networkd-wait-online.service 2>/dev/null

If this lab feels natural, foundations are landing.

---

Part F — Advanced doors (do not start here)

Only after A–C feel boring, open these from systemd.io when you need them:

1. Socket activation details & portable services
2. systemd-boot / UKI / measured boot / TPM topics
3. Credentials, factory reset, image building (mkosi)
4. Cgroup delegation for containers
5. Homed / user records
6. Full sandboxing directives catalog (ProtectSystem=, …)
7. Philosophical debates (modularity vs suite)—context, not day-1 ops

Advanced pages exist so foundations stay small; visit them with a concrete problem, not FOMO.

---

Closing

systemd becomes learnable when you refuse to swallow the whole suite at once. Start from the official spine—PID 1 starts the rest of the system—then master units, targets, systemctl, journalctl, and the few targets that confuse everyone (network-online).

For narrative history and debate map, Grokipedia’s systemd page remains a useful overview. For project-authored definitions and admin docs, start at systemd.io and the linked FAQ/debugging/network-online/API-FS pages—then deepen with man pages (systemd.unit, systemd.special, systemctl, journalctl) when a real unit forces the next question.