Paths

/usr/lib/systemd/system default unit files installed by packages. Should not be modified. Least priority.
/usr/lib/systemd/user
/etc/systemd/system custom unit files and overrides. Second priority.
/etc/systemd/user custom user unit files.
/etc/systemd/system/multi-user.target.wants symlinks to units started on multi-user target.
/run/systemd/system automatically generated unit files. Max priority.

Timers

man systemd.timer

OnActiveSec= Defines a timer relative to the moment the timer unit itself is activated.
OnBootSec= Defines a timer relative to when the machine was booted up.
OnStartupSec= Defines a timer relative to when the service manager was first started.
OnUnitActiveSec= Defines a timer relative to when the Unit= was last activated.
OnUnitInactiveSec= Defines a timer relative to when the Unit= was last deactivated.
OnCalendar= realtime (i.e. wallclock) timers. man 7 systemd.time
RandomizedDelaySec= Delay the timer by a randomly selected, evenly distributed amount of time between 0 and the specified time value.
Unit= The unit to activate when this timer elapses. If not specified, defaults to the unit with the same name as the timer.

Example to shutdown a machine at 01:30 every day. /usr/lib/systemd/system/shutdown.timer

[Unit]
Description=Shutdown the computer at 01:30 every day

[Timer]
OnCalendar=*-*-* 01:30:00
Unit=poweroff.target

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now shutdown.timer
systemctl status shutdown.timer

Units

Sections

[Unit] unit description and dependencies.
[Service] how to start and stop the service.
[Install] target to load the unit.

Show commands

systemctl --type=service show only service units.
systemctl list-units --type=service show all active service units.
systemctl list-units --type=service --all show also inactive units.
systemctl --failed --type=service show failed units.
systemctl status -l name.service shows detailed information about unit.
systemctl list-dependencies <unit> show a list of dependencies.
systemctl cat <unit> show the unit's file.

Control units

systemctl start <unit> start unit
systemctl stop <unit> stop unit
systemctl enable <unit> start at boot
systemctl enable --now <unit> start at boot and start now
systemctl disable <unit> do not start at boot
systemctl disable --now <unit> do not start at boot and stop now
systemctl mask <unit> make sure nothing starts the unit
systemctl unmask <unit> allow the unit to be started

Reload configuration

systemctl daemon-reload
systemctl reload <unit>

Status

Loaded has been processed and the unit is active.
Active(running) running with one or more active processes.
Active(exited) successfully completed a one-time run.
Active(waiting) running and waiting for an event.
Inactive(dead) not running.
Enabled will be started at boot time.
Disabled will not be started at boot time.
Static cannot be enabled but may be started by another unit automatically.

Dependencies

Wants: weak requirement; if the units fail it has no impact and the unit will start.
Requires: stronger requirement; if this unit gets activated, the units listed will be activated as well. If one of the other units fails to activate, and an ordering dependency After= on the failing unit is set, this unit will not be started. Besides, with or without specifying After=, this unit will be stopped if one of the other units is explicitly stopped.
Requisite: similar to Requires=, but if the units listed here are not started already, they will not be started and the starting of this unit will fail immediately.

There are more man 5 systemd.unit

Service types

simple (default): it will be considered up immediately. Active for as long as the process is running. The process must not fork (run in the background)(daemon). Do not use this type if other services need to be ordered on this service, unless it is socket activated (as long as it starts running systemd proceeds with other services).
exec: like simple but will wait for the process to finish before it proceeds with other services.
forking: Used for programs that have a daemon option. systemd considers the service started up once the process forks and the parent has exited. For classic daemons use this type unless you know that it is not necessary. You should specify PIDFile= as well so systemd can keep track of the main process.
oneshot: like simple, but it will be considered up when the process exits. This is useful for scripts that do a single job and then exit. You may want to set RemainAfterExit=yes as well so that systemd still considers the service as active after the process has exited.
dbus: like simple but the service is considered up when the specified BusName= appears on DBus's system bus.
notify: similar to exec, but with the stipulation that the daemon will send a signal to systemd when it is ready.
idle: like simple but systemd will delay execution of the service binary until all jobs are dispatched. Note that this type is useful only to improve console output.

man 5 systemd.service

Edit a unit

The unit file is most likely in /usr/lib/systemd/system/. You can find the location with systemctl status <unit>

Edit the unit file.

systemctl edit --full <unit>

Edit a drop-in replacement. This will overwrite the options without modifying the unit. It creates an override.conf in /etc/systemd/system/<unit>/.

systemctl edit <unit>

Mounts

man systemd.mounts

[Unit]
Description=Mount a directory
Conflicts=umount.target
Before=local-fs.target umount.target
After=swap.target

[Mount]
What=tmpfs
Where=/tmp
Type=tmpfs
Options=nosuid,nodev
DirectoryMode=1777

Manually mount and umount

Mount

systemd-mount /mnt/data
systemd-mount /mnt/usb /dev/sdc1

Umount

systemd-umount /mnt/data
systemd-mount --umount /mnt/usb

More information in man 1 systemd-mount

Sockets

man systemd.sockets

A socket creates a listening socket which enables the service only when someone connects to the socket. It can create an instance per connection or one instance for all connections. The socket will start the service with the same name as the socket.

[Unit]
Description=Example socket

[Socket]
ListenStream=9001
ExecStartPost=
ExecStopPost=

[Install]
WantedBy=sockets.target

Targets

A target is a groups of units.
They provide different states of functionality for the system.
runlevels is the way SysV init manages services and different states.
Services started on a target are located inside the wants directory like /etc/systemd/system/multi-user.target.wants

Targets (there are more):

poweroff.target: Shut down and power off the system
emergency.target: minimal set of units; enough to fix the system.
rescue.target: fully functional without non-essential units. Rescue shell.
multi-user.target: everything running. Used in servers.
graphical.target: desktop environment. Used in desktops.

Runlevels (SysV init):

0: runlevel0.target or poweroff.target
1: runlevel1.target or rescue.target
2: runlevel2.target or multi-user.target
3: runlevel3.target or multi-user.target
4: runlevel4.target or multi-user.target
5: runlevel5.target or graphical.target
6: runlevel6.target or reboot.target
Runlevels equivalences can be seen with: ls -l /usr/lib/systemd/system/runlevel?.target

Default target: systemctl get-default or ls -l /usr/lib/systemd/system/default.target Current loaded targets: systemctl --type target or systemctl list-units --type target All the targets: systemctl --type target --all Change default target: systemctl set-default graphical.target Change current target: systemctl isolate multi-user.target Enter rescue mode: systemctl rescue Enter emergency mode: systemctl emergency

Examples

Start after desktop manager

[Unit]
Description=App desktop startup
Requires=display-manager.service
After=display-manager.service
AssertPathExists=/home/username/folder
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Environment="DISPLAY=:0"
ExecStartPre=/usr/bin/sleep 30
ExecStart=/path/to/biin
Restart=always
RestartSec=60s
User=username
Group=username

[Install]
WantedBy=multi-user.target

User units

Lingering

By default user units only run when the user is logged in. This behavior can be changed by enabling lingering with the following command:

sudo loginctl enable-linger $USER

The service will not be started at boot and will run regardless if the user is logged in or not.

udev

Monitor udev rules for changes

udevadm monitor --property --udev

Monitor udev rules for a specific device type

udevadm monitor --property --udev --subsystem-match=block/partition

The main path is /etc/udev/rules.d/. To write a rule, simple create a file with the .rules extension.

ACTION=="add", SUBSYSTEM=="block", ENV{ID_PART_TABLE_UUID}=="<PTUUID>", TAG+="systemd", ENV{SYSTEMD_WANTS}="automatic-backup.service"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="01:23:45:67:89:ab", NAME="ethernet0"
ACTION=="remove", SUBSYSTEM=="usb", ENV{ID_VENDOR_ID}=="05a9", ENV{ID_MODEL_ID}=="4519", RUN+="/path/to/your/script"
ACTION=="change", SUBSYSTEM=="power_supply", ATTRS{type}=="Mains", ATTRS{online}=="0", RUN+="/usr/bin/systemd-run --machine=target_user@.host --user notify-send 'Changing Power States' 'Using battery power'"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:00:00:00:00:00", ATTR{type}=="1", NAME="eno1"

Get more information on all the options

man udev
man udevadm

systemd-run

Run programs in transient scope units, service units, or path-, socket-, or timer-triggered service units

Run a command after 5 minutes as a replacement for at.

systemd-run --user --on-active=5m /bin/touch /tmp/foo

Unbrick the firewall example. Not tested.

/usr/sbin/nft list ruleset > /tmp/nftables.conf
systemd-run --on-active=5m --unit=unbrick-the-firewall /usr/sbin/nft -f /tmp/nftables.conf
/usr/sbin/nft -f /tmp/new-nftables.conf
echo "If everything works, run:"
echo "systemctl stop unbrick-the-firewall.timer"

machinectl

Run a command that can interact with systemd and podman as another user.

sudo machinectl shell --uid username .host /bin/bash