systemd as an init system is pretty good. Once upon a time I worked on porting launchd to FreeBSD and so I have some familiarity with the silliness of most init systems.

systemd as a katamari at the root level of most Linux systems is not “pretty good.” There have been numerous tendrils of what is understood to be “systemd” which are of lesser quality and have resulted in security issues.

Anyways, I hope you get the point. systemd as an init system: good. systemd as a operating system: bad.

The drama du jour is about the latter.

One should not obey in advance. Especially in the domain free and open source software which is literally a political project.

I stumbled into this blog post through Planet Debian by a debian maintainer which is patently absurd.

Recently, the free software Nazi bar crowd styling themselves as “concerned citizens” has tried to start a moral panic by saying that systemd is implementing age verification checks or that somehow it will require providing personally identifiable information.

The author is correct insofar that systemd did not add age verification. However most of the folks upset with the change are upset that their Linux systems are obeying in advance.

systemd did make changes in order to obey. To take part in anti-free restrictions under the guise of “age verification” From the pull request

Stores the user’s birth date for age verification, as required by recent laws in California (AB-1043), Colorado (SB26-051), Brazil (Lei 15.211/2025), etc.

The whole motivation of the change was to obey in advance to these unjust laws.

The author then goes on to make some equally absurd claims about how this functionality is important for porents to implement controls on computers, for the children! Clearly this person must not know any actual children, or even parents for that matter. Children are excellent at finding ways to circumvent restrictions. The idea that a user-modifiable piece of data on a local machine should be trusted for “parental controls” is so detached from reality that I originally thought they were making a sarcastic joke.

I think this tongue-in-cheek systemd-censord post does better than anybody to exclaim how absolutely ludicrous this obeying in advance is:

Systemd units will be created for every desired censorship function, and will be started based on the user’s location. For example, a unit for Kazakhstan will implement the government-required backdoor, a unit for China will implement keyword scans and web access blocks (more on this later), a unit for Florida will ban all packages with “trans” in the name (201 packages in current stable distribution), a unit for Oklahoma will ensure all educational software is compliant with the Christian Holy Bible, a unit for the entire United States will prevent installation of any program capable of decoding DVD or BluRay media, and a unit for California will provide the user’s age to all applications and all web sites from which applications may be downloaded. As can be seen, multiple units may be started for a given location.

Do not obey in advance.

The biggest thing I like about Pulseaudio is having independent mixer controls for every application. The pulsemixer tool sits in an open terminal all day to make it easier for me to move audio levels around.

One big requirement was that I wanted to be able to make the volume in my headphones louder than it was for other participants in the call.

Using pacctl I set out to solve this with:

• Create two null sinks:
  • Music
  • Playback
• Create three loopback devices:
  • Two from Music sink
  • Microphone

pactl load-module module-null-sink sink_name=Playback sink_properties=device.description=Stream
pactl load-module module-loopback source=Music.Monitor
pactl load-module module-loopback source=Music.monitor
pactl load-module module-combine-sink sink_name=Dualie slaves="Music,Playback" channels=2 channel_map=left,right

Then I:

• assigned my music to the Music null sink.
• Move “Music loopback” and “Microphone loopback” to the Playback sink
• Move another Music loopback to my headphones.

Finally, I select “Monitor of Stream” as the input device in the video call.

With a rhythm driving in the background, the hacking can commence!

Only after a couple weeks of usage did I notice how peaceful everything was. No notifications, no alerts, no sound of any form!

As desirable as some peace and quiet might be, I do need to hear things from my computer every now and again. I scurried off to diagnose the problem.

I use a simple terminal-based UI for managing sound on my machines pulsemixer, which was not showing any sound cards attached to the machine. Searching for similar issues with this hardware, a search engine brought me to a Fedora-related thread where a particular package was referenced as having solved the problem: sof-firmware.

For well over a decade I have used openSUSE, the reasons are not relevant for this post, but Fedora-alikes and openSUSE are usually close enough in packages, naming, and ethos, that forum threads and discussions for one can apply to the other:

zypper in sof-firmware

Suddenly Slack was knocking down my door again, and I had sound! Ta-da! Of course I closed Slack, muted the sound, and went back to work.

Generally speaking Linux hardware support has gotten so good in the last decade that I forget the olden days when we had to walk uphill both ways in the snow to get a working set of desktop drivers.

Maybe next year will finally be the year..

It turns out that this is due to a backwards incompatible change which OpenSSH released in 9.2 earlier this year:

ssh(1): add a new EnableEscapeCommandline ssh_config(5) option that controls whether the client-side ~C escape sequence that provides a command-line is available. Among other things, the ~C command-line could be used to add additional port-forwards at runtime.

The reason for this change is to support some sandboxing use-case which I don’t entirely understand but also don’t need, so I needed to add the following option to my host entries in ~/.ssh/config:

Host foobar
    Hostname 172.16.1.1
    EnableEscapeCommandLine yes

This can also be configured on the command line with -o EnableEscapeCommandline=yes. Happy port forwarding!

The alternative for this post could be “unix permissions are insane”.

Regardless of whether spawning a child or doing something within the process for modifying “who” is running a program is effectively the same. The program runs with the user id (uid) of the caller, so in the case of your shell which is running as you (often uid of 1000) and spawned process will inherit the same uid .

In a C program, I can however reassign my process’ uid with the setuid(2) system call:

setuid() sets the effective user ID of the calling process. If the calling process is privileged (more precisely: if the process has the CAP_SETUID capability in its user namespace), the real UID and saved set-user-ID are also set.

(In Rust this function is helpfully exposed via the nix crate.)

My normal shell user is not the root user and therefore in just about every program I execute, they cannot use setuid(2) to change to any other user id.

If processes spawned by my user cannot assume other user ids, how the heck does sudo do it?

Based on the man page excerpt above, it’s easy to assume that perhaps the process has the CAP_SETUID capability? Per-file capabilities are a Linux-ism which exposed to user-land via libcap:

Libcap implements the user-space interfaces to the POSIX 1003.1e capabilities available in Linux kernels. These capabilities are a partitioning of the all powerful root privilege into a set of distinct privileges.

These capabilities can provide the desired setuid functionality, but they can also be dangerous and lead to vulnerabilities. This blog post has a simple demonstration of using CAP_SETUID to gain root on a machine. We can check whether the sudo binary has this capability with the getcap program, which is not always in the distribution’s default packages:

❯ sudo getcap `which sudo`

❯

Unfortunately it looks like sudo has no special file capabilities, fortunately there is one other way to for a program to run with root privileges, but if you don’t know where to look, good luck finding it!

The file system contains a little bit more information about an executable referred to as the “setuid bit”. Using chmod you take a binary that you own, set the “setuid bit” and then whenever any user runs the binary, it will run as your user. Naturally if a binary is owned by root with the setuid bit set, any caller of that binary will run the binary as root.

This is exactly what sudo does, which you can check just by exampling the file mode:

❯ ls -lah `which sudo`
-rwsr-xr-x 1 root root 181K May 27 07:49 /usr/bin/sudo
❯

Note the rws at the beginning of the mode string, this means read (r), write (w), and setuid (s) are all set at the user level on the file. Additionally the executable (x) bits allow all users within the root group, and everybody else to execute the file.

Sudo basically starts out as root and then will read its configuration file and validate/drop permissions as necessary. Since the program is running with the root level privileges, it can easily call setuid(2) type functions and change to any user on the system.

Should you wish to explore exactly how sudo works, the source code is on GitHub.

There is effectively no user hierarchy in Unix-inspired systems, there is root and then there is everybody else. Unfortunately for my hacking this means I cannot have a user (1000) launch a process with a “lateral” user permission such as uid 1001. I would first need to escalate privileges in some manner from 1000 to root and then drop back down again to 1001.

If you ever see a program that “drops privileges” such as Docker, Nginx, or Systemd, just remember that in there is no dropping privileges without first escalating to root in a Unix-inspired system!

OpenSSH provides a number of escape characters that can be used on a running connection, one of these is ~C which will open up a little command line interface:

❯
ssh> ?
Commands:
      -L[bind_address:]port:host:hostport    Request local forward
      -R[bind_address:]port:host:hostport    Request remote forward
      -D[bind_address:]port                  Request dynamic forward
      -KL[bind_address:]port                 Cancel local forward
      -KR[bind_address:]port                 Cancel remote forward
      -KD[bind_address:]port                 Cancel dynamic forward

From this simple command line you can add or remove local, remote, and dynamic port forwards! Pretty snazzy! When I was first learning how to use these escape characters I had a difficult time getting the key-combination correct, frequently I would somehow screw it up and just send ~C to the shell:

❯ ~C
zsh: no such user or named directory: 

The approach that I have found most reliable is to hold my left shift key down while I hit ~ and C in sequence. In essence, my left pinky finger keeps shift depressed while my middle finger and then my pointer finger consecutively hit the two keys. I share this because when I first started to use these escape sequences I could never get the timing/cadence correct the first time and it felt like I was wasting more time than it would take to just disconnect and reconnect.

Another tip is to make sure you have detached from any active tmux sessions, as tmux has it’s own escape sequences and key bindings that you likely want to avoid triggering.

Once you get the hang of it, you can dynamically allocate new ports for whatever service you’re developing and then easily use your local browser/client to access the service under development. Much better!

For more escape characters, refer to the ssh(1) manpage via man ssh which has a section dedicated to these magic key combos.

ESCAPE CHARACTERS
     When a pseudo-terminal has been requested, ssh supports a number of functions through the use of an escape character.

     A single tilde character can be sent as ~~ or by following the tilde by a character other than those described below.  The escape
     character must always follow a newline to be interpreted as special.  The escape character can be changed in configuration files us-
     ing the EscapeChar configuration directive or on the command line by the -e option.

     The supported escapes (assuming the default ‘~’) are:

     ~.      Disconnect.

     ~^Z     Background ssh.

     ~#      List forwarded connections.

     ~&      Background ssh at logout when waiting for forwarded connection / X11 sessions to terminate.

     ~?      Display a list of escape characters.

     ~B      Send a BREAK to the remote system (only useful if the peer supports it).

     ~C      Open command line.  Currently this allows the addition of port forwardings using the -L, -R and -D options (see above).  It
             also allows the cancellation of existing port-forwardings with -KL[bind_address:]port for local, -KR[bind_address:]port for
             remote and -KD[bind_address:]port for dynamic port-forwardings.  !command allows the user to execute a local command if the
             PermitLocalCommand option is enabled in ssh_config(5).  Basic help is available, using the -h option.

     ~R      Request rekeying of the connection (only useful if the peer supports it).

     ~V      Decrease the verbosity (LogLevel) when errors are being written to stderr.

     ~v      Increase the verbosity (LogLevel) when errors are being written to stderr.

Today’s example was a common situation, one in which I frequently find myself: I want to use a tool which is available in the distribution’s packages but doesn’t quite have all features enabled. In today’s prototype, I was tinkering around with rsyslog and Apache Kafka via the omkafka module.

One problem: the default rsyslog package did not come with omkafka enabled, and omkafka was not packaged through another means. Since my test instance was running openSUSE, I found the rsyslog package, branched it, and made some minor changes to build a rsyslog-module-kafka rpm.

But wait! That’s not even the cool part!

Using the power of OBS, the project I branched (imagine the GitHub fork model) started building my changes immediately, and published my development packages within minutes! Suddenly I was able to have a fully formed rpm repository with my package in it. One which I could enable in my test environment and immediately start using!

Rather than fooling around with an entire custom build of rsyslog, I spent around an hour in between a couple of meetings tweaking the official package to add Kafka support, and then was back to the real work of prototyping!

OBS was well worth the learning curve!

I was originally attrached to openSUSE due to their truly innovative work in packaging Linux. They championed reiserfs, xfs, and later btrfs. From my understanding, they shipped the first production “delta RPMs” to users. Most of all openSUSE packaged numerous desktop environments well and emphasized user choice and flexibility, at a time when Ubuntu was chasing the One True User Experience, attempting to emulate the Mac OS X playbook.

When I left for Debian, openSUSE was at a bit of a cross-roads. The openSUSE Tumbleweed project had just started to get focus, but overall the entire distribution felt trapped between two extremes: old packages and high quality, or new packages and low quality. Over time, I also became rather frustrated that newer tools were never packaged for openSUSE, but almost always provided for Red Hat and Debian users.

I started to feel the itch again after successive upgrades of Debian on my primary laptop. That old problem of new packages with low quality or old packages with high quality was creeping into my day-to-day more and more. Reflecting upon much of the software I was using, I realized that much of it wasn’t actually packaged for Debian, but relied on the comedy trio of curl, pipe, and bash.

The catalyst for my return was an infrastructure issue in the Jenkins project, wherein I needed to re-familiarize myself with the openSUSE Build Service. An installation of Open Build Service (OBS), yet another openSUSE innovation, OBS allows for users and developers to easily create packages for a myriad of operating systems. I have long had an account in OBS, but had not really taken advantage of it. Looking at a smattering of unpackaged garbage strewn across the file system, and then considering this excellent tool for packaging applications for Linux, I had found a solution to (some) of my woes. One afternoon of playing around with packaging sway (packages here), I was sold: back to openSUSE.

OBS is what drew me back in, but I have noticed that openSUSE Tumbleweed is far more stable and mature than it once was, providing much newer packages with good quality. I am not sure what other tunables openSUSE ships with, but I have noticed a dramatic reduction in resident memory usage, running similar workloads to what I had previously under Debian. Finally, everything I was using previously is packaged now, which makes me feel much better about the state of the system. Signal, Sway, Docker, and everything else already provided by most systems is coming through easily managed and upgraded packages. The only things I’ve curl-bashed have been RVM and Rust, but I have no expectations that RVM or Rust nightly would fit into any sane packaging scheme.

I recommend trying out openSUSE, for me it strikes the balance between stability of a curated system and providing power users flexibility, with minimal sacrifices in either direction.

What can I say, a tidy system brings me joy.

I have since learned that this is a Gtk+ 3.x dialog, which is changed the default behavior for handling key strokes to use search rather than entering in of a path.

I abhor using the trackpad on my laptop, which makes me feel like an old “hunt-and-peck” typist, and instead rely on a number of keyboard-heavy tools to navigate quickly and efficiently around web pages, terminals, and editors.

While waiting for a build to complete earlier today, I was frustrated enough to scour the internet for a potential solution and happened upon this blog post which contained a suitable workaround!

Ctrl+L in the Open File dialog will display a text field where the surly keyboard-heavy user can simply type in a relative, or absolute, path followed by Enter and navigate immediately to that directory or file.

Much better!

Last year I participated and helped man the openSUSE booth with the great folks from the openSUSE community. This year there is a distinct possibility that I will help man a Jenkins booth!

If you’ve not been, I recommend showing up and taking advantage of the opportunity to meet other local open source users, developers, packagers and sysadmins, all collected together under one roof.

The conference is still in the “call-for-papers” state, after which the session schedule will be posted.

If you’re within a 1-2 hour plane ride of Los Angeles, then I expect to see you there!