Systemd Linger

Posted by edward 1 day ago

Counter65Comment89OpenOriginal

Comments

Comment by chlorion 10 hours ago

Having user processes killed upon log out seems pretty reasonable to me, and its a configurable option if you don't like it. I don't get the fuss here.

I think it's just popular to dislike systemd, some sort of echo chamber effect or something.

Comment by walrus01 1 day ago

Default Debian Trixie out of the box configuration without changing any systemd config file allows gnu screen to work as it always has. There would have been great uproar and screeching if basic 'screen' detach and resume functionality had broken anywhere between Debian v11, v12 and v13, and it has not.

I'm assuming this is same for tmux but hasn't tested it.

Is there some major distro out there that has the config flag set the opposite of this by default?

Comment by mike_hock 1 day ago

> Default Debian Trixie out of the box configuration without changing any systemd config file allows gnu screen to work as it always has.

As mentioned in the second sentence of TFA. Debian enables linger by default.

Comment by yjftsjthsd-h 1 day ago

> Is there some major distro out there that has the config flag set the opposite of this by default?

It bit me on pop os (the Ubuntu derivative)

Comment by egorfine 10 hours ago

While perhaps maybe sometimes useful on desktops, all this logind crap absolutely must be removed from servers.

Personally me I fail to see any scenario where this is genuinely useful.

Comment by unixhero 23 hours ago

This is hideously annoying functionality for anyone heavily using detached tmux and screen sessions. Massively annoyed and such an unfriendly kdea.

Comment by graemep 1 day ago

This is why i dislike systemd. Things not working the way I expect means I have to do more work to figure something out.

Comment by Hendrikto 1 day ago

In my experience, systemd is much better documented, way more consistent, and way more reliable than what came before.

Comment by lucideer 1 day ago

I guess it depends on your source of info but I've found what came before was pretty comprehensively & accessibly documented, at least as well as systemd, which - while well documented - suffers from sprawl & overwhelm of the docs. There's just SO MUCH to grok in comparison.

Comment by BirAdam 1 day ago

I never found systemd difficult, but I do find prior solutions to be simpler. An init script is easy, and the order is also easy. If people are accustomed to UNIX systems, the sysvinit is better. For people who never experienced those systems, systemd is their friend. No need to use UNIX tools when Systemd ships with batteries included.

Comment by dietr1ch 1 day ago

Yeah, init scripts are simple, but things get tricky outside of the happy path. Think of service upgrades/restarts. Once you care about dependencies, then you are back to a graph like systemd's.

Comment by BirAdam 1 day ago

If ever handled thousands of servers running Linux with sysvinit, you get quite accustomed to it… and the tricky bits aren’t that tricky.

Comment by iririririr 1 day ago

talking "outside happy path" in defense of systemd is wild. because that's it's Achilles heel.

systemd is awfully documented (volume is not quality), things are still changing and haven't even caught up in functionality with any init hack.

the only saving grace of systemd is that it almost makes it easier to use namespace and other sandbox (but only because those are newer and init systems also didn't have their solutions up yet)

systemd only purpose is to mimic windows administration in linux. sorry of anyone who fails to see this

Comment by jamespo 1 day ago

How are hacky init scripts written in shellscript well documented? Can you point me to this documentation?

When are the init systems going to implement the basic security measures systemd has?

Comment by graemep 6 hours ago

Which security measures are init systems missing? Are you saying all other init systems are badly documented?

Comment by jamespo 1 hour ago

take your pick: https://github.com/alegrey91/systemd-service-hardening - and this was 6 years ago and barely scratches the surface

Comment by otabdeveloper4 11 hours ago

> An init script is easy

Oh boy. Lmao.

Comment by 7bit 1 day ago

No way. I was easily able to create Systemd units from 0-knowledge, where the init system remains a big black box riddled with question marks for a very long time. Very little was documented.

Comment by m0llusk 1 day ago

Absolutely not the case. Even relatively simple real world uses involved collections of shell scripts to carefully mount or initialize services in just the right order and would often fail with weird and infrequent timing glitches. With systemd that all becomes explicit, so of course it is awkwardly verbose. That is the whole point and a huge upgrade.

Comment by shevy-java 1 day ago

I never used shell scripts for that. Ruby worked much better and easier. No glitches either.

I also fail to see how verbosity is a "huge upgrade".

Comment by preg_match 22 hours ago

The problem is Ruby is not de-facto installed on most distros. Shell is the "universal" thing, but shell is actually not portable, even across distros. Perl is the only thing I'd qualify as both "sufficiently portal" and "universal". But then you have to write Perl, which nobody wants to do seemingly.

Comment by jamespo 1 day ago

So your systems have a dependency on ruby to startup and anyone who maintains it has to know ruby?

Comment by jauntywundrkind 1 day ago

We can argue about which has better docs, but, to me, systems offers a uniformity of practices and a consistency of service offerings that was never the case before.

Ever company used to have to make its own decisions about what logging solution to use. Each one had its own configuration paradigms! Every single daemon had some kind of bespoke /etc/init.d/ script that had its own rules and practices, having either forked off some other skeleton or having been made on it's own at some point.

And then companies wanted to do the right thing and apply some capabilities restrictions to processes. They wanted a little more than out of box security. That results in every company forming and modifying init scripts, changing the configuration paradigms, encoding config directly in or inventing their own new parameterization.

The fact is I don't have to look at docs any more, because everything has a (pleasant) uniformity. The fact is we are worlds better at securing and locking down our processes, because systemd has incredible built in options for capability, sandboxing, and other fantastic security processes.

> There's just SO MUCH to grok in comparison.

There always was before too. Most people skated along the surface, but when some daemon tried to actually have better practices, do some security tightening, operators were in for that exciting rabbit hole of trying to catch up and learn. Or just hand wave it away.

I think this statement really iconifies the systemd resistance really. Yes there is so much (imo: enjoy it, isn't that so great?) It's many many many latent possibilities (for you to over time learn): it's so often what the kernel is offering you, it's well exposed, and you have a pleasant uniform system for harnessing that, that you only have to learn once and which can pay dividends again and again and again once you learn it. Learn once, use many.

Or just ask your LLM. It knows. It will just tell you. It won't have to read docs or source. It knows systemd very well. Because it is something shared, something known. Imo it ought be something treasured.

I'm fine if some day we get rid of systemd. Nothing sacred here. But to attack it, it must be attacked from above. It must be replaced by something with more principled and more capabilities. Same as Kube. Neither is going away, neither will ever be displaced by smaller offerings. The only way out is through, the only way out is up. Godspeed, good luck, have fun.

Comment by M95D 23 hours ago

> Ever company used to have to make its own decisions about what logging solution to use. Each one had its own configuration paradigms! Every single daemon had some kind of bespoke /etc/init.d/ script that had its own rules and practices, having either forked off some other skeleton or having been made on it's own at some point.

> And then companies wanted to do the right thing and apply some capabilities restrictions to processes. They wanted a little more than out of box security. That results in every company forming and modifying init scripts, changing the configuration paradigms, encoding config directly in or inventing their own new parameterization.

IMHO, any competent sysadmin, when is installing the system, would edit all those templates into the exact shape the system and the user needs. Those file are just that - templates. The fact that overworked company sysadmins started using templates as final configuration and needed something automated instead, doesn't mean that this systemd that is fit for them is also fit for everybody else.

Comment by jauntywundrkind 11 hours ago

> IMHO, any competent sysadmin, when is installing the system, would edit all those templates into the exact shape the system and the user needs

That, IMHO, is the other 90% of the problem (to riff on the ninety-ninety rule).

Every competent sysadmin has their own particular style & preferences & opinions about what the exact shape should be. And will go rewrite the world, according to some particular set of tastes, that they usually won't write down or document or socialize.

So not only do you have to figure out all the special sauce in how the service you just walked in to is managed, you have to wade through a complete re-architecting that some sysop made on their own judgement 5 years ago that people have patched and hacked since.

You can keep adapting to these scenarios. But to me, the normativity, the incredibly great toolkit of options it comes with: yes it's sizable, but it's one thing that I have to learn, that is usually a direct representation of something the kernel gives me, that will then be familiar and portable and useful wherever I go. This feels so much more civilized than what came before.

Comment by M95D 9 hours ago

I'm no sysadmin, except for my own systems, but I think that the vast majority of systems are fixed-config for their entire lifetime. Deployed once on VM and stays the same until it's no longer needed. So every sysadmin can just set them up in whatever way he wants with no obligation to make it understandable to the next guy. The VM will be deleted anyway when no longer serves it's purpose.

That being said, I do understand the need for some standardization. But you missed this part:

> doesn't mean that this systemd that is fit for them is also fit for everybody else.

They practically forced everybody to use it, and not just the service manager, everything else too: init, logger, network, login, dbus, etc.

Comment by cwillu 18 hours ago

Days since my last exposure to “I'm sorry, you're absolutely right, that's wrong”: 1

Comment by vrighter 1 day ago

the oss community has a phobia of having anything in common against all its reimplementations of the same thing. So systemd offers some consistency between distros: bad.

X11 was pretty much standardized, that won't do.... let's invent wayland but make sure that there are multiple incompatible implementations of it!

and so on. I love linux, but this sort of mentality is ridiculous

Comment by inigyou 1 day ago

the systemd init system is complicated but fine. the problem with systemd is how much it wants your computer to itself. e.g. it can only run as PID 1, you aren't allowed to use control groups if you use systemd, it overwrites resolv.conf with its own address, it renames all your network adapters to lennart's preference (he calls them stable names but anyone who actually uses a computer will tell you they change more often now), and it binds itself to syslog

Comment by graemep 6 hours ago

The problem with systemd is not the init system. Most people who do not like systemd are happy with the init system, but do not want the rest of it.

Comment by po1nt 1 day ago

Maybe, but the scale of systemd features is humongous. It better be well documented. You don't need a user manual for hammer, you do need one for hydraulic press.

Comment by nubinetwork 1 day ago

> hydraulic press

What is there to know? It goes up and down, and don't stick your arm inside...

Comment by po1nt 1 day ago

Until it breaks, reports error, leaks, makes weird sound, needs a maintenance or even if you want to move it. You need to know how heavy it is, where are mounting points, where is the center of gravity, what needs to be affixed etc. What is the power it draws for the pump, what is the peak power and how long is the peak, what is the power factor? Does it have a safety certification, what kind of training operator needs, are safety elements up to date to modern standard? What kind of fire extinguisher do you need? What is the toxicity of all the materials under normal and extreme conditions? What kind of peak force is applied to floor and how is it distributed?

Comment by M95D 23 hours ago

You'd be surprised to know.

https://www.manualslib.com/manual/3389176/Cincinnati-90-Form...

Comment by 7bit 1 day ago

The scale is humongous? Syszemd is not one big thing. It's a collection of small things, each meticoulsouy documented and you can pick and choose.

If you still think there is only one Systemd, maybe learn about the tool instead of just talking others after their mouth.

Comment by po1nt 1 day ago

Just the PID 1 manager is multiple times larger in LoC than any alternative. There is no metric in which systemd is more lightweight that I could find.

Comment by preg_match 22 hours ago

I'm not sure this is true in practice, if we're counting init scripts themselves. Which we should, because init scripts are code and they often re-implement standard systemd features. Usually in wonky, broken ways.

Comment by graemep 3 hours ago

What about other init systems? The GP said "any alternative." I am not familiar with how complex they are but was under the impression that there are a number much simpler than sysv init?

Comment by po1nt 13 hours ago

I think that even if we count init scripts which are way different level of complexity than C LoC, the Systemd would still be larger. Especially if we count just the scripts that are deployed.

Comment by iririririr 1 day ago

that's wrong in the sense it won't be useful for anything unless you have a dozen of those small things setup in the very specific way the author envisioned you set them up.

you seem to be repeating all the marketing, while mentioning in other comments you never understood any init system. i don't think you're the authority to be adding so many comments here.

Comment by rnhmjoj 1 day ago

Yes, systemd not being a monolith is a myth.

You can forward logs to rsyslog or something, but you really can't disable systemd-journald. Technically you don't need to run systemd-boot, you can use grub or something else, but then bootctl and systemd kexec stop working. You don't need to run dbus, but if you don't several systemd features break, and it's not even documented what needs it. The list goes on and on.

Also, the fact that all components are in the same repository and need to be built together causes a lot of headaches with packaging due to dependency cycles (like, systemd -> cryptsetup -> lvm2 -> udev -> systemd).

Comment by preg_match 22 hours ago

Systemd is literally not a monolith because it's not one big piece of software. It's a collection of software. For example, systemd-boot has basically nothing to do with systemd-init.

Yes, some software is more tightly coupled than others, naturally. systemd-init and systemd-journald work tightly together because logging is a core requirement of any init system.

Comment by rnhmjoj 9 hours ago

It's literally not a monolith, but in practice it may well be, that was my point.

Comment by M95D 1 day ago

I'm still using sysvinit without anything on top (the default would have been sysvinit + openrc, but I dropped openrc, too complicated for my needs). I find it a lot easier than learning systemd, or learning anything else. To find out what's going on, I just need to open one file: /etc/inittab, and it's all in there. Nothing hidden somewhere else. Nothing undocumented. Simple commands, one after another. There are no dependencies, no conditions, not unless I put them there myself.

Comment by egorfine 10 hours ago

This has never been my experience for the whole duration of the systemd existence.

Comment by shevy-java 1 day ago

So, I disagree about your statement here, but let's for a moment assume it were correct - so let's not evaluate THAT particular statement, thus.

You most likely refer to sysinit or something like that, right? Because to what else do you compare it to? Shell scripts?

Shell scripts in general are crap. I don't understand why linux systems use them. When I transitioned to linux a long time ago, I wrote - and still write - pretty much all logic in ruby and .yml files. Literally my whole system is described via .yml files, ruby then just auto-expands all of this into what is meaningful. I also compile from source and manage the system via ruby as-is, so I don't even need systemd, nor shell scripts. But this is just one comment here and if you refer to shell scripts then I agree with you - shell scripts have always been horrible. I fail to see why we should like systemd merely because shell scripts are so awful. That makes no sense. I neither use systemd nor shell scripts (ok ok I bail ... I am currently using manjaro which uses systemd; I disabled most of the services, but the primary reason I was assimilated into systemd is mostly because the non-systemd linux distributions, kind of gave up for the most part or come with their own set of problems - slackware, void and so forth. And yeah, I was using and testing them for ages too, slackware I used for many, many years. It is still a great distribution but it is not really active anymore. No new .iso releases in years, sorry, that is dead. Even if Patrick still updates packages.)

The other part is ... IF you referred to an alternative init system, then THAT comparison is flawed too, because sysvinit is mostly just an init system. Systemd is some giant thing that does 100000 things. So the comparison is, and ALWAYS has been, unfair. Not the same things are compared here.

Comment by eptcyka 1 day ago

Systemd is not 1 giant thing that does 100000 things - it is a collection or 10000 services doing 10000 things, integrating amongst themselves where practical. You need not use networkd, logind, resolved or any other one piece of the system you dislike.

Comment by yjftsjthsd-h 1 day ago

My least favorite part is journald. When you find a way to rip that out, let me know.

Also logind is so hard to avoid that even other systems have ported it: https://wiki.alpinelinux.org/wiki/Elogind

Comment by yjftsjthsd-h 1 day ago

> You most likely refer to sysinit or something like that, right? Because to what else do you compare it to? Shell scripts?

dinit is a good service manager that doesn't feel obligated to reinvent the kitchen sink

Comment by dosisking 1 day ago

Well, in my experience, systemd has much worse documentation, is very inconsistent (or you could say it is constistantly brain-dead), and extremely unreliable and unpredictable, since it often hangs when shutting down and rebooting.

Comment by preg_match 22 hours ago

In my experience hangs usually happen from no time limits on units or excessively long time limits. I fried a NIC a while back and network-manager-wait-online has no time limit. Usually this isn't a problem, but the NIC was fried in a very unique way where it would continuously try to connect and then downgrade connections.

The bright side is this is all logged, trivially grep-able, and easy to fix.

Comment by bombcar 1 day ago

systemd is like the registry. If you drink the koolaid and fully admit and adopt the decisions made, you’ll do ok, perhaps even good.

If you have 20+ years of experience, you’ll will NEVER get it to work the way you expect, might as well install Gentoo.

Comment by NekkoDroid 1 day ago

> Things not working the way I expect means I have to do more work to figure something out.

I dunno, "things not working the way I expect" is very subjective.

I personally think it makes more sense that processes do not outlive the parent from which they come (in this case processes spawned from a user shouldn't outlive that user) unless you explicitly go through something which outlive the parent (process manager in this case (e.g. running via systemd-run)).

Though I am also strongly of the opinion of just because something is a feature, does not mean it is good to keep around, e.g SUID/SGID (which has a similar "I get more than the parent" problem).

Though I guess different people can have different opinions/values on this.

Comment by mike_hock 1 day ago

It's one of those systemd features that are annoying and in the way when you're trying to accomplish a specific task but deal well with shitware. To be fair to systemd, these features can't usually be implemented without writing new rules for what you have to do to accomplish the legitimate tasks.

Comment by Someone 1 day ago

> This is why i dislike systemd. Things not working the way I expect means I have to do more work to figure something out.

Even if the change is a genuine improvement?

Comment by egorfine 10 hours ago

Yes. Improvement or not, a thing not working the way one expects means one has to do more work to figure something out. And in case of pid1, this is never a time spent productively.

I'm all for improvements and I love updates of software, but what systemd usually does is ideology and destruction in the name of "new". Like, we have had cron perfectly working for _decades_ until one man told us we were all wrong on that.

Comment by dosisking 1 day ago

How can the change be a genuine improvement, if it does something unexpected that needs to be figured out?

Comment by graemep 1 day ago

This is not a genuine improvement:

> One of the features of systemd that is most controversial is the option to kill user processes when the user logs out. That initially killed screen/tmux/nohup processes too.

Killing processes that the user clearly did not want killed is a regression.

Reading the rest of article it seems that its fixed but made things a lot more complicated than "unless you deliberately start things so that they keep running when you logout they will be killed when you logout". Am I wrong?

Comment by preg_match 22 hours ago

I don't think it's a regression at all. It's been standard advice for decades that logout-login wipes user processes for updates, therefore you don't need to restart.

I'd argue the opposite is true: linger is surprising, and I turn it off.

Comment by graemep 6 hours ago

I I start a process with nohup and logout I clearly want it to keep running. That is the entire point of nohup.

If that process is killed when I logout then that is a regression.

I do not understand what this has to do with udaptes.

Comment by qweqwe14 1 day ago

"The way you expect" is probably inconsistent and, upon closer inspection, completely broken.

There is a reason why systemd became the default and other niche init systems have faded into obscurity.

Sure, it may be more complex than your pile of shell scripts, but that's because it does the same things better, and has a lot more functionality that you will need at some point, and good luck replicating that by hacking shell scripts.

Comment by yjftsjthsd-h 1 day ago

> "The way you expect" is probably inconsistent and, upon closer inspection, completely broken.

When I start a program, I want it to stay running. This is perfectly consistent and not broken.

Comment by graemep 1 day ago

You are framing this as the only choice being between systemd and shell scripts.

Comment by dosisking 1 day ago

> There is a reason why systemd became the default and other niche init systems have faded into obscurity.

Yes, it is part of the Embrace, Extend, Extinguish strategy

Comment by qweqwe14 1 day ago

No, it is part of the "I just want stuff to work" strategy

Comment by halJordan 1 day ago

I, for one, appreciate the honesty. I wish more systemd haters would admit this is why they dislike it. We could finally throw away the tired arguments that have been disproven in the past ten years

Comment by TacticalCoder 4 hours ago

> We could finally throw away the tired arguments that have been disproven in the past ten years ...

Here's one you cannot disprove: the Jian Tan XZ backdoor did not affect Linux distro that weren't using systemd.

I know, I know, it's got absolutely nothing to do with systemd.

We're talking about something insane like upstream OpenSSH being modified to add systemd logs "log compression" to freaking OpenSSH (yay security, makes perfect sense!).

So disprove this: not a single Linux systemd-less distro was affected by an attempt to place a worldwide backdoor on Linux systems exposing a SSH port.

P.S: I moved all my infra to VMs that are systemd-free and containers, by default, do not run systemd as PID 1 (they don't run it at all). It works flawlessly. I'll now move my hypervisors to FreeBSD+bhyve and I'll be, at long last, totally systemd-free again (I was using Linux before "systemd-I-compress-logs-so-I-make-you-modify-OpenSSH-to-add-liblzma-dependency" came on the scene with it's its Windows-style .INI files and these were much better days).

Comment by nurettin 1 day ago

> Things not working the way I expect means I have to do more work to figure something out.

You must love SELinux

Comment by ChocolateGod 1 day ago

The fault lies in the distribution for it's default configuration, not systemd.

The article even mentions that distros ship with it turned off

Comment by yjftsjthsd-h 1 day ago

The fault lies in the upstream defaulting the option on, so that distros have to

Comment by dietr1ch 1 day ago

> Things not working the way I expect means I have to do more work to figure something out.

Sound like you don't like computers to begin with

Comment by dosisking 1 day ago

He probably just doesn't like having to use poorly written software

Comment by BigTTYGothGF 1 day ago

That's pretty much all software.

Comment by shevy-java 1 day ago

Systemd is simply too complex.

Comment by surajrmal 20 hours ago

The Linux kernel is simply too complex.

Comment by dietr1ch 1 day ago

But how much of it is inherent complexity?

Comment by tremon 1 day ago

Only the pid 1 sigchild handling. Everything else is design choices.

Comment by dosisking 1 day ago

About 0.1% is inherent complexity

Comment by saturn_vk 1 day ago

Which part? Just saying systemd is like saying a distribution is too complex

Comment by agumonkey 1 day ago

I may be wrong but he was probably hinting at systemd being a constellation of too many parts.

Comment by realusername 1 day ago

As a user I can say that journald was a mistake, if you try to reinvent a database, you will reinvent it poorly and journald wasn't an exception.

Comment by benj111 1 day ago

I'm presumably missing something here. Why does tmux et al not work? Remotely? Ie you're logged in locally and remotely to the same machine, log out locally which kills everything running remotely, but then why would you log in twice like that?

And that being the case can systemd not just only close everything when your logins reach 0?

Comment by tux3 1 day ago

You can detach from screen/tmux, leave it running in the background, and log back in later to the same screen/tmux like you left it.

Except systemd can kill it when you log out, so you come back to nothing.

Comment by c-hendricks 1 day ago

What I don't get is why (on Arch) I have linger off yet have never had trouble with tmux/ zellij being killed.

Edit: ah, this post clears it up. Mentions linger has nothing to do with screen muxers, and also brings KillUserProcesses to peoples attention.

Damn systemd and its multiple levels of convenience.

Comment by benj111 1 day ago

Ah ok. Seems like the log out metaphor is kind of broken then?

It seems to me if you want persistence between logouts, the processes should belong to a different group/user.

(I'm spitballing hypotheticalshere, not saying anyone/thing in particular is wrong)

Comment by dietr1ch 1 day ago

Think about user services in a multi-user setup

Do you want user-level services, or a multi-tenant system-level service? - Who can tweak the service configuration? - How do you ensure there's no data leaks between users? - What about misbehaviour? Quotas, crashes

I think lingering is great here

Comment by benj111 1 day ago

Off the top of my head. Have a group that doesn't die on logout. So you can have an email Daemon that responds with out of office. And you tmux session can keep running?

I'm not necessarily talking about the Unix concept of a group. It seems to me you want some granularity between 'kill all processes' and 'keep all processes'

Comment by egorfine 10 hours ago

LP knows better what should work for you and what not.

Comment by kmbfjr 1 day ago

I never had a problem with systemd init despite its quirky habits like just giving up if a mount is bad in fstab, and I rather like systemd-networkd over the other four or five ways of managing the network. Systemd-timesyncd isn’t bad either.

But this, systemd-logind worrying about stuck processes in userspace is exactly the kind of scope creep that ends up justifying a lot of the hand wringing a decade ago. This thing bit me when it came out in a couple different ways and I am unapologetic in my dislike for it. So much so that it has caused me to drift away from Linux where I can. Even though it is a simple fix, I’ve become fatigued with userspace chaos.

Time to re-evaluate what the bazaar has become. Systemd is a common complaint, but it isn’t the only contributor to the fatigue.

Comment by iluvcommunism 1 day ago

[dead]