Using Python for Scripting
Posted by birdculture 7 days ago
Comments
Comment by rjmill 2 days ago
Quick rundown for the unfamiliar:
Give it a command as a list of strings (e.g., subprocess.run(["echo", "foo"]).)
It takes a bunch of flags, but the most useful (but not immediately obvious) ones are:
check=True: Raise an error if the command fails
capture_output=True: Captures stdout/stderr on the CompletedProcess
text=True: Automatically convert the stdout/stderr bytes to strings
By default, subprocess.run will print the stdout/stderr to the script's output (like bash, basically), so I only bother with capture_output if I need information in the output for a later step.Comment by bccdee 2 days ago
Comment by c6401 9 hours ago
Comment by theomega 1 day ago
Basically you can just `from sh import [command]` and then have an installed binary command available as function
from sh import ifconfig
print(ifconfig("eth0"))Comment by theamk 1 day ago
By default (1) captures stdout and stderr of all processes and (2) create tty for processs stdout.
Those are really bad defaults. The tty on stdout means many programs run in "interactive" rather then "batch" mode: programs which use pager get output truncated, auto-colors may get enabled and emit ESC controls into output streams (or not, depending on user's distro... fun!). And captured stderr means warnings and progress messages just disappear.
For example, this hangs forever without any output, at least if executed from interactive terminal:
from sh import man
print(man("tty"))
Compare to "subprocess" which does the right thing and returns manpage as a string: import subprocess
subprocess.check_output(["man", "tty"], text=True)
Can you fix "sh"? sure, you need to bake in option to disable tty. But you've got to do it in _every_ script, or you'll see failure sooner or later. So it's much easier, not to mention safer, to simply use "subprocess". And as a bonus, one less dependency!(Fun fact: back when "sh" first appeared, everyone was using "git log" as an example of why tty was bad (it was silently truncating data). They fixed it.. by disabling tty only for "git" command. So my example uses "man" :) )
Comment by petters 1 day ago
Wow... yes sounds like a library to avoid!
Comment by pxc 1 day ago
https://plumbum.readthedocs.io/en/latest/local_commands.html...
It also does argument parsing and validation, so it's generally pretty useful for writing little CLI tools that invoke other CLI tools.
Comment by Lvl999Noob 1 day ago
https://pypi.org/project/pypyp/
It takes cares of the input and output boilerplate so you can focus on the actual code that you wanted python for.
> seq 1 5 | pyp 'sum(map(int, lines))'
> ls | pyp 'Path(x).suffix'Comment by two_handfuls 19 hours ago
Comment by grumps 1 day ago
Comment by btown 1 day ago
https://github.com/amoffat/sh/blob/2a90b1f87a877e5e09da32fd4...
Comment by code_biologist 1 day ago
https://docs.astral.sh/uv/guides/scripts/#using-different-py...
Comment by grumps 1 day ago
Comment by zelphirkalt 1 day ago
Comment by albertzeyer 1 day ago
Comment by sevensor 2 days ago
Comment by zelphirkalt 1 day ago
Comment by capyba 23 hours ago
Pair it with numpy and matplotlib (two external dependencies that personally I consider part of Python itself), and you’ve got 80% of an interactive scientific simulation environment.
Comment by zelphirkalt 23 hours ago
Comment by bigstrat2003 1 day ago
Comment by BoppreH 1 day ago
The sqlite, tkinter, and shelve modules are the ones I find most impressive.
Comment by mubou2 7 days ago
C# scripts let you reference packages in a comment at the top of the file, for example:
https://devblogs.microsoft.com/dotnet/announcing-dotnet-run-...
Comment by chillaranand 6 days ago
https://avilpage.com/2025/04/learn-python-uv-in-100-seconds....
Comment by simonw 2 days ago
# /// script
# dependencies = [
# "cowsay",
# ]
# ///
import cowsay
cowsay.cow("Hello World")
Then: uv run cowscript.py
It manages a disposable hidden virtual environment automatically, via a very fast symlink-based caching mechanism.You can also add a shebang line so you can execute it directly:
#!/usr/bin/env -S uv run --script
#
# /// script
# dependencies = ["cowsay"]
# ///
import cowsay
cowsay.cow("Hello World")
Then: chmod 755 cowscript
./cowscriptComment by vovavili 1 day ago
Comment by emidln 2 days ago
Comment by networked 1 day ago
#! /bin/sh
"exec" "/usr/bin/env" "uv" "run" "--quiet" "--script" "$0" "$@"
# /// script
# dependencies = [
# "cowsay",
# ]
# ///
import cowsay
cowsay.cow("Hello, world!")
On systems that can't run uv, like NetBSD and OpenBSD, switch to pipx: #! /bin/sh
"exec" "/usr/bin/env" "pipx" "run" "$0" "$@"
# ...Comment by collinfunk 2 days ago
Comment by presbyterian 2 days ago
Comment by vovavili 1 day ago
Comment by ewidar 1 day ago
If you are installing packages, then starting with installing uv should be fine.
Comment by milliams 2 days ago
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "requests<3",
# "rich",
# ]
# ///
import requests
from rich.pretty import pprint
resp = requests.get("https://peps.python.org/api/peps.json")
data = resp.json()
pprint([(k, v["title"]) for k, v in data.items()][:10])Comment by eternityforest 7 days ago
Generally the only nontrivial scripting I ever do is associated with a larger project, so I often already have a pyproject.toml and a UV environment, and I just add the dependencies to the dev group.
Comment by mubou2 6 days ago
It just feels strange that C# of all languages is now a better scripting tool than Python, at least out of the box. I did notice uv has exactly the feature I'm looking for, though it's obviously third-party:
https://docs.astral.sh/uv/guides/scripts/#declaring-script-d...
Is everyone just using uv now instead of pip, perhaps? Or is just another alongside pipenv, conda, poetry, etc.? (Python's not my main these days, so I'm out of the loop.)
Comment by jollyllama 2 days ago
Comment by dizhn 2 days ago
That said, a lot of very complicated things are actually written in bash. Distrobox I think is for example.
Comment by DonHopkins 2 days ago
They're only complicated BECAUSE they're written in bash. If they were written in Python they would be much less complicated, more modular, able to use many existing industrial strength well tested python modules instead of rolling their own ad-hoc ones, and much easier to maintain.
Comment by eternityforest 6 days ago
I suspect conda still has some market share too but I've never needed it.
Comment by psunavy03 2 days ago
Comment by Rucadi 2 days ago
Comment by HumanOstrich 1 day ago
Might as well rewrite all your scripts in Rust too while you're layering on unholy amounts of complexity.
Comment by Diti 1 day ago
I’m so thankful to see a flake.nix file in every single cool project on code forges.
Comment by HumanOstrich 1 day ago
Seeing that flake.nix badge of complexity lets me know a project will be a nightmare to set up and will break every other week. It's usually right next to the Cargo.toml badge with 400 dependencies underneath.
Comment by Diti 17 hours ago
Sure, installing packages the proper way requires a little bit more setup (Home Manager, most likely, and understanding where is the list of packages and which command to build to switch configuration), but as trivial as other complex tasks most of us hackers are capable of doing (like using `jq` or Vim).
Comment by Rucadi 1 day ago
The easiest entry-point is to just use it like a package manager, you install nix (which is just a command...) and then you have available the whole set of packages which are searchable from here: https://search.nixos.org/packages
nix-shell is just to download&add programs temporary to your PATH.
I don't feel that this is harder than something like "sudo apt install -y xxxxx" but for sure more robust and portable, and doesn't require sudo.
If at some point you want to learn the language in order to create configurations or packaging software, it may require to check a lot more documentation and examples, but for this I think it's pretty straightforward and is not harder than any other package manager like aptitude, homebrew or pacman.
Comment by kokada 1 day ago
Comment by albertoCaroM 2 days ago
Comment by luz666 2 days ago
Then you can do e.g. use other persons python scripts without modifying their shebang.
Comment by DonHopkins 2 days ago
Not only does bash not have a module system like python, or a vast ecosystem of modules like python, but also that it's much too weak and brittle a language to implement most of those modules that Python has, and can't even call native code libraries directly.
Even with just its standard built in "batteries included" libraries and no extension modules or native code modules, Python is still much more powerful than bash and easier to code and maintain.
If you're complaining about having to install Python modules, you're usually already doing something that's impossible or incredibly difficult to do in bash anyway.
Even something as simple and essential as fetching files via http or parsing json. Bash has to call out to other programs to do that, but you have to install those programs too, and while Python can certainly call out to curl or wget or jq, it doesn't have to, since it has all that and more built in.
There really is no comparison, because bash loses along so many dimensions at once compared to Python.
Comment by lgas 1 day ago
#!/usr/bin/env -S uv run --with sh --script
from sh import ifconfig
print(ifconfig("en0"))
which is a pretty nice experience assuming you already have `uv` in the target environment.Comment by qznc 1 day ago
Python packages are fine for servers but not for CLI tools.
Comment by froh 1 day ago
once the script is non-trivial, 'install' it using pipx, in editable mode when you work on the script and as normal pipx installed cli utility otherwise.
the venv part is then completely under the hood.
Comment by kjkjadksj 2 days ago
Comment by a-dub 2 days ago
Comment by bigstrat2003 1 day ago
Comment by paulddraper 2 days ago
The same way you handle them with bash?
Install them?
What are we talking about here?
Comment by fridder 2 days ago
Comment by bigstrat2003 1 day ago
Comment by SahAssar 12 hours ago
There's 10 different package managers and none of them seem to have any common conventions.
To even run a script you need to setup a venv and enter it by sourcing scripts.
Try to move a venv, it breaks.
Try to understand how/what/where it installs things its a black box if you are not "in" the ecosystem.
Node seems easy in comparison. I hate how python makes it so hard to use.
Comment by bdangubic 1 day ago
Comment by paulddraper 2 days ago
Use that.
Comment by bccdee 2 days ago
Comment by skydhash 1 day ago
Comment by bccdee 1 day ago
No—system python is for the system's scripts, and user projects should have their dependencies sandboxed in a virtual environment. That's the only model that really works.
Comment by archargelod 2 days ago
But if a script I write needs to use arrays, sets, hashtable or processes many files - I use Nim[0]. It's a compiled systems-programming language that feels like a scripting language:
- Nim is easy to write and reads almost like a pseudocode.
- Nim is very portable language, runs almost anywhere C can run (both compiler and programs).
- `nim r script.nim` to compile and run (cached on subsequent runs) or use a shebang `#!/bin/env -S nim r`
- Nim programs are fast to compile (use debug mode and tcc compiler for almost instant compile times)
- Nim scripts run very fast <10ms (something that was very annoying to me with bash and Python)
- good chances you don't need external dependencies, because stdlib is batteries included and full of goodies.
- if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] for easy Nim cross-compilation).
[0] - https://nim-lang.org
Comment by slaymaker1907 1 day ago
I actually sped up a script the other day that had been written in bash by 200x by moving it over to Python and rewriting the regexes so they could run on whole files all at once instead of line by line. Performance problems are most often from poorly written code in my experience, not a slow language.
Comment by pxc 1 day ago
> But if a script I write needs to use arrays, sets, hashtable or processes many files
One option that I sometimes use at work (in addition to writing some Python CLIs) that is a pretty nice next step on this spectrum is Bash-with-all-the-fixins'.
I use a Nix-based dependency resolver for shell scripts called resholve¹ to parse scripts so that it can identify all of their dependencies (including Bash itself), then produce a "compiled" script as a Nix build where all of the references to external programs are replaced with pinned Nix store paths.
Then I have a fixed (and recent) version of GNU Bash regardless of platform, so I'm free to use Bash features that are newer or nicer than POSIX sh. My favorite such features are `mapfile` and `lastpipe` for writing in a more functional style, plus of course maps ("associative arrays").
I don't have to worry about portability problems with common utilities, because my scripts will bring along the expected implementations of coreutils, `find`, `grep`, etc. I'm free to use "modern" alternatives to classic utilities (like `rg` instead of GNU grep or `fd` instead of GNU findutils) if they offer better performance or more readable syntax. Polished interactivity is easy to build by just embedding a copy of `fzf`. And while I don't love Bash for working with structured data, it becomes a lot less painful when I can just pull in `jq`.
It's obviously got some disadvantages versus an option like Python, but the amount-of-code to functionality ratio is also much more favorable than Python's.
I typically use this for scripting build and development tasks in projects that already use Nix (via Devenv— I have some unpublished changes to the scripts module that add resholve and ShellCheck integration) to manage their environments, so there's no special packaging/distribution/setup burden.
IME it makes for vastly more readable and maintainable shell scripts than painstakingly limiting oneself to pure POSIX sh, so it can serve quite well as a middle ground option between barebones sh and a "real" programming language as your little script gradually grows in complexity.
> if you need external deps - just statically link them and distribute a cross-compiled binary (use zigcc[1] or easy Nim cross-compilation).
The deployment story sounds very slick and hard to beat! This is something I might want to try for scripts I want to be easy to distribute on macOS systems that don't have Nix installed.
--
Comment by gabrielsroka 2 days ago
> Python 3 is installed on basically every machine out there.
> Python will work the same on all the machines you run your script on
No, no, and no.
Comment by nilslindemann 2 days ago
Comment by qznc 1 day ago
Comment by ggm 1 day ago
/*
Oh, you're looking at my CSS. Here be dragons.
Let's bundle in a rant while you're here: I'd like to use variable width fonts
instead of fixed size fonts. I enjoy 300 (i.e. light) and 600 (i.e. semibold)
over regular and bold for Crimson Pro only. But Chrome doesn't like to show
anything but 400 and 700 with variable width fonts, so I'll have to trick Chrome
by declaring my 300 and 600 fonts as if they were 400 and 700 respectively.
... or well, I could use font-variation-settings, but that's a global override
which means I have to specify it EVERYWHERE and that's just too much effort for
poor me.
*/Comment by idoubtit 1 day ago
That's the bright side of Python. They should mention the dark side, or Why _not_ to use Python for scripting.
First of all, the promise of easy portability breaks as soon as the script has dependencies. Try to install some Python program on a server where you're not root and a minimal python3 is installed.
The stability isn't very good in my experience either. I've often seen programs not compatible with recent releases of Python, either explicitly in the README or implicitly at runtime. Unmaintained Python code breaks.
Unfortunately, there is no silver bullet. Posix shell or bash may be better for simple scripts; Perl or Python if you know you won't require dependencies or if you have a good control on where to install the script; languages that compile to static executables are not really "scripting", but may be a better choice for long(term usage. These past years, I tend to keep away from Python as much as I can.
Comment by wodenokoto 21 hours ago
And bash has a good dependency story? At least with python you can bundle your script with a requirements.txt file and it is doable for the target machine to get up and running.
Comment by skydhash 1 day ago
Comment by kazinator 2 days ago
cp version.template build/version.txt
sed -i "s/@VERSION@/${COMMIT_TAG:-dev}/" build/version.txt
sed -i "s/@BUILD_DATE@/${BUILD_DATE}/" build/version.txt
Eww! Mutating a file in place, and needing a GNU extension for it. sed \
-e "s/@VERSION@/${COMMIT_TAG:-dev}/" \
-e "s/@BUILD_DATE@/${BUILD_DATE}/" \
< build/version.template > build/version.txtComment by yoavsha1 1 day ago
Comment by miggol 16 hours ago
That being said, for real portability of programs that have slightly outgrown the script moniker I really like Janet.
https://janet.guide/scripting/
It can compile your script to a static binary for distribution.
Comment by Alifatisk 1 day ago
I do wonder, let's say the scripting file is using lots of libraries, do you have to include some kind of requirements.txt file with it aswell when you want to share it with other people?
In Ruby, there is inline bundler which makes sharing a single Ruby script very portable.
https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...
Comment by bccdee 1 day ago
# /// script
# dependencies = [
# "requests<3",
# "rich",
# ]
# ///
import requestsComment by zelphirkalt 1 day ago
Comment by davidkhess 2 days ago
Comment by bjoli 1 day ago
Comment by Rendello 2 days ago
Comment by seabrookmx 1 day ago
Comment by abhiyerra 2 days ago
Comment by tayo42 2 days ago
Would be cool if python had a pipe operator though.
The back ticks in ruby is pretty ergonomic too. Wish python had a simpler way to run commands. Kind of tedious to look up subprocess run arguments and also break things up into arrays.
Comment by kstrauser 1 day ago
For example,
subprocess.run(“rm -rf ~/ some file”, shell=True)
and subprocess.run([“rm”, “-rf”, “~/ some file”])
have significant different behavior.Comment by lgas 1 day ago
Comment by kstrauser 1 day ago
Edit: I’m typing this on my phone, so brevity won over explicitness. The latter probably wouldn’t expand ~. Imagine a file named “/home/me/ some file” for a better example.
Comment by vovavili 19 hours ago
Comment by mackeye 2 days ago
Comment by kazinator 2 days ago
Python:
forty_two = ( 42 )
tuple_containing_forty_two = ( 42, )Comment by sevensor 1 day ago
42,
This expression is a tuple all by itself.Comment by kkfx 2 days ago
Comment by pxc 1 day ago
His ugly sh example:
morning_greetings=('hi' 'hello' 'good morning')
energetic_morning_greetings=()
for s in "${morning_greetings[@]}"; do
energetic_morning_greetings+=( "${s^^}!" )
done
and his more readable Python equivalent: morning_greetings = ['hi', 'hello', 'good morning']
energetic_morning_greetings = \
[s.upper() + '!' for s in morning_greetings]
And I'd write the shell version in Bash something like this: morning_greetings=(hi hello 'good morning')
printf '%s!\n' "${morning_greetings[@]}" \
| tr '[:lower:]' '[:upper:]' \
| readarray -t energetic_morning_greetings
Does it still involve more syntax? Yeah. Printing arrays in Bash always involves some. But it's clearer at a glance what it does, and it doesn't involve mutating variables.The piece this example is too simple to show (since it's focused only on data operations and not interacting with the filesystem or running external programs) is how much shorter the Bash usually ends up being than the Python equivalent.
Comment by binary132 1 day ago
Comment by Alifatisk 1 day ago
Comment by N_Lens 2 days ago
Comment by GutenYe 2 days ago
Comment by headcrash 1 day ago
Comment by zbentley 23 hours ago
Talking super simple stuff here, too: database drivers, markdown formatters, structured data parsers.
Comment by wiseowise 1 day ago
Comment by DonHopkins 1 day ago
Comment by headcrash 1 day ago
Comment by doppelgunner 1 day ago