Remix.run Logo
Remix clone Hacker News
new | show | ask | jobsGithub
▲ theamk 15 days ago

My rule has always been that once a shell script gets too complex, it's time to rewrite it in real language. This library is way, way beyond that threshold.

My recommendation for rewrite is "python with only stdlib". Using pip-installed libraries brings in the famous packaging problems, but you don't have to use it! Python's stdlib is fully "batteries included", and it has lots of useful things like ini file parser. And if your system has bash, it often has python as well, or at least it's easy to install. Things might get a bit more verbose, but they would be much more reliable and debuggable.

▲rollcat 15 days ago | parent | next [-]

Long before systemd took over the init world, Debian led a huge effort to switch their entire infrastructure to dash[1], specifically making heavy use of checkbashisms[2]. The main reason behind it was performance: POSIX sh implementations were much simpler, and easier to optimise.

[1]: http://gondor.apana.org.au/~herbert/dash/

[2]: https://linux.die.net/man/1/checkbashisms

My rule of thumb is that if a shell script calls for a Bash-only feature, exceeds 100 lines, defines more than three functions, includes a nested loop, or a pipeline that exceeds 80 characters, it's probably time to reach for Python.

There are reasonable cases for exceptions - my backup script has about 200 lines and 6 functions, but it's 100% portable between Linux & macOS.

▲akx 14 days ago | parent | prev | next [-]

> Using pip-installed libraries brings in the famous packaging problems, but you don't have to use it!

These days, with uv and PEP 723 inline script metadata, that's not an issue either.

    # /// script
    # dependencies = ["numpy"]
    # requires-python = ">=3.12"
    # ///
    import numpy as np
and

    uv run myscript.py
will just Do The Right Thing.
▲indentit 14 days ago | parent [-]

And if you come back to this script in a few years time and it pulls a newer version of numpy with an incompatible api, there is no easy way of knowing which version it was designed to be used with...

▲zanecodes 14 days ago | parent | next [-]

Only if you didn't run `uv lock --script` [0] and commit the lockfile. If you don't want that second file sitting next to the script, you could instead specify an `exclude-newer` timestamp, and uv won't resolve any dependency versions newer than that.

It might be cool if uv ignored dependency versions newer than the script's last modified timestamp, but this behavior would probably need to be very explicit to avoid the situation where a working script is mysteriously broken by making an innocuous change which updates its last modified timestamp, causing uv to resolve a newer, incompatible dependency version.

[0] https://docs.astral.sh/uv/guides/scripts/#locking-dependenci...

▲akx 14 days ago | parent | prev [-]

You can of course absolutely use `"numpy~=1.12"` as a requirement.

▲quickslowdown 15 days ago | parent | prev | next [-]

Have you seen the x-cmd package? It's nuts

https://x-cmd.com

It's legitimately impressive, the person/people behind this wrote some pretty interesting code, I've learned some things about Bash reading through parts of the code. But it's crazy at the same time.

▲quotemstr 15 days ago | parent | prev | next [-]

People are generally bad about following their own advice. They're worst at following advice to avoid shell scripts. :-) We all know in theory we should move things to Python or another robust language once they become sufficiently complex, but in the moment, adding just this one feature takes precedence. Iterate "just one feature" a few times and you end up with a 1,000-line bash program. (The word "script" becomes inapplicable.) Now what? You want to add configuration support to your bash program? Well, you might as well use something robust to do it.

▲pwdisswordfishz 14 days ago | parent | next [-]

Which is why you rewrite in Python long before reaching that threshold. I start considering it as early as when I need to add any non-trivial command line parsing: argparse is just so much better than "while getopt".

▲thyristan 14 days ago | parent | prev | next [-]

This is why my threshold for not using any shell is "does it have more than one loop, more than one if-statement, an argument or file format parser or more than 10 lines?". Meaning that only very trivial things can be shell scripts and you rewrite early enough such that it isn't really a hassle or painful.

▲theamk 15 days ago | parent | prev [-]

It does not have to be all-at-once.

Your bash program takes 10 input args and requires 15 env variables to be set? Don't reach out for that bash ini parser, create a python script which parses configs and invokes the original bash script. Next time you change feature, move it to python. Eventually you'll be mostly python.

Something that helps greatly with that is that you can embed bash in python. So initial conversion can be as simple as file rename + 3 lines of code.

    import subprocess
    subprocess.run(["bash", "-c", """
    echo This is bash
    if hostname | grep box; then
       echo running on a box
    fi
    """], check=True)
this lets you start by trivial .sh -> .py conversion, and then you can change one block at a time. Variables are tricky so do them first; or in the worst case, "declare -p" + "eval" + temporary files will save you.
▲quotemstr 15 days ago | parent [-]

It's not that simple. How are you passing argv to the bash script? What about how `subprocess.run` changes the error output to include Python exception spam? Plus, in most editors, you lose all the bash syntax highlighting in the bash program chunk. I haven't seen a lot of hybrid programs like this work well.

▲pwdisswordfishz 14 days ago | parent | next [-]

That's why it's just the first stage of an incremental rewrite.

argv is easy to pass, though.

▲theamk 14 days ago | parent | prev [-]

It is that simple. You can pass arguments to `bash -c` with no problems(just don't forget a dummy $0). If you don't want exception spam, replace `check=True` with explicit exit. Here you go, it's 5 lines now.

    #!/usr/bin/env python3
    import sys
    import subprocess
    result = subprocess.run(["bash", "-c", """
    echo I am in bash, I have $# args, arg1 is "$1"
    set -x
    "$@"
    """, "dummy.sh"] + sys.argv[1:])
    sys.exit(result.returncode)
python's shell tooling is really good (if a bit verbose). And another cool thing is "shlex.quote", so you can write something like:

    subprocess.run(["bash", "-c", f"rsync {shlex.quote(filename)} somehost: | some-script"])
and have "filename" safely inserted into script fragment.. No command escapes, and newlines, quotes, backticks etc... are all handled properly.

The bash syntax highlighting will disappear, correct, and that is a downside. But hey, you are supposed to be rewriting this in python, so it's just more motivation :)

▲mschuster91 14 days ago | parent | prev | next [-]

> And if your system has bash, it often has python as well, or at least it's easy to install.

The problem is, bash is a universal target. Every unixoid system has bash in one way or another, usually at /bin/bash or in a pinch /usr/bin/env bash. No matter if your target system is Debian oldoldstable or bleeding edge Gentoo, it will run.

Python however is a hot mess. Some operating systems are at `python` yielding v2, others v3. Others (Macports) have python2 and python3 binaries if the user doesn't also do a `port select` afterwards. The user may have overridden this with virtualenv's (that this exists at all is a disgrace in itself). And that all is before `pip` even has entered the stage, or before you try to figure out what whitespace format Python expects and what whitespace format you just used in the last YAML file you edited.

(The whitespace shit is what will keep me from using Python if at all possible.)

▲thyristan 14 days ago | parent | next [-]

That is why you don't use Python, you use Perl. Every Unix has Perl, and Perl has backward compatibility into the last millenium. Always use 'strict' and proper linting for readability and typo avoidance.

If you are able to install modules, add 'IPC::Run' and 'Getopt::Declare' or 'Getopt::Euclid'. Never again will you be able to tolerate anything else ;)

▲theamk 14 days ago | parent | prev | next [-]

Yeah, the naming was pretty bad for a while, but it seems today `#!/usr/bin/env python3` is a safe bet. And pip/virtualenv should not matter - remember, we've are talking about "python-without-dependencies"

As for tabs vs spaces - I really don't get it... My editor (emacs) picks the style automatically, I don't think I even had to set this up. But most of the python code I see use 2 or 4 spaces, so if someone puts tabs, it's immediately visible 8-character indents, which makes me alert (and then I turn on whitespace highlighting in my editor).

▲IshKebab 14 days ago | parent | prev [-]

uv has pretty much solved those issues, but it does need to be installed. Fortunately it's very easy to install.

Another option I've used is Deno.

▲ndsipa_pomu 14 days ago | parent | prev | next [-]

That's all well and good, but if you're looking for a language that will survive for decades, then BASH or shell scripts are a better choice. Often, there'll be some ancient machine that's been almost forgotten about and will die and need replacing. I can trivially copy a 30-year old BASH script and run it on a new machine, but can you do that with a 30-year old Python programme? How will today's Python survive the next 30 years?

▲theamk 14 days ago | parent [-]

python 3.0 code, released back in 2008, is still compatible with latest released python, with some exceptions like asyncio or obscure file formats which are unlikely to appear in scripts. The pip packages likely did break, but we are talking about python-with-stdlib here.

Granted, it's not 30 years old, but 17 years is pretty impressive. And I don't think there will be another big break, the 2.0->3.0 transition was so painful they'll likely avoid another one for decades.

▲ndsipa_pomu 12 days ago | parent [-]

17 years is very good. I don't have anything against Python, but there's a particular niche that shell/BASH scripts are ideal for. I keep considering writing more stuff in Python, but BASH operates at just the right level for tying systems together (computer duck tape).

▲xelxebar 14 days ago | parent | prev | next [-]

> rewrite it in a real language

This refrain shows up in every shell script discussion here, and hot take, I find it mostly ill-founded hogwash. Some of the worst spaghetti code I've seen is such Python and Javascript rewrites.

Choice of language only really helps when it encourages developers to use patterns that fit the underlying problem domain. However, Python's path of least resistance is OOP architecture, which usually fits terribly in the domains where shell scripts are ubiquitous, e.g. glue code between systems, infra administrative tasks, etc.

Almost all shell script tasks boil down to executing a linear collection of steps, and by organizing code thusly, I find shell scripts to keep code short, sweet, and very maintainable. Using a data-first design, with well-designed global state typically meshes well with shell's strengths—dynamic scope, pipes, word-splitting, etc, not to mention coreutils, grep, awk, etc.

IMHO, all the shell hate boils down to not recognizing an impedance mismatch between the programming paradigms of the programmer and language. Try writing Haskell like you would write Java or vice-vesa, and you'll end up with just as much of a mess.

</rant> This topic is a bug-bear of mine, for whatever reason. :shrug:

▲hiccuphippo 14 days ago | parent | prev | next [-]

I used to do the same but lately I've been enjoying writing scripts using bun.js and its shell API: https://bun.sh/docs/runtime/shell

The main reason is piping commands, way easier than python. It's very easy to convert a bash script to bun.

▲lsferreira42 14 days ago | parent | prev | next [-]

I agree with you on that, but I had this task on a legacy project with over 50k lines of shell scripts, and out of nowhere, it needed to start reading INI files and loading those values into environment variables. That's why this was written!

▲sebazzz 14 days ago | parent | prev [-]

> My rule has always been that once a shell script gets too complex, it's time to rewrite it in real language

Is Powershell a real language? Because it is a shell script too.

I think you meant to say "data types" and "data structures"?