> I can't say I've ever experienced this. Are you sure it's not related to other things in the script? I wrote a single file Python script, it's a few thousand lines long.

It's because of module imports, primarily and generally. It's worse with many small files than a few large ones (Python 3 adds a little additional overhead because of needing extra system calls and complexity in the import process, to handle `__pycache__` folders. A great way to demonstrate it is to ask pip to do something trivial (like `pip --version`, or `pip install` with no packages specified), or compare the performance of pip installed in a venv to pip used cross-environment (with `--python`). Pip imports literally hundreds of modules at startup, and hundreds more the first time it hits the network.

▲

nickjj 4 days ago | parent | next [-]

Makes sense, most of my scripts are standalone zero dependency scripts that import a few things from the standard library.

`time pip3 --version` takes 230ms on my machine.

▲

maccard 4 days ago | parent [-]

That proves the point, right?

`time pip3 --version` takes ~200ms on my machine. `time go help` takes 25, and prints out 30x more lines than pip3 --version.

▲

nickjj 4 days ago | parent [-]

Yep, running time on my tool's --version takes 50ms and funny enough processing 10k CSV lines with ~2k lines of Python code takes 100ms, so 50ms of that is just Python preparing things to run by importing 20 or so standard library modules.

	▲	zahlman 3 days ago \| parent [-]
		> so 50ms of that is just Python preparing things to run by importing 20 or so standard library modules. Probably a decent chunk of that actually is the Python runtime starting up. I don't know what all you `import` that isn't implied at startup, though. Another chunk might be garbage collection at process exit.

▲

fwip 4 days ago | parent | prev [-]

And it's worse if your python libraries might be on network storage - like in a user's homedir in a shared compute environment.

▲

dekhn 4 days ago | parent [-]

Exactly this. The time to start python is roughly a function of timeof(stat) * numberof(stat calls) and on a network system that can often be magnitudes larger than a local filesystem.

▲

zahlman 4 days ago | parent [-]

I do wonder, on a local filesystem, how much of the time is statting paths vs. reading the file contents vs. unmarshaling code objects. (Top-level code also runs when a module is imported, but the cost of that is of course highly module-dependent.)

	▲	dekhn 4 days ago \| parent [-]
		Maybe you could take the stat timings, the read timings (both from strace) and somehow instrument Python to output timing for unmarshalling code (or just instrument everything in python). Either way, at least on my system with cached file attributes, python can startup in 10ms, so it's not clear whether you truly need to optimize much more than that (by identifying remaining bits to optimize), versus solving the problem another way (not statting 500 files, most of which don't exist, every time you start up).