To follow up, after experimenting a bit with the source code:

1. Please, for the love of God, and for scientific reproducibility, specify library versions explicitly, and use pyproject.toml instead of an incomplete requirements.txt.

2. The 1,000 Sudoku examples are augmented with hand-coded permutation algorithms, so the actual input data set is more like 1,000,000 examples, not 1,000.

▲

rudedogg 4 days ago | parent | next [-]

Do you have a fork or the changes? I might take a look, and python dependency hell on Sunday is no good

▲

mkagenius 3 days ago | parent | prev [-]

> specify library versions explicitly

Sometimes even that is not helpful. It's a pain we have to deal with.

▲

gavinray 3 days ago | parent [-]

How is it not helpful?

A dependency lock file with resolved versions for both direct and transient dependencies = reproducible build

	▲	blincoln 3 days ago \| parent \| next [-]
		I don't know how common this is, but the fschat library maintainers went for at least a year without making an official release or updating the version number in their GitHub repo, so the only way to both have current code and a reproducible build (without just including the fschat library directly, of course) was to pin it to a particular GitHub commit hash, which would get you code that was current, but with the version number from 12+ months earlier. fschat is pretty popular for LLM-related work, so I assume this is at least not unheard-of for other notable third-party libraries.
	▲	mkagenius 3 days ago \| parent \| prev [-]
		I don't remember the exact scenario but it might have been related to the underlying python or some sys library being a little different and then the dependency lock not being compatible with it.