I appreciate you doing this and sharing it. I had a similar experience with rust and tokenization library (BERTScore) and realized it was better to let the barely worse method stand because the effort was not worth it to maintain long term