And start with simpler regular rules and get more complex over time as words are imported and reimported, pronunciations shift, grammatical rules morph and evolve (often to simplify grammatical genders and cases) while leaving their mark, and spelling changes.

For example, goose/geese is the result of the plural form and singular form undergoing different paths in the Great Vowel Shift resulting in the different vowels in the modern form.

There's also evidence that Proto-Indo-European had laryngeal consonants that have disappeared in all modern languages derived from it [1], but have left their mark on the descendant languages.

[1] https://en.wikipedia.org/wiki/Laryngeal_theory

Then there are also the lovely instances of deliberate misspellings / insertion of letters into words that never had them in English.

Eg. receipt, which has the p only in Latin, but had long lost it by the time Old French brought it to Britain.