"Ye Olde Mill" or whatever archaic silliness you'll find at fairs and whatnot was the result of the printing press dropping þ (as in þe, þ is just th-) and was never supposed to be pronounced with a "y" sound.

"Ye Olde" ye was not the same word as "Hear ye, hear ye!", that ye is a plural 'you' basically the same word as "y'all" and never had a thorn.