Remix.run Logo
fy20 3 days ago

English often has a lot of redundancy, you could rewrite your comment to this and still have it convey the original meaning:

Regarding cost per token: is a token ideally a composable, atomic unit of information? Since English is often used as an encoding format, efficiency is limited by English's encoding capacity.

Could other languages offer higher information density per token? Could a more efficient language be invented for this purpose, one teachable to humans, especially aspiring prompt engineers?

67 tokens vs 106 for the original.

Many languages don't have articles, you could probably strip them from this and still understand what it's saying.