I think you're right with the current LLM/transformer architecture. There are several factors that affect model size:

- The number of token values supported by the model ("n_vocab").

- The number of parameters/features that are used to represent each token ("d_model").

- The number of attention layers there are ("n_layers").

such that the number of parameters is approximately:

   p ~= 12 * n_layers * d^2_model + n_vocab * d_model

It should be possible to keep the model size smaller by using better architectures, or making improvements to the existing model architecture.

For example, improving the token model by possibly using something similar to the image and audio data and getting the model to learn its own internal representation of the byte/character data instead of doing a tokenization pre-processing step. This way, instead of a separate model learning that several bytes/characters appear together, the transformer could learn things like language-specific prefices and suffices, character pairings (like in Japanese, Chinese, and Korean), and other syntactic morphology. It may also help with solving issues like "how many X characters are in the word/phrase Y". You could also experiment with using either 256 parameters (one per character in a byte) or using a single parameter per byte (that is 1/byte_value).