> TI-BASIC programs are stored as tokens, not text: every command, function, and variable is a token of 1 or 2 bytes. The OS detokenizes (token→display string) to show a program and tokenizes (keypress/text→token) on entry; the parser walks tokens to execute.

From my memory of using a TI-83 in the late 90s, I would not be surprised if the keypad UI injects tokens directly based on your keypress, rather than "tokenizing the text". I seem to recall, for example, you could not position the cursor in the middle of a BASIC token, and if you managed to type out the tokens it would not work; you needed to find the right menu item to inject the correct token.

▲ siraben 2 hours ago | parent | next [-]

Yes, to type a TI-BASIC program you have to go through the calculator menus which directly insert the tokenized input into the buffer.

The weird thing about TI-BASIC is how seemingly innocent changes in the input can cause huge performance regressions e.g. https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.htm...

  For(I,1,N
  If 0
  1
  End

is much slower than

  For(I,1,N)
  If 0
  1
  End

	▲	asveikau 2 hours ago \| parent [-]
		The open paren being part of the tokens was always weird. I could imagine that doing strange things for the parser; when it sees a close paren it needs to know that several of the preceding tokens may have an open paren even without having a '(' token.

▲ duskwuff 3 hours ago | parent | prev [-]

I can confirm that. On the TI-83, many of the TI-BASIC tokens contained lowercase characters which couldn't be typed at all - you could only type uppercase letters on the keyboard. (There were a few lowercase letters available as tokens for special purposes, but it wasn't a full set.)

Interestingly, you could print tokens in strings - e.g. you could Disp "Disp ".

	▲	suburban_strike 2 hours ago \| parent \| next [-]
		The 83+ let you type the full set of lowerchase chars as well, but they used 2x as many bytes per character for storage.
	▲	7jjjjjjj an hour ago \| parent \| prev [-]
		There's actually a hidden lowercase feature, you can use an assembly program to enable it.