> You basically need two pieces of state

That’s enough for INDENT, but for DEDENT you also need a stack of previous indentation levels. That’s how, when the amount of indentation decreases, you know how many DEDENTs to emit.

The requirement for a stack means that Python’s lexical grammar is not regular.