Depending on the state, you don't need to refer to each other as a spouse, just that people get (or could get) the impression you're spouses.

My father's last "marriage" was like this. After several divorces he was dead set on never getting married again, made sure only his (or her) name was on any purchases, and made sure his friends knew the situation. After a messy breakup where he discovered she had been diverting money for drugs and hiding bills, her lawyer managed to convince an Alabama court to find a common law marriage. Not long afterward she ended going to prison for arson after setting aflame some of my father's personal and real property, and his dog[1], in piques of spite. I think those felony convictions happened prior to or at least concurrent with the common law marriage proceedings.

Of course, his real mistakes occurred before he ever started cohabitating. Common law marriage rules were far down the list of wrong turns. But in any event, ironically a few years later Alabama abolished common law marriage.

[1] Dog survived after heroic efforts by, IIRC, University of Alabama, but had pretty severe scaring.