One thing I have always thought was missing in game theory (and it is probably there but I just haven't looked hard enough) is a mathematical framework for how to build trust to increase the infinite payout for everyone. If in the decision making the idea of an offering is added in then it brings up the possibility of gauging trust and building trust so that future actions can capture more value until an optimal infinite policy is attained. So, for instance, I look at all my possible options and I choose one based on how much I trust the other party AND how much I want to increase that trust in the future. So I give them an offering, select an option that gives them a little more but at a cost to me, to prove that I am willing to increase trust. If they reciprocate then I loose nothing and the next offering can be bigger. If they don't then I gained knowledge and my next offering is smaller. Basically, this is like tit for tat but over time and intended to get to the optimal solution instead of the min max solution. Clearly I'm not a mathematician, but I bet this could be refined to exact equations and formalized so that exact offerings could be calculated.

With an optimal way of determining fair splitting of gains like Shapley value[0] you can cooperate or defect with a probability that maximizes other participants expected value when everyone act fairly.

The ultimatum game is the simplest example; N dollars of prize to split, N/2 is fair, accept with probability M / (N /2) where M is what's offered to you; the opponents maximum expected value comes from offering N/2; trying to offer less (or more) results in expected value to them < N/2.

Trust can be built out of clearly describing how you'll respond in your own best interests in ways that achieve fairness, e.g. assuming the other parties will understand the concept of fairness and also act to maximize their expected value given their knowledge of how you will act.

If you want to solve logically harder problems like one-shot prisoners dilemma, there are preliminary theories for how that can be done by proving things about the other participants directly. It won't work for humans, but maybe artificial agents. https://arxiv.org/pdf/1401.5577

[0] https://en.wikipedia.org/wiki/Shapley_value

There's certainly academic work about game theory and reputation. Googling "reputation effects in repeated games" shows some mentions in university game theory courses. There's also loads of work about how to incentivizing actors to be truthful (e.g. when reviewing peers or products).

Signaling theory (in evolutionary biology) might also be vaguely related.

This is the TCP backoff algorithm, specifically the slow start to find the optimal bandwidth. In your analogy, it would find the optimal amount that a person is willing to reciprocate.

Not only does this algorithm exist, but we're using it to communicate right now!

https://en.wikipedia.org/wiki/TCP_congestion_control

Look into cooperative game theory. If I remember correctly, trust is modelled as a way of exchanging information and influencing the probabilities that other players place on your next action

Nature already solved that and implemented it. It's all around us with relationships. Not just among humans. They typically last over a single interaction. Especially if you don't ignore the rules of the game.

But the game theory of nature also leaves room for other sort of players to somehow win over fair play. I thought this was a bug but over time realised it is a feature, critical to making players as a whole stronger. Without it there would be no point for anyone to be creative.

If you can solve the issue and make a playbook so that everyone do tic for tac, it won't take long for a bad actor to exploit it, then more, then you are back to where we are now.

Do you have an example of this in mind where the known "tit for tat" strategy falls short?

Economist here (but not a game theorist): Repeated games are actually really hard to find closed form equilibria for in general. Additionally, many repeated games have multiple equilibria which makes every bit of follow on analysis that much more annoying. Only very specific categories of repeated game have known nice unique solutions like you are hoping for, and even then usually only under some idealized information set structure. But as other commenters have said, this is an active area of research in Economics and many grad classes are offered on it.