Peak fertility though is early 20s. By the time people are in their 30s, having trouble getting pregnant is a more common problem.

Are people who want to have kids really struggling to have them though? Keep in mind that fertility rate is not actually about fertility (not directly at least). It's a measure of the kids being had rather than a measure of the capability that the name would lead people to think.

Exactly. So since we observe that actual birth rates among women in their 30s displays a different trend than that of women in their 20s, despite decreased natural fertility, there has to be more going on.