>In the United States, Section 230 is a section of the Communications Act of 1934 that was enacted as part of the Communications Decency Act of 1996, which is Title V of the Telecommunications Act of 1996, and generally provides immunity for online computer services with respect to third-party content generated by their users. (Emphasis mine)

So either the content is user generated and their training of the model should be copyright infringement, or it's not and Section 230 does not apply and this is speech for which Open AI is responsible.