Remix.run Logo
munchbunny 9 hours ago

> We do this and then no one reads it.

My experience with AI-written documentation is that, as a category, it contains just enough technical errors that more often than not I end up spending time running down confidently stated errors the old fashioned way: cross-referencing API documentation, reading code, etc. At that point you inevitably end up trying to get in touch with a human from the other team.

In other words, it's not because I don't know it's there, or I don't want to read it, it's because my priors tell me it's going to waste my time. What I end up doing is searching, finding the docs, and reading until I start to get the feeling of hallucinated factual errors.

I'm sure there's AI-written documentation in there where the authors really did a good job reviewing it, and I just didn't notice. But I've also had engineers under me lose weeks of productivity at this point because of documentation confidently presented, insufficiently vetted, that turned out to be wrong.

tcoff91 9 hours ago | parent | next [-]

1-shotting documentation with AI is a bad idea. You then have to have multiple verification passes to have any chance of it being worth anything.

Like if you have an AI generate a runbook, then you better have the AI actually validate the runbook and make sure it is correct.

ink_13 9 hours ago | parent | prev [-]

The step that a lot of people leave out is the revising: it is worth the human hours to go over the outputs of AI for this kind of thing with a fine-tooth comb to make sure it is usable, readable, accurate, etc.

Imustaskforhelp 8 hours ago | parent [-]

I would then have to wonder if all the time saved in the first place at all. I have heard anecdotes of verifying AI output to be more time-consuming and generally even more painful/stressful than doing the whole work yourself perhaps.

As an employee, I would prefer to not be stressed and as an employer, I would prefer to not additionally stress my employee on all times so that I can have them use the stress for the works which truly matter the most (which is a limited resource as an employee shouldn't be burnt out or shouldn't be given unlimited stress)

Also its probably the ethical thing to do but even rational logic dictates so in my honest opinion (not sure how much of it works exactly on documentation stuff though*)