About a year ago I was subbing as a para and spent some time in a middle school history class. During a break between classes the teacher showed me their latest tech addition:
A LLM program, named Poe, I believe, that was set up to only answer queries based on texts specified by the teacher.
The sales pitch had been that middle school history students would ask Poe history questions and get factual answers.
Because, of course, that's how all of this works. 
The teacher showed me how it was supposed to work. She asked Poe a comparative question about the Declaration of Independence and the Articles of Confederation.
It answered correctly.
I asked her to ask for something that it shouldn't be able to find. IIRC she asked it about the topic of slavery in both.
It extruded a long and rich answer...
That was obvious bullshit.
Or at least it was obvious to her. Because she knew these documents. Her students wouldn't know any better.
And that is why, when I left her, she was angrily clicking around in Poe.
1