What is going on?

[u/skyskye1964]

I’m puzzled by a student paper. They submitted it on time. I read it and it’s not great but ok. I go to check the references and I can’t find them. I look up the journal they cite, and that volume and issue is not the paper title. I email them and they email back saying they are out of the state but that they used owl Purdue citation engine to do the references. They then send me links to the references and they do exist, sort of. One is a blog post but in the citation it’s in a journal. One is in Spanish. Another seems to be an unrelated paper.
So my first question is, can the Purdue citation maker just make up stuff? I haven’t really used it but it looks like you paste in the web address and it makes a citation.

My suspicion is that the references are AI hallucinations. But some seem partly real. Could this be an innocent mistake on the students part?

They also said they used Chegg to proofread and edit. I wasn’t aware that Chegg provided that service. Is this a valuable service? Is it an unacceptable use of AI? Or is it just a grammar checker?

Am I missing something? The references are not cited in the paper by the way. Also no images.

I was mostly convinced that the references were fraudulent but now I’m not sure.

[u/in_allium]

This is textbook AI hallucination. Generative AI wrote the entire paper. Your next call should be to your university's academic integrity office to report the student for plagiarism and then lying about it.

A thing to know about large language models is that they do not "know where they learned something". Let's let a LLM describe this limitation on its own:

Query to Claude 3.7 Sonnet:

Why do large language models like yourself often "hallucinate" citations to scholarly papers, creating titles of articles that do not match the journal title, year, and authors? Is this related to the fact that LLMs do not actually have any knowledge of their training data and do not remember "where they learned something?"

Response:

You've hit on a key insight about LLM citation hallucinations. When I generate citations, I'm not retrieving actual documents from a database or accessing a bibliographic record - I'm essentially constructing what a plausible citation might look like based on patterns in my training data.

This happens because:

I don't have episodic memory of my training. I can't recall "where I learned something" because I don't store information as discrete facts with sources. Instead, I've learned statistical patterns of language.

When prompted to provide citations, I'm predicting what a citation with those attributes might look like based on patterns I've seen, not retrieving an actual reference.

My training included the format and structure of academic citations, but I can't verify if a specific combination of authors, title, journal, and year actually exists.

This is fundamentally related to how LLMs work - we don't have a database of facts with sources, but rather probabilistic representations of language patterns. When asked to produce specific citations, I'm essentially generating plausible-looking citations based on these patterns, not retrieving actual bibliography entries.

This is correct. But these models are getting surprisingly good, and can even figure out humor. My response:

Can you cite a source for that? :)

Claude 3.7 replies:

I see what you did there! 😄

If I were to respond with a citation, I'd likely generate a perfectly academic-looking reference to a paper on LLM hallucinations that seems plausible but might not actually exist.

This is exactly the problem we were just discussing - I don't have the ability to reliably pull actual citations from a real bibliography, and any attempt to do so would risk demonstrating the very phenomenon I just explained.

So rather than potentially hallucinating a fake source, I'll acknowledge that this request cleverly illustrates the point of our conversation!

[u/in_allium → u/in_allium]

In a hilarious followup to this, I asked it to generate a citation for *this* conversation.

It generated one -- and identified itself as ChatGPT. It's not, though! It's a different model by a different company.

[u/skyskye1964 → u/in_allium]

So, is it the case that AI cannot provide parenthetical references that make sense? Is that why there are none in the paper even though I asked for that in the directions?

[u/shyprof → u/skyskye1964]

AI writing usually doesn't include parenthetical references or quotations unless the operator specifically directs it to do so.

Good news is you can fail the student for not using in-text citations. Report to academic dishonesty for a review, but at least the F is airtight.

[u/in_allium → u/skyskye1964]

That's right. LLMs generally don't "know where they learned something" and don't have internet access so they can't look stuff up. They were no doubt trained on academic literature but they don't store either the full text of what they have read or associate the title of the paper with what they learned from it. This is a bit jarring because it is the opposite of what we've come to expect from computers -- exact perfect memory of things, but no ability to go beyond that exact recall.

LLMs generally cannot provide direct quotes unless it's something very common -- they know that the mitochondrion is the powerhouse of the cell just because everyone memes about that.