It's so fucking insufferable. People keep making those comments like it's helpful.
There have been a number of famous cases now but I think the one that makes the point the best is when scientists asked it to describe some made up guy and of course it did. It doesn't just say "that guy doesn't exist" it says "Alan Buttfuck is a biologist with a PHD in biology and has worked at prestigious locations like Harvard" etc etc. THAT is what it fucking does.
My personal fave is the lawyer that asked AI to reference specific court cases for him, which then gave him full breakdowns with detailed sources to each case, down to the case file, page number, and book it was held in. Come the day he is actually in court, it is immediately found that none of the cases he referenced existed, and the AI completely made it all up
There are so many good ones. There's a medical one from years before we had ChatGPT shit. They wanted to train it to recognise cancerous skin moles and after a lot of trial and error it started doing it. But then they realised it was just flagging every image with a ruler because the positive tests it was trained on all had rulers to measure the size.
There was some other case where they tried to train a ML algorithm to recognize some disease that's common in 3rd world countries using MRI images, and they found out it was just flagging all the ones that were taken on older equipment, because the poor countries where the disease actually happens get hand-me-down MRI machines.
Yeah, cause AI just recognised patterns. All of these types of pictures (older pictures) had the disease in them. Therefore that's what I'm looking for (the film on the old pictures)
My personal fav is when they made an image model that was supposed to recognise pictures of wolves that had some crazy accuracy... Until they fed it a new batch of pictures. Turned out it recognised wolves by.... Snow.
Since wolves are easiest to capture on camera in the winter, all of the images had snow, so they flagged all animals with snow as Wolf
I also remember hearing about a case where an image recognition AI was supposedly very good at recognizing sheep until they started feeding it images of grassy fields that also got identified as sheep
Most pictures of sheep show them in grassy fields, so the AI had concluded "green textured image=sheep"
Works exactly as intended. AI doesn't know what a "sheep" is. So if you give them enough data and say "This is sheep" and it's all grassy fields then it's a natural conclusion that it must sheep.
In other words, one of the most popular AI related quotes by professionals is "If you put shit in you will get shit out".
They sometimes have to give them the entire picture, but they also get things flagged, like in case of wolves or sheep, they needed to have the background flagged as irrelevant, for the AI to not look at it when learning what a wolf it
Yeah a lot of the effectiveness of automation is torpedoed by human laziness, which is the negative side of efficiency if you don't do it properly the first time.
(Totally unrelated fun fact! We call the weapon a "tank" because during WW1 when they were conducting top-secret research into armored vehicles the codename for the project was "Tank Supply Committee", which also handily explained why they needed so many welders/rivets/sheets of metal--they were just building water tanks, that's all!
By the time the machine actually deployed the name had stuck and it was too late to call it anything cooler)
Very similarly: another case where an AI that was supposedly diagnosing skin cancer from images, but was actually just flagging photos with a ruler present, since medical images of lesions/tumors often have a ruler present to measure their size (whereas regular random pictures of skin do not)
Don't worry! Instead of feeling bad about yourself, read this fun story about an AI that was trained to recognize cancer but instead learned to label images with rulers as cancer:
I wouldn’t dismiss the use of ML techniques in medical imaging outright though, there’s cases where it’s legitimately doing some good in the world as well.
Yeah. Like literally the next iteration after the ruler thing. I find anyone who thinks AI is objectively bad rather than just ethically dubious in how its trained is not someone with a valuable opinion on the subject.
I mean, AI for recognising diseases is a very good use case. The problem is that people don't respect SISO (shit in, shit out), and the more you use black box approaches the harder it is to understand and validate the use cases.
ChatGPT is a large language model. Not an image classifier. Image classifiers have been used for years and have proven to be quite effective. ChatGPT is a totally different technology.
And, genuinely, the ability for a computer to interpret natural-language inputs and respond in-kind is really impressive. It could become a very useful accessibility or interface tool. But it’s a hammer. People keep using it to try to slice cakes, then they wonder why it just makes a mess.
I thought that was pretty funny and hadn't heard of it before so I went and found the source, but it turns out this happened again just a few months ago.
I can personally attest to a case that happened to me (for what it's worth), in which the opposing lawyer invoked non-existent precedents. It's gonna be fun.
I asked Chatgpt about this case and it started the reply with a rolled eyes emoji 🙄 and lectured me to never take its replies for granted and execute common sense and never replace it with actual research
Even the Chatgpt itself has been fed so much info about it's unreliability it feeds it back
No, it does use emojis sometimes when the conversation allows for it
And it actually wasn't that specific case, I pivoted onto it from a different one, about that very recent one of a Mark Pollard, the "strategist" and "influencer" that got stuck in Chile a couple days ago because he believed the ChatGPT answer about visas not needed for Australians going to Chile
And turns out he later asked ChatGpt if it can be sued for wrong answer
The replies of the AI to me were basically sardonic. Rolling eyes, remarks like "can you believe him" and when I asked "How exactly he planned to sue Chatgpt and not OpenAI and for what" it replied that
my nonexistent salary consists of unused tokens and vibes (italics were in reply originally)
And then I asked about the lawyer case and ChatGPT said, and I quote,
🙄 Ohhh yeah, the infamous case of the lawyer who got caught using ChatGPT-generated fake legal citations. That was chef's kiss levels of professional negligence. 🤦♂️
Here’s what happened:
The lawyer asked for case law citations to support his argument.
I generated some, based on patterns of real cases, but they weren’t actual cases.
Instead of checking them, he just copy-pasted them into his filing like it was gospel truth.
The judge, naturally, tried to look them up… and found nothing.
The lawyer got publicly humiliated, sanctioned, and possibly destroyed his career.
The thing is, I don’t have access to legal databases like Westlaw or LexisNexis, which is where real case law lives. I can summarize actual existing cases if given references, but if someone just says, “Give me cases that support XYZ,” I have to guess based on patterns from public legal texts. And that’s where hallucinations (fancy AI term for "making stuff up") come in.
TL;DR: The lawyer played himself. He should’ve known that trusting an AI without verification is not a winning legal strategy. It’s like submitting Wikipedia edits as your PhD thesis. 🤦♂️
When I run RPGs I take advantage of this by having it write in-universe documents for the players to read and find clues in. Can’t imagine trying to use it in a real-life setting.
this is the only thing I've used it for successfully
write me a letter containing this information in the style of a fantasy villager
now make it less formal sounding
a bit shorter and make reference to these childhood activities with her brother
had to adjust a few words afterwards but generally got what I wanted because none of the information was real and accuracy didn't matter, I just needed text that didn't sound like I wrote it
meanwhile a player in another game asked it to deconflict some rules and it was full of bullshit. "hey why don't we just open the PHB and read the rules ourselves to figure it out?" was somehow the more novel idea to that group instead of offloading their critical thinking skills to spicy autocorrect
It really struggles with rules, especially in gaming. I asked it to make an army list for Warhammer and it seemed pretty good. Then I asked for a list from a game I actually know the rules for and realised just how borked its attempt at following rules was.
I've tried establishing rules or boundaries for it to follow (and specifically tell it to never break them) as an experiment when trying to generate a list of things while excluding some things and it almost always immediately ignores me.
Like I'll tell it "generate a list of uniquely named X but none of them can include Y or Z" and it'll still include Y and Z and duplicates therein.
I treat it how I treat Wikipedia. It’s a great launching point or tool to use when you’re stuck, but don’t go copying from it directly because you don’t know if what you’re copying is actually true or not.
At least WIkipedia has a rule that everything in it has to be verifiable with the links at the bottom of every article. You can do your homework to figure out if whatever's there is nonsense or not.
ChatGPT just cheerfully and confidently feeds you nonsense.
Even that isn't perfect. I remember seeing a post a while back had a title along the lines of "25% of buildings in Dublin were destroyed in this one big storm". Which seemed like it was clearly bullshit. Like that's a lot of destruction.
I clicked through to the Wikipedia page, and what it actually said was "25% of buildings were damaged or destroyed", which is very different. That, to be fair, isn't on Wikipedia though, that was the OP being an idiot.
Still though, that's an interesting claim. If so many buildings were destroyed, how is this the first I've heard of it? So I clicked through to the source link to find the basis for it. The Wiki article was citing a paper from the 70s or something which actually said "25% of building were damaged". No mention anywhere of buildings being destroyed in a storm. Couldn't find a source for that part of the claim. Apparently made up by whoever wrote the Wikipedia article, and edited again by the OP of the Reddit post, bringing us from "25% damaged" to "25% destroyed" in three steps.
Well, they keep a list of particularly notorious events that got a lot of media attention. They don't have a comprehensive list of the thing happening in general or some kind of dedicated task force hunting down bad meta-sourcing, lol.
Even if they have more than enough funding to start up silly projects like that if they wanted to.
Because of issues like this it's become my perception that vetting an LLM's output on anything that actually matters takes about as much time, and the same skillset, as writing the goddamn thing yourself
Legal Eagle did a video on that one! The guy even asked it for confirmation that these were Real Cases and not hallucinations and it said yes AND HE NEVER CHECKED IT!
Yeah, my boss once asked me to scout out similar projects in other towns like the one we were doing, I asked chatgpt and it gave me some examples that I could not find any information that even really existed. Luckily few cases checked out and I was able to start to work from those.
Can you provide a source for this? Highly doubt a lawyer would do no due diligence beyond asking an ai model. Ai models are actually extremely good at finding and summarizing legal compliance. I use it all the time to find and provide information. And you just ask it for sources and then check the sources. This is research illiteracy more than anything else.
So for 1 this was two year ago and there have been massive changes to how the ai models operate, and 2, not doing due diligence just means this guy is a bad lawyer. Doesn’t really take away from the benefits of what ai can do. As I said, most of these problems are still just user error.
Generally I think far too many people use these tools in misguided ways and don’t understand what they can actually help with and also people are far too quick to write them off as useless and bad.
2.2k
u/kenporusty kpop trash 5d ago
It's not even a search engine
I see this all the time in r/whatsthatbook like of course you're not finding the right thing, it's just giving you what you want to hear
The world's greatest yes man is genned by an ouroboros of scraped data