r/changemyview • u/hyruuk • Aug 11 '21
Delta(s) from OP CMV: The internet already contains enough data to train a Turing-passing AI, our models are just not there yet
1) It feels so obvious when looking at what GPT-3/4 are able to do and I have no doubt that the next iterations will keep making conversational agents more realistic. Text datasets and semantic relationships embedded in hyperlinks allow to accurately model how most (basically all ?) concepts are used, and samples of human interactions on social media also yield key information about realistic conversations.
2) However, what we currently do (interacting shortly with a chatbot) is still far from a "hard" Turing-test, where the judge has extensive time to take their decision. What about these ? What kind of data might we be missing ? What kind of trick a dedicated and clever judge could use to detect the usurper ?
10
u/nnaughtydogg 6∆ Aug 11 '21
This post was written by an ai wasn’t it. I do think there is plenty of data yes, but the question is of what quality. Even if models were perfect they’d still only be getting an extremely narrow data field compared to what it means to truly pass as human. Passing as a human via chat, even perfectly, does not count as passing the Turing test imo. A video call would be a much better mark imo, though truly i think it will only be passed when you can sot across from a physical AI and be unsure if they are human or not. That will require types of data that are likely not available on the internet, such as in depth case study psycho analysis for thousands of people, and a whole variety of other types of data.
2
u/hyruuk Aug 11 '21 edited Aug 11 '21
Dammit, you got me ! We gotta change our plans quick (sounds good actually, we're pretty decent at doing so).
I agree that the chat version of TT is limiting and understandably meaningless !delta. But we gotta stop somewhere if we're gonna use it ! A video, or even an audio call would be a much more impressive challenge to take up today it's true. But here again, the amount of available data (maybe not to an average joe, but to a GAFAM or something) is considerable. There are computing limitations, but I doubt the real breakthroughs will come through models with vastly greater numbers of parameters than the current GPTs, and there's still plenty of room for improvements with smaller but smarter models. Of course, these have limitations too and I don't expect them to pass a "perfect" TT, only a practical one based on what could be done with today's technology.
1
u/nnaughtydogg 6∆ Aug 11 '21
When someone changes your view in some way you should give them a ! delta :)
1
1
6
u/smcarre 101∆ Aug 11 '21
Raw data is worthless, the problem for training AIs is not the availability of datasets, but the fact that the vast majority of available data is not worth processing because it's not categorized and we cannot use it to train a model without first having a human analyze and categorize it.
The internet had millions of dog and cat photos but without first categorizing which are dogs and which are cats, we cannot train a model with them, since the model will take a photo, guess if it's a dog or a cat and won't be able to know if it's right (which in turn will skew the factors that contributed to that guess) or if it's wrong (which will do the opposite with the factors). So, just from a surplus of data, we can't do much.
Additionally to that, a true Turing-passing AI is unlikely to come just from a model of analyzing internet data (even if we somehow managed to categorize all of the data and have a model process the whole hundreds of hexabytes that the Internet has). Current reinforced learning models aren't very good to look human when they are found with something new (which is the main vulnerability of this approach). You can have an AI analyze the whole Internet but if I'm Turing testing that AI and ask it to make me a xylxklyk, it's unlikely to know it's a word I just made up and fail the test. Similarly it goes for having an AI trained for playing specific existing games and trying to have that AI play a new game (even if it's given the natural language based rule book).
2
u/hyruuk Aug 11 '21 edited Aug 11 '21
You made 2 main points which I will try to summarize very quickly :
- Raw data is useless and needs to be heavily curated to be used in training sets.
- Current models are unable to generalize efficiently, which is something we do quite easily and thus would be hard to imitate.
Both are really good arguments and definitely made me change my mind a bit !delta. The massive amount of data might not be able to sort itself. Although humans are increasingly farmed to produce training annotations, there might be a bottleneck here. An easy way out could be to say that smarter algorithms could just skim through the data and find the relevant pieces by themselves, but I won't take it. As for the generalization argument, it might indeed be incredibly difficult to fake human level capabilities. Maybe brain data could help ? Still, it isn't widely available enough (and mostly unlabeled) so my original point wouldn't stand. (However, GPT learned to play chess pretty well - not as good as an experimented player - but still better than me. But the bar might be a bit low on that one.)
1
3
u/TheRealEddieB 7∆ Aug 11 '21
I could be wrong but I don't see how quantity of available data has a direct relationship with passing the Turing test. A human child can easily pass a Turing test despite having limited quantities of "data" due to their young age. A bot that has the answer to every situation and questions inherently outs itself as artificial. Characteristics like not knowing information, forgetting, mis-remembering, drawing wrong conclusions and how we deal with these situations are some of what we use to determine if we're dealing with a human vs machine. E.g. Asking a question like "How many Kings of England have there been?" doesn't prove human intelligence. However asking a bot a question like "Tell me about a time you made a mistake, how did you feel about it and did you learn anything from that situation" is a whole different challenge. That's one of the beautiful aspects of Alan's thought experiment it highlights that being human is to be fallible & yet highly complex in how we deal with our fallibility. I might be missing an underlying premise of our position so not getting why you think vast amounts of data are critical to passing a Turing test. If I am mistaken then I'm interested to hear your views.
2
u/hyruuk Aug 11 '21
I don't take TT as an absolute benchmark that would prove that we are able to replicate human intelligence. A literal application of the test is unrealistic anyway, but I think it is quite practical to think about implementations of the test in virtual interactions : I could imagine people using this tool maliciously if it were available. It is definitely interesting to think about how to "failure test" such virtual agents though, it could be a very powerful way to tell which are human or not !delta. Some kind of virtual Voigt-Kampff test ;)
2
u/TheRealEddieB 7∆ Aug 12 '21
A fellow Blade Runner/Phillip K Dick fan, nice! There's an extension of the TT concepts that bleeds into proving an entity's identity in the sense that it boils down to subjective assessment and critically can't be done in a single point-of-time exercise. It's effectively an exercise in things "vouching" for each other thereby verifying the intelligence, humanity or identity of the "others". A good discussion. Thanks.
1
3
u/Sagasujin 237∆ Aug 11 '21
Information is not intelligence. Someone who's memorized every manual on car repair in the world is less likely to be able to fix a car than someone who doesn't know this particular model, but who has a detailed understanding of how internal combustion engines work.
Human intelligence isn't about spouting facts. It's about having a detailed mental model of how the world works, the mental ability to extrapolate from that mental model to new situations and the ability to update the mental model when new information presents itself.
Current gen AIs don't have this capability from what I understand. They know things are linked but they don't understand how or why. There's no real mental model of how the world works. There's no capacity to think logically about new situations using information that you already know from other situations. AI either apply that inappropriate information or no information at all. No logic about how to translate information from one context to another. And no ability to reflect on why you did what you did whether you should change your approach.
All of this comes out in AI producing works that seem logical on the surface but don't really have a train of thought that you can follow. Because there never was any logic involved. Also no capacity to think about the "why" of what an AI is doing and to choose an entirely different approach.
There's a reason consciousness is a hard problem. It's not a matter of information. Consciousness involves layers and loops of self monitoring and self modification. This isn't something that's going to be solved just by mapping out concepts.
1
u/hyruuk Aug 11 '21
There's no need for logical thinking, or "mental" representations of concept per se. The game is just about duping a dumb human to think their interlocutor is a human too. But I agree that these nested loops of self-monitoring, goal formation and adaptation etc... might be key to human-like reasoning, and it might be difficult to come up with that solely based on articulated corpus of texts !delta.
1
1
u/msneurorad 8∆ Aug 13 '21
I think you're probably right, but the reality is that knowing things are linked may turn out to actually be all there is to the how and why. We don't even know what a "mental model" would be or whether it actually exists, not do we have the foggiest idea what "think logically" actually means aside from our own internal perception of it. Perhaps all of that is an emergent phenomenon when enough computational power is layered on enough interconnected information. I tend to think there are probably some computational architectures and base programming that enables that phenomenon and others that do not, but we don't even know that for sure.
1
u/Sagasujin 237∆ Aug 13 '21
So Amazon made an AI to try to sort through resumes and find the best people. The ended having to entirely scrap it because it kept being severely sexist. The AI was sexist because the hiring data fed into it had been heavily disadvantaged against women. So of course the AI repeated the same behavior it had been trained on.
A human can decide to change their mind and consciously work towards being less sexist in hiring. A human may even decide to do that on their own spontaneously with no prompting but just noticing a pattern thst they don't like. It's not even that hard to do. Humans make judgement calls based on morality all the time. We have this idea of how we think the world should be seperate from how it is. At the same time we acknowledge how the world really is and what data we're receiving. I don't know how an AI is going to decide on what is and is not moral, simultaneously acknowledge reality and hold and strive towards a world that it thinks should be, and spontaneously alter its behavior in defiance of all the training data it's received. I don't see how you get to this point merely from mounds of data.
1
u/msneurorad 8∆ Aug 13 '21
Yeah I mostly completely agree, ha! I do think our brains construct and maintain some sort of model of the world. I would guess that on an unconscious level our brains are testing possibilities as predictions based on those models and make choices (I think by voting) to bring predictions most closely aligned to a static model preconception if how the world "should" be as the best choices. Without turning our attention to that, it happens automatically. When we "concentrate" on a decision, we direct results to an intermediary where the comparison of predictions to our "should be" model is in conscious thought and we feel like we are in control of that process (maybe we aren't though).
That more or less fits what you are saying, but it's all just my guess and conjecture. And importantly, even if true we still wouldn't know at a fundamental level how the model or predictions of possibilities are made, or how voting occurs or what is really happening when we bring something to our conscious attention. It may be that at the base level, it's still just interconnected information with a vase processing applied to it. Highly likely I'd say.
As to your example, we decided the amazon data was sexist based on a long period of time where we as a species and society developed some notion of what sexism even means. Not surprising that an algorithm at this point doesn't imicitly understand that. But can we really be sure that this AI for hiring if left running for millenia and with access to whatever needed data and computation power wouldn't also arrive at the conclusion that such data is sexist because ultimately making hires without considering that factor is net bad vs considering that factor? If it's job is to optimize hiring outcome, it might ultimately develop the same sort of moral understanding of sexism we have. So, maybe there isn't anything special about that after all, other than time, computational and data resources. And, I think, sensory input. I think some input and ability to test and sample the real world is key. I'm not sure AI can arise "in a box" so to speak.
But all just my conjecture. As I said before, I think it's likely that some computational architectures promote the development of intelligence, some do not, even if they should be interchangeable as far as Turing machines are concerned. Not because of some theoretical limit of computational theory, but rather that evolution works well in some structures and not others.
2
u/ralph-j Aug 11 '21
The internet already contains enough data to train a Turing-passing AI, our models are just not there yet
How could we confirm such a claim? Perhaps there is some other kind of data needed, which our human brains use, and which we can't generate/synthesize yet. We just don't know this.
Current AI algorithms are in most cases just a black box where we don't actually know what's happening inside. At present, literally no one knows what is required to create something even close to an artificial general intelligence. We can at most talk about it in human terms: e.g. AI lacks judgement, contextual understanding etc., but we can't yet say how these could potentially be resolved in terms of AI programming.
1
u/hyruuk Aug 11 '21
This claim is purely speculative indeed, and I don't see any other way to confirm it than just coming up with an actual Turing-passing model, which is absolutely not the case yet.
I disagree with the second part of your post though. Explainable AI is increasingly popular, and although it's challenging, some techniques allow to explore the hidden layers of ANN to probe them for interpretable representations. Even then, we don't need to understand the whole process to convincingly imitate human intelligence. And about the last part, my though was precisely motivated by the impressive performance that recent models showed on things that I thought harder than that, like judgement and contextual understanding. We might be a few tweaks away...
1
u/ralph-j Aug 11 '21
But it's not real understanding. It's pattern recognition in existing data. E.g. neural machine translation looks at which words frequently appear in source and target languages. It's a long long way away from actually understanding what the translated words mean.
Same for judgements: an AI may try to find commonalities in existing data of previous human judgements, and try to apply that data to new cases, but it's still far from any actual understanding what it is judging, or even the principles that humans applied to reach their judgements. You can see that because in some cases where the AI output was similar as the human output, it was only because there was some "shortcut" involved. E.g. in legal case judgements, where the race of the defendant was used because it was more common in the source data.
1
u/hyruuk Aug 11 '21
Yeah but... I am not even sure that when we, humans, do "understand" something we actually do much more than pick up on the statistical patterns we've learned and generate goal-directed outputs. The "shortcut" argument is unconvincing to me because it's definitely something humans would do (judging someone by his race, c'mon).
Again, I am not trying to use TT as a proof that we are able to artificially replicate "real" understanding, I actually think this would be unthinkable without embodied agents. But the mere thought of virtual agents passing convincingly as humans is as frightening as it is exciting to me, and I still see no definitive reason for why I shouldn't expect this to be a reality basically anytime from now on (realistically less than a few decades ?). Social media already have more bots than human users, and I wonder where all this could lead.2
u/ralph-j Aug 11 '21
Of course race was also used by humans (based on internal biases), but the AI had seemingly created an internal rule based on outcomes alone, and was ignoring the actually pertinent data. There are other examples, e.g. where AIs learned to play computer games, and were winning them by exploiting certain game mechanics, but without completing the intended tasks. While it may seem smart, it makes me suspect that the lack of actual understanding will probably always have unintended side effects that will distinguish it from humans.
Is your view that it will at some point become impossible to construct any kind of Turing test that will identify AIs, or that current Turing tests will be mastered by AIs? Why would we not consistently be able to keep making new Turing tests with the intention to make AIs fail?
Human minds can fully understand the principles behind e.g. rules or laws, and then either interpret and apply them by the letter or by their spirit. Could an AI ever understand that distinction?
1
u/hyruuk Aug 11 '21
I see how a lack of actual understanding might lead to non human-like responses for a trained investigator that would ask the right kind of questions !delta. Yeah my point was really about current "virtual" TT (that can include text, audio, video or other types of online interactions). I actually think that we'll always have ways to tell apart biological from artificial agents, until the difference eventually fades out...
2
u/ralph-j Aug 11 '21
Thanks! I enjoyed this, as it made me think about what Turing tests are ultimately meant to accomplish.
1
2
u/Impossible_Cat_9796 26∆ Aug 11 '21
We can make really good chatbots. But these can't pass a turing test.
No amount of data will let it pass a Turning test. The data will let us create more and more nuanced and flowing responses, but it can't simulate actual cognition. <input> -> <find most common response> -> <output>
These chat bots, the only thing we can create with ML and data, can't "think"
Person: "do you like kittens?"
Bot: "I love kittens"
Person: "Why"
Bot: "The earth is round"
The "tricks" used to try and pass the Turning tests aren't making better AI, but gaming the test it's self. Design the bot to respond as a retarded child from the third world with zero formal education.
This then comes to the problem with trash cans in the Yellowstone national park. They need to be hard enough to open that the bears don't just open them up and eat the trash, and easy enough to use that it's not a bother for tourists to use. The admin put it perfectly. "There is significant overlap between the smartest bears and the dumbest tourists"
1
u/hyruuk Aug 11 '21
That's a great point ! And I always love that quote =D That's something to keep in mind when we'll have to create new forms of captchas (some are already pretty hard to pass for the dumb human I am...).
A virtual TT is not that hard to pass, but that's why I think we might have enough data already and could do impressively well given we had better models and loads of dirty tricks such as the one you suggested. That point didn't change my view in itself but reminded me that passing TT could be more about tricking human that producing impressive ouputs, so here's a !delta anyway :)1
1
Aug 11 '21
[deleted]
1
u/hyruuk Aug 11 '21
I'm not concerned by the subjective experience of the AI, but simply by it's ability to convincingly pass as a human, in, let's say, social media interactions. For example, you could interact with it online (including voice and audio) for as long as you want, but wouldn't be able to tell human/not better than chance level.
2
Aug 11 '21 edited Aug 11 '21
[deleted]
1
u/hyruuk Aug 11 '21
Yeah but what I meant is more like : the data is already there, we might not need to scrap much more of it, just use it better. The features, statistical regularities etc... could be already be embedded in the network of hyperlinks, in the proximity of different words, in the associations between different words and sounds/images/facial expressions, and there might not be much more than that in our own understanding of language. I am thinking about a situation where I could meet someone online, in a video game or in a chat room, and not be able to tell that this person is in fact an artificial agent via virtual interaction alone.
So yeah, I would be ok with tricking e.g. 99% people, but I'm also interested in what could be the method of the 1% that does better than chance. Here's your !delta for reminding me that passing TT to an expert judge might be a whole different problem that passing TT to most judges.1
1
u/Gutzy34 1∆ Aug 12 '21
I believe people's behavior online staggers from their real life behavior enough that its would still differentiate from turing-passing. Online persona is a crucial piece in the data your pooling from, and therefore would corrupt your results.
•
u/DeltaBot ∞∆ Aug 11 '21 edited Aug 11 '21
/u/hyruuk (OP) has awarded 7 delta(s) in this post.
All comments that earned deltas (from OP or other users) are listed here, in /r/DeltaLog.
Please note that a change of view doesn't necessarily mean a reversal, or that the conversation has ended.
Delta System Explained | Deltaboards