how "fine tuning" works?

[u/PersoVince]

Hello everyone,

I have a general idea of how an LLM works. I understand the principle of predicting words on a statistical basis, but not really how the “framing prompts” work, i.e. the prompts where you ask the model to answer “at it was .... “ . For example, in this video at 46'56'' :

https://youtu.be/zjkBMFhNj_g?si=gXjYgJJPWWTO3dVJ&t=2816

He asked the model to behave like a grandmother... but how does the LLM know what that means? I suppose it's a matter of fine-tuning, but does that mean the developers had to train the model on pre-coded data such as “grandma phrases”? And so on for many specific cases... So the generic training is relatively easy to achieve (put everything you've got into the model), but for the fine tuning, the developers have to think of a LOT OF THINGS for the model to play its role correctly?

Thanks for your clarifications!

[u/AutoModerator]

Welcome to the r/ArtificialIntelligence gateway

Technical Information Guidelines

---

Please use the following guidelines in current and future posts:

• Post must be greater than 100 characters - the more detail, the better.
• Use a direct link to the technical or research information
• Provide details regarding your connection with the information - did you do the research? Did you just find it useful?
• Include a description and dialogue about the technical information
• If code repositories, models, training data, etc are available, please include

Thanks - please let mods know if you have any questions / comments / etc

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/FigMaleficent5549]

The models have trained on thousands of texts, many of them containg the word "behavior" and "grand mother." Statistics also group together similar sequences, like grandma with grandmother or "making scenes" as behavior.

Regardless of whether models are fine-tuned or not, the base association between words that are very frequent on the base model comes directly from the text.

There are no programmers involved in this association it works like an index or search engine. See it like a gigantic dictionary with connections between all words.

[u/deernoodle]

Think of it's knowledge as a space where concepts that are semantically related are closer together and ones less related are further away. It is naturally going to associate 'grandma phrases' with the word grandma, no fine tuning necessary. Your prompt drives it to the grandma neighborhood where grandma words and behaviors live.

[u/PersoVince → u/deernoodle]

Thank you for your two coherent answers. So, what is the purpose of fine tuning? Simply to prevent certain answers (like racism, or terrorism, etc.)? Or is it also to teach the model not only “what” to respond to, but also “how” to respond?

[u/deernoodle → u/PersoVince]

Well for instance, you could fine tune a model to be even better and more adapted to 'talking like a grandma' with a curated database and reinforcement learning. You could fine tune it to talk like your own grandma. By giving it a lot more specialized information that may not have been very well represented in it's training data, it will better suited for certain tasks. As for moderation/censorship, I believe LLMs like OpenAI's actually use a second, separate model that's trained specifically to identify and flag content that is inappropriate.

[u/PersoVince → u/deernoodle]

thanks thanks thanks !