I think that the technology just isn’t there for most generated dialog.
What we’re doing today is taking a training corpus and then directly, without higher-level processing, producing more text like it, given a prompt.
What limitations exist here?
Written and spoken language are not the same, and a lot of training data is from written language. In English, written sentences are longer than spoken ones. People use some different words and grammatical structures. Try reading a play or transcribing what someone says, and it kind of drives the point home. That’s not an unsolvable problem, but it’s a good argument that gluing something like ChatGPT to a speech synthesizer is a long way from where you want to be.
You need a training corpus similar to the way a given character would speak. Maybe if you want “a generic American”, you’re okay. And there’s legitimate uses for that in games, certainly. But what if you want to have Celechir, high-elven guard in the kingdom of Arandie? How do you build up a list of things that Celechir would talk about? How much germane training data is there out there?
There is no strong association between text and game world state, which is normally desirable. Let’s say that I’m creating a character in Fallout 4. How am I going to get them to talk about the world around them? I can encode some world state and state about that character in a prompt, but that’s sharply-bounded using existing mechanisms – I can provide maybe a couple of hundred prompt terms, which is not a lot to try to describe the world and relevant characters and all that. I’d guess that any such generation mechanism is going to require some level of pre-processing as to world state that doesn’t exist today.
I am all for using generative AI to do speech synth. I’ve been impressed with output there. We may not be quite to the point of good, emotive speech yet, but we’re good-enough for a lot of uses, and it lets one do things that cannot be done with pre-recorded, static samples from a voice actor, like dynamically-generated text.
But for writing dialog via generative AI? I’m a lot more hesitant there in the near future, given what I’ve seen so far.
Now, I am sure that you can make video games in certain limited genres that do leverage what’s there. But I think that it’s far enough from a drop-in replacement for hand-written text that it’s not a great option. Maybe you can make a so-so sexy chatbot or something like that that’s isolated from a broader video game world. Maybe you can create characters that speak in fairly-constrained ways. But I don’t think that we can just create NPCs on par with human-written-dialog characters via gluing ChatGPT to them and providing a handful of human-language directives about how the character should act, the way we could for a human writer, which I think is what some people are dreaming of. Further down the line, maybe, but I think that it’s still a fair way from where we are in 2024.
I dont think LLMs will or should replace properly written dialogue.
Where they would shine is just generating inane background chatter. So instead of hearing an NPC say “i took an arrow to the knee”, or “jesus christ be praised! Henry’s come to see us!” 300 times, an LLM could generate some short one liners that are a bit more dynamic. It would go a long way to making the world feel more alive.
I don’t think it’ll solve the problem. Ask anyone in the sillytavern subreddit and they’ll tell you LLMs tend to repeat the same dialogue a lot (look up the “shivers up/down their spine” meme)
Edit: since it might not be obvious, here’s an example of people who use LLMs for character dialogue’s opinion on the content being produced: (Link Warning: reddit)
I bet that it’s not too expensive to write a lot of human-written dialog, but that hiring a bunch of voice actors to act out minor lines – especially if a given character has only a few lines – it is probably the more-expensive bit. Like, I think that a human dialog writer could probably affordably put together enough dialog that a player wouldn’t really exhaust it, but that you’d want to make any synthesis of the lines not have a lot of extra cost.
New text to speech models are incredible these days…
Again, we shouldnt replace actual voice actors for main dialog. But for generating thousands of lines of background chatter (which nobody would have time or resources to make anyway) LLM writing paired with text to speech could really help flesh out a living game world
I think that the technology just isn’t there for most generated dialog.
What we’re doing today is taking a training corpus and then directly, without higher-level processing, producing more text like it, given a prompt.
What limitations exist here?
Written and spoken language are not the same, and a lot of training data is from written language. In English, written sentences are longer than spoken ones. People use some different words and grammatical structures. Try reading a play or transcribing what someone says, and it kind of drives the point home. That’s not an unsolvable problem, but it’s a good argument that gluing something like ChatGPT to a speech synthesizer is a long way from where you want to be.
You need a training corpus similar to the way a given character would speak. Maybe if you want “a generic American”, you’re okay. And there’s legitimate uses for that in games, certainly. But what if you want to have Celechir, high-elven guard in the kingdom of Arandie? How do you build up a list of things that Celechir would talk about? How much germane training data is there out there?
There is no strong association between text and game world state, which is normally desirable. Let’s say that I’m creating a character in Fallout 4. How am I going to get them to talk about the world around them? I can encode some world state and state about that character in a prompt, but that’s sharply-bounded using existing mechanisms – I can provide maybe a couple of hundred prompt terms, which is not a lot to try to describe the world and relevant characters and all that. I’d guess that any such generation mechanism is going to require some level of pre-processing as to world state that doesn’t exist today.
I am all for using generative AI to do speech synth. I’ve been impressed with output there. We may not be quite to the point of good, emotive speech yet, but we’re good-enough for a lot of uses, and it lets one do things that cannot be done with pre-recorded, static samples from a voice actor, like dynamically-generated text.
But for writing dialog via generative AI? I’m a lot more hesitant there in the near future, given what I’ve seen so far.
Now, I am sure that you can make video games in certain limited genres that do leverage what’s there. But I think that it’s far enough from a drop-in replacement for hand-written text that it’s not a great option. Maybe you can make a so-so sexy chatbot or something like that that’s isolated from a broader video game world. Maybe you can create characters that speak in fairly-constrained ways. But I don’t think that we can just create NPCs on par with human-written-dialog characters via gluing ChatGPT to them and providing a handful of human-language directives about how the character should act, the way we could for a human writer, which I think is what some people are dreaming of. Further down the line, maybe, but I think that it’s still a fair way from where we are in 2024.
I dont think LLMs will or should replace properly written dialogue.
Where they would shine is just generating inane background chatter. So instead of hearing an NPC say “i took an arrow to the knee”, or “jesus christ be praised! Henry’s come to see us!” 300 times, an LLM could generate some short one liners that are a bit more dynamic. It would go a long way to making the world feel more alive.
I don’t think it’ll solve the problem. Ask anyone in the sillytavern subreddit and they’ll tell you LLMs tend to repeat the same dialogue a lot (look up the “shivers up/down their spine” meme)
Edit: since it might not be obvious, here’s an example of people who use LLMs for character dialogue’s opinion on the content being produced: (Link Warning: reddit)
https://www.reddit.com/r/SillyTavernAI/comments/1div11q/sends_shivers_down_your_fuing_spine/
That’s a thought.
considers
I still think that the limiting factor there is more one of speech synth than writing dialog. Like, “arrow to the knee” is Skyrim, right?
kagis
Yeah. And those were voiced.
Similarly, you had Fallout: New Vegas with stuff like “patrolling the Mojave almost makes you wish for a nuclear winter”.
I bet that it’s not too expensive to write a lot of human-written dialog, but that hiring a bunch of voice actors to act out minor lines – especially if a given character has only a few lines – it is probably the more-expensive bit. Like, I think that a human dialog writer could probably affordably put together enough dialog that a player wouldn’t really exhaust it, but that you’d want to make any synthesis of the lines not have a lot of extra cost.
New text to speech models are incredible these days…
Again, we shouldnt replace actual voice actors for main dialog. But for generating thousands of lines of background chatter (which nobody would have time or resources to make anyway) LLM writing paired with text to speech could really help flesh out a living game world