How Do We Do It Better?
The Bioware approach can only be as rich as the development work put into plotting out any given conversation. Each conversation and all of its possible options have to be fully scripted.
But in the kinds of genres games typically explore, a lot of dialogue doesn't need to be that rich. Tough-guy quips, snarled threats, or urgent requests for aid are commonplace and often don't even need to be contextualized.
Let's not try to simulate a conversation or pretend we're going to return to the full-text days of Infocom/Eliza. Instead, think about it from a problem-solving, dramatic approach.
Each NPC is a problem. When you encounter them, they are not doing what you want them to be doing. Your goal is to persuade/coerce them into compliance, with a risk that they may instead defy you. You decide whether the risk is worth the reward.
This suggests that a given NPC has three states: default, compliance, and defiance. We can imagine these as points on a continuum, with "default" in the middle. Your job is to move the NPC from default to compliance. (In fact, I ripped that off from EA's first Godfather game, where you could physically intimidate people into paying you protection money. The more you pushed them, the more money they'd give you, but if you pushed them too far they'd flip out and attack. It was a fun, quick, and universal mechanic.)
So now let's rip off Fable II.
In Fable II, you have a large library of physical gestures you can make that are divided into different categories, such as Social, Humorous, and Romantic. If an NPC likes humor, their opinion of you improves if you dance like a chicken. If they find you attractive, they are turned on by displays of strength or flirtation.
Let's take from Fable II the idea that you have groupings of expression forms. And let's say there are only two: Persuasion and Coercion. Persuasion generally means you're appealing to a sense of duty or morality or responsibility. Coercion generally means you're using intimidation or bribery. (Please note that I don't consider these as good/evil mappings. Both approaches should have their uses.)
Within each of those two forms, let's say there's ten different expressions. Unlike the gestures of Fable II, these are actual dialogue. On screen, they're presented as Bribe, Threaten, Implore, etc., Each one has a set of stock phrases that are randomly chosen each time, although context can override these with custom phrases.
When you encounter an NPC, that NPC may have one or several different interactions available. Typical interactions available with incidental NPCs, such as a farmer in his field, might include Information, Combat Assistance, and Healing. These basic interactions would be duplicated on many NPCs and would be available in the appropriate context.
You choose an interaction type and then begin choosing expressions. Not all expressions are available for all NPCs. Behind the scenes, the NPC corresponds to one of many personality types. Each type is defined by a list of allowable expressions paired with modifiers to the NPC's state on the compliance/default/defiance continuum. In addition, each type has a bias modifier for each type of interaction. Most personality types, for example, may be easily shifted towards Compliance when the interaction type is Information, but are easily shifted towards Defiance when the interaction type is Combat Assistance.
A really simple type, such as a Town Guard, may only have two interactions: Information and Combat Assistance. Because of his role, he is biased towards compliance in both. He's not meant to be a very interesting character so he probably only has a couple of allowable expression types.