Finite State Machine and Behavior Tree are often used in game AI. In game, designer asks expected actions to give users expected experience. So, generic offerings like machine learning is somewhat useless because it lacks control.
But there IS a difficulty in decision making. What action to take at when? There're many possibilities.
State --> Action --> Path Finding, Movement, etc.
A1 A2 A3 A4
PC
S1 S2 S3 S4 S5 S6 S7 S8
a1 a2 a3 a4
NPC
s1 s2 s3 s4 s5 s6 s7 s8
Action: A, a
State: S, s
* current in bold
Actions may be ordered by behavior trees.
States may be directed.
In FSM, there's only one state at a time, and is designed as directed graph.
So, in this case,
Decision Making is done as "Pick up an action or a chain of actions based on current states". And what leads to better decision making?
In FSM, for example in FPS, state is simple. In combat or not. Whenever NPC become aware of PC, it gets in combat mode to kill you. But the action could be complicated. Which path to get closer? With obstacles to hide? Can work as a team? Evaluation function? More variables gets you less control. If NPC is damn zombie, sending more zombies with damn pathfinder makes sense. But if it is not, it seems another layer of state machine is required. And, that would require another, and another, and another...
It may not help, but think about ourselves. We, human, does this in three stages.
-Define Goal
Often, defining goal is not easy. It is not plain, but hierarchical. In team, we share it.
-Observe
Collect information to describe the current situation towards the goal.
-Analyse
Analyse collected information, and list possible actions.
Wrong analysis on less information or wrong information likely leads to failure.
This happens also in game. Information given is limited both in its amount and its realtime ness due to system resource. Even if information is enough, the situation may not be obvious.
But a game is basically zero-sum game. I mean, there is an opponent. So if you know what the opponent does, you can choose a winning option. But if the opponent knows it, it can choose a counter solution. And if, and if, and if...
Now it seems we are in an infinite loop on a finite state machine. This may represent a common argue, "WE NEED MORE PROCESSING POWER". Seriously?
A sniper will put most of its efforts on collecting information. A soldier on actions to give more damages. If there is a diversity in NPC abilities, which fates its character, their fights to take their advantages to achieve their goals become a game. Doesn't it become an expectation of what your team mates do? Like, one tend to stay back and cover, another to go first and lead. That kind of character makes game play a lot easier to work with and more fun to fight against NPCs.
A character, an expectation, to count on others, simplifies decision making. PC reveals its character, too, through actions made in the past, which makes team play easier, and prevent bad users from the community to keep game experience.