In fact the actual three phase motors mostly have only three coil connections. Thus, they cannot be driven with a two phase driver like TMC239, which always need both ends of a single coil in order to drive it based on a constant current chopper. The TMC454 plus TMC239 can only drive three phase stepper motors in microstepping, which provide 6 connections and have no internal interconnection of the coils.
A star or triangle connection three phase stepper motor has special driving scheme requirements, which also differ slightly from a BLDC motor, but it can provide superior microstepping exactness. We can propose you the more modern solution TMC457 as high end motion controller plus TMC332 as motor driver controller. For test purpose, the TMC332-EVAL evaluation board will give you the possiblity to evaluate motion behaviour of a three phase motor controlled by the TMC332. It also allows evalution of encoder signals. The board can also be coupled to the TMC457-EVAL using step and direction interface.