I'm sure you can connector 32 ohm drivers directly to CSR8645's pins SPKR_LP/LN and SPKR_RP/RN. The reason to use 32 ohm drivers is the current driving capability of CSR8645 is not high enough to driver low impedance drivers such as 4 or 8 ohm.
You may notice all highly integrated IC use differential signal within the IC, both in RF section and audio section. Differential signal pair has high rejection to common mode noise.