Here's another thought ..
The main thing which differentiates normal matter from dark matter is electromagnetic interaction, yes ?
So why couldn't the added effects of the Electroweak and Electrostrong forces result in normal matter coming together more quickly than the DM clumps ?
Once the normal matter starts forming its 'aggregates', they then slowly begin to interact (gravitationally .. as the aggregates become significant), with the DM, which then passes straight through the aggregates on its way to another clump of DM ? (The process goes on over and over …??…).
What we see in the Bullet Cluster is that there is a difference in dynamic behaviours when comparing DM and normal matter.
Ah dunno …
… I just made all that up !! …
Mind you, there's a lot of research going into the behaviours of dusty matter and how it comes together .. and I think EM forces play a big role in the very early stages of it all (??)
Cheers