I think for me it depends on what class I'm playing. For ranged characters I think I find burning souls more annoying since they can send a flurry of lightning at you from off screen, and with more fragile builds like sorcs/necs/glass cannons, this can be dangerous, especially if your light res isn't awesome. Dolls are a little more manageable because you can keep a safe distance from them and let your ranged attacks and merc handle it.
For melee chars, dolls are just awful. Especially if you're mid-whirlwind and one or two decide to ambush you from around the corner and.... *pop* you're dead. Souls can also be a pain for the melee chars since you have to close the distance before you can attack them, and they can get in a lot of hits in that short period of time.