Hard to say what the issue might be. Barlow, EP, scope collimation or conditions? Collimation is something you need to learn how to do and get confident about. It's not hard but can be confusing at first. Follow good instructions, e.g., as described in Orion branded Dob user manuals (see
www.telescope.com) or look up Sky and Telescope's 'how to collimate a Newtonian' article. Simple Cheshire style visual collimating tool is the most reliable. Note that in collimating a Newtonian only the final step (adjusting the tilt of the primary) is critical. The rest (centring and orienting the secondary) are fairly forgiving and need to be done only once a year or less often. When you read collimation instructions they can seem daunting but once you learn it's quick and easy.
But what you're describing does not sound like a collimation problem: that would only lead to not being able to focus, and usually asymmetry in the best focussed or slightly defocussed images. It's easier to see this when looking at stars.
The rainbow colours you describe sound like atmospheric refraction when the subject is low in the sky. But if you're getting it on both Mars and Jup, assuming you're observing them at around the same time, that's unlikely.
So perhaps it's your EP and/or barlow. What are you using? And what make is your scope?
Re what's best for your scope on planets (assuming it's an f/4.7-5), as far as magnification goes, aim for ~150x as a first look-see (and on poor-seeing nights that may be the best you can do), ~200x on average nights, ~250x on good nights, ~300 on excellent nights, and up to ~400x higher on rare exceptional nights. It's hard to recommend specific EPs without knowing your budget and observing habits.
Most fully multi-coated Plossls and barlows will do a decent job and tend to be the best performers on the cheap. Longer barlows tend to perform better than "shorties" - especially per dollar and in fast scopes like yours. At ~10mm and under the eye relief of Plossls gets a bit too close for comfort for most people so it's probably best to use your barlow for ~120x and over. You can increase the magnification of any barlow with a barrel extension tube.
