You might want to try using POWERSTRIP to run the custom resolution, I hear its great with getting odd resolutions running with old video cards .
http://www.entechtaiwan.com/util/ps.shtm
By the sounds its syncing vertically and not horizontally: Powerstrip will allow you to force a horizontal sync rate but the Nvidea drivers won't.