I'm really struggling to find Galaxies guys

I think i am looking in the right places, by using stars as reference.
I think that possibly the locations are too affected by light pollution, as they tend to be in the eastern sky and here in Wollongong we get the orange glow of Port kembla.
Although Sombrero should be NE enough i think.
So if it is not light pollution, am i looking wrong ?
Atm i'm trying to find 2 stars that the Galaxy falls between on finder scope, then i have a few practice runs by panning from one star to another using the FS.
Then i switch to a 15mm Plossl and Pan again, nice and slowly, but i've not seen anything that resembles a Galaxy
What i have seen is the faintest of smudges, like a slightly less black patch in the sky. If this is the galaxy, is light pollution my nemesis, am i imagining the artifact, or am i expecting too much and thats what i should be seeing ?
Stu