I'd ponder that you'd have to be really good at mathematics, and look into what are the major fields of research going on at present and examine your value proposition of which endeavour you'd like to encompass and define how you add value to it.
Maybe write some e-mails to directors at several large observatories explaining your situation and asking their advice of how to get into the field. Stroke their egos and ask how they did it, what advice they can give you, what is most in demand now, what gets you in the front door (is it being on some famous astrophysict's coat tails as their research student) etc...
Just keep persevering is I suspect the secret!
|