I can only speak for myself but this is my experience:
I was doing very well in maths all the way through school and highschool, and always wanted to study astronomy at university. I began uni immediately after highschool but found that the maths required was simply killing my joy. I didn't have the necessary flair for it at such high level and/or the patience to learn and memorize countless mathematical proofs which we had to do in order to pass exams. And the astronomy content was minimal until you reached year 3-4. Maybe it was due to the way it was taught but I just found it dissappointing overall, especially because this was what I'd always dreamt of doing.
We had a simple introductory astronomy course during the first year, and I vividly recall one day when we had all recently been struggling with quantum mechanics (which was the most dreaded and feared subject): Our astronomy lecturer's response was "Oh quantum mechanics, I've never had any use for that". That day I realised i was wasting my time
So I left uni and pursued computer programming instead.
A few of those I studied with have gone on to become quite successful astronomers, one received the Hubble Postdoctoral Fellowship and works at Caltech now. I recall him also as being very very bright. Another one who cruised through it was Niels Bohr's great grandchild! I think all in all maybe 5% of those who started that year actually completed the studies and became actual scientists.
In the end I'm actually happy that I didn't complete the studies because my impression of the profession is that it involves a lot of paperwork and competing for funding etc. Now I know that I'd much rather have the scope in my backyard and be able to do with it what I want.
As I said, it's only my experience, and yours may be different. But if you have trouble with maths now then I don't think it'll be easier when you study astrophysics so you'd have to be prepared to put in some hard work.
Regards,
Rolf