Indeed it's not fair, and my back protests regularly in agreement.
It's equaly frustrating with time as well.
When you're working, money is flowing, so dropping a few thousand (or more) on mount, OTA and accessories seems reasonable ... but you don't have time to use it because you've got to get up for work most days.
If you're not working, having a late night (whenever the sky decides to be clear, that is) is easy ... but there's little if any money coming in, and you can't afford to splurge on gear.
I think astronomy requires:
(1) a modest approach early on and excellent saving discipline so you can afford decent gear in retirement; or
(2) a rich and understanding spouse who will support you; or
(3) winning lotto
... as well as an ongoing regimen of weight training.