I do the "mount, then counterweights, then OTA dance" to marked tiles in the backyard. and rarely need to re-polar align unless I've been travelling and have changed the mount around. You can usually be up and running (imaging) around 45mins from deciding the weather is worthwhile (which is almost never....
). Packing it all away is 30mins or so - less if I'm lazy and put a cover over it all for the AM.
The way I look at it is how many sports or offsite hobbies can you do without a trip in your car somewhere for 30mins there and back? An observatory isn't really practical in my backyard (let alone my skies!), but an hr now and then putting up and away isn't that big a deal if you're enthusiastic about it. You can always start setting up before dark to save valuable night hours.
At the end of the day this is a personal thing as to your own "pain threshold". If it sounds like too much then a small mount and refractor might be a better option?