I have no issues with computer games, its just another form of entertainment. If anything its probably a bit rich of people criticizing it, when astronomy could be seen in the same context.
You mention astronomy to most people and they'll think its interesting, but you start crapping on about eyepieces, scopes, NGC...and people will roll their eyes.
Thats my 2 cents worth
Moderation is the key.