Hi Matt
We have these and huntsmen too, mainly in the garage and rumpus room (under the house) and they seem to keep our place clean and tidy from other insects, such as cockroaches, so we let them stay!
We call it biological pest control!
Cheers
Dennis