Indeed, it would be wise to do so. Depending on how you mount the console, it may or may not be a pain in the ass to detach and reattach.
Real world example:
In this case, the console is simply placed in the glove box, no adhesives, so it's relatively simple to disconnect cables and remove...