Some people say you have to be soft in the head to want to climb peaks like Everest and K2. But a recent Italian study found it definitely works the other way, too: serious mountaineers who went over 4,000 meters (13,000 feet) several times a year showed decreased brain activity compared to non-climbers.
MRI scans and neuropsychological tests showed an affect on cognitive abilities like memory and motor functions. It's the altitude that does it, apparently.
Ah well. I've been drawn to mountaineering as I get older because a) the more I rock climb, the more scared of heights I get, and b) climbing mountains is something you can still do well in your 40s, as your body gets too ragged for more quick-pounding sports. (Although you don't want to push it too old.)