I think a person's mentality has a lot to do with it. As a woman, genetically I know I will never be as hot as my hubby but at the same time I also seem to get by with less clothes than a lot of my female friends.
A lot if that has to do with understanding that as it gets colder after a long hot summer it is easy to start bundling up at 60 degrees since it is so much colder than it had been. But if you brave it and force yourself to deal with it by wearing shorts and t shirts well into the 50's then your body slowly gets used to it and you're just throwing on a light pullover in the 40's instead of acting like an ice age has occurred.
You'll never get over basic genetics or the conditioning of the place where you were raised, things like that will always have an effect, but you can use mental power to overcome a lot.