Welcome to the forums. Sounds like you have a good setup. I'm with Nathan H and think a closed cell matt would really help out. Another thing you may want to consider is a weather shield. The Jacks - R - Better weather shield or Dri-Ducks poncho would add another 5-10 degrees of warmth as well.
I would start with the closed cell matt 1st though.