Silly question....since you'll empty out the panniers in camp can't you just put your riding gear back in there? I'm guessing maybe it doesn't fold sufficiently to fit?
Another storage option would be a "gear hammock" such as this one:
There are several different ways to rig it depending on your needs. It makes a great pack cover too (for you maybe a seat cover). I really think a chunk of silnylon will be similar in weight but much more multi-purpose than a mesh hammock. YMMV...