I think it is better to go thinner to save weight rather than smaller-- less fill. You have get some gains by the amount of surface area as well as thickness, although thickness prevails in real cold. My thought is if you are insulating for say, 40F and raining, an over all cover and thin insulating layer will cut wind and convection over the majority of the hammock. With my poncho under cover, you get a "free" under cover weight wise, because I will always have some sort of rain gear. A middling light rain shell is 12oz, to the poncho is a weight bargain.
Then it is getting a light, compressible and effective layer of insulation. All I'm trying to do there is to isolate the bottom of the hammock from circulating cold air that absorbs the heat off my backside and cools the air I have heated inside the hammock. It needs to be of a material that doesn't transfer heat efficiently or in the case of the space blanket, might even reflect some back to me. The system needs to keep it close to the hammock, but I don't think it needs to be snug-- you get that nice dead air space between the hammock and the space blanket. I could be wrong. I'm counting on the fact that the outer cover is keeping the air inside fairly still too.
The space blanket is fairly fragile, so compressing it for travel is the weak part. It isn't hard to roll it up and with 1/2" of polyfill inside, it can be rolled to Nalgene size or close without really crunching on it. A rubber band keeps it rolled. That is where the AMK Thermal Bivy comes in-- it is windproof and it have some reflective properties, but less than the space blanket. It is more on the quick and dirty side, but it can be manhandled more than the space blanket and still provides a good thermal barrier so heat doesn't pass through the shell and the insulated dead air inside. You could knock out a whipped-end outer cover from ripstop nylon and quilt part of it or all of it. You could have Velcro strips inside and add "pillows" of insulation in light covers to change length/thickness and weight to your needs.
As to the double layer hammock problem, I think you need a differential cut, with the inner weight-bearing hammock just a little smaller. That outer layer just needs to hold the insulation and it's own weight. Quilting thought the works goes back to my "hanging sleeping bag" question. I think that is the way to really deal with the whole "mountain hammock" issue. It would be easy to alter the top insulation to suit the season. So take a typical gathered end hammock with a matching outer shell that is baggy, add a couple layers of Primaloft, sew the sides together and run a few lines of stitching across to lock the batting in place--- spaced what, 12" apart? You need to be able to launder it, so I would go with a separate bug net. I like carabiners in channels for suspension and that would make laundering easier too. The outer layer should be windproof and have some DWR properties. The inner layer needs to handle the weight and stress. It doesn't sound that hard
If you are dealing with sub-zero stuff, it isn't raining. Hopefully your shelter has more coverage, taking care of most of the wind issues. Then it is just raw insulating value and it's time for the lofty under quilts.