There are a lot of great points here and I should mention, or at least emphasize what Kayak Karl highlighted about backpacking. If any of you are familiar with or at least interested in ultralight backpacking techniques, than finding a way to maximize weight loss and multi-use gear is a high-priority.
When weight or bulk are less of an issue, than having wearable sleeping insulation isn't really considered. Like Karl, I'll err on the side of ease and bring any number of things if I'm car camping, jacket included.
There are limitations and complications to be sure. Mobility in wet weather or if bushwhacking through thorns are issues to be aware of. Here are my plans for tackling some of the points raised so far.
FIRE - When I'm backpacking, I don't often build a fire. Sometimes it's just not a priority for my trip. YMMV. If I do build a fire, they are usually small, just to provide a little comfort and ambiance (I don't need it for warmth or cooking as my other systems are in place for that). I set these LNT fires a good distance from my hammock where I watch it in comfort. I don't want embers around any of my gear. I've rarely had problems with hot embers or ash burning my gear -- my fires are too small and I'm too far away.
I would speculate that even with a puffy jacket, if you're building a large enough fire where crackling wood logs spewed sparks and embers, you'd be in the same danger of burn holes as you would if you wore a wearable quilt. You would exercise the same caution, I would hope.
If I were relying on a campfire to keep me warm, I'd probably act like my Boy Scouts do around a fire: slow roast yourself by turning different sides of your body to face the fire. When you're done cooking yourself and it's time for bed, you rush and jump into your sleeping bag. All without wearing a jacket at all
HIKING IN COLD & WET - While down is warm, it's not the best idea to backpack or hike in a down jacket (or wearable quilt) because it is too easy to overheat, or worse, wetting out your insulation. As soon as i'm ready to start hiking, I stow my insulation and rely on the exercise to generate heat. I use a series of light layers that are extremely efficient in trapping warmth by blocking the wind and rain and helping me regulate my temperature.
Adding extra insulation typically happens when I'm back in camp, when my body is cooling off.
BUSHWHACKING & THORNS - Going back to my technique for hiking in wet and cold, I'm also not tackling prickly situations in my wearable sleeping gear or jacket while hiking. If "snagglies" were at camp, that might be an issue, but I think it is rare. I don't know many people who seek out briar patches as a suitable camp site. Tree branches and bushes are more common for me, but I haven't had a problem navigating them yet.
I think thorns and snags would be more common while hiking and less common in camp -- at least that is my own experience.
CAMPING IN COLD & WET - Let's say the weather has been nasty wet and cold. Sometimes when backpacking it's ideal to pitch a tarp along the trail and just wait out the storm. If at camp, often I just modify my technique. Cooking in the rain? I don't care who you are, you're probably finding a dry place, even if it is under you tarp and near your sleeping area. Maybe you packed a "patrol tarp" if you're hiking with a group so you have a shared kitchen area.
All of my rain jackets are large enough that I can layer insulation underneath, including a wearable quilt. Moving about when it's raining isn't a show stopper, but it is naturally limited if you're in camp unless you like "Singing In The Rain."
Again, I'm not advocating that wearable quilts are the ideal solution for everyone. They are a very viable option for those looking to lighten their pack and they're nice to have around when base camping for extra fluffy goodness, if you want it.
Lightweight backpacking typically focuses on systems and how those interrelate.