I'd take an eyeful of down over an eyeful of silica any day of the week. Silica is a very common dessicant. Just because it's included in my beef jerky doesn't mean I'm going to eat it.
The clothing manufacturer states that it is encapsulated - whatever that entails (whether it's simply a shell or the aerogel is sealed in some manner, as it has been in sleeping pads). Either way, I'd still be leery if the product happens to get punctured/ripped/torn. But thats just me.
Yep, it's expensive, most new/ limited tech is - not sure how that relates to it's safety. It's also been around since the early 80's and does show promising uses in a wide spectrum of applications. Perhaps they have overcome any limitations to make it usable in clothing - I just haven't seen any evidence of that beyond the manufacturers claims, and I'll take those with a very large grain of salt.