Stay away from the Jersey fabrics. They are knits and not suitable for weight bearing applications, unless of course you want to end up on the ground from the stretch. Polyester or nylon fabric untreated is what most people end up using. Ripstop or taffeta are both reasonable choices. My personal preference is polyester taffeta.
Neoprene sounds a little kinky but I doubt it would be a successful choice if for no other reason and bulk.