I say hike at a pace you feel comfortable with. If you are breathing hard, then slow down. If you are sweating profusely, then slow down. If on the other hand you're getting passed by grandma on her rascal, then speed up.
If you are wanting to hike to exercise and loose weight, then hike at a vigorous pace that lets you still talk normally. If you are wanting to hike to just get out in the woods, then go at a pace you are comfortable with.
My hiking speed is around 3.5 mph on the mostly flat trails in my area, which is much faster than my typical strolling around town pace. Something about being in the woods invigorates me. Now if I could quit stopping to look at every little thing I could kill some serious mileage.