From my Mechanics of Materials studies....
The shear strength of a grade 8 bolt is in fact higher than that of a grade 5 bolt. Asa general rule, shear strength is typically 60% of the ultimate strength in tension. Since Grade 5 bolts have an 85kpsi tensile strength and Grade 8's are rated at 130kpsi, obviously the Grade 8's have more shear strength (20% greater).
The brittleness argument is one that just doesn't seem to go away. But what is brittleness? Brittleness is a failure at a load below that of at which the object would normally be permanently changed (typically in diameter). Grade 8 bolts exhibit no higher failure of this kind than Grade 5. So what makes a Grade 8 bolt different? Its hardness. Hardness and brittleness are not the same. Hard means it won't bend as easily.....brittle means it has an inexplicable failure.
Ok..that much is settled now. So where does the myth stem from? Well....a Grade 5 bolt is far more likely than a Grade 8 to bend before failure. This is due to the hardness. However, under no circumstances will the Grade 5 bend without failing while the Grade 8 breaks and fails. Case closed.
So...why use Grade 5 bolts? There are several reasons, the first being cost. Grade 8 DOES cost more...significantly in some cases. The second reason is that you just plain don't need the strength of the grade 8. Trailer hitches are a good example of this. The final reason I would use a Grade 5 instead of a Grade 8 is that I WANT the bolt to fail first. It's sort of a shear pin situation, not unlike guys who upgrade ever aspect of an axle except the u-joints because u-joints are cheap...I would rather break a u-joint than an alloy axle if something absolutely has to give.
If you're worried about strength, remember one thing....torque those suckers down! Bolts are designed as clamping devices, not shear devices. If you clamp the hitch to the frame well, the forces will first have to overcome the static (and kinetic eventually) coefficient of friction and the forces resulting from that before the bolt is actually subjected to the shear load. In the case of a hitch where the bolts are in a vertical position, they will ultimately only be subjected to a tension load from clamping the hitch to the frame. The shear load is not passed onto the bolts because of the friction. Grade 8 bolts will have the distinct advantage in this use because they can be subjected to higher tensile loads. So why aren't they used with hitches? Because they aren't needed! The forces aren't such that it would be an issue. The hitch ball in its single shear configuration or the ball mount itself is far more likely to fail.
Thank you, Society of Automotive Engineers.