The damage to the brain is related to how quickly the skull stops moving while the brain inside it continues on until it smashes violently into the inside of the skull. The role of a helmet is to give the skull a 2-3 cm cushion to gradually slow it down, rather than have it come to a near-instantaneous stop. So the brain ends up making a more gentle impact with the skull.
In the case of a bike helmet, the foam should crumple, bringing the skull to a more gradual stop, in much the same way as cars are designed to crumple, thus exposing the occupants to a lesser rate of deceleration. But that means there is a trade-off between bulkiness and protection.
In the case of some players (ones like Leigh Matthews), a layer of padding could be applied to their shoulders, hips, and elbows, to lessen the damage they can do.