The loss of life, bodily integrity, and personal possession are reasons why your listed crimes are harmful. Their causes are the immediate physical actions that precipitate their loss. In contrast, speech precipitates no loss and no harm, and you only deem it "harmful" because they merely have the potential, down the line, to motivate or to lower the mental obstacles for actions that deprive life, bodily integrity, or personal possession. Your view of "harm" is suddenly made so expansive that it would force us to conclude that, for example, socialist slogans and ideas are forms of hate speech in the sense that they have the potential -- proven through historical precedent -- to motivate actions that deprive life, bodily integrity, and personal possession.
Ultimately, your argument would like us to take extra steps up the chain of causality to ban things that aren't directly related to harm. How far up the chain of causality can we really go, or should we go? 2 or 3 steps seem just as arbitrary a demarcation as 20 or 40 steps. If a butterfly flaps its wings and down the line someone is killed, must we then ban the butterfly from flapping its wings?