Worst Linux Annoyances?
My annoyance is not with X it is with the XFree86 implementation, which is a continual source of frustration for many people.
Yes, I do love the remote display capabilities and I've used them several times. I don't know what I'd do without them.
And x2x is really sweet! Control 2 (or more) desktops with a single mouse and keyboard without annoying KVM switches.
However, that said, XFree86 is a pain, should have better hardware configuration and autodetection, and is by far the crashiest part of my Linux and *BSD experience. It is, as far as I know, the only piece of software that can bring a Linux box to its knees, requiring you to ssh into it to safely reboot it.