A drone needs a camera to manually navigate the thing whilst out of line-of-sight.
And there is your problem. Operating it out of line-of-sight. Perhaps the Swedish ruling is based on the nature of constitutes acceptable use, and out of line-of-sight does not meet their criteria.
As to the argument about acceptable use versus unacceptable use, while the courts and legislatures often do side with if something has an acceptable use then it won't be banned for having an unacceptable use, there are exceptions, and those exceptions are often based on the nature of the unacceptable use, and how widespread that unacceptable use is compared to otherwise. Often that kind of consideration is based on how the unacceptable use affects other people.