And what happens when you go to any website with copyrighted content? You download it, making a copy, adding it to ram, making a copy, adding it to your web cache, making a copy. Yet no one considers any of those unlicensed acts copyright infringement.
If the content in question were kiddie porn, though, a majority would say the user is responsible for the user-agent's behavior. So while you didn't infringe the child pornographer's copyright, you did commit some sort of child pornography offense.
What this tells me, is that users are responsible for the download, but the infringing download has a Fair Use defense.
But that presumes the copy on the server wasn't made via infringement. But the user doesn't know that. So maybe it's Fair Use if the user believes it's Fair Use ("if you didn't want people to download your article, you shouldn't have published it on your web server!"), but infringement if the user believes it's not ("if you didn't want people to download your article, you shouldn't have let pirates put it on their web serv-- oh shit, you've got me!"). Very mens rea-ish.