From the VISA Website "If your card is lost or stolen you should notify your bank as soon as possible. If anyone has fraudulently used your contactless card to make a payment, providing you take reasonable precautions to protect your card and let your bank know as soon as you realise it’s gone, you will not be responsible for any losses incurred (subject to your bank’s terms and conditions)."
In other words you are not responsible for any losses. Provided you report it lost as soon as you realise.
They are very low value for any fraudster - the best they can hope is a few contactless transactions and probably each one runs the risk of being caught (and on CCTV).
VISA seem prepared to take the hit - so what's the issue?