Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Massive Email Crash Hits Canadian ISP Shaw

timothy posted about a year and a half ago | from the honey-did-you-bring-in-the-spam? dept.

Canada 150

rueger writes "One of Canada's biggest cable/Internet providers has their customers in an outrage. '... after an interruption of Shaw's email services Thursday led to millions of emails being deleted ... About 70 per cent of Shaw's email customers were affected when the company was troubleshooting an unrelated email delay problem and an attempted solution caused incoming emails to be deleted ... Emails were deleted for a 10-hour period between 7:45 a.m. and 6:15 p.m. Thursday, although customers did not learn about the problem until Friday, and only then by calling customer service or accessing an online forum for Shaw Internet subscribers.' To top it off, when Shaw did send out notices about this, they looked so much like every day phishing spam that many people deleted them unread."

cancel ×

150 comments

Sorry! There are no comments related to the filter you selected.

Hire the cheapest fucking help you can (0)

Anonymous Coward | about a year and a half ago | (#43134979)

Not that Shaw is particularly noted for this... bust just sayin' in general.

Re:Hire the cheapest fucking help you can (-1)

Anonymous Coward | about a year and a half ago | (#43135061)

I just wanted to say: NIGGERS!!

Re:Hire the cheapest fucking help you can (-1)

Anonymous Coward | about a year and a half ago | (#43135233)

Awwww. poor baby - you're suffering from low libido? Poor self esteem? The women won't look at you without laughing? Some skinny black boy stole your fat heifer? You didn't get enough breast feeding as an infant? And, it's all the nigger's fault? Poor, poor baby. You should just go down to a gay bar, and let one of those big black guys help you to feel better. A big black buck is just what you need. There's no need to be jealous, he'll share his big black dick with you. There's a gay bar near you, get moving or some other little queer bait will take your big black dick home!

Re:Hire the cheapest fucking help you can (-1)

Anonymous Coward | about a year and a half ago | (#43135349)

Vote BigBlackDick for president!

Re:Hire the cheapest fucking help you can (-1)

Anonymous Coward | about a year and a half ago | (#43135481)

FEMALE CREATURES!?

Re:Hire the cheapest fucking help you can (1)

Joe_Dragon (2206452) | about a year and a half ago | (#43136725)

This is wheel of fortune

Sometimes it's better to copy and forward... (0)

girlinatrainingbra (2738457) | about a year and a half ago | (#43134983)

I've got mail through one account that also automatically forwards to another account (not gmail or google, thanky god), so even if one provider loses my email dataset, the other still has a good copy. I also pop my mail in to my own machine, so I 've got a local archive. I wonder what the details of this canadian mishap really are... :>(

Re:Sometimes it's better to copy and forward... (4, Insightful)

Anonymous Coward | about a year and a half ago | (#43135049)

Except when the failing account is that forwarder account.

Re:Sometimes it's better to copy and forward... (1)

girlinatrainingbra (2738457) | about a year and a half ago | (#43135073)

oops, you're right about that. If the first hop fucks up, it's game over. :>( I guess I've been lucky that my school hasn't failed yet.

Re:Sometimes it's better to copy and forward... (-1)

Anonymous Coward | about a year and a half ago | (#43135459)

My diamond-hard cock wants to LIVE inside your asshole and donate a PLETHORA of sticky goodies to the deepest reaches of your rectum. What say you?

Re:Sometimes it's better to copy and forward... (-1)

Anonymous Coward | about a year and a half ago | (#43135555)

You took my advice, and went to the porn shop for a dildo? Good move. That little 2 1/4 inch thing between your legs wasn't going to catch any girls, in training bras or otherwise. Of course, you'll do better to wait until they no longer need training bras. Pedophilia is frowned upon in most countries.

Re:Sometimes it's better to copy and forward... (0)

Anonymous Coward | about a year and a half ago | (#43135939)

oops, you're right about that. If the first hop fucks up, it's game over. :>( I guess I've been lucky that my school hasn't failed yet.

... that you know of.

Re:Sometimes it's better to copy and forward... (4, Informative)

andymadigan (792996) | about a year and a half ago | (#43135109)

The details are that the messages were never delivered in the first place, your setup would not protect against such a problem.

Re:Sometimes it's better to copy and forward... (-1)

Anonymous Coward | about a year and a half ago | (#43135525)

I want to transform your pure, unviolated ass into a cummy mess!

Re:Sometimes it's better to copy and forward... (2)

Sipper (462582) | about a year and a half ago | (#43135551)

The details are that the messages were never delivered in the first place, your setup would not protect against such a problem.

That's true.

How bad this is depends on the system -- in this case it sounds like Shaw was doing "accept, then drop" which is the worst case because no one is notified of the failure. If however Shaw rejected spam rather than accept it, the sending mail system would notify the sender that the message was not delivered. It should be noted that this latter solution also does not cause backscatter because it doesn't generate a bounce. [For a bounce to occur, the message first needs to be accepted, but then for some reason cannot be delivered.]

Re:Sometimes it's better to copy and forward... (1)

gweihir (88907) | about a year and a half ago | (#43135141)

I do much the same with my own primary and secondary on hosted vservers and forward to my home-machine (via dyn-dns). So far, no loss.

Re:Sometimes it's better to copy and forward... (2)

ciotog (1098035) | about a year and a half ago | (#43135357)

How do you know?

Re:Sometimes it's better to copy and forward... (1)

gweihir (88907) | about a year and a half ago | (#43136691)

If the primary was down, the email would have gone to the secondary. That did not happen so far, ever, except when I tested it. Also, failure to forward results in an email that eventually gets delivered if email gets through at all. So unless the server is down for several days, I would have known. Except for very exotic scenarios, that means no loss. Postfix is a very, very reliable MTA and the whole email-system is designed for reliability.

Blame? (3, Funny)

Anonymous Coward | about a year and a half ago | (#43134987)

Who? [youtube.com]

So he sez (-1)

Anonymous Coward | about a year and a half ago | (#43134989)

"Yo gettin a new e-mail provida?"
"Fo Shaw!"

Hosers! (1)

KrazyDave (2559307) | about a year and a half ago | (#43135003)

Back bacon and beer, eh? This what happens when Bob and Doug McKenzie are the network troubleshooters.

Re:Hosers! (1)

Anonymous Coward | about a year and a half ago | (#43135063)

And Carly Rae Jepson. "We'll Send it Maybe!"

I thought MY US ISP sucked donkey schlongs... (0)

theoriginalturtle (248717) | about a year and a half ago | (#43135013)

WAIT.... "deleted?"

As in, "spilled the seed on the ground?"

I can understand if things maybe don't get delivered for a few hours, or maybe a few got munged up somewhere during the repairs, but to blissfully direct the firehose into the abyss that is /dev/null...

Who pays for that?

Oh, wait. Canadians are rather accepting of abuse on the part of their phone/cable/broadband suppliers, and the Tories back up the big businesses.

"Unacceptable! Unacceptable! Mepps, mepps!"

Re:I thought MY US ISP sucked donkey schlongs... (4, Interesting)

isopropanol (1936936) | about a year and a half ago | (#43135127)

Shaw is probably the least abusive of the canadian major telecom companies. I've been a shaw customer for 14 years and this is the only incedent I've had other than lines being blown down in a storm. My wife's email was effected but mine was not. This is a normal (and rare) human error... most of the actual abuse telecom companies dish out is abusive contracts and misleading advertizing like 3-year cellphone contracts and "Optik TV and Internet" ... which is actually satellite and DSL, not FTTH.

Re:I thought MY US ISP sucked donkey schlongs... (2)

slazzy (864185) | about a year and a half ago | (#43135315)

I would say Distributel is less abusive, similar prices, runs over Shaw lines but internet is unlimited - no overage fees. Teksavvy used to be good too, not sure if they still are.

Re:I thought MY US ISP sucked donkey schlongs... (1)

AdamWill (604569) | about a year and a half ago | (#43135709)

Teksavvy seems to be getting worse. I check them every so often, and last time I checked, their AUP now specifically says you can't run servers; I'm pretty sure it didn't used to say that. They don't appear to offer static IPs for residential accounts any more. Their business pricing isn't very competitive. Distributel's AUP looks similar to Teksavvy's and Shaw's. I can't really see much to recommend any of the resellers over just using Shaw directly any more unless you really need the 'unlimited' data. That's still an advantage, but I think that may be going away soon, with the recent legal settlement. Shaw's caps aren't really that terrible either, they're way higher than Telus' - I think 250GB for 25Mb/sec, and 400GB for 50Mb/sec. And they do offer unlimited accounts, though at higher prices.

It's worth noting that some of Shaw's business services are weirdly well priced. I'm on their 50Mb/s business account; it sounds expensive, but then you see that it includes TV, and you'd think there'd be a catch, but there really isn't. The business 50Mb/sec service gives you basically the same services you'd get in the home 50Mb/s internet + basic cable TV bundle, for the same price (or $10 cheaper, I forget), you get a static IP address, the AUP on business accounts permits servers, you get a higher bandwidth cap and your uplink speed is faster. So...yeah, there's absolutely no reason not to get it. I did.

Re:I thought MY US ISP sucked donkey schlongs... (1)

Mashiki (184564) | about a year and a half ago | (#43135863)

Teksavvy seems to be getting worse. I check them every so often, and last time I checked, their AUP now specifically says you can't run servers; I'm pretty sure it didn't used to say that.

No it's always said that. Though they really don't enforce it unless people are being abusive, it's more of a CYA clause. They only offer static IP's for DSL customers, that being the nature and problems with the cable plants used by the majority of companies(rogers/shaw/cogeco). Really though everyone got the shaft from the CRTC on the latest round of TPIA agreements, and tek is moving to a new ATPIA system which will cost more. But you'll get more, which is okay. Though they're still fighting the ruling, so is distrubutel, and electronic box.

Re:I thought MY US ISP sucked donkey schlongs... (2, Insightful)

Anonymous Coward | about a year and a half ago | (#43135397)

This is a normal (and rare) human error...

Making mistakes is human. This is why a competent professional acknowledges that he will make a mistake sooner or later, and designs his activities so that mistakes in execution won't have catastrophic consequences. These guys failed to properly do this.

Re:I thought MY US ISP sucked donkey schlongs... (1)

Anonymous Coward | about a year and a half ago | (#43135583)

Shaw might be less abusive of their customers of their internet service.

When it comes to other things, they're nasty as hell.

Barrett Xplore ring any bells for the Canadians here? Shaw taking over Starchoice ring any bells, too?

I ran my own store and decided to sell satellite service and equipment. Barrett Xplore wouldn't even give me the time of day. Eventually their answer was "FUCK OFF" (not joking, those were the actual words from the sales droid). I took their advice and sold FTA equipment after that. Oddly enough, just as I did that, after a few previous months of literally begging Bell and eventually phoning up the CRTC asking why the hell they regulate that only Shaw and Bell can sell service here when they monopolize the sales end of it as well, shuttering legitimate business, Bell called telling me I'm now an authorized dealer. It was too late at that point, I was knee deep in Pansats by then and making $1,000,000 a year in sales. Although their sign looked nice in the door.

If you ever wondered why there was an explosion in sales of FTA in Canada, now you know why. Those idiots caused it through their decision to be complete assholes to anyone running their own shop. All of those of us who decided to go into that business ended up giving up on those jokers and just rolled our own business. Plenty more profitable too...

Re:I thought MY US ISP sucked donkey schlongs... (-1)

Anonymous Coward | about a year and a half ago | (#43135619)

I have no idea what FTA is so I don't fucking care.

Re:I thought MY US ISP sucked donkey schlongs... (2)

hawkinspeter (831501) | about a year and a half ago | (#43136009)

Fibre To Aardvarks?

Re:I thought MY US ISP sucked donkey schlongs... (2, Informative)

Anonymous Coward | about a year and a half ago | (#43135603)

I work for TELUS, and can assure you that optik tv is NOT a satellite service. It is an IPTV service being delivered over either ADSL 2+ or VDSL connections (and sometimes fibre in new areas).

Satellite tv is offered in areas where we don't have the broadband infrastructure to support standard Optik TV. However we refer to that as TELUS satellite TV and not Optik tv.

I don't believe TELUS ever claimed optik was FTTH. The optik name refers to the fact that it is served by our new fibre network. The "last mile" is still copper in most neighbourhoods though (past the DSLAM essentially).

Minor part of your post i know, but thought i would help clarify since you seemed to be confused on what optik is.

Cheers.

http://www.telus.com/content/tv/sat/
http://www.telus.com/content/tv/optik/index.jsp
http://en.wikipedia.org/wiki/Telus_TV

-----

This post reflects my own views and do not necessarily represent the views of my employer.

Re:I thought MY US ISP sucked donkey schlongs... (0)

Anonymous Coward | about a year and a half ago | (#43135905)

Optik TV and Internet can mean a few things.
TELUS has 2 varieties of Optik TV. One is delivered via satellite and is simply rebranded Bell. The other services is the real Optik TV and is IPTV delivered either via DSL or GPON. So in some cases Optik TV is actually FTTH and in other cases it is ADSL2+ or VDSL (which can reach speeds of 60+Mbit).

Contracts...well, that's not as rosy.

Re:I thought MY US ISP sucked donkey schlongs... (1)

Anonymous Coward | about a year and a half ago | (#43136425)

Shaw is probably the least abusive of the canadian major telecom companies. I've been a shaw customer for 14 years and this is the only incedent I've had other than lines being blown down in a storm. My wife's email was effected but mine was not. This is a normal (and rare) human error... most of the actual abuse telecom companies dish out is abusive contracts and misleading advertizing like 3-year cellphone contracts and "Optik TV and Internet" ... which is actually satellite and DSL, not FTTH.

Less abusive to customer perhaps, not to their employees.
http://www.cbc.ca/news/canada/british-columbia/story/2013/03/08/bc-shaw-contracts.html

Things like this (4, Insightful)

suprcvic (684521) | about a year and a half ago | (#43135043)

Are one of the reasons I don't use ISP hosted email. Main reason is portability.

Re:Things like this (1)

theoriginalturtle (248717) | about a year and a half ago | (#43135077)

At least in the USA, then you get nailed for anywhere from 10-40% surcharges for "business-class service" which basically means you pay $400 a year for fixed IP and the "privilege" of running a server on your own broadband line."

My ISP tries to explain the ridiculous cost boost on "well, you get 'priority' on service calls. You have 10 email addresses and 10MB of free web space on our server!"

I shut them up one time:

I've been on "the internet" longer than you've existed as a company. Even at 2am, nobody has ever asked me if I was a business customer. And if I'm running my own server, whadduigivefuck about your "free email addresses and webspace?" I HAVE A SERVER.

#FAIL

Re:Things like this (0)

Anonymous Coward | about a year and a half ago | (#43135153)

At least in the USA, then you get nailed for anywhere from 10-40% surcharges for "business-class service" which basically means you pay $400 a year for fixed IP and the "privilege" of running a server on your own broadband line."

My ISP tries to explain the ridiculous cost boost on "well, you get 'priority' on service calls. You have 10 email addresses and 10MB of free web space on our server!"

I shut them up one time:

I've been on "the internet" longer than you've existed as a company. Even at 2am, nobody has ever asked me if I was a business customer. And if I'm running my own server, whadduigivefuck about your "free email addresses and webspace?" I HAVE A SERVER.

#FAIL

Yeah but you keep on paying them anyway... so for them,

#SUCCESS

Re:Things like this (0)

Anonymous Coward | about a year and a half ago | (#43135181)

Because there's no competition because the US is a corporatocracy.

Re:Things like this (1)

wvmarle (1070040) | about a year and a half ago | (#43135221)

You don't need a fixed IP to receive mail. For years I've used a dynamic DNS service and that worked just fine. I've even for a while ran a web site off a dynamic DNS, also worked. Not recommended of course, but it works.

Re:Things like this (1)

ls671 (1122017) | about a year and a half ago | (#43135425)

Be aware: that was the old days. Nowadays, most need a business class service just to have port 80 and 25 unblocked. You can't run a web server or a mail server on most customer grade connection because the ports are blocked by your ISP.

I used to do it too. Only risk was that during the IP switch, due to the lag of DNS updates depending mostly on TTL and how server respect it, you may have somebody impersonating your site or grab your mail.

Risk was very slim, the machine that gets your old IP would have to know in advance that it will get your old IP and be configured accordingly.

Re:Things like this (2)

pe1chl (90186) | about a year and a half ago | (#43136289)

How terrible!
And how do you access your own NAS or printer over the net?
These days, more and more devices come with their built-in webserver that enables the owner to contact home from his smartphone or tablet.
It is a security nightmare, but that is a different topic.

In the Netherlands, it is forbidden to filter internet other than for security or customer-opted convenience reasons (e.g. spam filtering).
Many providers also offer fixed or semi-fixed IP as standard feature.

Re:Things like this (0)

Anonymous Coward | about a year and a half ago | (#43135441)

Get a VPS and run backups yourself. You can get it for cheaper than $400/yr. Added bonus, when your home connection goes down, your email is still up and running.

Also, it sounds like you are running on cable. I've never had that issue with a DSL company. In fact, they gave me a couple IP addresses for free.

Even though I agree (0, Informative)

Anonymous Coward | about a year and a half ago | (#43135195)

with your comment, I modded you down because it began in the title

Because I DISagree (1)

Runaway1956 (1322357) | about a year and a half ago | (#43135249)

Imagine that - a registered member logs in as AC to tell us that he's a douche. Wow - at least he knows he's a douche!! There is hope for him. Not much, but some hope.

There was never much hope (0)

Anonymous Coward | about a year and a half ago | (#43135469)

only a douche's hope.

Re:Because I DISagree (0)

Anonymous Coward | about a year and a half ago | (#43135485)

He's "logging in" as AC because if he was actually logged in it would reverse his mod.

Whether he is a douche or not is left as an exercise to the reader. Although I will refer to the old adage, "takes one to know one."

Don't let your irrational hate of the AC blind you to the fact that correlation and causation are not the same thing.

Sincerely,

--Another AC

Re:Because I DISagree (0)

Anonymous Coward | about a year and a half ago | (#43135567)

"Even though I agree with your comment, I modded you down because it began in the title"

Douche move, plain and simple.

--YAAC

Don't worry (4, Funny)

the eric conspiracy (20178) | about a year and a half ago | (#43135055)

There will be more mail tomorrow.

Re:Don't worry (0)

Anonymous Coward | about a year and a half ago | (#43135273)

There will be more mail tomorrow.

^ Shaw's Director of Customer Service (they don't have a VP for that)

Re:Don't worry (1)

ls671 (1122017) | about a year and a half ago | (#43135509)

Which VP is he under then, sales?

DIY vs. ISP (1)

PuddleBoy (544111) | about a year and a half ago | (#43135087)

I've hosted my own mail server for about 15 years and I regularly think to myself, 'I'm tired of worrying about hardware and my circuit. Maybe I should let somebody else host it.'

Then it seems there's always an article like this that clears my head.

Re:DIY vs. ISP (1)

gweihir (88907) | about a year and a half ago | (#43135133)

I have primary and secondary on differently hosted vservers with local backups (via procmail) and immediate forwarding to my machine at home. Very little chance of loss and hardware is the provider's to worry about.

Re:DIY vs. ISP (1)

MysteriousPreacher (702266) | about a year and a half ago | (#43136443)

I've hosted my own mail server for about 15 years and I regularly think to myself, 'I'm tired of worrying about hardware and my circuit. Maybe I should let somebody else host it.'

Then it seems there's always an article like this that clears my head.

Why are you continuing to run a set-up that, by your own admission, is a great hassle ameliorated by a once in blue moon event? Seems like wearing a crash helmet whenever outdoors, and justifying it by pointing at an incident in which a pedestrian got clocked by a golf ball and ended up with a brain clot.

This isn't unusual (1)

kilodelta (843627) | about a year and a half ago | (#43135103)

Cox over the years has had some spectacular email outages and fuckups. To the point where I now use Gmail via IMAP and a private domain via IMAP.

/dev/null (4, Funny)

linatux (63153) | about a year and a half ago | (#43135119)

isn't a holding-bay?

Re:/dev/null (0)

Anonymous Coward | about a year and a half ago | (#43135203)

Maybe they thought /dev/null worked like /dev/RecycleBin or /dev/Trash....

Re:/dev/null (2)

Sipper (462582) | about a year and a half ago | (#43135569)

isn't a holding-bay?

/dev/null isn't, but sadly the "Trash" folder is.

A few years ago I was working as an email administrator and got a call to someone's desk that was having a problem with their mail client because some of the folders were too full. One of them was Trash, so I was about to erase messages from the folder when the user paniced; "wait, that's important!"

For whatever reason, they were using the Trash folder for "real work"
(Sigh.)

Re:/dev/null (2)

dbIII (701233) | about a year and a half ago | (#43135727)

I had that too - some insane bullshit of using it as a transfer point to sort stuff. I'd set up mail clients to empty Trash on exit, and when the guy that was doing this logged back in it was of course empty, so he came to rant at me in the lunch room to the amusement of all onlookers. Now every few months he rings me up about a full disk, and each time I have to suggest emptying the Trash mail folder and the "Recycle Bin" on his desktop.
It's not that the person in question is stupid, it's that such people don't understand the design purpose of such items and use them in ways opposed to what was intended, which can sometimes produce useful results as well as just utter facepalms.
A physical example of such a situation is the Australian company Telstra that was providing backup services for several government departments and where using large wheeled plastic trash containers (wheelie bins) to store very large numbers of backup tapes. A new cleaning company won a the cleaning contract and the obvious happened since the new cleaners didn't know that the tapes were intended to be kept, so the backups from four or more government departments, tens of thousands of tapes worth around $100 each not to mention that data on them, ended up in landfill. The people in government with the power to cancel the contract with Telstra were Telstra shareholders so they kept the backup contract even after such a monumental fuckup.

Re:/dev/null (0)

Anonymous Coward | about a year and a half ago | (#43136215)

more like:

# mailq
250000 items in /var/spool/mqueue

"oh, that's an easy fix..."

# rm /var/spool/mqueue/*

# mailq
0 items in /var/spool/mqueue

"my work here is done"

What about hiring competent engineers? (4, Insightful)

gweihir (88907) | about a year and a half ago | (#43135131)

Like, ones that make a backup before messing with critical data? As an elementary precaution known to anybody halfway competent in IT?

This just demonstrates a massive, massive management screwup, as they allowed unqualified personnel to work on their systems. Save a buck, loose a million.

Yep, backup/snapshot before mucking with productio (2)

raymorris (2726007) | about a year and a half ago | (#43135173)

Yeah, messing up is normal. Failing mark a snapshot before becoming with a million emails is incompetence. With a snapshot, human error might have resulted in losing three minutes worth of emails.

"To err is human, to fuck up the whole system requires root."

Re:Yep, backup/snapshot before mucking with produc (1)

gweihir (88907) | about a year and a half ago | (#43135253)

Indeed. And with preventing the server from accepting emails, and a snapshot, there would have been no loss at all. (Emails in the 3 minutes going to the secondary...)

Those truly incompetent are those not aware that they can make mistakes. Seems management is trying hard to make the engineers more like them.

Re:Yep, backup/snapshot before mucking with produc (1)

ls671 (1122017) | about a year and a half ago | (#43135547)

Yeap, the whole mail system is designed from the core so mail should never be lost as I learnt in my young days.

Managing to loose a single email never mind millions is quite an achievement.

Re:Yep, backup/snapshot before mucking with produc (2)

dbIII (701233) | about a year and a half ago | (#43135763)

MS Exchange lowered the bar. Yes I know it's supposed to do a dozen other things but it's MTA was crap for years and still seems to generate a lot of panic on sysadmin mailing lists.

Re:Yep, backup/snapshot before mucking with produc (1)

gweihir (88907) | about a year and a half ago | (#43136715)

panic on sysadmin mailing lists

Hehehe, good old Microsoft. Never transparent, never clear, always good for surprises and gets more obscure than Linux kernel hacking when you have to fix problems MS did not anticipate. In German we call these "Schoenwettersysteme" (translates as "nice-weather only systems"). Toys, not fit for any real-world use.

Re:Yep, backup/snapshot before mucking with produc (1)

dbIII (701233) | about a year and a half ago | (#43135755)

There's a lot of incompetence about, especially bullshit such as the secondary being /dev/null itself as some sort of stupid anti-spam bandaid. I was stung by that one when I had the situation where the primary that was accepting mail for a company I was working for was getting congested and the host they were paying the ISP to supply as the secondary had a management imposed policy of just dropping everything. Probably 2/3 of incoming mail during working hours was never delivered in that four month window before they admitted that we'd been paying to let them throw our incoming email away.

Re:Yep, backup/snapshot before mucking with produc (1)

gweihir (88907) | about a year and a half ago | (#43136583)

There's a lot of incompetence about, especially bullshit such as the secondary being /dev/null itself as some sort of stupid anti-spam bandaid.

How stupid is that? Incredible! The whole reason for secondaries seeing more spam is that some of them do not have spam filters because of incompetent mailadmins. The fix is to either have the secondaries forward to the primaries (when they are back up and storing for some time before that) or to have the same spam filter on the secondaries.

Re:Yep, backup/snapshot before mucking with produc (1)

flyingfsck (986395) | about a year and a half ago | (#43135873)

You don't even need a secondary. If your SMTP server goes off-line, the senders should retry for up to 4 hours. So you can quite literally unplug a mail server, do what you got to do within 4 hours, plug it back in and no mail wil be lost.

Re:Yep, backup/snapshot before mucking with produc (1)

gweihir (88907) | about a year and a half ago | (#43136569)

You are right. I was thinking of IMAP servers for clients sending outbound mail, but they should be separate and a secondary would not help.

Although the postfix/sendmail default for delivery failure is 2 days, I believe.

Re:What about hiring competent engineers? (-1)

Anonymous Coward | about a year and a half ago | (#43135187)

Assuming this was a planned action gone wrong:
Cost of Backing up Gigs of User Email vs Value of Data - If user email isn't considered worth cost, don't bother.

Assuming this was someone poking when they shouldn't
You're screwed anyway.

Re:What about hiring competent engineers? (3, Insightful)

gweihir (88907) | about a year and a half ago | (#43135265)

Not at all. If you are halfway competent, then you are prepared for this scenario and the backup is not expensive. And if you are competent, then you have a reasonable cost-estimate for making your customers mad. These clowns did not have either it seems.

same PHB that let there data center fire take out (1)

Joe_Dragon (2206452) | about a year and a half ago | (#43136761)

same PHB that let there data center fire take out 911 and other stuff in Calgary. Now I can see a fire killing all power or tripping the master power switch but not having a back up data center?

HR sees = cs degree as compentent in IT (1)

Joe_Dragon (2206452) | about a year and a half ago | (#43136825)

HR sees = cs degree as competent in IT while passing over people who went to tech schools or have years of experience

Re:What about hiring competent engineers? (1)

Maow (620678) | about a year and a half ago | (#43135259)

Like, ones that make a backup before messing with critical data? As an elementary precaution known to anybody halfway competent in IT?

This just demonstrates a massive, massive management screwup, as they allowed unqualified personnel to work on their systems. Save a buck, loose a million.

Speculation: backed up the email server settings, made minor change such that spam matched against "*" wildcard.

Spam of course gets deleted and not backed up.

Hours later(!), someone notices overly aggressive spam filter and restores backed-up rules.

Just prior to xmas, I was without internet for > a week due to some routing issues within Shaw, so it may well be that they have an over abundance of incompetence these days.

Also, lost->lose. Tight->loose. See how easy it is to make a simple mistake of only one character that changes drastically the intended outcome?

Re:What about hiring competent engineers? (0)

Anonymous Coward | about a year and a half ago | (#43135477)

I honestly doubt that he cares nor understands the subtleties of administering an information system. Many people on slashdot were programmers, many now are just tech junkies. Gweihir is in the first category, just enough knowledge to be dangerous in a production environment but not enough to troubleshoot live systems.

Re:What about hiring competent engineers? (1)

gweihir (88907) | about a year and a half ago | (#43136627)

Aaaand, fail! I have quite a bit of experience in that class, including several years of running network traffic capturing with very hard real-time conditions where all maintenance was done on the running system, because there was no other possibility. (Was easy though, as the whole thing was my own design. But it was listening on a backbone.) I have been running mail-servers for 15 years, DNS and web servers for 10. I have designed, built and administrated a computer cluster for 6 years. I admit that these things are more of a hobby, as I am vastly overqualified for these activities. My main activity today is IT risk analysis, and software architecture and design.

Well, I guess you can be excused that you cannot grasp the quality level of my comments. I doubt you have a fraction of my experience.

Re:What about hiring competent engineers? (1)

gweihir (88907) | about a year and a half ago | (#43136595)

Hmm. Good theory!

If they were halfway competent, they would store the spam for a few days though, just for this eventuality. Personally, I keep all spam on my own mail servers for a while.

Re:What about hiring competent engineers? (1)

Anonymous Coward | about a year and a half ago | (#43135675)

I was just thinking that. I realise that there is a lot of data, but backups are really important. I've lost stuff before (drives dying) and I'm getting quite anal about making backups. I have a website with a crapload of software that I've built, including a backup system (backups the sites, the databases, and the backup software). The test is that its all saved my poor miserable hide, more than once. Accidentally deleting data is bad. Drives crashing is worse. I used to play an old-timer game called space quest. You couldn't get through the game without backup and recover unless you were very lucky. So they didn't make a backup, don't have backups, have a broken email system, and incompetent engineers (apparently fix the system while deleting customer data). I'm shaking my head.

Re:What about hiring competent engineers? (1)

gweihir (88907) | about a year and a half ago | (#43136663)

Indeed. Everybody with real experience has lost or come close to losing data that they cared about. Those that can learn from experience learn to never be without backups and in particular to never, ever, ever work on life data without backups _before_ it is their job to handle critical data for others.

Not an "Isolated event" (2, Informative)

Anonymous Coward | about a year and a half ago | (#43135169)

From TFA : "The mistake was an “isolated event,” Lakshman said, and promised a detailed review, which would include a discussion about compensation."

Except it isn't. Few years ago I had a business's domain email hosted with Shaw (was included with the internet service and they provided IMAP), and they lost all of it. They wouldn't return my calls about it, and on the third time I called in a week or so later I was told it would not be recoverable, that there is no backup for their business email service at all, instead they would credit the account ~5 days of internet service. I was floored, but was too busy to get into it with them and I had our email backed up so I just moved on with a email hosting provider I felt more confident about.

Shaw's internet service has been decent, but I wouldn't trust them as anything more then a data pipe.

Re:Not an "Isolated event" (3, Interesting)

v1 (525388) | about a year and a half ago | (#43135281)

From TFA : "The mistake was an âoeisolated event,â Lakshman said, and promised a detailed review, which would include a discussion about compensation."

"A small number of our customers may be experiencing a minor problem with..." is standard big provider boilerplate for "between some and most of our customers are having a major problem with..."

Until proven otherwise, when I hear "a small number of...", I immediately translate that into "a few have yet to complain to us about..."

Ah... Shaw. Their email has caused grief for years (0)

WoTG (610710) | about a year and a half ago | (#43135183)

I've been working with a Vancouver based retailer's email newsletter for years. Around here, Shaw is by far the worst of the bigger email domains on the list for deliverability - at least on any of the big webmail providers, recipients can white-list the email address we use to send out emails. Further, some emails will be completely deleted and not put into the junk folder at all. And emails that are suspected spam, will be deleted after only 7 days - don't go on a 10 day vacation.

I could go on, but suffice it to say that I if I notice an associate or friend or family member using a shaw.ca email address, I will often strongly suggest that they move to any of the big webmail providers.

BTW: Gmail provides IMAP and POP access, which is a stumbling block for those who want a desktop email client. I'm not sure about Yahoo or Hotmail.

Re:Ah... Shaw. Their email has caused grief for ye (2)

FuzzNugget (2840687) | about a year and a half ago | (#43135321)

BTW: Gmail provides IMAP and POP access, which is a stumbling block for those who want a desktop email client. I'm not sure about Yahoo or Hotmail.

I'm sorry, I don't follow your logic. How is providing the option for POP and IMAP -- in addition to webmail -- considered a "stumbling block"?

Re:Ah... Shaw. Their email has caused grief for ye (0)

Anonymous Coward | about a year and a half ago | (#43135613)

Looks like one man's stumbling block is another man's cornerstone ;).

It's Shaw. This is not surprising. (2)

_Shorty-dammit (555739) | about a year and a half ago | (#43135191)

Also, gmail exists.

Re:It's Shaw. This is not surprising. (4, Interesting)

Almost-Retired (637760) | about a year and a half ago | (#43135679)

Damn, isn't there anybody here but me who has been locked out of their gmail account for about 2 weeks now? I have not changed a thing in my fetchmailrc or mailfilterrc's, and have been sucking my gmail account dry at 3 minute intervals with fetchmail for damned near 5 years.

2 weeks ago, both fetchmail and mailfilter started reporting password failures. It worked about 30 minutes a day for 5 or 6 days, but has not worked since the last week of February.

I call them up, get some yahoo whose command of English sucks dead toads through soda straws, he leaves to go get someone who speaks English, but the next guy isn't a hell of a lot better, and he finally speaks clear enough that he is telling me the account is blocked because my machine is compromised. I object, its a linux box, behind a router running DD-WRT. Doesn't make squat to him, my machine is compromised.

Seeing as how everything that comes in here has to run the clamav gauntlet, and that this is a linux machine which has not had java enabled anywhere near firefox in months, currently at V-19.0.2, AND that its behind a router running DD-WRT, AND neither chkrootkit nor rkhunter can find anything to complain about, I seriously doubt it has been compromised.

I had been gradually weaning my mailing list activities, moving them to other servers precisely because of their no dups policy, so that was all the impetus I needed to just move all my subs. I still scan them on schedule just in case they actually get someone who reads english wondering why a fetchmail instance is failing to login, telling fetchmail the password is toast when its the same pw I've been using for years, and its long enough John didn't get it in 6 hours of grinding on it when I last checked with john the ripper.

Until that happens, screw gmail, and the camel that rode in on them.

Cheers, Gene

Re:It's Shaw. This is not surprising. (1)

Anonymous Coward | about a year and a half ago | (#43136119)

Did you ask for your money back?

Re:It's Shaw. This is not surprising. (0)

Anonymous Coward | about a year and a half ago | (#43136173)

I'd been using SSH to get to my website for ages. And then it suddenly stopped working. I wondered, was it my password? But no. It turns out that, without telling me, my hosting company had changed things. Specifically, I now have to whitelist IP addresses if I wanted to use SSH. (This little fact is still not mentioned in the help documentation.) This is particularly frustrating as I have a dynamic IP address.

So, maybe Google went and changed something that your setup depends on, like requiring whitelisting or some such.

Re:It's Shaw. This is not surprising. (5, Informative)

DNS-and-BIND (461968) | about a year and a half ago | (#43136529)

You have to log in with the gmail interface and answer a captcha. Then your account's back on.

Cmon (2, Funny)

Anonymous Coward | about a year and a half ago | (#43135211)

Haven't we all fantasized about just deleting the goddamn queue and going home?

Re:Cmon (0)

Anonymous Coward | about a year and a half ago | (#43135269)

Haven't we all fantasized about just deleting the goddamn queue and going home?

One of my college profs had a medical problem and was away from his office for a good six weeks (in the 1970s when hospital stays could be much longer). On his return he trashed a big pile of mail with the comment, "I've been doing OK without this". As far as I know there were never any repercussions, but then again he probably didn't get any bills sent to his uni address...

Last year, their datacenter had fire/explosions... (1)

Anonymous Coward | about a year and a half ago | (#43135263)

http://it.slashdot.org/story/12/07/13/2050234/citys-it-infrastructure-brought-to-its-knees-by-data-center-outage

Like with everything digital.... (2)

trawg (308495) | about a year and a half ago | (#43135549)

...if your email is not in at least two physically separate places, you are at risk of losing all of it, forever.

It's weird Shaw can't restore from a backup - the article is a bit weird on the exact details about what happened and just ends with "the emails were not backed up".

If your online mail provider does not allow you to access or export your data to your own PC (via IMAP, POP, or whatever) then you should switch to one that does - and start backing up your own email if you want to be more confident that it's going to survive catastrophes.

Backups wouldn't help - storing 2 places would (2)

dbIII (701233) | about a year and a half ago | (#43135805)

While it's utterly trivial to alias everything incoming (or even outgoing) to another host that's another bit of infrastructure and often seen as an unnecessary expense. Their backups will be system files, whatever is in the mail spool at any given day is beneath their care factor and anything that arrives after the last backup is gone anyway.
Remember this folks before considering outsourcing, it's not their email so they don't care about it as much as you do. While you may want to keep stuff in two places they are not going to bother to go to the extra expense unless it's to their advantage.

Re:Like with everything digital.... (0)

Anonymous Coward | about a year and a half ago | (#43136509)

maybe you should set your expectations about what you get for free. Since you're not paying for it, you get exactly what you pay for.

Also, with your troubleshooting you didnt try a different machine?

Sounds like its more about you being a moron, and not google.

Shit happens (0)

Anonymous Coward | about a year and a half ago | (#43135633)

How do you not notice all emails being dumped until 11 hours later?

What? (5, Informative)

AdamWill (604569) | about a year and a half ago | (#43135687)

"To top it off, when Shaw did send out notices about this, they looked so much like every day phishing spam that many people deleted them unread."

Erm. No they didn't? I'm looking at one right now and it doesn't look remotely like 'every day phishing spam'. It doesn't offer me anything, threaten me with anything, or ask me to click on anything. It doesn't include any links except to a forum thread, which the text doesn't make any special effort to make you click on. It didn't trigger my mental 'phishing detector' in the slightest.

I got the email notification late Saturday, two days after the event happened, I guess. That's not a horrible delay. I also saw a bunch of delayed mails come through around that time - 10 or so - and they notified me of the sender and subject line of three mails that were lost, so looks like they managed to recover quite a lot.

I dunno, I guess I'm not TOTALLY OUTRAGED at this. As another commenter said, you know, admins screw up sometimes. Lord knows I have. The fact that they're at least able to identify the subject lines of all the lost mails makes a big difference; you could get any really vital ones re-sent.

This is news? (0)

Anonymous Coward | about a year and a half ago | (#43135897)

I apparently got hit by this.

I say apparently because I didn't notice anything had happened. A few people had to resend mail once they bothered to phone me, and then it was business as usual. I figured an MTA was just acting up somewhere along the line. No big deal.

I don't normally do this, but frankly, Shaw doesn't deserve any extreme bad press over this. They're a pretty good company. I've never had an issue with their service before. I continue to pay my bill, and they continue to provide me with the service for which I'm paying for.

Frankly, they're so "we don't care what you do with your pipe" that I probably wouldn't complain even if I had a reason to. No copyright alert system bullshit, no throttling (nothing that has effected me, anyways), no monthly limits. Well, they claim there's bandwidth caps but I've blown by them every month for the past two years and nothing has ever happened (and I'm not talking about by a few megs- I'll happily do 250GB/mo+ when my "cap" is supposedly 80GB). They never complain. And I keep paying my bill on time.

So really, as far as Shaw goes, cut them some slack. Shit happens. If you feel the need to complain, they'll usually do something for you with minimal effort (I had to call them once about a glitch with my PVR, they gave me a brand new unit and refunded me $30 on my next bill). There are far, far, FAR worse companies out there to deal with.

Email is designed for this. (1)

Anonymous Coward | about a year and a half ago | (#43136347)

Well if everybody has their mail servers configured correctly the incoming mail should be flag for redelivery by the sending MTA for at least 2-4 days, so hopefully nothing is lost. I believe Sendmail is 4 days. You would think with so many users Shaw would also have at least secondary MX records for failover. Despite being a horrible protocol email does have it's delivery protections. The problem these days is that everybody *expects* immediacy with a technology that was designed with broken connections in mind. Just think about it like this: "They don't deliver on Saturdays anymore.. don't worry you will get it on Monday"

Re:Email is designed for this. (0)

Anonymous Coward | about a year and a half ago | (#43136381)

Of course the spam mails will not be redelivered, as the spammers do not adhere to the RFC. They fire and forget. But losing spam mail is not really losing anything.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?