Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Chaos Monkey Released Into the Wild

Unknown Lamer posted about 2 years ago | from the infinite-monkeys-with-infinite-hammers dept.

Virtualization 76

Quince alPillan writes "Netflix revealed today that they've released Chaos Monkey, an open source Amazon Web Service testing tool that will randomly turn off instances in Auto Scaling Groups. 'We have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient. We are excited to make a long-awaited announcement today that will help others who embrace this approach. ...source code for the founding member of the Simian Army, Chaos Monkey, is available to the community.'"

cancel ×

76 comments

Sorry! There are no comments related to the filter you selected.

Into the wild? (4, Informative)

dubl-u (51156) | about 2 years ago | (#40824589)

And by "into the wild", they mean they're now letting it run on other people's sites.

Re:Into the wild? (5, Funny)

jcoy42 (412359) | about 2 years ago | (#40825273)

This is why we don't let you write headlines.

Re:Into the wild? (1)

Anonymous Coward | about 2 years ago | (#40825317)

I think the concept is good. That if the desire is to withstand failures of unforeseen natures, then test with random failures and observe how the software reacts to it.

In practice, it will likely get you fired in a production environment, but I still think the idea behind it is sound.

Re:Into the wild? (5, Insightful)

inKubus (199753) | about 2 years ago | (#40825983)

Sound idea, sure. But not a substitute for good engineering. You see this issue come up again and again with these cloud services. The pressure from sales and marketing to move quickly and monetize the idea (and support lots of subscribers quickly) is not conducive to building a solid infrastructure. Netflix's approach is actually the exact opposite of Amazon's. Amazon's system is highly engineered and designed to resist failures that take down Amazon.com for it's customers. That is their number one goal. Amazon.com has not been down for a long time. AWS is an offshoot of that effort to resell their extra cycles but it's not nearly as engineered at the Amazon.com application built on top, which redirects around the globe and does lots of other things. It seems that AWS always has some new service coming out, but think about this: all those services were probably made by Amazon 3 years ago and they are just now releasing them to you..

Netflix, on the other hand, seems to be just hacking together a site, if this is really what they primarily used to QA their application. What you're doing with this random failure thing is just statistically creating errors and finding bugs in failure handling code statistically. This means there's _up to_ an infinite number of bugs that will *not* be found with this method because they are unlikely or the tester is unlucky.

It certainly has to do with the math of it, but it also has to do with the human culture that arises when working like this. See, with this brute force iterative programming, you are building a nest of patches. So what you are going to end up with is going to be more complicated and less functional than if you do the hard work. And that's the issue. Thinking about stuff in terms of thousands or millions of nodes is "too hard" so the aforementioned cloud providers keep coming up with "creative solutions" like this. (I remember reading about Facebook hacking mysql a few years back and shaking my head as well..) But, like "creative accounting", it may not be illegal but it may get you into trouble. You're never going to be absolutely sure the application will stay up and available. Ok, fine, so it Netflix goes down no ones going to die, but still...there's millions of dollars and subscriber goodwill at stake and that's not nothing.

Anyway, don't think that I'm railing against creative testing, but they shouldn't think they are so clever as the release seems to imply they think they are ;)

Re:Into the wild? (2)

dave420 (699308) | about 2 years ago | (#40826839)

That's a lot of guesswork... I don't see many links backing your positions up.

Re:Into the wild? (1)

luis_a_espinal (1810296) | about 2 years ago | (#40831115)

That's a lot of guesswork... I don't see many links backing your positions up.

His positions are superficial and emotional, that's all.

Re:Into the wild? (0)

Anonymous Coward | about 2 years ago | (#40827201)

This means there's _up to_ an infinite number of bugs that will *not* be found with this method because they are unlikely or the tester is unlucky.

There are up to an infinite number of problems with that sentence, and with mine, and with up to an infinite number of other sentences.

But, at least we have up to an infinite number of monkeys working on the problem. That will at least find up to an infinite number of solutions, and the complete works of Shakespeare.

Re:Into the wild? (3, Insightful)

eyrieowl (881195) | about 2 years ago | (#40827557)

There are a lot of things that can go wrong in failover scenarios. Unless and until they are tested in real world situations, you can't be certain the system works. I happen to know of many systems which had failover processes which were "tested", and sounded fine on paper, but when it came to the real world, they had failed to account for this or that unexpected condition which ended up leading to far more downtime that was expected. If chaos monkey is their ONLY way of arriving at a resilient service, than sure, they have a deeper issue. But if they've spent time trying to design a solid system and then they're using Chaos Monkey to make sure it's as bullet-proof as they think it is, then it's good, solid engineering for the real world. I am reminded of the book "Inviting Disaster", on technology failures. All the systems described in the book which failed were well engineered systems. But due to a series of events working in concert, disaster happened. Any one link in the chain of failures wouldn't be enough; and it is not possible to fully engineer that out of your system; and certainly not possible to test for that in controlled testing environments. But if you can start causing failures in the real world (which is a luxury you have with systems that don't actually keep people alive), you have the opportunity to eliminate those sorts of weaknesses from the system. That's what I think is the value to something like this.

Re:Into the wild? (0)

Anonymous Coward | about 2 years ago | (#40851901)

One additional thing is that by using the chaos monkey, you know exactly what went down and you'll be able to immediately bring it back online because it's not actually broken. If this service went down due to some other reason such as a hardware failure and it caused problems with the rest of your system, then you would have to wait until you can fix that service before you can bring your whole system online (possibly hours or days). With the chaos monkey you can see the problem then get your system back online within minutes and create the fix before you actually have a real outage.

Re:Into the wild? (1)

Anonymous Coward | about 2 years ago | (#40828123)

The superiority of Amazon's engineering culture over Netflix must be the reason why during the last major EC2 outage, Netflix managed to stay up and operational...

Re:Into the wild? (1)

inKubus (199753) | about 2 years ago | (#40852667)

To clarify what I specifically wrote in my post, Amazon.com (Amazon's application, where they make the money), has not been down in a long time. The Virgina EC2 outage only affected the excess capacity they resell to AWS customers. I'm not singling out Netflix and I'm not saying that this is a bad or horrible or un-useful tool. I appreciate all the stuff Netflix is open-sourcing.

Re:Into the wild? (1)

Anonymous Coward | about 2 years ago | (#40828257)

I've been using Netflix for years now - and I've only had trouble with the streaming service once, maybe twice in that time. Hulu often freaks out, Vudu the same - so I think the proof is in the pudding. Lastly, this is just an additional testing piece, and frankly it is very cool. No where did they say it was a substitute for good engineering.

Re:Into the wild? (2)

metrometro (1092237) | about 2 years ago | (#40828431)

If Netflix is hacking together a site, why is their HD streaming more reliably pleasing than any other online service, including places like Comcast, which presumably has 100x the engineers on hand? Maybe they are good at teh hacking?

Re:Into the wild? (2)

JackieBrown (987087) | about 2 years ago | (#40828525)

It is better than Amazons on the PS3 for me. Amazon gets stuck buffering a lot for me.

Re:Into the wild? (2)

rwa2 (4391) | about 2 years ago | (#40830467)

Meh, what's the point of good engineering if you never test it? I've heard of a quite a few wonderfully expensive and over-engineered UPS and RAID deployments that failed completely because they never bothered to actually test the procedures. The last company I worked at would often have regular "emergency power off" events where they'd do a complete shutdown of the entire datacenter triggered by various environmental factors. And you know what? More times than not they'd still find a system that somehow missed the trap and didn't get shut down properly, and plenty of caveats with the enterprise-grade UPS infrastructure.

At one of the first companies I worked at, the idea was to engineer a cluster with no SPOF, so we'd actually invite customers (/monkeys) to go to the back and rip out / unplug something, anything, while the cluster was doing something like a distributed POVRay render. It was a pretty simple, elegant test, and a great mindset to have when designing any HA system, not just for fault tolerance, but also to architecturally enable for on-line upgradeability, scalability, and some other niceties.

Re:Into the wild? (0)

Anonymous Coward | about 2 years ago | (#40830983)

What you're doing with this random failure thing is just statistically creating errors and finding bugs in failure handling code statistically. This means there's _up to_ an infinite number of bugs that will *not* be found with this method because they are unlikely or the tester is unlucky.

While what you're saying is technically true, you're missing the point. How much money should a company be willing to spend to fix a bona fide bug that only gets triggered once in a septillion cases?

Re:Into the wild? (1)

luis_a_espinal (1810296) | about 2 years ago | (#40831105)

That this post was modded 5 is a sad testament to slashdot.

Sound idea, sure. But not a substitute for good engineering.

That argument only makes sense if it were the case that Netflix is using it in lieue of good engineering. But, it isn't, so...

Also, this is a false dichotomy. Chaos Monkey is in great part a form of fault injection, which itself is part of good engineering.

You see this issue come up again and again with these cloud services.

Like amazon EC2?

The pressure from sales and marketing to move quickly and monetize the idea (and support lots of subscribers quickly) is not conducive to building a solid infrastructure. Netflix's approach is actually the exact opposite of Amazon's.

You know this from a fact, or is it pure speculation?

Amazon's system is highly engineered and designed to resist failures that take down Amazon.com for it's customers. That is their number one goal. Amazon.com has not been down for a long time. AWS is an offshoot of that effort to resell their extra cycles but it's not nearly as engineered at the Amazon.com application built on top, which redirects around the globe and does lots of other things. It seems that AWS always has some new service coming out, but think about this: all those services were probably made by Amazon 3 years ago and they are just now releasing them to you..

Great non sequitur.

Netflix, on the other hand, seems to be just hacking together a site, if this is really what they primarily used to QA their application.

Seems? Seems? First you state in very certain terms that Netflix is doing the exact opposite to Amazon. And then you say that Netflix modus operandi seems hackey? You built an entire argument against Netflix from what it seems to you?

What you're doing with this random failure thing is just statistically creating errors and finding bugs in failure handling code statistically.

And this is bad because? Ever heard of failt injection? I don't know man, but fault injection has always been part of good engineering workbooks.

This means there's _up to_ an infinite number of bugs that will *not* be found with this method because they are unlikely or the tester is unlucky.

1. This method (and fault injection in general) is not meant to discover all the bugs, nor is it being billed by Netflix for that purpose. The argument makes for a good strawman, though.

2. Fault injection or not, for any large piece of software built, there will always be bugs that will remain undiscovered. Always. This is independent of whether fault injection is used or not as part of development/QA processes.

When you use a fault injection method independent of a developer's POV, the objective is to create scenarios where bugs will manifest themselves during the development process. This is distinct from a QA/Tester that verifies software according to established test scenarios. It is equally distinct from stress/load testing.

How different this is from manually injecting a fault in a system to see if it can cope with it? Say, kill -9 your test database while your app writes to it to see how it handles the error and brings an appropriate error page to the user (as opposed to an ugly http server 500 page)? Bring it back to see if your app can reconnect to it for future transactions? Kill your LDAP server while users are logging into your app to verify that already logged users are not affected by login failures (it shouldn't but most systems fail miserably at this.) Force your thread/connection pool to be size 1 and flood it with requests, injecting out-of-capacity failure, to see how your system manages it? Does it drop death? Can it recover?

What this Netflix tool is doing is simply automating the process of fault injection. What developers (good developers worth their shit that is) do manually, this tool facilitates its automation. It is like an Ant/Cron spawn for fault injection.

You rant about Netflix while comparing it with Amazon (according to the things that seem like to you) neglecting the recent EC2 failures and how Netflix still worked no problems. What have been the recent outages that cause you to decide Netflix is simply hacking its way out? Are you even aware of all the stuff the company does in terms of technology?

I'm not a Netflix fanboy, hell, I have never watched a single thing with its services. But c'mon. Can we at least pretend to try building solid arguments in /. for once?

It certainly has to do with the math of it, but it also has to do with the human culture that arises when working like this. See, with this brute force iterative programming, you are building a nest of patches.

What brute, iterative approach? How do you think Netflix is using this tool?

So what you are going to end up with is going to be more complicated and less functional than if you do the hard work. And that's the issue.

It is an issue that you are gratuituously imputing on Netflix usage of the Chaos Monkey tool thing.

Thinking about stuff in terms of thousands or millions of nodes is "too hard" so the aforementioned cloud providers keep coming up with "creative solutions" like this.

Non sequitur.

(I remember reading about Facebook hacking mysql a few years back and shaking my head as well..)

Shake your head as much as you want, Facebook is doing absolutely fine in its technical endeavours. It might have questionable practices, but technically speaking, they are doing well in anything they do, in development, architecture and hacking.

But, like "creative accounting", it may not be illegal but it may get you into trouble.

How do FUD?

You're never going to be absolutely sure the application will stay up and available.

Netflix isn't claiming that, but don't let that get between you and the road to strawmanland.

Ok, fine, so it Netflix goes down no ones going to die, but still...there's millions of dollars and subscriber goodwill at stake and that's not nothing.

Appeal to emotion. You are trying really hard to build a criticism of Netflix's approach, but the technical merits are severely lacking.

Anyway, don't think that I'm railing against creative testing, but they shouldn't think they are so clever as the release seems to imply they think they are ;)

But who says that they pretend to be so clever? Here is you building strawmans that you then topple to claim argumentative victory.

Re:Into the wild? (1)

inKubus (199753) | about 2 years ago | (#40852633)

Thanks for taking the time to reply to my post, I appreciate it.

Re:Into the wild? (3, Informative)

arkhan_jg (618674) | about 2 years ago | (#40826283)

Seems more about that they've just published the source code on github [github.com] under the Apache licence.

So you can run your own chaos monkey on your own amazon cloud systems, or modify it to run on your private cloud, or whatever.

Chaos Reigns. (0)

Anonymous Coward | about 2 years ago | (#40824645)

Chaos Reigns.

Re:Chaos Reigns. (1)

jimi1x (1105911) | about 2 years ago | (#40834009)

In the future there is only war.

That's racist! (-1)

Anonymous Coward | about 2 years ago | (#40824647)

Waiting for the first comment of someone asking why they released Obama into the wild... 3... 2...

Re:That's racist! (-1)

Anonymous Coward | about 2 years ago | (#40824989)

It says "Chaos Monkey" not "Porch Monkey". Get it right, libtard.

This ... (1)

broginator (1955750) | about 2 years ago | (#40824653)

This reads like something out of The Onion.

Re:This ... (1)

Anonymous Coward | about 2 years ago | (#40824687)

panic(cpu 0): Enraged Monkey Error: Out of bananas!

Very Erlang-y (3, Informative)

Anonymous Coward | about 2 years ago | (#40824703)

We have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient.

Sounds like what has been common in Erlang for decades. [wikibooks.org]

Off topic: when I watch the /. homepage, I am logged in. As soon as I click on a story, I become an Anonymous Coward. Did anybody else experience this bug too?

Re:Very Erlang-y (1)

colinrichardday (768814) | about 2 years ago | (#40824735)

I don't have such a problem.

Re:Very Erlang-y (0)

Anonymous Coward | about 2 years ago | (#40824865)

Thanks, but that does not help me.

I'm using Firefox 14.0.1 (with Adblock, NoScript (but /. is allowed) & Better Privacy) on Xubuntu.

Re:Very Erlang-y (1)

NVW55V (994264) | about 2 years ago | (#40824893)

Sounds like a cookie permission problem to me. Get rid of saved cookies, look through Tools-Options-Privacy-History section-Exceptions button, find relevant Slashdot entries, check their status, delete or modify as needed.

Re:Very Erlang-y (0)

Anonymous Coward | about 2 years ago | (#40824971)

I also have this problem, but I found ticking "public terminal" when logging in fixes it.

Re:Very Erlang-y (1)

FatdogHaiku (978357) | about 2 years ago | (#40825113)

Off topic: when I watch the /. homepage, I am logged in. As soon as I click on a story, I become an Anonymous Coward. Did anybody else experience this bug too?

Some would see this as a super power... of course they're already trolls, but mighty trolls with super powers.
Seriously, I've seen something like this in FF with some privacy plugins, but it's been awhile.

Re:Very Erlang-y (1)

19thNervousBreakdown (768619) | about 2 years ago | (#40825513)

You're probably disabling subdomain cookies. For instance right now we're not on slashdot.org, we're on it.slashdot.org.

DEEP WEB forums : I don't even need a condom! (-1)

Anonymous Coward | about 2 years ago | (#40824721)

for now, mature adults should visit and post at one or both of these unofficial tor discussion forums, these tinyurl's will take you to:

** HackBB:
http://www.tinyurl.com/hackbbonion [tinyurl.com]

** Onion Forum 2.0
http://www.tinyurl.com/onionforum2 [tinyurl.com]

Each tinyurl link will take you to a hidden service discussion forum. Tor is required to visit these links, even though they appear to be on the open web, they will lead you to .onion sites.

I know the Tor developers can do better, but how many years are we to wait?

Caution: some topics may be disturbing. You should be eighteen years or older. I recommend you disable images in your browser when viewing these two forums[1] and only enabling them if you are posting a message, but still be careful! Disable javascript and cookies, too.

If you prefer to visit the hidden services directly, bypassing the tinyurl service:

HackBB: (directly)
http://clsvtzwzdgzkjda7.onion/ [clsvtzwzdgzkjda7.onion]

Onion Forum 2.0: (directly)
http://65bgvta7yos3sce5.onion/ [65bgvta7yos3sce5.onion]

The tinyurl links are provided as a simple means of memorizing the hidden services via a link shortening service (tinyurl.com).

[1]: Because any content can be posted! Think 4chan, for example. onionforum2 doesn't appear to be heavily moderated so be aware and take precautions.

Missleading title (4, Funny)

valentinas (2692229) | about 2 years ago | (#40824727)

I though this was about monkeys...

Re:Missleading title (1)

Anonymous Coward | about 2 years ago | (#40824741)

I'm just wondering what makes Chaos Monkey different than Timetwister, and how much mana it costs.

Re:Missleading title (0)

Anonymous Coward | about 2 years ago | (#40824757)

I'm just wondering what makes Chaos Monkey different than Timetwister, and how much mana it costs.

I doesn't cost mana. It costs banannas. It's a monkey!

Re:Missleading title (0)

the simurgh (1327825) | about 2 years ago | (#40824805)

they just released the source to something that can be turned into a ddos tool in like 5 minutes? seriously?

Re:Missleading title (1)

gman003 (1693318) | about 2 years ago | (#40824863)

Freakin' PING can be turned into a DDOS tool in like five seconds. Doesn't mean it shouldn't be distributed.

Also, I imagine it needs some form of authentication to actually turn your site off. Which means you'd have to already have a privileged username/password for each site you want to attack. Pretty poor DDOS tool.

Re:Missleading title (2)

hawguy (1600213) | about 2 years ago | (#40824915)

they just released the source to something that can be turned into a ddos tool in like 5 minutes? seriously?

If someone else has the private keys that let them control your EC2 instances, then you probably have more to worry about than a tool that will randomly shut down your running instances.

Re:Missleading title (1)

azalin (67640) | about 2 years ago | (#40826119)

No.
Please try to read the summary again. If anyone has gained the access level required to run this software, you are already f*cked beyond rescue.

Chaos Monkey? (1)

Anonymous Coward | about 2 years ago | (#40824813)

One black, one red, one green, one blue and one white mana + X ,where X is random. Throw any in play instant through the room: if it lands face-down eat a banana. If it lands face-up the instant is played a normal.

Re:Missleading title (0)

Anonymous Coward | about 2 years ago | (#40824969)

I really should have said "Chaos Orb", but it's been too long since I played and I never did have the older cards.

Re:Missleading title (0)

Anonymous Coward | about 2 years ago | (#40824767)

I though this was about monkeys...

Well, if I understand it correctly, its about codemonkeys. It's all Cobol to me anyways.

Re:Missleading title (0)

Anonymous Coward | about 2 years ago | (#40824845)

I though this was about monkeys...

Follow the Mandelbrot Set shaped shitstains.

Re:Missleading title (0)

Anonymous Coward | about 2 years ago | (#40827545)

I was also very interested in what a chaos monkey was, and how they were observing the effects on normal monkeys.

Then I saw it was just some Netflix plugin.

Re:Missleading title (1)

slashmydots (2189826) | about 2 years ago | (#40827583)

You would get approximately the same result if you let an actual monkey loose in your server room though.

Chaos Monkey? (0)

GeneralTurgidson (2464452) | about 2 years ago | (#40824771)

What an apt name for our new help desk tech!

Tech journalists: Stop hyping unproven security... (0, Offtopic)

Anonymous Coward | about 2 years ago | (#40824875)

Tech journalists: Stop hyping unproven security tools
Monday, July 30, 2012 | Christopher Soghoian
http://paranoia.dubfire.net/2012/07/tech-journalists-stop-hyping-unproven.html [dubfire.net]

http://static.guim.co.uk/sys-images/Media/Pix/pictures/2010/3/25/1269523445370/Austin-Heap-001.jpg [guim.co.uk]

"Preface: Although this essay compares the media's similar hyping of Haystack and Cryptocat, the tools are, at a technical level, in no way similar. Haystack was at best, snake oil, peddled by a charlatan. Cryptocat is an interesting, open-source tool created by a guy who means well, and usually listens to feedback.

In 2009, media outlets around the world discovered, and soon began to shower praise upon Haystack, a software tool designed to allow Iranians to evade their government's Internet filtering. Haystack was the brainchild of Austin Heap, a San Francisco software developer, who the Guardian described as a "tech wunderkind" with the "know-how to topple governments."

The New York Times wrote that Haystack "makes it near impossible for censors to detect what Internet users are doing." The newspaper also quoted one of the members of the Haystack team saying that "It's encrypted at such a level it would take thousands of years to figure out what youâ(TM)re saying."

Newsweek stated that Heap had "found the perfect disguise for dissidents in their cyberwar against the worldâ(TM)s dictators." The magazine revealed that the tool, which Heap and a friend had in "less than a month and many all-nighters" of coding, was equipped with "a sophisticated mathematical formula that conceals someoneâ(TM)s real online destinations inside a stream of innocuous traffic."

Heap was not content to merely help millions of oppressed Iranians. Newsweek quoted the 20-something developer revealing his long term goal: "We will systematically take on each repressive country that censors its people. We have a list. Donâ(TM)t piss off hackers who will have their way with you.

The Guardian even selected Heap as its Innovator of the Year. The chair of the award panel praised Heap's "vision and unique approach to tackling a huge problem" as well as "his inventiveness and bravery."

This was a feel-good tech story that no news editor could ignore. A software developer from San Francisco taking on a despotic regime in Tehran.

There was just one problem: The tool hadn't been evaluated by actual security experts. Eventually, Jacob Appelbaum obtained a copy of and analyze the software. The results were not pretty -- he described it as "the worst piece of software I have ever had the displeasure of ripping apart."

Soon after, Daniel Colascione, the lead developer of Haystack resigned from the project, saying the program was an example of "hype trumping security." Heap ultimately shuttered Haystack.

After the proverbial shit hit the fan, the Berkman Center's Jillian York wrote:

        I certainly blame Heap and his partnersâ"for making outlandish claims about their product without it ever being subjected to an independent security review, and for all of the media whoring theyâ(TM)ve done over the past year.

        But I also firmly place blame on the media, which elevated the status of a person who, at best was just trying to help, and a tool which very well could have been a great thing, to the level of a kid genius and his silver bullet, without so much as a call to circumvention experts.

http://blogs-images.forbes.com/jonmatonis/files/2012/07/web_chat.png [forbes.com]

Cryptocat: The press is still hypin'

In 2011, Nadim Kobeissi, then a 20 year old college student in Canada started to develop Cryptocat, a web-based secure chat service. The tool was criticized by security experts after its initial debut, but stayed largely below the radar until April 2012, when it won an award at the Wall Street Journal's Data Transparency Codeathon. Days later, the New York Times published a profile of Kobeissi, which the newspaper described as a "master hacker."

Cryptocat originally launched as a web-based application, which required no installation of software by the user. As Kobeissi told the New York Times:

        "The whole point of Cryptocat is that you click a link and youâ(TM)re chatting with someone over an encrypted chat room... Thatâ(TM)s it. Youâ(TM)re done. Itâ(TM)s just as easy to use as Facebook chat, Google chat, anything.â

There are, unfortunately, many problems with the entire concept of web based crypto apps, the biggest of which is the difficulty of securely delivering javascript code to the browser. In an effort to address these legitimate security concerns, Kobeissi released a second version of Cryptocat in 2011, delivered as a Chrome browser plugin. The default version of Cryptocat on the public website was the less secure, web-based version, although users visiting the page were informed of the existence of the more secure Chrome plugin.

Forbes, Cryptocat and Hushmail

Two weeks ago, Jon Matonis, a blogger at Forbes included Cryptocat in his list of 5 Essential Privacy Tools For The Next Crypto War. He wrote that the tool "establishes a secure, encrypted chat session that is not subject to commercial or government surveillance."

If there is anyone who should be reluctant offer such bold, largely-unqualified praise to a web-based secure communications tool like Cryptocat, it should be Matonis. Several years ago, before he blogged for Forbes, Matonis was the CEO of Hushmail, a web-based encrypted email service. Like Cryptocat, Hushmail offered a 100% web-based client, and a downloadable java-based client which was more resistant to certain interception attacks, but less easy to use.

Hushmail had in public marketing materials claimed that "not even a Hushmail employee with access to our servers can read your encrypted e-mail, since each message is uniquely encoded before it leaves your computer." In was therefore quite a surprise when Wired reported in 2007 that Hushmail had been forced by a Canadian court to insert a backdoor into its web-based service, enabling the company to obtain decrypted emails sent and received by a few of its users.

The moral of the Hushmail story is that web based crypto tools often cannot protect users from surveillance backed by a court order.

Wired's ode to Cryptocat

This past Friday, Wired published a glowing, 2000 word profile on Kobeissi and Cryptocat by Quinn Norton. It begins with a bold headline: "This Cute Chat Site Could Save Your Life and Help Overthrow Your Government," after which, Norton describes the Cryptocat web app as something that can "save lives, subvert governments and frustrate marketers."

In her story, Norton emphasizes the usability benefits of Cryptocat over existing secure communications tools, and on the impact this will have on the average user for whom installing Pidgin and OTR is too difficult. Cryptocat, she writes, will allow "anyone to use end-to-end encryption to communicate without ... mucking about with downloading and installing other software." As Norton puts it, Cryptocat's no-download-required distribution model "means non-technical people anywhere in the world can talk without fear of online snooping from corporations, criminals or governments."

In short, Norton paints a picture in which Cryptocat fills a critical need: secure communications tools for the 99%, for the tl;dr crowd, for those who can't, don't know how to, don't have time to, or simply don't want to download and install software. For such users, Cryptocat sounds like a gift from the gods.

Journalists love human interest stories

Kobeissi presents the kind of human interest story that journalists dream about: A Lebanese hacker who has lived through 4 wars in his 21 years, whose father was killed, whose house was bombed, who was interrogated by the "cyber-intelligence authorities" in Lebanon and by the Department of Homeland Security in the US, and who is now building a tool to help others in the Arab world overthrow their oppressive governments.

As such, it isn't surprising that journalists and their editors aren't keen to prominently highlight the unproven nature of Cryptocat, even though I'm sure Kobeissi stresses it in every interview. After all, which journalist in their right mind would want to spoil this story by mentioning that the web-based Cryptocat system is vulnerable to trivial man in the middle, HTTPS stripping attacks when accessed using Internet Explorer or Safari? What idiot would sabotage the fairytale by highlighting that Cryptocat is unproven, an experimental project by a student interested in cryptography?

And so, such facts are buried. The New York Times waited until paragraph 10 in a 16 paragraph story to reveal that Kobeissi told the journalist that his tool "is not ready for use by people in life-and-death situations." Likewise, Norton waits until paragraph 27 of her Wired profile before she reveals that "Kobeissi has said repeatedly that Cryptocat is an experiment" or that "structural flaws in browser security and Javascript still dog the project." The preceding 26 paragraphs are filled with feel good fluff, including description of his troubles at the US border and a three paragraph no-comment from US Customs.

At best, this is bad journalism, and at worst, it is reckless. If Cryptocat is the secure chat tool for the tl;dr crowd, burying its known flaws 27 paragraphs down in a story almost guarantees that many users won't learn about the risks they are taking.

Cryptocat had faced extensive criticism from experts

Norton acknowledges in paragraph 23 of her story that "Kobeissi faced criticism from the security community." However, she never actually quotes any critics. She quotes Kobeissi saying that "Cryptocat has significantly advanced the field of browser crypto" but doesn't give anyone the opportunity to challenge the statement.

Other than Kobeissi, Norton's only other identified sources in the story are Meredith Patterson, a security researcher who is quoted saying "although [Cryptocat] got off to a bumpy start, heâ(TM)s risen to the occasion admirably" and an unnamed active member of Anonymous, who is quoted saying "if it's a hurry and someone needs something quickly, [use] Cryptocat."

It isn't clear why Norton felt it wasn't necessary to publish any dissenting voices. From her public Tweets, it is however, quite clear that Norton has no love for the crypto community, which she believes is filled with "privileged", "mostly rich 1st world white boys w/ no real problems who don't realize they only build tools [for] themselves."

Even though their voices were not heard in the Wired profile, several prominent experts in the security community have criticized the web-based version of Cryptocat. These critics include Thomas Ptacek, Zooko Wilcox-O'Hearn, Moxie Marlinspike and Jake Appelbaum. The latter two, coincidentally, have faced pretty extreme "real world [surveillance] problems" documented at length, by Wired.

Security problems with Cryptocat and Kobeissi's response

Since Cryptocat was first released, security experts have criticized the web-based app, which is vulnerable to several attacks, some possible using automated tools. The response by Kobeissi to these concerns has long been to point to the existence of the Cryptocat browser plugin.

The problem is that Cryptocat is described by journalists, and by Kobeissi in interviews with journalists, as a tool for those who can't or don't want to install software. When Cryptocat is criticized, Kobeissi then points to a downloadable browser plugin that users can install. In short, the only technology that can protect users from network attacks against the web-only Cryptocat also neutralizes its primary, and certainly most publicized feature.

Over the past few weeks, criticism of the web-based Cryptocat and its vulnerability to attacks has increased, primarily on Twitter. Responding to the criticism, on Saturday, Kobeissi announced that the the upcoming version 2 of Cryptocat will be browser-plugin only.

Kobeissi's decision to ditch the no-download-required version of Cryptocat came just one day after the publication of Norton's glowing Wired story, in which she emphasized that Cryptocat enables "anyone to use end-to-end encryption to communicate without ... mucking about with downloading and installing other software."

This was no doubt a difficult decision for Kobeissi. Rather than leading the development of a secure communications tool that Just Works without any download required, he must now rebrand Cryptocat as a communications tool that doesn't require operating system install privileges, or one that is merely easier to download and install. This is far less sexy, but, importantly, far more secure. He made the right choice.

Conclusion

The technology and mainstream media play a key role in helping consumers to discover new technologies. Although there is a certain amount of hype with the release of every new app or service (if there isn't, the PR people aren't doing their jobs), hype is dangerous for security tools.

It is by now well documented that humans engage in risk compensation. When we wear seatbelts, we drive faster. When we wear bike helmets, we drive closer. These safety technologies at least work.

We also engage in risk compensation with security software. When we think our communications are secure, we are probably more likely to say things that we wouldn't if our calls were going over a telephone like or via Facebook. However, if the security software people are using is in fact insecure, then the users of the software are put in danger.

Secure communications tools are difficult to create, even by teams of skilled cryptographers. The Tor Project is nearly ten years old, yet bugs and design flaws are still found and fixed every year by other researchers. Using Tor for your private communications is by no means 100% safe (although, compared to many of the alternatives, it is often better). However, Tor has had years to mature. Tools like Haystack and Cryptocat have not. No matter how good you may think they are, they're simply not ready for prime time.

Although human interest stories sell papers and lead to page clicks, the media needs to take some responsibility for its ignorant hyping of new security tools and services. When a PR person retained by a new hot security startup pitches you, consider approaching an independent security researcher or two for their thoughts. Even if it sounds great, please refrain from showering the tool with unqualified praise.

By all means, feel free to continue hyping the latest social-photo-geo-camera-dating app, but before you tell your readers that a new security tool will lead to the next Arab Spring or prevent the NSA from reading peoples' emails, step back, take a deep breath, and pull the power cord from your computer."

http://www.guardian.co.uk/technology/2010/mar/21/austin-heap-haystack-iran [guardian.co.uk]
http://www.nytimes.com/2010/02/19/opinion/19iht-edcohen.html [nytimes.com]
http://www.thedailybeast.com/newsweek/2010/08/06/needles-in-a-haystack.print.html [thedailybeast.com]
http://www.guardian.co.uk/technology/2010/sep/17/haystack-software-security-concerns [guardian.co.uk]
https://twitter.com/ioerror/status/24425326976 [twitter.com]
http://www.guardian.co.uk/technology/2010/sep/17/haystack-software-security-concerns [guardian.co.uk]
http://blog.austinheap.com/haystack-halting-testing/ [austinheap.com]
http://jilliancyork.com/2010/09/13/haystack-and-media-irresponsibility/ [jilliancyork.com]
http://nadim.cc/ [nadim.cc]
http://news.ycombinator.com/item?id=2855257 [ycombinator.com]
http://www.matasano.com/articles/javascript-cryptography/ [matasano.com]
https://twitter.com/random_walker/status/192745147040145408 [twitter.com]
http://datatransparency.wsj.com/ [wsj.com]
http://www.nytimes.com/2012/04/18/nyregion/nadim-kobeissi-creator-of-a-secure-chat-program-has-freedom-in-mind.html?_r=1 [nytimes.com]
http://www.matasano.com/articles/javascript-cryptography [matasano.com]
https://chrome.google.com/webstore/detail/gonbigodpnfghidmnphnadhepmbabhij [google.com]
http://www.forbes.com/sites/jonmatonis/2012/07/19/5-essential-privacy-tools-for-the-next-crypto-war/ [forbes.com]
http://www.wired.com/threatlevel/2007/11/encrypted-e-mai/ [wired.com]
http://www.wired.com/threatlevel/2012/07/crypto-cat-encryption-for-all/all/ [wired.com]
http://en.wikipedia.org/wiki/Wikipedia:Too_long;_didn't_read [wikipedia.org]
http://www.nytimes.com/2012/04/18/nyregion/nadim-kobeissi-creator-of-a-secure-chat-program-has-freedom-in-mind.html?_r=2 [nytimes.com]
http://www.thoughtcrime.org/software/sslstrip/ [thoughtcrime.org]
http://en.wikipedia.org/wiki/Inverted_pyramid [wikipedia.org]
http://twitter.com/quinnnorton/statuses/229177519704784897 [twitter.com]
http://twitter.com/quinnnorton/statuses/229178651059568640 [twitter.com]
http://twitter.theinfo.org/227813118108127232#id227966760802975744 [theinfo.org]
http://www.matasano.com/articles/javascript-cryptography/ [matasano.com]
http://news.ycombinator.com/item?id=2855257 [ycombinator.com]
http://www.wired.com/threatlevel/2010/11/hacker-border-search/ [wired.com]
http://www.wired.com/threatlevel/2011/10/doj-wikileaks-probe/ [wired.com]
http://twitter.com/kaepora/statuses/228247942723678208 [twitter.com]
http://twitter.theinfo.org/227813118108127232#id227966760802975744 [theinfo.org]
https://blog.crypto.cat/2012/07/cryptocat-2-deployment-notes/ [crypto.cat]
http://en.wikipedia.org/wiki/Risk_compensation [wikipedia.org]
http://bits.blogs.nytimes.com/2012/06/27/an-app-that-encrypts-shreds-hashes-and-salts/ [nytimes.com]
http://www.blogger.com/email-post.g?blogID=16750015&postID=8560613549686010811 [blogger.com]

This work by Christopher Soghoian is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

http://paranoia.dubfire.net/ [dubfire.net]
http://creativecommons.org/licenses/by-nc-sa/3.0/ [creativecommons.org]

i for one (0)

Anonymous Coward | about 2 years ago | (#40824883)

welcome our new simian army overlords

We've been doing this for years (0)

Anonymous Coward | about 2 years ago | (#40824957)

My employer has been doing this for years. We call it "segfaulting".

The Truth (1, Informative)

paleo2002 (1079697) | about 2 years ago | (#40825027)

War, famine, violence, addiction, pollution . . . truly, WE are the Chaos Monkeys!

Better go underground ... (1)

geofgibson (1332485) | about 2 years ago | (#40825065)

Now we see the beginning of the Army of the 12 Monkeys. We're doomed ...

Obligatory... (3, Funny)

CODiNE (27417) | about 2 years ago | (#40825223)

MonkeyLives [folklore.org]

Re:Obligatory... (1)

honestmonkey (819408) | about 2 years ago | (#40828657)

Don't know if you noticed this:

We kept our system flags in an area of very low memory reserved for the system globals, starting at address 256 ($100 in hexadecimal)

100 bucks for an address?

Cool story, though.

Bah! (0)

Anonymous Coward | about 2 years ago | (#40825325)

We don't need Chaos Monkey, we have Summer Intern!

I love this thing (3, Interesting)

ghostdoc (1235612) | about 2 years ago | (#40825463)

Not only for the idea that a serious company lets a masturbating-and-throwing-poo grinning idiot loose in their sensitive vitals, but also because it draws so many parallels with other resilient systems.

Allergies cured by parasitical worms? Chaos Monkey Effect - you need something attacking your defences for your system to stay healthy

Ecosystem that relies on bushfires to clear old vegetation? Chaos Monkey Effect

Something almost Zen about not only turning an attacker's violence against them, but deliberately introducing new attackers so your system is strengthened by them.

Well done chaps, carry on.

Re:I love this thing (0)

Anonymous Coward | about 2 years ago | (#40828121)

Chin strap on your motorcycle helmet comes loose and your helmet flies off behind you in the high wind, causing a thirty-two car pile up on the freeway that just happens to kill the future anti-Christ? Chaos Monkey Effect.

You decide on the "two entress plus rice or noodles" at the Chinese place in the food court. Your second entree is General Tso's chicken. Evidentally popular, the pan containing the chicken is nearly empty. The server spoons the required serving onto your plate, pauses, and instead of dumping the last serving of chicken onto the new, fresh pan, he gives it to you at no extra charge. Chaos Monkey Effect.

You slept with the still shrink-wrapped "Insanity Workout" DVD under your pillow for the past thirteen days. On the fourteenth day, you woke up with six pack abs and a healthy tan. Chaos Monkey Effect.

Re:I love this thing (1)

bdabautcb (1040566) | about 2 years ago | (#40831965)

I was going to attack your attack of bushfires... until I re-read your allergy attack sentence and realized you have it right. Good work, fellow ecology nerd.

Netflix? (0)

Anonymous Coward | about 2 years ago | (#40825489)

Shouldn't they be focusing on new customers and earnings per share?

Java, meh (4, Funny)

codepunk (167897) | about 2 years ago | (#40825537)

Leave it to some java developers to write 100k lines of code to do a shutdown -h now.

Re:Java, meh (3, Funny)

TubeSteak (669689) | about 2 years ago | (#40825827)

1 line to do shutdown -h now.
99,999 lines to build a GUI.

That sounds about right.

Re:Java, meh (1)

codepunk (167897) | about 2 years ago | (#40825933)

Personally I am shocked they did not write it in Scala.

Re:Java, meh (0)

Anonymous Coward | about 2 years ago | (#40828003)

If you think building a good GUI is simple, you're probably not any good at it.

Excellent idea and great work (1)

Anonymous Coward | about 2 years ago | (#40825677)

Congrats team, give yourselves a slap in the face!

Good idea for mobile devs too... (1)

Kelson (129150) | about 2 years ago | (#40825757)

Except they need to randomly turn off the network connection in their test envronment. It's amazing how many mobile apps assume you'll always have a solid connection and never be in an elevator, or walking between tall buildings, or the basement of a convention center, or any other place with a spotty or overloaded signal.

Wait.. (1)

moniker127 (1290002) | about 2 years ago | (#40825767)

I didn't tell anyone about the chaos monkey.... Oh. Its just some program. Carry on then.

The shot heard 'round the Web... (1)

BenJCarter (902199) | about 2 years ago | (#40825859)

The media war is getting serious. Chaos Monkeys? How about you get Stars back?

A Cure for "Unexpected" (3, Funny)

retroworks (652802) | about 2 years ago | (#40825895)

"We have found that the best defense against major unexpected failures is to fail often."

In other words, you'll never be disappointed if you expect total incompetence. I've already achieved this same thing on my own with my Netflix account, by completely and utterly lowering my expectations.

In other news... (1)

jaymemaurice (2024752) | about 2 years ago | (#40825917)

Script kiddies are released on the internet to improve security by exploiting unchecked buffers and unsanitized inputs...
Security of information at all time high.

Errrr...

For a bit more background about Chaos Monkey (2)

ZorroXXX (610877) | about 2 years ago | (#40826151)

Jeff Atwood has an blog Working with the Chaos Monkey [codinghorror.com] .

Re:For a bit more background about Chaos Monkey (0)

Anonymous Coward | about 2 years ago | (#40827045)

Chanos Monkey anppends n's randomly to an's.

Jeffrey Goines approves this message (0)

Anonymous Coward | about 2 years ago | (#40826513)

Jeffrey Goines and his Army of the 12 Monkeys approves this software release.

God, title sounded WAY more awesome (1)

PJ6 (1151747) | about 2 years ago | (#40828459)

than it actually was.

I was picturing a wild, multicolored, gene-spliced ball of fur tearing around, shoving badgers in lion's ears 'n shit.

They released it (1)

ntropia (939502) | about 2 years ago | (#40828983)

Fools. They don't know what they did... Let's hope that James Cole makes it, this time.

Chaos Monkey start up this morning... (1)

TheLoneGundam (615596) | about 2 years ago | (#40834267)

Chaos Monkey start up get working Chaos Monkey is a hoot Chaos Monkey's preferred new target is instance of a group... (the rest is left as an exercise for Coulton fans)
Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>