Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Cache On Delivery — Memcached Opens an Accidental Security Hole

timothy posted more than 4 years ago | from the everything-has-a-downside dept.

Security 149

jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."

cancel ×

149 comments

Sorry! There are no comments related to the filter you selected.

Firewall? (4, Insightful)

chx1975 (625070) | more than 4 years ago | (#33171862)

I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.

Re:Firewall? (4, Insightful)

MikeFM (12491) | more than 4 years ago | (#33171880)

My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.

Re:Firewall? (1)

tsm_sf (545316) | more than 4 years ago | (#33171988)

My memcached server[...]

Since you run memcached maybe you can answer this. From what I've read it seems like an alerted (or just savvy) admin could easily detect this kind of futzing and (if you're lucky just) inject honey directly onto your machine. Attempting to maliciously write to memcached seems equally problematic. You'd never know if you were successful or just seeing what someone wanted you to see. How off base am I?

Re:Firewall? (1)

neerolyte (878983) | more than 4 years ago | (#33172116)

Just run two. Only one of them has to have real world data, the other one siphens IPs to your firewall software :)

Re:Firewall? (4, Insightful)

PIBM (588930) | more than 4 years ago | (#33172494)

Ive been running memcached since it's out, even sent some patches in.

The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.

I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.

Nothing new, bad admins get bad things done to them, move along.

Re:Firewall? (1)

bill_mcgonigle (4333) | more than 4 years ago | (#33174042)

Memcached is designed to be fast AND non-secure, to be run on your local network

My entire training on memcached is a 45-minute LUG presentation I went to a couple years back, and even then this was readily apparent.

The affected websites need to review their security personnel and procedures.

Re:Firewall? (1)

MikeFM (12491) | more than 4 years ago | (#33172550)

I doubt most admins look and if memcached was open it's virtually a feast of access. You wouldn't necessarily find it easy to jump from memcached to the rest of the system but you could gather a lot of information and alter data going to users and internal systems. Depending how memcache is being used it could be a pretty big exploit. The hardest part would be finding the right data keys. All mine are md5 hashes of a per site secret token combined with the given key so pretty difficult but I'm sure the kind of developer that leaves the service open to the public might be bright enough to name something 'password'.

I'm hardly an expert but it seems pretty obvious. Unless the developer is pretty idiotic it's probably not a problem but if they are then maybe it's a big problem.

Re:Firewall? (4, Interesting)

IICV (652597) | more than 4 years ago | (#33171886)

Yeah, slide 52 (paraphrased) is as follows:

Fixes?

  1. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW....
  2. .....
  3. Also, FW

I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

Re:Firewall? (1, Insightful)

Penguinisto (415985) | more than 4 years ago | (#33171932)

I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

...depends if he's a FreeBSD fan [freebsd.org] or not. :)

Re:Firewall? (1)

marcoslaviero (1873058) | more than 4 years ago | (#33172394)

I prefer pf actually :)

Re:Firewall? (1)

bsDaemon (87307) | more than 4 years ago | (#33172762)

All the cool kids are using 'pf' these days: https://cipitunk.wordpress.com/2007/07/07/ipfw-vs-pf/ [wordpress.com]

Re:Firewall? (0)

Anonymous Coward | more than 4 years ago | (#33173634)

A slow and hopelessly outdated version. It surprises me that a community who wanks to benchmarks would run that kind of stuff. Update to pf 4.8 now!

Re:Firewall? (1)

newdsfornerds (899401) | more than 4 years ago | (#33173036)

If he's a FreeBSD guy he should be using ipfilter. HEH!

Re:Firewall? (0)

Anonymous Coward | more than 4 years ago | (#33172004)

You forgot step 4 and 5
4: ???
5: profit

Re:Firewall? (1)

marcoslaviero (1873058) | more than 4 years ago | (#33172020)

eh... this presentation was more an "in-person" thing than meant for later perusing.

that said, i'll stick to "firewall" next time.

Re:Firewall? (1)

WrongSizeGlass (838941) | more than 4 years ago | (#33172296)

Tech people aren't exactly known for being overly verbose (except maybe on /.):
* ar
* cp
* df
* ls
* mv
* ps
* rm
* sh
* tr
M'kay?

Re:Firewall? (1)

Call Me Black Cloud (616282) | more than 4 years ago | (#33172954)

Ah, I was wondering what FW meant. Thanks.

Never underestimate the ignorance of Web 2.0 devs. (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33172808)

Web 2.0 is an interesting phenomenon. In many ways, it's nothing but embracing ignorance, bad ideas, the worst possible way of doing things, and the stupidest people (both users and developers!) around.

First, let's ignore how Web 2.0 sites cater to the stupidest users, and because of this are willing to abuse private data given to them by their users. That's a whole different story. Let's just focus on the technology here.

The first thing we see is that the use of PHP is prevalent among Web 2.0 sites. If there's one thing that PHP is "good" for, it is quickly developing poorly-performing and insecure web applications, written by programmers with a minimum of technical knowledge. That's why it has blossomed, so to speak, among the Web 2.0 development community. It caters well to their dislike of quality and their lack of technical knowledge. In many cases, the developers are too ignorant to know that there are better options out there, let alone identify the flaws and problems with their own code.

The second thing we see is a lack of knowledge about data storage and scalability. This has manifest itself as the rise of the NoSQL movement. NoSQL is all about throwing away decades of accumulated knowledge concerning relational databases, and replacing it with hash tables. It's about embracing ignorance.

Many of the most popular Web 2.0 sites moved to NoSQL "databases" claiming it was necessary to do so for "scalability reasons", but that's obviously bunk after reading the many articles they publish describing their architectures. We find out that they're clueless about such basic things as indexes, and in some cases even perform joins by pulling ALL of the data back to their PHP code, and manually joining it there.

Worse yet, these Web 2.0 sites are prone to failure. Twitter's "fail whale" has become widely known, and reddit suffers from frequent downtime, including a lengthy incident just yesterday. What's funniest, however, is that the traffic they face is minuscule compared to the activity that many POS and financial systems, or even "Web 1.0" sites, handle with ease because they're developed by knowledgeable developers who know how to properly use relational databases.

Given that the Web 2.0 developers thrive on ignorance and doing things as incorrectly as possible, it's no wonder some of their flagship technologies and sites would have such glaring security holes. That's just what's to be expected from people who don't have a fucking clue about what they're doing.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (1, Funny)

Anonymous Coward | more than 4 years ago | (#33171864)

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 3, August 2010

  Copyright (C) 2010 Anonymous Coward

Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.

DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

    0. You just DO WHAT THE FUCK YOU WANT TO.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (0, Offtopic)

blue trane (110704) | more than 4 years ago | (#33171978)

http://tunes.org/legalese/bugroff.html [tunes.org] [tunes.org]

Richard Stallman of the Free Software Foundation devised, in addition to some marvelous software, the GNU General Public License (GPL for short). Or the CopyLeft it is sometimes called.

It is quite a revolutionary document, using the "copyright" tool to to protect your right to use free software.

Unfortunately using copyright to protect free software is a lot like using a Jackal to guard the hens.

In fact, various inconveniences relating to this have resulted in modifications such as the LGPL (Library General Public License) and more recently the NPL (Netscape Public License)

I call these matters mere inconveniences, the real damage will occur when the Jackal's, (sorry, I mean lawyers), actually get to test the GPL in court for the first time.

Thus enter my version.

Its very simple.

Entirely consistent.

Completely unrestrictive.

Easy to apply.

The "No problem Bugroff" license is as follows...

The answer to any and every question relating to the copyright, patents, legal issues of Bugroff licensed software is....

Sure, No problem. Don't worry, be happy. Now bugger off.

All portions of this license are important..

"Sure, no problem." Gives you complete freedom. I mean it. Utterly complete. A bit of a joke really. You have complete freedom anyway.
"Don't worry, be happy." Apart from being good advice and a good song, it also says :- No matter what anyone else says or does, you still have complete freedom.
Now bugger off. The only way to get rid of pushy Jackals is to ignore them and not feed them. The GPL is just begging somebody to take it to court. Can't you just see it. Exactly the same thing that happened when some twit (not Linus) registered Linux as his own personal trademark. People got upset, started a fund, and hired, off all ruddy things, a Jackal to try and defend the chicken! Who really benefits from this trademark / patent / copyright thing anyway? The lawyers. Who made it up in the first place? The lawyers.
OK so the last part of the license sounds a bit harsh, but seriously folks, if you are a :-

Lawyer asking these legalese questions... You should go off and learn an honest trade that will actually contribute to life instead of draining it.
Programmer asking these legalese questions... You have amazingly powerful tools in your hands and mind, use them to ask and answer the worthwhile questions of life, the universe and everything. Stop mucking about with such legal nonsense and get back to programming.
User/reader asking these question... Don't worry. Go off and be happy. Have fun. Enjoy what has been created for you.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (0, Offtopic)

flyingfsck (986395) | more than 4 years ago | (#33172036)

Sorry to pop your bubble, but there already is a Do What The Fuck You Want To license.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (0, Offtopic)

blue trane (110704) | more than 4 years ago | (#33172048)

I want to reinvent it!

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (0, Offtopic)

BradMajors (995624) | more than 4 years ago | (#33172078)

Sorry to pop your bubble, but there already is a Do What The Fuck You Want To license.

But, the "Do What The Fuck You Want To License" allows users to copy the license and claim they invented and own this license.

Re:DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (-1, Offtopic)

Anonymous Coward | more than 4 years ago | (#33172112)

Anonymous Coward here. Property is theft, therefore theft is property, therefore the license is mine.

I fail to see why this is news (5, Insightful)

OverlordQ (264228) | more than 4 years ago | (#33171868)

Much less 'memcached' being at fault. They say it themselves:

Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.

All this is is stupid admins doing stupid things story and those are dime a dozen.

Re:I fail to see why this is news (2, Insightful)

Etylowy (1283284) | more than 4 years ago | (#33171986)

This.

That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too).
Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers ;-)

Re:I fail to see why this is news (2, Insightful)

Anonymous Coward | more than 4 years ago | (#33172446)

You don't need a security team, just a clue.

Re:I fail to see why this is news (0, Insightful)

Anonymous Coward | more than 4 years ago | (#33172216)

Are there double standards on this site, depending on whether Microsoft are involved or not? You blame the admins here, but if it were Windows, I bet you'd blame Microsoft and not stupid users.

Re:I fail to see why this is news (0, Redundant)

flimflammer (956759) | more than 4 years ago | (#33172508)

I really wish I had mod points right now. +1

Re:I fail to see why this is news (4, Insightful)

MikeFM (12491) | more than 4 years ago | (#33172558)

The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.

Re:I fail to see why this is news (1)

Bigjeff5 (1143585) | more than 4 years ago | (#33174044)

I'm sorry, but I don't see how a system with absolutely no security can somehow be intrinsically more secure than a system without security.

The same security rules apply no matter what systems you use. If you're too stupid to understand basic security practices, then even Microsoft can't save you (though they do at least try).

Re:I fail to see why this is news (1, Informative)

Anonymous Coward | more than 4 years ago | (#33172576)

http://code.google.com/p/memcached/wiki/NewConfiguringServer

Networking
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.

Re:I fail to see why this is news (2, Informative)

vrmlguy (120854) | more than 4 years ago | (#33172640)

http://code.google.com/p/memcached/wiki/NewConfiguringServer

Networking
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.

Here's an idea that won't impact performance: At startup, issue a big multi-line warning if the IP addresses that are getting bound aren't on a Private Internet [faqs.org] :

The Internet Assigned Numbers Authority (IANA) has reserved the
      following three blocks of the IP address space for private internets:

          10.0.0.0 - 10.255.255.255 (10/8 prefix)
          172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
          192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

Re:I fail to see why this is news (1)

jon3k (691256) | more than 4 years ago | (#33173240)

Because in a lot of scenarios you might have a memcached server that uses public address space. You'd just use iptables to only allow ssh for remote management and then only allow your webservers to connect on your memcached port. No need for a big expensive hardware firewall. But I guess a warning wouldn't _hurt_ anything? Why not just "***WARNING - Only your webservers should be able to connect to memcache!"

Re:I fail to see why this is news (0, Flamebait)

grumbel (592662) | more than 4 years ago | (#33172600)

The "blame the user" is pretty much standard under Unix/Linux, after all it makes you:

1) Feel clever
2) Stops you from thinking of how to improve the situation

Stuff like this happens simply because currently day computer systems are extremely crappy in communicating what they export to the world, thus it is very easy to overlook cases like this where an app exposes more then you intended.

Re:I fail to see why this is news (2, Interesting)

Enleth (947766) | more than 4 years ago | (#33172622)

netstat -lpn seems simple enough. I tend to run it every time I change something in a configuration file of a network-enabled service, just to be sure. It would be irresponsible to do otherwise.

Re:I fail to see why this is news (1)

grumbel (592662) | more than 4 years ago | (#33172806)

"netstat -lpn" lists a lot of stuff that is not exported, you can get closer with "netstat -tulpen" or whatever, but even then you only have a very very rough overview about what you are exporting. For example you could have samba (or apache or whatever) running, which you want, but have it export all of '/' instead of just the directory you want. There is no easy way to find that out without digging through a lot of config files.

Re:I fail to see why this is news (1)

SanityInAnarchy (655584) | more than 4 years ago | (#33173804)

extremely crappy in communicating what they export to the world,

If you're an admin -- particularly one for, say, PBS -- and you're setting up a server, don't you think it'd be worth asking how that server does its communication? And it's not like this is undocumented -- from the wiki page on setting up a server [google.com] :

By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.

And maybe I am just "clever", but this was pretty much obvious to me within about two minutes of learning of memcached -- I'm amazed anyone actually ran it this way. Sorry, but this isn't a case of poor UI design -- if you're employed as a system administrator, it's your job to know this shit.

Re:I fail to see why this is news (0)

bickerdyke (670000) | more than 4 years ago | (#33172630)

I wouldn't call it double standard, as this is exactly about what MS is doing: Pretending that it's easy.

It's ok to design a cache-db thats only secure behind a firewall. As long as you're informing your users about it. I'd take any bet that it is in the memcachd manuals in bold.

The MS way usually is to pretend that it's easy. The docs consist of a leaflet that explains how to open your cd drive and place the cd the right side up, and how to handle the mouse to make the cursor move to the "Ok" button.

At first glance, making things easier sounds like a good idea. "Hey, installing WinXP is easy! Just insert the CD and hit OK three times. Anyone can do it." Yes, sounds like a way to make everyones life easier. But it's a really bad idea if anyone CAN install an OS if only very few people know about the additional steps needed to finish setting up a reasonybly secure machine. (Antivirus, Firewall/NAT, AdBlocker, Browser, YMMV)

Re:I fail to see why this is news (3, Insightful)

TheRaven64 (641858) | more than 4 years ago | (#33172372)

A good tool is designed in such a way that the correct use is the easiest. Does memcached, for example, default to only accepting connections from the local host and require other IPs to be explicitly added? That would be trivial to implement and, if done, would require the admin to implement something like the correct security policy. In contrast, defaulting to accepting connections from anywhere means that the admin can use it incorrectly without needing to do any thinking, but needs to think before using it correctly.

Re:I fail to see why this is news (4, Insightful)

MikeFM (12491) | more than 4 years ago | (#33172560)

It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.

Re:I fail to see why this is news (3, Insightful)

nebular (76369) | more than 4 years ago | (#33172710)

I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

Re:I fail to see why this is news (2, Insightful)

apoc.famine (621563) | more than 4 years ago | (#33173788)

That's like saying knives should be designed to be dull, because users are idiots, and will cut themselves.

It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.

Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair straightener has a warning on it that it can burn you because it's hot....do we really need to extend this to software?

I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out. Same for software. If you're not smart enough to run a website, perhaps you need a painful pointing out of that.

Re:I fail to see why this is news (5, Insightful)

TheRaven64 (641858) | more than 4 years ago | (#33172750)

Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.

If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.

Re:I fail to see why this is news (1)

SanityInAnarchy (655584) | more than 4 years ago | (#33173826)

More than that, once installed, it defaults to not being used at all.

You have to actively develop and deploy software which uses and trusts memcached before this is a security hazard.

In other words, while it might be nice if it was secure by default, you now need two groups of people to fuck up at the same time -- system administrators and software developers -- both of whom, as part of their job, are supposed to think about security.

Re:I fail to see why this is news (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33172810)

Disclaimer: I don't run a public site or run memcache.

From the description (and the memcache "NewConfigurinServer" instruction excerpts) that have been posted here, it looks like memcache is a knife in a box that says "do not handle by blade" on the outside. It comes sharp, but there's a warning. If you're smart, you either know not to wrap your fingers around the blade, or you read the warning and you learn that you shouldn't hold it that way. If you're dumb, you ignore the warning and reach into the box, and wrap your (remaining) fingers around what you find most convenient. I suppose they could also blunt the knife and force you to sharpen it first, but I can only have so much sympathy for those who ignore warnings and cry later when it is harder to play piano.

In the case of a big public site, if you're a professional that gets paid to set these things up, knowing what should be on the inside of the firewall and what should be on the outside of the firewall has to be job 1, and Rule #1 that goes with that should be "it goes on the inside." Rule #2 should be "see Rule #1." Rule #3 is, "are you absolutely, positively, stake your reputation, your job, you life on it sure it needs to be exposed to the outside? Still, unless the site fails to operate, see Rule #1." Rules 1 & 2 are also a good for setting up a home network, too.

Let me see if I understand this (-1, Troll)

Anonymous Coward | more than 4 years ago | (#33171882)

Let me see if I understand this and can boil it down:

Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?

Re:Let me see if I understand this (5, Informative)

Firehed (942385) | more than 4 years ago | (#33171894)

Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.

Re:Let me see if I understand this (2, Insightful)

Anonymous Coward | more than 4 years ago | (#33172030)

I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.

At the very least, for that extra micro-second of overhead, it means if some cracker gets into your network, you don't have to worry about that as much (not that you don't have to worry, hell, some crackers in your network!).

Also, wouldn't you at least want to restrict it from employees that shouldn't be fucking with it. Granted companies whose purpose it is to develop this application, likely want almost everyone having access, but most instances you wouldn't want this.

None the less, this is off topic, and the article is retarded.

Re:Let me see if I understand this (2, Insightful)

PIBM (588930) | more than 4 years ago | (#33172516)

It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.

Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.

So, what have you been able to achieve exactly ? Nothing beside slow downs everyone downs. People should just learn to use the tools at their disposal. And yes, you can kill yourself with a car. Should we lock them all down to 10mph ?

For the parent, here's memcached help. Notice that even if it's ADRANY, you have it in your face should you be on an unsecure network. And besides, locking it to your ip won't prevent someone inside your network from getting the content.

  memcached -help
memcached 1.2.6
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-a access mask for unix socket, in octal (default 0700)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory. Note that there is a
                            limit on how much memory you may lock. Trying to
                            allocate more than that would fail, so be sure you
                            set the limit correctly for the user you started
                            the daemon with (not for -u user;
                            under sh this is done with 'ulimit -S -l NUM_KB').
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48

Re:Let me see if I understand this (1)

lukas84 (912874) | more than 4 years ago | (#33172686)

Besides, one could easily secure this by using IPsec and machine certificates.

I'm not a Linux admin, but that's how i would do it on Windows.

Re:Let me see if I understand this (1)

MikeFM (12491) | more than 4 years ago | (#33172594)

I don't allow random systems to communicate with my database. It is only accessed directly from localhost. Any other system has to go through a predefined RPC function that sanitizes inputs and outputs and does a very specific task. Even that access is only granted to a few systems and never over the public network. I usually still use the built-in security too but I would consider removing it as I consider it almost useless.

Re:Let me see if I understand this (1)

jon3k (691256) | more than 4 years ago | (#33173272)

Sounds like a great reason to use something like SSL between memcached and the webservers. Forward the incoming SSL connections to memcached back to the memcached port over a loopback address. Now you can have authentication and encryption, if you want it, without adding any complexity to memcached. That way if someone wants to run it locally on a webserver you haven't really added any complexity.

Re:Let me see if I understand this (2, Funny)

riskeetee (1039912) | more than 4 years ago | (#33172038)

You don't leave the keys in the Batmobile when it's outside of the cave. That's just asking for trouble.

Re:Let me see if I understand this (1)

StripedCow (776465) | more than 4 years ago | (#33172150)

Yes, totally true. But it would still have been better to have authentication enabled by default. The user who actually knows what he is doing can then disable it.

Re:Let me see if I understand this (1)

j_sp_r (656354) | more than 4 years ago | (#33172152)

Because of MySQL injection attacks and other problems, it is nice to have different access rights for different parts of your site. So I would not recommend stripping out the auth layer, especially if you have more than 1 DB running.

Even the best programmers and practices can create bugs.

Re:Let me see if I understand this (1)

ewanm89 (1052822) | more than 4 years ago | (#33172776)

I always use MySQL permissions to separate databases so that one application can't access another database so easily, I also never allow access to it from the internet directly, same policy applies to all server software, unless it's made to be directly connected to and needs to then it stays on the internal network or loopback.

As for all programmers having bugs, even Rivest et all's MD6 reference implentation code sent for the SHA3 NIST competition had a buffer overflow in it: slashdot.org [slashdot.org] , and while we all should say security experts should know better, these mistakes happen from time to time.

Re:Let me see if I understand this (0)

Anonymous Coward | more than 4 years ago | (#33173458)

What the fuck is a "MySQL injection attack"?

More Boiled and Distilled. (5, Insightful)

SuperKendall (25149) | more than 4 years ago | (#33171902)

Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?

Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.

Is that an underwear design flaw?

Re:More Boiled and Distilled. (5, Insightful)

Farmer Tim (530755) | more than 4 years ago | (#33171914)

Best. Analogy. Ever.

Re:More Boiled and Distilled. (4, Funny)

davester666 (731373) | more than 4 years ago | (#33171936)

No. Car. Was. Involved.

Re:More Boiled and Distilled. (5, Funny)

Farmer Tim (530755) | more than 4 years ago | (#33171954)

That's. Why.

Re:More Boiled and Distilled. (0)

Anonymous Coward | more than 4 years ago | (#33172272)

Who said the underwear did not have cars on them?

Re:More Boiled and Distilled. (1)

icebraining (1313345) | more than 4 years ago | (#33172674)

No, they obviously have funny cartoon animals [tumblr.com] .

Re:More Boiled and Distilled. (1)

human-cyborg (450395) | more than 4 years ago | (#33172952)

Hear, hear!

Re:More Boiled and Distilled. (4, Funny)

pushing-robot (1037830) | more than 4 years ago | (#33171918)

As spokesman for the Justice League, I say yes.

Re:More Boiled and Distilled. (5, Funny)

outsider007 (115534) | more than 4 years ago | (#33171944)

That's actually more of a feature.

Re:More Boiled and Distilled. (0, Redundant)

Lord Kano (13027) | more than 4 years ago | (#33172016)

Is that an underwear design flaw?

Well, kinda. That's why I don't use them.

LK

Re:More Boiled and Distilled. (2, Funny)

sjames (1099) | more than 4 years ago | (#33172054)

Boiled and distilled underwear....Ewwwww.

Re:More Boiled and Distilled. (1)

imsabbel (611519) | more than 4 years ago | (#33172108)

Tell that superman face to face and lets see how strong your point is :)

Re:Let me see if I understand this (0)

Anonymous Coward | more than 4 years ago | (#33172144)

I'm guessing you've never used it and have no idea why it would be useful.

Re:Let me see if I understand this (2, Funny)

Anonymous Coward | more than 4 years ago | (#33172356)

Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?

"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"

Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal but they created a bunch of high performance distributed system tools - gearman, memcache, mogileFS, etc that allowed anyone to build massive social website prior to them, the tools were not there. Now I am sure some smug historical revisionists was come and explain how these things were all built in SNOBOL in the 1950s then reinvented in Java as apache foundation project with simple 12,000 line xml config files that future generations will claim are alien code for time travel devices but those historical revisionists would be dead wrong.

A few clarifications (5, Informative)

marcoslaviero (1873058) | more than 4 years ago | (#33171996)

In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]

Re:A few clarifications (1)

gabba_gabba_hey (309551) | more than 4 years ago | (#33172124)

It's just odd to me that people don't read the manual. Perhaps the memcached authors could make the fact that it doesn't do any sort of authentication more obvious. Still, it was quite plain to me when setting it up that it needs to be firewalled - just like everything else that you don't want to be accessed from outside.

Re:A few clarifications (5, Interesting)

marcoslaviero (1873058) | more than 4 years ago | (#33172182)

There's a deeper issue at play here as it relates to shifting apps and platforms away from your own hardware/networks. Developers are now often responsible for deploying apps onto cloud systems where they don't have experience with network-security or the tools for protecting network-based services, and this is an obvious difference from the traditional network/app split that occurs in most corporates. It doesn't help that memcached (by default) binds to * but they do make this pretty clear (also, remote enumeration of the cache is genuinely a debug feature).

Man pages help, but when the defaults don't aid developers we need to a rethink both of the software (memcached) and the systems were it's not running securely (cloud platforms).

Re:A few clarifications (1)

gabba_gabba_hey (309551) | more than 4 years ago | (#33172254)

Excellent points and I don't disagree with you at all.

I come from a different background where I would never think of exposing this sort of thing and I think that also needs to be drilled into anyone dealing with networked apps. Security 101 never really changes.

Re:A few clarifications (1)

DrSkwid (118965) | more than 4 years ago | (#33172264)

The deeper issue that's exposed is that your mythical developers are ignoring the fact that most attacks come from inside. Mostly through rouge employees or compromised machines. They are operating on the fallacy that you can have one big happy subnet and everybody attached it is a team player.

I don't use memcached but it sounds like one should have your memcached server machines on their own LAN and multiple network cards in the client machines.

It's really not that difficult, it's just :

1 developers are not security minded
2 people are lazy
3 people have a false sense of security
4 people have an overblown confidence in their own ability
5 the company hasn't been burned yet

Most things happen once 5 has changed - when was the last time you ran your disaster recovery plan?

Re:A few clarifications (1)

marcoslaviero (1873058) | more than 4 years ago | (#33172300)

Indeed, i've had confirmation from a number of other security people about encountering memcacheds internal to organisations, and your points are all valid for internal installations. The external-facing instances we found are not explained by assuming the internal network is trusted, hence the point about developers taking an increasing network-security role.

Re:A few clarifications (5, Funny)

IAmGarethAdams (990037) | more than 4 years ago | (#33172422)

Mostly through rouge employees

Luckily, they often get caught red-handed.

Re:A few clarifications (0)

Anonymous Coward | more than 4 years ago | (#33172424)

Mostly through rouge employees or compromised machines.

Bet that leaves them red in the face with embarrassment.

Re:A few clarifications (1)

MikeFM (12491) | more than 4 years ago | (#33172612)

I refuse to give even the boss access. I've even designed systems that wouldn't give me access except under very specific conditions such as a second admin also entering their credentials from a known system during business hours. Some data needs to be treated with extreme care.

Admin or distro? (5, Interesting)

shish (588640) | more than 4 years ago | (#33172012)

Debian's default config says:

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1

Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)

Re:Admin or distro? (0)

Anonymous Coward | more than 4 years ago | (#33172068)

Admin. The memcached man page for 1.4.5 is 126 lines long in a standard terminal.

You have no business administering a machine, if you can't be bothered to read 126 lines of text, don't know what the fuck you're doing, and are relying on the defaults of your distro to keep you safe.

Especially if you're getting paid to run large public services like the ones described in the presentation.

Re:Admin or distro? (1)

buchner.johannes (1139593) | more than 4 years ago | (#33172134)

From http://linux.die.net/man/1/memcached [die.net] :

-l
        Listen on ; default to INDRR_ANY. This is an important option to consider as there is no other way to secure the installation. Binding to an internal or firewalled network interface is suggested.

yeah ... it's hard to blame the memcached guys

Re:Admin or distro? (4, Insightful)

TheRaven64 (641858) | more than 4 years ago | (#33172390)

default to INDRR_ANY

And this is why they're to blame. Default should be the loopback, and enabling external access should require explicit configuration.

Re:Admin or distro? (1)

fearlezz (594718) | more than 4 years ago | (#33173226)

Sorry, I don't agree with that.
Memcached is not meant for single-server configurations. It's meant to combine the memory of all connected servers, so that if you connect 10 servers with 4GB each, you get 40GB of cache. This won't work if you only allow loopback connections.
The admin should block memcached connections on the outer interface. And set the default firewall policy to DENY to start with.

Re:Admin or distro? (3, Insightful)

bill_mcgonigle (4333) | more than 4 years ago | (#33174092)

Memcached is not meant for single-server configurations

That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.

If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.

Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.

Re:Admin or distro? (1)

akanouras (1431981) | more than 4 years ago | (#33172442)

I've met smartpants web developers that felt they were getting a more tailored-to-their-system installation by downloading tarballs and compiling the stack all the way down.

Nevermind updates or security though...

Re:Admin or distro? (1)

MikeFM (12491) | more than 4 years ago | (#33172626)

That's why you create your own packages and have a system to add your mods to the public versions of the packages. I don't custom compile everything but the components most key to what I'm doing are often custom compiled because the public packaged lack the functionality I need.

Re:Admin or distro? (1)

akanouras (1431981) | more than 4 years ago | (#33172716)

Your way is the proper one to go about it, provided you stay on top of security updates by yourself.

In the aforementioned developers' case, they just followed some shitty "How to install LAMP" "howto" and never bothered to check if their distro provided the relevant packages. On top of that, they had absolutely no clue on how to provide a secure or even performance-optimised configuration for the custom compiled packages. If all that wasn't enough, they also never applied security updates after the initial installation.

Re:Admin or distro? (2, Interesting)

Paul Jakma (2677) | more than 4 years ago | (#33172464)

So what if you want to run memcached on a multi-user machine?

It's slightly mad that software like this, which is designed without security, would use TCP per default, instead of local Unix sockets (access to which can be controlled with standard Unix filesystem permissions on the containing directory (careful about relying on permissions on the socket itself have any effect - not portable)). Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).

Re:Admin or distro? (1)

MikeFM (12491) | more than 4 years ago | (#33172634)

Don't run it on a multi-user machine. That's why we invented server farms and virtualization. Nothing should be on a multi-user machine.

Re:Admin or distro? (1)

Culture20 (968837) | more than 4 years ago | (#33173594)

Nothing should be on a multi-user machine.

Then Unix shouldn't be multi-user. Somehow I think there are practical issues with that sentiment.

mysql/ssh/postgres/ftp/etc. opens security hole (2, Insightful)

pyalot (1197273) | more than 4 years ago | (#33172102)

Yeah like... all these open a socket to listen to, and man, if somebody logs into it, they're like, owning your server! Geeze, what where mysql/ssh/postgres/ftp/etc. thinking?!!!

Re:mysql/ssh/postgres/ftp/etc. opens security hole (0)

Anonymous Coward | more than 4 years ago | (#33172130)

They all have authentication and authorisation mechanisms. Memcached, by design, does not. This is clearly stated on the memcached site and in its config file.

Re:mysql/ssh/postgres/ftp/etc. opens security hole (1)

Lord Bitman (95493) | more than 4 years ago | (#33172644)

..and presumably a developer would notice when at some point they write connect_to_memcached() and don't need to pass in a username or password..

Re:mysql/ssh/postgres/ftp/etc. opens security hole (1, Insightful)

Anonymous Coward | more than 4 years ago | (#33172938)

Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."

Re:mysql/ssh/postgres/ftp/etc. opens security hole (1)

pyalot (1197273) | more than 4 years ago | (#33173698)

If you somehow think that having user/password authentication would prevent you from being owned, then you're even more incompetent then the admins of the exploited memcached servers.

Flash required (0)

Anonymous Coward | more than 4 years ago | (#33172168)

Can we have a Flash warning like we have on PDFs? At least I can open PDFs but not Flash.

Re:Flash required (1)

Skapare (16644) | more than 4 years ago | (#33172384)

Flash for a freaking slide show? This is too easily doable without Flash. Even Javascript is overkill. And it requires Flash version 9 as if no earlier Flash version could do a slide show.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?