Firewall Failover With pfsync And CARP

Daniel Hartmeier writes "OpenBSD developer Ryan McBride explains the new firewall redundancy features in the upcoming OpenBSD 3.5 release in his article Firewall Failover with pfsync and CARP. CARP (Common Address Redundancy Protocol) is a free alternative to the patent-encumbered VRRP, responsible for electing masters in a firewall cluster, while pfsync syncronizes packet filter state information among nodes. The combination allows to replace single-point-of-failure firewalls with clusters of two (or more) nodes, which continue to filter ongoing and new connections when nodes fail. Additional features like arpbalance allow one to share a single IP address for multiple servers, transparently balancing load among them, and adapting to servers failing. Pre-order for OpenBSD 3.5 has started, CDs will ship May 1st."

That's really cool

[Anonymous Coward]

I think my office implemented such functionality for like $120k, and it doesn't even work too well.

Re:That's really cool

[hdw]

Yup, we have something like that too.

Except that our 50.000USD firewall solution fails to handle state sync (they've got problems enough with rules sync) and the the failover works so bad that the dudes that run it have failed over to manual fail over :)

I've been _soo_ tempted to suggest to replace the all the gunk with OpenBSD, since it has all the stuff we need, and it works ...

And it is a little bit cheaper. // hdw

HSRP

[bolix]

I love sniffing the Cisco equivalent to CARP. Lots of HSRP calls to 224.0.0.224 with no security built in. A simple ARP poison will fuck the switch. More advanced attack methods can be found c/o Phenolit [phenoelit.de]

Re:HSRP

[bolix]

Absolutely. Multiple mac addresses will cause the switch to fault.... to a default HUB state. At which point any protection afforded by switched port isolation is meaningless.

I wonder...

[Yarn]

What hardware would I need to do this on my 1000SX uplink. Admittedly, I've only peaked at 80Mbit/s so far, but I think even handling that will take some beefy hardware.

Re:I wonder...

[dhartmei]

Filtering ordinary traffic (not extreme test-cases of minimal packets, average number of packets/connection) statefully at 100Mbps doesn't require much hardware. Even little Soekris boxes (embedded 486 133MHz) can do that.

For Gbps, the limiting factor is the NIC and its driver. Some cards/drivers are reported to reach more than 70% of the maximum throughput. The reason they don't (yet) go further is not packet filtering, though.

If you want specific names/models, the mailing list archives contain the reports.

Re:I wonder...

[hdw]

I'm running an OpenBSD 3.4 firewall on a PII-400 with a 100Mb/s Internet feed.

And I know that I've reached over 40Mb/s without any sign of problem with the firewall.

So unless you're running lots of IpSec stuff or have a high rate of connects I don't think the firewall (or OpenBSD) will be the problem.

I think the selecting a good NIC is more important. // hdw

Re:I wonder...

[peripatetic_bum]

I wonder how a "little" p2 can filter 40MB/s of packets. when it seems like the same p2 will bog down in other stuff (im not talking about a gui)

can you explain this?

Thanks

Re:I wonder...

[Homology]

I wonder how a "little" p2 can filter 40MB/s of packets. when it seems like the same p2 will bog down in other stuff (im not talking about a gui)

can you explain this?

The grandparent wrote 40Mb/s, like in 40 mega bit, and a PII can handle this. However, you should have a good NIC and not one of those pisspor Realtek that offloads the work to the CPU.

Re:I wonder...

[hdw]

yup, I can.

First of all, I said Mb, not MB, call me conservative but I'm used to count bandwidth in bits, not bytes.

Second, as I stated, check your NIC and the drivers.
It means a lot when it comes to network handling.

(I remember how out old VAX 11/785 reacted when it shared an non-switched net with 2 sparc servers, the poor VAX was down on it's knees just by trying to ignore the traffic :)).

And as a wider note, the performance of a system isn't only down to processor speed. There's tons of parameters, b

Mailto link?

[duffbeer703]

Why would a /. editor include a mailto link to an OpenBSD developer in a story?

The poor bastard is going to be flooded with spam ad crap now.

Re:Mailto link?

[dhartmei]

@openbsd.org addresses are already readily available for harvesters through cvsweb, mailing list archives and usenet gates, putting one in a /. posting couldn't make things any worse.

The upside is that after a certain amount of spam received, people get really good at filtering it. That's where the motivation behind some of the anti-spam features in OpenBSD comes from, I guess :)

Re:Mailto link?

[SillyNickName4me]

I'd say they come mainly from Theo wanting them due to all the fan mail he receives ;P

Re:Mailto link?

[Shurhaian]

Maybe that caution in the spam-armour munging should be taken into account, hm? After all, if you spam the developers, they'll have brand new samples with which to test their anti-spam routines... get THEM fed up with spam and they're right at the source! Larger projects would take a little longer to have that effect(though, since nobody in their right mind likes spam, those larger projects have more people to work on it, too; that could be seen as good or bad, depending).

Spam an ordinary person until they

Re:This is awesome

[dhartmei]

What CARP/pfsync does is transparent balancing on IP level. Each client connection is redirected to an arbitrary available server. This works for applications where each server can independantly handle a client request, like serving stateless HTTP or DNS from multiple servers.

For SQL, clustering is much more involved. One client might insert data that must propagate to the other server, or locks across all servers must be obtained, etc. This cannot be done transparently on IP level, the servers themselves must support it.

Search for replication, clustering or redundancy together with postgresql, you'll find erserver [erserver.com] etc. Except for very special cases (like read-only databases), this way beyond IP level packet filtering ;)

Re:This is awesome

[afekz]

You say CARP/pfsync does transparent balancing on IP level.

You mean to say CARP, period.

pfsync is for synchronizing firewall state tables.

Sad.

[MisterP]

It's kinda of sad that something this cool gets so little discussion on a site like Slashdot. I guess it will be news when CARP gets ported to linux and iptables gets ip state sync'ing across hosts.

Re:Sad.

[hdw]

Userland CARP is already ported to Linux.

http://www.ucarp.org

/ hdw

Re:Sad.

[SoundGuy666]

Unfortunately, UCARP has not (yet?) implemented one of the most desirable features (to me) of OpenBSD's CARP - Arpbalance. I guess this would require kernel level changes though.

I'm a firewall admin amongst other things..

[harikiri]

...and this looks really attractive to me. Our environment comprises of Nokia IPSO-based firewalls running Checkpoint, so I'm very familiar with VRRP.

However, as excellent as this looks, I can only shudder in horror at the thought of migrating any of our existing rulesets across to openbsd/pf, let alone distributed management of policies across several 'clusters' of firewalls we have.

Yes my friends. I'm asking for a GUI. FW Builder [fwbuilder.org] is a good start, but it still needs work (porting to Windows would be a good start). Migration tools from Checkpoint (or other commercial firewalls) would be another good addition.

PS, I ask for Windows support not for my sake, but so that my co-workers would be able to use it. However, this criticism is levelled at FW Builder.

OpenBSD/pf/CARP has provided a brilliant technical starting block, but it needs these additional tools to make inroads into enterprise organisations.

Re:I'm a firewall admin amongst other things..

[harikiri]

I'm very aware that I could put together my own 'deployment' script with a combination of ssh/scp and rsync.

If you need a GUI and FW admin is your day job, I have to wonder why you're bothering with FW admin.

I do not need a GUI. My colleagues do not need one either (we previously used PIX... shudder). But when you start dealing with a large number of firewalls (we have over 25 deployed), and not simply firewall rules, but NAT, PAT, authentication and VPN's - having a GUI frontend that ties all that information up together and provides it in an easy to manage way, is a lot better than grepping and trawling through long configuration files to make additions or changes.

Yes any capable firewall admin should be able to implement rules once they read documentation for ipfilter/iptables/pf/ipfw/etc - but they shouldn't necessarily have to. The people I work with aren't stupid, they just don't want to have to work at the command-line across multiple systems to implement a single rule.

Re:I'm a firewall admin amongst other things..

[shurdeek]

What you describe as deficiencies aren't caused by lack of GUI, but by lack of automation. While I admit that e.g. iptables per se doesn't have an automation tool, it still can be automatized without a GUI.

Re:I'm a firewall admin amongst other things..

[Bensmum]

So what you're saying is "I don't want to do my job, cause that's too much work."? Seriously, if you can't be bothered to read the documentation, and learn to use a piece of software, then don't use it. If you had bothered to read some pf docs, you'd realize a GUI isn't going to help you. PF has literally the easiest to use, most helpful syntax around, and if you have a long pf.conf, then you are doing something wrong, it supports variables and macro expansion. The default config file has all sorts of c

Re:I'm a firewall admin amongst other things..

[sirket]

There is absolutely no benefit to a GUI at all

This is a idotic comment. I've been a firewall admin for years. I admin CheckPoint, PIX, NetScreen, ipfw, ipf, and pf firewalls.

Have you ever tried to configure a fully meshed VPN topology between 30 sites by hand? Are you really going to sit there and write 900 rules by hand and expect to do it without making a mistake?

What about defining a group of objects on one firewall (say a cluster of web servers) and then going to implement a rule on a different firewall that uses that web server group? With a central GUI, you can define the object once and not worry about changing it in 5 places or making a mistake when you copy it over to another firewall. (Yes this can be done with scripts but if you are going to write a whole management interface, why not stick a GUI on top of it to make browsing rules easier?)

What about when you need to print out the rule sets for a compliance officer or your CEO?

What about when you have have 25 firewalls and you forgot to backup the rule set on a firewall that just died. Wouldn't it be nice to have a management box with all the rule sets stored locally?

There are about 50 good reasons to have a GUI and very few reasons not to have one. As long as you can configure the boxes from the command line and the GUI doesn't generate gibberish rules, then it is an excellent addition to a great firewall package.

-sirket

Re:I'm a firewall admin amongst other things..

[sirket]

Perhaps you should to do your job instead of bitching on Slashdot. Maybe you've heard the expression "A picture is worth a thousand words?"

GUI's can convey more information in less time and do so more accurately than a text based rule set can. If used correctly, it is a valuable asset.

You think apache isn't as good as IIS because they don't have a GUI too? Oh, wait, there are *THOUSANDS* of tools to manage, edit, and distribute text based config files. Its no more difficult to admin dozens of firewalls

Re:I'm a firewall admin amongst other things..

[Bensmum]

I don't recognize a good thing because its not a good thing. This isn't a difficult concept. Just because you make the claim that a GUI is somehow required and you can't function without one doesn't make it so. If you insist on claiming that open source firewall solutions aren't good enough because they don't provide a GUI, how about you back it up with some facts, instead of just insulting the people who are giving you this stuff *for free*. Talk about a "world owes me" attitude.

And its not that I am

Re:I'm a firewall admin amongst other things..

[sirket]

So what you're saying is "I don't want to do my job, cause that's too much work."?

No. What he is saying is that unlike you, he is not an idiot. He recognizes how easy it is to make a typo when you have to enter the same rule and object definition on 25 firewalls. He recognizes the security advantages of a simple clean way to view firewall rules to help avoid a mistake in the ruleset.

The biggest information security threat to any company is the arrogance of its admins. Instead of bitching about a GUI a go

Reading isn't that hard.

[Bensmum]

Seriously, aren't listening. You don't have to enter the same rule and object definitions over and over, that's exactly what I am saying. You make a single template, and then any firewall from there is just changing some variables like $ext_if or $local_net. Plus there are lots of things you don't have to do with pf, like making a whole set of rules to stop spoofing, with pf you can just do antispoof on $ext_if. I am not complaining about a GUI tool, I am saying the parent poster is dumb for complaining

Re:Reading isn't that hard.

[sirket]

You're assuming I want a simple rule set that can be templated. That isn't how most firewalls work. They share objects, but rarely do they share rules. Can this be done through macros and from the command line? Of course it can. The problem is that when you are updating your firewall during the 1 hour 3 am maintenance window it is easy to make a mistake that you just overlook because you've been staring at rule sets all day. Different data representations (A GUI) are critical to making sure that you underst

Re:Reading isn't that hard.

[Bensmum]

I am not assuming anything, seriously, spend some time learning about PF, you don't realize just how huge a difference it makes to your rulesets. We're talking 1100 rules in ipf -> 85 rules in pf. And the only kind of typos GUIs prevent are the same ones pf prevents. If you try to add a rule with a typo in the syntax it will say so. The kind of typo that's a problem is typos in the IPs, which GUIs can't prevent either. I know they are complaing about not having a GUI, when they have never put in the

Re:Reading isn't that hard.

[harikiri]

If you read above, I was complaining about the lack of a windows port for FW Builder (see link above), because this would encourage more enterprises to consider an openbsd pf/carp solution.

When I speak to my colleagues about open source programs, their first questions are on - how easy is it to manage, and how easy is it to deploy. For something that requires configuration changes multiple times a day on multiple servers, responding that "you manage it from the command line" is not a valid option.

This i

Re:I'm a firewall admin amongst other things..

[hdw]

PF is not hard to understand and distributing common rules and specific rules is super easy and secure, with tools that come with a default install of OpenBSD (scp).

I have no problem understanding pf rules or distribution via scp (or cvs, works very well).

But it's not about understanding pf rules, it's about keeping track of, often hairy, network and system topology, of various security policies and in many cases a horde of users that need authentication (and that forget their PINs, break their tokens,

Conterpoint: Cisco PIX

[^BR]

Cisco PIXes are configured the old way thru SSH (ok, there's a Web interface, never heard of anyone using it) and they sell pretty well. Cisco do have a (laughable) management solution that includes a GUI but almost nobody use it as it plain sucks (simply installing it is a nightmare, plen,ty of dependencies...). The nice thing is that it provides a nice market for third party solutions to do that job...

So having a GUI is not a prerequisite for enterprise acceptance. Even if being Cisco sure helps...

Re:Conterpoint: Cisco PIX

[sirket]

I configure PIX's all day long and I love the simplicity of a PIX config file. That said, Cisco has been losing market share for years because they don't have a GUI. Ever try to set up a ton of VPN's through the command line? Doable? Certainly. Fun? Not a chance.

-sirket

Re:Conterpoint: Cisco PIX

[subzerorz]

Cisco PIX's GUI interface is actually very user friendly. It's a big improvement especially for Day to day administration.

Re:I'm a firewall admin amongst other things..

[_termx23]

If you want to run FWBuilder on Windows, install Cygwin/X. SSH to the firewall (with X11 forwarding turned on) and you are in business. I bet you could set this up with keys etc to the point that you could double click an icon on the Windows desktop and Firewall Builder would display in the Cygwin/X window.

CARP also works on Linux, NetBSD and OpenBSD 3.5

[chrysalis]

Try UCARP [ucarp.org] a portable userland implementation.

Re:CARP also works on Linux, NetBSD and OpenBSD 3.

[Lennie]

However there is no pfsync (or similair) for netfilter (if you'd like to have failover-firewalls).

But supposedly it doesn't matter, because netfilter doesn't have TCP window tracking.

And because existing connections are considered new by netfilter, it should work in theory (if you allow new connections, for all the established-connections).

Balancing won't work however, because UCARP doesn't do that, if I understand it correctly.

As there is no replication, rules should be replicated an other way (somethi

CARP/pf song for 3.5 Release

[RupertJ]

In keeping with OpenBSD's promo songs, the 3.5 release features a Monty Python-style sketch and song about CARP/pf and VRRP etc. Very funny stuff indeed. Lyrics and links to download the songs in MP3/OGG format at http://www.openbsd.org/lyrics.html [openbsd.org]

Interview with Ryan McBride

[dhartmei]

Jeremy Andrews from kerneltrap.org has just published an Interview with Ryan McBride [kerneltrap.org], which makes for an excellent read on CARP and pfsync.