Jails and loopback interfaces

I've got a server running FreeBSD 6.2 and PF. The server has
a couple  
dozen jails on it. Previously, I had a few
"private" services such as  
MySQL running on loopback IPs (127.0.0.2+) and the rest of
the jails  
running on the public IPs.

I have to renumber my machine with a new block of public IPs
so I  
thought I'd be clever and move all the jails onto loopback
IPs. Then  
I could use PF to redirect the new IPs and the old ones to
the  
corresponding jails.

The relevant parts of my PF config look like this:

> ext_if="em0"
> loop_if="lo0"
>
> ext_addr="72.29.111.130"
> loop_addr="127.0.0.101"
>
> ext_net ="72.29.111.128/27"
> ext_net2="208.75.180.64/27"
> loop_net="127.0.0.0/8"
>
> lo_simerson     = "127.0.0.4"
> lo_toaster      = "127.0.0.6"
> lo_tnpi         = "127.0.0.13"
>
> simerson     = "72.29.111.131"
> toaster      = "72.29.111.133"
> tnpi         = "72.29.111.140"
>
> # nat the jails running on loopback to their public ip
> nat on $ext_if from $lo_simerson     to any ->
$simerson
> nat on $ext_if from $lo_toaster      to any ->
$toaster
> nat on $ext_if from $lo_tnpi         to any ->
$tnpi
>
> # specific jail rules
> rdr on $ext_if from any to $simerson     ->
$lo_simerson
> rdr on $ext_if from any to $toaster      ->
$lo_toaster
> rdr on $ext_if from any to $tnpi         ->
$lo_tnpi

This works just great. Traffic from the outside world gets
redirected  
to the correct jail via a rdr rule. The corresponding nat
rule alters  
the source IP from each jail to its corresponding public
IP.

The problem I have with this arrangement is when a jail
attempts to  
connect to the public IP of another jails, the connection
fails. So,  
a client running in one jail can't send email to my mail
server  
running in another jail. This problem is described in detail
here:

     http://www.ope
nbsd.org/faq/pf/rdr.html

And I have attempted to configure the RDR and NAT
combination  
solution as shown. But after trying a few dozen iterations,
I can't  
get it to work. Based on my understand of the PF manual,  
conceptually, the problematic connection looks like this:

    simerson (127.0.0.4)  -> toaster public IP
(72.29.111.133)  
rewritten to toaster private IP (127.0.0.6)
    toaster (127.0.0.6) -> simerson (127.0.0.4)

So the simerson jail gets returned a packet from the
toasters  
loopback IP, which it ignores because it is not the IP it
sent the  
connection attempt to.

But no matter what I try, I can't make it work. PF is not
denying any  
connections. Without resorting to split-horizon DNS, has
anyone  
gotten this to work reliably on FreeBSD?   I'd love to see a
working  
configuration.

Matt

_______________________________________________
freebsd-securityfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-secu
rity
To unsubscribe, send any mail to
"freebsd-security-unsubscribefreebsd.org"

On Mon, 2007-07-02 at 12:43 -0500, Matt Simerson wrote:
> The problem I have with this arrangement is when a jail
attempts to  
> connect to the public IP of another jails, the
connection fails. So,  
> a client running in one jail can't send email to my
mail server  
> running in another jail. 


You can try keeping up-to-date version of /etc/hosts with
hostnames of
public services pointing to you 127.0.0.2+ IPs. It's dirty,
but at least
keeps your pf.conf clean as much as possible.
It works for me and it should for you if you dont move
services around a
lot. Then it requires lots of recursive changes in each jail
if you move
some service from one IP to another.




_______________________________________________
freebsd-securityfreebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-secu
rity
To unsubscribe, send any mail to
"freebsd-security-unsubscribefreebsd.org"