www.zeroshell.org Forum Index www.zeroshell.org
Linux Distribution for server and embedded devices
 
 SearchSearch  RegisterRegister  UsergroupsUsergroups 
 ProfileProfile  Log inLog in  Log in to check your private messagesPrivate Message 

Multiple ISP, servers on LAN

 
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> Signal a BUG
View previous topic :: View next topic  
Author Message
atheling



Joined: 24 Sep 2009
Posts: 212

PostPosted: Sun Nov 29, 2009 4:20 am    Post subject: Multiple ISP, servers on LAN Reply with quote

Please excuse my long post. Also, this is my first serious experince with iptables and ip rules. So I might be entirely wrong about the issue.

The environment:
I have a Soekris net5501 with an extra NIC interface giving a total of five ethernet interfaces. ETH00 and ETH01 are connected to two different ISPs. ETH00 is connected to a DSL modem that uses PPPoE while ETH01 has a fixed IP. All the other interfaces connect to different local LAN networks.

Looking at the output of "ip rules" and "ip route", I see that Zeroshell has setup routing realm 101 (0x65) for one ISP and realm 102 (0x66) for the other (at the moment I'm setup for failover but I recall loadbalance was the same):

Code:

root@zeroshell root> ip rule list
0:   from all lookup local
32764:   from all fwmark 0x66 lookup 102
32765:   from all fwmark 0x65 lookup 101
32766:   from all lookup main
32767:   from all lookup default
root@zeroshell root> ip route list table 101
nn.nn.nn.180/30 dev ETH01  proto kernel  scope link  src nn.nn.nn.181
10.7.52.0/24 dev ETH04  proto kernel  scope link  src 10.7.52.1
10.7.53.0/24 dev ETH03  proto kernel  scope link  src 10.7.53.1
10.7.54.0/24 dev ETH02  proto kernel  scope link  src 10.7.54.1
192.168.250.0/24 dev VPN99  proto kernel  scope link  src 192.168.250.254
10.4.27.0/24 dev ETH00  proto kernel  scope link  src 10.4.27.1
default via nn.nn.nn.182 dev ETH01
unreachable default  metric 99
root@zeroshell root> ip route list table 102
nn.nn.nn.180/30 dev ETH01  proto kernel  scope link  src nn.nn.nn.181
10.7.52.0/24 dev ETH04  proto kernel  scope link  src 10.7.52.1
10.7.53.0/24 dev ETH03  proto kernel  scope link  src 10.7.53.1
10.7.54.0/24 dev ETH02  proto kernel  scope link  src 10.7.54.1
192.168.250.0/24 dev VPN99  proto kernel  scope link  src 192.168.250.254
10.4.27.0/24 dev ETH00  proto kernel  scope link  src 10.4.27.1
default dev ppp0  scope link
unreachable default  metric 99
root@zeroshell root>


Looking at the mangle tables, ZS takes the routing realm for a new connection and saves it with the connection. When a new packet is received it restores the mark from connection to the packet:

Code:

Chain PREROUTING (policy ACCEPT 319K packets, 200M bytes)
 pkts bytes target     prot opt in     out     source               destination
 319K  200M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore
 319K  200M NetBalancer  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 316K packets, 200M bytes)
 pkts bytes target     prot opt in     out     source               destination
31739 3511K NB_CT_POST  all  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW
 316K  200M NB_STAT    all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain NB_CT_POST (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           realm 0x66 MARK set 0x66
 7895  591K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           realm 0x65 MARK set 0x65
31739 3511K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save


The end result of all this is that for connections initiated within the LAN, the intial routing decision is recorded and subsequent packets for that connection will use the same external interface. This is good.

The problem is when you have inbound traffic. I host a mail and web server, available through either link. My two links are quite different on speeds (the cable modem over ETH01 is 10x faster than my DSL) or if I am doing fail over the default route will always be over ETH01 for new connections. So traffic coming in over ppp0 will be responded to over ETH01.

The fix for this is to set the routing preference for new connections from the ISPs in the mangle prerouting chain. I've done it here by adding my own custom chain:

Code:

Chain PREROUTING (policy ACCEPT 319K packets, 200M bytes)
 pkts bytes target     prot opt in     out     source               destination
 319K  200M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore
 319K  200M NetBalancer  all  --  *      *       0.0.0.0/0            0.0.0.0/0
15687 2119K custom_preroute  all  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW

Chain custom_preroute (1 references)
 pkts bytes target     prot opt in     out     source               destination
 1963  113K MARK       all  --  ETH01  *       0.0.0.0/0            0.0.0.0/0           MARK set 0x65
 1963  113K CONNMARK   all  --  ETH01  *       0.0.0.0/0            0.0.0.0/0           CONNMARK save
 1630 79635 MARK       all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0           MARK set 0x66
 1630 79635 CONNMARK   all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0           CONNMARK save


Before I created this custom mangle prerouting rule attempts to connect via the backup link failed. With these rules it works perfectly.

It looks like the nb_setautomarking script would be the place to set this up. But when I explore the contents of /var/register/system/net/nb/Gateways/01/Interface I don't see ETH01 (I do see ppp0 in Gateways/02/Interface). So it is not clear to me where the correct location is to get the interface name needed to generate these rules.

There is an issue in the QoS table because it uses the same marking system as this routing. The above fix for inbound connections from the backup ISP exacerbates the QOS bug. But that is a topic of another bug which I'll post separately in a little while.
Back to top
View user's profile Send private message
fulvio
Site Admin


Joined: 01 Nov 2006
Posts: 1042

PostPosted: Sun Nov 29, 2009 6:26 am    Post subject: Reply with quote

Thanks a lot for the suggestion. Sureli I'll include it in the next release.
QoS and load balancer/failover cannot work simultaneously. You should use separately boxes.

Regards
Fulvio
Back to top
View user's profile Send private message Send e-mail
RDelpopolo



Joined: 04 Jul 2010
Posts: 21

PostPosted: Wed Jul 07, 2010 10:40 am    Post subject: Reply with quote

Can you explain easily how to solve that problem?
Back to top
View user's profile Send private message
atheling



Joined: 24 Sep 2009
Posts: 212

PostPosted: Wed Jul 07, 2010 1:44 pm    Post subject: Reply with quote

RDelpopolo wrote:
Can you explain easily how to solve that problem?


I've posted a patch that works for me and for some others. It did not get included in beta13 but the patch should work on that release too. See http://www.zeroshell.net/eng/forum/viewtopic.php?p=7600#7600 for more details.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> Signal a BUG All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group