[LastwagenMann]

Asymmetric Routing – It May Be The Solution
=================================
We were experiencing highly similar issues as the original poster, so I thought I would share my solution.

This method works well and is intended for those users that are focusing on increasing upload bandwidth only, as the goal is to use the bonded channel for upstream traffic only, and an Unbonded VPN Channel for the downstream, thus 2 x (25Mbps/5Mbps) will give you (25Mbps/10Mbps), which is a better ratio all together if upstream is your concern.

We are bonding two VDSL2 modems together from the same provider, each 25/5, however the upload performance was actually capped at 7Mbps as shown on the modem diagnostics page.

We had the same problem with excessive ACKs, except the problem was always traffic from the DC to the Bonded Site. Doesn’t matter what the source is. Traffic going from the Bonded Site to the DC is excellent! We are getting consistent 10.5-11.0Mbps of upload on TCP, and close to 12Mbps on UDP (test using iperf between bonded site and DC).

Downstream is horrible, any single connection would average 1.7Mbps and at best 2Mbps. Specifying -P 10 parameter to push 10 TCP connections resulted max 8.5Mbps, with WireShark showing excessive Duplicate ACKs, Fast Retransmissions, Previous Segment Not Captured, Packet out of Order.

As for the reason why, it likely has to do with the unpredictable nature of the internet, the QoS of your connection, congestion, the underlying architecture of the internet backbone, and the peering capacity between your two endpoints.

In our case, the peering capacity between the VDSL Provider and the DataCenter Provider isn’t the greatest. The download consistency is better than the upload at the DC end, which is on 100Mbps connection, but the CIR/Committed Information Rate we purchased was 10Mbps, so it is burstable rather than dedicated 100Mbps.

The algorithm used to control the connection at times of high congestion works by for example, allowing a 10ms burst of traffic at 100Mbps, followed by a 90ms pause, instead of a consistent stream of 10Mbps, which really messes up TCP.

The bonded DSL lines on the other hand allows for constant streaming up to it’s capped capacity. Because the modem is capped at 7Mbps, it will not have an opportunity to burst way beyond that, pause, and burst again to average 7Mbps.

To help troubleshoot, it is a good idea to measure the upstream/downstream consistency of the underlying connections. Visualware offers a free online speed test that graphs transmission speeds, and TCP forced delays.

So the connection ended up being 1.7/10 instead of 50/10 (minus the overhead). However our goal is increasing the upload and download is fine as-is at 25Mbps.

The solution was asymmetric routing:

66.119.x.y [ZeroShell Bonded]========[ZeroShell DC] 66.119.x.x

At ZeroShell DC:

---

bond0 = 10.32.161.10 (Slaves VPN00, VPN01)
vpn02 = 10.32.162.10

Static Route:
–>> ip route 66.119.x.y 255.255.255.248 10.32.162.20 [vpn02]

At ZeroShell Bonded Site:

---

bond0 = 10.32.161.20
vpn02 = 10.32.162.20

Static Route:
–>> ip route 0.0.0.0 0.0.0.0 10.32.161.20 [bond0]

Thus we are only routing upstream traffic through the bonded interfaces, and all downstream traffic thru vpn02 (unbonded). After doing that, we effectively got very close to 25/14Mbps – of course minus overhead with the occasional retransmission and out of order packet which is true of any network -bonded, vpn or not- so actual performance was 24/12Mbps.

The underlying OpenVPN Links were UDP. Just FYI doing TCP over TCP doesn’t make sense and as proof of theory, an experiment resulted in transfer speeds under 2Mbps/2Mbps and a whole array of serious TCP errors not only on downstream but upstream as well seen in WireShark.

Hope this helps someone solve their frustrations! 😀

[Author]

Posts