High system CPU, routeupd running multiple times

[squigley]

I have 4 WAN connections, and I run an openVPN tunnel over each. The system CPU load when running top gets so high that the system can’t keep up, and it seems to hang completely (won’t respond to ping), but then a little while later it will recover.

I should mention that this is a Neoware CA-10 thin client with a flash disk I am running ZS on:

VIA Nehemiah

Architecture:          i686

CPU op-mode(s):        32-bit

Byte Order:            Little Endian

CPU(s):                1

On-line CPU(s) list:   0

Thread(s) per core:    1

Core(s) per socket:    1

Socket(s):             1

Vendor ID:             CentaurHauls

CPU family:            6

Model:                 9

Stepping:              8

CPU MHz:               800.356

BogoMIPS:              1600.71

ie:

top - 23:56:34 up 12 days,  3:32,  1 user,  load average: 3.72, 3.95, 3.36

Tasks: 147 total,   5 running, 142 sleeping,   0 stopped,   0 zombie

Cpu(s):  44.4% user,  55.6% system,   0.0% nice,   0.0% idle,   0.0% IO-wait

Mem:    501644k total,   480428k used,    21216k free,   142876k buffers

Swap:   131068k total,     2852k used,   128216k free,   231296k cached

It appears to be related to the amount of traffic being passed over the VPN, since it causes the CPU load of the openvpn process to increase, when the system is already overloaded.

I did some experimentation, by turning off all but one vpn, and one route in the net balancer, and the system load dropped way down to about 30%, instead of the 300/400% it usually runs at.

Turning the routes back on, without the vpns, and the system load went up again. I noticed that there were multiple instances of routeupd running. I looked in this script, and it doesn’t make sense to me to have multiple copies of it running at the same time.

I added this quick dirty patch in the top to make it check to see if it is already running, and if so, just exit out, to block it running multiple times simultaneously:

pid=`ps auxww|grep routeupd|grep -v grep|wc -l|awk '{ print $1 }'`

if [ "$pid" -gt 2 ]; then

exit 1

fi

After doing this, I have been able to turn all 4 routes back on, and the 4 vpns, and the system load is slightly lower than it was before, and it hasn’t had the issue of getting so overloaded that it stops responding to ping.

[Author]

Posts