www.zeroshell.org Forum Index www.zeroshell.org
Linux Distribution for server and embedded devices
 
 SearchSearch  RegisterRegister  UsergroupsUsergroups 
 ProfileProfile  Log inLog in  Log in to check your private messagesPrivate Message 

Dramatically improve HAVP performance and save CF lifetime

 
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> ZeroShell
View previous topic :: View next topic  
Author Message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Sun Jan 24, 2010 10:03 pm    Post subject: Dramatically improve HAVP performance and save CF lifetime Reply with quote

Hi Folks,

This is a simple guide on how to dramatically improve HAVP transparent proxy performance while extending the lifetime of your Flash Memory medium)

Target audience: This guide is directed to those who use the Compact Flash image of Zeroshell, but may also be useful by those who boot Zeroshell from a CD and have the Database in a pendrive or other flash memory based medium (in this case, for cdrom installation, it may require minor adjustments - not tested)

In its original ZS compact flash image, HAVP is configured to write the temporary files (the ones yet to be scanned) into the "/Database/var/register/system/havp/tmp" folder which resides into the writable partition of the Compact Flash.
This original configuration is very performance impacting because writing to Flash Memory is way too slow. In addition, this also tends to reduce the life time of the medium since Flash Memory is subject to aging in the proportion of the number of writings.
This guide aims to implement a workaround for these issues by enabling a on-memory scan, thus also avoiding Compact Flash frequent writes.


Overall steps (UPDATE: The below guide became obsolete. A better approach is described in the 3rd post of this topic):
======================================================================================


Step 0 - Disable your HAVP proxy using the GUI, in the case it is currently enabled.

Step 1 - Create a 50MB ext2 file-system virtual medium:
    > cd /Database
    > dd if=/dev/zero of=HAVP.ext2 count=100000
    > mkfs.ext2 HAVP.ext2
    (answer yes since it will complain that HAVP.ext2 is not a block device)

NOTE: This will be a ram disk, so there is no need to have a journaled filesystem like ext3, thus ext2 should be fine.

Step 2 - Prepare the HAVP.ext2 directory ownership for havp:
    > mount -o loop HAVP.ext2 /mnt
    > chown havp.havp /mnt
    > umount /mnt

Step 3 - These commands have to be both executed in the Zeroshell's shell AND added to your pre-boot scripts:
    > cat /Database/HAVP.ext2 >/dev/ram3
    > mount -omand,noatime /dev/ram3 /Database/var/register/system/havp/tmp

NOTE: Be careful to not mistype it as /dev/ram2, which is your root ram filesystem Wink

Step 4 - Re-enable your HAVP proxy using the GUI

Have fun...


Performance Gains:
==============


I didn't use any tool to benchmark this, but I set a configuration for the Opera Browser to, at startup and with disk cache off, open several sites with tons of hundreds of images and started it twice, each time with one configuration, i.e., with this ram fs mounted and with it unmounted. Using the ram disk, I could notice a 10+ times performance increase.
Of course your performance gain will depend on the processing power of your box as well as the Compact Flash quality/speed.


Additional Considerations:
===================


The disadvantage of this on-memory scan approach is that it permanently consumes 50MB of ram, although I'm using this in my 512MB Fit-pc Slim (a GeodeLX 500MHz based gadget) with no problems.
I'm not yet sure if 50MB is a good value for the ram disk and, in the future, I may edit this post after some fine tunning.

Good luck!


Last edited by Marcelo on Wed Jan 27, 2010 1:00 am; edited 4 times in total
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Mon Jan 25, 2010 10:53 am    Post subject: Reply with quote

Improving the guide with a bit of diskspace optimization:

Add a new step between steps 2 and 3:
Quote:
Step 2.1 - gzip the HAVP.ext2 image:
    > gzip HAVP.ext2


Modify step3 accordingly:
Quote:
Step 3 - These commands have to be both executed in the Zeroshell's shell AND added to your pre-boot scripts:
    > gzip -dc /Database/HAVP.ext2.gz >/dev/ram3
    > mount -omand,noatime /dev/ram3 /Database/var/register/system/havp/tmp
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Wed Jan 27, 2010 12:43 am    Post subject: Improving the improvement :) Reply with quote

Hi Folks, I've just learned more about tmpfs.... Smile

...and this simply made my original guide (the first post in this topic) obsolete.

I'll describe here a newer and better (yet simpler) approach using tmpfs.
Using tmpfs solves the permanent memory allocation downside of the initial approach and makes the virus scanning even quicker.

The new guide:
===========


Step 0 - Disable your HAVP proxy using the GUI, in the case it is currently enabled.

Step 1 - Undo the original approach (This step is only needed IF you have implemented the first approach described in the first post of this topic, otherwise skip it and jump right to step 2):
    > cd /Database
    > rm HAVP.ext2
    > umount /Database/var/register/system/havp/tmp

    [ edit the pre-boot script and remove the lines you added there as part of the original guide ]

    NOTE1: Unfortunately, unless if you manage to get a copy of the "busybox" tool (which BTW is not that difficult to obtain), there is not an easy way to free up the space allocated by the ram disk mount used by the original approach, so, you will have to reboot your zeroshell box to free this memory. Fortunately the new approach will never suffer from the same problem.
    NOTE2: If you did manage to copy busybox to your Zeroshell instalation, instead of rebooting, just execute "busybox freeramdisk /dev/ram3"

Step 2 - The following command have to be both executed in the Zeroshell's shell AND added to your pre-boot scripts (through the Zeroshell's GUI):
    > mount -omand,noatime,uid=havp,gid=havp,size=50m -ttmpfs none /Database/var/register/system/havp/tmp

Step 3 - Re-enable your HAVP proxy using the GUI.

Have fun...

Why is tmpfs better?
===============


tmpfs is a memory filesysytem derived from ramfs that resides in the vfs layer and which framework is the same used by the kernel for caching all files of all mounted file systems. So, tmpfs reduces the overhead of having a format like ext2, ext3, etc, needed by the /dev/ramX devices. In addition, comparing to the original approach, it reduces cpu utilization and memory accesses for every file access when compared to /dev/ramX devices.

Tmpfs eliminates the permanently allocated memory downside of using the traditional ram disks. The "size=50m" option just specifies the maximum limit size of the filesystem, not a permanent allocation. Tmpfs allocates memory on demand and only the necessary amount to hold the existing files. Every time a file is deleted or truncated, memory is freed.

For more information on tmpfs refer to the documentation found on the kernel sources tree (note: this is not installed in your Zeroshell box):
    Documentation/filesystems/ramfs-rootfs-initramfs.txt
    Documentation/filesystems/tmpfs.txt

Best regards,


Last edited by Marcelo on Sat Feb 06, 2010 12:50 am; edited 2 times in total
Back to top
View user's profile Send private message
Pit



Joined: 14 Jan 2010
Posts: 45
Location: Germany

PostPosted: Fri Feb 05, 2010 7:07 pm    Post subject: Reply with quote

Hi Marcelo,

my df looks:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/ram2 31729 15404 16325 49% /
/dev/hda2 140524 140524 0 100% /cdrom
shm 484588 0 484588 0% /dev/shm
/dev/hda1 13197 4900 7616 40% /boot
/udev/hda3 806928 185428 580508 25% /DB
/DB/_DB.001 806928 185428 580508 25% /Database
none 51200 0 51200 0% /Database/var/register/system/havp/tmp

HTTP Proxy refuses to start after the changes. Do you have any idea?

Pit
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Fri Feb 05, 2010 7:21 pm    Post subject: Reply with quote

df output looks fine.

Sounds like a permission problem, are you sure you have executed the chown step?

to check, run "ls -lad /Database/var/register/system/havp/tmp" from the shell. You should see (only the text in bold is important, the remaining text may differ a bit when you run):

drwxrwxrwt 2 havp havp 540 Feb 5 15:32 /Database/var/register/system/havp/tmp
Back to top
View user's profile Send private message
Pit



Joined: 14 Jan 2010
Posts: 45
Location: Germany

PostPosted: Fri Feb 05, 2010 8:36 pm    Post subject: Reply with quote

Hi ,
this is my ls:

drwxrwxrwt 2 havp havp 40 Feb 5 21:30 /Database/var/register/system/havp/tmp

I posted your code to /Database/var/register/system/startup/rc.local.
After reboot there is no tmpfs. I changed also to /bin/mount and /bin/chown. But no tmpfs. Are there other pre-boot scripts?

Pit
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Fri Feb 05, 2010 9:12 pm    Post subject: Reply with quote

Ok, my tutorial could be clearer about what pre-boot scripts I'm refferring to: Pre-boot scripts are accessible from the Zeroshell's GUI under Setup -> Startup / Cron -> pre-boot scripts.

Editing rc.local in the filesystem won't get persisted after reboot as the root filesystem is a ramdisk on Zeroshell.

Getting back to your original problem: Did it work after fixing the directory permissions? If not, what is the output from the "free" command?
Back to top
View user's profile Send private message
Pit



Joined: 14 Jan 2010
Posts: 45
Location: Germany

PostPosted: Fri Feb 05, 2010 10:04 pm    Post subject: Reply with quote

No chanche, nothing works.

Here is my screenshot:

[/img]http://alstercom.de/img/scripting-editor.png


Pit
Back to top
View user's profile Send private message
Pit



Joined: 14 Jan 2010
Posts: 45
Location: Germany

PostPosted: Fri Feb 05, 2010 10:11 pm    Post subject: Reply with quote

And the output of free:

root@zeroshell root> free
total used free shared buffers cached
Mem: 969180 132216 836964 0 12108 36320
-/+ buffers/cache: 83788 885392
Swap: 131064 0 131064
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Fri Feb 05, 2010 11:46 pm    Post subject: Reply with quote

Ok, no reason for the previous commands not to work, but I edited the guide to make it simpler and replaced

Quote:
> mount -omand,noatime,size=50m -ttmpfs none /Database/var/register/system/havp/tmp
> chown havp.havp /Database/var/register/system/havp/tmp

with

Quote:
> mount -omand,noatime,uid=havp,gid=havp,size=50m -ttmpfs none /Database/var/register/system/havp/tmp

This will always make sure the ownership definition of the directory isn't skiped as it has been directly incorporated in the mount command (no more chown needed).

I also added a note that the pre-boot scripts I'm referring to is the Zeroshell's GUI one
Back to top
View user's profile Send private message
Pit



Joined: 14 Jan 2010
Posts: 45
Location: Germany

PostPosted: Sat Feb 06, 2010 12:49 am    Post subject: Reply with quote

Solved:

- Never never use Opera Browser for administering Zeroshell web-gui.

- Firefox works fine for me.

- Enter the new code from Marcelo into Setup/Startup-Cron/Pre-Boot.
- Check the status button and save.
- Reboot.

- Start HTTP Proxy from the gui. Checkbox is enabled andStatus shows DOWN.
- Reboot.
- HTTP Proxy Status shows ACTIVE

Thanks a lot for your help.

Pit
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Sat Feb 06, 2010 12:57 am    Post subject: Reply with quote

You are welcome.

It doesn't really require any reboot. Just execute the mount command both in the ZS shell and in the pre-boot script. If you are configuring your home router, rebooting is certainly a good way of testing that the pre-boot scripts are really working (or if you can afford doing this in a produciton router of your organization - in a maintenance window, maybe...).

I'm happy it is working for you now. Enjoy your new "boosted speed" proxy.
Back to top
View user's profile Send private message
apatheticsheep



Joined: 23 Feb 2010
Posts: 11

PostPosted: Tue Mar 30, 2010 2:45 pm    Post subject: Reply with quote

I tried this on my new zeroshell box because web browsing was painfully slow, even after I had disabled image scanning.

after following this procedure the proxy starts fine.

however there is absolutely no performance increase.

the proxy is still mind-numbingly slow.

I am running a p4 1500mhz with 128MB ram.

note: the only reason I am using the proxy is for the url blacklist, not the a/v scanning.
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Tue Mar 30, 2010 2:55 pm    Post subject: Reply with quote

Hi

please, post here the output of your df command when typed in the router's shell.
Back to top
View user's profile Send private message
apatheticsheep



Joined: 23 Feb 2010
Posts: 11

PostPosted: Tue Mar 30, 2010 2:59 pm    Post subject: Reply with quote

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/ram2 31729 15349 16380 49% /
/dev/hda2 140524 140524 0 100% /cdrom
shm 62848 0 62848 0% /dev/shm
/dev/hda1 13197 4900 7616 40% /boot
/udev/hda3 806928 273828 492108 36% /DB
/DB/_DB.001 806928 273828 492108 36% /Database
none 51200 0 51200 0% /Database/var/register/system/havp/tmp
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Tue Mar 30, 2010 3:02 pm    Post subject: Reply with quote

Seems correct.

Interesting that it didn't improve performance for you. At least it is preserving you CF life (in case you use a CF installation).

In that case, I see no other suggestion to you other than disabe the image scanning as you are not interested in the AV anyway.

BTW, chances are you are facing some other problem not directly related to the proxy itself.

Regards,
Marcelo Vianna
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Tue Mar 30, 2010 3:05 pm    Post subject: Reply with quote

I also see you have a very limited amount of RAM (128MB). In this circumnstance, even having On-Memory-Scan enabled probably is forcing the system to swap this memory, so disk access will be performed anyway.

Suggest increasing memory (at least to 256MB), if possible.
Back to top
View user's profile Send private message
apatheticsheep



Joined: 23 Feb 2010
Posts: 11

PostPosted: Tue Mar 30, 2010 3:06 pm    Post subject: Reply with quote

Not sure if i stated it in my original post but I have already disabled image scanning.
Back to top
View user's profile Send private message
apatheticsheep



Joined: 23 Feb 2010
Posts: 11

PostPosted: Tue Mar 30, 2010 3:08 pm    Post subject: Reply with quote

as far as the small amount of ram is concerned, I realize that 128mb is not alot, however as I stated in my original post I am only using the proxy for url blocking, if there is a way to disable the A/V scanning that would be ideal.
Back to top
View user's profile Send private message
Marcelo



Joined: 23 Jan 2010
Posts: 41

PostPosted: Tue Mar 30, 2010 3:56 pm    Post subject: Reply with quote

Understand, unfortunately, as far as I know, this is not possible.

Anyway, if your blacklist is not too big and doesn't make extensive use of wildcards, I believe that it would be preferrable to exclude the access using the IPs via firewall rules. This is specially important because your users might know the URL's IPs and skip your blacklist while firewall rules are more effective.

Good luck
Back to top
View user's profile Send private message
giancagianca



Joined: 23 Aug 2007
Posts: 36

PostPosted: Wed Mar 31, 2010 12:29 am    Post subject: Reply with quote

you can use opendns for Web content filter and parental control-

http://www.zeroshell.net/eng/opendns/

bye-
Back to top
View user's profile Send private message
apatheticsheep



Joined: 23 Feb 2010
Posts: 11

PostPosted: Thu Apr 01, 2010 7:32 pm    Post subject: Reply with quote

I have located a number of cidr blocks that I can block in the FORWARD chain to affect my blocking desires thank you for your input.

I will also consider using opendns, but I am not sure that I wish to use a 3rd party service for this. not that i do not believe it would work very well, but I would prefer to keep everything in one place.

I have a post in another topic regarding vpn if anyone would care to help me out with that.

http://www.zeroshell.net/eng/forum/viewtopic.php?t=2119
Back to top
View user's profile Send private message
Jiellen29



Joined: 20 Oct 2010
Posts: 5

PostPosted: Wed Oct 20, 2010 2:25 am    Post subject: Reply with quote

Thank you so much for the help,i really appreciate it,btw i'm newbie here too...

Thanks a lot aand God bless!!!
Back to top
View user's profile Send private message Yahoo Messenger
Rover



Joined: 02 Mar 2011
Posts: 6

PostPosted: Mon Mar 07, 2011 4:26 pm    Post subject: Reply with quote

well, i tried this tutorial, but there is no perfomance increase.

Im using a Alix 2d3 board with a CF Kingston 4 GB.

Sad
Back to top
View user's profile Send private message
siloen



Joined: 14 Mar 2012
Posts: 2

PostPosted: Wed Mar 14, 2012 1:07 pm    Post subject: Reply with quote

The tutorial hasn't worked for me either...
I have 1x Alix 2d3 Kit and 1GB CF card

Regards,
siloen

_____________________________
spy phone
Back to top
View user's profile Send private message
cyberman



Joined: 15 Feb 2013
Posts: 1

PostPosted: Fri Feb 15, 2013 11:27 am    Post subject: Reply with quote

I found a workaround to use UrlFiltering but "disable" che ClamAV.

Edit the HAVP configuration file at:
/root/kerbynet.cgi/template/havp.config

Change the MAXSCANSIZE from 5000000 to 1

The scan size is now 1 byte and speed up the Antivirus engine.
Back to top
View user's profile Send private message
Utteru



Joined: 19 Feb 2015
Posts: 10

PostPosted: Wed Apr 08, 2015 9:01 am    Post subject: Reply with quote

Thank you for sharing much nice information here.
Back to top
View user's profile Send private message
DiliaK



Joined: 26 Aug 2015
Posts: 11

PostPosted: Wed Aug 26, 2015 12:10 am    Post subject: Reply with quote

Rover wrote:
well, i tried this tutorial, but there is no perfomance increase.

Im using a Alix 2d3 board with a CF Kingston 4 GB.

Sad


Hello,

I use same board. If you dont see a performance increasing, your CF card say : "oh, i will be able to be used more time !".

I found it on the web that you should add to preserve CF cards :

Code:

mount -t tmpfs -o size=64m,mode=1777,nosuid,nodev,exec tmpfs /tmp
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /var/run
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /var/lock
mount -t tmpfs -o size=64m,mode=755,nosuid,nodev tmpfs /Database/LOG
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /Database/var/register/system/mrtg/counters
mount -t tmpfs -o size=32m,mode=755,nosuid,nodev tmpfs /Database/var/register/system/mrtg/html


Add it at the end of your pre-boot after the lines proposed here...

DiliaK
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    www.zeroshell.org Forum Index -> ZeroShell All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group