Problems with machine response and large amounts of RAM

I have over 64 MB in my Pentium or greater machine. However, it seems sluggish when doing anything. If I tell Linux to use only 64 MB with the mem=64M option in lilo.conf file and rerun lilo -V, it seems to speed up the machine. What is going on and can I do anything about it?
