Recently I had to step in urgently because a NetScaler deployment wasn’t working properly. When I tried to access the primary NetScaler of the HA pair over the web, I couldn’t get in, because of an invalid session error.
So I switched to a console connection to the virtual appliance, and right away I noticed a continuous stream of errors about a full filesystem.
Once in the shell, I ran the command “df -h” and it was clear from the output that the /var partition was full.
So I went ahead and cleaned up the filesystem, starting from the guidance in this official documentation page:
How to free space on the VAR directory for logging issues with a NetScaler appliance
Below are the directories that can be cleaned up:
/var/nstrace - This directory contains trace files.This is the most common reason for HDD being filled on the NetScaler appliance. This is due to an nstrace being left running for indefinite amount of time. All traces that are not of interest can and should be deleted. To stop an nstrace, go back to the CLI and issue stop nstrace command.
/var/log - This directory contains system specific log files.
/var/nslog - This directory contains NetScaler log files.
/var/tmp/support - This directory contains technical support files, also known as, support bundles. All files not of interest should be deleted.
/var/core - Core dumps are stored in this directory. There will be directories within this directory and they will be labeled with numbers starting with 1. These files can be quite large in size. Clear all files unless the core dumps are recent and investigation is required.
/var/crash - Crash files, such as process crashes are stored in this directory. Clear all files unless the crashes are recent and investigation is required.
/var/nsinstall - Firmware is placed in this directory when upgrading. Clear all files, except the firmware that is currently being used.In particular, firmware files are often left behind in the /var/nsinstall folder after NetScaler upgrades, but in this specific case I found that the logs in the /var/log folder were larger than usual (I found an ns.log2 file more than 6GB in size).
Looking into the system, I noticed that debug-level logging had been enabled a few days earlier, which is why the logs being generated were larger than usual. This NetScaler had been examined by Citrix support, who had enabled Loglevel DEBUG to gather more details on the issue they were investigating; too bad they then forgot to remove the Loglevel, along with several other log files scattered around the system.
So I disabled Loglevel DEBUG, to restore normal logging, with the following command:
Set syslogparams -loglevel EMERGENCY ALERT CRITICAL ERROR WARNING NOTICE INFORMATIONALThis way I told the NetScaler to log only events of the levels in the list, excluding all DEBUG events.
After cleaning up several folders, I was able to confirm that my work had paid off by running the “df -h” command again:
Finally, the partition containing /var was a good 57% free.
NB: on Linux/UNIX systems (and NetScaler is based on FreeBSD), running “df -h” right after deleting files may not immediately reflect the new state of the filesystem. You often need to wait and run the command again after several minutes to actually see the updated usage.