Understanding Disk Space Utilization

Like any other system resource, disk space is limited. Andrisoft Wanguard can collect and store very large quantities of data, depending on its configuration. If you think it is using up too much disk space, read through this article to understand what might be the cause and how to prevent the disk from filling up in the future. When the disk is full on the Console server, it is no longer possible to log in to the application and some components will fail to run.

Unless you changed the default paths for storing flows, graphs and dump files, you can use the ncdu utility or the following command to see what is using up your disk space (it may take up to a few minutes, depending on how many files you have):

du -ch -d 1 /var/lib/mysql/andrisoft /opt/andrisoft /var/lib/influxdb/data/andrisoft /var/lib/clickhouse

If the /var/lib/mysql/andrisoft directory occupies too much disk space, lower the parameters in Configuration » General Settings » Data Retention. The database is purged of old data once every hour, so changes to Data Retention will not be visible immediately. Also, on some distributions, MySQL/MariaDB will not free the deleted data and will instead reserve the resulting disk space for future use. To reclaim the newly freed disk space, execute the following command on the Console server (it might take a very long time to complete):

/opt/andrisoft/bin/WANmaintenance optimize_db

If the disk filled up completely, it is very likely that the database got corrupted and cannot be started. If this is the case, free up some disk space and execute:

/opt/andrisoft/bin/WANmaintenance repair_db

If the /opt/andrisoft/dumps directory occupies too much disk space, you have either taken large packet dumps manually, or configured the software to capture large packet dumps in Configuration » Network & Policy » Responses.

• Configure the software to delete old pcap files automatically in Configuration » General Settings » Data Retention » Packet Dumps.
• Recheck the configuration of the Traffic Sample Capturing Action side by side with the User Manual, which explains each field in detail. Configuring the Max. Packets parameter is the best way to avoid unnecessarily large dump files.
• If you need to free some disk space immediately, you can manually delete any files or directories from within /opt/andrisoft/dumps — but not the dumps directory itself. If you delete directories or files manually, let the software recreate them automatically; do not attempt to recreate them yourself.

If the /opt/andrisoft/flows directory occupies too much disk space, the Flow Sensor is receiving lots of flows and you have chosen to save them by enabling the Flow Collector option.

• As a first step to lower the disk space consumed by flows, select the "Save compressed flows" option in the Flow Sensor Configuration window. More information can be found here.
• Configure the software to delete old flows automatically in Configuration » General Settings » Data Retention » Flow Collectors.
• If you need to free some disk space immediately, you can manually delete any files or directories from within /opt/andrisoft/flows — but not the flows directory itself. If you delete directories or files manually, let the software recreate them automatically; do not attempt to recreate them yourself.

If the /opt/andrisoft/graphs directory occupies too much disk space, you have either configured the software to generate graph data files for a very large number of IPs, or configured the graph data files to contain an unnecessarily large amount of information.

• Since version 7.2 it is possible to use InfluxDB instead of RRD files to store graph data. InfluxDB uses less disk space, so you should seriously consider using it.
• To find out how much disk space each IP graph file requires, go to Configuration » General Settings » Storage & Graphs. Tweaking the Accuracy of the Round Robin Archive, the Decoders, the Stored Units or the Consolidation Functions modifies the IP graph file size, which is displayed dynamically in the bottom section of the Configuration window. A lot of unnecessary information may be stored for each IP, so it is best to reconsider the necessity of each option while consulting the User Manual here. The configuration applies to all IP graph files; it is currently not possible to choose different options for different IP prefixes.
• We strongly suggest disabling IP graphing for very large IPv4 prefixes or IPv6 prefixes in the IP Zone Configuration window (Configuration » Network & Policy » IP Zone). Depending on what you have configured in the Storage & Graphs window, the software calculates how much storage is required if you turn on IP graphing for any IPv4 prefix — see the "Storage Requirements" column of the IP Settings section in the IP Zone Configuration window. You will immediately see that turning IP graphing on for large prefixes is not a good idea.
• Important: the software generates IP graph files at their maximum size and updates them in place without changing the size afterward. You therefore cannot rely on the fact that an IP has little traffic and will not occupy much space: an IP graph file is created at the first packet that has that IP as a source or destination address. A simple network scan of a /8 prefix with IP graphing enabled can flood your hard disk with full-sized IP graph files.
• You can aggregate the IP graphing information collected by multiple Packet Sensors or Flow Sensor interfaces by using the Sensor Cluster component.
• If you need to free some disk space immediately, you can manually delete any files or directories from within /opt/andrisoft/graphs — but not the graphs directory itself — and let the software recreate them automatically. To delete the IP graph files that have not been updated in the last 90 days, execute:

  find /opt/andrisoft/graphs/ips -mtime +90 -type f -exec /bin/rm {} \;

If the /var/lib/influxdb/data/andrisoft directory occupies too much disk space, you probably configured the software to generate graph data for a very large number of IPs.

• The software can be configured to store graph data locally or on a remote server. If the existing Console server does not have enough disk space, you can use another server just for storing graphs.
• By default, the graph data is saved in InfluxDB in the "andrisoft" database. You can delete the existing database by executing DROP DATABASE andrisoft from the InfluxDB CLI, then recreate it in Configuration » General Settings » Storage & Graphs.

If the /var/lib/clickhouse directory occupies too much disk space, the software is storing flow data (Flow Sensors with ClickHouse flow storage enabled, which powers the interactive Flow Graphs) and/or graph data in ClickHouse. If you want to keep the data on another path or drive, symlink it to the original location.

• Flow records are held in four tables of decreasing detail — full per-flow records plus 1-minute, 5-minute and 1-hour aggregates — each aged out on its own retention, configurable in Configuration » General Settings » Data Retention. Keeping the raw detail short and the aggregates long lets you hold months of trend data while storing only a few days of full records. Each tier is aged out one whole partition at a time, so the Console rejects retention values below 7 days, and Flow Sensors that write to the same ClickHouse database share one retention policy.
• When the graphs are stored in ClickHouse, their retention is set by the storage profile in Configuration » General Settings » Data Retention » ClickHouse Graphs: Small (long-range archives dropped after 1 year), Balanced (up to 5 years, the default) or Large (up to 15 years).
• If you need to free some disk space immediately, run the following command on the Console server — it drops the oldest flow partitions (full detail first, aggregates last) until the flow tables fit the given size. Append dry_run to preview what would be deleted:

  /opt/andrisoft/bin/WANmaintenance clickhouse_trim_flows <bytes>[K|M|G|T] [dry_run]

• You can delete and re-create the graph database with the [Delete DB] and [Initialize DB] buttons of the ClickHouse Connector in Configuration » General Settings » Storage & Graphs.
• ClickHouse can also run on a separate server reachable over HTTP(S) — if the Console server does not have enough disk space, you can use another server just for storing flows and graphs.

If you require further assistance, please contact support andrisoft.com.