Checking size of one or more directories (System monitoring)

Sum over all files and additionally partial sum over more recent files.

Note: Can also be used on a DMZ server.

Parameters


Name

Description

dir

dir.allow

dir.deny

(MAX_LEVEL)

(optional) Single directory or comma-separated list of directories. Default: <Installation path of the Integration Server>.

Notes:

  • All relative directories under the Lobster installation directory may be specified in the comma-separated list of the dir parameter, without further permission. For directories leading out of the Lobster installation path, an additional permission rule is required in the configuration (dir.allow). A permission rule applies only to this directory, as a summation root. It does not allow the use of a subdirectory under the allowed external directory.

  • Permission for the root directory of a file system is not possible. Likewise, special directories of the operating system, from which one could guess the names or settings of user accounts, for example /home, are excluded.

  • Summation over / or C:/ is not allowed, because root directories are excluded.

  • Access to C:/Users and all subdirectories is forbidden, even if a permission rule exists for it.

  • Under Linux/UNIX there is a danger of endless recursion due to directory links. To avoid this, a property MAX_LEVEL can be specified in the security file (see below), which limits the maximum recursion depth into the subdirectories. However, this will only prevent a program crash due to infinite recursion. Directory links can lead to multiple counting of file sizes. The default value is MAX_LEVEL=100.

  • An additional property dir.deny is available, which has higher priority than dir.allow. If necessary, you can explicitly specify forbidden directories to prevent an erroneous permission in dir.allow.

Example:

dir.allow=/usr;/tmp;/usr/local;/sys;/proc;/usr/local/IS

dir.deny=/usr

In this configuration example, /tmp, /usr/local and /usr/local/IS are allowed in parameter dir.

Although /usr is allowed in dir.allow, it is prohibited by the rule dir.deny.

Although the paths /sys and /proc are in the permission rule, summation is not allowed there because they are pseudo-directories that do not really exist in the directory system.

cmd

(optional) Specification for partial sum. Files younger than n units (s, m, h, d), i.e. seconds, minutes, hours, days. If the value is negative, 'older than' applies. Several partial sum commands must be separated by commas.

recurse

(optional) false if no subdirectories are to be considered. Default: true.

parallel

(optional) true for summation in multiple low priority threads or false for summation in the current thread (maximum speed). Default: true.

nameRegex

(optional) Restricts the summation to files whose filename matches the specified regular expression, e.g. to distinguish between message and error logs.

Security configuration


Security configuration through optional file ./conf/dirsize/dirSize.config.


client.ip = regex:127\\..*;regex:192\\.168\\.21[367]\\..*;0:0:0:0:0:0:0:1
client.secure=false

Security configuration parameters


Name

Description

client.ip

(optional) Access allowed from these IPs. Default: localhost. You can use regular expressions with prefix regex: (see example above).

client.secure

(optional) false if HTTP is allowed. true, if only HTTPS is allowed. Default: true.

MAX_LEVEL

(optional) Maximum recursion depth into subdirectories. Default: 100. See parameter dir.

Call example


http(s)://<IP/URL Integration Server or DMZ>:<Port>/dw/monitor/v1/dirSize

Further examples:

http(s)://<IP/URL Integration Server>:<Port>/dw/monitor/v1/dirSize?dir=logs&cmd=1d&nameRegex=.*err.*&parallel=false
http(s)://<IP/URL Integration Server>:<Port>/dw/monitor/v1/dirSize?dir=logs&cmd=1d
http(s)://<IP/URL Integration Server>:<Port>/dw/monitor/v1/dirSize?dir=/tmp&cmd=+30d,-30d

Example result XML


Here for the call http(s)://<IP/URL Integration Server>:<Port>/dw/monitor/v1/dirSize?dir=logs&cmd=1d

<DIRSIZES>
<DIRSIZE>
<PATH>logs</PATH>
<SUM_MBYTES>140</SUM_MBYTES>
<totalSpace>6.4724250624E10</totalSpace>
<freeSpace>4.3113938944E10</freeSpace>
<recurse>true</recurse>
<DurationSec>0.129</DurationSec>
<SubSum>
<Label>all</Label>
<younger>true</younger>
<time>0</time>
<bytes>1.46772885E8</bytes>
<count>6208</count>
</SubSum>
<SubSum>
<Label>1d</Label>
<younger>true</younger>
<time>1615792154265</time>
<bytes>2.2280063E7</bytes>
<count>58</count>
</SubSum>
</DIRSIZE>
</DIRSIZES>

Meaning of the elements

Element

Description

DIRSIZES

The root element is called DIRSIZES, it has no attributes.

DIRSIZE

Under this element there is a sequence of or only one element DIRSIZE. If parameter dir has only one value, it is one element. In case of a comma separated list of directories there is one element DIRSIZE for each element of this list.

PATH

SUM_MBYTES

The PATH and SUM_MBYTES elements refer to a directory and optionally all its subdirectories if recurse=true.

totalSpace

freeSpace

The totalSpace and freeSpace elements refer to the file system where the subdirectory is physically located.

recurse

See parameter recurse and element PATH above.

DurationSec

DurationSec refers to the duration of the calculation of the file sizes of this directory and has at most informational value.

nameRegex

If a parameter nameRegex is specified in the call, an XML element nameRegex appears with the corresponding value. All summations then go only over the files that match the regular expression.

SubSum

Label

younger

time

One or more elements SubSum. There is always one element SubSum with element Label=all. This describes the sum of the byte sizes of all files over which the summation is done. The call may contain a cmd parameter with an age specification or a comma-separated list of age specifications. For each age specification another element SubSum is created, which then counts only those files from the set all, which correspond to the age specification.

The age information is returned as subelement Label. It can consist of an integer and a unit. Allowed units are s, m, h, d (seconds, minutes, hours, days). Value 1d describes the subset of all files whose last modification is less than 1 day (24 hours) ago. Positive values denote the younger files, negative values the older ones.

This fact is additionally marked by the subelements younger (with true or false) and time (timestamp in milliseconds since 01.01.1971).

byte

count

The elements byte and count indicate the sum size and the number of the captured files. All sizes (bytes) are represented as double values in scientific notation with exponent. The number (count) is an integer.


In the above example, the logs subdirectory under the Lobster installation directory and all its subdirectories (recurse) were searched. There was no selection by file name. The files take up about 140 Mbytes. The query took 129 ms. 6208 files were found with a total size of 146772885 bytes. In addition, a summation was performed over the subset of these files whose last modification was less than one day ago. This found 58 files with a total size of 22280063 bytes. The elements totalSpace and freeSpace also reveal that the logs subdirectory is physically in a file system that is about 64 GBytes in size, of which about 43 GBytes are still free.