Swapping

When a profile processes large amounts of data, a large amount of memory is required. If memory is running out, the overall performance is deteriorating and, in extreme cases, the profile terminates with an error (OutOfMemoryException). In order to counteract this, data can be swapped to the hard disk during the profile run, thus reducing the need for memory.

Records are swapped on the source and/or target side during the mapping. If no records were swapped during the parsing (phase 2), they remain in the main memory. It is not possible to retrospectively swap the records of the source structure.

Whether it should be swapped and how is configured individually for each profile. The settings can be found on page "Phase 2/Extended".

Settings


(1) Write data to disk temporarily: Specifies whether data should be swapped.

(2) On memory use of: Specifies the threshold for the virtual machine's memory usage, at which the swapping is to be triggered. The memory usage takes into account not only the memory consumption of the current profile but the entire memory usage, i.e. all profiles plus the program resources. If the threshold is 0, the swapping is started immediately. The following figure shows a schematic representation of the virtual machine's memory usage without swapping.

images/download/attachments/162409400/Image_018-version-1-modificationdate-1702361948718-api-v2.jpg

Lazy swapping



This is the history of the memory usage with lazy swapping .

images/download/attachments/162409400/Image_019-version-1-modificationdate-1702361948716-api-v2.jpg


(1) The threshold for the memory usage, e. g. 40 %.

(2) The threshold value was exceeded after x records. This is detected after completing the record and 1/3 (but not more than 100) of the records on the target side are swapped. If the memory usage is still too high, another third (but again a maximum of 100) of the records is swapped. This process is repeated until the memory usage has fallen under the threshold or all records have been swapped.

(3) Memory usage below the threshold after n repetitions of (2).

(4) After further y records, the threshold for the memory usage is exceeded again. The procedure in (2) is repeated.

(5) Memory usage below the threshold after another m repetitions of (2).


Since this method tries to swap only as many records as necessary, this method is called lazy swapping. This is the default swapping method.

If many profiles are started in parallel, the lazy swapping may no longer be able to ensure that sufficient memory is available. For this reason, busy swapping is offered as well.

Busy swapping


In the case of busy swapping, a check is performed after every record to determine whether the percentage memory usage within the virtual machine is greater than the set maximum. If this is the case, all records already created are swapped to the hard disk. The following figure shows the memory usage with busy swapping.



images/download/attachments/162409400/Image_020-version-1-modificationdate-1702361948714-api-v2.jpg




(1) The configured threshold value for the memory usage, e.g. 40 %.

(2) The threshold value was exceeded after x records. This is notoced after completing the current record and all created records are swapped.

(3) Memory usage after swapping all records.

(4) After further y records the memory usage threshold was exceeded again. All records are swapped again.


The disadvantage of busy swapping versus lazy swapping is that more records than necessary might be swapped. This can lead to performance losses during the further processing.

Configuration


The configuration of whether and how to swap is done separately for each profile. The configuration of which method to use for the swapping, on the other hand, is done globally for all profiles. The following listing shows the entry in the configuration file ./etc/startup.xml that activates the lazy swapping. To enable busy swapping, the value for parameter lazySwapping has to be changed from true to false.

Note: Switching from lazy swapping to busy swapping should only be done in exceptional cases and only in consultation with our support.

<!-- Enable lazy swapping of temp. data -->‌
<Set name="lazySwapping">true</Set>

Email notification


The system checks every 5 minutes whether there are jobs with a swap file >= 20 GB. These are reported once by email (→ ./etc/startup.xml). An internal flag is set to indicate that an e-ail has already been sent (regardless of the job). If there is no job with an exceeded limit during the next check, the flag is reset, otherwise the flag remains, but no new email is sent (even if new jobs have been added).