Swapping
When a profile processes large amounts of data, a large amount of memory is required. If memory is running out, the overall performance of Lobster_data is deteriorating and, in extreme cases, the profile terminates with an error (OutOfMemoryException). In order to counteract this, Lobster_data can swap data to the hard disk during the profile run, thus reducing the need for memory.
Records are swapped on the source and/or destination side during the mapping. If no records were swapped during the parsing (phase 2), they remain in the main memory. It is not possible to retrospectively swap the records of the source structure.
Whether it should be swapped and how is configured individually for each profile. The following screenshot shows the swapping settings in a profile .
(1) Specifies whether data should be swapped.
(2) Specifies the threshold for the virtual machine's memory usage, at which the swapping is to be triggered. The memory usage takes into account not only the memory consumption of the current profile but the entire memory usage, i.e. all profiles plus the program resources. If the threshold is 0, the swapping is started immediately. The following figure shows a schematic representation of the virtual machine's memory usage without swapping.
Lazy Swapping
This is the history of the memory usage with lazy swapping .
(1) The threshold for the memory usage, e. g. 40 %.
(2) The threshold value was exceeded after x records. Lobster_data detects this after completing the record and stores 1/3 (but not more than 100) of the records on the destination side. If the memory usage is still too high, another third (but again a maximum of 100) of the records is swapped. This process is repeated until the memory usage has fallen under the threshold or all records have been swapped.
(3) Memory usage below the threshold after n repetitions of (2).
(4) After further y records, the threshold for the memory usage is exceeded again. The procedure in (2) is repeated.
(5) Memory usage below the threshold after another m repetitions of (2).
Since this method tries to swap only as many records as necessary, this method is called lazy swapping. This is the default swapping method in Lobster_data.
If many profiles are started in parallel, the lazy swapping may no longer be able to ensure that sufficient memory is available. For this reason, Lobster_data also offers busy swapping.
Busy Swapping
In the case of busy swapping, a check is performed after every record to determine whether the percentage memory usage within the virtual machine is greater than the set maximum. If this is the case, all records already created are swapped to the hard disk. The following figure shows the memory usage with busy swapping.
(1) The configured threshold value for the memory usage, e.g. 40 %.
(2) The threshold value was exceeded after x records. Lobster_data notices this after completing the current record and swaps all created records.
(3) Memory usage after swapping all records.
(4) After further y records the memory usage threshold was exceeded again. All records are swapped again.
The disadvantage of busy swapping versus lazy swapping is that Lobster_data might swap more records than necessary.This can lead to performance losses during the further processing.
Configuration
The configuration of whether and how to swap is done separately for each profile. The configuration of which method to use for the swapping, on the other hand, is done globally for all profiles. The following listing shows the entry in the configuration file ./etc/startup.xml that activates the lazy swapping. To enable busy swapping, the value for parameter lazySwapping has to be changed from true to false.
Note: Switching from lazy swapping to busy swapping should only be done in exceptional cases and only in consultation with our support.
<!-- Enable lazy swapping of temp. data --><Set name="lazySwapping">true</Set>