General problem with mass data (memory overflow)

When working with mass data, it may be necessary to separate subsets from the source data, which are then processed individually to avoid memory overflow or extremely long processing times.

Previous solution (and its limitation)

One way to solve this is to split the source data into several records by suitable parsing in a first profile and then call another profile via a Response (in combination with option "Enter output channel for each record"), in which the processing of the respective subset then takes place.

New solution (range limits and token list)

This class is intended for such cases where the range limits for the subsets can only be determined by calculations in phase 3. The division of the data into subsets on the basis of the transferred range limits then takes place in a subsequent profile.

For this purpose, the subsequent profile (see parameter target) is called several times with the same source data ("as received" set as content in the Response), but with continuous range limits. The provision of the suitable range limits takes place in a profile variable (see parameter parameters) as token list within phase 3 before the call of the class. The class calls a subsequent profile as often as the token list has tokens. With each call only the current token is passed to the corresponding MSG_CALL_ variable of the subsequent profile.

Preventing hard disk overflow (backup files)

Each time the subsequent profile is called, the same source data is transferred. If this is bulk data, there is a risk of the hard disk overflowing, because each job of the called profile saves this data as a backup file. It is therefore advisable to set the retention time of the backup file to 0 days so that it is deleted immediately after a successful run.

Used message type

The processing of the Response (in which this class is used) is an overall process within which the multiple calls of the subsequent profile take place. The use of synchronous Messages (see parameter "msgtype") is not recommended, because in that case a not exactly defined state occurs if, after successful calls, a further call of the subsequent profile leads to an error abort.

We recommend the Message type "persistent", because then the jobs of the subsequent profile can be processed strictly one after the other (see option "Profile may only run in one instance"). If the subsequent profile jobs are processed in parallel, there is otherwise a risk of memory overflow again.

Counter variable

Furthermore, a counter variable can be defined (see parameter "counter"), whose value is passed to the corresponding MSG_CALL_ variable of the subsequent profile.

This counter variable (if it is defined) is first emptied before a call of the subsequent profile and then receives the number of the current call.

If (with synchronous Messages) an abort occurs due to an error in the subsequent profile, the counter contains the number of the previous call.

Encoding

The encoding of the data is converted into the profile encoding of the subsequent profile, depending on the content type.

The only exception is the content type "as received" (not "as received, uncompressed"), where the backup file of the job is copied byte by byte as the backup file of the subsequent profile. In this case, the conversion into an encoding is skipped. We therefore recommend this content type for processing purely binary data.

Chaining of Responses

When chaining several Responses (on error/in response no.) it must be considered that the variable with the token list (parameter parameters) has the content defined in phase 3 again with the complete token list after the execution of the Response, although only the current token was passed to the subsequent profile.