Main Settings (ETL/ELT)

(1) This is used to retrieve information about tables and columns during editing. To prevent long loading times, you can deactivate DB helper functions here. Note: If you change this setting, you should reload an open task so that you receive updated information.

(2) Here you can display the search index of the ETL/ELT pipeline. These values can be queried in the expert search.

(3) In this field any URL that contains a documentation of this ETL/ELT pipeline can be specified. It be called up with (4). Additionally, system constants can be inserted into the URL with (5).

(6) Files to be read in child tasks that are smaller than the specified size are kept in memory (and not on disk).

(7) Defines which information is logged. Attention: If debug information is logged, the contents of all ETL/ELT fields and ETL/ELT variables are written per run, resulting in a very large amount of data. So use this mode only if necessary.

(8) Relevant for REST calls, but not active in the standard installation.

(9) The processing of the ETL/ELT pipeline can take place locally or on a remote server (to conserve resources of the main system).

Important note:

Files to be processed by the remote server must also be located on the remote server and the paths in the ETL/ELT pipeline must correspond to those of the remote server. For example, if the file is located remotely at /opt/input/file.csv and locally at /opt/Lobster/Lobster_data/webapps/root/upload/file.csv, then the path /opt/input/file.csv must be specified as the input source in the ETL/ELT pipeline.

Likewise, destination files are saved on the remote server.

I. e. one must either mount the folders of the remote server on the _data server, or vice versa, so that the files can be processed directly. Alternatively, the distribution of the files can also be organised with profiles (e.g. in workflows).

If you use databases in the ETL/ELT pipeline, you must also have access to them on the remote server. I.e. you have to adapt the configuration file ./etc/database.xml of the used Lobster Bee.

(10) If set, multiple instances of this ETL/ELT pipeline cannot run at the same time. So if there is already a job running for this pipeline, you cannot start another one.

(11) When selecting this checkbox, read-in NULL values will not be replaced by an empty string in function chains.

(12) If this checkbox is set, the clipboard is kept in a file and not in memory.

(13) A backup can be loaded here, see also (16).

(14) The pipeline is saved and started. Note: Variables that can be changed at startup are ignored here.

(15) The pipeline is saved and a test run is started . The input data can be limited in a further dialogue. Note: Variables that can be changed at startup are ignored here.

(16) The pipeline is saved and closed. In addition, a backup is created, see also (13).

(17) You can upload an image for this entry. This image is shown in the overview when the tile view is selected.