Settings

(1) File folder: Files are searched for in this directory (not recursively). You can also specify a URL. Example: " https://example.com/path".

(2) Icon right of (1): Selection of ETL/ELT variables.

(3) File filter: Here you can filter the files based on their name . Example: *.txt returns all files with the file extension .txt . With the prefix "regex:" you can use regular expressions for filtering. Leave the field empty or use the expression "*.*" to return all files.

(4) Sorting: Sorting of the found files by file name or modification date. Ascending or descending.

(5) Test query/settings: Test query that returns a list of files.

(6) ETL field file name (Example: "$FILENAME"), ETL field path and file name (Example: "$PATH_AND_FILE"): These ETL/ELT fields can be used to read out the individual file names in the destination data or in subtasks. More about this in the following section.

Procedure

A list of files is created from the specified file folder (depending on the settings). Each file name corresponds to a data row.

You can either process these data rows in the destination data as usual. Use the ETL/ELT fields defined in (6) for access.

An alternative possibility is to process the files in a subtask. In the first task all files are read out, in the subtask you then process one file after the other. Use one of the ETL/ELT fields from (6) in the subtask to specify the file name. Example: "./webapps/root/upload/myfiles/$FILENAME".

Practical example

A practical example of a bulk insert from a file into a database can be found here.