FTP (Input Agent Cron)
Introduction: Phase 1.
Lobster_data contacts an FTP server and scans a directory for files that match the file pattern. Only the specified directory is scanned, no subdirectories. The files are processed in the order in which they are delivered by the FTP server. Sorting on the client side is not performed.
(1) You can choose between the protocols FTP, SFTP (via SSH) and FTPS (via SSL).
(2) Send data via DMZ server. If no DMZ is configured, this checkbox has no effect. The DMZ server is contacted, it executes the FTP commands, receives the files and stores them locally. Then these files are retrieved from the DMZ server via an internal FTP communication partner and processed. This option is only available if a channel is configured in (3). Attention: The internal communication partner DMZ Internal is created automatically, but must not be changed.
(3) Here you can select a channel of type FTP. Note: If possible, always use channels to centrally manage connection parameters. In addition to the increased clarity that also facilitates maintenance. If the connection parameters change, you only have to adjust the data in one place and you do not forget a profile.
(4) DNS name or IP address (and port) of the FTP server. If value 0 is specified for the port, then either the default port (./etc/ftp.xml) is used (if no channel is selected) or the port specified in the channel is used (if a channel is selected). If a value > 0 is specified, then this value is used as port number.
(5) User and password for the target system. When connecting to a Value Added Network (VAN), it may be necessary to specify the GXS account in addition to the user. It must be ensured that the entered user also has access rights for the specified directory (7).
(6) In passive mode, the client establishes the control and the data connection. Normally (in the active case) the server establishes the data connection, but it may be the case that the necessary port is blocked by the local firewall.
(7) Directory on the FTP server side. You can also use the placeholders for dates and variables in this field.
(8) The FTP command NLST is used instead of the command LIST to create the file list. The file list creation is faster this way, but only the names of the files and no other information (e.g. if it is a directory) are provided. Since the response of the LIST command is not standardised, there may be problems parsing the received data. The use of the fast list mode will eliminate errors of this kind.
(9) File name pattern. The files can be sorted for processing by filename in descending or ascending order or as supplied by the operating system.
(10) The read file can be handled in the following ways.
Do not delete. Causes Lobster_data not to delete the read file. An existing control file will also not be deleted.
Delete. Causes Lobster_data to delete the read file and any existing control file.
Rename. Causes Lobster_data to rename the read file. The new file name can be specified in the associated input field. Any existing control file is neither deleted nor renamed. The following placeholders can be used to refer to the name of the processed file.
<file> is the full name of the processed file.
<file-prefix> is the name of the processed file without the file extension.
(11) Causes Lobster_data to read a file only if there is an associated control file. See also (10). See also the example in (12). See also system variable VAR_SYS_CTRL_FILE.
(12) Suffix or prefix of the control file. Example: If the suffix .ctrl is entered in the field, the control file name1.txt.ctrl will be searched for the file name1.txt. Please note that the dot in the suffix .ctrl must be explicitly specified.
(13) If this option is set, you only need a control file for the directory and not a separate control file for each file. The job will be started if this file exists in directory (7).
(14) If this checkbox is set, several instances of this profile can work in parallel. The checkbox Profile may only run in one instance must not be set for this profile
(15) Empty files are always deleted on the remote system, regardless of the setting in (10).
(16) If this option is activated, jobs for this profile will even be created for empty files. Note: In a DMZ environment, the DMZ server can be started with the following startup option in the configuration file ./etc/startup.xml to control the behaviour.
<!-- Defines how to handle empty FTP, SSH and OFTP files (0=skip, 1=delete, 2=transfer) --><Set name="handleEmptyFiles">0</Set>skip: (Default) Empty file remains on the DMZ server. Log entry in DmzEvents.
delete: Empty file is deleted on the DMZ server.
transfer: Empty file is transferred to the profile and option (8) will have an effect.
(17) Specifies whether the AS400 main file should be deleted after the files have been fetched. Attention: If there are files within the main file that do not correspond to the file pattern (9), the main file will still be deleted.
(18) Additional FTP SITE commands and standard FTP commands can be specified here (according to RFC 959). Which SITE commands can be specified here depends on which commands the respective FTP server supports (with SITE HELP the FTP server sometimes provides a list of its specific SITE commands). If you specify the additional FTP command somecommand, the FTP command SITE somecommand will be sent internally. If you want to send standard FTP commands, please use the prefix rfc:. So if you enter the additional FTP command rfc:CDUP, the FTP command CDUP will be sent internally. If there is a prefix before: before the command, it will be sent before the file transfer, otherwise afterwards. So to send the command CDUP before the file transfer, before:rfc:CDUP must be used. Note: When sending commands, the return code is checked by the server and all values 2xx except 202 are accepted. For other values, the profile generates an error. Note: Variables in the form @VARNAME@ can also be used to specify the additional commands. Note: Additional FTP commands cannot be used with SFTP (1). Note: The successful execution of the SITE commands themselves is not checked, i.e. Lobster _data does not generate an error if a SITE command fails (except for an IO exception).
Further Notes
Some FTP servers also provide subdirectories without making them recognisable. To prevent jobs from running on non-existent files, the long file list can be requested from these FTP servers.
If value Do not delete is set for option File handling, it is necessary to delete/rename this file and a potential control file on the FTP server side after successfully reading this file, as
the files would otherwise be processed again during the next time the profile is run.If deleting or renaming of a file that is to be processed fails, for example, because a file of the same name already exists, no job is started. This will lead to an error if checkbox Missing data are not ignored is set, even if a file, that is to be processed, exists.
With extremely large files, the communication between Lobster_data and the DMZ server can lead to a timeout. A solution is described in section Dealing with Timeouts Due to Extremely Large Files.
If using this Input Agent, a maximum size of the input file can be checked. See section Maximum Size of Input Files.
Files that can not be assigned to any profile are stored in the Unresolved area. See also section Warning for Non-assignable Input Data.
FTP site commands can be sent just like in the FTP Response Route.