Settings

(1) Alias: The Kafka alias. See section Kafka connections.

(2) Topic: The topic to which the message is sent. Note: If the system variable KAFKA_PARTITION of type Integer is defined and has a value >= 0 , this value will be passed as the topic of the partition to be used.

(3) Persist too large messages: This is a special setting that you can (and should) ignore under normal circumstances. If a sent message is too large (broker limit 1 MB by default, 10 MB protocol limit), an exception is normally thrown. If this checkbox is set, this exception is caught, the file is stored in a local folder instead and a dummy message with a reference to the local file is sent, with which the recipient automatically obtains this local file. However, this currently only works if the consumers listening to topic (2) are in the same Load Balancing system. Otherwise, do not set this checkbox, as this will give the impression that the message has been sent successfully, but the recipient cannot receive it.

(4) Data type: The data type of the message data to be sent. Important note: The data type must always be specified. Make sure that you always use matching types when sending and receiving. If, for example, a message is defined and sent as Byte/String and then read as Integer/Byte, this leads to an error and the message cannot be read. Lobster Integration as a consumer is blocked until someone removes this erroneous message from the broker and it does not process any messages from this topic! Important note: If the data type AVRO is used, the path to a schema file and in (8) the address to the schema registry must also be specified. Name: schema.registry.url , Value: http://address:port. In addition, the JsonToAvroBinaryPostExecuter must be used (see there).

(5) Key type: The data type of the key of the message. Important note: The data type must always be specified. Make sure that you always use matching types when sending and receiving. If, for example, a message is defined and sent as Byte/String and then read as Integer/Byte, this leads to an error and the message cannot be read. Lobster Integration as a consumer is blocked until someone removes this erroneous message from the broker and it does not process any messages from this topic!

(6) Key: The (optional) key of the message.

(7) Client ID: An optional identifier of a Kafka consumer (in a consumer group) that is passed to a Kafka broker with every request.

(8) KafkaProducer properties: Additional producer properties can be defined via the context menu.

(9) Mark whole job as failed if this response fails: Normally a job is not necessarily considered to be erroneous if a single Response fails (see section Status of Response - Error Behaviour). But it can be enforced with this option here.

(10) Additional text on error: Here you can specify an additional log text for the error case.

Tombstone

If the system variable KAFKA_TOMBSTONE is set to true, a tombstone record is sent (this can be used to prematurely delete records in certain topics).

Header properties

You can transfer header properties with system variables of the form KAFKA_<keyname>. Note: Since only byte arrays can be specified as value for a header property, Lobster Integration always uses UTF-8 as encoding.

Autoserialized lists and maps

If you use a profile with a Kafka Response to send messages (to a Kafka server) that are retrieved by a profile with a Kafka Input Agent, autoserialized lists and maps (→ prefix autoserialize_) are automatically available in the profile with the Kafka Input Agent if the variable KAFKA_ADD_AUTOSERIALZE_DATA of type Boolean with the value true is created in the profile with the Kafka Response.

Example:

Profile 1 has a Kafka Response, a variable KAFKA_ADD_AUTOSERIALZE_DATA=true and creates the map autoserialize_mymap in phase 3.

Profile 2 has a Kafka Input Agent that retrieves the message sent by profile 1. In phase 3, the map autoserialize_mymap can be accessed directly.