Important note: Databases and JDBC drivers used to access databases are third-party products and are neither supported nor provided by Lobster. Any support or advice on databases or JDBC drivers that may nevertheless be provided by the Lobster support is voluntary and in no way implies a transfer of responsibility to them. The installation, operation and maintenance of databases/JDBC drivers, or measures carried out on them, are always and without exception the responsibility of the customer. The Lobster support will of course be happy to assist you with the configurations necessary in your system to connect functioning third-party systems.

This guide describes how to connect a MongoDB database (additional licence required). This does not cover the setup of MongoDB itself. Please use the MongoDB manuals for this.

Basic requirements

To use MongoDB as the logging server, your installation must have a valid license for "Thread Optimization".

You should already have the MongoDB driver "mongo-java-driver.jar" in your "./lib" directory. If not, please contact our support staff.

Configuration file ./etc/mongodb.xml

images/download/attachments/150864290/Image_003-version-1-modificationdate-1695353596891-api-v2.jpg

The database alias for logging must be named _data. If this is not done, the logging will not be switched to the MongoDB!

Pooling and caching is done by the driver itself. Option catalogName references to the database holding the so-called collections. Since MongoDB supports clustered servers, you can define multiple servers under one alias. Following an example.

Important note: The following configuration variant is only valid from Lobster Integration version 4.6.5 (before that, the old variant applies).

<Configure class="com.ebd.hub.services.nosql.mongo.MongoDBConnectionService">
	<Set name="verbose">False</Set>
	<Call name="initPool">
		<Arg>
			<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
				<Set name="alias">_data</Set>
				<Call name="addDatabaseEntry">
					<Arg>mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultAuthDB][?options]]</Arg>
				</Call>
				<Set name="catalogName">hugo</Set>
			</New>
		</Arg>
	</Call>
</Configure>

So, for example:

...
 <Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">_data</Set>
			<Call name="addDatabaseEntry">
				<Arg>mongodb://mongouser:mongopassword@localhost/admin</Arg>
			</Call>
			<Set name="catalogName">dw</Set>
		</New>
	</Arg>
</Call> 
...

The password can also be specified in obfuscated form with the placeholder <password> (in escaped form, exactly as shown below, whereby mongouser and myObfuscatedPassword still have to be replaced, of course.) and an additional line.

...

<Arg>mongodb://mongouser:<password>@localhost/admin</Arg>
<Arg>OBF:myObfuscatedPassword</Arg>
...

Configuration file ./etc/factory.xml

Enable or add the following section in your file ./etc/factory.xml (right after adding the MongoDBConnectionService).

<Call name="addService">
	<Arg>com.ebd.hub.services.nosql.mongo.MongoDBConnectionService</Arg>
	<Arg>etc/mongodb.xml</Arg>
</Call>

Stored procedure for next job number and archive option in "./etc/startup.xml"

The next available job number can be created by two methods. Using the internal storage service or by executing a stored procedure. When using MongoDB, the stored procedure must be activated. Important note: If the called stored procedure does not return a job number (e.g. because the database hub cannot be reached), Lobster Integration is terminated.

Example for MySQL

For a MySQL database, you should use the following sequence instead of a stored procedure (in the database (schema) that is used for (normal) logging), since stored procedures in MySQL are not thread-safe.

create sequence getNextJobNr as BIGINT start with <start_value> increment by 1;

For the start_value, use the currently highest job number (possibly a little higher).

Then add the following option to the configuration file ./etc/startup.xml.

<Set name="sequenceProcedureCall">SELECT NEXT VALUE FOR getNextJobNr</Set>

Example for MSSQL

Note: A sequence does not work in an MSSQL database because it cannot be queried through JDBC with a SELECT nor through a procedure. Calling a sequence in a function is also not possible. Instead, you have to make do with the following procedure.

First, a table must be created. This table contains a column IDENTITY, whose value is automatically increased for each entry.

CREATE TABLE [dbo].[jobnr](
    [SeqID] [int] IDENTITY(1,1) NOT NULL,
    [SeqVal] [varchar](1) NULL,
PRIMARY KEY CLUSTERED 
(
    [SeqID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

Then a procedure is needed to query the incremented value of column IDENTITY. This procedure makes an entry in the table created above so that the value of the IDENTITY column is incremented. With scope_identity(), this value is then fetched and returned by the procedure as the return value.

create procedure [dbo].[GetNewJobnr] @jobnr int Output as
begin
      set NOCOUNT ON
      insert into jobnr (SeqVal) values ('a')
      set @jobnr = scope_identity()
      delete from jobnr WITH (READPAST)
return
end
GO

Following is the entry in the configuration file ./etc/startup.xml to call the procedure.

<Set name="sequenceProcedureCall">declare @jobnr int exec GetNewJobnr @jobnr output select @jobnr</Set>

Example for Oracle

In this case, only the sequence must be created in the database.

CREATE SEQUENCE jobnr_dw_seq
 START WITH     1
 INCREMENT BY   1
 NOCACHE
 NOCYCLE;

Following is the entry in the configuration file ./etc/startup.xml to call the sequence.

<Set name="sequenceProcedureCall">Select jobnr_dw_seq.nextval from dual</Set>

By using MongoDB, you can move outdated log entries of jobs into a different collection. To do so, add the following.

<Call name="setLongTimeArchive"><Arg type="int">30</Arg><Arg type="int">900</Arg></Call>

In this example, all log entries older than "30" days will be moved to the archive collection and will be deleted from there after "900" days. When requesting log entries older than "30" days, they will be searched for in the archive collection rather than in the normal one.

Log migration

When your system has been started successfully, you can merge your old log messages of the SQL database into MongoDB. The migration will be done in the background, so do not stop the server until the job has finished!

Open the Admin Console and navigate to "Class Execution". Enter com.ebd.hub.datawizard.log.mongodb.TransferLogEntries as the class name and the used database alias of the SQL logging (normally this is hub). Optionally, you can enter the DataCockpit alias if configured. Execute the class and check file ./logs/wrapper.log on Windows or ./hub.txt on Unix/Linux systems. A message like Finished, overall time is XXXs will be output at the end. Only table dw_log_sum will be migrated to the MongoDB. If you need old detail logs, you have to select them from the old dw_log table.

MongoDB and DataCockpit

If you are using a MongoDB and DataCockpit (see section Installation), please insert an entry like the following in file ./etc/startup.xml

<Call name="addApplication"><Arg>
		<New class="de.lobster.webmon.apps.WebMonitor">
			<Set name="alias">hub</Set>
			<Set name="remoteHost"></Set>
			<Set name="remotePort">8020</Set>
			<Set name="mailSender">first.last@lobster.de</Set>
			<Set name="retainHeaderLogs">999</Set>
			<Set name="cleanUpTime">2</Set>
			<Set name="serverName">Main Server</Set>
		</New>
	</Arg></Call>

and additionally in file ./etc/mongodb.xml

<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">_data</Set>
			<Call name="addDatabaseEntry">
				<Arg type="String">mongodb://localhost:27017</Arg>
			</Call>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>
 
<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">hub</Set>
			<Call name="addDatabaseEntry">
				<Arg type="String">mongodb://localhost:27017</Arg>
			</Call>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>