Databases and JDBC drivers used to access databases are third-party products and are neither supported nor provided by Lobster. Any support or advice on databases or JDBC drivers that may nevertheless be provided by the Lobster support is voluntary and in no way implies a transfer of responsibility to them. The installation, operation and maintenance of databases/JDBC drivers, or measures carried out on them, are always and without exception the responsibility of the customer.

The Lobster support will of course be happy to assist you with the configurations necessary in Lobster_data to connect functioning third-party systems.

This guide describes how to set up a MongoDB database in Lobster_data (additional licence required). This does not cover the setup of MongoDB itself. Please use the MongoDB manuals for this.

Basic Requirements

To use MongoDB as the logging server of Lobster_data, your installation must have a valid license for Thread Optimization.

You should already have the MongoDB driver mongo-java-driver.jar in your ./lib directory. If not, please contact our support staff.

Configuration File ./etc/mongodb.xml

images/download/attachments/58591424/Image_003-version-2-modificationdate-1656055626713-api-v2.jpg

The database alias for logging must be named _data. If this is not done, the logging will not be switched to the MongoDB!

Option minSize does accept values 0 or 1 only, pooling and caching is done by the driver itself. A minSize of 0 does not build up a connection during server start. Option catalogName references to the database holding the so-called collections. Since MongoDB supports clustered servers, you can define multiple servers under one alias. Following an example.

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC
"-//EBD Integration//DTD Configure 1.0//EN"
"http://www.ebd-integration.de/dtd/configure_1_1.dtd">
<Configure class="com.ebd.hub.services.nosql.mongo.MongoDBConnectionService">
	<Set name="verbose">False</Set>
	<Call name="initPool">
		<Arg>
			<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
				<Set name="alias">_data</Set>
				<!-- connection string: 'mongodb:[gssapi|x509|sha1|sha256|plain]://server:port'-->
				<!-- sha256 refers to the authentication type SCRAM-SHA-256.-->
				<Call name="addDatabaseEntry">
					<Arg type="String">mongodb:plain://localhost:27017</Arg>
					<Arg type="String"/>
					<Arg type="String"/>
				</Call>
				<!--
				<Call name="addDatabaseEntry">
					<Arg type="String">mongodb:sha256://localhost:27017</Arg>
					<Arg type="String">user</Arg>
					<Arg type="String">password</Arg>
				</Call>
				-->
				<Set name="minSize">0</Set>
				<Set name="catalogName">hugo</Set>
			</New>
		</Arg>
	</Call>
</Configure>

Configuration File ./etc/factory.xml

Enable or add the following section in your file ./etc/factory.xml (right after adding the MongoDBConnectionService).

<Call name="addService">
	<Arg>com.ebd.hub.services.nosql.mongo.MongoDBConnectionService</Arg>
	<Arg>etc/mongodb.xml</Arg>
</Call>

Stored Procedure for Next Job Number and Archive Option in ./etc/startup.xml

Lobster_data can manage the next available job number by two methods. Using the internal storage service or by executing a stored procedure. When using MongoDB, the stored procedure must be activated. Important note: If the called stored procedure does not return a job number (e.g. because the database hub cannot be reached), Lobster_data is terminated.

Example for MySQL (and other databases)

A sample procedure for several databases can be found in file ./conf/samples/db_sequence.sql. Store the latest job number in the table used by the procedure to avoid duplicate job numbers. Add the following option in file ./etc/startup.xml, assuming you have used the sample procedure mentioned before.

General:
<Set name="sequenceProcedureCall">execute command</Set> (see example file for your database)

The procedure must be created within the database (schema) Lobster_data uses for the normal logging.

Important note: If you are using a MySQL database, you should use the following sequence instead, since stored procedures are not thread-safe there.

create sequence getNextJobNr as BIGINT start with <start_value> increment by 1;

For the start_value, use the currently highest job number (possibly a little higher).

Then add the following option to the configuration file ./etc/startup.xml.

<Set name="sequenceProcedureCall">SELECT NEXT VALUE FOR getNextJobNr</Set>

Archiving

When using a MongoDB, you can move outdated log entries of jobs into a different collection. To do so, add the following.

<Call name="setLongTimeArchive"><Arg type="int">30</Arg><Arg type="int">900</Arg></Call>

In this example, all log entries older than 30 days will be moved to the 'archive' collection and will be deleted from there after 900days. When requesting log entries older than 30 days, Lobster_data will search in the archive collection rather than in the normal one.

Log Migration

When your system has been started successfully, you can merge your old log messages of the SQL database into MongoDB. The migration will be done in the background, so do not stop the server until the job has finished!

Open the Admin Console and navigate to Class Execution. Enter com.ebd.hub.datawizard.log.mongodb.TransferLogEntries as the class name and the used database alias of the SQL logging (normally this is hub). Optionally, you can enter the DataCockpit alias if configured. Execute the class and check file ./logs/wrapper.log on Windows or ./hub.txt on Unix/Linux systems. A message like Finished, overall time is XXXs will be output at the end. Only table dw_log_sum will be migrated to the MongoDB. If you need old detail logs, you have to select them from the old dw_log table.

MongoDB and DataCockpit

If you are using a MongoDB and DataCockpit (see section Installation), please insert an entry like the following in file ./etc/startup.xml

<Call name="addApplication"><Arg>
		<New class="de.lobster.webmon.apps.WebMonitor">
			<Set name="alias">hub</Set>
			<Set name="remoteHost"></Set>
			<Set name="remotePort">8020</Set>
			<Set name="mailSender">first.last@lobster.de</Set>
			<Set name="retainHeaderLogs">999</Set>
			<Set name="cleanUpTime">2</Set>
			<Set name="serverName">Main Server</Set>
		</New>
	</Arg></Call>

and additionally in file ./etc/mongodb.xml

<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">_data</Set>
			<Call name="addDatabaseEntry"><Arg type="String">mongodb:plain://localhost:27017</Arg><Arg type="String"></Arg><Arg type="String"></Arg></Call>
			<Set name="minSize">0</Set>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>
 
<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">hub</Set>
			<Call name="addDatabaseEntry"><Arg type="String">mongodb:plain://localhost:27017</Arg><Arg type="String"></Arg><Arg type="String"></Arg></Call>
			<Set name="minSize">0</Set>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>