Databases and JDBC drivers used to access databases are third-party products and are neither supported nor provided by Lobster. Any support or advice on databases or JDBC drivers that may nevertheless be provided by the Lobster support is voluntary and in no way implies a transfer of responsibility to them. The installation, operation and maintenance of databases/JDBC drivers, or measures carried out on them, are always and without exception the responsibility of the customer.

The Lobster support will of course be happy to assist you with the configurations necessary in Lobster_data to connect functioning third-party systems.

Basic Requirements

To use MongoDB as the logging server of Lobster_data, your installation must have a valid license for Thread Optimization.

You should already have the MongoDB driver mongo-java-driver.jar in your ./lib directory. If not, please contact our support staff.

Configuration File ./etc/mongodb.xml

images/download/attachments/58591435/Image_003-version-1-modificationdate-1597826536821-api-v2.jpg

The database alias for logging must be named _data. If this is not done, the logging will not be switched to the MongoDB!

Option minSize does accept values 0 or 1 only, pooling and caching is done by the driver itself. A minSize of 0 does not build up a connection during server start. Option catalogName references to the database holding the so-called collections. Since MongoDB supports clustered servers, you can define multiple servers under one alias. Following an example.

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC
"-//EBD Integration//DTD Configure 1.0//EN"
"http://www.ebd-integration.de/dtd/configure_1_1.dtd">
<Configure class="com.ebd.hub.services.nosql.mongo.MongoDBConnectionService">
	<Set name="verbose">False</Set>
	<Call name="initPool">
		<Arg>
			<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
				<Set name="alias">_data</Set>
				<!-- connection string: 'mongodb:[gssapi|x509|sha1|sha256|plain]://server:port'-->
				<!-- sha256 refers to the authentication type SCRAM-SHA-256.-->
				<Call name="addDatabaseEntry">
					<Arg type="String">mongodb:plain://localhost:27017</Arg>
					<Arg type="String"/>
					<Arg type="String"/>
				</Call>
				<Set name="minSize">0</Set>
				<Set name="catalogName">hugo</Set>
			</New>
		</Arg>
	</Call>
</Configure>

Configuration File ./etc/factory.xml

Enable or add the following section in your file ./etc/factory.xml (right after adding the MongoDBConnectionService).

<Call name="addService">
	<Arg>com.ebd.hub.services.nosql.mongo.MongoDBConnectionService</Arg>
	<Arg>etc/mongodb.xml</Arg>
</Call>

Stored Procedure for Next Job Number and Archive Option in ./etc/startup.xml

Lobster_data can manage the next available job number by two methods. Using the internal storage service or by executing a stored procedure. When using MongoDB, the stored procedure must be activated. Important note: If the called stored procedure does not return a job number (e.g. because the database hub cannot be reached), Lobster_data is terminated.

Example for MySQL

For a MySQL database, you should use the following sequence instead of a stored procedure (in the database (schema) that Lobster_data uses for (normal) logging), since stored procedures in MySQL are not thread-safe.

create sequence getNextJobNr as BIGINT start with <start_value> increment by 1;

For the start_value, use the currently highest job number (possibly a little higher).

Then add the following option to the configuration file ./etc/startup.xml.

<Set name="sequenceProcedureCall">SELECT NEXT VALUE FOR getNextJobNr</Set>

Example for MSSQL

Note: A sequence does not work in an MSSQL database because it cannot be queried through JDBC with a SELECT nor through a procedure. Calling a sequence in a function is also not possible. Instead, you have to make do with the following procedure.

First, a table must be created. This table contains a column IDENTITY, whose value is automatically increased for each entry.

CREATE TABLE [dbo].[jobnr](
    [SeqID] [int] IDENTITY(1,1) NOT NULL,
    [SeqVal] [varchar](1) NULL,
PRIMARY KEY CLUSTERED 
(
    [SeqID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

Then a procedure is needed to query the incremented value of column IDENTITY. This procedure makes an entry in the table created above so that the value of the IDENTITY column is incremented. With scope_identity(), this value is then fetched and returned by the procedure as the return value.

create procedure [dbo].[GetNewJobnr] @jobnr int Output as
begin
      set NOCOUNT ON
      insert into jobnr (SeqVal) values ('a')
      set @jobnr = scope_identity()
      delete from jobnr WITH (READPAST)
return
end
GO

Following is the entry in the configuration file ./etc/startup.xml to call the procedure.

<Set name="sequenceProcedureCall">declare @jobnr int exec GetNewJobnr @jobnr output select @jobnr</Set>

Example for Oracle

In this case, only the sequence must be created in the database.

CREATE SEQUENCE jobnr_dw_seq
 START WITH     1
 INCREMENT BY   1
 NOCACHE
 NOCYCLE;

Following is the entry in the configuration file ./etc/startup.xml to call the sequence.

<Set name="sequenceProcedureCall">Select jobnr_dw_seq.nextval from dual</Set>

By using MongoDB, you can move outdated log entries of jobs into a different collection. To do so, add the following.

<Call name="setLongTimeArchive"><Arg type="int">30</Arg><Arg type="int">900</Arg></Call>

In this example, all log entries older than 30 days will be moved to the 'archive' collection and will be deleted from there after 900days. When requesting log entries older than 30 days, Lobster_data will search in the archive collection rather than in the normal one.

Log Migration

When your system has been started successfully, you can merge your old log messages of the SQL database into MongoDB. The migration will be done in the background, so do not stop the server until the job has finished!

Open the Admin Console and navigate to Class Execution. Enter com.ebd.hub.datawizard.log.mongodb.TransferLogEntries as the class name and the used database alias of the SQL logging (normally this is hub). Optionally, you can enter the Web Monitor alias if configured. Execute the class and check file ./logs/wrapper.log on Windows or ./hub.txt on Unix/Linux systems. A message like Finished, overall time is XXXs will be output at the end. Only table dw_log_sum will be migrated to the MongoDB. If you need old detail logs, you have to select them from the old dw_log table.

MongoDB and Web Monitor

If you are using a MongoDB and the Web Monitor (see section Installation Web Monitor), please insert an entry like the following in file ./etc/startup.xml

<Call name="addApplication"><Arg>
		<New class="de.lobster.webmon.apps.WebMonitor">
			<Set name="alias">hub</Set>
			<Set name="remoteHost"></Set>
			<Set name="remotePort">8020</Set>
			<Set name="mailSender">first.last@lobster.de</Set>
			<Set name="retainHeaderLogs">999</Set>
			<Set name="cleanUpTime">2</Set>
			<Set name="serverName">Main Server</Set>
		</New>
	</Arg></Call>

and additionally in file ./etc/mongodb.xml

<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">_data</Set>
			<Call name="addDatabaseEntry"><Arg type="String">mongodb:plain://localhost:27017</Arg><Arg type="String"></Arg><Arg type="String"></Arg></Call>
			<Set name="minSize">0</Set>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>
 
<Call name="initPool">
	<Arg>
		<New class="com.ebd.hub.services.nosql.mongo.MongoDBSettings">
			<Set name="alias">hub</Set>
			<Call name="addDatabaseEntry"><Arg type="String">mongodb:plain://localhost:27017</Arg><Arg type="String"></Arg><Arg type="String"></Arg></Call>
			<Set name="minSize">0</Set>
			<Set name="catalogName">hugo</Set>
		</New>
	</Arg>
</Call>