Synchronization procedures

The synchronization process can be started by a manual action, i.e. when the administrator clicks on a button in the Admin Tool, starting a DWH action, or the sync process can be triggered automatically in LIVE mode. Please compare the figure in section Components involved in transfer explaining all parts of the synchronization processes.

You can use the DWH monitor of the Admin Tool (navigation item Monitor in the navigation group Data Warehouse) to get an overview of the current DWH status including some statistics. For more information, please see the DWH chapter in the ConSol CM Administrator Manual.

Operations triggered using the Admin Tool

A DWH operation which is performed synchronously using the Admin Tool does not do anything but create a task and write one entry into the CM table cmas_dwh_synchronization! Starting with version 6.11 the tasks are also shown in the Actions table in the Admin Tool (navigation group Data Warehouse, navigation item Administration). But it can still take up to some minutes until the action is displayed in the table, even when using the refresh button.

Actions in this table are executed according to the timestamp of their creation in ascending order.

So please note that if you press a button several times, a new entry in cmas_dwh_synchronization is created with each click! The respective operations will then have to be executed one after another, ordered by the timestamp of their creation. Thus, there is the risk of accumulating DWH operations which might decrease CM performance considerably and might produce unwanted results.

Please wait until you see the result of an operation -- the Admin Tool is only blocked before version 6.11 -- and do not create a pipeline of pending operations which might interfere with one another. Starting with version 6.11. you have the Delete icon to delete such duplicate tasks, when they are still in state NEW.

Example: The admin clicks on Initialize, nothing happens (because the system is working behind the scenes preparing the initialization). The admin clicks on Update. Nothing happens. The admin clicks on Initialize again ...

What will happen?

Three entries have been created in the CM table cmas_dwh_synchronization. (1) INITIALIZE, (2) UPDATE, (3) INITIALIZE

The initialization (1) is performed. When this is done, the respective entry is deleted from cmas_dwh_synchronization.

The update (2) is performed. When this is done, the respective entry is deleted from cmas_dwh_synchronization.

Now you could start work with the DWH, but ...

The next initialization (3) is performed. When this is done, the respective entry is deleted from cmas_dwh_synchronization.

The DWH is now in a non-operable state, and a transfer or update is required.

Admin Tool: Initialize button (without Delete existing data)

A new entry in cmas_dwh_synchronization is created using the following values (all other fields are initially set to NULL and filled in when required in the process): 

The initialization is then processed in another thread (by DWH transfer service). The Admin Tool will be unblocked when the processing of the initialization is finished in CM (cmas_dwh_synchronization.dwh_status='SUCCESS'|'ERROR').

Admin Tool: Initialize button (with Delete existing data)

A new entry in cmas_dwh_synchronization is created using the following values (all other fields are initially set to NULL and filled in when required in the process): 

The reinitialization is then processed in another thread (by DWH transfer service). The Admin Tool will be unblocked when the processing of the reinitialization is finished in CM (cmas_dwh_synchronization.dwh_status='SUCCESS'|'ERROR')

Admin Tool: Transfer button

A new entry in cmas_dwh_synchronization is created using the following values (all other fields are initially set to NULL and filled in when required in the process): 

The transfer is then processed in another thread (by DWH transfer service). The Admin Tool will be unblocked when the processing of the transfer is finished in CM (cmas_dwh_synchronization.dwh_status='SUCCESS'|'ERROR').

Important: The transfer will delete all data in DWH. But this is not done not very efficiently. If you want to reinitialize the DWH you should run Initialize with Overwrite first.

Admin Tool: Update button

A new entry in cmas_dwh_synchronization is created using the following values (all other fields are initially set to NULL and filled in when required in the process):

The update is then processed in another thread (by DWH transfer service).

The Admin Tool will be unblocked when the processing of the update is finished in CM (dwh_status='SUCCESS'|'ERROR').

The DWH Update can also be started via command line or script. In this case you have to use a tool which can access the ConSol CM MBeans via command line, e.g. Twiddle for JBoss. The MBean to use is consol.cmas.global.dwh.synchronizationService. The command (method) is update.

The following command line shows an example command with Twiddle. Twiddle as standalone installation.

$TWIDDLE_STANDALONE_HOME/bin/twiddle.sh -s service:jmx:remoting-jmx://127.0.0.1:9999 invoke consol.cmas:type=admin,topic=global,name=dwh.synchronizationService update

Please note that the CM package size, the CMRF cache size, and the retry counts cannot be set using the MBean. The default values will be used.

Please see also section JMX (CM).

Admin Tool: Enabling LIVE mode (OFF/ADMIN -> LIVE)

When the synchronization mode is set to LIVE using the Admin Tool (navigation group Data Warehouse, navigation item Administration, Configuration button) the CM system property cmas-dwh-server, live.start is created and set to the current date.

If an update is needed, a new entry in cmas_dwh_synchronization is created using the following values (all other fields are initially set to NULL and filled in when required in the process)

The update is then processed in another thread (by DWH transfer service).

Admin Tool: Run tasks button

This button is available/active if there are tasks which have not yet been performed. They are stored in the CM table cmas_dwh_task and are displayed in the Admin Tool (navigation group Data Warehouse, navigation item DWH Tasks).

Figure 13: Pending DWH tasks in Admin Tool and in CM database

When the admin has clicked the Run task button, a new entry in cmas_dwh_synchronization is created for each selected task using the following values (all other fields are initially set to NULL and filled in when required in the process):

The tasks are then processed in another thread (by DWH transfer service). The execution status of such tasks is also visible in the Admin Tool (navigation group Data Warehouse, navigation item Administration).

A task can also be started automatically if the option Automatic commit of administrative changes is enabled.

Operations performed by CM and by DWH services

DWH transfer service

The DWH transfer service works in the same way in ADMIN and LIVE mode.

The DWH transfer service reads entries from the CM table cmas_dwh_synchronization (with dwh_status = NEW or ACTIVE) and executes the respective jobs in the order of the timestamp of the entry (in ascending order). The service inserts serialized packages into the INT_TRANSFER_QUEUE. cmas_dwh_synchronization.dwh_status is changed during processing (NEW -> ACTIVE -> SUCCESS | ERROR). dwh_status is set to ACTIVE when the entry is processed (only one entry can be active). dwh_status is set to SUCCESS if the processing is finished successfully. dwh_status is set to ERROR if the processing is not finished successfully.

During the ACTIVE period, the DWH transfer service inserts serialized packages into the INT_TRANSFER_QUEUE. The package size taken from cmas_dwh_synchronization.package_size, which the is set when setting the CM package size via the Admin Tool GUI. When this value is not set, the CM system property cmas-dwh-server, batch-commit-interval is used (1000 if the property is not set). This is for example the case when a job was created via the JMX operations.

Each package has a serial number. When a package has been written to the INT_TRANSFER_QUEUE, the value of the next serial number (cmas_dwh_synchronization.next_serial_number) is incremented.

Processing of one entry:

  1. The entry is read from cmas_dwh_synchronization (dwh_status = ACTIVE | NEW)
  2. If dwh_status = NEW, dwh_status is updated (dwh_status = ACTIVE)
  3. Data are sent to CMRF in packages.
  4. Each package has a serial number.
    1. Sending of one package:
      1. A portion of data is read from the CM database (table cmas_dwh_synchronization).
      2. The package is created.

      3. The serialized package is inserted into the INT_TRANSFER_QUEUE. The value of the next serial number (cmas_dwh_synchronization.next_serial_number) is incremented. The progress of the processing (cmas_dwh_synchronization.position) is updated. These operations are done in one transaction.

  5. Processing in CM is finished, cmas_dwh_synchronization.dwh_status is updated (dwh_status = SUCCESS | ERROR).

Error handling:

After an exception the processing will be continued from the point where it was stopped (cmas_dwh_synchronization.position) if:

Otherwise the processing will not be continued (dwh_status = ERROR).

CMRF service

CMRF reads and processes TRANSFER data (types INITIALIZATION, REINITIALIZATION, TRANSFER, UPDATE, TASK), LIVE data and CONTROL data (pause and resume). TRANSFER data have a higher priority than LIVE data. LIVE data are processed only if there are no TRANSFER data available.

TRANSFER data

When the DWH Transfer Service has written entries into the INT_TRANSFER _QUEUE, the CMRF service can process these entries (CMRF service will also process other entries, this is treated below). The processing of each initialization/reinitialization/transfer/update/task is split into multiple transactions. The current state of the processing is saved in the database at the end of each transaction.

The performance and memory usage can be influenced by the cache size parameter. The cache size is taken from cmas_dwh_synchronization.cmrf_cache_size, which the is set when using the CMRF cache size via the Admin Tool GUI. When this value is not set, 10000 is used. This is for example the case, when a job was created via the JMX operations.

After restart the state of the processing is loaded from the database and the processing is continued.

A package (BLOB entry in the data column) of the INT_TRANSFER_QUEUE table contains the new/modified object (e.g. engineer, unit, resource) in total, i.e. "The ticket <Number> now looks like this: <modified ticket data>".

Error handling:

After an exception the processing will be continued from the point where it was stopped if:

Otherwise the processing will be not continued but the remaining packages are extracted from the table and discarded.

LIVE data

When the DWH live service has written entries into the INT_LIVE _QUEUE, the CMRF service can process these entries.

Error handling:

The entries which causes the abort together with the error message is logged in hlp_transfer_errors.

CONTROL data

Pause and resume entries written by a JMX into the INT_CONTROL_QUEUE are read by the CMRF service. They pause and resume the CMRF service.

DWH log service

The DWH log service works in the same way in ADMIN and LIVE mode.

The DWH log service is responsible for providing log information to CM administrators (e.g., in the Admin Tool). The service reads serialized packages from INT_LOG_QUEUE and writes them into the CM table cmas_dwh_log. Entries in this table can be displayed in the Admin Tool in the navigation group Data Warehouse, navigation item Administration. Select an action from the Actions table and open the log entries for this action using the Logbutton. The messages which are displayed are the entries in the cmas_dwh_log table, column operation_message.

Figure 14: DWH log entries in Admin Tool

One iteration (transaction):

  1. Log entries are read and deleted from the INT_LOG_QUEUE.
  2. A new entry in cmas_dwh_log is created for each log entry.
  3. The CM system property last.success.live.timestamp is updated for each LIVE log
  4. cmas_dwh_synchronization.cmrf_status is updated for each log entry except for LIVE log.

If any error should occur and you need to find out more details about the possible cause, you can increase the log level in the ConSol CM properties file cm6-cmrf-config.properties or cmrf-config.properties. Please see Configuring the log files in JBoss for details.

CM in LIVE mode

If LIVE mode is enabled and CMRF needs to be informed about changes done in a CM transaction, CM writes entries into the table cmas_dwh_ser_sync_object, one entry for each CM transaction which has been performed (e.g., a ticket field has been changed). The entry is written at the end of the transaction, before the commit operation is executed. Each entry contains a serialized package which needs to be sent to CMRF. The respective information will then be forwarded to CMRF by the DWH live service, see next section.

DWH live service

The DWH live service is only active if the synchronization mode is set to LIVE (i.e. if the CM system property cmas-dwh-server, live.start is set) and if all data older than the date of the property live.start were already sent to CMRF. If LIVE mode is not enabled and there is no data in cmas_dwh_ser_sync_object, the property live.start is deleted.

The DWH live service controls the just-in-time DWH update, i.e. it is responsible for sending information about changes of reportable data fields and data field groups in CM to CMRF. To do this, the DWH live service reads entries from the CM table cmas_dwh_ser_sync_object and inserts the respective serialized packages into the INT_LIVE_QUEUE in the DWH.

One iteration (transaction):

  1. Packages are read from cmas_dwh_ser_sync_object (max. 100, in order of creation).
  2. The property live.serial.number is read.
  3. A serial number is set for each package.
  4. The packages are deleted from cmas_dwh_ser_sync_object.
  5. The property live.serial.number is updated.
  6. The serialized packages are inserted into INT_LIVE_QUEUE.

Packages from INT_LIVE_QUEUE will then be processed by DWH transfer service.

Operations via JMX

JMX (CM)

JMX (CMRF)