Synchronization mechanism

Synchronization mode

There are three different values for the synchronization mode for transferring data from ConSol CM to a DWH database. This is the value of the CM system property cmas-dwh-server, dwh.mode.

• OFF
  No data transfer to the DWH.
• LIVE
  In this mode, every change that is submitted to the ConSol CM database is immediately synchronized with the DWH.
• ADMIN
  In this mode, the administrator has to trigger the synchronization manually.

Always use the graphical configuration in the Admin Tool (navigation group Data Warehouse, navigation item DWH Administration, button Configuration) to set the synchronization mode! Setting it directly in the CM system properties does not work!

Components involved in transfer

Figure 12: Services and tables involved in CM/CMRF/DWH data transfer (DIRECT mode)

Tables in the CM database (or database schema)

• cmas_dwh_operation_status
  The DWH log service writes into this table to document the DWH status.
  These are the info and error log entries which you can also find in the log file cmrf.log.
  You can also see these entries in the Data Warehouse part of Admin Tool when you select the logs.
  
• cmas_dwh_ser_sync_object
  This table is only used in LIVE mode. CM inserts an entry into this table for a transaction which was performed in the CM database. The entry contains a serialized package which needs to be sent to CMRF. The DWH live service reads from this table. See details below.
• cmas_dwh_synchronization
  When a DWH operation is performed using the Admin Tool (see details below), an entry is created in this table. The column type contains one of the possible values: INITIALIZATION, REINITIALIZATION, TRANSFER, UPDATE, TASK. The DWH transfer service reads from this table. See details below. Starting with version 6.11 you can see the entries also in the Admin tool (navigation group Data Warehouse, navigation item Administration).
  
• cmas_dwh_task
  An entry in this database is created when an annotation which is relevant for DWH transfer has been changed, e.g. reportable group, reportable, dwh-no-history, dwh-no-history-field. Tasks listed in this table are visible in the Admin Tool (navigation group Data Warehouse, navigation item Tasks). When a task is started (automatically or manually), a new entry in cmas_dwh_synchronization is created. When the data has been processed successfully, the entry from cmas_dwh_task is deleted.

Tables in the DWH (database or database schema)

• INT_CONTROL_QUEUE
  INT_CONTROL_QUEUE is used to control the processing in CMRF. Two actions are available: pause and resume. Only the JMX components write into INT_CONTROL_QUEUE. Only the CMRF service reads and deletes entries from INT_CONTROL_QUEUE. It is done during processing of other data (e.g. transfer). The CMRF service pauses processing after pause action and continues after resume action.
• INT_LIVE_QUEUE
  The DWH live service writes serialized packages into this table. CMRF reads from this table to process the LIVE data.
• INT_TRANSFER_QUEUE
  The DWH Transfer Service writes serialized packages into this table. CMRF reads from this table to process the TRANSFER data.
• INT_LOG_QUEUE
  This table is used for log entries written by the CMRF. Based on those entries, the DWH log service creates entries in the table cmas_dwh_operation_status. See section DWH log service.
• hlp_parameter
  This table is not directly involved in CM/CMRF/DWH transfer but provides some useful information. The most important value in this context are

  • the parameter_name last_data_transfer. The timestamp of the end of the last DHW update or transfer in the local time of the CMRF server.

  • the parameter_name dwh_status. The possible values are explained in the following table.
    

    Value of dwh_status (ID)	Description of the value	Explanation
    0	DWH uninitialized	This is the status after application startup, when DWH sees that DB has not been initialized (e.g. no entries in static data). This is a theoretical value, because when DWH is uninitialized, the DWH does not yet exist.
    1	DWH waiting for initial transfer	DWH waiting for initial transfer (ID=1) is set during (re)initialization. This is the status after DWH application startup when DWH sees that there has not been a data transfer yet. When action transfer sends transfer messages, DWH switches to status DWH: data transfer in progress (ID=2).
    2	DWH: data transfer in progress	CMRF is still working on transfer messages. In this status, DWH ignores all incoming non-transfer messages. There may have been problems with the transmission. DWH enters this state when action transfer is executed. It remains in this state until either a fatal error occurs (then state is switched to DWH: data transfer finished unsuccessfully (ID=3)) or all transfer messages have been handled (in which case the subsequent status is either (DWH: data transfer finished unsuccessfully (ID=3) when there where errors, or DWH operational (ID=4) when there were no errors).
    3	DWH: data transfer finished unsuccessfully	All data transfer messages have been handled, however, there were some errors.
    4	DWH operational	DWH is operational and will handle all incoming update messages from CM. This status is reached when all data transfers or updates have finished successfully.

• hlp_transfer_error:
  Keeps a copy of the error message from exceptions of CMRF Live Mode since the last REINITIALIZE. Here you can look up the object_uid for which the exception occurred.
• int_parameter:
  This table show technical details about the data transfer applied on the DWH database.
  • live-serial-number: the last applied serial number during live mode.

  • live-error: Boolean. If false than live mode did not abort.

  • last-success-id: epoch time stamp in milliseconds of last successful live mode update

  • last-success-live-id: epoch time stamp in milliseconds of last successful live update in of operation transfer or update. Note: this value can be lower than last-success-id, when a transfer or update task is executing.

CM processes and services

The following processes and services are relevant for the CMDB-DWH data transfer. The details are explained in the following sections.

• DWH transfer service
• CMRF service
• DWH log service
• CM in LIVE mode
• DWH live service