Zum Hauptinhalt springen

FAQs and tips for troubleshooting

Can I remove DWH actions?​

DWH actions can only be removed if the operation has not been started yet, i.e. have the status NEW. In this case, a Delete icon is shown in the row of the action on the Administration page. This is useful to remove an action, if you have created it accidentally.

The action entries on the Administration page are never removed, but you can filter them to locate a certain entry more easily.

On the Tasks page, the action entries are automatically removed once the processing has finished successfully.

In which order are tasks executed in live mode?​

Tasks are created automatically in the cmas_dwh_task table, when there are configuration changes affecting the DWH, e.g. the setting Transfer to the DWH was applied to a data field. On the Tasks page, you can choose whether you want to apply administrative changes automatically or not. When a task is executed, a new entry is created in the cmas_dwh_synchronization table. The DWH transfer service reads the data from the table and writes chunks into INT_TRANSFER_QUEUE in the DWH. The CMRF service reads the data from INT_TRANSFER_QUEUE and INT_LIVE_QUEUE. Transfer data has a higher priority than the live data. Therefore, the new data which results from the configuration change is processed first, then the live data processing is continued.

How can I reduce the transfer and update duration?​

When starting a transfer or update operation on the Administration page, you can provide appropriate advanced settings to improve performance.

The following settings are available:

  • CM package size: Number of transferred objects per transaction from CM to the DWH control tables. Higher values mean better performance and bigger memory usage.

  • CM retry count: Number of retries performed before the action is finished with the CM status Error.

  • Language: Select one or several languages for localized names of entities in the DWH. This adds localizations to various DWH tables which are not localized by default.

    Performance hint

    Adding additional localizations may have a significant performance impact. Only select languages here if they are really needed for reporting. Consider that some entities, such as enum values, are localized by default.

  • CMRF cache size: Number of transferred objects per transaction from the DWH control tables to the DWH tables.

  • CMRF retry count: Number of retries performed before the action is finished with the CMRF status Error.

  • CMRF thread count: Number of threads used to transfer the history data. The default value is 1, it can be increased to improve the transfer and update performance.

You can either modify these settings manually or choose one of the predefined configurations:

  • Default: default settings providing a balance between safety and velocity
  • Safe: settings to avoid out of memory errors, the action will take longer
  • Fast: settings for fast execution, might cause out of memory errors
  • Last: same settings as the last execution

How can I increase the log level of CMRF?​

The log level is determined in the CMRF configuration file, <JBOSS_CMRF_HOME>/standalone/configuration/cmrf-<DB>.xml:

<logger category="com.consol.cmrf">
  <level name="${logger.com.consol.cmrf.level:INFO}"/>
  <handlers>
    <handler name="CMRF_FILE"/>
    <handler name="ERROR_FILE"/>
  </handlers>
</logger>

It can be changed in the properties file cm6-config.properties or cmrf.properties. For example, you can change the level from the default INFO to DEBUG to get more information for troubleshooting. Add an entry logger.com.consol.cmrf.level=DEBUG to the properties file. Please see Configuring the log files.

Why are updates missing although the live mode is active?​

The latest changes to objects in ConSol CM are not visible instantaneously in the DWH because the live mode runs asynchronously. It can take several minutes or longer depending on the update volume until the updates are applied to the DWH database. In addition, the CMRF only checks for updates in certain intervals.

If you need live reports you must implement them using the production database. Live reports can be implemented in CM by using the detail search, case list or dashboard widgets.

How do I know if there are performance problems?​

The following kinds of log entries indicate that the execution of SQL queries is especially slow:

  • Check the server.log for slow CM queries:

      2022-10-14 10:40:56,303 INFO [Thread-timerFactory] [rvice.impl.TransferServiceImpl] Still waiting on messages...
      2022-10-14 10:55:56,394 INFO [Thread-timerFactory] [rvice.impl.TransferServiceImpl] Still waiting on messages...

  • Check the cmrf.log for slow insert operations in the DWH:

      2022-03-20 10:23:46,757 INFO [rvice.impl.TransferServiceImpl] Still processing unit history(estimated progress 0%, estimated remaining time 40:23:10)...
      2022-03-20 10:24:46,757 INFO [rvice.impl.TransferServiceImpl] Still processing unit history(estimated progress 0%, estimated remaining time 41:23:10)...

Special case Oracle

SQL performance problems due to stale or missing table statistics can occur for large data transfers when the tables size is significantly extended or during the initial transfer when the DWH tables are empty. The update time can also be significantly degraded if the DWH is not updated regularly.

You can significantly reduce the duration of the actions by forcing an update of the table statistics on the whole CM and CMRF schemes. The following example command creates statistics for the schema CM6:

exec DBMS_STATS.GATHER_SCHEMA_STATS('CM6', OPTIONS => 'GATHER')

With the standard Oracles statistics setting, this creates histograms statistics which are needed for the specific DWH update SQLs, which allow the optimizer to find a better execution plan. You may need to repeat several times, as the different objects have different SQLs. If the data volume is very large, this can still take several hours.

How can I avoid that the transfer stops after an error?​

Usually, the transfer stops when an error occurs. You can avoid this by defining recoverable exceptions which are ignored so that the transfer continues. You can provide a comma-separated list of the Java classes of the exceptions which should be ignored. Use + at the end of the class name to ignore all exceptions which inherit from the class.

Example: java.lang.Exception+ . In this case, all exceptions will be ignored, because java.lang.Exception is the parent class of all exceptions in Java.

A regular expression can be provided as optional parameter. The regular expression is applied to the message of the exception as a filter.

Escaped characters

The following characters need to be escaped in the regular expression:

  • , -> ,
  • : -> :
  • \ -> \

Example: java.sql.SQLRecoverableException,java.lang.RuntimeException+:.*T.{1\,2}T.*

Please note that the value for recoverable exceptions has to be set at two locations:

  • CM: in the system property cmas-dwh-server, recoverable.exceptions
  • CMRF: in the start command of the application server 
    -Dcmrf.recoverable.exceptions=java.sql.SQLRecoverableException,java.lang.RuntimeException+:.*T.{1\,2}T.*

In addition, you can also set a large number of retries.

What should I do when the live mode was aborted?​

When the live mode was aborted, there will be an error message An error occurred. Update is needed. Live packages will be processed again after update. in the log files. In this case, you need to perform an update. All changes since the last successful live message will be retransmitted. This date is displayed on the Monitor page of the Web Admin Suite in the Status section (Last Live message date) and also available as epoch timestamp in the system property last.success.live.timestamp of the module cmas-dwh-server.

How should problems after a database connection failure be handled?​

If the connection to the DWH database has been interrupted, it can happen that the live mode or DWH tasks do not work correctly anymore.

Please perform the following steps to restart the live mode:

  1. Restart the CMRF by either restarting the application server or restarting the service via MBean.
  2. Perform a DWH update to reenable the live mode.

If a DWH task does not update the DWH tables anymore, i.e. the CM progress works, but the CMRF progress is stuck and the task cannot be resumed on the Tasks page, you need to perform the following step to resume task processing

  • Restart the CMRF by either restarting the application server or restarting the service via MBean.

What should I do after recovering the DWH from a backup?​

When the DWH database had to be recovered from a full database backup, the log files will probably show errors like An error occurred. Update is needed. Live packages will be processed again after update. and many warnings like Update is needed. Live packages will be processed again after update.('from' date must be equal or lesser than X, 'to' date must be equal or greater than Y.)

If you want to restart the live mode without reinitializing the DWH and performing a full transfer, you need to perform the following step:

  1. Execute the following command: 
     update cmas_dwh_synchronization set cmrf_status='ERROR' where to_date > X;
  2. Start an update on the Administration page.

What should I do if the DWH database size increases faster than CM database on Oracle?​

When you initialize the DWH with the option to delete existing data, all existing DWH tables are deleted and recreated. Nevertheless, by default, Oracle does not delete tables but keeps them in the recycle bin. To free the data space you must purge the deleted tables from the recycle bin.

Why is data which was deleted on scene import still present in the DWH?​

When you import a scene with the option to delete existing data, the data is deleted from the CM database but not from the DWH database. It is neither removed by the live mode nor by regular updates. If you want to remove the old data from the DWH database you need to reinitialize the DWH with the option to delete existing data and start a transfer afterwards.

Why did reports stop working?​

DWH reports, which have been working before, now throw SQL errors or fields are missing or empty. In this case, you need to check if there were changes to the contact data model which affected the table structure in the DWH database, see chapter DWH database in the ConSol CM DWH Manual.

Additionally, DWH tables structures may change due to updates of ConSol CM, see ConSol CM Release Notes.