Index

Introduction to the index in ConSol CM

ConSol CM stores most of its data in a relational database. In order to improve the performance of search operations, Apache Lucene indexes are used. For each data field that should serve as search criterion, an index is created. The indexes are stored on the file system, in a subdirectory of the data directory. The component for index management is called indexer.

There are three types of searches:

• Quick search: Simple free-text search using a search field in the upper right corner of the Web Client
• Detail search: Advanced search on a specific page which allows combining several search criteria
• Autocomplete search: Context-dependent free-text search integrated into specific pages

The index is required to be able to search for objects in ConSol CM. The scope of the index are the basic user and case data, the entire case text, the attachments (can be configured in the system property cmas-core-index-common, index.attachment) and the data fields which have the setting Indexed for search. The data fields configured for the index are available as search criteria in the detailed search, and they can trigger an autocomplete search on contact and resource pages. The quick search shows the objects which contain the search term in an indexed field.

In addition, fields need to be indexed for the following purposes:

• The field is used as a table column and the users should be able to sort the table by this column.
• The field is used in scripts to search for entities, e.g. using criteria objects as TicketCriteria, UnitCriteria or ResourceCriteria.

Available settings for the index

The configuration of the index is done in a number of different places. The items described in the following sections need to be configured for the index to work correctly. In addition, you can fine-tune the indexer in the system properties, see Fine-tuning the indexer.

Data directory

The indexes are stored in the file system, in a subdirectory of the data directory indicated during system setup. The path of the data directory is stored in the system property cmas-core-shared, data.directory.

The following requirements need to be fulfilled for the index to work correctly:

• The data directory is always available for the ConSol CM server, also if it was created on another server and is linked to or mounted on the application server.
• The data directory provides enough space for storing indexes.

Field settings

The data fields which should be available as search criteria need to be specified by setting Indexed for search for the respective field. This is done in the data field administration, see TODO.

There are four possible values:

• True: An index is created for the field, so the object (case, contact or resource) can be searched by this field. Only the object itself is found.
• True (company + persons): Only relevant for contacts. An index is created for the field, so the contact can be searched by this field. For persons, only the person itself is found. For companies, the persons of the company are found as well.
• True (company + persons + cases): Only relevant for contacts. An index is created for the field, so the contact can be searched by this field. For persons, the cases of the person are also found. For companies, the cases of the company, the persons of the company, and the cases of the persons of the company are found as well.
• False:No index is created for the field, so it cannot be searched.

Performance hint

Including cases and persons for contact fields can have a negative impact on the indexer performance if the company or person has many cases or the company has many persons. This is due to the fact that all cases or persons have to be reindexed when there are changes to the field.

Nested fields, as used in lists or tables, all have to have the same indexing type, otherwise they cannot be searched. For example, when you work with a table, the data fields of the type List, Columns, and all data fields within the table which should be searchable, need to have the value "True" for the setting Indexed for search.

A phonetic search can be enabled for string fields. The phonetic search results do not include only exact matches but also results which sound similar even if the spelling differs from the entered search term, e.g. "Claire" and "Clare". In order to activate the phonetic search for a data field, set Phonetic search to "true" for this field.

Update mode

There are two kinds of changes which require index updates:

• Operative changes: Changes to cases, contacts, resources and users
• Administrative changes: Changes to the configuration

Operative changes are always processed directly. For the processing of administrative changes, there are two modes:

• Directly start processing: Administrative changes are processed directly. Select the checkbox Administrative changes are applied to the index automatically.
• Start processing manually: Administrative changes are processed when the user clicks the Apply administrative changes button on the Index page. Deactivate the checkbox Administrative changes are applied to the index automatically.

Direct processing of administrative changes can have a negative impact on the indexer performance if there are changes which require reindexing lots of objects, e.g. if a data field used in all cases is added to the index.

See Technical background of the index for further details.

Basic tasks

There are two types of tasks related to the index:

• Configuration-related tasks: These tasks are performed when configuring the ConSol CM system.
• Operation-related tasks: These tasks are performed when operating a ConSol CM system.

Configuration-related tasks

You can perform the following configuration-related tasks on the index:

• Decide which data fields should be indexed, i.e. available for search, see Field settings.
• Decide if attachments should be indexed, see system property cmas-core-index-common, index.attachment. The default is "true", meaning that attachments are indexed.
• Decide if administrative changes should be applied automatically, see Update mode.

Operation-related tasks

You can perform the following operation-related tasks on the index:

• Check the current index tasks: The current index tasks are displayed in the table on the Index page. You can see the ID, the type of task (indicates how the task was originated), the status (pending, running or paused), the creation date, the progress, and the details (which information is updated in the index).

• Manage current index tasks: You can pause a running index task by clicking the Pause icon in the task’s row. Paused index tasks can be resumed by clicking the Resume icon.

• Check the index status: The index status is shown below the header buttons. In addition, it is saved to the system property cmas-core-index-common, index.status. There are three status:

  • GREEN: All index tasks have run correctly. At the beginning of the synchronization process, the index status is set to GREEN. If there are any problems, it will change to YELLOW or RED.
  • YELLOW: Fixable problems were identified. This status is set when there are pending administrative changes (Administrative changes are applied to the index automatically not selected) or a retry task was created.
  • RED: Errors have occurred. The index needs to be rebuilt.

• Rebuild the index: If the index status is RED, you need to rebuild the index by clicking the Rebuild index button. All open index tasks are discarded and the index is rebuilt according to the options selected in the modal window:

  • Cases: Determines the order in which the cases are indexed. You can add queues to the list of Prioritized queues and sort them to determines the indexing order. The cases from the Remaining queues are indexed afterwards. The options above the lists allow to decide how to handle closed cases:
    • Index open cases by queue priorization first: Indexes first the open cases according to the queue priorization and then the closed cases.
    • Index open and closed cases by queue priorization: Indexes open and closed cases according to the queue priorization.
  • Attachments: Determines when to index attachments:
    • Index cases directly with attachments: The attachments are indexed together with the cases which they belong to. With this option, it takes longer until all cases are indexed.
    • Index cases first, them attachments: First the cases are indexed without attachments. The attachments are indexed afterwards. With this option, the cases are available for search more quickly, because indexing the attachments takes more time.

  Click Start rebuild to synchronize the index.

  info

  The existing index is not deleted during the rebuild process. Therefore, the search features are available while the process runs. Changes made after the rebuild was started are immediately reflected in the index, because these data have a higher priority than the data which is synchronized due to a manually triggered index update.

• Rebuild the index for a certain period: You can rebuild the index for a certain period if there were known indexer problems during this time. Click the button Rebuild index by period to select the time range. All the changes (creation and update) which were made in the selected time period are reindexed. Deletions and field setting changes are not considered.

• Repair the index: You can repair the index by clicking the Execute retry tasks button. All the pending index tasks of the type Retry tasks are executed.

• Apply administrative changes: If the checkbox Administrative changes are applied to the index automatically is not selected, you need to click the button Apply administrative changes to update the index after configuration changes.

Rebuilding the index using MBeans

You can rebuild the index using the MBean method recreateIndex for the respective type of objects:

• Cases: core.ticketIndexService
• Contacts: core.unitIndexService
• Resources: core.resourceIndexService
• Users: core.engineerIndexService

Advanced tasks

Fine-tuning the indexer

The behavior of the indexer can be fine-tuned using system properties The relevant properties are located in the module cmas-core-index-common. These properties allow to adjust parameters to improve the indexer performance, e.g. the number of threads, or whether attachments should be indexed or not.

Fine-tuning the search

The search features can be fine-tuned in several ways:

• Autocomplete search: The autocomplete search can be fine-tuned in the page customization of the respective place:

  • Email editor: mailTemplate
  • Contact section of a case: unitAutocomplete

• Quick search: The behavior of the quick search can be fine-tuned in the page customization type globalSearchField. You can define the number of displayed results in the system property cmweb-server-adapter, globalSearchResultSizeLimit. The display names of the found contacts and resources are determined using templates.

• Detail search: The behavior of the detail search can be fine-tuned in the page customization type detailSearch. You can define the length of the result table in the system property cmweb-server-adapter, searchPageSize and the available paging options in the system property cmweb-server-adapter, searchPageSizeOptions. For cases, the results table contains the assignee, main contact, case name and subject by default. For contacts and resources, it contains the display name. You can add additional columns to the right of the default ones by adding the desired position in the setting Column position in detail search results of the data field.

  info

  The user preferences overwrite the default configuration, i.e. once a user modifies the displayed columns, their order or their sorting in the Web Client, this configuration will be maintained for this particular user.

Operating the indexer

Indexer directory structure

The data directory is set using the system property cmas-core-shared, data.directory.

Monitoring the indexer

The indexer status is displayed on the Index page of the Web Admin Suite. Alternatively, you can check the system property cmas-core-index-common, index.status. The following status can occur:

• GREEN: The index is fine. No action is required.
• YELLOW: There are solvable problems with the index, either pending administrative changes or pending retry tasks. You need to perform the required action in the Web Admin Suite.
• RED: There are errors. You need to rebuild the index in the Web Admin Suite.

In addition, you need to monitor the file system where the index is located to ensure that there are enough space and RAM available, that the file system is available (if it is mounted on another server), and that the synchronization between the master and slave nodes works correctly (if you have a clustered environment).

The following problems can indicate that the indexer is not running smoothly:

• The search in the Web Client does not work correctly. Newly created objects are not found or the search does not work at all.
• Incoming emails are not processed (processing requires a search for case and contact).

Backup and recovery of the index

All ConSol CM indices are stored on the file system.

Please follow the following procedure to create an index backup, in order to avoid inconsistent states:

1. Back up the index using the following HTTP request. Replace the variables with the correct values for you system (URL, and name and password of the admin user):

wget http://${indexing.master.host:port}/index/snapshot --user ${admin} --password ${password} -Obackup.jar

This will download the full index into a backup.jar file. The received full snapshot will have the timestamp of the moment when the command was executed.

Please perform the following steps to restore the index:

1. Stop the ConSol CM server.
2. Clean the directories inside the index folder.
3. Unpack the backup.jar file to the index folder.
4. Restart the ConSol CM server.
5. Rebuild the index for the missing time period (between the creation of the backup and the restore) using the Web Admin Suite.

Fine-tuning the index

If the search is slow, or you notice delays, you might increase the number of indexer threads in the system property cmas-core-index-common, index.task.worker.threads. The default value is 1. Do not set the value to a number higher than the number of CPU cores of the machine.

Technical background of the index

Which changes require index updates?

There are two kind of changes relevant for the index:

• Operative changes: These are changes to the runtime data. The index is automatically updated when these changes are carried out.

  • Data field values: values of fields (case, contact and resource fields) which are configured for indexing using the Indexed for search setting.
  • User data: email, first name, last name
  • Case data: attachments (unless configured otherwise), creation date, assignee, history, name, queue, participants, subject, view

• Administrative changes: These are changes of certain configuration data:

  • scopes
  • queues
  • list values
  • user functions
  • supported locales
  • roles
  • Indexed for search setting of case, contact or resource fields

  The processing of administrative changes depends on the setting Administrative changes are applied to the index automatically:

  • If the checkbox is selected, the changes are processed automatically.
  • If the checkbox is not selected, the changes are stored in the database for deferred processing. Processing can be started by clicking the Apply administrative changes button.

Which is the performance impact of changes?

The performance impact of the index changes depends on the number of objects which need to be reindexed.

Change	Requires re-indexing of	Performance impact
Add or delete a system language	All cases, contacts and resources	Critical
Change internal or localized name of a queue	All cases residing in that queue	High
Add, remove or change value of the Indexed for search setting for a data field	All cases, contacts or resources which have this field set	High
Change internal name, localized name or order of a list value	All cases, contacts or resources which have the changed list value set	High
Add or remove the Rich text display in view mode setting to / from an indexed data field of the type Text (long)	All cases, contacts or resources which have this field set	Medium
Change internal name, localized name or order of a scope	All cases currently residing in the changed scope	Medium
Add or remove the Phone number setting of a contact field	All contacts which have the data field	Medium
Change phone numbers configuration of a customer group	All contacts of the customer group which have a phone number field	Medium
Change internal or localized name of a user function	All cases which have at least one participant in that function	Low
Change login, first name or last name of a user	All cases which the user is assigned to (as primary or referenced user)	Low
Change internal name or localized name of a resource type or resource category	All resources of that resource type / all resources which belong to the resource category	Low

How are index updates processed?

Two services are involved in index updates:

• Index changes notifier: Stores the entities which require an index update in the cmas_index_update_serialized table.

• Index changes receiver: Polls the cmas_index_update_serialized table for new entries and creates index tasks in the tables cmas_index_update_task and cmas_index_update_part. Administrative tasks are directly created in the database table cmas_index_administrative_task. The indexing server then executes the tasks from these tables and updates the Lucene files accordingly. If the execution of an update task fails, a new update task with the type Retry task is created and the index status is set to YELLOW. You need to execute the task by clicking the Execute retry tasks button.

  Note for clustered systems

  On clustered systems, the index is built on the master indexing server. The slave indexing servers download the index files from the master indexing server.

Stopping services

Do not stop the Index changes notifier service. If it is stopped and there is a change which requires an index update, the index status is set to RED and a rebuild is necessary. You can stop the Index changes receiver. After restart it will pick up all the missing changes from the respective table.