Index

Introduction to the index in ConSol CM

ConSol CM stores most of its data in a relational database. In order to improve the performance of search operations, Apache Lucene indexes are used. For each data field that should serve as search criterion, an index is created. The indexes are stored on the file system, in a subdirectory of the data directory.

Concepts, terms and definitions

Concept	Other terms	Definition
indexer		ConSol CM component for index management
quick search		Simple free-text search using a search field in the upper right corner of the Web Client
detailed search		Advanced search on a specific page which allows combining several search criteria
autocomplete search		Context-dependent free-text search integrated into specific pages

Purpose and usage

The index is required to be able to search for objects in ConSol CM. The scope of the index are the basic user and case data, the entire case text, the attachments (can be configured in the system property cmas-core-index-common, index.attachment) and the data fields which have the setting Indexed for search. The data fields configured for the index are available as search criteria in the detailed search, and they can trigger an autocomplete search on contact and resource pages. The quick search shows the objects which contain the search term in an indexed field.

In addition, fields need to be indexed for the following purposes:

• Field is used as a table column and the users should be able to sort the table by this column

• Field is used in scripts to search for entities, e.g. using criteria objects as TicketCriteria, UnitCriteria or ResourceCriteria

Available settings for the index

The configuration of the index is done in a number of different places. The items described in the following sections need to be configured for the index to work correctly. In addition, you can fine-tune the indexer in the system properties, see Fine-tuning the indexer.

Data directory

The indexes are stored in the file system, in a subdirectory of the data directory indicated during system setup. The path of the data directory is stored in the system property cmas-core-shared, data.directory.

The following requirements need to be fulfilled for the index to work correctly:

• The data directory is always available for the ConSol CM server, also if it was created on another server and is linked to or mounted on the application server.
• The data directory provides enough space for storing indexes.

Field settings

The data fields which should be available as search criteria need to be specified by setting Indexed for search for the respective field. This is done in the data field administration.

There are four possible values:

• True:
  An index is created for the field, so the object (case, contact or resource) can be searched by this field. Only the object itself is found.
• True (company + persons):
  Only relevant for contacts. An index is created for the field, so the contact can be searched by this field. For persons, only the person itself is found. For companies, the persons of the company are found as well.
• True (company + persons + cases):
  Only relevant for contacts. An index is created for the field, so the contact can be searched by this field. For persons, the cases of the person are also found. For companies, the cases of the company, the persons of the company, and the cases of the persons of the company are found as well.
• False:
  No index is created for the field, so it cannot be searched.

Including cases and persons for contact fields can have a negative impact on the indexer performance if the company or person has many cases or the company has many persons. This is due to the fact that all cases or persons have to be reindexed when there are changes to the field.

Nested fields, as used in lists or tables, all have to have the same indexing type, otherwise they cannot be searched. For example, when you work with a table, the data fields of the type List, Columns, and all data fields within the table which should be searchable, need to have the value True for the setting Indexed for search.

A phonetic search can be enabled for string fields. The phonetic search results do not include only exact matches but also results which sound similar even if the spelling differs from the entered search term, e.g. “Claire” and “Clare”. In order to activate the phonetic search for a data field, set Phonetic search to true for this field.

Update mode

There are two kinds of changes which require index updates:

• Operative changes:
  Changes to cases, contacts, resources and users
• Administrative changes:
  Changes to the configuration

Operative changes are always processed directly. For the processing of administrative changes, there are two modes:

• Directly start processing:
  Administrative changes are processed directly. Select the checkbox Administrative changes are applied to the index automatically.
• Start processing manually:
  Administrative changes are processed when the user clicks the Apply administrative changes button on the Index page. Deactivate the checkbox Administrative changes are applied to the index automatically.

Direct processing of administrative changes can have a negative impact on the indexer performance if there are changes which require reindexing lots of objects, e.g. if a data field used in all cases is added to the index.

See Technical background of the index for further details.

Basic tasks

There are two types of tasks related to the index:

• Configuration-related tasks: These tasks are performed when configuring the ConSol CM system.
• Operation-related tasks: These tasks are performed when operating a ConSol CM system.

Configuration-related tasks

You can perform the following configuration-related tasks on the index:

• Decide which data fields should be indexed, i.e. available for search, see Field settings.
• Decide if attachments should be indexed, see system property cmas-core-index-common, index.attachment. The default is true, meaning that attachments are indexed.
• Decide if administrative changes should be applied automatically, see Update mode.

Operation-related tasks

You can perform the following operation-related tasks on the index:

• Check the current index tasks:
  The current index tasks are displayed in the table on the Index page. You can see the ID, the type of task (indicates how the task was originated), the status (pending, running or paused), the creation date, the progress, and the details (which information is updated in the index).
• Manage current index tasks:
  You can pause a running index task by clicking the Pause icon in the task’s row. Paused index tasks can be resumed by clicking the Resume icon.
• Check the index status:
  The index status is shown below the header buttons. In addition, it is saved to the system property cmas-core-index-common, index.status. There are three status:
  • GREEN
    All index tasks have run correctly. At the beginning of the synchronization process, the index status is set to GREEN. If there are any problems, it will change to YELLOW or RED.
  • YELLOW
    Fixable problems were identified. This status is set when there are pending administrative changes (Administrative changes are applied to the index automatically not selected) or a retry task was created.
  • RED
    Errors have occurred. The index needs to be rebuilt.
• Rebuild the index:
  If the index status is RED, you need to rebuild the index by clicking the Rebuild index button. All open index tasks are discarded and the index is rebuilt according to the options selected in the modal window:
  • Cases
    Determines the order in which the cases are indexed. You can add queues to the list of Prioritized queues and sort them to determines the indexing order. The cases from the Remaining queues are indexed afterwards. The options above the lists allow to decide how to handle closed cases:
    • Index open cases by queue priorization first: Indexes first the open cases according to the queue priorization and then the closed cases.
      
    • Index open and closed cases by queue priorization: Indexes open and closed cases according to the queue priorization.
  • Attachments
    Determines when to index attachments:
    • Index cases directly with attachments: The attachments are indexed together with the cases which they belong to. With this option, it takes longer until all cases are indexed.
    • Index cases first, them attachments: First the cases are indexed without attachments. The attachments are indexed afterwards. With this option, the cases are available for search more quickly, because indexing the attachments takes more time.

  Click Start rebuild to synchronize the index.

  The existing index is not deleted during the rebuild process. Therefore, the search features are available while the process runs. Changes made after the rebuild was started are immediately reflected in the index, because these data have a higher priority than the data which is synchronized due to a manually triggered index update.

• Rebuild the index for a certain period:
  You can rebuild the index for a certain period if there were known indexer problems during this time. Click the button Rebuild index by period to select the time range. All the changes (creation and update) which were made in the selected time period are reindexed. Deletions and field setting changes are not considered.
• Repair the index:
  You can repair the index by clicking the Execute retry tasks button. All the pending index tasks of the type Retry tasks are executed.
• Apply administrative changes:
  If the checkbox Administrative changes are applied to the index automatically is not selected, you need to click the button Apply administrative changes to update the index after configuration changes.

You can rebuild the index using the MBean method recreateIndex for the respective type of objects:

• Users:
  core.engineerIndexService
• Resources:
  core.resourceIndexService
• Cases:
  core.ticketIndexService
• Contacts (i.e., persons and companies):
  core.unitIndexService

Advanced tasks

Fine-tuning the indexer

The behavior of the indexer can be fine-tuned using system properties. The relevant properties are located in the module cmas-core-index-common. These properties allow to adjust parameters to improve the indexer performance, e.g. the number of threads, or whether attachments should be indexed or not.

Technical background of the index

Which changes require index updates?

There are two kind of changes relevant for the index:

• Operative changes
  These are changes to the runtime data:

  • Data field values: values of fields (case, contact and resource fields) which are configured for indexing using the Indexed for search setting.
  • User data: email, first name, last name
  • Case data: attachments (unless configured otherwise), creation date, assignee, history, name, queue, participants, subject, view

  The index is automatically updated when these changes are carried out.

• Administrative changes
  These are changes of certain configuration data:

  • scopes
  • queues
  • list values
  • user functions
  • supported locales
  • roles
  • Indexed for search setting of case, contact or resource fields

  The processing of administrative changes depends on the setting Administrative changes are applied to the index automatically:

  • If the checkbox is selected, the changes are processed automatically.
  • If the checkbox is not selected, the changes are stored in the database for deferred processing. Processing can be started by clicking the Apply administrative changes button.

Which is the performance impact of changes?

The performance impact of the index changes depends on the number of objects which need to be reindexed.

Change	Requires re-indexing of	Performance impact
Add or delete a system language	All cases, contacts and resources	Critical
Change internal or localized name of a queue	All cases residing in that queue	High
Add, remove or change value of the Indexed for search setting for a data field	All cases, contacts or resources which have this field set	High
Change internal name, localized name or order of a list value	All cases, contacts or resources which have the changed list value set	High
Add or remove the Rich text display in view mode setting to / from an indexed data field of the type Text (long)	All cases, contacts or resources which have this field set	Medium
Change internal name, localized name or order of a scope	All cases currently residing in the changed scope	Medium
Add or remove the Phone number setting of a contact field	All contacts which have the data field	Medium
Change phone numbers configuration of a customer group	All contacts of the customer group which have a phone number field	Medium
Change internal or localized name of a user function	All cases which have at least one participant in that function	Low
Change login, first name or last name of a user	All cases which the user is assigned to (as primary or referenced user)	Low
Change internal name or localized name of a resource type or resource category	All resources of that resource type / all resources which belong to the resource category	Low

How are index updates processed?

Two services are involved in index updates:

• Index changes notifier:
  Stores the entities which require an index update in the cmas_index_update_serialized table.

• Index changes receiver:
  Polls the cmas_index_update_serialized table for new entries and creates index tasks in the tables cmas_index_update_task and cmas_index_update_part. Administrative tasks are directly created in the database table cmas_index_administrative_task. The indexing server then executes the tasks from these tables and updates the Lucene files accordingly. If the execution of an update task fails, a new update task with the type Retry task is created and the index status is set to YELLOW. You need to execute the task by clicking the Execute retry tasks button.

  On clustered systems, the index is built on the master indexing server. The slave indexing servers download the index files from the master indexing server.

Do not stop the Index changes notifier service. If it is stopped and there is a change which requires an index update, the index status is set to RED and a rebuild is necessary. You can stop the Index changes receiver. After restart it will pick up all the missing changes from the respective table