Index

Introduction to the index in ConSol CM

ConSol CM stores most of its data in a relational database. In order to improve the performance of search operations, Apache Lucene indexes are used. For each data field that should serve as search criterion, an index is created. The indexes are stored on the file system, in a subdirectory of the data directory.

Concepts, terms and definitions

Concept	Other terms	Definition
indexer		ConSol CM component for index management
quick search		Simple free-text search using a search field in the upper right corner of the Web Client
detailed search		Advanced search on a specific page which allows combining several search criteria
autocomplete search		Context-dependent free-text search integrated into specific pages

Purpose and usage

The index is required to be able to search for objects in ConSol CM. The scope of the index is the entire ticket text, the attachments (can be configured in the system property cmas-core-index-common, index.attachment) and the data fields which are annotated with field-indexed. The data fields configured for the index are available as search criteria in the detailed search, and they can trigger an autocomplete search on customer and resource pages. The quick search shows the objects which contain the search term in an indexed field.

Available settings for the index

The configuration of the index is done in a number of different places. The items described in the following sections need to be configured for the index to work correctly. In addition, you can fine-tune the indexer in the system properties, see Fine-tuning the indexer.

Data directory

The indexes are stored in the file system, in a subdirectory of the data directory indicated during system setup. The path of the data directory is stored in the system property cmas-core-shared, data.directory.

The following requirements need to be fulfilled for the index to work correctly:

• The data directory is always available for the ConSol CM server, also if it was created on another server and is linked to or mounted on the application server.
• The data directory provides enough space for storing indexes.

Annotations

The data fields which should be available as search criteria need to be specified by assigning the annotation field-indexed to the respective field. This is done in the data field administration, see Ticket Fields (Setting Up the Ticket Data Model) for tickets, Customer Field Management and GUI Design for Customer Data for customer and CM/Resource Pool - Setting Up the Basic Resource Model for resources.

The field-indexed annotation can have four possible values:

• transitive: Only relevant for customers. An index is created for the field, so the customer can be searched by this field. For contacts, the tickets of the contact are also found. For companies, the tickets of the company and the contacts of the company are found as well.
• unit: Only relevant for customers. An index is created for the field, so the customer can be searched by this field. For contacts, only the contact itself is found. For companies, the contacts of the company are found as well.
• local: An index is created for the field, so the object (ticket, customer or resource) can be searched by this field. Only the object itself is found.
• not indexed: No index is created for the field, so it cannot be searched.

Using the value transitive for customer fields can have a negative impact on the indexer performance if the company or contact has many tickets or the company has many contacts. This is due to the fact that all tickets or contacts have to be reindexed when there are changes to the annotated field.

Nested fields, as used in lists or tables, all have to have the same indexing type, otherwise they cannot be searched. For example, when you work with a table, the data fields of the type list, struct, and all data fields within the table which should be searchable, need to have the value local for the annotation field-indexed.

A phonetic search can be enabled for string fields. The phonetic search results do not include only exact matches but also results which sound similar even if the spelling differs from the entered search term, e.g. “Claire” and “Clare”. In order to activate the phonetic search for a data field, assign the annotation phonetic with the value true to this field.

Update mode

There are two kinds of changes which require index updates:

• Operative changes: Changes to tickets, customers, resources and engineers
• Administrative changes: Changes to the configuration

Operative changes are always processed directly. For the processing of administrative changes, there are two modes:

• Directly start processing: Administrative changes are processed directly. Deactivate the checkbox No automatic commit of administrative changes.
• Start processing manually: Administrative changes are processed when the user clicks the Commit administrative changes button on the Index page. Select the checkbox No automatic commit of administrative changes.

Direct processing of administrative changes can have a negative impact on the indexer performance if there are changes which require reindexing lots of objects, e.g. if a data field used in all tickets is added to the index.

See Technical background of the index for further details.

Basic tasks

There are two types of tasks related to the index:

• Configuration-related tasks: These tasks are performed when configuring the ConSol CM system.
• Operation-related tasks: These tasks are performed when operating a ConSol CM system.

Configuration-related tasks

You can perform the following configuration-related tasks on the index:

• Decide which data fields should be indexed, i.e. available for search, see Annotations.
• Decide if attachments should be indexed, see system property cmas-core-index-common, index.attachment. The default is true, meaning that attachments are indexed.
• Decide if administrative changes should be applied automatically, see Update mode.

Operation-related tasks

You can perform the following operation-related tasks on the index:

• Check the current index tasks:
  The current index tasks are displayed in the table on the Index page. You can see the ID, the type of task (indicates how the task was originated), the status (open or running), the creation date, the progress, and the details (which information is updated in the index).
• Check the index status:
  The index status is shown below the header buttons. In addition, it is saved to the system property cmas-core-index-common, index.status. There are three status:
  • GREEN
    All index tasks have run correctly. At the beginning of the synchronization process, the index status is set to GREEN. If there are any problems, it will change to YELLOW or RED.
  • YELLOW
    Fixable problems were identified. This status is set when there are pending administrative changes (No automatic commit of administrative changes not selected) or a retry task was created.
  • RED
    Errors have occurred. The index needs to be rebuilt.
• Rebuild the index:
  If the index status is RED, you need to rebuild the index by clicking the Synchronize index button. All open index tasks are discarded and the index is rebuilt according to the options selected in the modal window:
  • Tickets
    Determines the order in which the tickets are indexed. You can add queues to the list of Prioritized queues and sort them to determines the indexing order. The tickets from the Remaining queues are indexed afterwards. The options above the lists allow to decide how to handle closed tickets:
    • Index open tickets of all queues, then closed tickets of all queues: Indexes first the open tickets according to the queue priorization and then the closed tickets.
      
    • Index open and closed tickets of each queue: Indexes open and closed tickets according to the queue priorization.
  • Attachments
    Determines when to index attachments:
    • Index tickets along with attachments: The attachments are indexed together with the tickets which they belong to. With this option, it takes longer until all tickets are indexed.
    • Index tickets without attachments, then again with attachments: First the tickets are indexed without attachments. The attachments are indexed afterwards. With this option, the tickets are available for search more quickly, because indexing the attachments takes more time.

  Click OK to synchronize the index.

  The existing index is not deleted during the rebuild process. Therefore, the search features are available while the process runs. Changes made after the rebuild was started are immediately reflected in the index, because these data have a higher priority than the data which is synchronized due to a manually triggered index update.

• Rebuild the index for a certain period:
  You can rebuild the index for a certain period if there were known indexer problems during this time. Click the button Recover index to select the time range. All the changes (creation and update) which were made in the selected time period are reindexed. Deletions and annotation changes are not considered.
• Repair the index:
  You can repair the index by clicking the Repair index button. All the pending index tasks of the type Retry tasks are executed.
• Apply administrative changes:
  If the checkbox No automatic commit of administrative changes is not selected, you need to click the button Commit administrative changes to update the index after configuration changes.

Advanced tasks

Fine-tuning the indexer

The behavior of the indexer can be fine-tuned using system properties, see System Properties. The relevant properties are located in the module cmas-core-index-common. These properties allow to adjust parameters to improve the indexer performance, e.g. the number of threads, or whether attachments should be indexed or not. The most important indexer properties are described in Indexer and Search Configuration.

Technical background of the index

Which changes require index updates?

There are two kind of changes relevant for the index:

• Operative changes

  These are changes to the runtime data:

  • Data field values: values of fields (ticket, customer and resource fields) which are configured for indexing using the field-indexed annotation
  • Engineer data: email, first name, last name
  • Ticket data: attachments (unless configured otherwise), creation date, assigned engineer, history, name, queue, additional engineers, subject, view

  The index is automatically updated when these changes are carried out.

• Administrative changes

  These are changes of certain configuration data:

  • scopes
  • queues
  • enum values
  • engineer functions
  • supported locales
  • roles
  • field-indexed annotation of ticket, customer or resource fields

  The processing of administrative changes depends on the setting No automatic commit of administrative changes:

  • If the checkbox is selected, the changes are processed automatically.
  • If the checkbox is not selected, the changes are stored in the database for deferred processing. Processing can be started by clicking the Commit administrative changes button.

How are index updates processed?

Two services, see CM Services, are involved in index updates:

• Index changes notifier:
  Stores the entities which require an index update in the cmas_index_update_serialized table.

• Index changes receiver:
  Polls the cmas_index_update_serialized table for new entries and creates index tasks in the tables cmas_index_update_task and cmas_index_update_part. Administrative tasks are directly created in the database table cmas_index_administrative_task. The indexing server then executes the tasks from these tables and updates the Lucene files accordingly. If the execution of an update task fails, a new update task with the type Retry task is created and the index status is set to YELLOW. You need to execute the task by clicking the Repair index button.

  On clustered systems, the index is built on the master indexing server. The slave indexing servers download the index files from the master indexing server.

Do not stop the Index changes notifier service. If it is stopped and there is a change which requires an index update, the index status is set to RED and a rebuild is necessary. You can stop the Index changes receiver. After restart it will pick up all the missing changes from the respective table