You are here:

Search Configuration and Indexer Management

ConSol CM provides a powerful search mechanism for all objects involved in the business processes (customers, tickets and resources). Technically, the search is based on the Indexer, a module of ConSol CM.

The following paragraphs explain the entire topic Search in ConSol CM from an administrative point of view. Please refer to the ConSol CM User Manual for a detailed explanation about how to use the search as an engineer.

Search Modes

A ConSol CM engineer can use several search modes:

Quick Search

This is performed using the Quick Search field in the upper right-hand corner of the Web Client. The display of the results (i.e., the fields and the order of the fields in the result list) can be formatted using templates, as detailed in sections Templates for Customer Data and CM.Resource Pool - Templates for Resource Data. Please keep in mind that you can adapt the length of the result set using the system property cmweb-server-adapter, globalSearchResultSizeLimit - see System Properties for details. The result set will also be influenced by the Page Customization attribute appendWildcardAutomatically, see section Page Customization, appendWildcardAutomatically for details.

Figure 349: ConSol CM Web Client - Quick Search

Phonetic Search

For the Quick Search, a phonetic search can be configured for String fields. Please see section Configuring the Phonetic Search for the explanation of how to prepare your ConSol CM system for this feature.

With a phonetic search, the engineer cannot only find exact matches in the search results but also results which sound similar even if the spelling differs from the entered search term. The implementation is based on the Apache Commons Codec libraries.

Figure 350: ConSol CM Web Client - Search results without phonetic search

Figure 351: ConSol CM Web Client - Search results with phonetic search

Detailed Search

This is performed using the Detailed Search page. To open this page, click on the magnifier icon next to the Quick Search entry field.

Figure 352: ConSol CM Web Client - Detailed Search

Please keep in mind that the result list length and paging of the result list for the Detailed Search can be configured using the system properties cmweb-server-adapter, searchPageSize and cmweb-server-adapter, searchPageSizeOptions. See System Properties for an explanation.

For the engineer search, you, as an administrator, can define engineer names which should not be offered in the pull-down menu in the search criteria. This is configured using page customization. Please see section excludedUserNames for details.

Please refer to the ConSol CM User Manual, section Searching for Tickets, Customers and Resources to learn how to use the search functionality.

Configuring the Search Result List

You, as an administrator, can configure the layout of the search result list using the annotation order-in-result. This annotation influences the columns of object-specific search result sets. For example, the customer field group ResellerCompanyData, belonging to the customer data model ResellerModel, contains the following customer fields:

company_name: order-in-result = 1
company_number: order-in-result = 2

Thus, in a search result list, e.g., a Detailed Search which shows a result set with companies of a customer group which uses the ResellerModel will contain the two columns company_name and company_number.

For a field (i.e., column in the result table) which should not be displayed in the Web Client (i.e., in the field selector for the result list and as column in the table) at all, set the value of the order-in-result annotation to 0 (zero).

The following figure shows the order-in-result annotation of a customer field. You reach this screen by opening the navigation item Data Models in the navigation group Customers.

Figure 353: ConSol CM Admin Tool - Customers, Data Models: Setting the annotation order-in-result for a customer field

Figure 354: ConSol CM Web Client - Search result set with the two annotated columns (order-in-result)

Notes about the Order of Search Result Columns

Since the annotation order-in-result is not the only parameter which influences the order of columns in search results (tables), please find here a detailed explanation.

In the following section, user preferences means the layout of a search result table which the engineer has configured manually, e.g., the columns which should be displayed, the order of those columns or the sorting.

Rules:

The columns of a search result table can display different fields for each of the CM object types:
1. for tickets: ticket fields as well as internal fields like queue, creation date, engineers
2. for customers (contacts or companies): customer fields as well as internal fields like object name of a customer
3. for resources: resource fields as well as internal fields like object name of a resource
The order-in-result annotation organizes only the order of the ticket fields, customer fields, and resource fields, not of internal fields.
The internal field types have the following visibility and order by default:
1. for ticket lists:
  1. Engineer
  2. Main customer
  3. Ticket name
  4. Ticket subject
2. for customer (contact/company) lists:
  1. object (i.e. contact or company) name
3. for resources lists:
  1. object (=resource) name
The data fields (ticket field, customer fields, and resource fields) are primarily ordered by the user preferences. Those will overwrite any other configuration.
When, e.g., as default configuration, a ticket field has the annotation order-in-result = 5 and is visible, it will (case a) not be displayed if the engineer has deselected the column, and (case b) it will be displayed as column #2 if the engineer has placed it there.
If no order is defined by the user preferences, the order is the ascending order of numbers from the data field annotation values order-in-result. Those numbers are evaluated globally across all data fields within all ticket field groups, all customer field groups, and all resource field groups. Identical values of the annotation order-in-result of different data fields are theoretically possible.
For an exactly defined order, use some convention for an entire CM installation and apply unique numbers, e.g., use high four digit values.
If there are still two fields with identical values for order-in-result which should both be displayed, the data fields are alphabetically ordered based on their localized names. The locale of the engineer's browser is used.

Configuring Detail Search Behavior for Enum and MLA Values

The values within an enum (a sorted list) or an MLA might change over time, e.g. a new software version is added to a list, an old one should no longer be available. In this case, the deprecated/old value will be deactivated. This means that there might be tickets which contain enum or MLA values which are deactivated in the current system. Of course it is indispensable that engineers can still search for tickets which contain this value. Therefore deactivated enum or MLA values are offered in the Detailed Search as shown in the following figure for the deactivated enum value Resource Pool.

Figure 355: ConSol CM Web Client - Detailed Search for modules, one enum value deactivated

The ConSol CM behavior concerning the Detailed Search of deactivated enum and MLA values can be configured using a Page Customization attribute: enableSearchForDeactivatedEnums .

Autocomplete Search (Search by Using Intelligent Fields)

The Autocomplete Search is performed implicitly when you start entering a word in an Autocomplete field, e.g., in company data or customer data when you create a ticket (see figures below).

Figure 356: ConSol CM Web Client - Autocomplete Search

Figure 357: ConSol CM Web Client - Suggestions for an Autocomplete Search

Starting with ConSol CM version 6.10.2, a search in all contacts of the ConSol CM database is also performed in the Ticket Email Editor when an engineer starts typing in the To, Cc or Bcc field:

Figure 358: ConSol CM Web Client - Ticket Email Editor: Automatic search for contacts as email receivers

The number of characters which must be typed before automatic search is activated can be configured using the page customization attribute minMailInputLength (Integer, Default 1) in the scope mailTemplate.

Fields Which Can Be Searched

For ticket fields, customer fields and resource fields which should be searched, the annotation field indexed has to be set. See section The Annotation field indexed. This makes the field available for the Quick Search and for the Detailed Search.

Two types of fields are processed by the search engine:

data fields and content which are/is indexed by default
data fields which are annotated using the annotation field indexed

Data Fields and Content Which Are Indexed by Default

Engineer data
- Email
- First name
- Last name
- ID
Ticket data
- Attachment content (this is controlled by the CM system property cmas-core-index-common, index.attachment which is true per default)
- Creation date
- Engineer
- History
- ID (technically the ticket name)
- Queue
- Referenced engineers
- Subject
- View

Data Fields Which Are Annotated Using the Annotation field indexed

For all data fields which have been added and configured by the ConSol CM administrators, the annotation field indexed has to be set to make the fields searchable. A field which is indexed is available in

the Quick Search
the Detailed Search
all searches in workflow and Admin Tool scripts which use, for example, criteria classes/objects (e.g., TicketCriteria, UnitCriteria, ResourceCriteria). This includes
- scripts which have been manually added by the administrator
- scripts which are present by default (e.g., the script createTicket.groovy. In this script, the correct contact for a new ticket can only be set if the customer field which is used to retrieve the correct contact or company is indexed)

Please note that if values of a table, i.e. a list of structs, should be available for the search, all elements of the data structure have to be annotated with field-indexed = true. This means that the list, the struct and every single data field within the struct have to be annotated that way!

You, as an administrator, can define the system's behavior with regards to search results. Depending on the values of the field indexed annotation, the search results for a contact might include all the tickets of the contact as well. The following table shows the implications of all possible values of the field indexed annotation for the different object types. The field indexed annotation has to be set for each individual data field, e.g., name and email for a contact, zip and address for a company, priority and software module for tickets, or name and model for a resource type.

Please note that two distinct perspectives are addressed here:

the search criterion, i.e., the field for which you start the search, e.g., all companies that start with ConS*. A field is made available in the search by setting the field-indexed annotation.
the result set, i.e., the objects which are displayed in the result lists. The objects in the result set are defined by the value of the field-indexed annotation. Depending on this value, you might see only the objects you have searched for (e.g., only contacts) or also objects which are related to the objects you have searched for (e.g., companies of those contacts). Please see the following table for a detailed explanation.

Object type/ value of field indexed annotation	transitive	unit	local	not indexed
TICKET	no difference between the three values the ticket data will be available for searching no other objects which are related to a ticket in any way will be retrieved we recommend that you use the default value transitive			the ticket data will be NOT available for searching
CONTACT	the contact data will be available for searching the tickets of the contact will also be found when you search for the contact searching by company by its contact field is NOT possible	the contact data will be available for searching searching for a company by its contact field is NOT possible	the contact data will be available for searching no other objects which are related to a contact in any way will be retrieved	the contact data will be NOT available for searching
COMPANY	the company data will be available for searching the tickets of the company will also be found when you search for the company the contacts of the company will also be found when you search for the company	the company data will be available for searching the contacts of the company will also be found when you search for the company	the company data will be available for searching no other objects which are related to a company in any way will be retrieved	the company data will be NOT available for searching
RESOURCE	no difference between the three values the resource data will be available for searching no other objects which are related to a resource in any way will be retrieved we recommend to use the default value transitive			the resource data will be NOT available for searching

Please note that if it should be possible to sort result tables (in the Web Client) according to a column (by clicking on the column header), the respective field has to be indexed!

Configuring the Phonetic Search

In order to activate the phonetic search for a data field, set the annotation phonetic to true for this field. The annotation can only be used for String fields.

Administrator Tasks Concerning the Indexer

It is important for the administrator to know how to configure ConSol CM in a way that ...

all required fields can be searched.
no overhead is produced (i.e., not too many fields are configured for searching).
the search results are displayed in the desired way.
- in the result table in the Detailed Search
- as suggestions for Autocomplete Search

These tasks are described in the following sections.

First, some background knowledge about the Indexer, the system which manages the search in ConSol CM, is provided. This will help you, as an administrator, to look behind the scenes and understand the configuration in a better way.

Introduction to the ConSol CM Indexer

The Indexer is a module of ConSol CM which creates search indexes. For each data field that should serve as search criterion (see section Fields Which Can Be Searched above), an index is created.

The indexes are stored in a sub-directory of the data directory that you have indicated during system set-up (the data directory is stored as a system property: cmas-core-shared, data.directory). The following picture shows an example for index files of a ConSol CM installation (here used for a demo environment). The demo_Datadir is the data directory specified during set-up, and all subdirectories are created automatically by ConSol CM.

Figure 359: ConSol CM Indexer - Directory demo_Datadir

Please make sure that ...

the data directory is always available for the ConSol CM server system if it was created on another server and is linked to (or mounted on) the application server.
the data directory provides enough space for storing indexes.
the ConSol CM datadir is part of the daily backup (and can be restored if required).

If the index directory should be corrupted or the index is not available, it can be rebuilt or repaired. Please see the following sections for details.

Indexer and Index Management Using the Admin Tool

The Annotation field indexed

By default, the entire ticket text and all attachments are indexed. For all ticket fields, customer fields and resource fields, the fields which should be indexed have to have the annotation field-indexed. Please refer to the sections Ticket Field Administration (Setting Up the Ticket Data Model), Customer Field Management and GUI Design for Customer Data and CM.Resource Pool - Setting Up the Basic Resource Model for details about setting annotations to ticket fields (ticket data), customer fields (customer data) and resource fields (resource data). There are four possible values for this annotation:

local
unit
transitive
not indexed

A detailed explanation of the system behavior triggered by those values is provided in the table above.

Nested fields all have to have the same index type, otherwise they cannot be searched. For example, when you work with a list of structs, the list, the struct and all data fields in the struct which should be searched have to have the value transitive for the annotation field-indexed.

Indexer Management: Navigation Item Index

Usually, the index requires no manual maintenance. ConSol CM will handle everything regarding indexing automatically. There are only two cases where you have to perform manual administrative operations:

You would like to change the configuration, e.g., by changing field-indexed annotations.
Errors have occurred in the indexing process.

In the Admin Tool, open the navigation group Services, navigation item Index to configure and manage the Indexer. If Indexer tasks are still running, this will be indicated by an exclamation mark next to the name of the navigation group (Services) and by the number of open tasks next to the name of the navigation item (Index). In this way, even when the navigation group is not opened, you, as an administrator, can immediately notice that some tasks are open in the group Services and can then quickly identify the number of open tasks.

Figure 360: ConSol CM Admin Tool - Services, Index

The Index screen contains three main elements:

Status of the indexer (1)
Buttons to manage the indexer (2)
List of open/active indexer tasks (3)

The number of open indexer tasks is also indicated next to the navigation item Index.

Figure 361: ConSol CM Admin Tool - Services: Indication of open indexer tasks

In the first line the current status of the Indexer is displayed (this is the value of the system property cmas-core-index-common, index.status):

GREEN
All Indexer tasks have run correctly, no action required. At the beginning of the synchronization process, the index status is set to green. If it completes successfully, it remains green. If there are any problems, it will change to yellow or red.
YELLOW
Fixable problems were identified, collected and persisted. The status is set when an administrative task (with auto-commit off) or a retry task is created.
RED
Errors have occurred. Please check. The index needs full synchronization.

The following operations can be performed:

Synchronize index
The index is rebuilt completely (from scratch), all open Indexer tasks are discarded.
Repair index
Indexer tasks which have not run successfully are restarted. The tasks can be selected in the Indexer task list.
Recover index
A time range can be selected. All changes which have been committed to ConSol CM during this period of time will be (re-)indexed.
Commit administrative changes
Click this button to commit the changes when you have set a ticket field, customer field or resource field to field-indexed that was not indexed before. This has to be used if the checkbox No automatic commit of administrative changes has been selected. If the checkbox is inactive, the changes will be committed automatically when you have set the new annotation(s).

Please note, that the automatic commit of administrative changes can affect the system performance.
There is a difference in ConSol CM versions concerning the meaning of administrative changes which influences manual index management!

In versions prior to 6.9.3.0, setting the annotation field-indexed from false (or not set) to true is not considered an administrative change. That means when you have added the annotation field-indexed = true for a data field which was present before, you have to modify the index manually by using either Synchronize index or Recover index.

Starting with version 6.9.3.0, setting the annotation field-indexed from false (or not set) to true is considered an administrative change. That means when you have added the annotation field-indexed = true for a data field which was present before, you have to
- If No automatic commit of administrative changes is set, then modify the index manually by clicking on Commit administrative changes
or
- If No automatic commit of administrative changes is not set, no action is required.

For experienced administrators only:

The operation Commit administrative changes can also be executed using the ConSol CM MBean consol.cmas.admin.global.core.indexManagement, method commitAdministrativeChanges(). You can use graphical tools like JConsole or use a command, e.g. with REST and Jolokia. If you need help with this topic, please ask your ConSol CM consultant.

If there are open tasks in the Indexer task list, the following data is displayed for each task:

ID
Task ID
Task type
Three types are available:
- Synchronization
  - Recreates the whole index.
  - Triggered manually using the Admin Tool, Synchronize index command.
  - Before starting, all other index tasks are removed.
- Administrative changes
  - Created automatically when one of the following is updated:
    scope, queue, enum value, ticket function, ticket engineer, supported locale, role.
  - Processed automatically if the No automatic commit of administrative changes option is unchecked.
  - The Commit administrative changes command will start all administrative changes tasks.
- Retry
  - Created automatically when errors are encountered during the index update process.
  - Holds information about entities which caused problems.
  - The Repair index command will start all retry tasks.
Status
E.g., RUNNING
Created at
Time stamp when the task was created.
Progress
A progress bar that indicates the tasks progress as a percentage.
Details
A list of objects which are (re-)indexed by the task.

Please note that data in the index is always synchronized with the ConSol CM database, i.e. during an Index update no data is deleted/removed from the index files. The index is fully usable during the synchronization process, i.e. the search operations (detail as well as quick search) can be used with their full functionality and the complete data set. Changes made after the synchronization was started are immediately reflected in the index, because these data have a higher priority than the data which is synchronized due to an index update triggered manually using the Admin Tool.

When an admin has started the index update manually (Synchronize Index), all other index tasks are removed, before this update starts.

Indexer and Index-Relevant System Properties

The following system properties are also relevant for the Indexer, see following figure. Please refer to System Properties for a detailed explanation of the Indexer system properties and the Indexer and Search Configuration section of List of System Properties by Area.

The following figure shows the properties in the module cmas-core-index-common. You reach this screen by opening the navigation item System Properties in the navigation group System and selecting the module.

Figure 362: ConSol CM Admin Tool - System, System Properties: Indexer properties

Indexer Update

For a detailed description of the ConSol CM Indexer, including the configuration of the Indexer in an environment with master and slave indexing servers, please refer to the ConSol CM Operations Manual. Here in the current section, some short information will be provided as quick introduction for CM administrators.

Indexer Update Modes

The indexer stores entities which require an index update (e.g. modified tickets, modified customers, new engineers) in a persistent store. This can be either a JMS queue of the application server or a table in the CM database. You can select the mode by setting the system property database.notification.enabled, default value in CM version 6.10 is false.

Index Persistent Store as JMS Queue

database.notification.enabled = false

The JMS queue queue/cm6-index will be used.

Index Persistent Store as Database Table

database.notification.enabled = true (should not be used with CM versions before 6.9.4.1!)

The database table cmas_index_update_serialized will be used for indexer transactions (as persistent store). The master indexing server (or the only indexing server in single-server environments) polls this database

Indexer Update Principle

The master indexing server polls the persistent store for new entries. If there are new entries, indexer tasks are created in the tables cmas_index_update_task and cmas_index_update_part. The master indexing server then executes the tasks from cmas_index_update_task and cmas_index_batch_update_task. In this way, the (master) index is updated.

On the master (or only) indexing server, administrative tasks are stored in the database table cmas_ index_administrative_task.

Every slave indexing server polls the master indexing server for updates and then updates its own / local index.

Indexer-Related System Properties

Please refer to the Indexer and Search Configuration section of List of System Properties by Area.

CM Indexer Services

For indexing, two ConSol CM services are important:

Index changes notifier
Creates JMS (Java Message Service) messages with notifications that changes have been made which concern the index.
Index changes receiver
Reads the JMS queue and initiates updates in the Indexer.

Please see also section CM Services.

Stopping index changes notifier is NOT safe. If the indexer module discovers that the notifier is stopped and there is a message that has to be sent to the persistent store, the indexer will set the index status configuration property to RED, i.e., signal that the index needs a full synchronization.

However, stopping index changes receiver is safe. After restarting, it will pick up all of the missing changes from the persistent store.

Systems with More Than One Indexing Server

When ConSol CM is run in a cluster of application servers, each cluster node holds a local index in its ${cmas.data}/index/ directory. The index gets updated on the (single) master server, which is then polled by all slave servers to update their copies. As a result, all indexing servers are eventually consistent, i.e., updates are not visible right away, usually with a small delay.

JGroups is used for indexer synchronization (see the JGroups.org web site for more information).