Before you install Kiali you must have already installed Istio along with its telemetry storage addon (i.e. Prometheus). You might also consider installing Istio’s optional tracing addon (i.e. Jaeger) and optional Grafana addon but those are not required by Kiali. Refer to the Istio documentation for details.

Install via Istio Addons

If you downloaded Istio, the easiest way to install and try Kiali is by running:

kubectl apply -f ${ISTIO_HOME}/samples/addons/kiali.yaml

To uninstall:

kubectl delete -f ${ISTIO_HOME}/samples/addons/kiali.yaml --ignore-not-found

Install via Helm

Only Helm v3 has been tested. Previous Helm versions may or may not work.

To install the latest version of Kiali Server using Helm, run the following command:

helm install \
  --namespace istio-system \
  --set auth.strategy="anonymous" \
  --repo https://kiali.org/helm-charts \
  kiali-server \
  kiali-server

If you get a validation error, you may have to pass the option --disable-openapi-validation (this is needed on some versions of OpenShift, for example).

To uninstall:

helm uninstall --namespace istio-system kiali-server

Access to the UI

Run the following command:

kubectl port-forward svc/kiali 20001:20001 -n istio-system

Then, access Kiali by visiting https://localhost:20001/ in your preferred web browser.

1.2 - Installation Guide

Installing Kiali for production.

This section describes the production installation methods available for Kiali.

The recommended way to deploy Kiali is via the Kiali Operator, either using Helm Charts or OperatorHub.

The Kiali Operator is a Kubernetes Operator and manages your Kiali installation. It watches the Kiali Custom Resource (Kiali CR), a YAML file that holds the deployment configuration.

It is only necessary to install the Kiali Operator once. After the operator is installed you only need to create or edit the Kiali CR. Never manually edit resources created by the Kiali Operator.

If you previously installed Kiali via a different mechanism, you must first uninstall Kiali using the original mechanism’s uninstall procedures. There is no migration path between older installation mechanisms and the install mechanisms explained in this documentation.

1.2.1 - Prerequisites

Hardware and Software compatibility and requirements.

Istio

Enable the Debug Interface

Like istioctl, Kiali makes use of Istio’s port 8080 “Debug Interface”. Despite the naming, this is required for accessing the status of the proxies and the Istio registry.

The ENABLE_DEBUG_ON_HTTP setting controls the relevant access. Istio suggests to disable this for security, but Kiali requires ENABLE_DEBUG_ON_HTTP=true, which is the default.

For more information, see the Istio documentation.

Version Compatibility

Each Kiali release is tested against the most recent Istio release. In general, Kiali tries to maintain compatibility with older Istio releases and Kiali versions later than those posted in the below table may work, but such combinations are not tested and will not be supported. Known incompatibilities are noted in the compatibility table below.

It is always recommended that users run a supported version of Istio. The Istio news page posts end-of-support (EOL) dates. Supported Kiali versions include only the Kiali versions associated with supported Istio versions.

Istio	Kiali Min	Kiali Max	Notes
1.23	1.87.0
1.22	1.82.0	1.86.1	Kiali v1.86 is the recommended minimum for Istio Ambient users. v1.22 is required starting with Kiali v1.86.1.
1.21	1.79.0	1.81.0	Kiali 1.82 - 1.86.0 may work, but v1.22 is required starting with Kiali 1.86.1.
1.20	1.76.0	1.78.0	Istio 1.20 is out of support.
1.19	1.72.0	1.75.0	Istio 1.19 is out of support.
1.18	1.67.0	1.73.0	Istio 1.18 is out of support.
1.17	1.63.2	1.66.1	Istio 1.17 is out of support. Avoid 1.63.0,1.63.1 due to a regression.
1.16	1.59.1	1.63.2	Istio 1.16 is out of support. Avoid 1.62.0,1.63.0,1.63.1 due to a regression.
1.15	1.55.1	1.59.0	Istio 1.15 is out of support.
1.14	1.50.0	1.54	Istio 1.14 is out of support.
1.13	1.45.1	1.49	Istio 1.13 is out of support.
1.12	1.42.0	1.44	Istio 1.12 is out of support.
1.11	1.38.1	1.41	Istio 1.11 is out of support.
1.10	1.34.1	1.37	Istio 1.10 is out of support.
1.9	1.29.1	1.33	Istio 1.9 is out of support.
1.8	1.26.0	1.28	Istio 1.8 removes all support for mixer/telemetry V1, as does Kiali 1.26.0. Use earlier versions of Kiali for mixer support.
1.7	1.22.1	1.25	Istio 1.7 istioctl no longer installs Kiali. Use the Istio samples/addons for quick demo installs. Istio 1.7 is out of support.
1.6	1.18.1	1.21	Istio 1.6 introduces CRD and Config changes, Kiali 1.17 is recommended for Istio < 1.6.

Maistra Version Compatibility

Maistra	SMCP CR	Kiali	Notes
2.6	2.6	1.73	Using Maistra 2.6 to install service mesh control plane 2.6 requires Kiali Operator v1.73. Other versions are not compatible.
2.6	2.5	1.73	Using Maistra 2.6 to install service mesh control plane 2.5 requires Kiali Operator v1.73. Other versions are not compatible.
2.6	2.4	1.65	Using Maistra 2.6 to install service mesh control plane 2.4 requires Kiali Operator v1.73. Other versions are not compatible.
2.5	2.5	1.73	Using Maistra 2.5 to install service mesh control plane 2.5 requires Kiali Operator v1.73. Other versions are not compatible.
2.5	2.4	1.65	Using Maistra 2.5 to install service mesh control plane 2.4 requires Kiali Operator v1.73. Other versions are not compatible.
2.4	2.4	1.65	Using Maistra 2.4 to install service mesh control plane 2.4 requires Kiali Operator v1.65. Other versions are not compatible.
n/a	2.3	n/a	Service mesh control plane 2.3 is out of support.
n/a	2.2	n/a	Service mesh control plane 2.2 is out of support.
n/a	2.1	n/a	Service mesh control plane 2.1 is out of support.
n/a	2.0	n/a	Service mesh control plane 2.0 is out of support.
n/a	1.1	n/a	Service mesh control plane 1.1 is out of support.
n/a	1.0	n/a	Service mesh control plane 1.0 is out of support.

OpenShift Console Plugin (OSSMC) Version Compatibility

Kiali server with the same version of OSSMC plugin must be installed previously in your OpenShift cluster.

OpenShift	OSSMC Min	OSSMC Max	Notes
4.15+	1.84
4.12+	1.73	1.83	All OSSMC versions from v1.73 to v1.83 are only compatible with Kiali server v1.73

OpenShift Service Mesh Version Compatibility

OpenShift

If you are running Red Hat OpenShift Service Mesh (OSSM), use only the bundled version of Kiali.

OSSM	Kiali	Notes
2.6	1.73	Same version as 2.5
2.5	1.73
2.4	1.65
2.3	1.57	OSSM 2.3 is out of support
2.2	1.48	OSSM 2.2 is out of support

Browser Compatibility

Kiali requires a modern web browser and supports the last two versions of Chrome, Firefox, Safari or Edge.

Hardware Requirements

Any machine capable of running a Kubernetes based cluster should also be able to run Kiali.

However, Kiali tends to grow in resource usage as your cluster grows. Usually the more namespaces and workloads you have in your cluster, the more memory you will need to allocate to Kiali.

Platform-specific requirements

OpenShift

If you are installing on OpenShift, you must grant the cluster-admin role to the user that is installing Kiali. If OpenShift is installed locally on the machine you are using, the following command should log you in as user system:admin which has this cluster-admin role:

$ oc login -u system:admin

For most commands listed on this documentation, the Kubernetes CLI command kubectl is used to interact with the cluster environment. On OpenShift you can simply replace kubectl with oc, unless otherwise noted.

Google Cloud Private Cluster

Private clusters on Google Cloud have network restrictions. Kiali needs your cluster’s firewall to allow access from the Kubernetes API to the Istio Control Plane namespace, for both the 8080 and 15000 ports.

To review the master access firewall rule:

gcloud compute firewall-rules list --filter="name~gke-${CLUSTER_NAME}-[0-9a-z]*-master"

To replace the existing rule and allow master access:

gcloud compute firewall-rules update <firewall-rule-name> --allow <previous-ports>,tcp:8080,tcp:15000

Istio deployments on private clusters also need extra ports to be opened. Check the Istio installation page for GKE to see all the extra installation steps for this platform.

1.2.2 - Install via Helm

Using Helm to install the Kiali Operator or Server.

Introduction

Helm is a popular tool that lets you manage Kubernetes applications. Applications are defined in a package named Helm chart, which contains all of the resources needed to run an application.

Kiali has a Helm Charts Repository at https://kiali.org/helm-charts. Two Helm Charts are provided:

The kiali-operator Helm Chart installs the Kiali operator which in turn installs Kiali when you create a Kiali CR.
The kiali-server Helm Chart installs a standalone Kiali without the need of the Operator nor a Kiali CR.

The kiali-server Helm Chart does not provide all the functionality that the Operator provides. Some features you read about in the documentation may only be available if you install Kiali using the Operator. Therefore, although the kiali-server Helm Chart is actively maintained, it is not recommended and is only provided for convenience. If using Helm, the recommended method is to install the kiali-operator Helm Chart and then create a Kiali CR to let the Operator deploy Kiali.

Make sure you have the helm command available by following the Helm installation docs.

Helm version 3.10 is the minimum required Helm version. Older versions will not work. Newer versions have not been tested.

Adding the Kiali Helm Charts repository

Add the Kiali Helm Charts repository with the following command:

$ helm repo add kiali https://kiali.org/helm-charts

All helm commands in this page assume that you added the Kiali Helm Charts repository as shown.

If you already added the repository, you may want to update your local cache to fetch latest definitions by running:

$ helm repo update

Installing Kiali using the Kiali operator

This installation method gives Kiali access to existing namespaces as well as namespaces created later. See Namespace Management for more information.

Once you’ve added the Kiali Helm Charts repository, you can install the latest Kiali Operator along with the latest Kiali server by running the following command:

$ helm install \
    --set cr.create=true \
    --set cr.namespace=istio-system \
    --set cr.spec.auth.strategy="anonymous" \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

The --namespace kiali-operator and --create-namespace flags instructs to create the kiali-operator namespace (if needed), and deploy the Kiali operator on it. The --set cr.create=true and --set cr.namespace=istio-system flags instructs to create a Kiali CR in the istio-system namespace. Since the Kiali CR is created in advance, as soon as the Kiali operator starts, it will process it to deploy Kiali. After Kiali has started, you can access Kiali UI through ‘http://localhost:20001’ by executing kubectl port-forward service/kiali -n istio-system 20001:20001 because of --set cr.spec.auth.strategy="anonymous". But realize that anonymous mode will allow anyone to be able to see and use Kiali. If you wish to require users to authenticate themselves by logging into Kiali, use one of the other auth strategies.

The Kiali Operator Helm Chart is configurable. Check available options and default values by running:

$ helm show values kiali/kiali-operator

You can pass the --version X.Y.Z flag to the helm install and helm show values commands to work with a specific version of Kiali.

The kiali-operator Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_root setting which you can configure in the kiali-operator Helm Chart by passing --set cr.spec.server.web_root=/your-path to the helm install command.

For more information about the Kiali CR, see the Creating and updating the Kiali CR page.

Operator-Only Install

To install only the Kiali Operator, omit the --set cr.create and --set cr.namespace flags of the helm command previously shown. For example:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

This will omit creation of the Kiali CR, which you will need to create later to install Kiali Server. This option is good if you plan to do large customizations to the installation.

Installing Multiple Instances of Kiali

By installing a single Kiali operator in your cluster, you can install multiple instances of Kiali by simply creating multiple Kiali CRs. For example, if you have two Istio control planes in namespaces istio-system and istio-system2, you can create a Kiali CR in each of those namespaces to install a Kiali instance in each control plane.

If you wish to install multiple Kiali instances in the same namespace, or if you need the Kiali instance to have different resource names than the default of kiali, you can specify spec.deployment.instance_name in your Kiali CR. The value for that setting will be used to create a unique instance of Kiali using that instance name rather than the default kiali. One use-case for this is to be able to have unique Kiali service names across multiple Kiali instances in order to be able to use certain routers/load balancers that require unique service names.

Since the spec.deployment.instance_name field is used for the Kiali resource names, including the Service name, you must ensure the value you assign this setting follows the Kubernetes DNS Label Name rules. If it does not, the operator will abort the installation. And note that because Kiali uses this as a prefix (it may append additional characters for some resource names) its length is limited to 40 characters.

Standalone Kiali installation

To install the Kiali Server without the operator, use the kiali-server Helm Chart:

$ helm install \
    --namespace istio-system \
    kiali-server \
    kiali/kiali-server

The kiali-server Helm Chart mirrors all settings of the Kiali CR as chart values that you can configure using regular --set flags. For example, the Kiali CR has a spec.server.web_fqdn setting which you can configure in the kiali-server Helm Chart by passing the --set server.web_fqdn flag as follows:

$ helm install \
    --namespace istio-system \
    --set server.web_fqdn=example.com \
    kiali-server \
    kiali/kiali-server

Upgrading Helm installations

If you want to upgrade to a newer Kiali version (or downgrade to older versions), you can use the regular helm upgrade commands. For example, the following command should upgrade the Kiali Operator to the latest version:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    kiali-operator \
    kiali/kiali-operator

WARNING: No migration paths are provided. However, Kiali is a stateless application and if the helm upgrade command fails, please uninstall the previous version and then install the new desired version.

By upgrading the Kiali Operator, existent Kiali Server installations managed with a Kiali CR will also be upgraded once the updated operator starts.

Managing configuration of Helm installations

After installing either the kiali-operator or the kiali-server Helm Charts, you may be tempted to manually modify the created resources to modify the installation. However, we recommend using helm upgrade to update your installation.

For example, assuming you have the following installation:

$ helm list -n kiali-operator
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
kiali-operator  kiali-operator  1               2021-09-14 18:00:45.320351026 -0500 CDT deployed        kiali-operator-1.40.0   v1.40.0

Notice that the current installation is version 1.40.0 of the kiali-operator. Let’s assume you want to use your own mirrors of the Kiali Operator container images. You can update your installation with the following command:

$ helm upgrade \
    --namespace kiali-operator \
    --reuse-values \
    --set image.repo=your_mirror_registry_url/owner/kiali-operator-repo \
    --set image.tag=your_mirror_tag \
    --version 1.40.0 \
    kiali-operator \
    kiali/kiali-operator

Make sure that you specify the --reuse-values flag to take the configuration of your current installation. Then, you only need to specify the new settings you want to change using --set flags.

Make sure that you specify the --version X.Y.Z flag with the version of your current installation. Otherwise, you may end up upgrading to a new version.

Uninstalling

Removing the Kiali operator and managed Kialis

If you used the kiali-operator Helm chart, first you must ensure that all Kiali CRs are deleted. For example, the following command will agressively delete all Kiali CRs in your cluster:

$ kubectl delete kiali --all --all-namespaces

The previous command may take some time to finish while the Kiali operator removes all Kiali installations.

Then, remove the Kiali operator using a standard helm uninstall command. For example:

$ helm uninstall --namespace kiali-operator kiali-operator
$ kubectl delete crd kialis.kiali.io

You have to manually delete the kialis.kiali.io CRD because Helm won’t delete it.

If you fail to delete the Kiali CRs before uninstalling the operator, a proper cleanup may not be done.

Known problem: uninstall hangs (unable to delete the Kiali CR)

Typically this happens if not all Kiali CRs are deleted prior to uninstalling the operator. To force deletion of a Kiali CR, you need to clear its finalizer. For example:

$ kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge

This forces deletion of the Kiali CR and will skip uninstallation of the Kiali Server. Remnants of the Kiali Server may still exist in your cluster which you will need to manually remove.

Removing standalone Kiali

If you installed a standalone Kiali by using the kiali-server Helm chart, use the standard helm uninstall commands. For example:

$ helm uninstall --namespace istio-system kiali-server

1.2.3 - Install via OperatorHub

Using OperatorHub to install the Kiali Operator.

Introduction

The OperatorHub is a website that contains a catalog of Kubernetes Operators. Its aim is to be the central location to find Operators.

The OperatorHub relies in the Operator Lifecycle Manager (OLM) to install, manage and update Operators on any Kubernetes cluster.

The Kiali Operator is being published to the OperatorHub. So, you can use the OLM to install and manage the Kiali Operator installation.

Installing the Kiali Operator using the OLM

Go to the Kiali Operator page in the OperatorHub: https://operatorhub.io/operator/kiali.

You will see an Install button at the right of the page. Press it and you will be presented with the installation instructions. Follow these instructions to install and manage the Kiali Operator installation using OLM.

Afterwards, you can create the Kiali CR to install Kiali.

Installing the Kiali Operator in OpenShift

The OperatorHub is bundled in the OpenShift console. To install the Kiali Operator, simply go to the OperatorHub in the OpenShift console and search for the Kiali Operator. Then, click on the Install button and follow the instruction on the screen.

Afterwards, you can create the Kiali CR to install Kiali.

1.2.4 - The Kiali CR

Creating and updating the Kiali CR.

The Kiali Operator watches the Kiali Custom Resource (Kiali CR), a custom resource that contains the Kiali Server deployment configuration. Creating, updating, or removing a Kiali CR will trigger the Kiali Operator to install, update, or remove Kiali.

If you want the operator to re-process the Kiali CR (called “reconciliation”) without having to change the Kiali CR’s spec fields, you can modify any annotation on the Kiali CR itself. This will trigger the operator to reconcile the current state of the cluster with the desired state defined in the Kiali CR, modifying cluster resources if necessary to get them into their desired state. Here is an example illustrating how you can modify an annotation on a Kiali CR:

$ kubectl annotate kiali my-kiali -n istio-system --overwrite kiali.io/reconcile="$(date)"

The Operator provides comprehensive defaults for all properties of the Kiali CR. Hence, the minimal Kiali CR does not have a spec:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali

Assuming you saved the previous YAML to a file named my-kiali-cr.yaml, and that you are installing Kiali in the same default namespace as Istio, create the resource with the following command:

$ kubectl apply -f my-kiali-cr.yaml -n istio-system

Often, but not always, Kiali is installed in the same namespace as Istio, thus the Kiali CR is also created in the Istio namespace.

Once created, the Kiali Operator should shortly be notified and will process the resource, performing the Kiali installation. You can wait for the Kiali Operator to finish the reconcilation by using the standard kubectl wait command and ask for it to wait for the Kiali CR to achieve the condition of Successful. For example:

kubectl wait --for=condition=Successful kiali kiali -n istio-system

You can check the installation progress by inspecting the status attribute of the created Kiali CR:

$ kubectl describe kiali kiali -n istio-system
Name:         kiali
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>
API Version:  kiali.io/v1alpha1
Kind:         Kiali

  (...some output is removed...)

Status:
  Conditions:
    Last Transition Time:  2021-09-15T17:17:40Z
    Message:               Running reconciliation
    Reason:                Running
    Status:                True
    Type:                  Running
  Deployment:
    Instance Name:  kiali
    Namespace:      istio-system
  Environment:
    Is Kubernetes:       true
    Kubernetes Version:  1.21.2
    Operator Version:    v1.40.0
  Progress:
    Duration:  0:00:16
    Message:   5. Creating core resources
Events:        <none>

Never manually edit resources created by the Kiali Operator, only the Kiali CR.

You may want to check the example install page to see some examples where the Kiali CR has a spec and to better understand its structure. Most available attributes of the Kiali CR are described in the pages of the Installation and Configuration sections of the documentation. For a complete list, see the Kiali CR Reference.

It is important to understand the spec.deployment.accessible_namespaces setting in the CR. See the Namespace Management page for more information.

Once you created a Kiali CR, you can manage your Kiali installation by editing the resource using the usual Kubernetes tools:

$ kubectl edit kiali kiali -n istio-system

To confirm your Kiali CR is valid, you can utilize the Kiali CR validation tool.

1.2.5 - The OSSMConsole CR

Creating and updating the OSSMConsole CR.

OpenShift ServiceMesh Console (aka OSSMC) provides a Kiali integration with the OpenShift Console; in other words it provides Kiali functionality within the context of the OpenShift Console. OSSMC is applicable only within OpenShift environments.

The main component of OSSMC is a plugin that gets installed inside the OpenShift Console. Prior to installing this plugin, you are required to have already installed the Kiali Operator and Kiali Server in your OpenShift environment. Please the Installation Guide for details.

There are no helm charts available to install OSSMC. You must utilize the Kiali Operator to install it. Installing the Kiali Operator on OpenShift is very easy due to the Operator Lifecycle Manager (OLM) functionality that comes with OpenShift out-of-box. Simply elect to install the Kiali Operator from the Red Hat or Community Catalog from the OperatorHub page in OpenShift Console.

The Kiali Operator watches the OSSMConsole Custom Resource (OSSMConsole CR), a custom resource that contains the OSSMC deployment configuration. Creating, updating, or removing a OSSMConsole CR will trigger the Kiali Operator to install, update, or remove OSSMC.

Never manually edit resources created by the Kiali Operator, only edit the OSSMConsole CR.

Creating the OSSMConsole CR to Install the OSSMC Plugin

With the Kiali Operator and Kial Server installed and running, you can install the OSSMC plugin in one of two ways - either via the OpenShift Console or via the “oc” CLI. Both methods are described below. You choose the method you want to use.

Installing via OpenShift Console

From the Kiali Operator details page in the OpenShift Console, create an instance of the “OpenShift Service Mesh Console” resource. Accept the defaults on the installation form and press “Create”.

Install Plugin

Installing via “oc” CLI

To instruct the Kiali Operator to install the plugin, simply create a small OSSMConsole CR. A minimal CR can be created like this:

cat <<EOM | oc apply -f -
apiVersion: kiali.io/v1alpha1
kind: OSSMConsole
metadata:
  namespace: openshift-operators
  name: ossmconsole
EOM

Note that the operator will deploy the plugin resources in the same namespace where you create this OSSMConsole CR - in this case openshift-operators but you can create the CR in any namespace.

For a complete list of configuration options available within the OSSMConsole CR, see the OSSMConsole CR Reference.

To confirm your OSSMConsole CR is valid, you can utilize the OSSMConsole CR validation tool.

Installation Status

After the plugin is installed, you can see the “OSSMConsole” resource that was created in the OpenShift Console UI. Within the operator details page in the OpenShift Console UI, select the OpenShift Service Mesh Console tab to view the resource that was created and its status. The CR status field will provide you with any error messages should the deployment of OSSMC fail.

Installed Plugin

Once the operator has finished processing the OSSMConsole CR, you must then wait for the OpenShift Console to load and initialize the plugin. This may take a minute or two. You will know when the plugin is ready when the OpenShift Console pops up this message - when you see this message, refresh the browser window to reload the OpenShift Console:

Plugin Ready

Uninstalling OSSMC

This section will describe how to uninstall the OpenShift Service Mesh Console plugin. You can uninstall the plugin in one of two ways - either via the OpenShift Console or via the “oc” CLI. Both methods are described in the sections below. You choose the method you want to use.

If you intend to also uninstall the Kiali Operator, it is very important to first uninstall the OSSMConsole CR and then uninstall the operator. If you uninstall the operator before ensuring the OSSMConsole CR is deleted then you may have difficulty removing that CR and its namespace. If this occurs then you must manually remove the finalizer on the CR in order to delete it and its namespace. You can do this via: oc patch ossmconsoles <CR name> -n <CR namespace> -p '{"metadata":{"finalizers": []}}' --type=merge

Uninstalling via OpenShift Console

Remove the OSSMConsole CR by navigating to the operator details page in the OpenShift Console UI. From the operator details page, select the OpenShift Service Mesh Console tab and then select the Delete option in the kebab menu.

Uninstall Plugin

Uninstalling via “oc” CLI

Remove the OSSMConsole CR via oc delete ossmconsoles <CR name> -n <CR namespace>. To make sure any and all CRs are deleted from any and all namespaces, you can run this command:

for r in $(oc get ossmconsoles --ignore-not-found=true --all-namespaces -o custom-columns=NS:.metadata.namespace,N:.metadata.name --no-headers | sed 's/  */:/g'); do oc delete ossmconsoles -n $(echo $r|cut -d: -f1) $(echo $r|cut -d: -f2); done

1.2.6 - Accessing Kiali

Accessing and exposing the Kiali UI.

Introduction

After Kiali is succesfully installed you will need to make Kiali accessible to users. This page describes some popular methods of exposing Kiali for use.

If exposing Kiali in a custom way, you may need to set some configurations to make Kiali aware of how users will access Kiali.

The examples on this page assume that you followed the Installation guide to install Kiali, and that you installed Kiali in the istio-system namespace.

Accessing Kiali using port forwarding

This method should work in any kind of Kubernetes cluster.

You can use port-forwarding to access Kiali by running any of these commands:

# If you have oc command line tool
oc port-forward svc/kiali 20001:20001 -n istio-system
# If you have kubectl command line tool
kubectl port-forward svc/kiali 20001:20001 -n istio-system

These commands will block. Access Kiali by visiting https://localhost:20001/ in your preferred web browser.

Please note that this method exposes Kiali only to the local machine, no external users. You must have the necessary privileges to perform port forwarding.

Accessing Kiali through an Ingress

You can configure Kiali to be installed with an Ingress resource defined, allowing you to access the Kiali UI through the Ingress. By default, an Ingress will not be created. You can enable a simple Ingress by setting spec.deployment.ingress.enabled to true in the Kiali CR (a similar setting for the server Helm chart is available if you elect to install Kiali via Helm as opposed to the Kiali Operator).

Exposing Kiali externally through this spec.deployment.ingress mechanism is a convenient way of exposing Kiali externally but it will not necessarily work or be the best way to do it because the way in which you should expose Kiali externally will be highly dependent on your specific cluster environment and how services are exposed generally for that environment.

When installing on an OpenShift cluster, an OpenShift Route will be installed (not an Ingress). This Route will be installed by default unless you explicitly disable it via spec.deployment.ingress.enabled: false. Note that the Route is required if you configure Kiali to use the auth strategy of openshift (which is the default auth strategy Kiali will use when installed on OpenShift).

The default Ingress that is created will be configured for a typical NGinx implementation. If you have your own Ingress implementation you want to use, you can override the default configuration through the settings spec.deployment.ingress.override_yaml and spec.deployment.ingress.class_name. More details on customizing the Ingress can be found below.

The Ingress IP or domain name should then be used to access the Kiali UI. To find your Ingress IP or domain name, as per the Kubernetes documentation, try the following command (though this may not work if using Minikube without the ingress addon):

kubectl get ingress kiali -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

If it doesn’t work, unfortunately, it depends on how and where you had setup your cluster. There are several Ingress controllers available and some cloud providers have their own controller or preferred exposure method. Check the documentation of your cloud provider. You may need to customize the pre-installed Ingress rule or expose Kiali using a different method.

Customizing the Ingress resource

The created Ingress resource will route traffic to Kiali regardless of the domain in the URL. You may need a more specific Ingress resource that routes traffic to Kiali only on a specific domain or path. To do this, you can specify route settings.

Alternatively, and for more advanced Ingress configurations, you can provide your own Ingress declaration in the Kiali CR. For example:

When installing on an OpenShift cluster, the deployment.ingress.override_yaml will be applied to the created Route. The deployment.ingress.class_name is ignored on OpenShift.

spec:
  deployment:
    ingress:
      class_name: "nginx"
      enabled: true
      override_yaml:
        metadata:
          annotations:
            nginx.ingress.kubernetes.io/secure-backends: "true"
            nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        spec:
          rules:
          - http:
              paths:
              - path: /kiali
                backend:
                  serviceName: kiali
                  servicePort: 20001

Accessing Kiali in Minikube

If you enabled the Ingress controller, the default Ingress resource created by the installation (mentioned in the previous section) should be enough to access Kiali. The following command should open Kiali in your default web browser:

xdg-open https://$(minikube ip)/kiali

Accessing Kiali through a LoadBalancer or a NodePort

By default, the Kiali service is created with the ClusterIP type. To use a LoadBalancer or a NodePort, you can change the service type in the Kiali CR as follows:

spec:
  deployment:
    service_type: LoadBalancer

Once the Kiali operator updates the installation, you should be able to use the kubectl get svc -n istio-system kiali command to retrieve the external address (or port) to access Kiali. For example, in the following output Kiali is assigned the IP 192.168.49.201, which means that you can access Kiali by visiting http://192.168.49.201:20001 in a browser:

NAME    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                          AGE
kiali   LoadBalancer   10.105.236.127   192.168.49.201   20001:31966/TCP,9090:30128/TCP   34d

If you are using the LoadBalancer service type to directly expose the Kiali service, you may want to check the available options for the HTTP Server and Metrics server.

Accessing Kiali through an Istio Ingress Gateway

If you want to take advantage of Istio’s infrastructure, you can expose Kiali using an Istio Ingress Gateway. The Istio documentation provides a good guide explaining how to expose the sample add-ons. Even if the Istio guide is focused on the sample add-ons, the steps are the same to expose a Kiali installed using this Installation guide.

Accessing Kiali in OpenShift

By default, Kiali is exposed through a Route if installed on OpenShift. The following command should open Kiali in your default web browser:

xdg-open https://$(oc get routes -n istio-system kiali -o jsonpath='{.spec.host}')/console

Specifying route settings

If you are using your own exposure method or if you are using one of the methods mentioned in this page, you may need to configure the route that is being used to access Kiali.

In the Kiali CR, route settings are broken in several attributes. For example, to specify that Kiali is being accessed under the https://apps.example.com:8080/dashboards/kiali URI, you would need to set the following:

spec:
  server:
    web_fqdn: apps.example.com
    web_port: 8080
    web_root: /dashboards/kiali
    web_schema: https

If you are letting the installation create an Ingress resource for you, the Ingress will be adjusted to match these route settings. If you are using your own exposure method, these spec.server settings are only making Kiali aware of what its public endpoint is.

It is possible to omit these settings and Kiali may be able to discover some of these configurations, depending on your exposure method. For example, if you are exposing Kiali via LoadBalancer or NodePort service types, Kiali can discover most of these settings. If you are using some kind of Ingress, Kiali will honor X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-Port HTTP headers if they are properly injected in the request.

The web_root receives special treatment, because this is the path where Kiali will serve itself (both the user interface and its api). This is useful if you are serving multiple applications under the same domain. It must begin with a slash and trailing slashes must be omitted. The default value is /kiali for Kubernetes and / for OpenShift.

Usually, these settings can be omitted. However, a few features require that the Kiali’s public route be properly discoverable or that it is properly configured; most notably, the OpenID authentication.

1.2.7 - Advanced Install

Advanced installation options.

Canary upgrades

During a canary upgrade where multiple controlplanes are present, Kiali will automatically detect both controlplanes. You can visit the mesh page to visualize your controlplanes during a canary upgrade. You can also manually specify a controlplane you want Kiali to connect to through the Kiali CR however if you do this then you will need to update these settings during each canary upgrade.

spec:
  external_services:
    istio:
      config_map_name: "istio"
      istiod_deployment_name: "istiod"
      istio_sidecar_injector_config_map_name: "istio-sidecar-injector"

Installing a Kiali Server of a different version than the Operator

When you install the Kiali Operator, it will be configured to install a Kiali Server that is the same version as the operator itself. For example, if you have Kiali Operator v1.34.0 installed, that operator will install Kiali Server v1.34.0. If you upgrade (or downgrade) the Kiali Operator, the operator will in turn upgrade (or downgrade) the Kiali Server.

There are certain use-cases in which you want the Kiali Operator to install a Kiali Server whose version is different than the operator version. Read the following section «Using a custom image registry» section to learn how to configure this setup.

Using a custom image registry

Kiali is released and published to the Quay.io container image registry. There is a repository hosting the Kiali operator images and another one for the Kiali server images.

If you need to mirror the Kiali container images to some other registry, you still can use Helm to install the Kiali operator as follows:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set image.repo=your.custom.registry/owner/kiali-operator-repo
    --set image.tag=your_custom_tag
    --set allowAdHocKialiImage=true
    kiali-operator \
    kiali/kiali-operator

Notice the --set allowAdHocKialiImage=true which allows specifying a custom image in the Kiali CR. For security reasons, this is disabled by default.

Then, when creating the Kiali CR, use the following attributes:

spec:
  deployment:
    image_name: your.custom.registry/owner/kiali-server-repo
    image_version: your_custom_tag

Change the default image

As explained earlier, when you install the Kiali Operator, it will be configured to install a Kiali Server whose image will be pulled from quay.io and whose version will be the same as the operator. You can ask the operator to use a different image by setting spec.deployment.image_name and spec.deployment.image_version within the Kiali CR (as explained above).

However, you may wish to alter this default behavior exhibited by the operator. In other words, you may want the operator to install a different Kiali Server image by default. For example, if you have an air-gapped environment with its own image registry that contains its own copy of the Kiali Server image, you will want the operator to install a Kiali Server that uses that image by default, as opposed to quay.io/kiali/kiali. By configuring the operator to do this, you will not force the authors of Kiali CRs to have to explicitly define the spec.deployment.image_name setting and you will not need to enable the allowAdHocKialiImage setting in the operator.

To change the default Kiali Server image installed by the operator, set the environment variable RELATED_IMAGE_kiali_default in the Kiali Operator deployment. The value of that environment variable must be the full image tag in the form repoName/orgName/imageName:versionString (e.g. my.internal.registry.io/mykiali/mykialiserver:v1.50.0). You can do this when you install the operator via helm:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set "env[0].name=RELATED_IMAGE_kiali_default" \
    --set "env[0].value=my.internal.registry.io/mykiali/mykialiserver:v1.50.0" \
    kiali-operator \
    kiali/kiali-operator

Development Install

This option installs the latest Kiali Operator and Kiali Server images which are built from the master branches of Kiali GitHub repositories. This option is good for demo and development installations.

helm install \
  --set cr.create=true \
  --set cr.namespace=istio-system \
  --set cr.spec.deployment.image_version=latest \
  --set image.tag=latest \
  --namespace kiali-operator \
  --create-namespace \
  kiali-operator \
  kiali/kiali-operator

1.2.8 - Example Install

Installing two Kiali servers via the Kiali Operator.

This is a quick example of installing Kiali. This example will install the operator and two Kiali Servers - one server will require the user to enter credentials at a login screen in order to obtain read-write access and the second server will allow anonymous read-only access.

For this example, assume there is a Minikube Kubernetes cluster running with an Istio control plane installed in the namespace istio-system and the Istio Bookinfo Demo installed in the namespace bookinfo:

$ kubectl get deployments.apps -n istio-system
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
grafana                1/1     1            1           8h
istio-egressgateway    1/1     1            1           8h
istio-ingressgateway   1/1     1            1           8h
istiod                 1/1     1            1           8h
jaeger                 1/1     1            1           8h
prometheus             1/1     1            1           8h

$ kubectl get deployments.apps -n bookinfo
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
details-v1       1/1     1            1           21m
productpage-v1   1/1     1            1           21m
ratings-v1       1/1     1            1           21m
reviews-v1       1/1     1            1           21m
reviews-v2       1/1     1            1           21m
reviews-v3       1/1     1            1           21m

Install Kiali Operator via Helm Chart

First, the Kiali Operator will be installed in the kiali-operator namespace using the operator helm chart:

$ helm repo add kiali https://kiali.org/helm-charts
$ helm repo update kiali
$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    kiali-operator \
    kiali/kiali-operator

Install Kiali Server via Operator

Next, the first Kiali Server will be installed. This server will require the user to enter a Kubernetes token in order to log into the Kiali dashboard and will provide the user with read-write access. To do this, a Kiali CR will be created that looks like this (file: kiali-cr-token.yaml):

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: "token"
  deployment:
    accessible_namespaces: ["bookinfo"]
    view_only_mode: false
  server:
    web_root: "/kiali"

This Kiali CR will command the operator to deploy the Kiali Server in the same namespace where the Kiali CR is (istio-system). The operator will configure the server to: respond to requests to the web root path of /kiali, enable read-write access, use the authentication strategy of token, and be given access to the bookinfo namespace:

$ kubectl apply -f kiali-cr-token.yaml

Get the Status of the Installation

The status of a particular Kiali Server installation can be found by examining the status field of its corresponding Kiali CR. For example:

$ kubectl get kiali kiali -n istio-system -o jsonpath='{.status}'

When the installation has successfully completed, the status field will look something like this (when formatted):

$ kubectl get kiali kiali -n istio-system -o jsonpath='{.status}' | jq
{
  "conditions": [
    {
      "ansibleResult": {
        "changed": 21,
        "completion": "2021-10-20T19:17:35.519131",
        "failures": 0,
        "ok": 102,
        "skipped": 90
      },
      "lastTransitionTime": "2021-10-20T19:17:12Z",
      "message": "Awaiting next reconciliation",
      "reason": "Successful",
      "status": "True",
      "type": "Running"
    }
  ],
  "deployment": {
    "accessibleNamespaces": "bookinfo,istio-system",
    "instanceName": "kiali",
    "namespace": "istio-system"
  },
  "environment": {
    "isKubernetes": true,
    "kubernetesVersion": "1.20.2",
    "operatorVersion": "v1.41.0"
  },
  "progress": {
    "duration": "0:00:20",
    "message": "7. Finished all resource creation"
  }
}

Access the Kiali Server UI

The Kiali Server UI is accessed by pointing a browser to the Kiali Server endpoint and requesting the web root /kiali:

xdg-open http://$(minikube ip)/kiali

Because the auth.strategy was set to token, that URL will display the Kiali login screen that will require a Kubernetes token in order to authenticate with the server. For this example, you can use the token that belongs to the Kiali service account itself:

$ kubectl get secret -n istio-system $(kubectl get sa kiali-service-account -n istio-system -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 -d

The output of that command above can be used to log into the Kiali login screen.

Install a Second Kiali Server

The second Kiali Server will next be installed. This server will not require the user to enter any login credentials but will only provide a view-only look at the service mesh. To do this, a Kiali CR will be created that looks like this (file: kiali-cr-anon.yaml):

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: kialianon
spec:
  installation_tag: "Kiali - View Only"
  istio_namespace: "istio-system"
  auth:
    strategy: "anonymous"
  deployment:
    accessible_namespaces: ["bookinfo"]
    view_only_mode: true
    instance_name: "kialianon"
  server:
    web_root: "/kialianon"

This Kiali CR will command the operator to deploy the Kiali Server in the same namespace where the Kiali CR is (kialianon). The operator will configure the server to: respond to requests to the web root path of /kialianon, disable read-write access, not require the user to authenticate, have a unique instance name of kialianon and be given access to the bookinfo namespace. The Kiali UI will also show a custom title in the browser tab so the user is aware they are looking at a “view only” Kiali dashboard. The unique deployment.instance_name is needed in order for this Kiali Server to be able to share access to the Bookinfo application with the first Kiali Server.

$ kubectl create namespace kialianon
$ kubectl apply -f kiali-cr-anon.yaml

The UI for this second Kiali Server is accessed by pointing a browser to the Kiali Server endpoint and requesting the web root /kialianon. Note that no credentials are required to gain access to this Kiali Server UI because auth.strategy was set to anonymous; however, the user will not be able to modify anything via the Kiali UI - it is strictly “view only”:

xdg-open http://$(minikube ip)/kialianon

Reconfigure Kiali Server

A Kiali Server can be reconfigured by simply editing its Kiali CR. The Kiali Operator will perform all the necessary tasks to complete the reconfiguration and reboot the Kiali Server pod when necessary. For example, to change the web root for the Kiali Server:

$ kubectl patch kiali kiali -n istio-system --type merge --patch '{"spec":{"server":{"web_root":"/specialkiali"}}}'

The Kiali Operator will update the necessary resources (such as the Kiali ConfigMap) and will reboot the Kiali Server pod to pick up the new configuration.

Uninstall Kiali Server

To uninstall a Kiali Server installation, simply delete the Kiali CR. The Kiali Operator will then perform all the necessary tasks to remove all remnants of the associated Kiali Server.

kubectl delete kiali kiali -n istio-system

Uninstall Kiali Operator

To uninstall the Kiali Operator, use helm uninstall and then manually remove the Kiali CRD.

You must delete all Kiali CRs in the cluster prior to uninstalling the Kiali Operator. If you fail to do this, uninstalling the operator will hang and remnants of Kiali Server installations will remain in your cluster and you will be required to perform some manual steps to clean it up.

$ kubectl delete kiali --all --all-namespaces
$ helm uninstall --namespace kiali-operator kiali-operator
$ kubectl delete crd kialis.kiali.io

1.3 - Deployment Options

Simple configuration options related to the Kiali deployment, like the installation namespace, logger settings, resource limits and scheduling options for the Kiali pod.

There are other, more complex deployment settings, described in dedicated pages:

All examples on this page are focused on the Kiali CR (when installing via the Kiali operator). Remember that Helm charts mirror all these configurations.

Kiali and Istio installation namespaces

By default, the Kiali operator installs Kiali in the same namespace where the Kiali CR is created. However, it is possible to specify a different namespace for installation:

spec:
  deployment:
    namespace: "custom-kiali-namespace"

It is assumed that Kiali is installed to the same namespace as Istio. Kiali reads some Istio resources and may not work properly if those resources are not found. Thus, if you are installing Kiali and Istio on different namespaces, you must specify what is the Istio namespace:

spec:
  istio_namespace: "istio-system"

Log level and format

By default, Kiali will print up to INFO-level messages in simple text format. You can change the log level, output format, and time format as in the following example:

spec:
  deployment:
    logger:
      # Supported values are "trace", "debug", "info", "warn", "error" and "fatal"
      log_level: error  
      # Supported values are "text" and "json".
      log_format: json  
      time_field_format: "2006-01-02T15:04:05Z07:00"

The syntax for the time_field_format is the same as the Time.Format function of the Go language.

The json format is useful if you are parsing logs of your applications for further processing.

In Kiali, there are some special logs called audit logs that are emitted each time a user creates, updates or deletes a resource through Kiali. Audit logs are INFO-level messages and are enabled by default. If audit logs are too verbose, you can disable them without reducing the log level as follows:

spec:
  server:
    audit_log: false

Kiali instance name

If you plan to install more than one Kiali instance on the same cluster, you may need to configure an instance name to avoid conflicts on created resources:

spec:
  deployment:
    instance_name: "secondary"

The instance_name will be used as a prefix for all created Kiali resources. The exception is the kiali-signing-key secret which will always have the same name and will be shared on all deployments of the same namespace, unless you specify a custom secret name.

Since the instance_name will be used as a name prefix in resources, it must follow Kubernetes naming constraints.

Since Kubernetes resources cannot be renamed, you cannot change the instance_name of an existing Kiali installation. The workaround is to uninstall Kiali and re-install with the desired instance_name.

Resource requests and limits

You can set the amount of resources available to Kiali using the spec.deployment.resources attribute, like in the following example:

spec:
  deployment:
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "1Gi"
        cpu: "500m"

Please, read the Managing Resources for Containers section in the Kubernetes documentation for more details about possible configurations.

Custom labels and annotations on the Kiali pod and service

Although some labels and annotations are set on the Kiali pod and on its service (depending on configurations), you can add additional ones. For the pod, use the spec.deployment.pod_labels and spec.deployment.pod_annotations attributes. For the service, you can only add annotations using the spec.deployment.service_annotations attribute. For example:

spec:
  deployment:
    pod_annotations:
      a8r.io/repository: "https://github.com/kiali/kiali"
    pod_labels:
      sidecar.istio.io/inject: "true"
    service_annotations:
      a8r.io/documentation: "https://kiali.io/docs/installation/deployment-configuration"

Kiali page title (browser title bar)

If you have several Kiali installations and you are using them at the same time, there are good chances that you will want to identify each Kiali by simply looking at the browser’s title bar. You can set a custom text in the title bar with the following configuration:

spec:
  installation_tag: "Kiali West"

The installation_tag is any human readable text of your desire.

Kubernetes scheduler settings

Replicas and automatic scaling

By default, only one replica of Kiali is deployed. If needed, you can change the replica count like in the following example:

spec:
  deployment:
    replicas: 2

If you prefer automatic scaling, creation of an HorizontalPodAutoscaler resource is supported. For example, the following configuration automatically scales Kiali based on CPU utilization:

spec:
  deployment:
    hpa:
      api_version: "autoscaling/v1"
      spec:
        minReplicas: 1
        maxReplicas: 2
        targetCPUUtilizationPercentage: 80

You must omit the scaleTargetRef field of the HPA spec, because this field will be populated by the Kiali operator (or by Helm) depending on other configuration.

Read the Kubernetes Horizontal Pod Autoscaler documentation to learn more about the HPA.

Allocating the Kiali pod to specific nodes of the cluster

You can constrain the Kiali pod to run on a specific set of nodes by using some of the standard Kubernetes scheduler configurations.

The simplest option is to use the nodeSelector configuration which you can configure like in the following example:

spec:
  deployment:
    node_selector:
      worker-type: infra

You can also use the affinity/anti-affinity native Kubernetes feature if you prefer its more expressive syntax, or if you need more complex matching rules. The following is an example for configuring node affinity:

spec:
  deployment:
    affinity:
      node:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: worker-type
              operator: In
              values:
              - infra

Similarly, you can also configure pod affinity and pod anti-affinity using the spec.deployment.affinity.pod and spec.deployment.affinity.pod_anti attributes.

Finally, if you want to run Kiali in a node with taints, the following is an example to configure tolerations:

spec:
  deployment:
    tolerations: # Allow to run Kiali in a tainted master node
    - key: "node-role.kubernetes.io/master"
      operator: "Exists"
      effect: "NoSchedule"

Read the following Kubernetes documentation to learn more about these configurations:

Priority class of the Kiali pod

If you are using priority classes in your cluster, you can specify the priority class that will be set on the Kiali pod. For example:

spec:
  deployment:
    priority_class_name: high-priority

For more information about priority classes, read Pod Priority and Preemption in the Kubernetes documentation.

Adding host aliases to the Kiali pod

If you need to provide some static hostname resolution in the Kiali pod, you can use HostAliases to add entries to the /etc/hosts file of the Kiali pod, like in the following example:

spec:
  deployment:
   host_aliases:
   - ip: 192.168.1.100
     hostnames:
     - "foo.local"
     - "bar.local"

HTTP server

Kiali is served over HTTP. You can configure a few options of the HTTP server. The following are the defaults, but you can change them to suit your needs.

spec:
  server:
    # Listen/bind address of the HTTP server. By default it is empty, which means to
    # listen on all interfaces.
    address: ""

    # Listening port of the HTTP server. If you change it, also Kiali's Kubernetes
    # Service is affected to use this port.
    port: 20001

    # Use GZip compression for responses larger than 1400 bytes. You may want to disable
    # compression if you are exposing Kiali via a reverse proxy that is already
    # doing compression.
    gzip_enabled: true

    # For development purposes only. Controls if "Cross-Origin Resourse Sharing" is
    # enabled. 
    cors_allow_all: false

There is one additional spec.server.web_root option that affects the HTTP server, but that one is described in the Specifying route settings section of the Instalation guide.

Metrics server

Kiali emits metrics that can be collected by Prometheus. Most of these metrics are performance measurements.

The metrics server is enabled by default and listens on port 9090:

spec:
  server:
    metrics_enabled: true
    metrics_port: 9090

The bind address is the same as the HTTP server. Thus, make sure that the HTTP Server and the metrics server are not configured to the same port.

2 - Configuration

How to configure Kiali to fit your needs.

The pages in this Configuration section describe most available options for managing and customizing your Kiali installation.

Unless noted, it is assumed that you are using the Kiali operator and that you are managing the Kiali installation through a Kiali CR. The provided YAML snippets for configuring Kiali should be placed in your Kiali CR. For example, the provided configuration snippet for setting up the Anonymous authentication strategy is the following:

spec:
  auth:
    strategy: anonymous

You will need to take this YAML snippet and apply it to your Kiali CR. As an example, an almost minimal Kiali CR using the previous configuration snippet would be the following:

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  namespace: kiali-namespace
  name: kiali
spec:
  istio_namespace: istio-system
  deployment:
    namespace: kiali-namespace
  auth:
    strategy: anonymous

Then, you can save the finished YAML file and apply it with kubectl apply -f.

It is recommended that you read The Kiali CR and the Example Install pages of the Installation Guide for more information about using the Kiali CR.

Also, for reference, see Kiali CR Reference which documents all available options.

2.1 - Authentication Strategies

Choosing and configuring the appropriate authentication strategy.

Kiali supports five authentication mechanisms.

The default authentication strategy for OpenShift clusters is openshift.
The default authentication strategy for all other Kubernetes clusters is token.

All mechanisms other than anonymous support limiting per-user namespace access control.

For multi-cluster, only anonymous and openid are currently supported.

Read the dedicated page of each authentication strategy to learn more.

2.1.1 - Anonymous strategy

Access Kiali with no authentication.

Introduction

The anonymous strategy removes any authentication requirement. Users will have access to Kiali without providing any credentials.

Although the anonymous strategy doesn’t provide any access protection, it’s valid for some use-cases. Some examples known from the community:

Exposing Kiali through a reverse proxy, where the reverse proxy is providing a custom authentication mechanism.
Exposing Kiali on an already limited network of trusted users.
When Kiali is accessed through kubectl port-forward or alike commands that allow usage of the cluster’s RBAC capabilities to limit access.
When developing Kiali, where a developer has a private instance on his own machine.

It’s worth to emphasize that the anonymous strategy will leave Kiali unsecured. If you are using this option, make sure that Kiali is available only to trusted users, or access is protected by other means.

Set-up

To use the anonymous strategy, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: anonymous

The anonymous strategy doesn’t have any additional configuration.

Access control

When using the anonymous strategy, the content displayed in Kiali is based on the permissions of the Kiali service account. By default, the Kiali service account has cluster wide access and will be able to display everything in the cluster.

OpenShift

If you are running Kiali in OpenShift, access can be customized by changing privileges to the Kiali ServiceAccount. For example, to reduce permissions to individual namespaces, first, remove the cluster-wide permissions granted by default:

  oc delete clusterrolebindings kiali

Then grant the kiali role only in needed namespaces. For example:

  oc adm policy add-role-to-user kiali system:serviceaccount:istio-system:kiali-service-account -n ${NAMESPACE}

View only

You can tell the Kiali Operator to install Kiali in “view only” mode (this does work for either OpenShift or Kubernetes). You do this by setting the view_only_mode to true in the Kiali CR, which allows Kiali to read service mesh resources found in the cluster, but it does not allow any change:

spec:
  deployment:
    view_only_mode: true

2.1.2 - Header strategy

Run Kiali behind a reverse proxy responsible for injecting the user’s token, or a token with impersonation.

Introduction

The header strategy assumes a reverse proxy is in front of Kiali, such as OpenUnison or OAuth2 Proxy, injecting the user’s identity into each request to Kiali as an Authorization header. This token can be an OpenID Connect token or any other token the cluster recognizes.

In addition to a user token, the header strategy supports impersonation headers. If the impersonation headers are present in the request, then Kiali will act on behalf of the user specified by the impersonation (assuming the token supplied in the Authorization header is authorized to do so).

The header strategy allows for namespace access control.

The header strategy is only supported for single cluster.

Set-up

The header strategy will work with any Kubernetes cluster. The token provided must be supported by that cluster. For instance, most “on-prem” clusters support OpenID Connect, but cloud hosted clusters do not. For clusters that don’t support a token, the impersonation headers can be injected by the reverse proxy.

spec:
  auth:
    strategy: header

The header strategy doesn’t have any additional configuration.

HTTP Header

The header strategy looks for a token in the Authorization HTTP header with the Bearer prefix. The HTTP header should look like:

Authorization: Bearer TOKEN

Where TOKEN is the appropriate token for your cluster. This TOKEN will be submitted to the API server via a TokenReview to validate the token ONLY on the first access to Kiali. On subsequent calls the TOKEN is passed through directly to the API server.

Security Considerations

Network Policies

A policy should be put in place to make sure that the only “client” for Kiali is the authenticating reverse proxy. This helps limit potential abuse and ensures that the authenticating reverse proxy is the source of truth for who accessed Kiali.

Short Lived Tokens

The authenticating reverse proxy should inject a short lived token in the Authorization header. A shorter lived token is less likely to be abused if leaked. Kiali will take whatever token is passed into the reqeuest, so as tokens are regenerated Kiali will use the new token.

Impersonation

TokenRequest API

The authenticating reverse proxy should use the TokenRequest API instead of static ServiceAccount tokens when possible while using impersonation. The ServiceAccount that can impersonate users and groups is privileged and having it be short lived cuts down on the possibility of a token being leaked while it’s being passed between different parts of the infrastructure.

Drop Incoming Impersonation Headers

The authenticating proxy MUST drop any headers it receives from a remote client that match the impersonation headers. Not only do you want to make sure that the authenticating proxy can’t be overriden on which user to authenticate, but also what groups they’re a member of.

2.1.3 - OpenID Connect strategy

Access Kiali requiring authentication through a third-party OpenID Connect provider.

Introduction

The openid authentication strategy lets you integrate Kiali to an external identity provider that implements OpenID Connect, and allows users to login to Kiali using their existing accounts of a third-party system.

If your Kubernetes cluster is also integrated with your OpenId provider, then Kiali’s openid strategy can offer namespace access control.

Kiali only supports the authorization code flow of the OpenId Connect spec.

Requirements

The Kiali’s signing key needs to be 16, 24 or 32 byte long. If you install Kiali via the operator and don’t set a custom signing key, the operator should create a 16 byte long signing key.

If you don’t need namespace access control support, you can use any working OpenId Server where Kiali can be configured as a client application.

If you do need namespace access control support, you need either:

A Kubernetes cluster configured with OpenID connect integration, which results in the API server accepting tokens issued by your identity provider.
A replacement or reverse proxy for the Kubernetes cluster API capable of handling the OIDC authentication.

The first option is preferred if you can manipulate your cluster API server startup flags, which will result in your cluster to also be integrated with the external OpenID provider.

The second option is provided for cases where you are using a managed Kubernetes and your cloud provider does not support configuring OpenID integration. Kiali assumes an implementation of a Kubernetes API server. For example, a community user has reported to successfully configure Kiali’s OpenID strategy by using kube-oidc-proxy which is a reverse proxy that handles the OpenID authentication and forwards the authenticated requests to the Kubernetes API.

Set-up with namespace access control support

Assuming you already have a working Kubernetes cluster with OpenId integration (or a working alternative like kube-oidc-proxy), you should already had configured an application or a client in your OpenId server (some cloud providers configure this app/client automatically for you). You must re-use this existing application/client by adding the root path of your Kiali instance as an allowed/authorized callback URL. If the OpenID server provided you a client secret for the application/client, or if you had manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

where $NAMESPACE is the namespace where you installed Kiali and $CLIENT_SECRET is the secret you configured or provided by your OpenId Server. If Kiali is already running, you may need to restart the Kiali pod so that the secret is mounted in Kiali.

It’s worth emphasizing that to configure OpenID integration you must re-use the OpenID application/client that you created for your Kubernetes cluster. If you create a new application/client for Kiali in your OpenId server, Kiali will fail to properly authenticate users.

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

This assumes that your Kubernetes cluster is configured with OpenID Connect integration. In this case, the client-id and issuer_uri attributes must match the --oidc-client-id and --oidc-issuer-url flags used to start the cluster API server. If these values don’t match, users will fail to login to Kiali.

If you are using a replacement or a reverse proxy for the Kubernetes API server, the minimal configuration is like the following:

spec:
  auth:
    strategy: openid
    openid:
      api_proxy: "https://proxy.domain.com:port"
      api_proxy_ca_data: "..."
      client_id: "kiali-client"
      issuer_uri: "https://openid.issuer.com"

The value of client-id and issuer_uri must match the values of the configuration of your reverse proxy or cluster API replacement. The api_proxy attribute is the URI of the reverse proxy or cluster API replacement (only HTTPS is allowed). The api_proxy_ca_data is the public certificate authority file encoded in a base64 string, to trust the secure connection.

Set-up with no namespace access control support

Register Kiali as a client application in your OpenId Server. Use the root path of your Kiali instance as the callback URL. If the OpenId Server provides you a client secret, or if you manually set a client secret, issue the following command to create a Kubernetes secret holding the OpenId client secret:

kubectl create secret generic kiali --from-literal="oidc-secret=$CLIENT_SECRET" -n $NAMESPACE

Then, to enable the OpenID Connect strategy, the minimal configuration you need to set in the Kiali CR is like the following:

spec:
  auth:
    strategy: openid
    openid:
      client_id: "kiali-client"
      disable_rbac: true
      issuer_uri: "https://openid.issuer.com"

As namespace access control is disabled, all users logging into Kiali will share the same cluster-wide privileges.

Additional configurations

Configuring the displayed user name

The Kiali front-end will, by default, retrieve the string of the sub claim of the OpenID token and display it as the user name. You can customize which field to display as the user name by setting the username_claim attribute of the Kiali CR. For example:

spec:
  auth:
    openid:
      username_claim: "email"

If you enabled namespace access control, you will want the username_claim attribute to match the --oidc-username-claim flag used to start the Kubernetes API server, or the equivalent option if you are using a replacement or reverse proxy of the API server. Else, any user-friendly claim will be OK as it is purely informational.

Configuring requested scopes

By default, Kiali will request access to the openid, profile and email standard scopes. If you need a different set of scopes, you can set the scopes attribute in the Kiali CR. For example:

spec:
  auth:
    openid:
      scopes:
      - "openid"
      - "email"
      - "groups"

The openid scope is forced. If you don’t add it to the list of scopes to request, Kiali will still request it from the identity provider.

Configuring authentication timeout

When the user is redirected to the external authentication system, by default Kiali will wait at most 5 minutes for the user to authenticate. After that time has elapsed, Kiali will reject authentication. You can adjust this timeout by setting the authentication_timeout with the number of seconds that Kiali should wait at most. For example:

spec:
  auth:
    openid:
      authentication_timeout: 60 # Wait only one minute.

Configuring allowed domains

Some identity providers use a shared login and regardless of configuring your own application under your domain (or organization account), login can succeed even if the user that is logging in does not belong to your account or organization. Google is an example of this kind of provider.

To prevent foreign users from logging into your Kiali instance, you can configure a list of allowed domains:

spec:
  auth:
    openid:
      allowed_domains:
      - example.com
      - foo.com

The e-mail reported by the identity provider is used for the validation. Login will be allowed if the domain part of the e-mail is listed as an allowed domain; else, the user will be rejected. Naturally, you will need to configure the email scope to be requested.

There is a special case: some identity providers include a hd claim in the id_token. If this claim is present, this is used instead of extracting the domain from the user e-mail. For example, Google Workspace (aka G Suite) includes this hd claim for hosted domains.

Using an OpenID provider with a self-signed certificate

If your OpenID provider is using a self-signed certificate, you can disable certificate validation by setting the insecure_skip_verify_tls to true in the Kiali CR:

spec:
  auth:
    openid:
      insecure_skip_verify_tls: true

You should use self-signed certificates only for testing purposes.

However, if your organization or internal network has an internal trusted certificate authority (CA), and your OpenID server is using a certificate issued by this CA, you can configure Kiali to trust certificates from this CA, rather than disabling verification. For this, create a ConfigMap named kiali-cabundle containing the root CA certificate (the public component) under the openid-server-ca.crt key:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali-cabundle
  namespace: istio-system # This is Kiali's install namespace
data:
  openid-server-ca.crt: <the public component of your CA root certificate>

After restarting the Kiali pod, Kiali will trust this root certificate for all HTTPS requests related to OpenID authentication.

Using an HTTP/HTTPS Proxy

In some network configurations, there is the need to use proxies to connect to the outside world. OpenID requires outside world connections to get metadata and do key validation, so you can configure it by setting the http_proxy and https_proxy keys in the Kiali CR. They use the same format as the HTTP_PROXY and HTTPS_PROXY environment variables.

spec:
  auth:
    openid:
      http_proxy: http://USERNAME:PASSWORD@10.0.1.1:8080/
      https_proxy: https://USERNAME:PASSWORD@10.0.0.1:8080/

Passing additional options to the identity provider

When users click on the Login button on Kiali, a redirection occurs to the authentication page of the external identity provider. Kiali sends a fixed set of parameters to the identity provider to enable authentication. If you need to add an additional set of parameters to your identity provider, you can use the additional_request_params setting of the Kiali CR, which accepts key-value pairs. For example:

spec:
  auth:
    openid:
      additional_request_params:
        prompt: login

The prompt parameter is a standard OpenID parameter. When the login value is passed in this parameter, the identity provider is instructed to ask for user credentials regardless if the user already has an active session because of a previous login in some other system.

If your OpenId provider supports other non-standard parameters, you can specify the ones you need in this additional_request_params setting.

Take into account that you should not add the client_id, response_type, redirect_uri, scope, nonce nor state parameters to this list. These are already in use by Kiali and some already have a dedicated setting.

Provider-specific instructions

Using with Keycloak

When using OpenId with Keycloak, you will need to enable the Standard Flow Enabled option on the Client (in the Administration Console):

Client configuration screen on Keycloak

The Standard Flow described on the options is the same as the authorization code flow from the rest of the documentation.

Using with Google Cloud Platform / GKE OAuth2

If you are using Google Cloud Platform (GCP) and its products such as Google Kubernetes Engine (GKE), it should be straightforward to configure Kiali’s OpenID strategy to authenticate using your Google credentials.

First, you’ll need to go to your GCP Project and to the Credentials screen which is available at (Menu Icon) > APIs & Services > Credentials.

Credentials Screen on in GCP Project

On the Credentials screen you can select to create a new OAuth client ID.

Select OAuth on Credentials Screen

If you’ve never setup the OAuth consent screen you will need to do that before you can create an OAuth client ID. On screen you’ll have multiple warnings and prompts to walk you through this.

On the Create OAuth client ID screen, set the Application type to Web Application and enter a name for your key.

Select Web Application

Then enter in the Authorized Javascript origins and Authorized redirect URIs for your project. You can enter in localhost as appropriate during testing. You can also enter multiple URIs as appropriate.

Enter URLs

After clicking Create you’ll be shown your newly minted client id and secret. These are important and needed for your Kiali CR yaml and Kiali secrets files.

Get Credentials

You’ll need to update your Kiali CR file to include the following auth block.

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your client id from GCP>"
      disable_rbac: true
      issuer_uri: "https://accounts.google.com"
      scopes: ["openid", "email"]
      username_claim: "email"

Don’t get creative here. The issuer_uri should be https://accounts.google.com.

Finally you will need to create a secret, if you don’t have one already, that sets the oidc-secret for the openid flow.

apiVersion: v1
kind: Secret
metadata:
  name: kiali
  namespace: istio-system
  labels:
    app: kiali
type: Opaque
data:
  oidc-secret: "<base64 encode your client secret from GCP and enter here>"

Once all these settings are complete just set your Kiali CR and the Kiali secret to your cluster. You may need to refresh your Kiali Pod to set the Secret if you add the Secret after the Kiali pod is created.

Using with Azure: AKS and AAD

The OpenID authentication strategy can be used with Azure Kubernetes Service (AKS) and Azure Active Directory (AAD) with Kiali versions 1.33 and later. Prior Kiali versions do not support namespace access control on Azure.

AKS has support for a feature named AKS-managed Azure Active Directory, which enables integration between AKS and AAD. This has the advantage that users can use their AAD credentials to access AKS clusters and can also use Kubernetes RBAC features to assign privileges to AAD users.

However, Azure is implementing this integration via the Kubernetes Webhook Token Authentication rather than via the Kubernetes OpenID Connect Tokens authentication (see the Azure AD integration section in AKS Concepts documentation). Because of this difference, authentication in AKS behaves slightly different from a standard OpenID setup, but Kiali’s OpenID authentication strategy can still be used with namespace access control support by following the next steps.

First, enable the AAD integration on your AKS cluster. See the official AKS documentation to learn how. Once it is enabled, your AKS panel should show the following:

AKS-managed AAD is enabled,700

Create a web application for Kiali in your Azure AD panel:

Go to AAD > App Registration, create an application with a redirect url like https://<your-kiali-url>/kiali
Go to Certificates & secrets and create a client secret.
1. After creating the client secret, take note of the provided secret. Create a Kubernetes secret in your cluster as mentioned in the Set-up with namespace access control support section. Please, note that the suggested name for the Kubernetes Secret is kiali. If you want to customize the secret name, you will have to specify your custom name in the Kiali CR. See: secret_name in Kial CR Reference.
Go to API Permissions and press the Add a permission button. In the new page that appears, switch to the APIs my organization uses tab.
Type the following ID in the search field: 6dae42f8-4368-4678-94ff-3960e28e3630 (this is a shared ID for all Azure clusters). And select the resulting entry.
Select the Delegated permissions square.
Select the user.read permission.
Go to Authentication and make sure that the Access tokens checkbox is ticked.

Access tokens enabled

Then, create or modify your Kiali CR and include the following settings:

spec:
  auth:
    strategy: "openid"
    openid:
      client_id: "<your Kiali application client id from Azure>"
      issuer_uri: "https://sts.windows.net/<your AAD tenant id>/"
      username_claim: preferred_username
      api_token: access_token
      additional_request_params:
        resource: "6dae42f8-4368-4678-94ff-3960e28e3630"

You can find your client_id and tenant_id in the Overview page of the Kiali App registration that you just created. See this documentation for more information.

2.1.4 - OpenShift strategy

Access Kiali requiring OpenShift authentication.

Introduction

The openshift authentication strategy is the preferred and default strategy when Kiali is deployed on an OpenShift cluster.

When using the openshift strategy, a user logging into Kiali will be redirected to the login page of the OpenShift console. Once the user provides his OpenShift credentials, he will be redireted back to Kiali and will be logged in if the user has enough privileges.

The openshift strategy supports namespace access control.

The openshift strategy is only supported for single cluster.

Set-up

Since openshift is the default strategy when deploying Kiali in OpenShift, you shouldn’t need to configure anything. If you want to be verbose, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: openshift

The Kiali operator will make sure to setup the needed OpenShift OAuth resources to register Kiali as a client for the most common use-cases. The openshift strategy does have a few configuration settings that most people will never need but are available in case you have a situation where the customization is needed. See the Kiali CR Reference page for the documentation on those settings.

2.1.5 - Token strategy

Access Kiali requiring a Kubernetes ServiceAccount token.

Introduction

The token authentication strategy allows a user to login to Kiali using the token of a Kubernetes ServiceAccount. This is similar to the login view of Kubernetes Dashboard.

The token strategy supports namespace access control.

The token strategy is only supported for single cluster.

Set-up

Since token is the default strategy when deploying Kiali in Kubernetes, you shouldn’t need to configure anything, unless your cluster is OpenShift. If you want to be verbose or if you need to enable the token strategy in OpenShift, use the following configuration in the Kiali CR:

spec:
  auth:
    strategy: token

The token strategy doesn’t have any additional configuration other than the session expiration time.

2.1.6 - Session options

Session timeout and signing key configuration

There are two settings that are available for the user’s session. The first one is the session expiration time, which is only applicable to token and header authentication strategies:

spec:
  login_token:
    # By default, users session expires in 24 hours.
    expiration_seconds: 86400

The session expiration time is the amount of time before the user is asked to extend his session by another cycle. It does not matter if the user is actively using Kiali, the user will be asked if the session should be extended.

The second available option is the signing key configuration, which is unset by default, meaning that a random 16-character signing key will be generated and stored to a secret named kiali-signing-key, in Kiali’s installation namespace:

spec:
  login_token:
    # By default, create a random signing key and store it in
    # a secret named "kiali-signing-key".
    signing_key: ""

If the secret already exists (which may mean a previous Kiali installation was present), then the secret is reused.

The signing key is used on security sensitive data. For example, one of the usages is to sign HTTP cookies related to the user session to prevent session forgery.

If you need to set a custom fixed key, you can pre-create or modify the kiali-signing-key secret:

apiVersion: v1
kind: Secret
metadata:
  namespace: "kiali-installation-namespace"
  name: kiali-signing-key
type: Opaque
data:
  key: "<your signing key encoded in base64>"

The signing key must be 16, 24 or 32 bytes length. Otherwise, Kiali will fail to start.

If you prefer a different secret name for the signing key and/or a different key-value pair of the secret, you can specify your preferred names in the Kiali CR:

spec:
  login_token:
    signing_key: "secret:<secretName>:<secretDataKey>"

It is possible to specify the signing key directly in the Kiali CR, in the spec.login_token.signing_key attribute. However, this should be only for testing purposes. The signing key is sensitive and should be treated like a password that must be protected.

2.2 - Console Customization

Default selections, find and hide presets and custom metric aggregations.

Custom metric aggregations

The inbound and outbound metric pages, in the Metrics settings drop-down, provides an opinionated set of groupings that work both for filtering out metric data that does not match the selection and for aggregating data into series. Each option is backed by a label on the collected Istio telemetry.

It is possible to add custom aggregations, like in the following example:

spec:
  kiali_feature_flags:
    ui_defaults:
      metrics_inbound:
        aggregations:
        - display_name: Istio Network
          label: topology_istio_io_network
        - display_name: Istio Revision
          label: istio_io_rev
      metrics_outbound:
        aggregations:
        - display_name: Istio Revision
          label: istio_io_rev

Notice that custom aggregations for inbound and outbound metrics are defined separately.

You can find some screenshots in Kiali v1.40 feature update blog post.

Default metrics duration and refresh interval

Most Kiali pages show metrics per refresh and refresh interval drop-downs. These are located at the top-right of the page.

Metrics per refresh specifies the time range back from the current instant to fetch metrics and/or distributed tracing data. By default, a 1-minute time range is selected.

Refresh interval specifies how often Kiali will automatically refresh the data shown. By default, Kiali refreshes data every 15 seconds.

spec:
  kiali_feature_flags:
    ui_defaults:
      # Valid values: 1m, 5m, 10m, 30m, 1h, 3h, 6h, 12h, 1d, 7d, 30d
      metrics_per_refresh: "1m"

      # Valid values: pause, 10s, 15s, 30s, 1m, 5m, 15m
      refresh_interval: "15s"

User selections won’t persist a reload.

Default namespace selection

By default, when Kiali is accessed by the first time, on most Kiali pages users will need to use the namespace drop-down to choose namespaces they want to view data from. The selection will be persisted on reloads.

However, it is possible to configure a predefined selection of namespaces, like in the following example:

spec:
  kiali_feature_flags:
    ui_defaults:
      namespaces:
      - istio-system
      - bookinfo

Namespace selection will reset to the predefined set on reloads. Also, if for some reason a namespace becomes deleted, Kiali will simply ignore it from the list.

Graph find and hide presets

In the toolbar of the topology graph, the Find and Hide textboxes can be configured with presets for your most used criteria. You can find screenshots and a brief description of this feature in the feature update blog post for versions 1.31 to 1.33.

The following are the default presets:

spec:
  kiali_feature_flags:
    ui_defaults:
      graph:
        find_options:
        - auto_select: false  
          description: "Find: slow edges (> 1s)"
          expression: "rt > 1000"
        - auto_select: false
          description: "Find: unhealthy nodes"
          expression:  "! healthy"
        - auto_select: false
          description: "Find: unknown nodes"
          expression:  "name = unknown"
        hide_options:
        - auto_select: false
          description: "Hide: healthy nodes"
          expression: "healthy"
        - auto_select: false
          description: "Hide: unknown nodes"
          expression:  "name = unknown"

Hopefully, the attributes to configure this feature are self-explanatory.

To enable one of the configurations by default, it is possible to set auto_select to true, available for find and hide settings.

Note that by providing your own presets, you will be overriding the default configuration. Make sure to include any default presets that you need in case you provide your own configuration.

Graph default traffic rates

Traffic rates in the graph are fetched from Istio telemetry and there are several metric sources that can be used.

In the graph page, you can select the traffic rate metrics using the Traffic drop-down (next to the Namespaces drop-down). By default, Requests is selected for GRPC and HTTP protocols, and Sent bytes is selected for the TCP protocol, but you can change the default selection:

spec:
  kiali_feature_flags:
    ui_defaults:
      graph:
        traffic:
          grpc: "requests" # Valid values: none, requests, sent, received and total
          http: "requests" # Valid values: none and requests
          tcp:  "sent"     # Valid values: none, sent, received and total

Note that only requests provide response codes and will allow health to be calculated. Also, the resulting topology graph may be different for each source.

2.3 - Custom Dashboards

Configuring additional, non-default dashboards.

Custom Dashboards require some configuration to work properly.

Declaring a custom dashboard

When installing Kiali, you define your own custom dashboards in the Kiali CR spec.custom_dashboards field. Here’s an example of what it looks like:

custom_dashboards:
- name: vertx-custom
  title: Vert.x Metrics
  runtime: Vert.x
  discoverOn: "vertx_http_server_connections"
  items:
  - chart:
      name: "Server response time"
      unit: "seconds"
      spans: 6
      metrics:
      - metricName: "vertx_http_server_responseTime_seconds"
        displayName: "Server response time"
      dataType: "histogram"
      aggregations:
      - label: "path"
        displayName: "Path"
      - label: "method"
        displayName: "Method"
  - chart:
      name: "Server active connections"
      unit: ""
      spans: 6
      metricName: "vertx_http_server_connections"
      dataType: "raw"
  - include: "micrometer-1.1-jvm"
  externalLinks:
  - name: "My custom Grafana dashboard"
    type: "grafana"
    variables:
      app: var-app
      namespace: var-namespace
      version: var-version

The name field corresponds to what you can set in the pod annotation kiali.io/dashboards.

The rest of the field definitions are:

runtime: optional, name of the related runtime. It will be displayed on the corresponding Workload Details page. If omitted no name is displayed.
title: dashboard title, displayed as a tab in Application or Workloads Details
discoverOn: metric name to match for auto-discovery. If omitted, the dashboard won’t be discovered automatically, but can still be used via pods annotation.
items: a list of items, that can be either chart, to define a new chart, or include to reference another dashboard
- chart: new chart object
  - name: name of the chart
  - chartType: type of the chart, can be one of line (default), area, bar or scatter
  - unit: unit for Y-axis. Free-text field to provide any unit suffix. It can eventually be scaled on display. See specific section below.
  - unitScale: in case the unit needs to be scaled by some factor, set that factor here. For instance, if your data is in milliseconds, set 0.001 as scale and seconds as unit.
  - spans: number of “spans” taken by the chart, from 1 to 12, using bootstrap convention
  - metrics: a list of metrics to display on this single chart:
    - metricName: the metric name in Prometheus
    - displayName: name to display on chart
  - dataType: type of data to be displayed in the chart. Can be one of raw, rate or histogram. Raw data will be queried without transformation. Rate data will be queried using promQL rate() function. And histogram with histogram_quantile() function.
  - min and max: domain for Y-values. When unset, charts implementations should usually automatically adapt the domain with the displayed data.
  - xAxis: type of the X-axis, can be one of time (default) or series. When set to series, only one datapoint per series will be displayed, and the chart type then defaults to bar.
  - aggregator: defines how the time-series are aggregated when several are returned for a given metric and label set. For example, if a Deployment creates a ReplicaSet of several Pods, you will have at least one time-series per Pod. Since Kiali shows the dashboards at the workload (ReplicaSet) level or at the application level, they will have to be aggregated. This field can be used to fix the aggregator, with values such as sum or avg (full list available in Prometheus documentation). However, if omitted the aggregator will default to sum and can be changed from the dashboard UI.
  - aggregations: list of labels eligible for aggregations / groupings (they will be displayed in Kiali through a dropdown list)
    - label: Prometheus label name
    - displayName: name to display in Kiali
    - singleSelection: boolean flag to switch between single-selection and multi-selection modes on the values of this label. Defaults to false.
  - groupLabels: a list of Prometheus labels to be used for grouping. Similar to aggregations, except this grouping will be always turned on.
  - sortLabel: Prometheus label to be used for the metrics display order.
  - sortLabelParseAs: set to int if sortLabel needs to be parsed and compared as an integer instead of string.
- include: to include another dashboard, or a specific chart from another dashboard. Typically used to compose with generic dashboards such as the ones about MicroProfile Metrics or Micrometer-based JVM metrics. To reference a full dashboard, set the name of that dashboard. To reference a specific chart of another dashboard, set the name of the dashboard followed by $ and the name of the chart (ex: include: "microprofile-1.1$Thread count").
externalLinks: a list of related external links (e.g. to Grafana dashboards)
- name: name of the related dashboard in the external system (e.g. name of a Grafana dashboard)
- type: link type, currently only grafana is allowed
- variables: a set of variables that can be injected in the URL. For instance, with something like namespace: var-namespace and app: var-app, an URL to a Grafana dashboard that manages namespace and app variables would look like: http://grafana-server:3000/d/xyz/my-grafana-dashboard?var-namespace=some-namespace&var-app=some-app. The available variables in this context are namespace, app and version.

Label clash: you should try to avoid labels clashes within a dashboard.

In Kiali, labels for grouping are aggregated in the top toolbar, so if the same label refers to different things depending on the metric, you wouldn’t be able to distinguish them in the UI. For that reason, ideally, labels should not have too generic names in Prometheus. For instance labels named “id” for both memory spaces and buffer pools would better be named “space_id” and “pool_id”. If you have control on label names, it’s an important aspect to take into consideration. Else, it is up to you to organize dashboards with that in mind, eventually splitting them into smaller ones to resolve clashes.

Modifying Built-in Dashboards: If you want to modify or remove a built-in dashboard, you can set its new definition in the Kiali CR’s spec.custom_dashboards. Simply define a custom dashboard with the same name as the built-in dashboard. To remove a built-in dashboard so Kiali doesn’t use it, simply define a custom dashboard by defining only its name with no other data associated with it (e.g. in spec.custom_dashboards you add a list item that has - name: <name of built-in dashboard to remove>.

Dashboard scope

The custom dashboards defined in the Kiali CR are available for all workloads in all namespaces.

Additionally, new custom dashboards can be created for a given namespace or workload, using the dashboards.kiali.io/templates annotation.

This is an example where a “Custom Envoy” dashboard will be available for all applications and workloads for the default namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: default
  annotations:
    dashboards.kiali.io/templates: |
      - name: custom_envoy
        title: Custom Envoy
        discoverOn: "envoy_server_uptime"
        items:
          - chart:
              name: "Pods uptime"
              spans: 12
              metricName: "envoy_server_uptime"
              dataType: "raw"

This other example will create an additional “Active Listeners” dashboard only on details-v1 workload:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: details-v1
  labels:
    app: details
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: details
      version: v1
  template:
    metadata:
      labels:
        app: details
        version: v1
      annotations:
        dashboards.kiali.io/templates: |
          - name: envoy_listeners
            title: Active Listeners
            discoverOn: "envoy_listener_manager_total_listeners_active"
            items:
              - chart:
                  name: "Total Listeners"
                  spans: 12
                  metricName: "envoy_listener_manager_total_listeners_active"
                  dataType: "raw"          
    spec:
      serviceAccountName: bookinfo-details
      containers:
      - name: details
        image: docker.io/istio/examples-bookinfo-details-v1:1.16.2
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9080
        securityContext:
          runAsUser: 1000

Units

Some units are recognized in Kiali and scaled appropriately when displayed on charts:

unit: "seconds" can be scaled down to ms, µs, etc.
unit: "bytes-si" and unit: "bitrate-si" can be scaled up to kB, MB (etc.) using SI / metric system. The aliases unit: "bytes" and unit: "bitrate" can be used instead.
unit: "bytes-iec" and unit: "bitrate-iec" can be scaled up to KiB, MiB (etc.) using IEC standard / IEEE 1541-2002 (scale by powers of 2).

Other units will fall into the default case and be scaled using SI standard. For instance, unit: "m" for meter can be scaled up to km.

Prometheus Configuration

Kiali custom dashboards work exclusively with Prometheus, so it must be configured correctly to pull your application metrics.

If you are using the demo Istio installation with addons, your Prometheus instance should already be correctly configured and you can skip to the next section; with the exception of Istio 1.6.x where you need customize the ConfigMap, or install Istio with the flag --set meshConfig.enablePrometheusMerge=true.

Using another Prometheus instance

You can use a different instance of Prometheus for these metrics, as opposed to Istio metrics. This second Prometheus instance can be configured from the Kiali CR when using the Kiali operator, or ConfigMap otherwise:

# ...
external_services:
  custom_dashboards:
    prometheus:
      url: URL_TO_PROMETHEUS_SERVER_FOR_CUSTOM_DASHBOARDS
    namespace_label: kubernetes_namespace
  prometheus:
    url: URL_TO_PROMETHEUS_SERVER_FOR_ISTIO_METRICS
# ...

For more details on this configuration, such as Prometheus authentication options, check the Kiali CR Reference page.

You must make sure that this Prometheus instance is correctly configured to scrape your application pods and generates labels that Kiali will understand. Please refer to this documentation to setup the kubernetes_sd_config section. As a reference, here is how it is configured in Istio.

It is important to preserve label mapping, so that Kiali can filter by app and version, and to have the same namespace label as defined per Kiali config. Here’s a relabel_configs that allows this:

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace

Pod Annotations and Auto-discovery

Application pods must be annotated for the Prometheus scraper, for example, within a Deployment definition:

spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"

prometheus.io/scrape tells Prometheus to fetch these metrics or not
prometheus.io/port is the port under which metrics are exposed
prometheus.io/path is the endpoint path where metrics are exposed, default is /metrics

Kiali will try to discover automatically dashboards that are relevant for a given Application or Workload. To do so, it reads their metrics and try to match them with the discoverOn field defined on dashboards.

But if you can’t rely on automatic discovery, you can explicitly annotate the pods to associate them with Kiali dashboards.

spec:
  template:
    metadata:
      annotations:
        # (prometheus annotations...)
        kiali.io/dashboards: vertx-server

kiali.io/dashboards is a comma-separated list of dashboard names that Kiali will look for. Each name in the list must match the name of a built-in dashboard or the name of a custom dashboard as defined in the Kial CR’s spec.custom_dashboards.

2.4 - Debugging Kiali

How to debug Kiali using traces.

Kiali provides the ability to emit debugging traces to the distributed tracing platform, Jaeger or Grafana Tempo.

From Kiali 1.79, the feature of Kiali emitting tracing data into Jaeger format has been removed.

The traces can be sent in HTTP, HTTPS or gRPC protocol. It is also possible to use TLS. When tls_enabled is set to true, one of the options skip_verify or ca_name should be specified.

The traces are sent in OTel format, indicated in the collector_type setting.

server:
  observability:
    tracing:
      collector_type: "otel"
      collector_url: "jaeger-collector.istio-system:4317"
      enabled: false
      otel:
        protocol: "grpc"
        tls_enabled: true
        skip_verify: false
        ca_name: "/tls.crt"

Usually, the tracing platforms expose different ports to collect traces in distinct formats and protocols:

The Jaeger collector accepts OpenTelemetry Protocol over HTTP (4318) and gRPC (4317).
The Grafana Tempo distributor accepts OpenTelemetry Protocol over HTTP (4318) and gRPC (4317). It can be configured to accept TLS.

The traces emitted by Kiali can be searched in the Kiali workload:

Kiali traces

2.5 - Istio Environment

Kiali’s default configuration matches settings present in Istio’s installation configuration profiles. If you are customizing your Istio installation some Kiali settings may need to be adjusted. Also, some Istio management features can be enabled or disabled selectively.

Labels and resource names

Istio recommends adding app and version labels to pods to attach this information to telemetry. Kiali relies on correctness of these labels for several features.

In Istio, it is possible to use a different set of labels, like app.kubernetes.io/name and app.kubernetes.io/version, however you must configure Kiali to the labels you are using. By default, Kiali uses Istio’s recommended labels:

spec:
  istio_labels:
    app_label_name: "app"
    version_label_name: "version"

Although Istio lets you use different labels on different pods, Kiali can only use a single set.

For example, Istio lets you use the app label in one pod and the app.kubernetes.io/name in another pod and it will generate telemetry correctly. However, you will have no way to configure Kiali for this case.

Monitoring port of the IstioD pod

Kiali connects directly to the IstioD pod (not the Service) to check for its health. By default, the connection is done to port 15014 which is the default monitoring port of the IstioD pod.

Under some circumstances, you may need to change the monitoring port of the IstioD pod to something else. For example, when running IstioD in host network mode the network is shared between several pods, requiring to change listening ports of some pods to prevent conflicts.

It is possible to map the newly chosen monitoring port of the IstioD pod in the related Service to let other services continue working normally. However, since Kiali connects directly to the IstioD pod, you need to configure the assigned monitoring port in the Kiali CR:

spec:
  external_services:
    istio:
      istiod_pod_monitoring_port: 15014

Root namespace

Istio’s root namespace is the namespace where you can create some resources to define default Istio configurations and adapt Istio behavior to your environment. For more information on this Istio configuration, check the Istio docs Global Mesh options page and search for “rootNamespace”.

Kiali uses the root namespace for some of the validations of Istio resources. If you customized the Istio root namespace, you will need to replicate that configuration in Kiali. By default, it is unset:

spec:
  external_services:
    istio:
      root_namespace: ""

Sidecar injection, canary upgrade management and Istio revisions

Kiali can assist into configuring automatic sidecar injection, and also can assist when you are migrating workloads from an old Istio version to a newer one using the canary upgrade method. Kiali uses the standard Istio labels to control sidecar injection policy and canary upgrades.

Management of sidecar injection is enabled by default. If you don’t want this feature, you can disable it with the following configuration:

spec:
  kiali_feature_flags:
    istio_injection_action: false

Assistance for migrating workloads between Istio revisions when doing a canary upgrade is turned off by default. This is because it is required to know what is the revision name that was used when installing each Istio control plane. You can enable and configure the canary upgrade support with the following configuration:

spec:
  external_services:
    istio:
      istio_canary_revision:
        # Revision string of old Istio version
        current: "1-10-3"
        # Revision string of new Istio version
        upgrade: "1-11-0"
  kiali_feature_flags:
    # Turns on canary upgrade support
    istio_upgrade_action: true

Please note that Kiali will use revision labels to control sidecar injection policy only when canary upgrade support is enabled; else, non-revision labels are used. Make sure you finish the canary upgrade before turning off the canary upgrade support. If you need to disable Kiali’s canary upgrade feature while an upgrade is unfinished, it will be safer if you also disable the sidecar injection management feature.

It is important to note that canary upgrades require adding a revision name during the installation of control planes. You will notice that the revision name will be appended to the name of Istio resources. Thus, once/if you are using Kiali with an Istio control plane that has a revision name you will need to specify what is the name of a few Istio resources that Kiali uses. For example, if your control plane has a revision name 1-11-0 you would need to set these configurations:

spec:
  external_services:
    istio:
      config_map_name: "istio-1-11-0"
      istio_sidecar_injector_config_map_name: "istio-sidecar-injector-1-11-0"
      istiod_deployment_name: "istiod-1-11-0"

The progress of the canary upgrade process can be tracked on the mesh page, which displays the namespaces pending migration to the canary Istio control plane.

Canary upgrade process

There following are links to sections of Kiali blogs posts that briefly explains these features:

2.6 - Kiali CR Reference

Reference page for the Kiali CR. The Kiali Operator will watch for resources of this type and install Kiali according to those resources’ configurations.

Example CR

(all values shown here are the defaults unless otherwise noted)

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  annotations:
    ansible.sdk.operatorframework.io/verbosity: "1"
spec:
  additional_display_details:
  - title: "API Documentation"
    annotation: "kiali.io/api-spec"
    icon_annotation: "kiali.io/api-type"

  installation_tag: ""

  istio_namespace: ""

  version: "default"

  api:
    namespaces:
      exclude:
      - "^istio-operator"
      - "^kube-.*"
      - "^openshift.*"
      - "^ibm.*"
      - "^kiali-operator"
      include: []
      label_selector_exclude: ""
      # default: label_selector_include is undefined
      label_selector_include: "kiali.io/member-of=istio-system"
  auth:
    strategy: ""
    openid:
      # default: additional_request_params is empty
      additional_request_params:
        openIdReqParam: "openIdReqParamValue"
      # default: allowed_domains is an empty list
      allowed_domains: ["allowed.domain"]
      api_proxy: ""
      api_proxy_ca_data: ""
      api_token: "id_token"
      authentication_timeout: 300
      authorization_endpoint: ""
      client_id: ""
      disable_rbac: false
      http_proxy: ""
      https_proxy: ""
      insecure_skip_verify_tls: false
      issuer_uri: ""
      scopes: ["openid", "profile", "email"]
      username_claim: "sub"
    openshift:
      #token_inactivity_timeout:
      #token_max_age:

  clustering:
    autodetect_secrets:
      enabled: true
      label: "kiali.io/multiCluster=true"
    clusters: []
    kiali_urls: []

  # default: custom_dashboards is an empty list
  custom_dashboards:
  - name: "envoy"

  deployment:
    # default: accessible_namespaces is undefined
    accessible_namespaces: ["my-mesh.*"]
    # default: additional_service_yaml is empty
    additional_service_yaml:
      externalName: "kiali.example.com"
    affinity:
      # default: node is empty
      node:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: kubernetes.io/e2e-az-name
              operator: In
              values:
              - e2e-az1
              - e2e-az2
      # default: pod is empty
      pod:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S1
          topologyKey: topology.kubernetes.io/zone
      # default: pod_anti is empty
      pod_anti:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: security
                operator: In
                values:
                - S2
            topologyKey: topology.kubernetes.io/zone
    # default: cluster_wide_access is undefined
    cluster_wide_access: false
    # default: configmap_annotations is empty
    configmap_annotations:
      strategy.spinnaker.io/versioned: "false"
    # default: custom_secrets is an empty list
    custom_secrets:
    - name: "a-custom-secret"
      mount: "/a-custom-secret-path"
      optional: true
    - name: "a-csi-secret"
      mount: "/a-csi-secret-path"
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: kiali-secretprovider
    dns:
      # default: config is empty
      config:
        options:
        - name: ndots
          value: "1"
      # default: policy is empty
      policy: "ClusterFirst"
    hpa:
      api_version: ""
      # default: spec is empty
      spec:
        maxReplicas: 2
        minReplicas: 1
        metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 50
    # default: host_aliases is an empty list
    host_aliases:
    - ip: "192.168.1.100"
      hostnames:
      - "foo.local"
      - "bar.local"
    image_digest: ""
    image_name: ""
    image_pull_policy: "IfNotPresent"
    # default: image_pull_secrets is an empty list
    image_pull_secrets: ["image.pull.secret"]
    image_version: ""
    ingress:
      # default: additional_labels is empty
      additional_labels:
        ingressAdditionalLabel: "ingressAdditionalLabelValue"
      class_name: "nginx"
      # default: enabled is undefined
      enabled: false
      # default: override_yaml is undefined
      override_yaml:
        metadata:
          annotations:
            nginx.ingress.kubernetes.io/secure-backends: "true"
            nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        spec:
          rules:
          - http:
              paths:
              - path: "/kiali"
                pathType: Prefix
                backend:
                  service:
                    name: "kiali"
                    port:
                      number: 20001
    instance_name: "kiali"
    logger:
      log_level: "info"
      log_format: "text"
      sampler_rate: "1"
      time_field_format: "2006-01-02T15:04:05Z07:00"
    namespace: "istio-system"
    # default: node_selector is empty
    node_selector:
      nodeSelector: "nodeSelectorValue"
    # default: pod_annotations is empty
    pod_annotations:
      podAnnotation: "podAnnotationValue"
    # default: pod_labels is empty
    pod_labels:
      sidecar.istio.io/inject: "true"
    priority_class_name: ""
    replicas: 1
    # default: resources is undefined
    resources:
      requests:
        cpu: "10m"
        memory: "64Mi"
      limits:
        memory: "1Gi"
    secret_name: "kiali"
    security_context: {}
    # default: service_annotations is empty
    service_annotations:
      svcAnnotation: "svcAnnotationValue"
    # default: service_type is undefined
    service_type: "NodePort"
    # default: tolerations is an empty list
    tolerations:
    - key: "example-key"
      operator: "Exists"
      effect: "NoSchedule"
    version_label: ""
    view_only_mode: false

  external_services:
    custom_dashboards:
      discovery_auto_threshold: 10
      discovery_enabled: "auto"
      enabled: true
      is_core: false
      namespace_label: "namespace"
      prometheus:
        auth:
          ca_file: ""
          insecure_skip_verify: false
          password: ""
          token: ""
          type: "none"
          use_kiali_token: false
          username: ""
        cache_duration: 10
        cache_enabled: true
        cache_expiration: 300
        # default: custom_headers is empty
        custom_headers:
          customHeader1: "customHeader1Value"
        health_check_url: ""
        is_core: true
        # default: query_scope is empty
        query_scope:
          mesh_id: "mesh-1"
          cluster: "cluster-east"
        thanos_proxy:
          enabled: false
          retention_period: "7d"
          scrape_interval: "30s"
        url: ""
    grafana:
      auth:
        ca_file: ""
        insecure_skip_verify: false
        password: ""
        token: ""
        type: "none"
        use_kiali_token: false
        username: ""
      dashboards:
      - name: "Istio Service Dashboard"
        variables:
          namespace: "var-namespace"
          service: "var-service"
      - name: "Istio Workload Dashboard"
        variables:
          namespace: "var-namespace"
          workload: "var-workload"
      - name: "Istio Mesh Dashboard"
      - name: "Istio Control Plane Dashboard"
      - name: "Istio Performance Dashboard"
      - name: "Istio Wasm Extension Dashboard"
      enabled: true
      health_check_url: ""
      # default: in_cluster_url is undefined
      in_cluster_url: ""
      is_core: false
      url: ""
    istio:
      component_status:
        enabled: true
      config_map_name: "istio"
      egress_gateway_namespace: ""
      envoy_admin_local_port: 15000
      gateway_api_classes: []
      ingress_gateway_namespace: ""
      istio_api_enabled: true
      # default: istio_canary_revision is undefined
      istio_canary_revision:
        current: "1-9-9"
        upgrade: "1-10-2"
      istio_identity_domain: "svc.cluster.local"
      istio_injection_annotation: "sidecar.istio.io/inject"
      istio_sidecar_annotation: "sidecar.istio.io/status"
      istio_sidecar_injector_config_map_name: "istio-sidecar-injector"
      istiod_deployment_name: "istiod"
      istiod_pod_monitoring_port: 15014
      root_namespace: ""
      url_service_version: ""
    prometheus:
      auth:
        ca_file: ""
        insecure_skip_verify: false
        password: ""
        token: ""
        type: "none"
        use_kiali_token: false
        username: ""
      cache_duration: 10
      cache_enabled: true
      cache_expiration: 300
      # default: custom_headers is empty
      custom_headers:
        customHeader1: "customHeader1Value"
      health_check_url: ""
      is_core: true
      # default: query_scope is empty
      query_scope:
        mesh_id: "mesh-1"
        cluster: "cluster-east"
      thanos_proxy:
        enabled: false
        retention_period: "7d"
        scrape_interval: "30s"
      url: ""
    tracing:
      auth:
        ca_file: ""
        insecure_skip_verify: false
        password: ""
        token: ""
        type: "none"
        use_kiali_token: false
        username: ""
      # default: custom_headers is empty
      custom_headers:
        customHeader1: "customHeader1Value"
      enabled: false
      grpc_port: 9095
      health_check_url: ""
      in_cluster_url: ""
      is_core: false
      namespace_selector: true
      provider: "jaeger"
      # default: query_scope is empty
      query_scope:
        mesh_id: "mesh-1"
        cluster: "cluster-east"
      query_timeout: 5
      tempo_config:
        datasource_uid: ""
        org_id: ""
      url: ""
      use_grpc: true
      whitelist_istio_system: ["jaeger-query", "istio-ingressgateway"]

  health_config:
    # default: rate is an empty list
    rate:
    - namespace: ".*"
      kind: ".*"
      name: ".*"
      tolerance:
      - protocol: "http"
        direction: ".*"
        code: "[1234]00"
        degraded: 5
        failure: 10

  identity:
    # default: cert_file is undefined
    cert_file: ""
    # default: private_key_file is undefined
    private_key_file: ""

  istio_labels:
    app_label_name: "app"
    egress_gateway_label: "istio=egressgateway"
    ingress_gateway_label: "istio=ingressgateway"
    injection_label_name: "istio-injection"
    injection_label_rev: "istio.io/rev"
    version_label_name: "version"

  kiali_feature_flags:
    certificates_information_indicators:
      enabled: true
      secrets:
      - "cacerts"
      - "istio-ca-secret"
    disabled_features: []
    istio_annotation_action: true
    istio_injection_action: true
    istio_upgrade_action: false
    ui_defaults:
      graph:
        find_options:
        - description: "Find: slow edges (> 1s)"
          expression: "rt > 1000"
        - description: "Find: unhealthy nodes"
          expression: "! healthy"
        - description: "Find: unknown nodes"
          expression: "name = unknown"
        hide_options:
        - description: "Hide: healthy nodes"
          expression: "healthy"
        - description: "Hide: unknown nodes"
          expression: "name = unknown"
        traffic:
          grpc: "requests"
          http: "requests"
          tcp: "sent"
      i18n:
        language: "en"
        show_selector: false
      list:
        include_health: true
        include_istio_resources: true
        include_validations: true
        show_include_toggles: false
      metrics_per_refresh: "1m"
      # default: metrics_inbound is undefined
      metrics_inbound:
        aggregations:
        - display_name: "Istio Network"
          label: "topology_istio_io_network"
        - display_name: "Istio Revision"
          label: "istio_io_rev"
      # default: metrics_outbound is undefined
      metrics_outbound:
        aggregations:
        - display_name: "Istio Network"
          label: "topology_istio_io_network"
        - display_name: "Istio Revision"
          label: "istio_io_rev"
      # default: namespaces is an empty list
      namespaces: ["istio-system"]
      refresh_interval: "1m"
    validations:
      ignore: ["KIA1301"]
      skip_wildcard_gateway_hosts: false

  kubernetes_config:
    burst: 200
    cache_duration: 300
    cache_token_namespace_duration: 10
    excluded_workloads:
    - "CronJob"
    - "DeploymentConfig"
    - "Job"
    - "ReplicationController"
    qps: 175

  login_token:
    expiration_seconds: 86400
    signing_key: ""

  server:
    address: ""
    audit_log: true
    cors_allow_all: false
    gzip_enabled: true
    # default: node_port is undefined
    node_port: 32475
    observability:
      metrics:
        enabled: true
        port: 9090
      tracing:
        collector_type: "jaeger"
        collector_url: "http://jaeger-collector.istio-system:14268/api/traces"
        enabled: false
        otel:
          ca_name: ""
          protocol: "http"
          skip_verify: false
          tls_enabled: false
    port: 20001
    profiler:
      enabled: false
    web_fqdn: ""
    web_history_mode: ""
    web_port: ""
    web_root: ""
    web_schema: ""
    write_timeout: 30

Validating your Kiali CR

A Kiali tool is available to allow you to check your own Kiali CR to ensure it is valid. Simply download the validation script and run it, passing in the location of the Kiali CRD you wish to validate with (e.g. the latest version is found here) and the location of your Kiali CR. You must be connected to/logged into a cluster for this validation tool to work.

For example, to validate a Kiali CR named kiali in the namespace istio-system using the latest version of the Kiali CRD, run the following:

bash <(curl -sL https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/bin/validate-kiali-cr.sh) \
  -crd https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/crd/kiali.io_kialis.yaml \
  --kiali-cr-name kiali \
  -n istio-system

If you wish to test your Kiali CR with an older version of Kiali, replace master in the above -crd option with the version you wish to test. For example, to test your Kiali CR with Kiali version v1.53.0, pass in the option -crd https://raw.githubusercontent.com/kiali/kiali-operator/v1.53.0/crd-docs/crd/kiali.io_kialis.yaml in the above command.

For additional help in using this validation tool, pass it the --help option.

Properties

.spec

(object)

This is the CRD for the resources called Kiali CRs. The Kiali Operator will watch for resources of this type and when it detects a Kiali CR has been added, deleted, or modified, it will install, uninstall, and update the associated Kiali Server installation. The settings here will configure the Kiali Server as well as the Kiali Operator. All of these settings will be stored in the Kiali ConfigMap. Do not modify the ConfigMap; it will be managed by the Kiali Operator. Only modify the Kiali CR when you want to change a configuration setting.

.spec.additional_display_details

(array)

A list of additional details that Kiali will look for in annotations. When found on any workload or service, Kiali will display the additional details in the respective workload or service details page. This is typically used to inject some CI metadata or documentation links into Kiali views. For example, by default, Kiali will recognize these annotations on a service or workload (e.g. a Deployment, StatefulSet, etc.):

annotations:
  kiali.io/api-spec: http://list/to/my/api/doc
  kiali.io/api-type: rest

Note that if you change this setting for your own custom annotations, keep in mind that it would override the current default. So you would have to add the default setting as shown in the example CR if you want to preserve the default links.

.spec.additional_display_details[*]

(object)

.spec.additional_display_details[*].annotation

(string) *Required*

The name of the annotation whose value is a URL to additional documentation useful to the user.

.spec.additional_display_details[*].icon_annotation

(string)

The name of the annotation whose value is used to determine what icon to display. The annotation name itself can be anything, but note that the value of that annotation must be one of: rest, grpc, and graphql - any other value is ignored.

.spec.additional_display_details[*].title

(string) *Required*

The title of the link that Kiali will display. The link will go to the URL specified in the value of the configured annotation.

.spec.api

(object)

.spec.api.namespaces

(object)

Settings that control what namespaces are returned by Kiali.

.spec.api.namespaces.exclude

(array)

A list of namespaces to be excluded from the list of namespaces provided by the Kiali API and Kiali UI. Regex is supported. This does not affect explicit namespace access.

.spec.api.namespaces.exclude[*]

(string)

.spec.api.namespaces.include

(array)

A list of namespaces to be included in the list of namespaces provided by the Kiali API and Kiali UI (if those namespaces exist). Regex is supported. An undefined or empty list is ignored. This does not affect explicit namespace access.

.spec.api.namespaces.include[*]

(string)

.spec.api.namespaces.label_selector_exclude

(string)

A Kubernetes label selector (e.g. myLabel=myValue) which is used for filtering out namespaces when fetching the list of available namespaces. This does not affect explicit namespace access.

.spec.api.namespaces.label_selector_include

(string)

A Kubernetes label selector (e.g. myLabel=myValue) which is used when fetching the list of available namespaces. This does not affect explicit namespace access.

If deployment.accessible_namespaces does not have the special value of '**' then the Kiali operator will add a new label to all accessible namespaces - that new label will be this label_selector_include (this label is added regardless if the namespace matches the label_selector_exclude also).

Note that if you do not set this label_selector_include setting but deployment.accessible_namespaces does not have the special “all namespaces” entry of '**' then this label_selector_include will be set to a default value of kiali.io/[<deployment.instance_name>.]member-of=<deployment.namespace> where [<deployment.instance_name>.] is the instance name assigned to the Kiali installation if it is not the default ‘kiali’ (otherwise, this is omitted) and <deployment.namespace> is the namespace where Kiali is to be installed.

.spec.auth

(object)

.spec.auth.openid

(object)

(object)

To learn more about these settings and how to configure the OpenShift authentication strategy, read the documentation at https://kiali.io/docs/configuration/authentication/openshift/

.spec.auth.openshift.token_inactivity_timeout

(integer)

Timeout that overrides the default OpenShift token inactivity timeout. This value represents the maximum amount of time in seconds that can occur between consecutive uses of the token. Tokens become invalid if they are not used within this temporal window. If 0, the Kiali tokens never timeout. OpenShift may have a minimum allowed value - see the OpenShift documentation specific for the version of OpenShift you are using. WARNING: existing tokens will not be affected by changing this setting.

.spec.auth.openshift.token_max_age

(integer)

A time duration in seconds that overrides the default OpenShift access token max age. If 0 then there will be no expiration of tokens.

.spec.auth.strategy

(string)

Determines what authentication strategy to use when users log into Kiali. Options are anonymous, token, openshift, openid, or header.

Choose anonymous to allow full access to Kiali without requiring any credentials.
Choose token to allow access to Kiali using service account tokens, which controls access based on RBAC roles assigned to the service account.
Choose openshift to use the OpenShift OAuth login which controls access based on the individual’s RBAC roles in OpenShift. Not valid for non-OpenShift environments.
Choose openid to enable OpenID Connect-based authentication. Your cluster is required to be configured to accept the tokens issued by your IdP. There are additional required configurations for this strategy. See below for the additional OpenID configuration section.
Choose header when Kiali is running behind a reverse proxy that will inject an Authorization header and potentially impersonation headers.

.spec.clustering.clusters[*].name

(string)

The name of the cluster.

.spec.clustering.clusters[*].secret_name

(string)

The name of the secret that contains the credentials necessary to connect to the remote cluster. This secret must exist in the Kiali deployment namespace. If a secret name is not provided then it’s assumed that the cluster is inaccessible.

.spec.clustering.kiali_urls

(array)

A map between cluster name, instance name and namespace to a Kiali URL. Will be used showing the Mesh page’s Kiali URLs. The Kiali service’s ‘kiali.io/external-url’ annotation will be overridden when this property is set.

.spec.clustering.kiali_urls[*]

(object)

.spec.clustering.kiali_urls[*].cluster_name

(string)

The name of the cluster.

.spec.clustering.kiali_urls[*].instance_name

(string)

The instance name of this Kiali installation. This should be the value used in deployment.instance_name for Kiali resource name.

.spec.clustering.kiali_urls[*].namespace

(string)

The namespace into which Kiali is installed.

.spec.clustering.kiali_urls[*].url

(string)

The URL of Kiali in the cluster.

.spec.custom_dashboards

(array)

A list of user-defined custom monitoring dashboards that you can use to generate metrics charts for your applications. The server has some built-in dashboards; if you define a custom dashboard here with the same name as a built-in dashboard, your custom dashboard takes precedence and will overwrite the built-in dashboard. You can disable one or more of the built-in dashboards by simply defining an empty dashboard.

An example of an additional user-defined dashboard,

- name: myapp
  title: My App Metrics
  items:
  - chart:
      name: "Thread Count"
      spans: 4
      metricName: "thread-count"
      dataType: "raw"

An example of disabling a built-in dashboard (in this case, disabling the Envoy dashboard),

- name: envoy

To learn more about custom monitoring dashboards, see the documentation at https://kiali.io/docs/configuration/custom-dashboard/

.spec.custom_dashboards[*]

(object)

.spec.deployment

(object)

.spec.deployment.accessible_namespaces

(object)

.spec.deployment.cluster_wide_access

(boolean)

Determines if the Kiali server will be granted cluster-wide permissions to see all namespaces. When true, this provides more efficient caching within the Kiali server. It must be true if deployment.accessible_namespaces is left unset. To limit the namespaces for which Kiali has permissions, set to false and list the desired namespaces in deployment.accessible_namespaces. When not set, this value will default to false if deployment.accessible_namespaces is set to a list of namespaces; otherwise this will be true.

.spec.deployment.configmap_annotations

(object)

Custom annotations to be created on the Kiali ConfigMap.

.spec.deployment.custom_secrets

(array)

Defines additional secrets that are to be mounted in the Kiali pod.

These are useful to contain certs that are used by Kiali to securely connect to third party systems (for example, see external_services.tracing.auth.ca_file).

These secrets must be created by an external mechanism. Kiali will not generate these secrets; it is assumed these secrets are externally managed. You can define 0, 1, or more secrets. An example configuration is,

custom_secrets:
- name: mysecret
  mount: /mysecret-path
- name: my-other-secret
  mount: /my-other-secret-location
  optional: true

(array)

This is content for the Kubernetes ‘hostAliases’ setting for the Kiali server. This allows you to modify the Kiali server pod ‘/etc/hosts’ file. A typical way to configure this setting is,

host_aliases:
- ip: 192.168.1.100
  hostnames:
  - "foo.local"
  - "bar.local"

For details on the content of this setting, see https://kubernetes.io/docs/tasks/network/customize-hosts-file-for-pods/#adding-additional-entries-with-hostaliases

.spec.deployment.host_aliases[*]

(object)

.spec.deployment.host_aliases[*].hostnames

(array)

.spec.deployment.host_aliases[].hostnames[]

(string)

.spec.deployment.host_aliases[*].ip

(string)

.spec.deployment.hpa

(object)

Determines what (if any) HorizontalPodAutoscaler should be created to autoscale the Kiali pod. A typical way to configure HPA for Kiali is,

hpa:
  api_version: "autoscaling/v2"
  spec:
    maxReplicas: 2
    minReplicas: 1
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

.spec.deployment.image_pull_secrets[*]

(string)

.spec.deployment.image_version

(string)

Determines which version of Kiali to install. Choose ‘lastrelease’ to use the last Kiali release. Choose ‘latest’ to use the latest image (which may or may not be a released version of Kiali). Choose ‘operator_version’ to use the image whose version is the same as the operator version. Otherwise, you can set this to any valid Kiali version (such as ‘v1.0’) or any valid Kiali digest hash (if you set this to a digest hash, you must indicate the digest in deployment.image_digest).

Note that if this is set to ‘latest’ then the deployment.image_pull_policy will be set to ‘Always’.

Note that override_yaml.metadata.labels is not allowed - you cannot override the labels; to add labels to the default set of labels, use the deployment.ingress.additional_labels setting. Example,

override_yaml:
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/secure-backends: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  spec:
    rules:
    - http:
        paths:
        - path: /kiali
          pathType: Prefix
          backend:
            service
              name: "kiali"
              port:
                number: 20001

.spec.deployment.ingress.override_yaml.metadata

(object)

.spec.deployment.ingress.override_yaml.metadata.annotations

(object)

(object)

Custom labels to be created on the Kiali pod. An example use for this setting is to inject an Istio sidecar such as,

sidecar.istio.io/inject: "true"

.spec.deployment.priority_class_name

(string)

The priorityClassName used to assign the priority of the Kiali pod.

.spec.deployment.replicas

(integer)

The replica count for the Kiail deployment. If deployment.hpa is specified, this setting is ignored.

.spec.deployment.resources

(object)

Defines compute resources that are to be given to the Kiali pod’s container. The value is a dict as defined by Kubernetes. See the Kubernetes documentation (https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container). If you set this to an empty dict ({}) then no resources will be defined in the Deployment. If you do not set this at all, the default is,

requests:
  cpu: "10m"
  memory: "64Mi"
limits:
  memory: "1Gi"

(string)

Kiali resources will be assigned a ‘version’ label when they are deployed. This setting determines what value those ‘version’ labels will have. When empty, its default will be determined as follows,

If deployment.image_version is ‘latest’, version_label will be fixed to ‘master’.
If deployment.image_version is ‘lastrelease’, version_label will be fixed to the last Kiali release version string.
If deployment.image_version is anything else, version_label will be that value, too.

.spec.deployment.view_only_mode

(boolean)

When true, Kiali will be in ‘view only’ mode, allowing the user to view and retrieve management and monitoring data for the service mesh, but not allow the user to modify the service mesh.

.spec.external_services

.spec.external_services.grafana.auth.username

(string)

Username to be used when making requests to Grafana with basic authentication.

.spec.external_services.grafana.dashboards

(array)

A list of Grafana dashboards that Kiali can link to.

.spec.external_services.grafana.dashboards[*]

(object)

.spec.external_services.grafana.dashboards[*].name

(string)

The name of the Grafana dashboard.

(array)

A specific Istio component whose status will be monitored by Kiali.

(array)

A list declaring all the Gateways used in Istio. If left empty or undefined, the default is a single list item whose name is Istio and class_name is istio.

.spec.external_services.istio.gateway_api_classes[*]

.spec.external_services.prometheus.thanos_proxy.scrape_interval

(string)

Thanos Scrape interval value expresed as a string.

(object)

This section defines what it means for nodes to be healthy. For more details, see https://kiali.io/docs/configuration/health/

.spec.health_config.rate

(array)

.spec.health_config.rate[*]

(object)

.spec.health_config.rate[*].kind

(string)

The type of resource that this configuration applies to. This is a regular expression.

.spec.health_config.rate[*].name

(string)

The name of a resource that this configuration applies to. This is a regular expression.

.spec.health_config.rate[*].namespace

(string)

The name of the namespace that this configuration applies to. This is a regular expression.

.spec.health_config.rate[*].tolerance

(array)

A list of tolerances for this configuration.

(boolean)

.spec.kiali_feature_flags.certificates_information_indicators.secrets

(array)

.spec.kiali_feature_flags.certificates_information_indicators.secrets[*]

(string)

.spec.kiali_feature_flags.disabled_features

(array)

There may be some features that admins do not want to be accessible to users (even in ‘view only’ mode). In this case, this setting allows you to disable one or more of those features entirely.

(array)

A list of commonly used and useful find expressions that will be provided to the user out-of-box.

.spec.kiali_feature_flags.ui_defaults.graph.find_options[*]

(object)

.spec.kiali_feature_flags.ui_defaults.graph.find_options[*].auto_select

(boolean)

If true this option will be selected and take effect automatically. Note that only one option in the list can have this value be set to true.

.spec.kiali_feature_flags.ui_defaults.graph.find_options[*].description

(string)

Human-readable text to let the user know what the expression does.

.spec.kiali_feature_flags.ui_defaults.graph.find_options[*].expression

(string)

The find expression.

.spec.kiali_feature_flags.ui_defaults.graph.hide_options

(array)

A list of commonly used and useful hide expressions that will be provided to the user out-of-box.

(boolean)

If true list pages display checkbox toggles for the include options, Otherwise the configured settings are applied but can not be changed by the user. Default is false.

.spec.kiali_feature_flags.ui_defaults.metrics_inbound

(object)

Additional label aggregation for inbound metric pages in detail pages. You will see these configurations in the ‘Metric Settings’ drop-down. An example,

metrics_inbound:
  aggregations:
  - display_name: Istio Network
    label: topology_istio_io_network
  - display_name: Istio Revision
    label: istio_io_rev

.spec.kiali_feature_flags.ui_defaults.metrics_inbound.aggregations

(array)

.spec.kiali_feature_flags.ui_defaults.metrics_inbound.aggregations[*]

.spec.kiali_feature_flags.ui_defaults.namespaces

(array)

Default selections for the namespace selection dropdown. Non-existent or inaccessible namespaces will be ignored. Omit or set to an empty array for no default namespaces.

(string)

.spec.kubernetes_config.qps

(integer)

The QPS value of the Kubernetes client.

(string)

Defines the context root path for the Kiali console and API endpoints and readiness probes. When providing a context root path that is not /, do not add a trailing slash (i.e. use /kiali not /kiali/). When empty, this will default to / on OpenShift and /kiali on other Kubernetes environments.

.spec.server.web_schema

(string)

Defines the public HTTP schema used to serve Kiali. Value must be one of: http or https. When empty, Kiali will try to guess this value from HTTP headers. On non-OpenShift clusters, you must populate this value if you want to enable cross-linking between Kiali instances in a multi-cluster setup.

.spec.server.write_timeout

(integer)

The maximum duration, in seconds, before timing out writes of the HTTP response back to the client. Default is 30.

In OpenShift clusters, the route request time out should be also increased as the default is 30 seconds. This can be done by annotating the specific route with haproxy.router.openshift.io/timeout. See https://docs.openshift.com/container-platform/4.16/networking/routes/route-configuration.html#nw-configuring-route-timeouts_route-configuration for further details.

.spec.version

(string)

The version of the Ansible role that will be executed in order to install Kiali. This also indirectly determines the version of Kiali that will be installed. You normally will want to use default since this is the only officially supported value today.

If not specified, the value of default is assumed which means the most recent Ansible role is used; thus the most recent release of Kiali will be installed.

Refer to this file to see what the valid values are for this version field (as defined in the master branch), https://github.com/kiali/kiali-operator/blob/master/playbooks/kiali-default-supported-images.yml

This version setting affects the defaults of the deployment.image_name and deployment.image_version settings. See the documentation for those settings below for additional details. In short, this version setting will dictate which version of the Kiali image will be deployed by default. However, if you explicitly set deployment.image_name and/or deployment.image_version to reference your own custom image, that will override the default Kiali image to be installed; therefore, you are responsible for ensuring those settings are compatible with the Ansible role that will be executed in order to install Kiali (i.e. your custom Kiali image must be compatible with the rest of the configuration and resources the operator will install).

.status

(object)

The processing status of this CR as reported by the Kiali operator.

2.7 - Multi-cluster

Configuring Kiali for a multi-cluster mesh.

Kiali has experimental support for Istio multi-cluster installations.

Kiali multi-cluster

Before proceeding with the setup, ensure you meet the requirements.

Requirements

Aggregated metrics and traces. Kiali needs a single endpoint for metrics and a single endpoint for traces where it can consume aggregated metrics/traces across all clusters. There are many ways to aggregate metrics/traces such as Prometheus federation or using OTEL collector pipelines but setting these up are outside of the scope of Kiali.
Anonymous or OpenID authentication strategy. The unified multi-cluster configuration currently only supports anonymous or OpenID authentication strategies. In addition, current support varies by provider for OpenID across clusters.

Setup

The unified Kiali multi-cluster setup requires the Kiali Service Account (SA) to have read access to each Kubernetes cluster in the mesh. This is separate from the user credentials that are required when a user logs into Kiali. The user credentials are used to check user access to a namespace and to perform write operations. In anonymous mode, the Kiali SA is used for all operations and write access is also required. To give the Kiali SA access to each remote cluster, a kubeconfig with credentials needs to be created and mounted into the Kiali pod. While the location of Kiali in relation to the controlplane and dataplane may change depending on your istio deployment model, the requirements will remain the same.

If you would like to keep a separate Kiali per cluster and do not want to give Kiali access to remote clusters, you can still manually specify the remote cluster and remote Kiali URLs in the Kiali configuration and the UI will try to provide links to the external Kiali where appropriate. See below for more details.

Create a remote cluster secret. In order to access a remote cluster, you must provide a kubeconfig to Kiali via a Kubernetes secret. You can use this script to simplify this process for you. Running this script will:
- Create a Kiali Service Account in the remote cluster.
- Create a role/role-binding for this service account in the remote cluster.
- Create a kubeconfig file and save this as a secret in the namespace where Kiali is deployed.
In order to run this script you will need adequate permissions configured in your local kubeconfig, for both the cluster on which Kiali is deployed and the remote cluster. You will need to repeat this step for each remote cluster.
```
curl -L -o kiali-prepare-remote-cluster.sh https://raw.githubusercontent.com/kiali/kiali/master/hack/istio/multicluster/kiali-prepare-remote-cluster.sh
chmod +x kiali-prepare-remote-cluster.sh
./kiali-prepare-remote-cluster.sh --kiali-cluster-context east --remote-cluster-context west --view-only false
```

Use the option --help for additional details on using the script to create and delete remote cluster secrets.

Configure Kiali. The Kiali CR provides configuration settings that enable the Kiali Server to use remote cluster secrets in order to access remote clusters. By default, the Kiali Operator will auto-detect any remote cluster secret that has a label kiali.io/multiCluster=true and is found in the Kiali deployment namespace. The secrets created by the kiali-prepare-remote-cluster.sh script will be created that way and thus can be auto-detected. Alternatively, in the Kiali CR you can explicitly specify each remote cluster secret rather than rely on auto-discovery. Given the remote cluster secrets it knows about (either through auto-discovery or through explicit configuration) the Operator will mount the remote cluster secrets into the Kiali Server pod effectively putting Kiali in “multi-cluster” mode. Kiali will begin using those credentials to communicate with the other clusters in the mesh.
Optional - Configure tracing with cluster ID. By default, traces do not include their cluster name in the trace tags however this can be added using the istio telemetry API.

kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  # no selector specified, applies to all workloads
  tracing:
  -  customTags:
      cluster:
        environment:
          name: "ISTIO_META_CLUSTER_ID"
EOF

As an alternative, it can be added in the Istio tracing config in the primary cluster(s) (Without using the Telemetry API):

meshConfig:
  defaultConfig:
    tracing:
      custom_tags:
        cluster:
          environment:
            name: ISTIO_META_CLUSTER_ID

Optional - Configure user access in your OIDC provider. When using anonymous mode, the Kiali SA credentials will be used to display mesh info to the user. When not using anonymous mode, Kiali will check the user’s access to each configured cluster’s namespace before showing the user any resources from that namespace. Please refer to your OIDC provider’s instructions for configuring user access to a kube cluster for this.
Optional - Narrow metrics to mesh. If your unified metrics store also contains data outside of your mesh, you can limit which metrics Kiali will query for by setting the query_scope configuration.

That’s it! From here you can login to Kiali and manage your mesh across both clusters from a single Kiali instance.

Removing a Cluster

To remove a cluster from Kiali, you must delete the associated remote cluster secret. If you originally created the remote cluster secret via the kiali-prepare-remote-cluster.sh script, run that script again with the same command line options as before but also pass in the command line option --delete true.

After the remote cluster secret has been removed, you must then tell the Kiali Operator to re-deploy the Kiali Server so the Kiali Server no longer attempts to access the now-deleted remote cluster secret. If you are using auto-discovery, you can tell the Kiali Operator to do this by touching the Kiali CR. The easiest way to do this is to simply add or modify any annotation on the Kiali CR. It is recommended that you use the kiali.io/reconcile annotation as described here. If you did not rely on auto-discovery but instead explicitly specified each remote cluster secret in the Kiali CR, then you simply have to remove the now-deleted remote cluster secret’s information from the Kiali CR’s kiali_feature_flags.clustering.clusters section.

Adding an Inaccessible Cluster

In situations where Kiali does not have access to remote clusters, you can manually specify the remote cluster info along with any external Kialis running on the remote clusters and Kiali will try to provide links to these in the UI. For example, if there is a Kiali on the east cluster that does not have access to the west cluster and a Kiali on the west cluster that does not have access to the east cluster, you can add the following to your Kiali configurations to have each Kiali generate links to the external Kiali for that cluster.

East Kiali configuration

clustering:
  clusters:
    name: west
  kiali_urls:
    cluster_name: west
    instance_name: kiali
    namespace: istio-system
    url: https://kiali-external.west.example.com

West Kiali configuration

clustering:
  clusters:
    name: east
  kiali_urls:
    cluster_name: east
    instance_name: kiali
    namespace: istio-system
    url: https://kiali-external.east.example.com

2.8 - Namespace access control

Configuring per-user authorized namespaces.

Introduction

In authentication strategies other than anonymous Kiali supports limiting the namespaces that are accessible on a per-user basis. The anonymous authentication strategy does not support this, although you can still limit privileges when using an OpenShift cluster. See the access control section in Anonymous strategy.

To authorize namespaces, the standard Roles resources (or ClusterRoles) and RoleBindings resources (or ClusterRoleBindings) are used.

The Kubernetes RBAC documentation describe how to use Roles, ClusterRoles, RoleBindings and ClusterRoleBindings resources. If you are using OpenShift, read the OpenShift RBAC documentation.

Kiali can only restrict or grant read access to namespaces as a whole. So, keep in mind that while the RBAC capabilities of the cluster are used to give access, Kiali won’t offer the same privilege granularity that the cluster supports. For example, a user that does not have privileges to get Kubernetes Deployments via typical tools (e.g. kubectl) would still be able to get some details of Deployments through Kiali when listing Workloads or when viewing detail pages, or in the Graph.

Some features allow creating or changing resources in the cluster (for example, the Wizards). For these write operations which may be sensitive, the users will need to have the required privileges in the cluster to perform updates - i.e. the cluster RBAC takes effect.

Kiali is going to reject login to users that aren’t authorized to see any namespace.

Granting access to namespaces

In general, Kiali will give read access to namespaces where the logged in user is allowed to “GET” its definition – i.e. the user is allowed to do a GET call to the api/v1/namespaces/{namespace-name} endpoint of the cluster API. Users granted the LIST verb would get access to all namespaces of the cluster (that’s a GET call to the api/v1/namespaces endpoint of the cluster API).

You, probably, will want to have this small ClusterRole to help you in authorizing individual namespaces in Kiali:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiali-namespace-authorization
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
- apiGroups: ["project.openshift.io"] # Only if you are using OpenShift
  resources:
  - projects
  verbs:
  - get

The pods/log privilege is needed for the pods Logs view. Since logs are potentially sensitive, you could remove that privilege if you don’t want users to be able to fetch pod logs.

Once you have created this ClusterRole, you would authorize a namespace foobar to user john with the following RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: authorize-ns-foobar-to-john
  namespace: foobar
subjects:
- kind: User
  name: john
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kiali-namespace-authorization # The name of the ClusterRole created previously
  apiGroup: rbac.authorization.k8s.io

Note that in this example, the subject kind is User, which is the case when using openid or openshift authentication strategies. For other authentication strategies you would need to adjust the RoleBinding to use the right subject kind.

If you want to authorize a user to access all namespaces in the cluster, the most efficient way to do it is by creating a ClusterRole with the list verb for namespaces and bind it to the user using a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiali-all-namespaces-authorization
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods/log
  verbs:
  - get
  - list
- apiGroups: ["project.openshift.io"] # Only if you are using OpenShift
  resources:
  - projects
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: authorize-all-namespaces-to-john
subjects:
- kind: User
  name: john
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: kiali-all-namespaces-authorization
  apiGroup: rbac.authorization.k8s.io

Note that the only addition to the ClusterRole is the list verb in the first rule.

Alternatively, you could also use the previously mentioned kiali-namespace-authorization rather than creating a new one with the list privilege, and it will work. However, Kiali will perform better if you grant the list privilege.

Please read your cluster RBAC documentation to learn more about the authorization system.

Granting write privileges to namespaces

Changing resources in the cluster can be a sensitive operation. Because of this, the logged in user will need to be given the needed privileges to perform any updates through Kiali. The following ClusterRole contains all read and write privileges that may be used in Kiali:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiali-write-privileges
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  - replicationcontrollers
  - services
  verbs:
  - patch
- apiGroups: ["extensions", "apps"]
  resources:
  - daemonsets
  - deployments
  - replicasets
  - statefulsets
  verbs:
  - patch
- apiGroups: ["batch"]
  resources:
  - cronjobs
  - jobs
  verbs:
  - patch
- apiGroups:
  - networking.istio.io
  - security.istio.io
  - extensions.istio.io
  - telemetry.istio.io
  - gateway.networking.k8s.io
  resources: ["*"]
  verbs:
  - get
  - list
  - watch
  - create
  - delete
  - patch

If needed, you can reduce the set of write privileges to prevent users from changing unwanted resources. However read privileges are require to read the resources.

Similarly to giving access to namespaces, you can either use a RoleBinding to give read and write privileges only to specific namespaces, or use a ClusterRoleBinding to give privileges to all namespaces.

2.9 - Namespace Management

Configuring the namespaces accessible and visible to Kiali.

Introduction

The default Kiali installation (as mentioned in the Installation guide) gives Kiali access to all namespaces available in the cluster.

It is possible to restrict Kiali to a set of desired namespaces by providing a list of the ones you want, excluding the ones you don’t want. You can also filter namespaces by using Istio’s discovery selectors. You can use a combination of these options together.

As of Kiali 1.67, the following settings are deprecated:

api.namespaces.exclude
api.namespaces.include
api.namespaces.label_selector_exclude
api.namespaces.label_selector_include

It is recommended to instead use Istio Discovery Selectors to limit the namespaces in the mesh.

Accessible Namespaces

The “accessible namespaces” setting configures which namespaces are accessible to the Kiali server itself. The accessible namespaces are defined by a list of regex expressions that match against the namespace names visible to the Kiali operator. If left unset, the Kiali server will be given access to all namespaces in the cluster.

The list of accessible namespaces is specified in the Kiali CR via the accessible_namespaces setting, under the main deployment section. As an example, if Kiali is to be installed in the istio-system namespace, and is expected to monitor all namespaces prefixed with mycorp_, the setting would be:

spec:
  deployment:
    accessible_namespaces:
    - istio-system
    - mycorp_.*

When you set accessible_namespaces to a list of namespaces as the example above illustrates, by default the operator will create a Role for each accessible namespace. Each Role will be assigned to the Kiali Service Account thus providing Kiali permissions to those namespaces. To change that default behavior, deployment.cluster_wide_access can be set to true. This will tell the operator not to create these Roles and just create a single ClusterRole and assign it to the Kiali Server (see below).

Because Kiali server utilizes Kubernetes watches to watch all namespaces in deployment.accessible_namespaces, this may cause performance issues. To increase performance you can set deployment.cluster_wide_access to true when specifying a list of namespaces in accessible_namespaces. When you do this, the Kiali Server will be given access to the entire cluster and thus it can use a single cluster watch which increases performance and efficiency. However, you must be aware that when you do this, the Kiali Server will be granted access to the cluster via a ClusterRole - individual Roles will not be created per namespace. The deployment.accessible_namespaces will still be used to determine which namespaces to make available to users.

If you install Kiali using the Server Helm Chart, these Roles will not be created. This security feature is provided by the operator only, and is one reason why it is recommended to use the operator. The Server Helm Chart is provided only as a convenience.

Note that the namespaces declared here (including any regex expressions) are evaluated and discovered by the operator at install time. Namespaces that do not exist at the time of install will not be accessible to Kiali until the operator has a chance to reconcile the Kiali CR. If you enable namespace watching in the operator, this should happen fairly quickly after the new namespace is created. Namespace watching is an advanced feature; see the FAQ describing the operator WATCHES_FILE environment variable. Adding the new namespace to the list of accessible_namespaces will also trigger the operator to reconcile the Kiali CR. When the operator reconciles the Kiali CR, the necessary Role will be created giving the Kiali Server access to the new namespace.

As you can see in the example, the namespace where Kiali is installed must be listed as accessible (often, but not always, this is the same namespace as the Istio control plane namespace). If it is not in the list, it will be added for you by the operator.

As mentioned earlier, if you leave accessible_namespaces unset, this denotes that the Kiali server is given access to all namespaces in the cluster, including any namespaces that are created in the future. If you do this, individual Roles are not created per namespace; instead a single ClusterRole is created and assigned to the Kiali Service Account thus providing Kiali access to all namespaces in the cluster.

spec:
  deployment: {}

If you install the operator using the Helm Charts, to be able to use this default accessible_namespaces behavior, you must specify the --set clusterRoleCreator=true flag when invoking helm install.

When installing multiple Kiali instances into a single cluster, accessible_namespaces must be mutually exclusive. In other words, a namespace set must be matched by only one Kiali CR. Regular expressions must not have overlapping patterns.

A cluster can have at most one Kiali instance with accessible_namespaces unset.

Istio Discovery Selectors

In Istio’s MeshConfig, you can provide a list of discovery selectors that Istio will consider when processing namespaces. The Kiali server will also utilize these same discovery selectors when determining which namespaces to make available to users. Therefore, these discovery selectors work in conjunction with the accessible_namespaces setting. The accessible_namespaces setting dictates which namespaces are accessible to the Kiali Server, and hence to users of Kiali. The Istio discovery selectors tell Kiali to make available to users those namespaces within the accessible_namespaces list. So discovery selectors are a subset of accessible_namespaces.

You must make sure you configure accessible_namespaces with a list of namespaces that are equal to, or are a superset of, the namespaces that result from evaluating the Istio discovery selectors.

Included Namespaces

This feature is deprecated as of 1.67. It is recommended to instead use Istio Discovery Selectors.

When accessible_namespaces is unset, you can still limit the namespaces a user will see in Kiali. The api.namespaces.include setting allows you to configure the subset of namespaces that Kiali will show the user. This list is specified in a way similar to accessible_namespaces, as a list of namespaces that can include regex patterns. The difference is that this list is processed by the server, not the operator, each time the list of namespaces needs to be obtained by Kiali. This means it will handle namespaces that exist when Kiali is installed, or namespaces created later.

spec:
  api:
    namespaces:
      include:
      - istio-system
      - mycorp_.*
  deployment: {}

The api.namespaces.include setting is ignored if accessible_namespaces is set.

The api.namespaces.include will implicitly include the control plane namespace (e.g. istio-system) even if you do not explicitly specify it in the list.

Note that this setting is merely a filter and does not provide security in any way.

Excluded Namespaces

This feature is deprecated as of 1.67. It is recommended to instead use Istio Discovery Selectors.

The api.namespaces.exclude setting configures namespaces to exclude from being shown to the user. This setting can be used both when accessible_namespaces is unset or set to an explicit list of namespaces. It can also be used regardless of whether api.namespaces.include is defined. The exclude filter has precedence - if a namespace matches any regex pattern defined for api.namespaces.exclude, it will not be shown in Kiali. Like api.namespaces.include, the exclude setting is only a filter and does not provide security in any way.

For example, if the accessible_namespaces configuration includes mycorp_.* but it is not desirable to see test namespaces, the following configuration can be used:

spec:
  api:
    namespaces:
      exclude:
      - mycorp_test.*
  deployment:
    accessible_namespaces:
    - istio-system
    - mycorp_.*

You cannot exclude the control plane namespace. If you specify it in api.namespaces.exclude, it will be ignored.

Namespace Selectors

This feature is deprecated as of 1.67. It is recommended to instead use Istio Discovery Selectors.

In addition to the Include and Exclude lists (as explained above), Kiali supports optional Kubernetes label selectors for both including and excluding namespaces.

The include label selector is defined in the Kiali CR setting api.namespaces.label_selector_include.

The exclude label selector is defined in the Kiali CR setting api.namespaces.label_selector_exclude.

The example below selects all namespaces that have a label kiali-enabled: true:

spec:
  api:
    namespaces:
      label_selector_include: kiali-enabled=true
  deployment: {}

The example below hides all namespaces that have a label infrastructure: system:

spec:
  api:
    namespaces:
      label_selector_exclude: infrastructure=system

Important Note About `label_selector_include`

It is very important to understand how the api.namespaces.label_selector_include setting is used when deployment.accessible_namespaces is set to an explicit list of namespaces versus when it is unset.

When accessible_namespaces is unset then label_selector_include simply provides a filter just like api.namespaces.include. The difference is that label_selector_include filters by namespace label whereas include filters by namespace name. Thus, label_selector_include provides an additional way to limit which namespaces the Kiali user will see when accessible_namespaces is unset.

It is recommended to leave label_selector_include unset when accessible_namespaces is set to an explicit list of namespaces. This is because the operator adds the configured label (name and value) to each namespace defined by accessible_namespaces. This allows the Kiali code to use the label_selector_include value to easily select all of the accessible_namespaces. Unless you have a good reason to customize, the default value is highly recommended in this scenario.

If Kiali is installed via the Server Helm Chart, labels are not added to the accessible namespaces. The user must ensure api.namespaces.label_selector_include (if defined) selects all namespaces declared in accessible_namespaces (when unset). It is a user-error if this is not configured correctly.

For further information on how this api.namespaces.label_selector_include interacts with deployment.accessible_namespaces read the Kiali CR Reference documentation.

Soft Multi-Tenancy

When deploying multiple control planes in the same cluster, the label_selector_include’s value must be unique to each control plane. This allows each Kiali instance to select only the namespaces relevant to each control plane. Thus you can have multiple control planes in the same cluster, with each control plane having its own Kiali instance. In this “soft-multitenancy” mode, deployment.accessible_namespaces is typically set to an explicit set of namespaces. In that case, you do not have to do anything with this label_selector_include because the default value of label_selector_include is kiali.io/member-of: <spec.istio_namespace>. If you set your own Kiali instance name in the Kiali CR (i.e. you set deployment.instance_name to something other than kiali), then the default will be kiali.io/<spec.deployment.instance_name>.member-of: <spec.istio_namespace>.

2.10 - No Istiod Access

Kiali behavior with no access to Istiod (the /debug endpoints are not available)

Introduction

Kiali makes use of the Istiod /debug endpoints for introspection into the control plane. If this API is unavailable Kiali continues to perform, but the feature set will be degraded. The Istio API can be unavailable for various reasons:

The Istio API has been explicitly disabled in the Istio configuration.
The deployment model prevents access to the Istio API (firewalls, other networking concerns or limitations).
The API is configured but for some, potentially unexpected, reason can not be reached by Kiali.

Configuration

When the Istio API is known to be inaccessible Kiali should be configured via the istio_api_enabled configuration item.
By default, istio_api_enabled is true.

# ...
spec:
external_services:
  istio:
    istio_api_enabled: false
# ...

How does it affect Kiali

When the Istio API is not available there is expected feature degradation in Kiali:

The control plane metrics won’t be available.
The proxy status won’t be available in the workloads details view.
The Istio validations may not be available.
The Kiali validations will not be available.
The Istio Registry Services that are not present in the Kubernetes list won’t be available.

Note that Istio Configurations will be available. This is because the list of Istio configurations is obtained using the Kubernetes API.

Istio Validations

The Istio validations won’t be available as this logic is provided by the Istio API. But, if the Istio Config was created when the validatingwebhookconfiguration web hook was enabled, the validation messages will be available and the Istio validations can be found:

The Kiali validations won’t be available, as they are degraded, so they have been disabled too.

Istio Registry Services

The Istio Registry Services won’t be available in the service list when the Istio API is disabled.

The following image shows a service list when Istio API is enabled:

The following image shows the same list when it is disabled:

Istio Configurations

The Istio Configurations are available in view and edit mode. It is important to know that the validations are disabled, so the configurations created or modified won’t be validated.

There is one scenario where the creation/deletion/edition could fail: If the Istio validation webhook is enabled but the Istio registry is not available. In this case, the webhook should be removed in order for this to work.

It can be checked with the following command:

kubectl get ValidatingWebhookConfiguration

2.11 - OSSMConsole CR Reference

Reference page for the OSSMConsole CR. The Kiali Operator will watch for a resource of this type and install the OSSM Console plugin according to that resource’s configuration. Only one resource of this type should exist at any one time.

Example CR

(all values shown here are the defaults unless otherwise noted)

apiVersion: kiali.io/v1alpha1
kind: OSSMConsole
metadata:
  name: ossmconsole
  annotations:
    ansible.sdk.operatorframework.io/verbosity: "1"
spec:
  version: "default"

  deployment:
    imageDigest: ""
    imageName: ""
    imagePullPolicy: "IfNotPresent"
    # default: image_pull_secrets is an empty list
    imagePullSecrets: ["image.pull.secret"]
    imageVersion: ""
    namespace: ""

  kiali:
    graph:
      impl: "pf"
    serviceName: ""
    serviceNamespace: ""
    servicePort: 0

Validating your OSSMConsole CR

A tool is available to allow you to check your own OSSMConsole CR to ensure it is valid. Simply download the validation script and run it, passing in the location of the OSSMConsole CRD you wish to validate with (e.g. the latest version is found here) and the location of your OSSMConsole CR. You must be connected to/logged into a cluster for this validation tool to work.

For example, to validate an OSSMConsole CR named ossmconsole in the namespace istio-system using the latest version of the OSSMConsole CRD, run the following:

bash <(curl -sL https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/bin/validate-ossmconsole-cr.sh) \
  -crd https://raw.githubusercontent.com/kiali/kiali-operator/master/crd-docs/crd/kiali.io_ossmconsoles.yaml \
  --cr-name ossmconsole \
  -n istio-system

For additional help in using this validation tool, pass it the --help option.

Properties

.spec

(object)

This is the CRD for the resources called OSSMConsole CRs. The OpenShift Service Mesh Console Operator will watch for resources of this type and when it detects an OSSMConsole CR has been added, deleted, or modified, it will install, uninstall, and update the associated OSSM Console installation.

.spec.deployment

(object)

.spec.deployment.imageDigest

(string)

If deployment.imageVersion is a digest hash, this value indicates what type of digest it is. A typical value would be ‘sha256’. Note: do NOT prefix this value with a ‘@’.

.spec.deployment.imageName

(string)

Determines which OSSM Console image to download and install. If you set this to a specific name (i.e. you do not leave it as the default empty string), you must make sure that image is supported by the operator. If empty, the operator will use a known supported image name based on which version was defined. Note that, as a security measure, a cluster admin may have configured the operator to ignore this setting. A cluster admin may do this to ensure the operator only installs a single, specific OSSM Console version, thus this setting may have no effect depending on how the operator itself was configured.

.spec.deployment.imagePullPolicy

(string)

The Kubernetes pull policy for the OSSM Console deployment. This is overridden to be ‘Always’ if deployment.imageVersion is set to ‘latest’.

.spec.deployment.imagePullSecrets

.spec.kiali.serviceNamespace

(string)

The namespace where the Kiali service is deployed. If empty, an attempt will be made to auto-discover it from the Kiali OpenShift Route. It will assume that the OpenShift Route and the Kiali service are deployed in the same namespace.

.spec.kiali.servicePort

(integer)

The internal port used by the Kiali service for the API. If empty, an attempt will be made to auto-discover it from the Kiali OpenShift Route.

.spec.version

(string)

The version of the Ansible role that will be executed in order to install OSSM Console. This also indirectly determines the version of OSSM Console that will be installed. You normally will want to use default since this is the only officially supported value today.

If not specified, the value of default is assumed which means the most recent Ansible role is used; thus the most recent release of OSSM Console will be installed.

This version setting affects the defaults of the deployment.imageName and deployment.imageVersion settings. See the documentation for those settings below for additional details. In short, this version setting will dictate which version of the OSSM Console image will be deployed by default. However, if you explicitly set deployment.imageName and/or deployment.imageVersion to reference your own custom image, that will override the default OSSM Console image to be installed; therefore, you are responsible for ensuring those settings are compatible with the Ansible role that will be executed in order to install OSSM Console (i.e. your custom OSSM Console image must be compatible with the rest of the configuration and resources the operator will install).

.status

(object)

The processing status of this CR as reported by the OpenShift Service Mesh Console Operator.

2.12 - Prometheus, Tracing, Grafana

Kiali data sources and add-ons.

Prometheus is a required telemetry data source for Kiali. Jaeger is a highly recommended tracing data source. Kiali also offers a simple Grafana add-on integration. This page describes how to configure Kiali to communicate with these dependencies.

Read the dedicated configuration page to learn more.

2.12.1 - Grafana

This page describes how to configure Grafana for Kiali.

Grafana configuration

Istio provides preconfigured Grafana dashboards for the most relevant metrics of the mesh. Although Kiali offers similar views in its metrics dashboards, it is not in Kiali’s goals to provide the advanced querying options, nor the highly customizable settings, that are available in Grafana. Thus, it is recommended that you use Grafana if you need those advanced options.

Kiali can provide a direct link from its metric dashboards to the equivalent or most similar Grafana dashboard, which is convenient if you need the powerful Grafana options. For these links to appear in Kiali you need to manually configure the Grafana URL and the dashboards that come preconfigured with Istio, like in the following example:

Kiali will query Grafana and try to fetch the configured dashboards. For this reason Kiali must be able to reach Grafana, authenticate, and find the Istio dashboards. The Istio dashboards must be installed in Grafana for the links to appear in Kiali.

spec:
  external_services:
    grafana:
      enabled: true
      # Grafana service name is "grafana" and is in the "telemetry" namespace.
      in_cluster_url: 'http://grafana.telemetry:3000/'
      # Public facing URL of Grafana
      url: 'http://my-ingress-host/grafana'
      dashboards:
      - name: "Istio Service Dashboard"
        variables:
          namespace: "var-namespace"
          service: "var-service"
      - name: "Istio Workload Dashboard"
        variables:
          namespace: "var-namespace"
          workload: "var-workload"
      - name: "Istio Mesh Dashboard"
      - name: "Istio Control Plane Dashboard"
      - name: "Istio Performance Dashboard"
      - name: "Istio Wasm Extension Dashboard"

The described configuration is done in the Kiali CR when Kiali is installed using the Kiali Operator. If Kiali is installed with the Helm chart then the correct way to configure this is via regular –set flags.

2.12.2 - Prometheus

This page describes how to configure Prometheus for Kiali.

Prometheus configuration

Kiali requires Prometheus to generate the topology graph, show metrics, calculate health and for several other features. If Prometheus is missing or Kiali can’t reach it, Kiali won’t work properly.

By default, Kiali assumes that Prometheus is available at the URL of the form http://prometheus.<istio_namespace_name>:9090, which is the usual case if you are using the Prometheus Istio add-on. If your Prometheus instance has a different service name or is installed in a different namespace, you must manually provide the endpoint where it is available, like in the following example:

spec:
  external_services:
    prometheus:
      # Prometheus service name is "metrics" and is in the "telemetry" namespace
      url: "http://metrics.telemetry:9090/"

Notice that you don’t need to expose Prometheus outside the cluster. It is enough to provide the Kubernetes internal service URL.

Kiali maintains an internal cache of some Prometheus queries to improve performance (mainly, the queries to calculate Health indicators). It would be very rare to see data delays, but should you notice any delays you may tune caching parameters to values that work better for your environment.

See the Kiali CR reference page for the current default values.

Compatibility with Prometheus-like servers

Although Kiali assumes a Prometheus server and is tested against it, there are TSDBs that can be used as a Prometheus replacement despite not implementing the full Prometheus API.

Community users have faced two issues when using Prometheus-like TSDBs:

Kiali may report that the TSDB is unreachable, and/or
Kiali may show empty metrics if the TSBD does not implement the /api/v1/status/config.

To fix these issues, you may need to provide a custom health check endpoint for the TSDB and/or manually provide the configurations that Kiali reads from the /api/v1/status/config API endpoint:

spec:
  external_services:
    prometheus:
      # Fix the "Unreachable" metrics server warning.
      health_check_url: "http://custom-tsdb-health-check-url"
      # Fix for the empty metrics dashboards
      thanos_proxy:
        enabled: true
        retention_period: "7d"
        scrape_interval: "30s"

Prometheus Tuning

Production environments should not be using the Istio Prometheus add-on, or carrying over its configuration settings. That is useful only for small, or demo installations. Instead, Prometheus should have been installed in a production-oriented way, following the Prometheus documentation.

This section is primarily for users where Prometheus is being used specifically for Kiali, and possible optimizations that can be made knowing that Kiali does not utilize all of the default Istio and Envoy telemetry.

Metric Thinning

Istio and Envoy generate a large amount of telemetry for analysis and troubleshooting. This can result in significant resources being required to ingest and store the telemetry, and to support queries into the data. If you use the telemetry specifically to support Kiali, it is possible to drop unnecessary metrics and unnecessary labels on required metrics. This FAQ Entry displays the metrics and attributes required for Kiali to operate.

To reduce the default telemetry to only what is needed by Kiali¹ users can add the following snippet to their Prometheus configuration. Because things can change with different versions, it is recommended to ensure you use the correct version of this documentation based on your Kiali/Istio version.

The metric_relabel_configs: attribute should be added under each job name defined to scrape Istio or Envoy metrics. Below we show it under the kubernetes-pods job, but you should adapt as needed. Be careful of indentation.

    - job_name: kubernetes-pods
      metric_relabel_configs:
      - action: drop
        source_labels: [__name__]
        regex: istio_agent_.*|istiod_.*|istio_build|citadel_.*|galley_.*|pilot_[^psx].*|envoy_cluster_[^u].*|envoy_cluster_update.*|envoy_listener_[^dh].*|envoy_server_[^mu].*|envoy_wasm_.*
      - action: labeldrop
        regex: chart|destination_app|destination_version|heritage|.*operator.*|istio.*|release|security_istio_io_.*|service_istio_io_.*|sidecar_istio_io_inject|source_app|source_version

Applying this configuration should reduce the number of stored metrics by about 20%, as well as reducing the number of attributes stored on many remaining metrics.

Metric Thinning with Crippling

The section above drops metrics unused by Kiali. As such, making those configuration changes should not negatively impact Kiali behavior in any way. But some very heavy metrics remain. These metrics can also be dropped, but their removal will impact the behavior of Kiali. This may be OK if you don’t use the affected features of Kiali, or if you are willing to sacrifice the feature for the associated metric savings. In particular, these are “Histogram” metrics. Istio is planning to make some improvements to help users better configure these metrics, but as of this writing they are still defined with fairly inefficient default “buckets”, making the number of associated time-series quite large, and the overhead of maintaining and querying the metrics, intensive. Each histogram actually is comprised of 3 stored metrics. For example, a histogram named xxx would result in the following metrics stored into Prometheus:

xxx_bucket
- The most intensive metric, and is required to calculate percentile values.
xxx_count
- Required to calculate ‘avg’ values.
xxx_sum
- Required to calculate rates over time, and for ‘avg’ values.

When considering whether to thin the Histogram metrics, one of the following three approaches is recommended:

If the relevant Kiali reporting is needed, keep the histogram as-is.
If the relevant Kiali reporting is not needed, or not worth the additional metric overhead, drop the entire histogram.
If the metric chart percentiles are not required, drop only the xxx_bucket metric. This removes the majority of the histogram overhead while keeping rate and average (non-percentile) values in Kiali.

These are the relevant Histogram metrics:

istio_request_bytes

This metric is used to produce the Request Size chart on the metric tabs. It also supports Request Throughput edge labels on the graph.

Appending |istio_request_bytes_.* to the drop regex above would drop all associated metrics and would prevent any request size/throughput reporting in Kiali.
Appending |istio_request_bytes_bucket to the drop regex above, would prevent any request size percentile reporting in the Kiali metric charts.

istio_response_bytes

This metric is used to produce the Response Size chart on the metric tabs. And also supports Response Throughput edge labels on the graph

Appending |istio_response_bytes_.* to the drop regex above would drop all associated metrics and would prevent any response size/throughput reporting in Kiali.
Appending |istio_response_bytes_bucket to the drop regex above would prevent any response size percentile reporting in the Kiali metric charts.

istio_request_duration_milliseconds

This metric is used to produce the Request Duration chart on the metric tabs. It also supports Response Time edge labels on the graph.

Appending |istio_request_duration_milliseconds_.* to the drop regex above would drop all associated metrics and would prevent any request duration/response time reporting in Kiali.
Appending |istio_request_duration_milliseconds_bucket to the drop regex above would prevent any request duration/response time percentile reporting in the Kiali metric charts or graph edge labels.

Scrape Interval

The Prometheus globalScrapeInterval is an important configuration option². The scrape interval can have a significant effect on metrics collection overhead as it takes effort to pull all of those configured metrics and update the relevant time-series. And although it doesn’t affect time-series cardinality, it does affect storage for the data-points, as well as having impact when computing query results (the more data-points, the more processing and aggregation).

Users should think carefully about their configured scrape interval. Note that the Istio addon for prometheus configures it to 15s. This is great for demos but may be too frequent for production scenarios. The prometheus helm charts set a default of 1m, which is more reasonable for most installations, but may not be the desired frequency for any particular setup.

The recommendation for Kiali is to set the longest interval possible, while still providing a useful granularity. The longer the interval the less data points scraped, thus reducing processing, storage, and computational overhead. But the impact on Kiali should be understood. It is important to realize that request rates (or byte rates, message rates, etc) require a minumum of two data points:

rate = (dp2 - dp1) / timePeriod

That means for Kiali to show anything useful in the graph, or anywhere rates are used (many places), the minimum duration must be >= 2 x globalScrapeInterval. Kiali will eliminate invalid Duration options given the globalScrapeInterval.

Kiali does a lot of aggregation and querying over time periods. As such, the number of data points will affect query performance, especially for larger time periods.

For more information, see the Prometheus documentation.

TSDB retention time

The Prometheus tsdbRetentionTime is an important configuration option. It has a significant effect on metrics storage, as Prometheus will keep each reported data-point for that period of time, performing compaction as needed. The larger the retention time, the larger the required storage. Note also that Kiali queries against large time periods, and very large data-sets, may result in poor performance or timeouts.

The recommendation for Kiali is to set the shortest retention time that meets your needs and/or operational limits. In some cases users may want to offload older data to a secondary store. Kiali will eliminate invalid Duration options given the tsdbRetentionTime.

For more information, see the Prometheus documentation.

Some non-essential telemetry remains in order to not over-complicate the configuration change. The remaining telemetry is typically negligible. ↩︎
Note that Prometheus can be configured such that individual scrape points can override the global setting, but Kiali is not currently concerned with this corner case. ↩︎

2.12.3 - Tracing

Configuration to setup Kiali with Jaeger or Grafana Tempo.

Jaeger is the default tracing provider for Kiali. From Kiali version 1.74, Tempo support is also included. This page describes how to configure Jaeger and Grafana Tempo in Kiali.

2.12.3.1 - Jaeger

This page describes how to configure Jaeger for Kiali.

Jaeger configuration

Jaeger is a highly recommended service because Kiali uses distributed tracing data for several features, providing an enhanced experience.

By default, Kiali will try to reach Jaeger at the GRPC-enabled URL of the form http://tracing.<istio_namespace_name>:16685/jaeger, which is the usual case if you are using the Jaeger Istio add-on. If this endpoint is unreachable, Kiali will disable features that use distributed tracing data.

If your Jaeger instance has a different service name or is installed to a different namespace, you must manually provide the endpoint where it is available, like in the following example:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Jaeger is unreachable.
      enabled: true
      # Jaeger service name is "tracing" and is in the "telemetry" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # if you set "use_grpc" to false.
      in_cluster_url: "http://tracing.telemetry:16685/jaeger"
      use_grpc: true
      # Public facing URL of Jaeger
      url: "http://my-jaeger-host/jaeger"

Minimally, you must provide spec.external_services.tracing.in_cluster_url to enable Kiali features that use distributed tracing data. However, Kiali can provide contextual links that users can use to jump to the Jaeger console to inspect tracing data more in depth. For these links to be available you need to set the spec.external_services.tracing.url to the URL where you expose Jaeger outside the cluster.

Default values for connecting to Jaeger are based on the Istio’s provided sample add-on manifests. If your Jaeger setup differs significantly from the sample add-ons, make sure that Istio is also properly configured to push traces to the right URL.

2.12.3.2 - Grafana Tempo

This page describes how to configure Grafana Tempo for Kiali.

Grafana Tempo Configuration

There are two possibilities to integrate Kiali with Grafana Tempo:

Using the Grafana Tempo API: This option returns the traces from the Tempo API in OpenTelemetry format.
Using the Jaeger frontend with the Grafana Tempo backend.
Appendix: Configuration table

Using the Grafana Tempo API

There are two steps to set up Kiali and Grafana Tempo:

Set up the Kiali CR updating the Tracing and Grafana sections.
Set up a Tempo data source in Grafana.

Set up the Kiali CR

This is a configuration example to set up Kiali tracing with Grafana Tempo:

spec:
  external_services:
    tracing:
      # Enabled by default. Kiali will anyway fallback to disabled if
      # Tempo is unreachable.
      enabled: true
      health_check_url: "https://tempo-instance.grafana.net"
      # Tempo service name is "query-frontend" and is in the "tempo" namespace.
      # Make sure the URL you provide corresponds to the non-GRPC enabled endpoint
      # It does not support grpc yet, so make sure "use_grpc" is set to false.
      in_cluster_url: "http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/"
      provider: "tempo"
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"
      use_grpc: false
      # Public facing URL of Tempo 
      url: "https://tempo-tempo-query-frontend-tempo.apps-crc.testing/"

The default UI for Grafana Tempo is Grafana, so we should also set the Grafana URL in the Kiali configuration, such as this example:

spec:
  external_services:
    grafana:
      in_cluster_url: http://grafana.istio-system:3000
      url: https://grafana.apps-crc.testing/

Set up a Tempo Datasource in Grafana

We can optionally set up a default Tempo datasource in Grafana so that you can view the Tempo tracing data within the Grafana UI, as you see here:

Kiali grafana_tempo

To set up the Tempo datasource, go to the Home menu in the Grafana UI, click Data sources, then click the Add new data source button and select the Tempo data source. You will then be asked to enter some data to configure the new Tempo data source:

Kiali grafana_tempo

The most important values to set up are the following:

Mark the data source as default, so the URL that Kiali uses will redirect properly to the Tempo data source.
Update the HTTP URL. This is the internal URL of the HTTP tempo frontend service. e.g. http://tempo-tempo-query-frontend.tempo.svc.cluster.local:3200/

Additional configuration

The Traces tab in the Kiali UI will show your traces in a bubble chart:

Kiali grafana_tempo

Increasing performance is achievable by enabling gRPC access, specifically for query searches. However, accessing the HTTP API will still be necessary to gather information about individual traces. This is an example to configure the gRPC access:

spec:
  external_services:
    tracing:
      enabled: true
      # grpc port defaults to 9095
      grpc_port: 9095 
      in_cluster_url: "http://query-frontend.tempo:3200"
      provider: "tempo"
      use_grpc: true
      url: "http://my-tempo-host:3200"

Service check URL

By default, Kiali will check the service health in the endpoint /status/services, but sometimes, this is exposed in a different url, which can lead to a component unreachable message:

component_unreachable

This can be changed with the health_check_url configuration option.

spec:
  external_services:
    tracing:
      health_check_url: "http://query-frontend.tempo:3200"

Configuration for the Grafana Tempo Datasource

In order to correctly redirect Kiali to the right Grafana Tempo Datasource, there are a couple of configuration options to update:

spec:
  external_services:
    tracing:
      tempo_config:
        org_id: "1"
        datasource_uid: "a8d2ef1c-d31c-4de5-a90b-e7bc5252cd00"

org_id is usually not needed since “1” is the default value which is also Tempo’s default org id. The datasource_uid needs to be updated in order to redirect to the right datasource in Grafana versions 10 or higher.

Using the Jaeger frontend with Grafana Tempo tracing backend

It is possible to use the Grafana Tempo tracing backend exposing the Jaeger API. tempo-query is a Jaeger storage plugin. It accepts the full Jaeger query API and translates these requests into Tempo queries.

Since Tempo is not yet part of the built-in addons that are part of Istio, you need to manage your Tempo instance.

Tanka

The official Grafana Tempo documentation explains how to deploy a Tempo instance using Tanka. You will need to tweak the settings from the default Tanka configuration to:

Expose the Zipkin collector
Expose the GRPC Jaeger Query port

When the Tempo instance is deployed with the needed configurations, you have to set meshConfig.defaultConfig.tracing.zipkin.address from Istio to the Tempo Distributor service and the Zipkin port. Tanka will deploy the service in distributor.tempo.svc.cluster.local:9411.

The external_services.tracing.in_cluster_url Kiali option needs to be set to: http://query-frontend.tempo.svc.cluster.local:16685.

Tempo Operator

The Tempo Operator for Kubernetes provides a native Kubernetes solution to deploy Tempo easily in your system.

After installing the Tempo Operator in your cluster, you can create a new Tempo instance with the following CR:

kubectl create namespace tempo
kubectl apply -n tempo -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: smm
spec:
  storageSize: 1Gi
  storage:
    secret:
      type: s3
      name: object-storage
  template:
    queryFrontend:
      component:
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
      jaegerQuery:
        enabled: true
        ingress:
          type: ingress
EOF

Note the name of the bucket where the traces will be stored in our example is called object-storage. Check the Tempo Operator documentation to know more about what storages are supported and how to create the secret properly to provide it to your Tempo instance.

Now, you are ready to configure the meshConfig.defaultConfig.tracing.zipkin.address field in your Istio installation. It needs to be set to the 9411 port of the Tempo Distributor service. For the previous example, this value will be tempo-smm-distributor.tempo.svc.cluster.local:9411.

Now, you need to configure the in_cluster_url setting from Kiali to access the Jaeger API. You can point to the 16685 port to use GRPC or 16686 if not. For the given example, the value would be http://tempo-ssm-query-frontend.tempo.svc.cluster.local:16685.

There is a related tutorial with detailed instructions to setup Kiali and Grafana Tempo with the Operator.

Configuration table

Supported versions

Kiali Version	Jaeger	Tempo	Tempo with JaegerQuery
<= 1.79 (OSSM 2.5)	✅	❌	✅
> 1.79	✅	✅	✅

Minimal configuration for Kiali <= 1.79

In external_services.tracing

	http	grpc
Jaeger	`.in_cluster_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.in_cluster_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`.in_cluster_url = 'http://query_frontend_url:16686'` `.use_grpc = false`	`.in_cluster_url = 'http://query_frontend_url:16685'` `.use_grpc = true (Not required: by default)`

Minimal configuration for Kiali > 1.79

	http	grpc
Jaeger	`.in_cluster_url = 'http://jaeger_service_url:16686/jaeger'` `.use_grpc = false`	`.in_cluster_url = 'http://jaeger_service_url:16685/jaeger'` `.use_grpc = true (Not required: by default)`
Tempo	`in_cluster_url = 'http://query_frontend_url:3200'` `.use_grpc = false` `.provider = 'tempo'`	`.in_cluster_url = 'http://query_frontend_url:3200'` `.grpc_port: 9095` `.provider: 'tempo'` `.use_grpc = true (Not required: by default)`

2.13 - Traffic Health

Customizing Health for Request Traffic.

There are times when Kiali’s default thresholds for traffic health do not work well for a particular situation. For example, at times 404 response codes are expected. Kiali has the ability to set powerful, fine-grained overrides for health configuration.

Default Configuration

By default Kiali uses the traffic rate configuration shown below. Application errors have minimal tolerance while client errors have a higher tolerance reflecting that some level of client errors is often normal (e.g. 404 Not Found):

For http protocol 4xx are client errors and 5xx codes are application errors.
For grpc protocol all 1-16 are errors (0 is success).

So, for example, if the rate of application errors is >= 0.1% Kiali will show Degraded health and if > 10% will show Failure health.

# ...
  health_config:
    rate:
      - namespace: ".*"
        kind: ".*"
        name: ".*"
        tolerance:
          - code: "^5\\d\\d$"
            direction: ".*"
            protocol: "http"
            degraded: 0
            failure: 10
          - code: "^4\\d\\d$"
            direction: ".*"
            protocol: "http"
            degraded: 10
            failure: 20
          - code: "^[1-9]$|^1[0-6]$"
            direction: ".*"
            protocol: "grpc"
            degraded: 0
            failure: 10
# ...

Custom Configuration

Custom health configuration is specified in the Kiali CR. To see the supported configuration syntax for health_config see the Kiali CR Reference.

Kiali applies the first matching rate configuration (namespace, kind, etc) and calculates the status for each tolerance. The reported health will be the status with highest priority (see below).

Rate Option	Definition	Default
namespace	Matching Namespaces (regex)	.* (match all)
kind	Matching Resource Types (workload\|app\|service) (regex)	.* (match all)
name	Matching Resource Names (regex)	.* (match all)
tolerance	Array of tolerances to apply.

Tolerance Option	Definition	Default
code	Matching Response Status Codes (regex) [1]	required
direction	Matching Request Directions (inbound\|outbound) (regex)	.* (match all)
protocol	Matching Request Protocols (http\|grpc) (regex)	.* (match all)
degraded	Degraded Threshold(% matching requests >= value)	0
failure	Failure Threshold (% matching requests >= value)	0

[1] The status code typically depends on the request protocol. The special code -, a single dash, is used for requests that don’t receive a response, and therefore no response code.

Kiali reports traffic health with the following top-down status priority :

Priority	Rule (value=% matching requests)	Status
1	value >= FAILURE threshold	FAILURE
2	value >= DEGRADED threshold AND value < FAILURE threshold	DEGRADED
3	value > 0 AND value < DEGRADED threshold	HEALTHY
4	value = 0	HEALTHY
5	No traffic	No Health Information

Examples

These examples use the repo https://github.com/kiali/demos/tree/master/error-rates.

In this repo we can see 2 namespaces: alpha and beta (Demo design).

Alpha

Where nodes return the responses (You can configure responses here):

App (alpha/beta)	Code	Rate
x-server	200	9
x-server	404	1
y-server	200	9
y-server	500	1
z-server	200	8
z-server	201	1
z-server	201	1

The applied traffic rate configuration is:

# ...
health_config:
  rate:
   - namespace: "alpha"
     tolerance:
       - code: "404"
         failure: 10
         protocol: "http"
       - code: "[45]\\d[^\\D4]"
         protocol: "http"
   - namespace: "beta"
     tolerance:
       - code: "[4]\\d\\d"
         degraded: 30
         failure: 40
         protocol: "http"
       - code: "[5]\\d\\d"
         protocol: "http"
# ...

After Kiali adds default configuration we have the following (Debug Info Kiali):

{
  "healthConfig": {
    "rate": [
      {
        "namespace": "/alpha/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/404/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/[45]\\d[^\\D4]/",
            "degraded": 0,
            "failure": 0,
            "protocol": "/http/",
            "direction": "/.*/"
          }
        ]
      },
      {
        "namespace": "/beta/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/[4]\\d\\d/",
            "degraded": 30,
            "failure": 40,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/[5]\\d\\d/",
            "degraded": 0,
            "failure": 0,
            "protocol": "/http/",
            "direction": "/.*/"
          }
        ]
      },
      {
        "namespace": "/.*/",
        "kind": "/.*/",
        "name": "/.*/",
        "tolerance": [
          {
            "code": "/^5\\d\\d$/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/^4\\d\\d$/",
            "degraded": 10,
            "failure": 20,
            "protocol": "/http/",
            "direction": "/.*/"
          },
          {
            "code": "/^[1-9]$|^1[0-6]$/",
            "degraded": 0,
            "failure": 10,
            "protocol": "/grpc/",
            "direction": "/.*/"
          }
        ]
      }
    ]
  }
}

What are we applying?

For namespace alpha, all resources
Protocol http if % requests with error code 404 are >= 10 then FAILURE, if they are > 0 then DEGRADED
Protocol http if % requests with others error codes are> 0 then FAILURE.
For namespace beta, all resources
Protocol http if % requests with error code 4xx are >= 40 then FAILURE, if they are >= 30 then DEGRADED
Protocol http if % requests with error code 5xx are > 0 then FAILURE
For other namespaces Kiali will apply the defaults.
Protocol http if % requests with error code 5xx are >= 20 then FAILURE, if they are >= 0.1 then DEGRADED
Protocol grpc if % requests with error code match /^[1-9]$|^1[0-6]$/ are >= 20 then FAILURE, if they are >= 0.1 then DEGRADED

Alpha	Beta

2.14 - Virtual Machine workloads

Ensuring Kiali can visualize a VM WorkloadEntry.

Introduction

Kiali graph visualizes both Virtual Machine workloads (WorkloadEntry) and pod-based workloads, running inside a Kubernetes cluster. You must ensure that the Istio Proxy is running, and correctly configured, on the Virtual Machine. Also, Prometheus must be able to scrape the metrics endpoint of the Istio Proxy running on the VM. Kiali will then be able to read the traffic telemetry for the Virtual Machine workloads, and incorporate the VM workloads into the graph.

Kiali does not currently distinguish between pod-based and VM-based workloads nor does Kiali support viewing additional details for the VM-based workloads beyond what is displayed on the graph. One way to distinguish between the two is to give the VM-based workloads a different version label than the pod-based workloads.

Configuring Prometheus to scrape VM-based Istio Proxy

Once the Istio Proxy is running on a Virtual Machine, configuring Prometheus to scrape the VM’s Istio Proxy metrics endpoint is the only configuration Kiali needs to display traffic for the VM-based workload. Configuring Prometheus will vary between environments. Here is a very simple example of a Prometheus configuration that includes a job to scrape VM based workloads:

- job_name: bookinfo-vms
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /stats/prometheus
  scheme: http
  follow_redirects: true
  static_configs:
  - targets:
    - details-v1:15020
    - productpage-v1:15020
    - ratings-v1:15020
    - reviews-v1:15020
    - reviews-v2:15020
    - reviews-v3:15020

3 - Features

An overview of important Kiali features.

3.1 - Application Wizards

Using Kiali wizards to generate application and request routing configuration.

Istio Application Wizards

Kiali provides Actions to create, update and delete Istio configuration, driven by wizards.

Actions can be applied to a Service

Service Detail Actions

Actions can also be applied to a Workload

Workload Detail Actions

And, actions are available for an entire Namespace

Namespace Actions

Service Actions

Kiali offers a robust set of service actions, with accompanying wizards.

Traffic Management: Request Routing

The Request Routing Wizard allows creating multiple routing rules.

Every rule is composed of a Request Matching and a Routes To section.
The Request Matching section can add multiple filters using HEADERS, URI, SCHEME, METHOD or AUTHORITY HTTP parameters.
The Request Matching section can be empty, in this case any HTTP request received is matched against this rule.
The Routes To section can specify the percentage of traffic that is routed to a specific workload.

Request Routing

Istio applies routing rules in order, meaning that the first rule matching an HTTP request (top-down) performs the routing. The Matching Routing Wizard allows changing the rule order.

Traffic Management: Fault Injection

The Fault Injection Wizard allows injecting faults to test the resiliency of a Service.

HTTP Delay specification is used to inject latency into the request forwarding path.
HTTP Abort specification is used to immediately abort a request and return a pre-specified status code.

Fault Injection

Traffic Management: Traffic Shifting

The Traffic Shifting Wizard allows selecting the percentage of traffic that is routed to a specific workload.

Traffic Shifting

Kiali also provides an analogous action for TCP traffic shifting.

Traffic Management: Request Timeouts

The Request Timeouts Wizard sets up request timeouts in Envoy, using Istio.

HTTP Timeout defines the timeout for a request.
HTTP Retry describes the retry policy to use when an HTTP request fails.

Request Timeouts

Traffic Management: Gateways

Traffic Management Wizards have an Advanced Options section that can be used to extend the scenario.

One available Advanced Option is to expose a Service to external traffic through an existing Gateway or to create a new Gateway for this Service.

Gateway

Traffic Management: Circuit Breaker

Traffic Management Wizards allows defining Circuit Breakers on Services as part of the available Advanced Options.

Connection Pool defines the connection limits for an upstream host.
Outlier Detection implements the Circuit Breaker based on the consecutive errors reported.

Circuit Breaker

Security: Traffic Policy

Traffic Management Advanced Options allows defining Security and Load Balancing settings.

TLS related settings for connections to the upstream service.
Automatically generate a PeerAuthentication resource for this Service.
Load balancing policies to apply for a specific destination.

Traffic Policy

Workload Actions

Automatic Sidecar Injection

A Workload can be individually managed to control the Sidecar Injection.

A default scenario is to indicate this at Namespace level but there can be cases where a Workload shouldn’t be part of the Mesh or vice versa.

Kiali allows users to alter the Deployment template and propagate this configuration into the Pods.

Workload-specific Disable Sidecar Injection

Namespace Actions

The Kiali Overview page offers several Namespace actions, in any of its views: Expanded, Compacted or Table.

Namespace Actions

Show

Show actions navigate from a Namespace to its specific Graph, Applications, Workloads, Services or Istio Config pages.

Automatic Sidecar Injection

When Automatic Sidecar Injection is enabled in the cluster, a Namespace can be labeled to enable/disable the injection webhook, controlling whether new deployments will automatically have a sidecar.

Canary Istio upgrade

When Istio Canary revision is installed, a Namespace can be labeled to that canary revision, so the sidecar of canary revision will be injected into workloads of the namespace.

Security: Traffic Policies

Kiali can generate Traffic Policies based on the traffic for a namespace.

For example, at some point a namespace presents a traffic graph like this:

Traffic Policies: Graph

And a user may want to add Traffic Policies to secure that communication. In other words, to prevent traffic other than that currently reflected in the Graph’s Services and Workloads.

Using the Create Traffic Policies action on a namespace, Kiali will generate AuthorizationPolicy resources per every Workload in the Namespace.

Traffic Policies: Sidecars and Authorization Policies

3.2 - Detail Views

Kiali provides list and detail views for your mesh components.

Kiali provides filtered list views of all your service mesh definitions. Each view provides health, details, YAML definitions and links to help you visualize your mesh. There are list and detail views for:

Applications
Istio Configuration
Services
Workloads

Detail list apps Detail list service Detail list workload Detail list Istio config

Selecting an object from the list will bring you to its detail page. For Istio Config, Kiali will present its YAML, along with contextual validation information. Other mesh components present a variety of Tabs.

Overview Tab

Overview is the default Tab for any detail page. The overview tab provides detailed information, including health status, and a detailed mini-graph of the current traffic involving the component. The full set of tabs, as well as the detailed information, varies based on the component type.

Each Overview provides:

links to related components and linked Istio configuration.
health status.
validation information.
an Action menu for actions that can be taken on the component.
- several Wizards are available.

And also type-specfic information. For example:

Service detail includes Network information.
Workload detail provides backing Pod information.

Detail overview app Detail overview service Detail overview workload

Both Workload and Service detail can be customized to some extent, by adding additional details supplied as annotations. This is done through the additional_display_details field in the Kiali CR.

Detail overview additional details

Traffic

The Traffic Tab presents a service, app, or workload’s Inbound and Outbound traffic in a table-oriented way:

Detail traffic

Logs

Workload detail offers a special Logs tab. Kiali offers a special unified logs view that lets users correlate app and proxy logs. It can also add-in trace span information to help identify important traces based on associated logging. More powerful features include substring or regex Show/Hide, full-screen, and the ability to set proxy log level without a pod restart.

Detail logs

Metrics

Each detail view provides Inbound Metrics and/or Outbound Metrics tabs, offering predefined metric dashboards. The dashboards provided are tailored to the relevant application, workload or service level. Application and workload detail views show request and response metrics such as volume, duration, size, or tcp traffic. The service detail view shows request and response metrics for inbound traffic only.

Kiali allows the user to customize the charts by choosing the charted dimensions. It can also present metrics reported by either source or destination proxy metrics. And for troublshooting it can overlay trace spans.

Detail metric inbound Detail metrics outbound

Traces

Each detail view provides a Traces tab with a native integration with Jaeger. For more, see Tracing.

Dashboards

Kiali will display additional tabs for each applicable Built-In Dashboard or Custom Dashboard.

Built-in dashboards

Kiali comes with built-in dashboards for several runtimes, including Envoy, Go, Node.js, and others.

Envoy

The most important built-in dashboard is for Envoy. Kiali offers the Envoy tab for many workloads. The Envoy tab is actually a Built-In Dashboard, but it is very common as it applies to any workload injected with, or that is itself, an Envoy proxy. Being able to inspect the Envoy proxy is invaluable when troublshooting your mesh. The Envoy tab itself offers five subtabs, exposing a wealth of information.

Detail Envoy

Istio’s Envoy sidecars supply some internal metrics, that can be viewed in Kiali. They are different than the metrics reported by Istio Telemetry, which Kiali uses extensively. Some of Envoy’s metrics may be redundant.

Note that the enabled Envoy metrics can be tuned, as explained in the Istio documentation: it’s possible to get more metrics using the statsInclusionPrefixes annotation. Make sure you include cluster_manager and listener_manager as they are required.

For example, sidecar.istio.io/statsInclusionPrefixes: cluster_manager,listener_manager,listener will add listener metrics for more inbound traffic information. You can then customize the Envoy dashboard of Kiali according to the collected metrics.

Go

Contains metrics such as the number of threads, goroutines, and heap usage. The expected metrics are provided by the Prometheus Go client.

Example to expose built-in Go metrics:

        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":2112", nil)

As an example and for self-monitoring purpose Kiali itself exposes Go metrics.

The pod annotation for Kiali is: kiali.io/dashboards: go

Kiali

Kiali has its own built-in dashboard that helps you observe performance of the Kiali server itself. To view this dashboard, navigate to either the application or workload view of the Kiali server and select the Kiali Internal Metrics tab to see Kiali’s own internal metrics:

Kiali Metrics (app view) Kiali Metrics (workload view)

Node.js

Contains metrics such as active handles, event loop lag, and heap usage. The expected metrics are provided by prom-client.

Example of Node.js metrics for Prometheus:

const client = require('prom-client');
client.collectDefaultMetrics();
// ...
app.get('/metrics', (request, response) => {
  response.set('Content-Type', client.register.contentType);
  response.send(client.register.metrics());
});

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/ratings

The pod annotation for Kiali is: kiali.io/dashboards: nodejs

Quarkus

Contains JVM-related, GC usage metrics. The expected metrics can be provided by SmallRye Metrics, a MicroProfile Metrics implementation. Example with maven:

    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-smallrye-metrics</artifactId>
    </dependency>

The pod annotation for Kiali is: kiali.io/dashboards: quarkus

Spring Boot

Three dashboards are provided: one for JVM memory / threads, another for JVM buffer pools and the last one for Tomcat metrics. The expected metrics come from Spring Boot Actuator for Prometheus. Example with maven:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/details

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: springboot-jvm,springboot-jvm-pool,springboot-tomcat

By default, the metrics are exposed on path /actuator/prometheus, so it must be specified in the corresponding annotation: prometheus.io/path: "/actuator/prometheus"

Thorntail

Contains mostly JVM-related metrics such as loaded classes count, memory usage, etc. The expected metrics are provided by the MicroProfile Metrics module. Example with maven:

    <dependency>
      <groupId>io.thorntail</groupId>
      <artifactId>microprofile-metrics</artifactId>
    </dependency>

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/productpage

The pod annotation for Kiali is: kiali.io/dashboards: thorntail

Vert.x

Several dashboards are provided, related to different components in Vert.x: HTTP client/server metrics, Net client/server metrics, Pools usage, Eventbus metrics and JVM. The expected metrics are provided by the vertx-micrometer-metrics module. Example with maven:

    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-micrometer-metrics</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Init example of Vert.x metrics, starting a dedicated server (other options are possible):

      VertxOptions opts = new VertxOptions().setMetricsOptions(new MicrometerMetricsOptions()
          .setPrometheusOptions(new VertxPrometheusOptions()
              .setStartEmbeddedServer(true)
              .setEmbeddedServerOptions(new HttpServerOptions().setPort(9090))
              .setPublishQuantiles(true)
              .setEnabled(true))
          .setEnabled(true));

Full working example: https://github.com/jotak/bookinfo-runtimes/tree/master/reviews

The pod annotation for Kiali with the full list of dashboards is: kiali.io/dashboards: vertx-client,vertx-server,vertx-eventbus,vertx-pool,vertx-jvm

Custom Dashboards

When the built-in dashboards don’t offer what you need, it’s possible to create your own. See Custom Dashboard Configuration for more information.

3.3 - Health

How Kiali reflects your Service Mesh Health.

Kiali help users know whether their service mesh is healthy. This includes the health of the mesh infrastructure itself, and the deployed application services.

Service Mesh Infrastructure Health

Users can quickly confirm the health of their infrastructure by looking at the Kiali Masthead. If Kiali detects any health issues with the infrastructure of the mesh it will show an indication in the masthead, severity will be reflected via color, and hovring will show the detail:

Masthead Health

For more detail on how Kiali tracks the Istio infrastructure status, see the Istio Status Feature.

Overview Health

The default Kiali page is an Overview Dashboard. This view will quickly allow you to identify namespaces with issues. It provides a summary of configuration health, component health and request traffic health. The component health is selectable via a dropdown and the page offers various filter, sort and presentation options:

Overview Health

Graph Health

The Kiali Graph offers a rich visualization of your service mesh traffic. The health of Nodes and Edges is represented via a standard color system using shades of orange and red to reflect degraded and failure-level traffic health. Red or orange nodes or edges may need attention. The color of an edge represents the request health between the relevant nodes. Note that node shape indicates the type of component, such as service, workload, or app.

The health of nodes and edges is refreshed automatically based on the user’s preference. The graph can also be paused to examine a particular state, or replayed to re-examine a particular time period.

Graph Health

Health Configuration

Kiali calculates health by combining the individual health of several indicators, such as pods and request traffic. The global health of a resource reflects the most severe health of its indicators.

Health Indicators

The table below lists the current health indicators and whether the indicator supports custom configuration for its health calculation.

Indicator	Supports Configuration
Pod Status	No
Traffic Health	Yes

Icons and colors

Kiali uses icons and colors to indicate the health of resources and associated request traffic.

No Health Information (NA)
Healthy
Degraded
Failure

Custom Request Health

3.4 - Internationalization

How Kiali is displayed in mutliple languages.

Kiali is used worldwide and some users prefer to display Kiali in a language that they are more comfortable with than English. For this reason Kiali supports internationalization, and it can be localized into multiple languages.

Current supported languages are English and Chinese.

Language Selector

Kiali provides a language selector in the Masthead to be able to switch Kiali between supported languages:

Masthead language selector

By default the language selector is hidden, and English is the default language.

In order to show the language selector a feature flag must be enabled in the Kiali CR:

spec:
  kiali_feature_flags:
    ui_defaults:
      i18n:
        language: en
        show_selector: true

You can also modify the default language shown in Kiali (values according to ISO 639-1 codes):

Language	Code
Chinese	zh
English	en

As an example, this is how Kiali displays the Overview page in Chinese:

Overview page in Chinese

If you want to collaborate with us on adding a new language or improving the translation of an existing one, please refer to the internationalization section of the kiali project’s README for UI

3.5 - Istio Ambient Mesh

Visualizing Ambient Mesh with Kiali

Kiali provides visualization for Ambient Mesh components:

Control Plane Ambient Mesh
Ambient nampespace
Workloads in Mesh
Ambient Telemetry

The Kiali Ambient features, as well as Ambient Mesh, are evolving. Some of these features are in alpha status. For enhancements or detected issues, don’t hesitate to open a GitHub issue.

Control Plane Ambient Mesh

When the control plane is in Ambient mode, Kiali will show an Ambient badge on the Overview page control plane namespace card. It will also be reflected in the control plane side-panel on the Mesh page. This badge indicates that Kiali has detected a ztunnel (the L4 component for Ambient) in the control plane.

Ambient Control Plane

Ambient Namespace

When a namespace is labeled with istio.io/dataplane-mode=ambient it is included in Ambient Mesh, and Kiali will show the Ambient badge on that Overview page namespace card:

Ambient Data Plane

Workloads in Ambient Mesh

When a workload, application, or service is part of the Ambient Mesh, a badge will appear in the namespace details. When hovering over this badge, further information about the workload will be displayed:

In Mesh: Indicating that it was included in Ambient, and the traffic is redirected to ztunnel to provide L4 features (L4 authorization and telemetry, and encrypted data transport)
In Mesh with waypoint enabled: Additionally, it can include the L7 badge which means that a waypoint proxy is deployed (providing additional L7 capabilities):

Workloads Captured by Ambient

Ambient Telemetry

The Traffic graph generated with Ambient telemetry differs slightly from the usual graph, as the HTTP traffic and TCP traffic have different reporters.

The telemetry reported with sidecars represents the kind of traffic for the request (green edges for HTTP, blue edges for TCP). In Ambient, this information depends on the element reporting the Telemetry. The Ztunnel will report all the traffic as TCP:

ztunnel graph

The following bookinfo namespace is in Ambient Mesh with a waypoint proxy enabled. Therefore, the telemetry is reported from ztunnel and from the Waypoint, resulting in double edges connecting different nodes (Note that the Graph page toolbar offers a Traffic menu, letting you be selective about the protocols shown):

Ambient Telemetry

There is an additional display option, Waypoint proxies for the Ambient Mesh, that will display the waypoint proxies in the graph (This is still an alpha-level feature and will likely change):

Ambient Telemetry

3.6 - Istio Configuration

Using Kiali to generate Istio mesh-wide configuration.

Kiali is more than observability, it also helps you to configure, update and validate your Istio service mesh.

The Istio configuration view provides advanced filtering and navigation for Istio configuration objects, such as Virtual Services and Gateways. Kiali provides inline config editing and powerful semantic validation for Istio resources.

Istio Config List

Validations

Kiali performs a set of validations on your Istio Objects, such as Destination Rules, Service Entries, and Virtual Services. Kiali’s validations go above and beyond what Istio offers. Where Istio offers mainly static checks for well-formed definitions, Kiali performs semantic validations to ensure that the definitions make sense, across objects, and in some cases even across namespaces. Kiali validations are based on the runtime status of your service mesh.

Check the complete list of validations for further information.

Istio Config Validation

Wizards

Kiali Wizards provide a way to apply an Istio service mesh pattern, letting Kiali generate the Istio Configuration. Wizards are launched from Kiali’s Action menus, located across the UI on relevant pages. Wizards can create and update Istio Config for Routing, Gateways, Security scenarios and more.

Istio Config Page Wizards

These Create Actions are available on the Istio Config page:

Istio Config Create Actions

Authorization Wizards

Kiali allows creation of Istio AuthorizationPolicy resources:

AuthorizationPolicy

Istio PeerAuthentication resources:

PeerAuthentication

Istio RequestAuthentication resources:

RequestAuthentication

Traffic Wizards

Kiali also allows creation of Istio Gateway resources.

Gateway

Istio ServiceEntry resources:

ServiceEntry

Istio Sidecar resources:

Sidecar

Other Kiali Wizards

Kiali also has Wizards available from the Overview page, and many details pages, such as Service Detail to create routing rules. The Kiali Travel Tutorial goes into several of these wizards.

Overview Wizards

The Overview page has namespace-specific actions for creating traffic policies:

Overview Actions

Service Wizards

The Service Detail page offers several wizards to create traffic control config:

Service Actions

3.7 - Istio Infrastructure Status

How Kiali monitors your Istio infrastructure.

A service mesh simplifies application services by deferring the non-business logic to the mesh. But for healthy applications the service mesh infrastructure must also be running normally. Kiali monitors the multiple components that make up the service mesh, letting you know if there is an underlying problem.

Istio component status

A component status will be one of: Not found, Not ready, Unreachable, Not healthy and Healthy. Not found means that Kiali is not able to find the deployment. Not ready means no pods are running. Unreachable means that Kiali hasn’t been successfully able to communicate with the component (Prometheus, Grafana and Jaeger). Not healthy means that the deployment doesn’t have the desired amount of replicas running. Otherwise, the component is Healthy and it won’t be shown in the list.

Regarding the severity of each component, there are only two options: core or add-on. The core components are those shown as errors (in red) whereas the add-ons are displayed as warnings (in orange).

By default, Kiali checks that the core components “istiod”, “ingress”, and “egress” are installed and running in the control plane namespace, and that the add-ons “prometheus”, “grafana” and “jaeger” are available.

Mesh page

Detailed information about the Istio infrastructure status is displayed on the mesh page. It shows an infrastructure topology view with core and add-on components, their health, and how they are connected to each other.

Similar to the traffic graph, the left side of the page shows the topology view, while the right side displays information about the selected node. If no node is selected, global infrastructure information is shown, including the status, version, and cluster of every component.

Mesh overview information

Connection issues between Kiali and any component are indicated with a red dotted line and a red health indicator in the target side panel.

Connection issue in the mesh

The specific information shown in the target side panel depends on the type of node selected:

Kiali

When you click on the Kiali node, you can check information such as the version, health status, and configuration values.

Kiali information

Istio control plane

When you click on the Istio control plane, you can check information such as the Istio version, mTLS status, outbound policy, CPU and memory metrics, configuration table, and more.

Istio control plane information

Data plane

When you click on the cluster data plane, you can check the basic information of each namespace belonging to that data plane (Istio configuration, traffic inbound/outbound), similar to what you can see on the overview page.

Data plane information

Add-on components

When you click on the “prometheus”, “grafana” or “jaeger” node, , its health status, version, and configuration values are displayed:

Add-on information

Certificate Information Indicators

In some situations, it is useful to get information about the certificates used by internal mTLS, for example:

Know whether the default CA is used or if there is another CA configured.
Check the certificates issuer and their validity timestamps to troubleshoot any issue with certificates.

The certificates shown depends on how Istio is configured. The following cases are possible:

Using Istio CA certificates (default), the information shown is from a secret named istio-ca-secret.
Using Plug in CA certificates, the information shown is from a secret named cacerts.
Using DNS certificates, the information shown is from reading many secrets found in Istio configuration.

The following is an example of viewing the default case:

Certificates information

Note that displaying this configuration requires permissions to read secrets (istio-ca-secret by default, possibly cacerts or any secret configured when using DNS certificates).

Having these permissions may concern users. For this reason, this feature is implemented as a feature flag and not only can be disabled, avoiding any extra permissions to read secrets, but also a list of secrets can be configured to explicitly grant read permissions for some secrets in the control plane namespace. By default, this feature is enabled with a Kiali CR configuration equivalent to the following:

spec:
  kiali_feature_flags:
    certificates_information_indicators:
      enabled: true
      secrets:
      - cacerts
      - istio-ca-secret

You can extend this default configuration with additional secrets, remove secrets you don’t want, or disable the feature.

If you add additional secrets, the Kiali operator also needs the same privileges in order to configure Kiali successfully. If you used the Helm Charts to install the operator, specify the secretReader value with the required secrets:

$ helm install \
    --namespace kiali-operator \
    --create-namespace \
    --set "secretReader={cacerts,istio-ca-secret}"
    kiali-operator \
    kiali/kiali-operator

If you installed the operator via the OperatorHub you need to update the operator privileges as a post-installation step, as follows:

$ kubectl patch $(kubectl get clusterroles -o name | grep kiali-operator) --type "json" -p '[{"op":"add","path":"/rules/0","value":{"apiGroups":[""],"resources":["secrets"],"verbs":["get"],"resourceNames":["secret-name-to-be-read"]}}]'

Replace secret-name-to-be-read with the secret name you want the operator to read and restart the Kiali operator pod after running the previous command.

3.8 - Multi-cluster

Advanced Mesh Deployment and Multi-cluster support.

Support for multi-cluster deployments is currently experimental and subject to change.

A basic Istio mesh deployment has a single control plane with a single data plane, deployed on a single Kubernetes cluster. But Istio supports a variety of advanced deployment models. It allows a mesh to span multiple primary (control plane) and/or remote (data plane only) clusters, and can use a single or multi-network approach. The only strict rule is that within a mesh service names are unique. A non-basic mesh deployment generally involves multiple clusters. See installation instructions for more detail on installing advanced mesh deployments.

A single Kiali install can currently work with at most one mesh, one metric store and one trace store but it can be configured for “single cluster” or “multi-cluster”. All clusters must be part of the same mesh and report to the same metric and trace store, whether directly or via some sort of aggregator. See the multi-cluster configuration page for more information on requirements.

For multi-cluster configurations, Kiali provides a unified view and management of your mesh across clusters.

Graph View: Cluster and Namespace Boxing

Istio provides cluster names in the traffic telemetry for multi-cluster installations. The Kiali graph can use this information to better visualize clusters and namespaces. The Display menu offers two multi-cluster related options: Cluster Boxes and Namespace Boxes. When enabled, either separately or together, the graph will generate boxes to help identify the relevant nodes and edges, and to see traffic traveling between them.

Each box type supports selection and provides a side-panel summary of traffic. Below we see a Bookinfo traffic graph for when Bookinfo services are deployed across the East and West clusters. The West cluster box is selected. You see traffic for all services and workloads across both clusters. Single cluster configurations will show some traffic across clusters but not all.

Multi-cluster traffic graph

Unified Multi-cluster configuration

Unlike single-cluster configurations, multi-cluster configurations show list/details pages across all clusters.

List Views: Aggregated mesh view

With a multi-cluster Kiali configuration, you can view all apps, workloads, services, and Istio config in your mesh from a single place.

Multi-cluster list pages

Detail Views: Dig into details across clusters

The detail pages provide the same functionality across all clusters that you have access to for a single cluster, including viewing logs, metrics, traces, envoy details, and more.

Multi-cluster detail pages

Overview: Cross cluster namespace info

The overview page shows namespace information across all configured clusters.

Multi-cluster overview

Mesh view: Cluster, Istio and data plane boxing

The mesh graph displays infrastructure information for multiple clusters, Istio control planes, and data planes according to the Istio deployment (primary-remote or multi-primary).

Multi-cluster mesh

Roadmap

See this issue to see the multi-cluster roadmap for Kiali.

3.9 - Security

How Kiali visualizes mTLS.

Kiali gives support to better understand how mTLS is used in Istio meshes. Find those helpers in the graph, the masthead menu, the overview page and specific validations.

At the right side of the Masthead, Kiali shows a lock when the mesh has strictly enabled mTLS for the whole service mesh. It means that all the communications in the mesh uses mTLS.

mTLS mesh-wide strict mTLS mesh-wide permissive

Kiali shows a hollow lock when either the mesh is configured in PERMISSIVE mode or there is a misconfiguration in the mesh-wide mTLS configuration.

Overview locks

The overview page shows all the available namespaces with aggregated data. Besides the health and validations, Kiali shows also the mTLS status at namespace-wide. Similar to the masthead, it shows a lock when strict mTLS is enabled or a hollow lock when permissive.

Overview: Namespace mTLS

Graph

The mTLS method is used to establish communication between microservices. In the graph, Kiali has the option to show which edges are using mTLS and with what percentatge during the selected period. When an edge shows a lock icon it means at least one request with mTLS enabled is present. In case there are both mTLS and non-mTLS requests, the side-panel will show the percentage of requests using mTLS.

Enable the option in the Display dropdown, select the security badge.

Graph: Edge mTLS

Validations

Kiali has different validations to help troubleshoot configurations related to mTLS such as DestinationRules and PeerAuthentications.

Validation supporting mTLS configuration

3.10 - Topology

How Kiali visualizes the mesh topology.

Kiali offers multiple ways for users to examine their mesh Topology. Each combines several information types to help users quickly evaluate the health of their service architecture.

Overview

Kiali’s default page is the topology Overview. It presents a high-level view of the namespaces accessible to Kiali, for this user. It combines service and application information, along with telemetry, validations and health, to provide a holistic summary of system behvior. The Overview page provides numerous filtering, sorting and presentation options. From here users can perform namespace-level Actions, or quickly navigate to more detailed views.

Topology namespace overview

Graph

The Kiali Graph offers a powerful visualization of your mesh traffic. The topology combines real-time request traffic with your Istio configuration information to present immediate insight into the behavior of service mesh, allowing you to quickly pinpoint issues. Multiple Graph Types allow you to visualize traffic as a high-level service topology, a low-level workload topology, or as an application-level topology.

Graph nodes are decorated with a variety of information, pointing out various route routing options like virtual services and service entries, as well as special configuration like fault-injection and circuit breakers. It can identify mTLS issues, latency issues, error traffic and more. The Graph is highly configurable, can show traffic animation, and has powerful Find and Hide abilities.

You can configure the graph to show the namespaces and data that are important to you, and display it in the way that best meets your needs.

Topology graph

Health

Colors in the graph represent the health of your service mesh. A node colored red or orange might need attention. The color of an edge between components represents the health of the requests between those components. The node shape indicates the type of component such as services, workloads, or apps.

Topology graph health

Side-Panel

The collapsible side-panel summarizes the current graph selection, or the graph as a whole. A single-click will select the node, edge, or box of interest. The side panel provides:

Charts showing traffic and response times.
Health details.
Links to fully-detailed pages.
Response Code and Host breakdowns.
Traces involving the selected component.

Topology graph side-panel app Topology graph side-panel service Topology graph side-panel workload

Node Detail

A single-click selects a graph node. A double-click drills in to show the node’s Detail Graph. The node detail graph visualizes traffic from the point-of-view of that node, meaning it shows only the traffic reported by that node’s Istio proxy.

You can return back to the main graph, or double-click to change to a different node’s detail graph.

Topology graph node detail

Traffic Animation

Kiali offers several display options for the graph, including traffic animation.

For HTTP traffic, circles represent successful requests while red diamonds represent errors. The more dense the circles and diamonds the higher the request rate. The faster the animation the faster the response times.

TCP traffic is represented by offset circles where the speed of the circles indicates the traffic speed.

Topology graph animation

Ranking

Nodes can be ranked in the graph based on pre-defined criteria such as ’number of inbound edges’. Combined with the find/hide feature, this allows you to quickly highlight or filter for important workloads, services, and applications.

Rankings are normalized to fit between 1..100 and nodes may tie with each other in rank. Ranking starts at 1 for the top ranked nodes so when ranking nodes based on ’number of inbound edges’, the node(s) with the most inbound edges would have rank 1. Node(s) with the second most inbound edges would have rank 2. Each selected criteria contributes equally to a node’s ranking. Although 100 rankings are possible, only the required number of rankings are assigned, starting at 1.

Topology graph ranking

Graph Types

Kiali offers four different traffic-graph renderings:

The workload graph provides the a detailed view of communication between workloads.
The app graph aggregates the workloads with the same app labeling, which provides a more logical view.
The versioned app graph aggregates by app, but breaks out the different versions providing traffic breakdowns that are version-specific.
The service graph provides a high-level view, which aggregates all traffic for defined services.

Topology graph type workload Topology graph type app Topology graph type versioned app Topology graph service

Replay

Graph replay allows you to replay traffic from a selected past time-period. This gives you a chance to thoroughly examine a time period of interest, or share it with a co-worker. The graph is fully bookmarkable, including replay.

Operation Nodes

Istio v1.6 introduced Request Classification. This powerful feature allows users to classify requests into aggregates, called “Operations” by convention, to better understand how a service is being used. If configured in Istio the Kiali graph can show these as Operation nodes. The user needs only to enable the “Operation Nodes” display option. Operations can span services, for example, “VIP” may be configured for both CarRental and HotelRental services. To see total “VIP” traffic then display operation nodes without service nodes. To see “VIP” traffic specific to each service then also enable the “Service Nodes” display option.

When selected, an Operation node also provides a side-panel view. And when double-clicked a node detail graph is also provided.

Because operation nodes represent aggregate traffic they are not compatible with Service graphs, which themselves are already logical aggregates. For similar reasons response time information is not available on edges leading into or out of operation nodes. But by selecting the edge the response time information is available in the side panel (if configured).

Operation nodes are represented as pentagons in the Kiali graph:

Topology graph operation

3.11 - Tracing

How Kiali integrates Distributed Tracing with Jaeger.

Kiali offers a native integration with two different Distributed Tracing platforms, Jaeger and Grafana Tempo. As such, users can access trace visualizations. But more than that, Kiali incorporates tracing into several correlated views, making your investment in trace data even more valuable.

For a quick glimpse at Kiali tracing features, see below. For a detailed explanation of tracing in Kiali, see this 3-part Trace my mesh blog series. There is a detailed guide to configure access to the different distributed tracing platform, Jaeger and Grafana Tempo.

Workload detail

When investigating a workload, click the Traces tab to visualize your traces in a chart. When selecting a trace Kiali presents a tab for trace detail, and a tab for span details. Kiali always tries to surface problem areas, Kiali uses a heatmap approach to help the user identify problem traces or spans.

Trace detail Span detail

Heatmaps

A heatmap that you see in the Workload’s Tracing tab is a matrix that compares a specific trace’s request duration against duration metrics aggregated over time.

Heatmap

Each trace has a corresponding heatmap matrix. Each cell in the matrix corresponds to a specific metric aggregate; the value and color of a cell represents the difference between that metric and the duration of the matrix’s associated trace.

For example, the top-right cell of the heatmap above shows that the duration of the request represented by the trace (5.96ms) was 17.5ms faster than the 99th percentile of all inbound requests to the workload within the last 10 minutes.

The color of a cell will range between red and green; the more green a cell is, the faster the duration of the trace was compared to the aggregate metric data. A red cell indicates the associated trace was much slower compared to the aggregate metric data and so examining that trace could help detect a potential bottleneck or problem.

Metric Correlation

Kiali offers span overlays on Metric charts. The user can simply enable the spans option to generate the overlay. Clicking any span will navigate back to the Traces tab, focused on the trace of interest.

Metrics with Tracing

Graph Correlation

Kiali users often use the Graph Feature to visualize their mesh traffic. In the side panel, When selecting a graph node, the user will be presented with a Traces tab, which lists available traces for the time period. When selecting a trace the graph will display an overlay for the trace’s spans. And the side panel will display span details and offer links back to the trace detail views.

Graph with Tracing

Logs Correlation

Kiali works to correlate the standard pillars of observability: traces, metrics and logs. Kiali can present a unified view of log and trace information, in this way letting users use logs to identify traces of interest. When enabling the spans option Kiali adds trace entries to the workload logs view. Below, in time-sorted order, the user is presented with a unified view of application logs (in white), Envoy proxy logs (in gold), and trace spaces (in blue). Clicking a span of interest brings you to the detail view for the trace of interest.

Logs with Tracing

3.12 - Validation

A description and complete list of Kiali validations.

Istio Config Validation

The complete list of validations:

AuthorizationPolicy

KIA0101 - Namespace not found for this rule

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request. For instance, all requests started from a workload on a list of namespaces. The present validation points out those rules referencing a namespace that don’t exist in the cluster.

Resolution

Either remove the namespace from the list, correct if there is any typo or create a new namespace.

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
         - non-existing # warning
         - unexisting # warning
   to:
   - operation:
       methods: ["GET"]
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0102 - Only HTTP methods and fully-qualified gRPC names are allowed

An AuthorizationPolicy has an Operation field where is defined the oprations allowed for a request. In the method field are listed all the allowed methods that request can have. This validation appears when a problem is found in there. The only methods accepted are: either HTTP valid methods or fully-qualified names of gRPC service in the form of “/package.service/method”

Resolution

Either change or remove the violating method. It has to be either a HTTP valid method or a fully-qualified names of a gRPC service in the form of “/package.service/method”

Severity

Warning

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       methods:
         - "GET"
         - "/package.service/method"
         - "WRONG" # Warning
         - "non-fully-qualified-grpc" # Warning
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0104 - This host has no matching entry in the service registry

AuthorizationPolicy enables access control on workloads. Each policy effects only to a group of request going to a specific destination. For instance, allow all the request going to details host.

The present validation points out those rules referencing a host that don’t exist in the authorization policy namespace. Kiali considers services, virtual services and service entries. Those hosts that refers to hosts outside of the object namespace will be presented with an unknown error.

Resolution

Either remove the host from the list, correct if there is any typo or deploy a new service, service entry or a virtual service pointing to that host.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: default
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       hosts:
         - wrong # Error
         - ratings
         - details.default
         - reviews.default.svc.cluster.local
         - productpage.outside # Unknown
         - google.com # Service Entry present. No error
         - google.org # Service Entry not present, wrong domain. Error.
       methods:
         - "GET"
         - "/package.service/method"
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when:
   - key: request.auth.claims[iss]
     values: ["https://accounts.google.com"]

KIA0105 - This field requires mTLS to be enabled

AuthorizationPolicy has a Source field which specifies the source identities of a request. The Source field accepts principals and namespaces which require mTLS be enabled.

A validation Error message on a principals or namespaces fields means that mTLS is not enabled.

This validation appears only when autoMtls is disabled.

Resolution

Either remove this field or enable autoMtls.

Severity

Error

Example

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: httpbin
 namespace: bookinfo
spec:
 selector:
   matchLabels:
     app: httpbin
     version: v1
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep"]
   - source:
       namespaces:
         - default
   to:
   - operation:
       methods: ["GET"]
       paths: ["/info*"]
   - operation:
       methods: ["POST"]
       paths: ["/data"]
   when