This section describes some known issues in Kiali that may not be fixed in the very short term, and lead to limitations in a restricted number of cases. Hopefully, you won’t even notice them.
If you are hitting a problem, whether it is listed here or not, do not hesitate to use the GitHub issue tracker. You can simply vote (using emojis) for any existing bug or feature request, this will help us to understand the most demanded enhancements, or you can create a new ticket. For existing JIRA tickets, use the vote link or add a comment describing your expectation or scenario.
The whole list of JIRA tickets can be found here.
In general, we use the GitHub tracker for issues raised by the community, and JIRA for issues raised by the core engineering team. Both are publicly accessible.
Why is the graph badly laid out?
If you search in the mailing list, you will find a number of discussions about the graph layouts.
The layout for Kiali Graph may render differently, depending on the data to display (number of graph nodes and their interactions) and it’s sometimes difficult, not to say impossible, to have a single layout that renders nicely in every situations. That’s why Kiali offers a choice of several layout algorithms. However, we may still find some scenarios where none of the proposed algorithm offer a satisfying display. If Kiali doesn’t not render your graph layout in a satisfactory manner please switch to another layout option. This can be done from the Graph Toolbar located on the bottom left of the graph.
Why are there many unknown nodes in the graph?
In some situations you can see a lot of connections from an "Unknown" node to your services in the graph, because some software external to your mesh might be periodically pinging or fetching data. This is typically the case when you setup Kubernetes liveness probes, or have some application metrics pushed or exposed to a monitoring system such as Prometheus. Perhaps you wouldn’t like to see these connections because they make the graph harder to read.
From the Graph page, you can filter them out by typing
node = unknown in the Graph Hide input box.
For a more definitive solution, there are several ways to prevent Istio from gathering this kind of telemetry.
The first is to have these endpoints (like
/metrics) exposed on a different port and server than your main application, and to not declare this port in your Pod's container definition as containerPort. This way, the requests will be completely ignored by the Istio proxy, as mentioned in Istio documentation (at the bottom of that page).
The second way is to modify Istio’s Prometheus rule to explicitly exclude some requests based on the User Agent. This is the
Rule resource named
promhttp located in
istio-system. To edit it:
kubectl edit rule promhttp -n istio-system
or for OpenShift:
oc edit rule promhttp -n istio-system
Then locate the
match field under
spec section. Change it to filter out, for instance, the Kubernetes probes:
match: (context.protocol == "http" || context.protocol == "grpc") && (match((request.useragent | "-"), "kube-probe*") == false)
Note that, starting from Istio 1.1, Kubernetes probes will be filtered out in that way by default. But you may still want to do something equivalent with other User Agents.
Why do I have missing edges?
Kiali just shows the telemetry in Prometheus and if you don’t see what you expect it probably means that it has not been reported in Prometheus. Which means ususally that 1) the requests are not actually sent, 2) sidecars are missing, 3) requests are leaving the mesh and are not configured for telemetry
Kiali builds the graph from Istio’s telemetry. If you don’t see what you expect it probably means that it has not been reported in Prometheus. This usually means that:
1- The requests are not actually sent.
2- Sidecars are missing.
For example, If you don’t see traffic going from node A to node B, but you are sure there is traffic, the first thing you should be doing is checking the telemetry by querying the metrics, for example, if you know that MyWorkload-v1 is sending requests to ServiceA try looking for metrics of the type:
If telemetry is missing then it may be better to take it up with Istio
Why can't I see traffic leaving the mesh?
See Why do I have missing edges?, and additionally consider:
3- Requests are not configured for telemetry.
You need to create a ServiceEntry (or several) to allow the requests to be mapped correctly.
You can check this article on how to visualize your external traffic in Kiali for more information.
Why my external HTTPS traffic is showing as TCP?
Istio can’t recognize HTTPS request that go directly to the service, the reason is that these requests are encrypted and are recognized as TCP traffic.
You can however configure your mesh to use TLS origination for your egress traffic. This will allow to see your traffic as HTTP instead of TCP.
Why does my workload graph show service nodes?
Display Service Nodes is disabled a workload graph can show service nodes.
Display Service Nodes ensures that you will see the service nodes between two other nodes, providing an edge to the destination service node, and a subsequent edge to the node handling the request. This option injects service nodes where they previously would not be shown. But Kiali will always show a terminal service node when the request itself fails to be routed to a destination workload. This ensures the graph visualizes problem areas. This can happen in a workload or app graph. Of course in a service graph the
Display Service Nodes option is simply ignored.
Why do I see traffic to PassthroughCluster?
Requests going to PassthroughCluster (or BlackHoleCluster) are requests that did not get routed to a defined service or service entry, and instead end up at one of these built-in Istio request handlers. See Monitoring Blocked and Passthrough External Service Traffic for more information.
Unexpected routing to these nodes does not indicate a Kiali problem, you’re seeing the actual routing being performed by Istio. In general it is due to a misconfiguration and/or missing Istio sidecar. Less often but possible is an actual issue with the mesh, like a sync issue or evicted pod.
Use Kiali’s Workloads list view to ensure sidecars are not missing. Use Kiali’s Istio Config list view to look for any config validation errors.
Why don't I see response times on my service graph?
Users can select Response Time to label their edges with 95th percentile response times. The response time indicates the amount of time it took for the destination workload to handle the request. In the Kiali graph the edges leading to service nodes represent the request itself, in other words, the routing. Kiali can show the request rate for a service but response time is not applicable to be shown on the incoming edge. Only edges to app, workload, or service entry nodes show response time because only those nodes represent the actual work done to handle the request. This is why a Service graph will typically not show any response time information, even when the Response Time option is selected.
Because Service graphs can show Service Entry nodes the Response Time option is still a valid choice. Edges to Service Entry nodes represent externally handled requests, which do report the response time for the external handling.