--- layout: handbook-page-toc title: Using Kibana category: Infrastructure for troubleshooting --- ## On this page {:.no_toc .hidden-md .hidden-lg} - TOC {:toc .hidden-md .hidden-lg} ### Overview This document provides information on what Kibana is, how to search it, interpret its results, and contains tips and tricks on getting specific information from it. ### Using Kibana [Kibana](https://www.elastic.co/kibana) is an open source data visualization plugin for [Elasticsearch](https://en.wikipedia.org/wiki/Elasticsearch). It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Support Engineering uses Kibana to both search for error events on GitLab.com and to detect when specific changes were made to various aspects of it by a user. #### Parameters Knowing *where* to search in Kibana is paramount to getting the proper results. The area of the application (GitLab.com) that you're searching is known as the `Index` in Kibana. The default index used for searching is `pubsub-rails-inf-gprd-*` but you can change this by clicking the `(change)` button: ![Changing search index](/images/support/kibana_index-selection.jpg) For example, if you're trying to track down failed logins you would search the index `pubsub-application-inf-gprd*`. To search for [`500` errors](https://about.gitlab.com/handbook/support/workflows/500_errors.html) involving a controller you'd search in `pubsub-rails-inf-gprd-*`, the default index. Along with the index, knowing *when* a specific error or event ocurred that you're trying to track down is important and it's important to keep in mind that Kibana logs on GitLab.com persist for seven days. Kibana allows you to choose relative and absolute time ranges for search results and this can be changed by manipulating the date range: ![Changing the date range](/images/support/kibana_changing-dates.jpg) #### Fields and Filters Each log entry is comprised of a number of `Fields` in which the specific information about the entry is displayed. Knowing which fields to apply to your search results and how to filter for them is just as important as knowing *where* and *when* to search. The most important fields are: - `json.method` - `json.action` - `json.controller` - for more on this see the [controller definitions in the GitLab sourcecode](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/controllers/projects) - `json.status` - `json.path` All available fields are displayed along the left-hand side menu and you can add them to your search results by hovering over each and clicking the `add` button. ![Viewing the list of fields](/images/support/kibana_available-fields.jpg) If you don't filter for specific fields it can be difficult to find specific log entries if a large number of them are returned on your search query. For example, we're trying to locate any log events generated by the GitLab.com user `tristan` that returned a `404` status code within the last 15 minutes. We can start by searching the `pubsub-rails-inf-gprd-*` index for `json.username : tristan` within that time range and we'd get results similar to the following once we click `add` next to the `json.status` field along the left-hand side bar: ![Unfiltered results for a specific username](/images/support/kibana_unfiltered-results.jpg) The majority of results as entries that returned `200`, which aren't in the scope of what we're looking for. To search for results where `404` was returned we can click `+ Add Filter` underneath the search bar and place a positive filter on `json.status is 404`, which will give us these results, exactly what we're looking for. ![Filtered results for a specific username](/images/support/kibana_filtered-results.jpg) ### Tips and Tricks This section details how you can find very specific pieces of information in Kibana by searching for and filtering out specific fields. Each tip includes a link to the group, subgroup, or project that was used in the example for reference. #### Runner Registration Token Reset Example group: [gitlab-bronze](https://gitlab.com/gitlab-bronze) We can determine if the GitLab Runner registration token was reset for a group or project and see which user reset it and when. For this example the Runner registration token was reset at the group-level in `gitlab-bronze`. To find the log entry: 1. Set the date range to a value that you believe will contain the result. Set it to `Last 7 days` if you're unsure. 1. Add a positive filter on `json.path` for the path of the group, which is just `gitlab-bronze` in this example. 1. Add a positive filter on `json.action` for `reset_registration_token`. 1. Observe the results. If there were any they will contain the username of the user that triggered the reset in the `json.username` field of the result. #### Deleted Group/Subgroup/Project - Example group: [gitlab-silver](https://gitlab.com/gitlab-silver/) - Example project: [gitlab-silver/test-project-to-delete](https://gitlab.com/gitlab-silver/test-project-to-delete) Kibana can be used to determine who triggered the deletion of a group, subgroup, or project on GitLab.com. To find the log entry: 1. Set the date range to a value that you believe will contain the result. Set it to `Last 7 days` if you're unsure. 1. Add a positive filter on `json.path` for the path of the project, including the group and subgroup, if applicable. This is `gitlab-silver/test-project-to-delete` in this example. 1. Add a positive filter on `json.method` for `DELETE`. 1. Observe the results. If there were any they will contain the username of the user that triggered the deletion in the `json.username` field of the result.