Sidekiq stats are collected by gitlab-monitor by talking to Redis, and scraped by Prometheus.
Table of Contents
[TOC] If you see no stats in the Sidekiq dashboard then something could be wrong with these three components.
- A warning message in prometheus-alerts
- Total Running Jobs or Running Jobs panels are showing flat lines
On any of the redis nodes redis0X.db.gitlab.com run:
/opt/gitlab/embedded/bin/redis-cli -p 26379
then type this in Redis console:
sentinel master gitlab-redis
you should see output like this:
1) "name"
2) "gitlab-redis"
3) "ip"
4) "10.66.2.103"
5) "port"
6) "6379"
the master is the node with private IP of 10.66.2.103, to get the actual node run the following on your machine:
knife ssh 'roles:gitlab-base-db-redis' "ifconfig | grep '10.66.2.103'"
the first column of the output is IP you should ssh to.
On the master node, run:
sudo sv status gitlab-monitor
which ideally should return something like this:
run: gitlab-monitor: (pid 1271) 19889s; run: log: (pid 1267) 19889s
if not, run:
sudo sv start gitlab-monitor
On the master node, run:
curl http://localhost:4567/sidekiq
it should return something like:
sidekiq_queue_size{name="system_hook"} 0
sidekiq_queue_size{name="update_merge_requests"} 0
sidekiq_queue_latency_seconds{name="admin_emails"} 0
sidekiq_queue_latency_seconds{name="archive_repo"} 0
<snip>
If it returned some Ruby errors, open an issue in gitlab-monitor project.
Login to the prometheus node, run:
less /opt/prometheus/prometheus/inventory/gitlab-monitor-redis.yml
it should have multiple entries for the redis nodes we have, make sure it got an entry
for with the private IP you obtained from step 1. If not, then make sure the prometheus-server Chef role
is configured to scrape both roles of gitlab-cluster-redis-master and gitlab-cluster-redis-slave (more on that in
gitlab-prometheus README), then run:
sudo chef-client