Skip to content

Latest commit

 

History

History
579 lines (519 loc) · 38.8 KB

prom-metrics.md

File metadata and controls

579 lines (519 loc) · 38.8 KB

Prometheus metrics

This guide covers the performance metrics availabe through Prometheus and assumes Prometheus in Sourcegraph Data Center has already been set up. For a Prometheus overview and setup guide, see the Data Center admin guide.

Recommended metrics

We recommend tracking the following key performance metrics through Prometheus. Typically this is done by creating dashboards of related metrics, so the metrics are grouped into logical dashboards.

If you're using Grafana for dashboards and have connected Sourcegraph's Prometheus instance to it, you can import these dashboards directly from JSON.

Dashboard: frontend

  1. frontend QPS, by HTTP code (5m avg):
sum(label_replace(job:src_http_request_count:rate5m, \"c\", \"00s\", \"code\", \"([0-9]).*\")) by (c)
  1. frontend QPS, by route (5m avg):
sum(route:src_http_request_count:rate5m{route=~\"blob|graphql|home|page.def.landing|page.repo.landing|repo|repo-branches|search|settings|sign-in|site-admin|tree|xlang\"}) by (route)
  1. frontend 90th-percentile request duration (5m avg):
histogram_quantile(0.90, route:src_http_request_duration_seconds_bucket:rate5m{route=~\"blob|graphql|home|page.def.landing|page.repo.landing|repo|repo-branches|search|settings|sign-in|site-admin|tree|xlang\"})
  1. frontend 75th-percentile request duration (5m avg):
histogram_quantile(0.75, route:src_http_request_duration_seconds_bucket:rate5m{route=~\"blob|graphql|home|page.def.landing|page.repo.landing|repo|repo-branches|search|settings|sign-in|site-admin|tree|xlang\"})
  1. frontend CPU usage, by pod (%)
label_replace( irate(process_cpu_seconds_total{app=\"sourcegraph-frontend\"}[10m]), \"inst\", \"\", \"instance\", \"[a-z0-9\\-]+\\-([a-z0-9]+)\" ) * 100
  1. frontend resident memory, by pod (GB)
label_replace( process_resident_memory_bytes{app=\"sourcegraph-frontend\"}, \"inst\", \"\", \"instance\", \"[a-z0-9\\-]+\\-([a-z0-9]+)\" ) / 1024 / 1024 / 1024
  1. frontend virtual memory, by pod (GB)
label_replace( process_virtual_memory_bytes{app=\"sourcegraph-frontend\"}, \"inst\", \"\", \"instance\", \"[a-z0-9\\-]+\\-([a-z0-9]+)\" ) / 1024 / 1024 / 1024

Dashboard: gitserver

  1. gitserver total QPS (10m avg):
sum(rate(src_gitserver_exec_duration_seconds_count{status=~\"[0-9]+\"}[10m])) by (job)"
  1. gitserver QPS, by operation (10m avg):
sum(rate(src_gitserver_exec_duration_seconds_count{status=~\"[0-9]+\"}[10m])) by (cmd, job)"
  1. gitserver p90 request duration, by operation (10m avg):
histogram_quantile(0.9, sum(rate(src_gitserver_exec_duration_seconds_bucket{status=~\"[0-9]+\"}[10m])) by (le, cmd, job))"
  1. gitserver p75 request duration, by operation (10m avg):
histogram_quantile(0.75, sum(rate(src_gitserver_exec_duration_seconds_bucket{status=~\"[0-9]+\"}[10m])) by (le, cmd, job))"
  1. git execs running
sum by (job)(src_gitserver_exec_running)"
  1. exit status QPS, by operation (10m avg):
sum by (status, cmd)(rate(src_gitserver_exec_duration_seconds_count{status!=\"\"}[10m]))"
  1. free disk space (TB):
min(src_gitserver_disk_space_available) BY (job) / 1000 / 1000 / 1000"

Dashboard: lang

  1. open workspaces, by language:
avg by (mode)(src_xlang_open_lsp_server_connections{mode!=\"\"})
  1. QPS, by language (5m avg):
prod:xlang_requests:rate5m
  1. request error ratio, by language:
prod:xlang_errors:ratio5m
  1. restarts, by service:
sum by (container)(increase(kube_pod_container_status_restarts{container=~\"(lsp|xlang).*\"}[10m]))
  1. max CPU, by service:
max by (container_name)(task:container_cpu_usage_seconds_total:sum{container_name=~\"(lsp|xlang).*\"})
  1. max RAM, by service:
max by (container_name)(container_memory_rss{container_name=~\"(xlang|lsp).*\"}) / 1024 / 1024 / 1024

Dashboard: resources

  1. max CPU, by service
(max by (job)(irate(process_cpu_seconds_total[5m]))) * 100
  1. max resident memory, by service (GB):
(max by (job)(process_resident_memory_bytes)) / 1024 / 1024 / 1024
  1. max virtual memory, by service (GB):
max by (job)(process_virtual_memory_bytes) / 1024 / 1024 / 1024

All metrics

This is a list of all metrics available through Prometheus. Though most critical information is covered by the recommended dashboards above, these additional metrics can be useful for creating custom dashboards or investigating specific issues.

<style> span.gray { color: gray; } </style>
  • frontend_graphql_parse_fails : counter : sourcegraph-frontend
    Total number of times a request fails query parsing.
  • frontend_graphql_strict_validation_fails : counter : sourcegraph-frontend
    Total number of times a request fails only in the new validation.
  • frontend_graphql_validation_fails : counter : sourcegraph-frontend
    Total number of times a request fails validation.
  • go_gc_duration_seconds : summary : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    A summary of the GC invocation durations.
  • go_goroutines : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of goroutines that currently exist.
  • go_memstats_alloc_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes allocated and still in use.
  • go_memstats_alloc_bytes_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total number of bytes allocated, even if freed.
  • go_memstats_buck_hash_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes used by the profiling bucket hash table.
  • go_memstats_frees_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total number of frees.
  • go_memstats_gc_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes used for garbage collection system metadata.
  • go_memstats_heap_alloc_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of heap bytes allocated and still in use.
  • go_memstats_heap_idle_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of heap bytes waiting to be used.
  • go_memstats_heap_inuse_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of heap bytes that are in use.
  • go_memstats_heap_objects : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of allocated objects.
  • go_memstats_heap_released_bytes_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total number of heap bytes released to OS.
  • go_memstats_heap_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of heap bytes obtained from system.
  • go_memstats_last_gc_time_seconds : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of seconds since 1970 of last garbage collection.
  • go_memstats_lookups_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total number of pointer lookups.
  • go_memstats_mallocs_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total number of mallocs.
  • go_memstats_mcache_inuse_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes in use by mcache structures.
  • go_memstats_mcache_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes used for mcache structures obtained from system.
  • go_memstats_mspan_inuse_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes in use by mspan structures.
  • go_memstats_mspan_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes used for mspan structures obtained from system.
  • go_memstats_next_gc_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of heap bytes when next garbage collection will take place.
  • go_memstats_other_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes used for other system allocations.
  • go_memstats_stack_inuse_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes in use by the stack allocator.
  • go_memstats_stack_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes obtained from system for stack allocator.
  • go_memstats_sys_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of bytes obtained by system. Sum of all system allocations.
  • golangserver_build_open_connections : gauge : xlang-go
    Number of open connections to the langserver.
  • golangserver_build_path_has_git_suffix : counter : sourcegraph-frontend, xlang-go
    Temporary counter to determine if paths have a git suffix.
  • golangserver_symbol_cache_size : gauge : sourcegraph-frontend, xlang-go
    Number of items in the symbol cache
  • golangserver_typecheck_cache_request_total : counter : xlang-go
    Count of requests to cache.
  • golangserver_typecheck_cache_size : gauge : sourcegraph-frontend, xlang-go
    Number of items in the typecheck cache
  • http_request_duration_microseconds : summary : github-proxy, pgsql, redis-cache, redis-store
    The HTTP request latencies in microseconds.
  • http_request_size_bytes : summary : github-proxy, pgsql, redis-cache, redis-store
    The HTTP request sizes in bytes.
  • http_requests_total : counter : github-proxy, pgsql, redis-cache, redis-store
    Total number of HTTP requests made.
  • http_response_size_bytes : summary : github-proxy, pgsql, redis-cache, redis-store
    The HTTP response sizes in bytes.
  • pg_exporter_last_scrape_duration_seconds : gauge : pgsql
    Duration of the last scrape of metrics from PostgresSQL.
  • pg_exporter_last_scrape_error : gauge : pgsql
    Whether the last scrape of metrics from PostgreSQL resulted in an error (1 for error, 0 for success).
  • pg_exporter_scrapes_total : counter : pgsql
    Total number of times PostgresSQL was scraped for metrics.
  • pg_locks_count : gauge : pgsql
    Number of locks
  • pg_runtime_variable_max_connections : gauge : pgsql
    Sets the maximum number of concurrent connections.
  • pg_runtime_variable_max_files_per_process : gauge : pgsql
    Sets the maximum number of simultaneously open files for each server process.
  • pg_runtime_variable_max_function_args : gauge : pgsql
    Shows the maximum number of function arguments.
  • pg_runtime_variable_max_identifier_length : gauge : pgsql
    Shows the maximum identifier length.
  • pg_runtime_variable_max_index_keys : gauge : pgsql
    Shows the maximum number of index keys.
  • pg_runtime_variable_max_locks_per_transaction : gauge : pgsql
    Sets the maximum number of locks per transaction.
  • pg_runtime_variable_max_pred_locks_per_transaction : gauge : pgsql
    Sets the maximum number of predicate locks per transaction.
  • pg_runtime_variable_max_prepared_transactions : gauge : pgsql
    Sets the maximum number of simultaneously prepared transactions.
  • pg_runtime_variable_max_standby_archive_delay_milliseconds : gauge : pgsql
    Sets the maximum delay before canceling queries when a hot standby server is processing archived WAL data.
  • pg_runtime_variable_max_standby_streaming_delay_milliseconds : gauge : pgsql
    Sets the maximum delay before canceling queries when a hot standby server is processing streamed WAL data.
  • pg_runtime_variable_max_wal_senders : gauge : pgsql
    Sets the maximum number of simultaneously running WAL sender processes.
  • pg_stat_activity_count : gauge : pgsql
    number of connections in this state
  • pg_stat_activity_max_tx_duration : gauge : pgsql
    max duration in seconds any active transaction has been running
  • pg_stat_bgwriter_buffers_alloc : counter : pgsql
    Number of buffers allocated
  • pg_stat_bgwriter_buffers_backend : counter : pgsql
    Number of buffers written directly by a backend
  • pg_stat_bgwriter_buffers_backend_fsync : counter : pgsql
    Number of times a backend had to execute its own fsync call (normally the background writer handles those even when the backend does its own write)
  • pg_stat_bgwriter_buffers_checkpoint : counter : pgsql
    Number of buffers written during checkpoints
  • pg_stat_bgwriter_buffers_clean : counter : pgsql
    Number of buffers written by the background writer
  • pg_stat_bgwriter_checkpoint_sync_time : counter : pgsql
    Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in milliseconds
  • pg_stat_bgwriter_checkpoint_write_time : counter : pgsql
    Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in milliseconds
  • pg_stat_bgwriter_checkpoints_req : counter : pgsql
    Number of requested checkpoints that have been performed
  • pg_stat_bgwriter_checkpoints_timed : counter : pgsql
    Number of scheduled checkpoints that have been performed
  • pg_stat_bgwriter_maxwritten_clean : counter : pgsql
    Number of times the background writer stopped a cleaning scan because it had written too many buffers
  • pg_stat_bgwriter_stats_reset : counter : pgsql
    Time at which these statistics were last reset
  • pg_stat_database_blk_read_time : counter : pgsql
    Time spent reading data file blocks by backends in this database, in milliseconds
  • pg_stat_database_blk_write_time : counter : pgsql
    Time spent writing data file blocks by backends in this database, in milliseconds
  • pg_stat_database_blks_hit : counter : pgsql
    Number of times disk blocks were found already in the buffer cache, so that a read was not necessary (this only includes hits in the PostgreSQL buffer cache, not the operating system's file system cache)
  • pg_stat_database_blks_read : counter : pgsql
    Number of disk blocks read in this database
  • pg_stat_database_conflicts : counter : pgsql
    Number of queries canceled due to conflicts with recovery in this database. (Conflicts occur only on standby servers; see pg_stat_database_conflicts for details.)
  • pg_stat_database_conflicts_confl_bufferpin : counter : pgsql
    Number of queries in this database that have been canceled due to pinned buffers
  • pg_stat_database_conflicts_confl_deadlock : counter : pgsql
    Number of queries in this database that have been canceled due to deadlocks
  • pg_stat_database_conflicts_confl_lock : counter : pgsql
    Number of queries in this database that have been canceled due to lock timeouts
  • pg_stat_database_conflicts_confl_snapshot : counter : pgsql
    Number of queries in this database that have been canceled due to old snapshots
  • pg_stat_database_conflicts_confl_tablespace : counter : pgsql
    Number of queries in this database that have been canceled due to dropped tablespaces
  • pg_stat_database_deadlocks : counter : pgsql
    Number of deadlocks detected in this database
  • pg_stat_database_numbackends : gauge : pgsql
    Number of backends currently connected to this database. This is the only column in this view that returns a value reflecting current state; all other columns return the accumulated values since the last reset.
  • pg_stat_database_stats_reset : counter : pgsql
    Time at which these statistics were last reset
  • pg_stat_database_temp_bytes : counter : pgsql
    Total amount of data written to temporary files by queries in this database. All temporary files are counted, regardless of why the temporary file was created, and regardless of the log_temp_files setting.
  • pg_stat_database_temp_files : counter : pgsql
    Number of temporary files created by queries in this database. All temporary files are counted, regardless of why the temporary file was created (e.g., sorting or hashing), and regardless of the log_temp_files setting.
  • pg_stat_database_tup_deleted : counter : pgsql
    Number of rows deleted by queries in this database
  • pg_stat_database_tup_fetched : counter : pgsql
    Number of rows fetched by queries in this database
  • pg_stat_database_tup_inserted : counter : pgsql
    Number of rows inserted by queries in this database
  • pg_stat_database_tup_returned : counter : pgsql
    Number of rows returned by queries in this database
  • pg_stat_database_tup_updated : counter : pgsql
    Number of rows updated by queries in this database
  • pg_stat_database_xact_commit : counter : pgsql
    Number of transactions in this database that have been committed
  • pg_stat_database_xact_rollback : counter : pgsql
    Number of transactions in this database that have been rolled back
  • process_cpu_seconds_total : counter : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Total user and system CPU time spent in seconds.
  • process_max_fds : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Maximum number of open file descriptors.
  • process_open_fds : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Number of open file descriptors.
  • process_resident_memory_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Resident memory size in bytes.
  • process_start_time_seconds : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Start time of the process since unix epoch in seconds.
  • process_virtual_memory_bytes : gauge : github-proxy, gitserver-1, indexer, lsp-proxy, pgsql, query-runner, redis-cache, redis-store, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Virtual memory size in bytes.
  • redis_aof_current_rewrite_duration_sec : gauge : redis-cache, redis-store
  • redis_aof_enabled : gauge : redis-cache, redis-store
  • redis_aof_last_rewrite_duration_sec : gauge : redis-cache, redis-store
  • redis_aof_rewrite_in_progress : gauge : redis-cache, redis-store
  • redis_aof_rewrite_scheduled : gauge : redis-cache, redis-store
  • redis_blocked_clients : gauge : redis-cache, redis-store
  • redis_client_longest_output_list : gauge : redis-cache, redis-store
  • redis_cluster_enabled : gauge : redis-cache, redis-store
  • redis_command_call_duration_seconds_count : gauge : redis-cache, redis-store
    Total number of calls per command
  • redis_command_call_duration_seconds_sum : gauge : redis-cache, redis-store
    Total amount of time in seconds spent per command
  • redis_commands_processed_total : gauge : redis-cache, redis-store
  • redis_connected_clients : gauge : redis-cache, redis-store
  • redis_connected_slaves : gauge : redis-cache, redis-store
  • redis_connections_received_total : gauge : redis-cache, redis-store
  • redis_db_avg_ttl_seconds : gauge : redis-cache, redis-store
    Avg TTL in seconds
  • redis_db_keys : gauge : redis-cache, redis-store
    Total number of keys by DB
  • redis_db_keys_expiring : gauge : redis-cache, redis-store
    Total number of expiring keys by DB
  • redis_evicted_keys_total : gauge : redis-cache, redis-store
  • redis_expired_keys_total : gauge : redis-cache, redis-store
  • redis_exporter_build_info : gauge : redis-cache, redis-store
    redis exporter build_info
  • redis_exporter_last_scrape_duration_seconds : gauge : redis-cache, redis-store
    The last scrape duration.
  • redis_exporter_last_scrape_error : gauge : redis-cache, redis-store
    The last scrape error status.
  • redis_exporter_scrapes_total : counter : redis-cache, redis-store
    Current total redis scrapes.
  • redis_instance_info : gauge : redis-cache, redis-store
    Information about the Redis instance
  • redis_instantaneous_ops_per_sec : gauge : redis-cache, redis-store
  • redis_keyspace_hits_total : gauge : redis-cache, redis-store
  • redis_keyspace_misses_total : gauge : redis-cache, redis-store
  • redis_latest_fork_usec : gauge : redis-cache, redis-store
  • redis_loading_dump_file : gauge : redis-cache, redis-store
  • redis_master_repl_offset : gauge : redis-cache, redis-store
  • redis_memory_fragmentation_ratio : gauge : redis-cache, redis-store
  • redis_memory_max_bytes : gauge : redis-cache, redis-store
  • redis_memory_used_bytes : gauge : redis-cache, redis-store
  • redis_memory_used_lua_bytes : gauge : redis-cache, redis-store
  • redis_memory_used_peak_bytes : gauge : redis-cache, redis-store
  • redis_memory_used_rss_bytes : gauge : redis-cache, redis-store
  • redis_net_input_bytes_total : gauge : redis-cache, redis-store
  • redis_net_output_bytes_total : gauge : redis-cache, redis-store
  • redis_process_id : gauge : redis-cache, redis-store
  • redis_pubsub_channels : gauge : redis-cache, redis-store
  • redis_pubsub_patterns : gauge : redis-cache, redis-store
  • redis_rdb_changes_since_last_save : gauge : redis-cache, redis-store
  • redis_rdb_current_bgsave_duration_sec : gauge : redis-cache, redis-store
  • redis_rdb_last_bgsave_duration_sec : gauge : redis-cache, redis-store
  • redis_rejected_connections_total : gauge : redis-cache, redis-store
  • redis_replication_backlog_bytes : gauge : redis-cache, redis-store
  • redis_up : gauge : redis-cache, redis-store
  • redis_uptime_in_seconds : gauge : redis-cache, redis-store
  • redis_used_cpu_sys : gauge : redis-cache, redis-store
  • redis_used_cpu_sys_children : gauge : redis-cache, redis-store
  • redis_used_cpu_user : gauge : redis-cache, redis-store
  • redis_used_cpu_user_children : gauge : redis-cache, redis-store
  • searcher_service_request_total : counter : searcher
    Number of returned search requests.
  • searcher_service_running : gauge : searcher
    Number of running search requests.
  • searcher_store_cache_size_bytes : gauge : searcher
    The total size of items in the on disk cache.
  • searcher_store_evictions : counter : searcher
    The total number of items evicted from the cache.
  • searcher_store_fetch_failed : counter : searcher
    The total number of archive fetches that failed.
  • searcher_store_fetch_queue_size : gauge : searcher
    The number of fetch jobs enqueued.
  • searcher_store_fetching : gauge : searcher
    The number of fetches currently running.
  • src_backend_client_request_duration_seconds : histogram : sourcegraph-frontend
    Total time spent on backend endpoints.
  • src_backend_client_requests : gauge : sourcegraph-frontend
    Current number of requests running for a method.
  • src_github_rate_limit_remaining : gauge : github-proxy
    Number of calls to GitHub's API remaining before hitting the rate limit.
  • src_github_requests_total : counter : repo-updater
    Total number of requests sent to the GitHub API.
  • src_gitserver_client_deadline_exceeded : counter : indexer, lsp-proxy, repo-updater, searcher, sourcegraph-frontend, symbols, xlang-go
    Times that Client.sendExec() returned context.DeadlineExceeded
  • src_gitserver_clone_queue : gauge : gitserver-1
    number of repos waiting to be cloned.
  • src_gitserver_disk_space_available : gauge : gitserver-1
    Amount of free space disk space on the repos mount.
  • src_gitserver_echo_duration_seconds : gauge : gitserver-1
    Duration of executing the echo command.
  • src_gitserver_exec_duration_seconds : histogram : gitserver-1
    gitserver.Command latencies in seconds.
  • src_gitserver_exec_running : gauge : gitserver-1
    number of gitserver.Command running concurrently.
  • src_gitserver_lsremote_queue : gauge : gitserver-1
    number of repos waiting to check existence on remote code host (git ls-remote).
  • src_gitserver_repos_recloned : counter : gitserver-1
    number of repos removed and recloned due to age
  • src_gitserver_repos_removed : counter : gitserver-1
    number of repos removed during cleanup due to inactivity
  • src_gitserver_update_queue : gauge : gitserver-1
    number of repos waiting to be updated (enqueue-repo-update)
  • src_graphql_field_seconds : histogram : sourcegraph-frontend
    GraphQL field resolver latencies in seconds.
  • src_graphql_search_results_stats_cache_hit : counter : sourcegraph-frontend
    Counts cache hits and misses for search results stats (e.g. sparklines).
  • src_http_request_duration_seconds : histogram : sourcegraph-frontend
    The HTTP request latencies in seconds.
  • src_http_requests_last_timestamp_unixtime : gauge : sourcegraph-frontend
    Last time a request finished for a http endpoint.
  • src_indexer_queue_length : gauge : indexer
    Lengh of the indexer's queue of repos to check/index.
  • src_legacy_def_landing_webapp : counter : sourcegraph-frontend
    Number of times a legacy def landing page has been served.
  • src_legacy_repo_landing_webapp : counter : sourcegraph-frontend
    Number of times a legacy repo landing page has been served.
  • src_pgsql_app_open_connections : gauge : sourcegraph-frontend
    Number of open connections to pgsql DB, as reported by pgsql.DB.Stats()
  • src_repos_github_cache_hit : counter : repo-updater
    Counts cache hits and misses for GitHub repo metadata.
  • src_repoupdater_time_last_bitbucketserver_sync : gauge : repo-updater
    The last time a comprehensive Bitbucket Server sync finished
  • src_repoupdater_time_last_github_sync : gauge : repo-updater
    The last time a comprehensive GitHub sync finished
  • src_repoupdater_time_last_gitlab_sync : gauge : repo-updater
    The last time a comprehensive GitLab sync finished
  • src_repoupdater_time_last_gitolite_sync : gauge : repo-updater
    The last time a comprehensive Gitolite sync finished
  • src_repoupdater_time_last_repolist_sync : gauge : repo-updater
    The time the last repository sync loop completed
  • src_updatecheck_requests : counter : sourcegraph-frontend
    Number of requests to the update check handler.
  • src_updatecheck_requests_has_update : counter : sourcegraph-frontend
    Number of requests to the update check handler where an update is available.
  • src_vfs_gitserver_bytes_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of bytes read into memory by ArchiveFileSystem.
  • src_xlang_client_rate_limited : counter : lsp-proxy, sourcegraph-frontend
    The number of times a client request was rate limited.
  • src_xlang_cumu_client_proxy_connections : counter : lsp-proxy, sourcegraph-frontend
    Cumulative number of connections to the xlang client proxy (total of open + previously closed since process startup).
  • src_xlang_cumu_lsp_server_connections : counter : lsp-proxy
    Cumulative number of connections (initialized + uninitialized) to the language servers (total of open + previously closed since process startup).
  • src_xlang_cumu_lsp_server_method_calls : counter : lsp-proxy
    Total number of calls sent for a (method, mode).
  • src_xlang_lsp_server_alive_seconds : histogram : lsp-proxy
    The number of seconds a proxied connection is kept alive.
  • src_xlang_lsp_server_failed_method_calls : histogram : lsp-proxy
    Total number of failed calls sent to a server proxy before it is shutdown.
  • src_xlang_lsp_server_method_calls : histogram : lsp-proxy
    Total number of calls sent to a server proxy before it is shutdown.
  • src_xlang_open_client_proxy_connections : gauge : lsp-proxy, sourcegraph-frontend
    Number of open connections to the xlang client proxy.
  • src_xlang_open_lsp_server_connections : gauge : lsp-proxy
    Open connections (initialized + uninitialized) to the language servers.
  • src_xlang_request_duration_seconds : histogram : sourcegraph-frontend
    The xlang request latencies in seconds.
  • symbols_parse_parse_failed : counter : symbols
    The total number of parse jobs that failed.
  • symbols_parse_parse_queue_size : gauge : symbols
    The number of parse jobs enqueued.
  • symbols_parse_parsing : gauge : symbols
    The number of parse jobs currently running.
  • symbols_store_cache_size_bytes : gauge : symbols
    The total size of items in the on disk cache.
  • symbols_store_evictions : counter : symbols
    The total number of items evicted from the cache.
  • symbols_store_fetch_failed : counter : symbols
    The total number of archive fetches that failed.
  • symbols_store_fetch_queue_size : gauge : symbols
    The number of fetch jobs enqueued.
  • symbols_store_fetching : gauge : symbols
    The number of fetches currently running.
  • xlang_cache_get_total : counter : lsp-proxy
    Total number of gets for a mode.
  • xlang_cache_get_total_bytes : counter : lsp-proxy
    Total number of bytes fetched from the cache for a mode.
  • xlang_cache_set_total : counter : lsp-proxy
    Total number of sets for a mode.
  • xlang_cache_set_total_bytes : counter : lsp-proxy
    Total number of bytes set to the cache for a mode.
  • xlang_vfs_cached_file_evict : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of evictions to cachedFetch archives.
  • xlang_vfs_github_fetch_failed_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of fetches by GitHubRepoVFS that failed.
  • xlang_vfs_github_fetch_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of fetches by GitHubRepoVFS.
  • xlang_vfs_gitserver_fetch_failed_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of fetches to GitServer that failed.
  • xlang_vfs_gitserver_fetch_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of fetches to GitServer.
  • xlang_vfs_remote_bytes_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of bytes cached into memory by RemoteProxyFS.
  • xlang_vfs_xremote_bytes_total : counter : lsp-proxy, sourcegraph-frontend, xlang-go
    Total number of bytes cached into memory by XRemoteFS.