Health check

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

GitLab provides liveness and readiness probes to indicate service health and reachability to required services. These probes report on the status of the database connection, Redis connection, and access to the file system. These endpoints can be provided to schedulers like Kubernetes to hold traffic until the system is ready or restart the container as needed.

Health check endpoints are normally used for load balancers and other Kubernetes scheduling systems that need to determine service availability before redirecting traffic.

You should not use these endpoints to determine effective uptime on large Kubernetes deployments. Doing so can show false negatives when pods are removed by autoscaling, node failure, or for other normal and otherwise non-disruptive operational needs.

To determine uptime on large Kubernetes deployments, look at traffic to the UI. This is properly balanced and scheduled, and therefore is a better indicator of effective uptime. You can also monitor the sign-in page /users/sign_in endpoint.

On GitLab.com, tools such as Pingdom and Apdex measurements are used to determine uptime.

IP allowlist

To access monitoring resources, the requesting client IP needs to be included in the allowlist. For details, see how to add IPs to the allowlist for the monitoring endpoints.

Using the endpoints locally

With default allowlist settings, the probes can be accessed from localhost using the following URLs:

GET http://localhost/-/health

GET http://localhost/-/readiness

GET http://localhost/-/liveness

Health

Checks whether the application server is running. It does not verify the database or other services are running. This endpoint circumvents Rails Controllers and is implemented as additional middleware BasicHealthCheck very early into the request processing lifecycle.

GET /-/health

Example request:

curl "https://gitlab.example.com/-/health"

Example response:

GitLab OK

Readiness

The readiness probe checks whether the GitLab instance is ready to accept traffic via Rails Controllers. The check by default does validate only instance-checks.

If the all=1 parameter is specified, the check also validates the dependent services (Database, Redis, Gitaly etc.) and gives a status for each.

GET /-/readiness
GET /-/readiness?all=1

Example request:

curl "https://gitlab.example.com/-/readiness"

Example response:

{
   "master_check":[{
      "status":"failed",
      "message": "unexpected Master check result: false"
   }],
   ...
}

On failure, the endpoint returns a 503 HTTP status code.

This check is being exempt from Rack Attack.

Liveness

WARNING: In GitLab 12.4 the response body of the Liveness check was changed to match the example below.

Checks whether the application server is running. This probe is used to know if Rails Controllers are not deadlocked due to a multi-threading.

GET /-/liveness

Example request:

curl "https://gitlab.example.com/-/liveness"

Example response:

On success, the endpoint returns a 200 HTTP status code, and a response like below.

{
   "status": "ok"
}

On failure, the endpoint returns a 503 HTTP status code.

This check is being exempt from Rack Attack.

Sidekiq

Learn how to configure the Sidekiq health checks.