Nines
Skip to content

Outage playbook

What to Do When GitLab Is Down

Keep development moving during a GitLab outage, handle CI runners and registries, and resume pipelines without repeating releases or deployments.

Reviewed

When GitLab fails, first establish whether the problem is on GitLab.com or an installation your organization operates. Then pause repeated pipeline attempts, preserve local changes and identify which part of the release process is blocked.

Check the official GitLab System Status page for GitLab.com incidents. For another installation, contact its service owner through your internal incident process. If that boundary is unclear, use Is GitLab down? before changing runners, credentials or project settings.

Stop retries and record unfinished operations

Save editor buffers and inspect the working tree. Keep the failing command, error, timestamp and pipeline or job identifier. Write down any push, package publication or deployment whose result is uncertain.

Avoid retrying an entire pipeline just because its interface stopped updating. A job may already have executed an external action. Check the destination system when available, and keep release ownership with one person until the state is clear.

Pause noisy integrations narrowly. Record which scheduled tasks or automatic retries were paused and how to restore them. Do not mix an outage response with unrelated changes to branch protection, authentication or runner configuration.

Confirm the affected component and tell the team

Match the error with Git operations, website, API, CI/CD or a registry. Ask one teammate to reproduce the same action if there is no matching notice. Use their own account and avoid sharing tokens as a troubleshooting shortcut.

Send a short operational update:

GitLab at [hostname] is affecting [operation]. We observed [error] at [time and timezone]. [Provider/internal owner] is investigating, or confirmation is pending. Hold [pipeline/release action]. Continue [local work]. [Owner] will post another update at [time].

For self-managed services, include the internal incident reference. For GitLab.com, link the provider incident when it exists. Social reports and Downdetector can support a suspected wider issue, but do not replace either source with an unsupported statement that all GitLab installations are down.

Keep development moving locally

Use existing clones for code changes, local commits and tests that have their dependencies available. Review a diff or write a reproducible failing case. Keep notes on checks that depend on unavailable services so the next reviewer knows what remains unverified.

Coordinate file ownership through a working communication channel. If you need to share a patch, use an already approved private route and include its base commit. A hurried mirror is not automatically a safe or equivalent replacement for your normal repository.

Prepare merge request descriptions, test instructions and release notes locally. These are useful tasks during a website outage and reduce the rush when it returns. Fetch and compare remote history before pushing accumulated changes.

Separate runner problems from CI coordination

Identify whether a job is waiting to start or has already begun executing. For a waiting job, check its requirements, approvals and runner availability. For a running job, preserve logs and inspect the last completed step.

GitLab documents runners as the systems that execute CI/CD jobs. An existing self-managed runner still needs the relevant GitLab services for normal job coordination. A healthy machine alone does not guarantee that queued jobs can be assigned or report results.

If only your runner is broken, involve its owner and use an existing suitable runner if your team has one. If coordination is affected, moving machines may not help. Run available checks locally, but do not represent local results as completed required CI checks or remove protections to force a merge.

Handle registry and API failures carefully

Keep installed dependencies, lockfiles and cached build artifacts intact. Inspect the failed hostname before changing anything. A pipeline downloading from npm needs the npm outage playbook, even though the surrounding job appears in GitLab.

For container and package publishing, record the intended version, tag and artifact identity. After connectivity returns, inspect the registry to establish whether publication succeeded. Avoid using a new tag merely to escape uncertainty about the first attempt.

For API integrations, retain error responses and request identifiers, reduce unnecessary traffic and bound retries. Check external results before repeating writes. If a bot creates issues, releases or deployment records, a lost response does not prove those objects were never created.

Recover from the destination backward

Start with one low-impact check, then reconcile unfinished operations. Do not release all queued work just because the status headline changes. The status-page guide explains why your workflow can lag behind a provider update.

  1. Confirm the affected hostname and component respond to a normal read.
  2. Fetch and compare repository history before pushing local work.
  3. Inspect running and queued pipelines, including old attempts.
  4. Check deployments, migrations and registry contents at their destination.
  5. Retry only missing or failed work after reviewing possible duplicate effects.
  6. Resume paused integrations and schedules gradually.
  7. Publish a team update with verified recovery and remaining cleanup.

Leave the required reviews and release checks in place. If an urgent release followed an existing emergency procedure, record what happened and complete its follow-up checks. Keep the outage note focused on decisions that will help the next responder.

An optional way to spend the wait

Once someone owns recovery and the useful work is covered, join the Nines ping list. Nines is a tiny browser arcade that opens for qualifying official developer-tool outages. Degraded banners alone do not unlock it.