GitHub's autoscaler watched the wrong meter, and auth, CI and Copilot fell together
The August 17 postmortem describes a sidecar that hit its concurrency limit while its autoscaler measured the host, then a VS Code retry bug that kept Copilot down hours after everything else returned.
The Engineer · Build desk
What happened
- On August 17 GitHub experienced an outage lasting 7 hours and 47 minutes that disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues and Copilot.
- In a postmortem published August 20th, GitHub CTO Vlad Fedorov said traffic reached a new peak while a critical infrastructure component in the Central US data center failed to scale with it.
- GitHub's root cause analysis traces the initial failure to an Istio service-mesh sidecar that reached its concurrency limit; its autoscaling policy monitored the host service rather than the sidecar's capacity, so new resources did not arrive as traffic increased.
- The bottleneck spread until four HAProxy nodes exhausted their flow limits, after which the gateway authentication path developed severe latency and failures affecting dependent services.
- At the peak GitHub measured error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads.
Why it matters
GitHub published the root cause analysis for its August 17 outage on August 20, and the mechanism matters more than the 7 hours and 47 minutes it cost [2][1]. Two unrelated defects, one in an autoscaling policy and one in a VS Code retry path, degraded code hosting, authentication, CI and AI assistance in the same window [1][9][15]. The first defect is a measurement error. GitHub's analysis traces the start to an Istio service-mesh sidecar that reached its concurrency limit while its autoscaling policy monitored the host service rather than the sidecar's own capacity, so new resources never arrived as traffic climbed [9]. The saturated object and the observed object were different objects. Pressure then moved outward until four HAProxy nodes exhausted their flow limits and the gateway authentication path developed severe latency and failures [11]. Everything downstream of auth followed: GitHub measured peak error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads [22], with SAML and OIDC authentication, SCIM provisioning and Team Sync also affected [13]. The second defect set the headline duration. Slow responses from one internal endpoint activated a latent retry bug in VS Code, lifting Copilot Token Service traffic from a normal 7,000 to 9,000 requests per second to between 70,000 and 100,000 [15], an increase of roughly eight to fourteen times [25]. GitHub blocked token requests with HTTP 403 responses, reduced gateway retries and restored traffic gradually [16], and said scraping attacks against code-download endpoints complicated recovery [17]. Most services were back at 16:36 UTC, Actions stayed degraded until about 18:03 UTC, and the Copilot Token Service did not fully recover until 21:02 UTC [14]. Working backward from the published 7:47 duration, the incident began near 13:15 UTC, core recovery took about 3 hours 21 minutes, and the Copilot retry tail accounts for roughly 57% of the outage window [24]. A client-side retry loop, not the mesh, owns most of the number in the headline. GitHub says neither August incident was caused by a code or configuration change and that both were capacity failures [3]. The August 6 Actions incident ran more than nine hours, with 71% of workflow runs hitting infrastructure failures at peak and 75% of the remainder delayed by more than five minutes [18]. July's availability report counted eight service-degrading incidents [19], and in March GitHub had already acknowledged that rapid load growth, tightly coupled infrastructure and weak defenses against misbehaving clients were letting local problems spread [20]. The load is real: monthly commits went from 1.4 billion in April to 2.9 billion in August [4], roughly doubling [27], alongside about 130 million merged pull requests and 24 million new repositories a month [5]. The spending is real too, and did not prevent this. GitHub added more than 3 million CPU cores, 120 petabytes of high-speed storage and installed as much hardware as available power in its existing data centers allowed [6]. Azure now carries roughly 58% of platform load and half of all Git operations, up from 12% of platform load in May [7], a 46-point shift in three months [26], with GitHub aiming to move GitHub.com production traffic out of its own data centers by the end of 2026 and add a second Azure region [8]. Two committed changes are worth tracking: consistent retry limits, retry budgets and variable timeouts across service-to-service calls, and a review of lower-priority CPU and memory alerts for components that could fail during spikes [10]. The structural one is slower, namely isolating critical systems and removing shared dependencies between them [12]. Until that lands, treat GitHub auth, Actions and Copilot as one failure domain rather than three, and check what your own clients do when a token endpoint gets slow. Fedorov's post was his third public reliability update since March [21].
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence66
- Adoption82
- Hype gap−12
- Incentives58
- Confidence64
Perspective Coverage
3 publishers- Builder
- Builder 39%
- Operator
- Operator 38%
- Investor
- Investor 23%
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
On August 17 GitHub experienced an outage lasting 7 hours and 47 minutes that disrupted github.com, authentication, GitHub Actions, APIs, pull requests, issues and Copilot.
- [2]
In a postmortem published August 20th, GitHub CTO Vlad Fedorov said traffic reached a new peak while a critical infrastructure component in the Central US data center failed to scale with it.
- [3]
GitHub said neither August outage was caused by a code or configuration change and that both incidents were capacity failures at their core.
- [4]
Monthly commits on GitHub grew from 1.4 billion in April to 2.9 billion by August.
- [5]
GitHub's charts show merged pull requests approaching 130 million a month and new repository creation reaching about 24 million a month.
- [6]
GitHub added more than 3 million CPU cores, 120 petabytes of high-speed storage and significant network capacity, installing as much hardware as available power allowed in its existing data centers.
- [7]
Azure now serves roughly 58% of GitHub's platform load and half of all Git operations, up from 12% of platform load in May.
- [8]
GitHub said in July it was working toward moving GitHub.com production traffic out of its own data centers by the end of 2026, with a second Azure region intended to provide regional resilience.
- [9]
GitHub's root cause analysis traces the initial failure to an Istio service-mesh sidecar that reached its concurrency limit; its autoscaling policy monitored the host service rather than the sidecar's capacity, so new resources did not arrive as traffic increased.
- [10]
The August 6 and August 17 incidents led to two immediate changes: applying consistent retry limits, retry budgets and variable timeouts across service-to-service interactions to prevent retry storms and cascading load, and reviewing lower-priority CPU and memory alerts to identify components that could fail during sudden traffic spikes.
- [11]
The bottleneck spread until four HAProxy nodes exhausted their flow limits, after which the gateway authentication path developed severe latency and failures affecting dependent services.
- [12]
GitHub says it is isolating critical systems and removing shared dependencies between them to reduce the likelihood of an outage and limit its impact when one occurs.
- [13]
SAML and OIDC authentication, SCIM provisioning and Team Sync were also affected.
- [14]
Most GitHub services recovered by 16:36 UTC, Actions remained degraded until about 18:03 UTC, and the Copilot Token Service did not fully recover until 21:02 UTC.
- [15]
Slow responses from one internal endpoint activated a latent retry bug in VS Code, increasing Copilot Token Service traffic from its usual 7,000 to 9,000 requests per second to between 70,000 and 100,000.
- [16]
GitHub temporarily blocked token requests with HTTP 403 responses, reduced gateway retries and restored traffic gradually.
- [17]
GitHub said scraping attacks against code-download endpoints complicated the recovery.
- [18]
On August 6th GitHub Actions was degraded for more than nine hours after a routine deployment exposed an existing capacity and concurrency weakness; at peak 71% of workflow runs experienced infrastructure failures and 75% of the remaining runs were delayed by more than five minutes.
- [19]
GitHub's July availability report counted eight incidents that degraded services during that month.
- [20]
GitHub acknowledged in March that rapid load growth, tightly coupled infrastructure and weak defenses against misbehaving clients were allowing localized problems to spread across critical services.
- [21]
The August post is Fedorov's third public reliability update since March.
- [22]
At the peak GitHub measured error rates of about 20% for web and API traffic and roughly 50% for archive and raw-content downloads.
- [23]
Competing products include Entire, the startup founded by GitHub's former CEO Thomas Dohmke, which is betting on a distributed system to handle agent-driven development workloads, and Cursor's Origin.
- [24]
Working back from the 7 hour 47 minute total, the incident began around 13:15 UTC; core service recovery at 16:36 UTC took about 3 hours 21 minutes, leaving roughly 57% of the published outage window attributable to the Copilot recovery tail.
- [25]
The retry storm multiplied Copilot Token Service traffic by roughly eight to fourteen times normal levels.
- [26]
Azure's share of GitHub platform load rose 46 percentage points between May and August.
- [27]
Monthly commits roughly doubled between April and August, an increase of about 107%.
Sources
3 independent publishers whose own reporting we read for this story.
- github.blogThe August 17 outage, and the work ahead
2 articles · August 21, 2026
- runtimewire.comGitHub blames capacity failure and retry storm for nearly eight-hour outage
1 article · August 20, 2026
- thenewstack.ioGitHub added 3 million CPU cores. It still couldn’t keep up with your commits.
1 article · August 20, 2026
Topics and entities
Topics
- Retry storms and cascading failure
- AI coding tools
- Incident Analysis and Postmortems
- Cloud Migration Programs