Darius Baruo
May 14, 2025 13:41
GitHub’s services experienced three significant disruptions in April 2025 due to configuration errors and database resource contention, affecting Codespaces users and causing migration service failures.
GitHub experienced several service disruptions in April 2025, impacting various services including Codespaces and migration functionalities, according to the GitHub Availability Report.
Incident on April 11
On April 11, GitHub’s Codespaces users faced significant challenges when approximately 75% encountered failures in creating and starting services. This incident, lasting 39 minutes from 03:05 UTC, was attributed to manual configuration changes to an internal dependency, which had not been adequately covered by tests. GitHub’s monitoring and detection systems promptly identified the issue, allowing for rapid resolution by reverting the changes.
Database Resource Contention on April 23
Later in the month, on April 23, multiple GitHub services were affected by a 20-minute degradation due to resource contention on database hosts. The error rates, affecting 2-5% of total requests, were linked to an interaction between query load and an ongoing schema change, leading to connection saturation. The completion of the schema migration resolved the issue. GitHub has since identified a regression in schema change tooling and reverted to a more stable version to mitigate future risks.
Migration Service Failures
On the same day, GitHub’s Migration service experienced elevated failures from 19:13 to 19:55 UTC. This was caused by a configuration change that inadvertently removed access for repository migration workers, affecting 837 migrations across 57 organizations. Users received error messages prompting them to seek support. Once access was restored, services returned to normal.
In response to these incidents, GitHub has implemented enhanced test coverage and refined monitoring thresholds. The company is also reviewing database capacity and improving monitoring systems to prevent similar issues in the future.
For real-time updates and further insights, users can follow GitHub’s status page and the GitHub Engineering Blog.
Image source: Shutterstock
Credit: Source link
