September 2025: Web services go down due to Caddy update

Summary: Web services become unresponsive.

Reported by: N/A

Date Reported: September 26th, 2025

Date Resolved: September 26th, 2025 (~30 minutes)

Root Cause: Caddy was reinstalled using it's generic package.

Resolution: Custom Caddy binary re-installed.

Narrative

SierraKomodo observed after logging into the EDC machine that there were 50 packages ready to be updated. She then ran the update command in line with usual best practice. Following this, all web services operating on the machine became unresponsive. Sushiloid happened to be available in text while discussing an unrelated incident. After brief investigation, he determined that the update command replaced the machine's custom Caddy binary, which included within it a module that allowed Caddy to communicate with Cloudflare.

Shortly thereafter Sushiloid re-installed the relevant binary and web services were promptly restored.