At approximately 06:00 (GMT+8) we observed a critical stability issue on our core network infrastructure following an upgrade to the latest Linux kernel. The new kernel introduced a regression in the network stack / NIC driver, which caused the affected network interfaces to crash and reset repeatedly under load. This behavior manifested as intermittent link flaps, packet loss, and periods of full connectivity loss for services routed through the impacted nodes.
Our engineering team immediately initiated a controlled rollback to our previously validated, stable kernel version. This required a reboot of key network systems to safely revert the kernel and restore interface stability. However, the recovery process took longer than expected due to an unforeseen issue with our IPMI. The IPMI interface for the affected device became inaccessible, preventing us from performing the usual remote power-cycle and recovery procedures.
Our on-site engineer was dispatched to the data center to restore IPMI functionality and manually reboot the impacted network equipment. After IPMI access was recovered and the systems were rebooted into the stable kernel, the network interfaces returned to normal operation, and traffic flows were restored.
We are keeping the affected systems on the previous stable kernel while we further analyze the regression in the newer kernel version, review NIC driver compatibility, and tighten our kernel rollout and soak-testing procedures to prevent similar incidents in the future. Continuous monitoring is in place to ensure ongoing stability of the network stack and interfaces.
![[object Object]](/_next/image?url=https%3A%2F%2Finstatus.com%2Fuser-content%2Fv1759684197%2Fuycisdc0oxs0rc6ps6pu.png&w=3840&q=75)