DDoS Protection Software Update
Date: January 17, 2025
Time Window: 13:00 - 20:00 CET
Current Status: In Progress
Details and Updates:
17:55 CET - In Progress
By announcing only test IP addresses to our DDoS scrubbing cluster, we traced the issue to a specific filter without causing further impact on customers. This filter will not be used in production again until the issue—potentially a kernel bug—has been analyzed and resolved. The remaining filters are functioning properly and will begin handling production traffic again shortly, once their functionality has been fully verified. Please rest assured that the scrubbing capacity remains sufficient even without the malfunctioning filter, as our scrubbing cluster consists of multiple redundant machines capable of handling the load, even if debugging takes a few days.
16:18 CET - Update
We rolled back to the previous software version and started routing the first prefixes through the DDoS scrubbing cluster about 5 minutes ago. However, our monitoring immediately alerted us to some timeouts. As a result, we have terminated the DDoS filter sessions again and are currently analyzing the issue before taking further action.
15:54 CET - Verifying
After receiving customer complaints about timeouts that have not yet been fully analyzed, we temporarily stopped the update process about 15 minutes ago. We apologize for any inconvenience caused. Unfortunately, we cannot estimate the actual impact at this point since the root cause has not been identified. We know there have been issues with a small portion of outgoing connections (e.g., the "apt update" command under Linux) and timeouts in applications running on IPs without permanent mitigation. IPs with permanent mitigation did not experience any issues. Consequently, we have terminated all DDoS filter BGP sessions for now (basic DDoS protection remains in place) and are evaluating whether the bug can be fixed immediately or if rolling back to the previous version is the best option. The maintenance window has been extended accordingly.
15:17 CET - Update
We are experiencing issues with some applications that are not permanently protected. We are already working on a fix with the highest priority.
14:54 CET - Temporary Fix
We implemented a temporary fix for Teamspeak servers and are working on a permanent solution.
14:46 CET - Teamspeak Issues
We are currently facing an issue with Teamspeak servers due to the new update. We are investigating the root cause.
13:00 CET - Maintenance Started
The DDoS protection cluster update to a new software version is underway. Filters are being updated one by one, with established TCP and UDP sessions seamlessly synced to the remaining filters. However, as this is a major update with significant changes, a residual risk remains despite extensive lab testing over the past few weeks. If you notice any unusual behavior in your applications during or after the update, especially with game and voice servers, please contact our support team immediately. While DDoS protection capacity is reduced during maintenance, redundancy ensures sufficient protection remains in place.
