Increased failure rate for VODs and Clips on streams ingested to NYC data center
Incident Report for Mixer
Resolved
This incident has been resolved.
Posted Mar 21, 2020 - 11:19 PDT
Monitoring
Network maintenance completed around 2:16AM PDT and normal data center network operation resumed. Streams that were online during the degraded network performance with NYC as their ingest point will likely continue to have VOD and Clip issues until the stream is restarted outside of the maintenance window. Engineers have monitored and confirmed the restoration of network reliability and are routing traffic back to the NYC data center.
Posted Mar 21, 2020 - 09:23 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 21, 2020 - 00:49 PDT
Update
Network maintenance is limiting the attempted mitigation. Re-routing traffic away from NYC data center.
Posted Mar 21, 2020 - 00:45 PDT
Update
Engineers are working to restore normal network connectivity between the NYC data center and other Mixer services.
Posted Mar 20, 2020 - 23:18 PDT
Identified
The connectivity issue between data centers has been identified and engineers are debugging the issue.
Posted Mar 20, 2020 - 23:12 PDT
Update
Streams that are connected to the NYC data center ingest are experiencing an increased failure rate when trying to create VODs or Clips as a result of diminished connectivity between the NYC data center and other Mixer services. Viewers that happen to connect to the NYC data center for distribution of FTL streams may experience a degradation of stream quality.
Posted Mar 20, 2020 - 23:10 PDT
Investigating
We are currently investigating this issue.
Posted Mar 20, 2020 - 23:05 PDT
This incident affected: Video Distribution (FTL (Low Latency) Video Delivery), VOD (VOD Uploads), and Clips.