To support the increased number of users (over 2.27 billion monthly active users) and data usage, Facebook has been deploying 100G CWDM4 technology (QSFP 28). Currently, when there is a link failure, it is a very tedious process to troubleshoot the optical interconnects with the existing tools. These troubleshoot tools are very expensive and can characterize one optical transceiver at a time. This is a very time consuming process and the operational cost is very high. To reduce the operational costs and to optimize the troubleshoot time, we have developed a unique tool called “BER Illusion Methodology” (BIM) which is based on OCP network switches. This methodology can be used to troubleshoot 128 optical interconnects simultaneously in less than 5 minutes including all necessary information such as Transmitter power, Transmitter bias current, Receiver power, Case Temperature, complete BER test, supplier details, etc. The number of troubleshoot ports can be scaled, just by adding more switches if needed in the topology. This endeavor describes a post production troubleshoot/sustaining process which is being implemented to identify and characterize optical interconnect failures at our data centers so that we can reduce the “No Trouble Found (NTF)” numbers significantly and with minimal effort.
