October 02, 2023
Is IP over WDM finally here?
By
Loi Nguyen, Executive Vice President, Cloud Optics Business Group, Marvell
Some twenty years ago the concept of IP over Wavelength Division Multiplexing (WDM) was proposed as a way to simplify the optical infrastructures. In this vision, all optical networks are connected via point-to-point mesh networks with a router at the center. The concept was elegant, but never took off because the optical technology at the time was not able to keep up with the faster innovation cycle of CMOS, driven by Moore’s law. The larger form factor of WDM optics does not allow them to be directly plugged into a router port. Adopting a larger form factor on the router in order to implement IP over WDM in a massive scale would be prohibitively expensive.
For routers to interface with the networks, a “transponder” is needed, which is connected to a router via short-reach optics on one side and WDM optics to the network on the other. The market for transponders grew quickly to become a multi-billion-dollar market.
A Star is Born
About 10 years ago, I was building a team at Inphi, where I was a co-founder, to further develop a nascent technology called silicon photonics. SiPho, as it’s called, leverages commercial CMOS foundries to develop photonics integrated circuits (PIC) that integrate hundreds of components ranging from high-speed modulators and detectors to passive devices such as couplers, waveguides, monitoring diodes, attenuators and so on. We were looking for ideas and customers to bring silicon photonics to the marketplace.
Fortunately, good technology and market need found one another. A group of Microsoft executives had been considering IP over WDM to launch a new concept of “distributed data centers,” in which multiple data centers in a region are connected by high speed WDM optics using the same form factor as shorter reach “client optics” used in switches and routers. By chance, we met at ECOC 2013 in London for the initial discussion, and then some months later, a product that enabled IP over WDM at cloud scale was born.
At Inphi, we called this product COLORZ® to clearly differentiate it from “grey” client optics, while adopting the same QSFP28 form factor. Silicon photonics technology allows us to shrink the optics onto a tiny 5x6mm2 photonics chip to fit inside the tiny QSFP28 module. But bandwidth, reach, and power present huge challenges. We need to deliver 100 Gbps of bandwidth over 80km while consuming a mere 3.5 watts. For comparison, a leading 100G WDM optics technology consumed 25W at the time and had to be packaged in a much larger CFP or CFP2 form factor.
The final product turned out to be something no one ever thought possible. COLORZ use pulse amplitude modulation (PAM4), dual wavelengths, proprietary Inphi® Forward Error Correction (iFEC), fixed WDM lasers, and a long list of other innovations to realize the program goals. The final power consumption turned out to be 4.8W, but this was not a showstopper given that the competing technology consumed 25W!
COLORZ went on to become one of the most successful products for Inphi (now Marvell). It is the industry’s first, true IP over WDM to be deployed at cloud scale.
Crossing the Chasm
The benefits of IP over WDM can’t be overstated. By eliminating the transponder, an expensive network element that requires high-touch provisioning, power, space, cooling, and double the number of optics (one for client side, one for line side), operators save thousands of dollars per wavelength in capex and even more in opex. For large operators that deploy 100,000s of wavelengths in their networks, we estimated the savings in total cost of ownership (TCO) would reach billions of dollars over the life of the program.
For IP over WDM to be fully embraced by the industry, however, it needed a second act. While coherent optics was too power hungry for this use case in 2014, advances in CMOS made it possible to develop 400G coherent optics that could fit in a form factor such as QSFP-DD or OSFP, like that of client optics. Learning from the past, the industry pushed for 400G coherent optics to be an open standard such as OIF or IEEE. Thus was born 400ZR coherent optics, which allows, for the first time ever, metro WDM optics based upon open standards, enabling interoperability among multiple vendors.
400ZR optics was launched in 2021 and went on to become “the fastest growing coherent optics ever” according to Andrew Schmitt, of the Cignal AI industry analyst group. According to Cignal AI, over 200,000 of these ZR (and its variant, OpenZR+) optics are being shipped this year. The forecast calls for up to one million optics to be shipped by 2027!
What’s Next?
At Marvell, we recently announced the launch of COLORZ 800, our third generation of WDM optics enabling IP over WDM. Operating at extreme rates of up to 140 Gbaud, COLORZ 800 is capable of transmitting 800 Gbps over more than 500km. In addition, COLORZ 800 integrates an erbium-doped fiber amplifier (EDFA) inside the module, which enables up to 0dbm of output power—the industry standard for ROADM-based telecom networks.
Re-architecting metro and regional networks is an audacious task that will take decades and billions of dollars. Transponders won’t disappear overnight, but given the rapid advances in ZR/ZR+ optics, transponder share will diminish while ZR/ZR+ optics share will increase dramatically over time.
Artificial Intelligence (AI) will accelerate demand for ZR/OpenZR+ optics to power massive data center interconnect (DCI) bandwidth growth within regions. Leading hyperscale providers have already described deployment scenarios with tens of thousands of GPUs in AI clusters connected across multiple data centers in a region. Those interconnects will likely be served by the 800ZR/OpenZR+ optics.
It’s clear that IP over WDM is here to stay. ZR/OpenZR+ optics will continue to gain share, reduce their power consumption, all while allowing customers to save, literally, billions over time.
To close on a personal note, I’d like to express my deepest thanks to the many, many team members who toiled long hours, weeks, months, and years to make these products successful. I’m forever in your debt.
This blog contains forward-looking statements within the meaning of the federal securities laws that involve risks and uncertainties. Forward-looking statements include, without limitation, any statement that may predict, forecast, indicate or imply future events or achievements. Actual events or results may differ materially from those contemplated in this blog. Forward-looking statements are only predictions and are subject to risks, uncertainties and assumptions that are difficult to predict, including those described in the “Risk Factors” section of our Annual Reports on Form 10-K, Quarterly Reports on Form 10-Q and other documents filed by us from time to time with the SEC. Forward-looking statements speak only as of the date they are made. Readers are cautioned not to put undue reliance on forward-looking statements, and no person assumes any obligation to update or revise any such forward-looking statements, whether as a result of new information, future events or otherwise.
Tags: data center interconnect, IP over WDM, PAM4, Wavelength Division Multiplexing, WDM
Recent Posts
- 2006: The Twelve Months That Changed the Chip Industry
- Encrypting Data in Use: How HSMs Can Advance Next-Generation Confidential Compute
- Nine Things to Remember About the Future of Copper in Computing
- Cloud-Managed Enterprise (CME) Switches Powered by SONiC
- Marvell COLORZ 800 Named Most Innovative Product at ECOC 2024
