System Context

More than 1,300 communication cards are deployed across CERN cryogenic systems, primarily in the Large Hadron Collider (LHC). These cards collect data from peripheral electronics and transmit it through the WorldFIP network.

The cards were designed approximately 20 years ago and use non-reprogrammable FPGAs, making firmware modifications impossible. Replacing the existing communication cards would cost more than 1 million CHF, so reliability improvements had to be implemented through additional electronics while preserving the installed hardware.

Intermittent Power-On Failures

During annual maintenance tests, approximately 20% of communication cards failed to initialize correctly after power cycling, requiring local resets and manual technical interventions in the accelerator tunnel. Rare communication corruptions were also occasionally observed during operation.

An FPGA-based add-on had previously been developed and installed on fewer than 100 critical cards. However, extending this solution to the entire installed base would cost more than 100,000 CHF, making it impractical for large-scale deployment.

Both the earlier FPGA-based add-on and the solution described here are called add-ons because they are installed directly on top of the existing communication card, connecting through the card's existing header pins together with two dedicated cables. Only minor rework of the existing card is required, allowing either solution to be deployed without redesigning the legacy hardware.

Root Cause Analysis

I led the investigation in the laboratory, reproducing the failure using a cryogenic crate and performing repeated power-off/power-on cycles while monitoring the power rails and communication signals.

The communication card combines a 3.3 V FPGA with a 5 V MicroFIP communication module, which implements the WorldFIP protocol. During startup, the FPGA transmits initialization data to the MicroFIP module.

I discovered that the time difference between the availability of the 3.3 V and 5 V supplies varied between power cycles. Depending on this timing difference, the FPGA could begin transmitting initialization data before the MicroFIP module was ready, causing part of the initialization sequence to be lost.

This intermittent startup race condition explained the observed power-on failures and the rare communication corruptions.

Engineering Approach & Solution

Based on the root-cause analysis, I designed a new low-cost add-on PCB with two functions.

First, it synchronizes the startup sequence, ensuring the required voltage levels are available with the correct timing before initialization data is exchanged between the FPGA and MicroFIP module.

Second, it provides remote reset functionality for communication cards where remote reset was not previously available. An analog pulse-detection circuit monitors the expected signal activity and generates a reset command when the pulse sequence stops.

Instead of using an FPGA, the circuit relies entirely on transistors, capacitors, resistors, and diodes.

I simulated the circuit using MATLAB/Simulink, performing worst-case calculations to ensure reliable operation under component tolerances and radiation-induced degradation. Radiation exposure reduces transistor gain over the product's lifetime: for the radiation levels involved, a transistor with a nominal gain of 200–300 could degrade to a worst-case gain of roughly 100 over the card's operational lifetime. To provide substantial margin beyond this worst case, I designed the transistor stages assuming a gain of only 20.

I then completed the PCB design and built prototype units for laboratory validation.

Validation, Qualification & Deployment

I led the complete validation and qualification process, including laboratory, temperature, and radiation testing. These tests verified both the electrical functionality and reliability of the design under the expected operating conditions.

Following successful laboratory and environmental testing, I supervised the installation of 19 prototype PCBs on communication cards in the accelerator, where they were monitored for approximately one year of operation.

After successful field validation, I led the transition to full production. This included component selection and procurement, coordination with the PCB manufacturer, production and assembly planning, and managing the technical preparation required for large-scale deployment.

I also supervised a junior engineer and an intern supporting laboratory testing and PCB assembly, while retaining responsibility for the technical direction, validation, and overall development of the project.

Impact

The discrete analog solution made large-scale deployment economically practical, costing a fraction of the previous FPGA-based approach, while addressing both the intermittent startup failures and the remote-reset limitation of the legacy cards.

The solution also eliminated the need for manual resets in the accelerator tunnel during annual maintenance tests, removing an intervention colleagues had previously carried out every year. The new remote-reset capability further allows affected cards to be reset remotely during operation, reducing the need for colleagues to access the accelerator for local interventions.