A false smoke alarm almost drowned Jane Street's only NYC datacenter
Learn how a single sensor failure can threaten an entire data center and what safeguards were missing.
A false smoke alarm caused a water‑based fire suppression system to activate, flooding Jane Street’s sole New York City data center and nearly destroying its hardware. The alarm was mistaken for a real fire, so the system released thousands of gallons of water into the hall. The incident stopped only when staff manually shut off the flow, after the building’s alarms had already filled the space.
The suppression system is tied directly to the smoke detector without an intermediate verification step, so any alarm, real or not, triggers a full‑scale discharge. The design follows code that allows fire suppression to act on low‑voltage (<4000 V) wiring, but it omits a dedicated Emergency Power Off (EPO) button inside the hall. Without an EPO, operators could not cut power to the pumps or stop the water release from the control room.
Jane Street’s own Sequence of Operations for data halls deliberately keeps the CRAC units running during a smoke alarm, per the International Mechanical Code, to prevent heat buildup. However, that rule does not address the water discharge path, leaving the facility vulnerable to exactly this scenario. The lack of an interlinked EPO and smoke alarm meant the system behaved as designed, rapidly, but without a human override.
For anyone building or managing a data center, install a physical EPO button inside each hall and wire it to both power and suppression controls. Verify that smoke alarms trigger a verification delay or a secondary sensor before releasing water. Regular drills should include testing the EPO and confirming that staff can access it quickly, even on higher floors.
TakeawayNever rely on a single alarm to trigger suppression; always provide a manual cut‑off inside the hall.