Don’t Crash the Factory: How to Pentest OT Networks Without Halting Production

In 2021, a water treatment facility in Oldsmar, Florida, watched an operator’s cursor move on its own. An intruder had reused a leaked TeamViewer password, hopped into the plant’s HMI, and moved a sodium hydroxide setpoint from 100 ppm to 11,100 ppm. The operator caught it because he was watching the screen. The dose would have taken hours to reach the water; nothing broke that day. But every OT security conversation since Oldsmar has circled the same fear: we know we need to test this network, and we’re terrified the test itself will stop the plant.

That fear is justified — and it’s also why untested OT networks stay untested for decades. The resolution isn’t more caution; caution applied to the wrong method still breaks things. The resolution is a different method: rules of engagement built for environments where availability beats confidentiality, test windows treated as production events, and findings written for the people who keep the line running. This article lays out that method end-to-end: how to scope an OT engagement, the rules that make active testing survivable, the finding patterns that repeat across every plant we’ve assessed — and how to translate all of it into language that gets fixes funded. (For how the IT-side foothold that leads here usually starts, see One Key to Rule Them All.)

Why OT Testing Is a Different Discipline

Copy-pasting an IT pentest into a plant is how plants get broken. Five properties make OT different, and every rule in this article is downstream of them:

1. Availability outranks everything. In IT, a reboot is a fix. In OT, a reboot can be a safety event, a batch loss, or a six-figure scrap incident. The CIA triage inverts to AIC — often with safety first above all.
2. Protocols were never designed with authentication. Modbus, DNP3, OPC Classic, EtherNet/IP, Profinet — many have no authentication and no encryption whatsoever in their most common deployments. Any device that can reach a PLC can command it.
3. The assets are unpatchable by policy. Vendor-certified configurations, Windows XP-era HMIs still receiving quarter-century support contracts, controllers whose firmware update requires a plant shutdown to recertify. Scanning them like IT hosts isn’t “risky” — it’s guaranteed disruption.
4. Engagement reach is asymmetric: attackers only need one working path; testers need to prove paths without tripping watchdogs, stale PLC code, or industrial protocol quirks that crash on malformed input.
5. The customer isn’t the CISO. The deliverable’s real audience is the controls engineer and plant manager — people whose definition of “critical” is measured in downtime minutes and safety interlocks, not CVSS.

Phase 0: Rules of Engagement — Signed Before Anything Plugs In

Every safe OT engagement starts weeks before testing, in a scoping room with operations in it. The output is a rules-of-engagement document that operations signs, not just IT security. It contains, at minimum:

• Asset spectrum and demarcation: what’s in (PLCs, HMIs, engineering stations, historians, safety-instrumented systems?) and what’s categorically out (SIS is almost always out-of-scope for active testing — test its separation, not its internals).
• The stop-condition grid. The heart of the document. For every technique and asset class, one of four verdicts: GREEN (passive listening only), AMBER (active but no writes — TCP connects, banner grabs, version queries), ORANGE (writes simulated on cloned or lab assets only), RED (never — e.g., anything touching safety controllers, drive systems under load, or vendor-locked appliances). Testers don’t ask at 2 a.m. whether something is okay; they look at the grid.
• Windows and communication cadence. Active phases happen in agreed maintenance windows with an operations engineer on a bridge who holds a veto card — anyone, either side, can call stop, no debate, testing halts inside 60 seconds.
• Emergency protocol. If a process anomaly appears during testing — even if unrelated — testing stops, contacts on both sides are listed with phone numbers, and an “all clear + must-check” checklist names who verifies the process state before resuming.
• Evidence handling. Passive captures may contain proprietary process recipes; agree retention, encryption, and destruction up front.

The engagement that spends a full day on this document and zero packets is the one that doesn’t make the news.

Phase 1: Passive Discovery — Weeks of Listening, Zero Packets Sent

The safest packet is the one you never send. Passive discovery builds the network map by listening only:

• SPAN/tap captures at the plant-floor switches and the IT/OT boundary: every device announces itself eventually — Modbus function codes, Profinet DCP hellos, CIP connection registrations, EtherCAT frames.
• Protocol metadata alone is a finding stream. Which masters talk to which slaves, which HMIs hold engineering sessions open for months, which controllers respond to anyone (a Modbus slave has no concept of an unauthorized master — if you can route to it, you own it).
• Historian and engineering-station configs (on-linux file reads) reveal the device inventory better than any scan: controller projects name every node, tag, and setpoint in the plant.
• IT/OT boundary watch: the captures show VPN tunnels, dual-homed workstations, and the “temporary” 2019 cable that outlived its project. (This is also where the leaked-credential path from the IT side — the Oldsmar pattern — becomes visible.)

Passive findings frequently justify the engagement on their own: rogue dual-homed devices, flat networks bridging office and plant floor, and unauthenticated engineering protocols are all visible without sending anything into the control network.

Phase 2: Active Non-Writing Testing — Precision, Never Sweeps

Where the stop-grid allows, active testing proceeds with OT-grade discipline — which is nothing like an IT nessus sweep:

• Enumerate before touching. Read-type function codes only: Modbus read coils/registers (function codes 1–4), CIP identity objects, Profinet read services. Enumerate I/O counts and module inventories — never write coils/registers (function codes 5, 6, 15, 16).
• Handshake scale, not host scale. Rate-limit to a handful of transactions per second, one asset at a time, watched by the ops engineer. Industrial stacks have tiny buffers and watchdogs; an “innocent” full port scan can reboot a PLC into fault mode.
• Banner and version fingerprinting via side channels: engineering software exports, historian driver strings, HTTP on embedded webservers (found on alarming numbers of PLCs), and EDR/syslog from engineering stations.
• Path testing, not payload testing. The question is “can an attacker reach this?” — proven by reading, not by commanding. If you can read a setpoint over an unauthorized path, writing it is a trivial inference; document the path and stop.

The output of Phase 2 is a reachability proof: a map of every path from every untrusted zone to every controller, with the exact protocol permissions each hop grants. That map — not a scanner scorecard — is what plant leadership needs to see.

Phase 3: Where Writes Are Proven — Clones, Labs, and Paper POCs

Some risks can only be demonstrated by doing the dangerous thing. The OT answer: do it somewhere that isn’t the plant.

• Clone benches: the same controller family, firmware, and a small I/O board cost thousands, not millions. “We changed this register and the drive command changed” is demonstrated on the bench, filmed, and shown to management — the plant never sees the packet.
• Vendor/simulator environments: PLC simulators (and vendor virtual controllers) accept the same engineering commands as production; exploit paths validate there.
• Maintenance-window exceptions: occasionally the plant agrees to a controlled write test on non-safety I/O during a scheduled stop — outputs lifted to an unused channel, effect verified, then restored. This is the only “production write” that ever happens, and it’s choreographed like a lab experiment.
• Paper POC discipline: for everything else, the report documents the exact function codes, register addresses, and preconditions an attacker would use — sufficient for any controls engineer to reproduce mentally in seconds. In OT, a precise paper POC from a trusted tester moves mountains; a theatrical live exploit moves budgets to your competitor.

Phase 4: The Findings That Repeat in Every Plant

Ten years of OT assessments, and the pattern list is embarrassingly stable. If your plant hasn’t been assessed, bet on finding:

1. Flat IT/OT bridging — a direct route from user LAN to PLC VLAN through a dual-homed host or an “temporary” firewall rule now old enough to vote.
2. Unauthenticated protocols everywhere — Modbus TCP 502 exposed to anyone routable; engineering access with no per-user identity.
3. Shared engineering credentials — one admin across every HMI and engineering station, password on a sticky note under the keyboard since commissioning.
4. Eternal vendor remote access — unsupported-teamviewer-era tools, always-on, MFA-less (the Oldsmar shortcoming).
5. Unpatchable Windows on the floor — XP/7 HMIs inside support contracts that forbid change, stacked on the same segment as controllers.
6. Safety-instrumented-system reachability — not hacked, but routable: a compromised engineering station can ping the SIS, which is one misconfiguration away from being commandable.
7. No asset inventory — the plant discovers unknown devices during our passive capture; nobody knew the modem was still attached.
8. Default cellular/serial fallbacks — forgotten out-of-band links that bypass every firewall ever purchased.
9. Watchdog brittleness — devices that fault on scan traffic specifically, meaning the plant’s own monitoring tools could DoS it (discovered, safely, by us — first).
10. Zero detection on the plant floor — no logging, no IDS on OT segments; the capture we ran for two weeks is the longest visibility the network has ever had.

Rank these by downtime and safety impact, not CVSS. “#2 unauthenticated protocols” is nonsense to a plant manager; “anyone on the corporate network can open the valve you told us is your scrap risk” gets a PO signed.

Phase 5: Reporting for the Room That Funds Fixes

The final translation step. Three artifacts, one audience each:

• Executive one-pager: the risk story in downtime currency (“a compromised laptop can stop Line 3 for an estimated shift”), the three fixes that remove 80% of risk, and a 90-day plan. No CVSS, no ports.
• The controls-engineer annex: per-finding: affected assets (by P&ID tag, not IP), exploitation preconditions, compensating controls that work without a shutdown (firewall rules, ACLs on engineering stations, protocol gateways, disabling unused services), and the certification path for the real fix.
• The roadmap tie-in: findings mapped to a recognised maturity model — ISA/IEC 62443 zones-and-conduits is the lingua franca (Zone = group of assets with identical security requirements; Conduit = the controlled pathway between them). “Establish Z0 demarcation” lands better with plant leadership than 47 raw findings.

And the deliverable plant teams can actually use comes from looping back to the stop-grid: retest inside the same windows, same rules, proving fixes — an OT network where the fixes are verified is the only version of “secure” that means anything on a factory floor.

FAQ: OT/ICS Penetration Testing

Can you actually pentest OT without stopping production?

Yes — by changing the method, not just the attitude. Passive discovery sends zero packets; active testing is limited to read-type protocol functions at handshake rates; writes are proven on clone benches or simulators; everything dangerous is RED-listed in a stop-condition grid operations signed. Disruption incidents in OT testing almost always trace to IT-style scanning applied to PLCs — the method, not the mission.

What’s the first thing to fix in an OT network?

The IT/OT boundary. An enforced demarcation (dedicated firewall or unidirectional gateway, no dual-homed hosts, no “temporary” rules) removes the paths that make every other finding exploitable from the corporate network. It’s also fixable without touching a single controller — no downtime, no recertification.

Why can’t we just install EDR/antivirus on the HMIs?

Because many HMIs run Windows versions or vendor-locked configurations where third-party agents void support certification, and agent resource overhead can affect HMI responsiveness. Where hosts can’t take agents, compensate at the network layer: segmentation, protocol whitelisting, jump hosts for engineering access, and passive OT detection listening on SPAN taps.

What is ISA/IEC 62443 and do we need it?

It’s the international standard for industrial automation and control systems security — its core ideas are zones (assets grouped by security requirements) and conduits (controlled connections between them). You don’t need to “do 62443” wholesale to benefit: using its zone/conduit vocabulary in your assessment report aligns findings with what vendors, insurers, and regulators increasingly ask about.

How often should OT networks be assessed?

Passive visibility continuously (it’s non-disruptive by design), a full rules-of-engagement assessment every 12–18 months, and a retest after any major change — new equipment, network rework, or a new remote-access path. Big-control-system changes shift the attack surface more than a year of routine operations does.

We have no budget for clone benches. What’s the minimum viable safe test?

Passive capture at two points (plant-floor core switch, IT/OT boundary) for two weeks, plus engineering-station and historian config review. Zero disrupt risk, and it reliably surfaces flat bridging, rogue devices, unauthenticated protocols, and the real asset inventory — often 70% of the findings, for the cost of two taps and an analyst week.

Key Takeaways

1. OT testing fails by method, not by intent. IT-style scanning is what breaks plants; passive-first, read-only active, and simulated writes are what don’t.
2. The stop-condition grid is the engagement. GREEN/AMBER/ORANGE/RED verdicts signed by operations before testing begin beat any amount of tester skill at 2 a.m.
3. Rights are proven by reading; writes are proven on clones. If you can read a setpoint over an untrusted path, the write is an inference — document it and never send the packet.
4. The finding list repeats everywhere: flat IT/OT bridging, unauthenticated protocols, shared engineering credentials, eternal remote access, unpatchable HMIs, SIS reachability. If your plant is unassessed, these are already there.
5. Report in downtime and safety currency. CVSS doesn’t fund fixes in a plant; “anyone on the corporate LAN can open the scrap-risk valve” does. Map findings to 62443 zones and conduits for the roadmap conversation.

References

• Oldsmar water facility incident (2021) — public reporting and advisory analyses of the TeamViewer-access intrusion
• ISA/IEC 62443 series — industrial automation and control systems security (zones and conduits model)
• MITRE ATT&CK for ICS — adversary tactics and techniques against industrial control systems
• NIST SP 800-82 — Guide to Industrial Control Systems (ICS) Security
• Dragos/Microsoft/Deloitte OT security surveys — recurring unauthenticated-protocol and remote-access findings
• Internal: One Key to Rule Them All — the IT-side credential foothold that OT intrusions so often begin with