SYMPTOM Phase stuck in RUNNING state at 2am. Operator could not advance batch. Batch deviation raised. DeltaV v14.3, bioreactor unit, pharma client.
ENVIRONMENT DeltaV v14.3 · Life Sciences · Bioreactor Batch · ISA-88 Phase/EM Architecture
ROOT CAUSE EM transitioned directly RUNNING→ABORTED on a conductivity high-high alarm. The calling phase was waiting for IDLE before advancing to HOLD. Three other EMs in the same system used the ABORTING→IDLE pattern — inconsistency between engineers writing different EMs. No project-wide EM state machine template had been defined.
FIX APPLIED Defined a mandatory EM state transition template enforced at design review. All EMs now follow RUNNING→ABORTING→ABORTED→IDLE. Added shared ABORT function block reused across all units. Phase waits for IDLE (not ABORTED) before transitioning. Retrospectively corrected all EMs in the system.
KEY LESSONS
-
Define EM state machine template before any DeltaV code is written — not after FAT failures expose the gap
-
Standardize the phase-to-EM completion handshake across all engineers on the project (IDLE vs ABORTED — pick one)
-
Build one shared ABORT function block and reference it everywhere rather than each engineer writing their own
-
Add EM state transition coverage to phase review checklist — walk every abort path explicitly before FAT
-
HMI must display EM state in plain text alongside phase state so operators can diagnose without engineering support