SFC core library

Background for D-17 and D-18.

Question: build a library analogous to valence-works/bpmn for the SFC engine? Checked 2026-11-06 (README, repository metadata, NuGet).

What that library is

A BPMN 2.0 interpreter whose whole design is one pure function, (process definition, token state, one event) -> (next state, commands for the host). No clock, no I/O, no threads, no DI, no persistence, no external dependencies; the host does everything else. MIT, net8.0/net10.0, packages Bpmn.Model, Bpmn.Interchange, Bpmn.Semantics. Maturity: repository created 2026-08-08, last push 2026-10-31, 17 stars, 12 open issues, NuGet 0.5.0 (about 9k downloads), by the Elsa/valence-works team. Young, but actively developed and honest about its limits (its README lists "when to use something else").

Why the pattern fits us

Today sfc_engine.py (~1.4k lines) mixes three things: the SFC semantics (steps, transitions, selection and simultaneous branches, qualifiers), the Redis mechanics (work stream, XAUTOCLAIM, Lua CAS) and the equipment I/O (callback_outgoing, _pending_commands). The semantics are the part that is hard to get right (branch convergence, priorities, qualifier edge cases), and they are tangled with Valkey. A pure core gives:

• Deterministic tests without containers: a bug report is four JSON documents (recipe, state, event, expected result). Most of the Python test suite would run in milliseconds.
• An explicit CAS contract: the host loads state, calls the core, and writes the new state plus the emitted commands in one atomic step (the Lua script only compares the old version and stores the result). The core never needs to know about Lua.
• AOT and allocation friendly: immutable records, no reflection, no I/O.
• Replaceable execution strategy: already a stated goal in docs/overall-design.md ("The SFC engine can be replaced with a different execution strategy without changing any processor code").
• It fits the existing dataschema dispatch on the Work Master: a recipe format is selected by URI and handled by an interpreter.

Recommendation (D-17, proposal)

1. Yes to the architecture, no to a separate product. Build the SFC core as its own dependency-free project in this repository (for example MicroDCS.Sfc, no reference to MicroDCS.Core, Valkey or transports). Keep it public-ready, but do not set up a separate repository, release train or documentation site now. It can be split out later if there is outside interest.
2. Shape of the core: see "Design rules taken from the BPMN docs" below (read in full on 2026-11-06); the first sketch (pure Step(recipe, state, event)) is confirmed and refined there.
3. The host stays thin: work stream consumption, loading and storing state with CAS, executing commands (call the southbound processor, trigger the northbound state machine), recovery scan.
4. Interpreter interface for later formats. Define IRecipeInterpreter on the dataschema boundary with the same pure signature. The SFC core is the first implementation.
5. BPMN: do not adopt now.
6. SFC qualifiers N/P/S/R/L/D, step/transition alternation and the equipment interaction patterns have no direct BPMN counterpart. Adopting BPMN means redesigning the recipe language and tooling, and our recipes are JSON while BPMN is XML.
7. The dependency is young (0.5.0, 17 stars).
8. It targets business processes; the domain here is ISA-88/IEC 61131-3 style sequence control.
9. Our hard requirements (idempotency, CAS, XAUTOCLAIM recovery) are in the host, which the BPMN library explicitly leaves out.
10. Revisit when someone wants BPMN recipes: an adapter interpreter on the IRecipeInterpreter boundary is a self-contained addition (candidate after S26). Until then it is a design reference only.
11. Do not build a general workflow engine. Scope stays what docs/sfc_engine.md describes: sequential function charts with the two interaction patterns, no expression language (conditions are resolved by the host or by declarative selectors).

Findings from the BPMN library docs

Read: docs/concepts (interpreter-not-engine, host-port, capabilities), ADRs 0002, 0005 and 0010, the hosting and simulation guides, the packages reference and the 0.5.0 release notes (clone of valence-works/bpmn, about 17.5k lines of source in four packages plus 21k lines of tests). Not read in detail: errors-and-escalation, ADRs 0003, 0004, 0006 to 0009, 0011 to 0013, the Interchange (XML) guides.

Confirmed:

• The contract is exactly our idea: synchronous, no clock, no I/O, no DI, no persistence, no randomness. Four inputs, (graph, prior state, host snapshot, event) -> (next state, continuation, commands, error disposition?). A bug report is a paste of those four values (as JSON); replaying the same events against the same state gives byte-identical state because every generated id is a pure function of a Sequence counter held in the state.
• The "host is real work" cost is real: the docs say so, and that is why they ship a reference host (Bpmn.Runtime.InMemory, with a virtual clock and a transcript) so the interpreter is tested against a host and recipes can be simulated.
• Their host invariants map onto our existing design: "pass back the state you were last given for that scope instance" is exactly what our Lua CAS enforces across replicas; "never reuse a handle" and "echo the token id" correspond to mdcsactionkey and the command CE id.

New or changed compared with the first sketch (design rules for S07 and S15):

1. Four inputs, not three: add a compiled graph. SfcGraph.Build(recipe, capabilities) validates once and is reused; the interpreter only runs graphs that built. Structural problems are refused at build time, not at step 17 of a job: unknown step in a transition, branch without paths, two selectors that cannot be told apart (R-1), and a cycle with no wait (a loop of steps whose actions all pass through would spin inside one evaluation; they refuse it at build time and add a hard propagation limit of 10,000 dispatches per evaluation that returns a Fault). The same build step is where a station refuses a recipe it cannot run (for example no pull_event support): it fits the existing job-acceptability check (is_job_acceptable) and their "capabilities refuse at graph-build time" rule (ADR 0004).
2. Three commands, waiting is also work. BPMN has only StartWork, CancelWorkSubtree and SignalEnclosingScope; a timer, a message wait and an HTTP call are all StartWork, and the interpreter does not distinguish them. For SFC: StartWork (push_command dispatch, pull_event wait and time-limited/delayed actions as work kinds, carrying binding ref = type_id, a stable work token id = the action key, and a correlation dictionary) and CancelWork (abort, simultaneous-branch teardown, re-dispatch). The "complete job / fail job / start" signals to the northbound state machine are not commands but the continuation (Complete(outcome), Defer, Fault(code, message)), which the host maps to northbound triggers. Commands are inert data, applied in the order returned.
3. A fault is a verdict, not a command. The core reports what happened (Caught, Propagated, Absorbed) and never says "retry", "open an incident" or "page someone"; that is host policy. For us: an equipment failure or timeout yields Fault(code) and the host decides job abort or re-dispatch (R-2).
4. Work identity (ADR 0010) is the key design lesson for R-2. Callbacks must echo the token id from StartWork (they found that without it two concurrent invocations of the same binding cannot be told apart and callbacks can complete the wrong sibling). Our action key plays that role. Two rules to adopt: (a) a callback with an explicit id selects exactly that record and is rejected, never matched to a sibling, when binding or id contradict; (b) handles identify physical attempts and are never reused. One place where our semantics must deliberately differ: BPMN absorbs a late completion for work it tore down (one-shot "stopped handle" marker). For push_command the equipment may finish attempt 1 after we re-dispatched attempt 2 under the same action key; that completion is real. Rule proposal: completion matches on the action key; an earlier attempt's response completes the action if it is still open, and any further response for a closed action is absorbed as a duplicate. Late responses for aborted or cancelled jobs are absorbed (this is the D-9 "TTL makes it stale" case, now explicit and testable).
5. Time stays outside; the host owns waiting. "The library's entire relationship with time is that the host has some." For the dispatch deadline (R-2) and the L/D qualifiers, the proposal is events that carry a timestamp (Tick(now)) and absolute deadlines in the state, with a durable due-index in Valkey maintained by the host. Reasoning against the Kubernetes multi-instance model, the alternatives and the open points are in "Time model and the multi-instance model" below (proposal D-18). To be decided in S07.
6. State is an opaque, immutable, JSON-serializable document with Prune(): persisted verbatim in Valkey (one value per job), bounded diagnostics ("what was decided", capped at 200 entries, usable as audit trail), a sequence counter, and an explicit persist state before applying commands rule ("applying a command whose state was never durably recorded is how a crash produces work with no token behind it"). The reverse failure (state recorded, command never applied) is not solved by their library; their host is not durable. For us the outbox covers it: one script atomically compares the version, stores state and appends the commands as work-stream entries; execution is at-least-once and idempotent by action key; the recovery scan re-derives missing commands. Suggested core support: a Resume event that compares active work with the snapshot's LiveWork (the host's ledger of work it knows about) and re-emits StartWork for work that has no live handle. This turns our _recovery_scan into a pure, testable function.
7. Versioned payload format, schema published, explicit wire names (ADR 0005). The library owns the JSON format of both the definition and the execution state, versions it separately from the package (BpmnPayloadFormat.Version, additive changes bump the minor), checks in a generated JSON Schema, and has tests that regenerate and compare it and that fail on any serialized property without an explicit wire name. Hosts wrap it in an envelope. Apply this to us: SfcRecipe and SfcState are versioned payload roots; the recipe's dataschema URI already carries a version; the Work Master data + dataschema envelope is the same pattern. Direction reverses compared with Python: there the JSON Schema is the source and the code is generated (dataclassgen); there the C# model is the source and the schema is generated from it. Decide in S07 (it also answers the JSON-schema-generation exploration item for the SFC part). Their schema generator uses reflection, which is fine as a build-time tool.
8. No expressions; outcomes or a synchronous evaluator port. Their conditional flows carry an outcome name that the completed work reported; expression conditions are answered by a host-supplied synchronous IBpmnConditionEvaluator, enabled by a declared capability that fails the build otherwise (ADR 0011). Finding about our code: SfcTransition.condition ("condition identifier, resolved at runtime") exists in the schema and docs, but the Python engine never reads it (no occurrence in sfc_engine.py); only priority is used. Decision for S07: drop it, or adopt the outcome model (an action result names an outcome, a transition fires on that outcome name, which also gives selection branches a real decision input and a place for the R-1 selector).
9. Reference host and simulation as a product feature. MicroDCS.Sfc.Testing (or in MicroDCS.Testing): an in-memory host with a virtual clock, scripted equipment responses and a transcript of every evaluation, so (a) the core is tested against a host, (b) a recipe can be dry-run before it is accepted (a recipe check in the job acceptance path or a design-time tool), (c) the Python scenarios can be ported as simulations. Not durable, not distributed, and says so.
10. Architecture tests enforce the zero-dependency rule. Their build checks that shipped packages have no external dependencies (a dedicated architecture test project). Add the equivalent: MicroDCS.Sfc references no other MicroDCS project and no external package, enforced in CI (S15).
11. Conformance suite. They keep a conformance test project for the specification semantics. Ours is the IEC 61131-3 SFC semantics (qualifiers, divergence/convergence) plus the Python scenarios from python-archive.

What does not transfer or needs care:

• AOT: the library does not claim AOT compatibility (no IsAotCompatible, no source-generated JSON context, schema tool uses reflection). Irrelevant while we only take the design, relevant if the BPMN adapter (below) is ever added.
• Immutable state copying has a cost ("every dispatch grows the state the next one copies"). Fine for SFC state (dozens of steps), but it is in tension with the low-allocation driver; measure (R-18) and use ImmutableArray/records with structural sharing where it matters.
• Size: their semantics package is ~7.7k lines for the whole of BPMN. SFC is far smaller (steps, transitions, two branch types, six qualifiers, two interaction patterns); a first core of roughly 1.5 to 2.5k lines plus tests is a realistic estimate (not verified), which fits S15 (core) and S19 (host).
• BPMN adapter later: the primitives they expose (StartWork, tokens, outcomes) map well onto our StartWork/work-token model, so a BPMN interpreter behind IRecipeInterpreter stays a realistic add-on, but the recipe language decision above (SFC first) is unchanged.

Revised verdict: D-17 stands and becomes more concrete. The main change to the plan is that S07 must settle the work-identity and time rules (items 4 and 5) and the schema direction (item 7) before S15, because they shape every type in the core.

Time model and the multi-instance model

Question: how does the Kubernetes multi-instance model affect the timer decision (timers as commands versus timestamped events and absolute deadlines)? Answer: it decides it. The relevant facts about the deployment, and what the Python code does today, come first.

Facts about the deployment (design assumptions, not new decisions)

• N replicas run the SFC engine, no leader (docs/overall-design.md, multi-instance model). Work is shared through a consumer-group stream; state changes are CAS on the Valkey document.
• Pods are ephemeral: rolling updates, node drains, OOM kills and scale-down kill them at any time, and new pods start empty. Any in-memory timer on a pod has to be assumed lost.
• There is no scale-to-zero for the engine; but a full outage (all pods down, or Valkey down) is possible and must be survivable.
• Each node has its own wall clock; clocks are NTP-synchronised but not identical.
• Push-command responses are instance-affine on purpose (D-9): the sending instance holds the pending map and the wait.

What Python does today (from reading sfc_engine.py, mqtt.py, the processors; not run)

• The only timer is a local asyncio task in MQTTHandler that sleeps for the message expiry interval and then calls processor.handle_cloudevent_expiration. It lives and dies with that one pod.
• That hook fails the SFC action only in the demo processor (greetings.py calls the registered sfc_engine.fail_action; wiring in app/__main__.py). The machinery jobs processor just logs.
• SfcActionAssociation.timeout_seconds is in the schema and docs but is never read by the engine. recovery_idle_threshold_s is stored and never used.
• The recovery scan runs once at pod start and enqueues a resume for every active job; there is no periodic scan. So after a pod death an action stays dispatched until some pod happens to restart.
• Conclusion: in effect the SFC engine has no working job-level timeout today. This is the gap R-2 describes, and the .NET design should close it deliberately.

Consequences for the two options

Recommendation (D-18, proposal): (b), with this concrete shape

1. Core stays clock-free. The time-carrying event is Tick(now); every event also carries its At. When the core starts an action it computes absolute deadlines (At + timeout) and stores them in the state. State exposes NextDeadline (the earliest open deadline of the job, or none). The host does not need to understand deadline kinds (wait timeout, L limit, D delay).
2. One due-index entry per job. Valkey sorted set (for example sfc:due:{scope}), member = job id, score = NextDeadline in epoch milliseconds. The same atomic script that compares the version and stores the new state also ZADDs or ZREMs the index entry (and appends the commands as outbox entries). The index is derived data; the state is the truth; it can be rebuilt from the active-job scan.
3. All replicas run the sweep, no leader. Every pod polls (about every second, with jitter) a script that pops due members (ZRANGEBYSCORE then ZREM in Lua; only the pod whose ZREM succeeds gets the entry) and XADDs a tick work item to the scope's work stream in the same script. From there the existing machinery applies: consumer group, XAUTOCLAIM if the pod dies mid-processing, CAS on the state. No Kubernetes Lease and no singleton lock.
4. Tick is idempotent. A duplicate tick (redelivery after XAUTOCLAIM) finds the deadline already handled in the state (version changed) and does nothing; a tick for a deadline that was extended meanwhile re-inserts the index entry with the new NextDeadline.
5. One clock for everyone: Valkey server time. now for ticks and for event timestamps comes from TIME inside the Valkey script, not from the pod clock, so all replicas agree and skew between nodes does not matter. Needs a check against Valkey (scripts that call TIME and then write are allowed since Redis 5 / Valkey, effects replication; verify in the S04 spike).
6. Response routing unchanged (D-9). The sending pod keeps its pending map. If the deadline is fired by another pod (the sender died, or it is simply the one that popped the entry), the core turns the open action into a re-dispatch or a fault with CAS; a response that then arrives at the old sender is handled by the late-response rule (finding 4): completes the action if it is still open, otherwise absorbed. The local expiry task in the transport layer may remain, but only to clean the pending map; it no longer decides anything.
7. Wire TTL and domain deadline are different things. The MQTT message expiry interval (D-10, native feature) tells the broker when to drop an undelivered command. The action deadline is the job-level timeout. Set the deadline slightly after the wire expiry (grace of a few seconds) so a response in flight at the edge is not lost; make the grace configurable.

Open points for S07 (decision or spike needed)

• Outage grace. After a full outage (or Valkey down) every deadline is overdue at once, and the first sweep would time out or re-dispatch everything for a reason that has nothing to do with the equipment. Options: (i) ignore, accept; (ii) the host keeps a heartbeat key written by the sweeps; if the gap since the last sweep exceeds a threshold (this is what recovery_idle_threshold_s was probably meant for), the host marks the Tick as afterOutage and the core extends open deadlines by the outage duration instead of failing them. Recommendation: (ii), because for a manufacturing DCS a restart of the platform must not abort running jobs.
• Sweep interval and cost. One poll per second per pod is O(log N) on the sorted set; with 20 pods that is 20 cheap calls per second. Make it configurable; jitter the start.
• Valkey cluster mode (HA deferred, D-12). The script touches the state key, the index key and the stream key; in cluster mode they need a common hash tag (for example the scope). Choose key names with a {scope} hash tag now so that HA later does not change the key schema.
• Rejected alternatives: Valkey keyspace expiry notifications (pub/sub, fire and forget, lost when no subscriber is connected); a Kubernetes CronJob or external scheduler (extra component, coarse, not fitted to per-job deadlines); a leader-elected scheduler (reintroduces a single point of failure for something CAS already solves); delayed stream entries (Valkey Streams cannot delay delivery, which is the reason for the sorted set).
• Local fast path. A pod-local timer next to the sweep would give exact timing but is a second mechanism with the same CAS outcome; not worth it unless the sweep latency turns out to be a problem (measure in S20).

Open points from the plan review (2026-11-07), also for S07

• Scope discovery across replicas. Python fills _known_scopes only locally (start_recipe, the in-process register_scope callback, the startup recovery scan). With shared subscriptions only the replica that received the job learns the scope; the others neither read nor XAUTOCLAIM its work stream until they restart. The job DAO maintains active_scopes, but the engine does not read it. The same question applies to the due-index: which sfc:due:{scope} keys does a pod sweep? With {scope} hash tags a single global index is not possible in cluster mode. Proposal: a persistent scope registry (a set written by the script that creates the first job of a scope), read by every pod at start and every few seconds, plus a removal rule once a scope has no active jobs. S20 gets a test: start a job through replica A, kill A, check that replica B picks up its work and deadlines.
• Retry limit and one timeout source. Re-dispatch (D-9, R-2, D-18 item 6) has no attempt limit or backoff; Python counts attempt but sets no limit, so failed equipment would be re-dispatched forever. Two timeouts compete: the recipe's SfcActionAssociation.timeout_seconds (required, never read by Python) and the wire expiry that D-18 item 7 derives the deadline from. Proposal: the deadline comes from the recipe (timeout_seconds) and the wire expiry is derived from it (deadline minus grace); add max_attempts (recipe field with a host default) that decides between re-dispatch and Fault(timeout).
• Stream trimming deletes pending work. All Python streams are written with MAXLEN ~ 5000, which also removes entries still pending in a consumer group; XAUTOCLAIM reports them as deleted ids and Python discards them. In the D-17 outbox that is a lost command. Rule for S09 and S19: no MAXLEN on work and outbox streams; trim only acknowledged entries (XTRIM MINID below the group's lowest pending id, or XACK plus XDEL); log and count the deleted ids XAUTOCLAIM returns. Change streams read by cursor can keep MAXLEN, because the publisher republishes the state index at start.
• Hash tags make the job identity (scope, id). If state, due-index and work stream share a {scope} hash tag, every job key contains the scope, so every lookup by job id needs the scope too: the REST job query (R-19), the MCP tools valkey_document and sfc_job, the maintenance tools, the MES-facing methods. Python keys jobs by id only. Decide explicitly in the key-schema ADR: either (scope, job id) with API shapes such as /scopes/{scope}/jobs/{id}, or no hash tags now and a key-schema change when HA arrives (D-8 allows it).
• Take At from the stream entry id instead of a TIME call (D-18 item 5). The core runs in the host before the CAS script, so now is needed before evaluation. A work item's id (<ms>-<seq>) is assigned by the server at XADD, so it already carries server time; ticks included, because their id carries the sweep's server time. No extra round trip, and a redelivered item keeps its id, so replay stays deterministic. Events that do not pass through the stream (push responses handled directly by the sending instance) still need TIME. Ids only increase within a stream; after a clock step back they run ahead of the clock, which is acceptable for deadlines. Check in S04.

Test implications (S15 and S20): core unit tests use constructed Tick events (no waiting); the in-memory host has the virtual clock; the compose scenario in S20 kills the pod that dispatched a command and asserts that another pod re-dispatches or faults after the deadline (and, as a second scenario, stops all pods for longer than the deadline and asserts that no job is aborted because of the outage, once the grace rule is decided).

Costs and risks

• Extra up-front design in S15 (state and event model, versioned state for CAS); the first running recipe comes about a session later.
• Commands are intentions recorded in the new state (for example action dispatched with deadline); the host's execution must be idempotent (mdcsactionkey), exactly as today. A command the host fails to execute must leave the state consistent.
• Atomicity: Python does the state change and the next work item in one Lua call. In .NET the host writes new state plus an outbox of work items in one script; the core must not assume more than that.

Follow-ups

• S07 decides: project name, IRecipeInterpreter, work identity and late-callback rules (finding 4), time model (finding 5), schema direction and payload versioning (finding 7), whether SfcTransition.condition is dropped or becomes outcome-based (finding 8), the capability flags for recipe acceptance (finding 1). Output as ADRs in docs/adr.
• S15: write the core first, with tests ported from test_sfc_engine.py and test_sfc_recipe.py in python-archive, plus the in-memory host (finding 9) and the architecture test (finding 10).
• Still to read if a decision depends on it: the BPMN docs on errors and escalation, ADR 0003 (fault disposition), ADR 0004 (capabilities) and ADR 0011 (condition port), and the in-memory host source (BpmnSimulation, BpmnTranscript) for the transcript format.
• Check whether one generic Valkey script ("compare version, set state, XADD outbox entries") can serve the whole engine.