SFC host

Background for D-52 (S19). The host is the thin part around the pure core (SFC core, SFC rules). User documentation: SFC core in .NET.

How one event runs

1. Load the state document, build or fetch the graph of the pinned recipe.
2. Call Evolve with the event.
3. If the core applied it, one compare-and-set stores the state, the due-index entry and the active-set entry, and appends the work entries. A lost race reads again.

Work entries: start, cancel, hold, release, tick, resume, event, dispatch:{action} and job (the job order transitions of one step). A wait is state and a withdrawn attempt concerns only its sender, so neither becomes an entry.

Findings

• IOutgoingCommands had no implementation and the pipeline never reported responses; IActionSink carried only a command id. S19 built both ends (D-35 revision).
• The time of start cannot be the entry id. An order can wait in the stream longer than its first timeout, and the attempt was then lost without a command being sent. Found by the compose tests under load. start reads the server clock; a repeated start is absorbed, so nothing replays this time. Ticks keep the entry id (D-44 revision).
• The heartbeat is per scope and is written in the sweep script. A global heartbeat lets only the first scope of a round see an outage, and a second instance could pop due jobs between the heartbeat and the pop.
• After an outage the sweep takes every job of the index, not only the due ones: all open deadlines have to move by the gap.
• A tick that finds nothing writes nothing, so the host puts the job back on the index; otherwise a moved deadline would never fire.
• Pending entries are not removed on a response or a re-dispatch. They hold the round, which tells a late answer to an earlier pass of a loop from an answer to the current one. They run out 60 s after their deadline.
• A step that cannot be done with what is stored (recipe missing, state unreadable, contention) on the response path is logged, not raised: a fault on MQTT stops the host (D-27), and a redelivery would fail the same way. The deadline settles the action.

Hold and release (S20a)

Python never held a recipe (H-19), so this part is new behaviour. Design notes:

• Where the hold lives. In the core state (heldAt, format 1.2), not in the host. The core already owns deadlines, and the release reuses the outage rule: open deadlines move by the length of the hold. NextDeadline is empty while held, so the compare-and-set takes the job off the due-index and no tick is queued. Tick and resume of a held job are absorbed; a tick that was queued before the hold finds nothing to do.
• What is recorded. Responses to commands that are out, and events for waits that began before the hold (maintainer's choice in S20a), are recorded on their actions; the step stays active until the release. The waits index keeps a held job, so its event is not dropped. A recorded failure is routed on release.
• Order of the entries. A Pause and a Resume write hold and release entries in the order script. Two consumers can take them in either order, and a Pause, Resume, Pause can leave the last entry in front. Each entry reads the order and aligns the recipe with it (Interrupted means held), then reads the order again after its write and repeats until both agree (at most four rounds, then the entry is left for a redelivery). The read is not atomic with the write, so a single pass could apply a hold the order had already left; the last writer always sees the latest order. The time is the server clock when the hold or release is applied, not the entry time: entries taken out of order would pair a hold with the release of another Pause (/code-review finding). Held and Suspended both count as Interrupted: the internal Hold operation is a constraint, not a decision, but the recipe cannot run on either way.
• Which commands the release sends. The routing map (SfcPendingCommands) is in-process, so no instance knows whether another one sent a command. The core re-issues every open push attempt with its command id and attempt. The dispatch reads the marker sent:{action} = {startedAt}:{commandId} together with the state (one HGETALL) and skips the send when it matches; after the command is queued it writes the marker (a script that writes only while the state exists). The compare-and-set of the state writes only its own fields, so the marker lives as long as the job and goes with its delete and its snapshot. The start time tells apart a re-stored job id, whose command ids begin again at #1. The deadline of the send comes from the state, because a dispatch entry from before the hold carries the old one.
• Side effect. A dispatch entry redelivered after its command was queued and marked no longer sends the command again. A crash between the queue and the marker sends the command id twice on redelivery, as before S20a; receivers deduplicate by id. The first version wrote the marker before the queue, which could lose a command on a crash (/code-review finding).

Not chosen: dropping events during a hold and re-adding the wait on release (an answer from equipment that finishes its work would be lost and the action would time out); a set of skipped dispatches per job written by the consumer (a consumer that reads the held state, then loses the race against the release, would add its entry after the release read the set, and the command would wait for its deadline); a marker key with a time to live (a hold can outlast any fixed lifetime).

Equipment hold (S27)

Python never told equipment to stop (parity row H-29), so this is new behaviour. The contract is D-57. Design notes:

• Matching by command id. The answer to a hold command and the answer to the work it holds have the same action key. The host always knows the command id a response answers (the causation id, ActionResponse.CommandId), so the core gets it on SfcActionCompleted and SfcActionFailed. Hold command ids end in :hold. A late answer to any earlier hold is then recognised and absorbed without keeping a list of old hold ids. A response without a command id (other hosts, the simulation) belongs to the open command, as before.
• Deadlines while held. S20a empties NextDeadline during a hold. A hold command must still expire, so NextDeadline while held is the earliest hold deadline, and a tick while held expires only those. An outage moves hold deadlines in both cases. The deadline of held work is cleared on held; the restart sets its own.
• Release before the answer. The release moves the deadline of the work but does not issue it again, because the answer decides: held restarts at once, anything else issues the work again with its command id (the sent marker keeps it from going out twice). Recovery (SfcResume) skips work whose hold is open and issues a restart again with its id.
• Work that never went out. A dispatch that reaches a held job is skipped (S20a), but the core counts the work as dispatched and sends a hold for it. The host compares the work's sent marker with its command id before sending the hold. If they differ, the host runs SfcActionFailed for the hold itself (not held), so the release sends the work. The equipment never sees a hold for work it does not know. The check is not atomic with a dispatch in flight: when one consumer has queued the work but not yet written its marker, a hold taken by another consumer in those milliseconds is answered as not held, and the work runs to its end as in S20a (/code-review finding, accepted: the fallback is the safe side, and closing the window would need the marker before the queue, which S20a rejected because a crash could lose the command).
• Late held. A held answer to the current hold after its deadline still counts: the equipment did hold the work, and only a restart gets it going again (its command was sent already, so issuing it again would be skipped). The restart clears the hold command id, so a repeated answer is absorbed (/code-review finding).
• Format version. A work master with dataschema v1.0.0 that declares hold or restart commands is refused as an unusable recipe and its job is aborted: the version says what a recipe does (/code-review finding).
• Markers. sent:{action} keeps the work's command (the restart takes it over), and sent:{action}:hold the hold's. Action names contain no :, so the fields cannot clash. One HGETALL reads both.
• Compose tests (SfcHostTests): a pause holds running work and resume restarts it to the end of the job, with a late answer to the held attempt absorbed; a refused hold lets the work finish without a resend; a hold of work that was never sent is answered by the host and the release sends the work.

Python parity list: host part of sfc_engine.py and the tests

Rows S-1 to S-40 of SFC rules hold the decisions from S07. This list records what S19 and S20 did with the ones they own.

No row is open.