Deontic arrays would be applied to the model being trained in the [[hfdt-takeover-scenario]] as an additional component of the optimization target. The approach would be applicable to all training regimes present in the scenario: self-supervised, supervised, and reinforcement learning.
In its basic form, the technique would be employed as follows. First, a large charter of normative principles expressed in written language would be collected from various sources. Those should contain huge amounts of redundancy (i.e. expressing the same principle in various formulations). The charter does not have to be internally consistent -- principles are allowed to occasionally clash.
Second, the charter would (automatically) be converted into an anti-charter which contains a negated version of each principle mentioned in the original charter.
Third, the two charters would be treated as collections of token sequences (i.e. sequences of words/subwords/characters). Given those two sequence sets, the model being trained would be incentived to output action sequences from which the original principles are more likely to follow compared to the negated versions. This requires a multi-modal model (e.g. the same model, a separate one, a past copy, etc.) to turn the action history and principles into a numerical reward signal. This signal would then join the main reward to form a final objective through an aggregation scheme (e.g. linear combination, geometric mean, etc.).
Fourth, the charter (and anti-charter) would be extended in parallel to overspecify the desired normative framework in an attempt to e.g. avoid quibbling over the specific letter of the law. The vanilla version of deontic arrays relies on what could be called "counterfactual cross-validation," which goes as follows. If in a completely sandboxed environment with only one output bit we temporarily discard part of the charter, is the model prevented from violating those anyways thanks to the remaining principles redundantly making up for them? If not, target external red teaming efforts to automatically patch up that brittle part of the normative framework. In a sense, new magnet-like structures would be deployed across Alex's state space to keep it in stable orbit around the safe region.
As a bonus, deontic arrays might help instill an aversion towards taking decisions which might then in turn lead to charter-violating actions later on, especially in a reinforcement learning regime. The model might grow to value actions which don't place it in morally ambiguous situations in the future, giving itself less opportunity to err.