Note that I don't want to argue that this is an ideal solution for everything; I just indicate what is already possible in XBoard. The possibility to pop up complex dialogs was inspired by Evert Glebbeek's remark that he would like an engine to be able to use the GUI for design of a new variant (rather than requiring the user to edit config files), through an interface similar to the Design Wizard I put on the Interactive Diagram article here. The described method was trivial to implement in the GUI, as everything was already there. So it was just a matter of allowing the engine to invoke it (rather than a menu click).

Some steps occurring in typical preludes could indeed be covered by regular moves, so that they would not need any special provisions at all on the GUI side. E.g. variants that start with setting up the army turnwise on an empty board could use drop moves, and the engine can simply refuse premature attempts to move around pieces in the setup phase. This can be extended by adopting the conventon that a drop move on a square that is occupied by a friendly piece indicates a swap, and takes the original occupant back into the hand. That way a Superchess prelude could be implemente as: white substitution 1, black substitution 2, obligatory white symmetrizing substitution 2, obligatory black symmetrizing substitution 1, white substitution 3, etc. In Musketeer Chess you don't have to pick just pieces, but also assign them to a file. (Very bad idea, btw...) This could be done by making extra (initially empty) board ranks to drop the selected pieces on. But this would then turn their later gating into a double move, where they have to step forward as a side effect of the piece in front of them moving away. Of course if the protocol extension is used that allows general multi-moving, (as in the WinBoard Alien Edition), this would be no problem. It does seem a bit unclean, though (and for clarity the extra ranks should be differently colored than the normal checkering, so a lot of GUI support). Some people would attach value to the piece and its gating location being visible at all times, but I don't consider that an absolute necessity. (There is also no optical clue for whether you are allowed to castle, so not all game state is always visible.)

I am dwelling on this to decide whether there is any market for a protocol extension that could exchange arbitrary messages between the players (whether engine or human), which would then also switch the clocks and appear as a move in the PGN. E.g. "pseudomove <string>" both for engine->GUI and GUI->engine, depending on who's turn it is. At first glance this seems of little use if the GUI does not know what <string> means. But it could be used to change hidden game state, for instance to assign a file to a piece in hand, to play Musketeer as a modified Seirawan Chess (where the GUI assumes you would always be allowed to gate anywhere). The engines could then communicate the allowed gatings to each other through the pseudomove command, send the actual gating in S-Chess format (e.g. f1g3c to gate a Cannon), which would be understood by an S-Chess-supporting GUI. GUIs especially designed for Musketeer could apply some visible clue to indicate the gating pieces and files, which could go as far as displaying a piece just beyond the board edge. But you would not be dependent on such a GUI feature for playing engine-engine or human-engine games. (If you are willing to put up with the hidden game state.) In the latter the GUI would have to present the <string> to the opponent, and allow the user to enter one by typing. (The general mechanism.)

Note that playing different engines against each other is really something that almost no one does. TalkChess is really a very unrepresentative environment in that respect, full of engine developers and testers. But engine developers, and especially variant-engine developers, are a rare breed. In XBoard the 'second engine' is only used in 'Two Machines' mode. But XBoard is very tightly coupled to the 'first engine', taking it along with everything you do (e.g. in Edit Game mode). This because it originally was dependent on that engine for legality checking of the entered user moves. So what XBoard does is ignore the second engine completely (it is not even started up) until the user presses 'Two Machines'. Then that engine is started, and if it then turns out to not support the current variant, the mode switch is refused, and an error popup appears reporting "Second engine does not support this variant". For variants that are not fully specified (such as shuffle games) the first engine is in charge, and any start position it specifies will be loaded through setboard into the second engine. The engines should better be compatible w.r.t. how the pieces are called and how they move, but that always applies. If they don't agree on the initial setup (like the probably would, in shuffle games) it is no problem, the position indicated by the first (if the GUI or user had no desires on this) would be force-fed to the second to setboard. If they would disagree about piece names or moves this would wreck things, and XBoard currently doesn't check this. But this would always be the case, even in standard variants known to the GUI, where there isn't anything that could be checked. It just means one of the engines is non-compliant.

I guess the real problem is that different CwDA flavors are in fact different variants, so that treating them as a single variant leads to all kinds of unnatural problems. In a single variant, only partially supporting it (like: "I can play Chess, but the Knight is not implemented") is not an option. But there is nothing wrong with supporting CwDA except the Nutters. The only real problem is the unmanageable proliferation of variants this causes when you support many armies. Perhaps the GUIs should just provide a smarter selection method for variants in general, not just allowing ticking of one radio button from a list of 'full' variant names, but allowing a variant name to be synthesized from two parts, each selectable by a radio-button from a list. E.g. the engine could specify it plays rookies~CWDA and CWDA~nutters, where the ~ in the variant name would be recognized by the GUI as having the special significance of defining a partial variant name, where the all-capitals part then would be considered a place holder as well as a group identification. It could then present lists of such 'left' and 'right' partial names in the New Variant dialog, each with their own group of radio buttons and allow selection of two compatible (i.e. belonging to the same group, here 'CWDA') partial variants (and refuse 'OK'ing the dialog for incompatible selections), and combine the partial names to 'rookies~nutters' to get the selected variant. Depending on the preference of the GUI developer this could also be two combo-boxes (for left and right partial name), or even an arbitrary number of pairs of comboboxes (one for each variant group). The list of 'full' names would probably list 'selected below' as one of the possible choices to enable the comboboxes for selecting the partial names. Or one radio button for each variant group. E.g.

o normal

o xiangqi

o shogi

o CWDA: [combobox1] ~ [combobox2]

[Cancel] [OK]

The (first) engine would have full control (through its variants feature) over how this New Variant dialog would look. (And pedantic GUIs could ignore things in the list that they don't like or don't know; e.g. the standard edition of WinBoard would ignore variant tenjiku.) If a match with the second engine is attempted, the GUI will check if the current variant is in that engine's variant feature directly, or as a combination of partials. And refuse the request to play the engines against each other when it isn't.

This moves the selection of armies from the engine's private option settings to the GUI's variant-selection dialog, which is what you want. The variant is a common setting for both engines, so it also removes the problem of  'engine coherence', which is what I like about it.

I have some misgivings about engine-defined options that should be standardized, recognized by the GUI, and displayed in a separate dialog. This is just not what the option feature was desigintended for. For comparison: WB protocol allows 'size overrides' on variant names, like 5x5+5_shogi. (And users can select such variants in XBoard by using size-override rank, file and holdings controls in the New Variant dialog.) Normally the engine would consider each differently overridden version of a variant a separate variant, and list it as such in its variants feature. (E.g. Shokidoki lists shogi, 5x5+5_shogi (= mini-Shogi) 6x6+6_shogi (= Judkin's Shogi), 7x7+6_shogi (= Tori Shogi) and 8x8+6_shogi (Euro-Shogi). But I also did define a standard pseudo-variant 'size', where 10x10+0_size in the variants feature would imply that the engine will accept size overrides for board sizes up to 10x10 (but no holdings) on any of the reported variants. This is very similar in purpose to your concept of game variables, like having spin options for ranks and files with maximum 10. The equivalent of your castling option in this form of the protocol would be to allow a 'flex_' prefix on any variant name to indicate that variant uses flexible castling, (e.g. flex_capablanca) and define a standard pseudo-variant flex_castling to indicate that every supported variant also is offered in the 'flex' flavor. (Basically just a short-cut, to prevent very long variant lists.) GUIs understanding the flex_ thing could display / enable a checkbox 'flexible castling' in their New Variant dialog. This way of doing things looks more natural to me, because it puts the modifiers where they really belong: they basically create different variants, which deserve to have different names.