I don't understand why this shouldn't work exactly like vacuum_index_cleanup (cf. vacuum_rel lines 2170ff). That would require no new mechanism.
That reloption is already an enum and there is no GUC to defer to when the value is unset. It doesn't seem like an equivalent scenario. AUTO is a perfectly useful value as opposed to an undocumented sentinel for unset/missing.
Sorry, the "already an enum" comment is wrong - I see the commit now where we basically re-implemented boolean value processing logic and added an "auto" option.
Basically we'd do this to make a boolean-compatible enum adding an undocumented value "null" as a valid and default set value and then interpret "null" as meaning "go use the vacuum_truncate GUC".
It's too late to argue against sentinel values so I suppose this would have to be acceptable.
