I got the confirmation from the product team.
Mixed-mode authentication has not been supported in SF, irrespective of what the cluster definition allowed. Any mismatch was simply an omission in setting validation – the actual authentication is really ‘either or’. The CU2 release notes refer to the case where the cluster credential type is set to ‘none’, which is the default value for that setting and was unfortunately allowed as a creation-time configuration.
Prior to CU2, the security validation enforced (incorrectly) the starting (== current) state of the cluster at the beginning of an upgrade; this has been fixed and shipped with CU2. However, the enforcement on the ‘target’ cluster configuration is still in effect, and both credential types are expected to be x509.
Note that, CU3 bring proper support for specifying a server certificate for the http gateway in Windows-authentication-based SF clusters.