I think I had the same issue, and I finally figured it out. You need to specify "common" as your tenant, rather than your tenant ID in your firebase call. This sends you to the correct url for the Microsoft Identity provider: https://login.microsoftonline.com/common/oauth2/v2.0 whereas providing my tenant id there requested the personal login against my tenant, which is not what I wanted. Not sure if it will help you, but this solved my issue with Firebase MS Identity, so figured I would share.
const provider = new OAuthProvider("microsoft.com");
provider.setCustomParameters({
prompt: "login",
// tenant: "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", <-- points to incorrect url
tenant: "common",
});