Hi,
What's the CU version of your Exchange 2016 server? If you are not in the latest CU version, you could try uptating it then access OWA and ECP again.
Please provide the complete error information or the screenshot of your issue here.
Use the command below to check if Authentication Settings for both Virtual Directories ECP and OWA are identical
Get-EcpVirtualDirectory | fl identity, *url*, *auth*
Get-OwaVirtualDirectory | fl identity, *url*, *auth*
We could also check the application log to get any related error information when we failed login OWA or ECP.
The thread below discusses about the similar issue as yours:
Exchange 2016 - Cannot Access OWA or ECP using Forms Authentication?
And I also found a KB here, which introduces the configuration like yours.
If an Answer is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.