Which IP address get you if you run nslookup adfs.domain.com on your WAP, internal oder external?
Which operatation system you use for AD FS and WAP servers? Especially for Windows Server 2012(R2) with AD FS 3.0 it exist some updates, which must be installed.
Did you check the claim rule and their access control already?
Double check your current AD FS configuration for Office 365 with follow blog articles:
https://simpleitpro.com/index.php/2018/08/24/step-by-step-o365-configuration-for-single-sign-on-with-adfs-2016-2/
http://www.thatlazyadmin.com/configure-adfs-office-365/