With Azure CDN, requests can come from any of the PoP or Edge locations. In order to enforce them, you will need to whitelist all of them.
Here is a doc which outlines the IPs by CDN offering. You will need to whitelist all of the IPs in your CDN offering.