I want to apply the CDN but I need to control the usage of each request, how I could achieve this?

Eduard Tomàs 25

Hi everyone I would like help on knowing how to do a particular thing with de Azure CDN. My use case is I need to implement a CDN on a service that we have, that retrieve files basically. The key is that we control the usage of our users (downloading size for each request) and this usage it's control on our end-host service. So I want to apply the CDN but I need to control the usage of each request, how I could achieve this? And what's the best way to do it?

Lex Li (Microsoft) 5,157 Reputation points Microsoft Employee

2023-09-25T16:44:21.11+00:00

Not sure what do you mean by "I want to apply the CDN but I need to control the usage of each request". Can you describe more details and be specific?
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-09-27T11:59:50.37+00:00

@Eduard Tomàs , could you please provide more details about your requirement to discuss it further?
Eduard Tomàs 25 Reputation points

2023-09-27T12:11:36.4966667+00:00

Yes! I implement a Microsoft classic CDN, so I want to cached the requests of an endpoint of my end. The problem is that my backend keep tracks of every request to get the information of the petition and then extract data to bill to my client. All this logic it's implemented on my end. Now when the CDN get a HIT will not passed by my backend service and so I would not be able to get this tracking. So then I don't know what's the best way to implement this, talking of software architecture level.Right now I'm getting the log queries from the Log Analytics resource and make a query to get the info I needed.
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-09-27T12:50:08.88+00:00

@Eduard Tomàs , I'm still not sure that I understand your requirement correctly.

But I believe this is what you are saying - You've a backend application which keep tracks of every request to get information and then extract data to bill your client and all this logic is implemented on your end. You also have a Microsoft classic CDN with caching enabled in front of your backend application and when the CDN gets a HIT, the content is served from the CDN edge cache and the request doesn't reach the backend, so you are not able to collect the required data. And hence you are getting the data from log analytics by running queries. But you would like to know if there is any other way to implement this setup, so that your backend gets the data even when CDN is HIT. Correct me if my understanding is wrong or let me know if this is right.

You mentioned "Right now I'm getting the log queries from the Log Analytics resource and make a query to get the info I needed". May I know what information you are collecting from Log analytics?

Regards,

Gita
Eduard Tomàs 25 Reputation points

2023-09-27T12:52:26.92+00:00

Perfectly! You described my question right!
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-09-27T12:56:47.99+00:00

@Eduard Tomàs , thank you for the confirmation. Please allow me some time to check on this and get back. Will keep you posted.
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-09-27T12:58:19.7566667+00:00

@Eduard Tomàs , Also, may I know if there is any specific information/data that you are collecting from Log analytics? Could you share some examples?

Eduard Tomàs 25

Yes sure! Here you have some example of the query I've made:

So I want to get the number of times it' called the different request Uri's

AzureDiagnostics 
| where OperationName == "Microsoft.Cdn/Profiles/AccessLog/Write" and Category == "AzureCdnAccessLog" and endpoint_s == "xxxx" and httpStatusCode_s == 200 and TimeGenerated >= ago(2m) and TimeGenerated <= now()
| summarize Count=count() by requestUri_s

GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-09-29T15:14:24.6233333+00:00
@Eduard Tomàs , thank you for the information.

I don't think it is possible to configure the CDN in a way that your backend gets the data even when CDN is HIT.

When CDN is hit, the content is served from the cache and the request doesn't go to the backend.

And since you want to use CDN caching but also want to get the billing data, the only thing that I could think of is to use the CDN raw logs.

Azure CDN from Microsoft Service currently provides Raw logs, and it is called AzureCdnAccessLog.

As mentioned in the Azure CDN monitoring document,

Raw logs include logs generated from both CDN edge (child POP) and origin shield. Origin shield refers to parent nodes that are strategically located across the globe. These nodes communicate with origin servers and reduce the traffic load on origin.

For every request that goes to origin shield, there are two log entries:

One for edge nodes.

One for origin shield.

To differentiate the egress or responses from the edge nodes vs. origin shield, you can use the field isReceivedFromClient to get the correct data.

If the value is false, then it means the request is responded from origin shield to edge nodes. This approach is effective to compare raw logs with billing data. Charges aren't incurred for egress from origin shield to the edge nodes. Charges are incurred for egress from the edge nodes to clients.

You can use the below Kusto query sample to exclude logs generated on origin shield in Log Analytics.

AzureDiagnostics | where OperationName == "Microsoft.Cdn/Profiles/AccessLog/Write" and Category == "AzureCdnAccessLog" | where isReceivedFromClient == true

Kindly let us know if the above helps or you need further assistance on this issue.

Regards,

Gita
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-10-04T15:41:10.0366667+00:00

@Eduard Tomàs , Could you please provide an update on this post?

Kindly let us know if the above helps or you need further assistance on this issue.
Eduard Tomàs 25 Reputation points

2023-10-05T06:57:42.27+00:00

I've already get with a solution! Finally using the raw logs was the best solution!
GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-10-05T11:51:50.4+00:00

@Eduard Tomàs , thank you for the update. Glad to hear that using Azure CDN raw logs helped achieve your requirement. I will summarize the answer below for visibility.

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.

Accepted answer

GitaraniSharma-MSFT 49,256 Reputation points Microsoft Employee

2023-10-05T11:52:01.63+00:00
Hello @Eduard Tomàs ,

I understand that you've a backend application which keep tracks of every request to get information and then extract data to bill your client and all this logic is implemented on your end. You also have a Microsoft classic CDN with caching enabled in front of your backend application and when the CDN gets a HIT, the content is served from the CDN edge cache and the request doesn't reach the backend, so you are not able to collect the required data. And hence you are getting the data from log analytics by running queries. But you would like to know if there is any other way to implement this setup, so that your backend gets the data even when CDN is HIT.

I don't think it is possible to configure the CDN in a way that your backend gets the data even when CDN is HIT.

When CDN is hit, the content is served from the cache and the request doesn't go to the backend.

And since you want to use CDN caching but also want to get the billing data, the only thing that I could think of is to use the CDN raw logs.

Azure CDN from Microsoft Service currently provides Raw logs, and it is called AzureCdnAccessLog.

As mentioned in the Azure CDN monitoring document,

Raw logs include logs generated from both CDN edge (child POP) and origin shield. Origin shield refers to parent nodes that are strategically located across the globe. These nodes communicate with origin servers and reduce the traffic load on origin.

For every request that goes to origin shield, there are two log entries:

One for edge nodes.

One for origin shield.

To differentiate the egress or responses from the edge nodes vs. origin shield, you can use the field isReceivedFromClient to get the correct data.

If the value is false, then it means the request is responded from origin shield to edge nodes. This approach is effective to compare raw logs with billing data. Charges aren't incurred for egress from origin shield to the edge nodes. Charges are incurred for egress from the edge nodes to clients.

You can use the below Kusto query sample to exclude logs generated on origin shield in Log Analytics.

AzureDiagnostics | where OperationName == "Microsoft.Cdn/Profiles/AccessLog/Write" and Category == "AzureCdnAccessLog" | where isReceivedFromClient == true

You confirmed that Azure CDN raw logs helped you achieve your requirement, and the issue is now resolved.

Kindly let us know if you need further assistance on this issue.

Please don’t forget to close the thread by clicking "Accept the answer", as this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

I want to apply the CDN but I need to control the usage of each request, how I could achieve this?

0 additional answers