Stream Analytics output system property override not working

CareyBoldenow 96

Hello,

I have a streams analytics query that is outputting events to a Service Bus Queue. I would like to manually set the "EventEnqueuedUtcTime" system property on the output. In my stream analytics set up, I set the "System Properties Column Json value to:

{"ScheduledEnqueueTimeUtc":"triggerDate"}

My stream analytics query looks like this:

SELECT *,
CAST('2019-12-29' AS datetime) AS triggerDate
INTO "future-output-servicebus"
FROM "golden-input"

When I then run the query whether in Test Mode or from the running stream analytics job, the "EventEnqueuedUtcTime" system property value is just the current timestamp and not the.value I am explicitly setting the triggerDate value. Below is a screenshot when I run the query in Test Mode. As you can see the triggerDate value looks correct, but is not getting picked up by my System Property Column config.

Any ideas as to why this is not working? Thanks for any help you can offer!

Regards,
Carey

PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2020-12-18T06:21:52.71+00:00

Hello @CareyBoldenow ,

Welcome to the Microsoft Q&A platform.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2020-12-21T04:52:36.28+00:00

Hello @CareyBoldenow ,

What do you mean by it’s not getting picked up by my System Property Column config?
What we see here there is nothing that is telling the Trigger Date to be the System Property Column. Could you please share the example of the input and the desired output that would help us provide the solution?
CareyBoldenow 96 Reputation points

2020-12-21T13:29:35.203+00:00

Hello @PRADEEPCHEEKATLA-MSFT ,

As you can see above in my initial note I mentioned that I am setting the System Property Column value in my output configuration. I have attached a screenshot to better represent what I am talking about.
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2020-12-22T05:45:55.347+00:00

Hello @CareyBoldenow ,

Thanks for sharing the details.

We are reaching out to the internal team to get more help on this, I will update you once we hear back from them.
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2020-12-24T04:46:51.413+00:00

Hello @CareyBoldenow ,

Can you see the column in the output from the service bus when you run the ASA job?
CareyBoldenow 96 Reputation points

2020-12-24T12:22:40.973+00:00

Hello @PRADEEPCHEEKATLA-MSFT

Well I do not see the triggerDate column in my output, but I do see the EventEnqueuedUtcTime, but it reflects the current datetime and not the triggerDate value. Below is a copy of the event from service bus.
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2020-12-29T11:08:16.867+00:00

Hello @CareyBoldenow ,

We are still waiting for a response from the product team. I will let you know once I heard back from the product team.

Stay Tuned!
CareyBoldenow 96 Reputation points

2021-01-05T12:54:35.497+00:00

Hello @PRADEEPCHEEKATLA-MSFT ,

When do you anticipate having an answer for this from the product team?

Thanks,
Carey
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2021-01-06T04:44:29.68+00:00

Hello @CareyBoldenow ,

We are still waiting for a response from the product team. I will let you know once I heard back from the product team.

In case if you need immediate assistance on this issue, you may file a support ticket.

Stay Tuned!
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2021-01-08T09:38:48.207+00:00
Hello @CareyBoldenow ,

We had setup your job in our subscription and tested. The System Property Column configuration worked. I think there are two things the customer misunderstood.

ScheduledEnqueueTimeUtc and EventEnqueuedUtcTime are different. ScheduledEnqueueTimeUtc is used to give the service bus a future time, and after this future time, the message will be enqueued. But in this case this time is set to the past, so I think it makes sense the actual EventEnqueuedUtcTime is the current timestamp.

The System Properties of the service bus is not shown in our test results table. We also tried the Service Bus Explorer (preview) on the Service Bus Portal, it does support showing system properties either. We would suggest you try this standalone Service Bus Explorer https://github.com/paolosalvatori/ServiceBusExplorer.
CareyBoldenow 96 Reputation points

2021-01-08T12:50:56.747+00:00

Hello @PRADEEPCHEEKATLA-MSFT

I respectfully disagree with the answer above. When I initially submitted this question (12-17-2020), 12-31-2020 was not in the past, it was in the future. I am disappointed that whoever was reviewing this, would not have at least done that due diligence. The other thing that does not make sense is that they say above that when they tested the System Property Column configuration, it worked, yet they say they had no visibility into the System Properties, how do they know then that it worked?

Thanks for the help on this, but I think I am going to open up a formal support ticket for this.
PRADEEPCHEEKATLA-MSFT 89,386 Reputation points Microsoft Employee

2021-01-11T05:58:57.04+00:00

Hello @CareyBoldenow ,

For a deeper investigation and immediate assistance on this issue, you may file a support ticket on the same.

1 answer

Marcello Leijer 1 Reputation point

2020-12-29T21:52:00.75+00:00

Click to vote
0 Votes"
0
MarcelloLeijer-4844 answered · 14 minutes ago
Mocrosoft geweldig product. Ik durf U uit ervaring te zeggen! dat verbetering ver van ten allertijden goed is. Jammer genoeg spreek ik uit er varing. Ik he nu 1 van de letterlijk vele foto's verstuurd. Wat letter lijk nooit gedaan is. Sterker ik heb Niet eens een creditkaart. Ik hoop dat deze foto helemaal door komt. Zo niet Zie ik graag Uw reactie dan weet ik dat voor de volgende keer. Sorry dat dit totaal geen andwoord is op Uw vraag. Ik wilde dit engelijk op de gehele pagina versturen. Om di even zo kor mogelijk te gaan houden. Er word op me naam gefraudeerd als maar mogelijk kan zijn. Wil ik gerust details van zeggen. In dien vraag heb je iemand je wachtwoord gegeven nee. Als ik op me microsoft account ga voo mijn labtop plaats na te zien geeft hij letterlijk meer dan 5 plekken aan. Oft ik vraag om hulp. id opgelost of werkelijk vooruitzicht wil ik U er ook gerust voo tegemoet komen. Graag verneem ik Uw reactie gr Marcello 52097-img-20201128-wa0020-1.jpg
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Stream Analytics output system property override not working

1 answer

Your answer