Thanks for asking question! The streaming endpoint wasn’t designed to be turned off and on like this, especially for video on demand (VOD) content. With VOD the assumption is that the streaming endpoint would run 24x7. The resources that the streaming endpoint needs do take a few moments to start. And if you have the CDN enabled, a while longer for that to be effective.
Also, The best way to share your feedback would be to create a User Voice feedback item and upvote it. The product group monitors this site for feedback. This is the best way to ensure you are heard and you may receive a response depending on how much they information they can currently share.
Hope this helps.