Hello @john john,
Even when you stop (or delete) Azure VMs, the Bastion solution is still there and keeps on consuming Azure resources.
It keeps reserving resources like some computing and a public IP address.
We also run into this and now use this runbook approach to deleting the Bastion, time-based in this example.
When we need a Bastion again, once every few days we accept the penalty for the 10-15 minutes it takes to set up.
If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.