I think I found the reason or issue.
I configured SSO and the identifier contained a slash at the end.
Removing the slash, disabling SSO and deletion worked for me .
The error messages are not helpful and lead in the wrong direction when using the web interface. Trying to delete the app via PowerShell gave more helpful error messages.