Per my test, the rich text editor will not work in document library. However it works in lists. We cannot change it through designer or PowerShell as it is just set as we expect. It is checked on enhanced rich text while we cannot use it.
If you have further concerns about this issue, please open a service request, support team will provide more help about it.