What mode do you use? Modern or classic?
There is no OOB method to enforce unique values in a lookup field. Also, currently, it is hard to insert code into modern experience.
You should switch to classic experience for this list. Then you can use code to set unique value for the lookup column. Just add the code into NewForm.aspx page.
You can refer to below article:
Removing Duplicate values from a Lookup field
Please note: Microsoft provides third-party contact information to help you find technical support. This contact information may change without notice. Microsoft does not guarantee the accuracy of this third-party contact information.