Custom Translator Phrase Dictionary only working with exact case

Eric McStravick 0

We have an issue that our Fully trained Custom Translator engine is not using the right phrases.

It's trained with about 18,000 segments + 700 terms in phrase dictionaries.

If we match the case exactly, it seems to work, but if we don't it's hit or miss what we will get.

I can't give specific examples since the data is PIP; but I have multiple entries I tested with. For example, if the dictionary entry is "The ABC Corp" and the sentence contains "we at the ABC corp believe....," it doesn't work, but "we at The ABC corp.." might work and "we at The ABC Corp" will work.

The other thing we found is astrerisk completely rules out a match, "The ABC* Corp" will never match "The ABC Corp" regardless of case variation. This one does makes more sense, but I would expected that some punctation would be ignored.

Is there something I can do to make the Phrase dictionary less observant of capitalization/punctuation?

dupammi 7,955 Reputation points Microsoft Vendor

2024-06-25T04:09:46.4733333+00:00
Hi @Eric McStravick

Thank you for using the Microsoft Q&A forum.

Here is a snippet from the documentation relevant to Custom Translator and to build a dictionary:

If you're using a phrase dictionary, capitalization and punctuation are important. Dictionary entries are case- and punctuation-sensitive. Custom Translator only matches words and phrases in the input sentence that use exactly the same capitalization and punctuation marks as specified in the source dictionary file.

For the specific example you provided:

If your dictionary has "The ABC Corp", you might also include "the ABC Corp", "The ABC corp", and "the ABC corp" to handle different capitalizations. Avoid using asterisks or other punctuation that might cause mismatches unless they are necessary for your entries.To achieve better results, you might consider the following approaches:

Pre-processing: Normalize the text before translation to match the case and punctuation with your dictionary entries.

Dynamic Dictionary: Try Using the dynamic dictionary feature, which might allow you to customize translations for specific terms or phrases dynamically. This might offer more flexibility for handling case and punctuation variations.

Neural Phrase Dictionary: Try utilizing the neural phrase dictionary, which adjusts both the term and the context, if it suits your use case.

I hope this helps. Thank you.
Eric McStravick 0 Reputation points

2024-06-28T19:18:33.9266667+00:00

Can anyone give me some insight into how to use a Dynamic/Neural Phrase Dictionary - neither of these appear to an option when I create Dictionaries. When I create a dictionary, I goto Documents > Add Document Set > Dictionary Set

I only get 2 options: Phrase Dictionary (which I have been using) and Sentence Dictionary.

I expect a Neural Phrase Dictionary would solve my issues, but the feature seems to be missing - is there something I am missing (like I should have created account in a different way?)

Share via

Custom Translator Phrase Dictionary only working with exact case