Skip to content

Do not normalise ngrams terms that ends in a patchtable

While investigating #631 (closed) I have tracked down the problem (or part of the problem anyway) with the way ngrams terms are normalised in the frontend before creating a PatchTable.

I have done some investigation on #631, and I wasn't able to replicate the issue with "normal" flow tests which added elements as map terms.

However, on the frontend I was able to replicate the issue with things like home—brew where — is an Unicode glyph for "em dash" https://www.compart.com/en/unicode/U+2014.

If I look at the payload that the frontend is sending at the backend, I can see how the data is being already cleaned up in a non-desireable way (see Elements/Matrix).