No announcement yet.

Diacritic Character removal. Those strange non text characters...

  • Filter
  • Time
  • Show
Clear All
new posts

    Diacritic Character removal. Those strange non text characters...

    Has anyone ever come up with a universal way to removed all control and special characters from stings in Miva Script?

    Removing characters below char 32 and above char 127 does not work because some, but not all, of the codes are two bytes long.

    Every example I find uses Regex expressions or functions built into the language.

    Here are JavaScript examples:
    const str = "Crme Brule"
    str.normalize("NFD").replace(/[\u0300-\u036f]/g, "")
    >"Creme Brulee"
    Or the more modern version:
    str.normalize("NFD").replace(/\p{Diacritic}/gu, "")
    Last edited by RayYates; 07-27-21, 06:21 AM.
    Ray Yates
    "If I have seen further, it is by standing on the shoulders of giants."
    --- Sir Isaac Newton

    Just out of curiosity, why do you want to remove them? Any business that wants to support a language other than English may need them. Are they causing any specific problems?
    Kent Multer
    Magic Metal Productions
    * Web developer/designer
    * E-commerce and Miva
    * Author, The Official Miva Web Scripting Book -- available on-line:


      You could iterate the string and use the "isprint" builtin and remove non-printable characters that way. However that will not replace the characters with their closet "ascii" character. If the text is UTF-8 you should be able to detect that a multi-byte sequence and know how many bytes to remove too.
      David Carver
      Miva, Inc. | Software Developer


        For Posterity, I've written a JavaScript function that converts unicode and diacritic characters to readable content.

        function utoh(text, min = 127, max = 255) {
            /*     Ray Yates 3/2022
            utoh() = Unicode to Html
                Translate diacritic and unicode characters to printable characters.
                Diacritic like 'ăşţ' etc.
                and Unicode like 
            Given a string that contain special characters,
            converts characters to plain text or html code. (hex version)
                 becomes a,  becomes c
                \u0092 becomes &#x92 ( displays the curled single quote ’ );
                \u00A9 becomes &#xA9 ( displays the copyright symbol  );
                1.    let domObject = document.querySelector("#tab-Products");
                    domObject.innerHTML = utoh("&mvt:product:descrip;");
                2.    $("#tab-Products").html( utoh(data.description) );
                3.    $("#tab-Products").html( utoh(data.description, 150, 160) );
                    Limit unicode characters to replace.
                text: the string to clean up.
                min, max: optional unicode characters to search for.
                If omitted, defaults set to 127, 255
            See: for unicode character set.
            let norm_text = text.normalize("NFKD");
            for (let index = min; index <= max; index++) {
                norm_text = norm_text.replaceAll( String.fromCodePoint(index), `&#x${index.toString(16).toUpperCase()};` );
            return norm_text;
        Last edited by RayYates; 03-30-22, 07:48 AM.
        Ray Yates
        "If I have seen further, it is by standing on the shoulders of giants."
        --- Sir Isaac Newton