Error with html inside JSON using miva_json_decode

[paul.oreilly]

HI,

I'm running into an issue using miva_json_decode(). It looks like there are <br> tags inside of some of the text description of a paged JSON response I'm trying to process. It looks like miva attempts re-evaluate the JSON 3 times, then it moves on. I can see that, because it's writing the contents of the response to a file, and I see duplicate entries prior to hitting a problematic json string. It doesn't give up, but it does seem to be reprocessing the request.

Is there anything I can use to debug what's going on? I saw there's miva_json_decode_last_error() . But I'm not sure how I'd use that.

Is there a way to pre-process the JSON response and remove the <br.> tags?

Here's the code as is:

<mvt:comment>Start making requests</mvt:comment>
<mvt:assign name="g.api_request" value="'https://api.resellerratings.com/v1/product/reviews/full.json?sku=' $ g.code $ '&per_page=100' $ '&page=1'" />
<mvt:call action="g.api_request" method="'GET'" headers="'Authorization:Bearer ' $g.rr_token $ asciichar( 13 ) $ asciichar( 10 )">
<mvt:comment>Turn the JSON response into a miva structure</mvt:comment>
<mvt:assign name="l.success" value="miva_json_decode( s.callvalue, l.settings:review_data)" />
<mvt:comment>The first request sets the context for how many pages we have, it's page 1</mvt:comment>
<mvt:assign name="g.rr_rating_count" value="l.settings:review_data:data:rating:count" />
<mvt:assign name="g.rr_rating_score" value="rnd(l.settings:review_data:data:rating:scor e, 2)" />
<mvt:comment>Store the rating and review count in custom fields</mvt:comment>
<mvt:item name="customfields" param="Write_Product_Code( g.code, 'reviewcount', g.rr_rating_count )" />
<mvt:item name="customfields" param="Write_Product_Code( g.code, 'reviewstars', g.rr_rating_score )" />
<mvt:assign name="g.rr_review_user" value="l.settings:review_data:data:pagedReviews:da ta:author:username" />
<mvt:assign name="g.current_page" value="1" />
<mvt:assign name="g.total_pages" value="l.settings:review_data:data:pagedReviews:me ta:pagination:total_pages" />
<mvt:assign name="g.results" value="'<ul>' $ asciichar( 13 ) $ asciichar( 10 )" />
<mvt:comment>the end of the first call</mvt:comment>
</mvt:call>


<mvt:while expr="g.current_page LE g.total_pages">
<mvt:assign name="g.api_request" value="'https://api.resellerratings.com/v1/product/reviews/full.json?sku=' $ g.code $ '&per_page=100' $ '&page=' $ g.current_page" />
<mvt:call action="g.api_request" method="'GET'" headers="'Authorization:Bearer ' $g.rr_token $ asciichar( 13 ) $ asciichar( 10 )">
<mvt:assign name="l.success" value="miva_json_decode(miva_html_strip( s.callvalue),''), l.settings:review_data)" />
<mvt:assign name="g.rr_rating_count" value="l.settings:review_data:data:rating:count" />
<mvt:assign name="g.rr_rating_score" value="rnd(l.settings:review_data:data:rating:scor e, 2)" />

<mvt:foreach iterator="reviews" array="review_data:data:pagedReviews:data">
<mvt:comment>Do stuff with results, just getting the review users name here</mvt:comment>
<mvt:assign name="g.rr_review_user" value="l.settings:reviews:author:username" />
<mvt:assign name="g.results" value="g.results $ '<li>' $ g.rr_review_user $ '</li>' $ asciichar( 13 ) $ asciichar( 10 )" />
</mvt:foreach>
</mvt:call>
<mvt:assign name="g.current_page" value="g.current_page + 1" />
</mvt:while>


Example of bad JSON:
{
"id": 505117,
"author": {
"username": "Victor R F",
"location": null
},
"date": "2016-09-27 23:41:23",
"title": "This product is really good. Every CPAP user should have one.",
"text": "I normally do not write reviews. Don't like nor want to. Will make an exception here. It woks as described, is easy to install and it really takes care of the cleaning and bacterial issues a normal CPAP machine has. I had certain respiratory problems before I used this machine, in spite of hand washing the mask every day and soaking the rest of the system in water and vinegar for 12 hours plus once every 3 weeks. Those issues are now gone completely. <br /><br />The unpleasant smell that is present after the use of the system and that some reviewers complain about, does disappear by turning it on a few times for a few minutes after the daily cleaning. One of those purchases that is worth every penny paid, in spite of the price. ",
"rating": {
"score": 5,
"min": 0,
"max": 5
},
"user_recommend": null,
"reply": {
"text": null,
"date": null,
"vote_yes": 0,
"vote_no": 0
}
}

Any help greatly appreciated.
Paul

[Gordon - Phosphor Media]

You can use miva_html_strip() to remove any html tags:
http://www.mivascript.com/item/mivas...tml_strip.html

[paul.oreilly]

I tried:
<mvt:assign name="l.success" value="miva_json_decode(miva_html_strip( s.callvalue),''), l.settings:review_data)" />

but that complains I need to supply the other parameter. Which I tried as well, but still didn't compile. Is there an example of how to use this method?

[Gordon - Phosphor Media]

Typo, try this instead:
<mvt:assign name="l.success" value="miva_json_decode(miva_html_strip( s.callvalue, '' ) ), l.settings:review_data)" />

your code put the second parameter into the decode function instead of the html_strip function.

[Bruce - PhosphorMedia]

If that doesn't work, assign the 'null' value of strings to remove to a variable:

<mvt:assign name="g.source" value="'This has html <b> HERE </b>'"/>
<mvt:assign name="g.tags" value="''"/>
<mvt:assign name="g.result" value="miva_html_strip(g.source, g.tags)"/>

SMT sometimes isn't has 'forgiving' with its syntax. One kind of gets into a habit of always using variables inside SMT functions.

[paul.oreilly]

Thanks for the help guys - I got the method to work, but unfortunately it's not striping because it fails at the request level. I should have thought of that - it doesn't get to the point of downloading JSON records that have <br> tags in it, so the request is skipped. I guess I'll reach out to the api vendor and see if they can't sanitize it.

thanks again,
Paul

[paul.oreilly]

HI, I'm resurrecting this old thread, hoping someone can help. I think there are some issue with how mvt:call works, and how json_decode expects data to be formatted.

Using mvt:call i'm hitting an external Reviews API and json_decoding the results. All works great unless someone has inserted <br> tags in their review - this make json_decode choke, and the process terminates.. I've tried miva_html_strip on the call value:

<mvt:assign name="g.tags" value="''"/> (i've also tried setting this to null as suggested above)
<mvt:assign name="g.sanitized" value="miva_html_strip( s.callvalue, g.tags )" />
<mvt:assign name="l.success" value="miva_json_decode(g.sanitized , l.settings:review_data)" />

this doesn't work, I think because the br tag is contained in a json string, as in the br tags are not escaped. So i'm wondering if miva_html_strip() ignores the tags, but that does seem odd.Or I'm guessing you can't use this function directly on the call object?

I also tried to deal with the tags based on an example in the docs:

"Simple MvCall Example: This example skips the tags and just returns the text."
<MvCALL ACTION = "g.url" METHOD = "GET">
<MvIF EXPR = "{ s.callobjecttype EQ 'tag'}">
<MvCALLCONTINUE>
</MvIF>
<MvASSIGN NAME = "l.text" VALUE = "{ l.text $ s.callvalue $ ' '}">
</MvCALL>

translated version below. if i leave in callcontinue, the file just hangs and times out, the data that gets' returned is not very large, so something else is going on . if i take it out callContinue, ishtml evaluates to true, so I'm able to see when there are tags in the call object, but i'm not able to skip them like the example above.

<mvt:if expr="s.callobjecttype EQ 'tag' AND tolower(s.callobjectelement) EQ 'br'">
<mvt:callcontinue />
<mvt:assign name="g.ishtml" value="'contains html - I found a br tag but i skipped it' " /> // this gets returned if tags, just for testing
</mvt:if>


Ideally, i'd like to be able to remove the tags, but if i could turn them into entities, that would work too. I tried :
encodeentities(s.callobjectelement) inside the IF statement above, but that didn't work. Any guidance greatly appreciated.
Paul

[Kent Multer]

Have you tried glosub() to remove the offending tags? You should be able to replace them with null strings, or maybe with spaces or some other white-space characters. Of course, it can only find specific tags, as opposed to looking for any HTML in the text. If you tell it to replace "<br>" then it won't find "<BR>" or <br/>" -- although you can write three separate glsub's if necessary.

HTH --

[paul.oreilly]

here's what I tried, no luck. should I be trying to alter s.callvalue? or am i doing this wrong?

<mvt:while expr="g.current_page LE g.total_pages">
<mvt:assign name="g.api_request" value="'https://api.resellerratings.com/v1/product/reviews/full.json?sku=' $ g.code $ '&per_page=100' $ '&page=' $ g.current_page" />
<mvt:call action="g.api_request" method="'GET'" headers="'Authorization:Bearer ' $g.rr_token $ asciichar( 13 ) $ asciichar( 10 )">
<mvt:assign name="g.tags" value="''"/>
<mvt:if expr="s.callobjecttype EQ 'tag' AND tolower(s.callobjectelement) EQ 'br'">
<mvt:assign name="s.callvalue" value="glosub('<br>', s.callvalue, ' ' )" />
<mvt:assign name="s.callvalue" value="glosub('<br/>', s.callvalue, ' ' )" />
<mvt:assign name="s.callvalue" value="glosub('<BR/>', s.callvalue, ' ' )" />


<mvt:assign name="g.ishtml" value="'contains html - I found a br tag but i skipped it' $ asciichar( 13 ) $ asciichar( 10 )" />
</mvt:if>


<mvt:assign name="l.success" value="miva_json_decode(s.callvalue , l.settings:review_data)" />

[Kent Multer]

Since you mentioned that the br tags are contained inside JSON text, it may be that mvt:call doesn't recognize them as tags. You might try applying that glosub() to the non-tag pieces of text.

[paul.oreilly]

i think mvt:call IS seeing the tags, and that's whats causing the problem after looking at the docs... If i check for <mvt:if expr="s.callobjecttype EQ 'tag' AND tolower(s.callobjectelement) EQ 'br'"> it let's me know there's a tag in that sea of text. And then that's what messes up json_decode, because i think the returned value is not text. I'm going to try saving out everything to a flat file and take a look. Part of the issue is that there are multiple pages in the request.

Happy Thanks Giving Kent, and All!

and thanks for your help.
Paul

[paul.oreilly]

So miva_html_strip() was the answer, I was just doing it in the wrong place.

<mvt:while expr="g.current_page LE g.total_pages">
    <mvt:assign name="g.api_request" value="'https://api.resellerratings.com/v1/product/reviews/full.json?sku=' $ g.code $ '&per_page=100' $ '&page=' $ g.current_page" />
        <mvt:call action="g.api_request" method="'GET'" headers="'Authorization:Bearer ' $g.rr_token $ asciichar( 13 ) $ asciichar( 10 )">
                <mvt:if expr="s.callobjecttype EQ 'tag' AND tolower(s.callobjectelement) EQ 'br'">
                    <mvt:comment>
                    none of these work on the call value... will identify the tag, but you can't alter it here.
                    <mvt:assign name="g.ishtml" value="miva_html_strip( s.callvalue, ' ' )" />
                    <mvt:assign name="g.return_value1" value="glosub( s.callobjectelement, s.callobjectelement, 'test1' )" />
                    <mvt:assign name="g.return_value2" value="glosub( s.callobjectelement, '<br />', 'test2' )" />
                    <mvt:assign name="g.return_value3" value="glosub( g.raw_results , 's.callobjectelement', 'test3' )" />
                    <mvt:assign name="g.return_value4" value="glosub( s.callvalue , 'br', 'test4' )" />
                    </mvt:comment>
                </mvt:if>

            <mvt:assign name="g.raw_results" value="g.raw_results $ s.callvalue" />
            <mvt:assign name="g.allowed_tags" value="''" />
            <mvt:comment>works here... </mvt:comment>
            <mvt:assign name="g.raw_results" value="miva_html_strip( g.raw_results, g.allowed_tags )" />
        </mvt:call>
    <mvt:assign name="g.current_page" value="g.current_page + 1" />
</mvt:while>
<mvt:assign name="l.success" value="miva_json_decode(g.raw_results , l.settings:review_data)" />
<mvt:eval expr="decodeattribute(glosub(miva_array_serialize(l.settings:review_data), ',', '<br>'))" />