Cloudflare CDN Replacement

[alphabet]

Thank you.

[ILoveHostasaurus]

I'd recommend posting in the MivaScript / Empresa channel on the apitype question as I'm not familiar with that variable. For the forwarding headers, a visitor can send any header if it's otherwise valid from a protocol standpoint, so x-forwarded-for, etc. generally should not be trusted unless they've come from a trusted IP. For sites with us that have Cloudflare active, we only trust that header if it has come from a Cloudflare IP. However, at the Miva Merchant / script level, you would not know if it's trustworthy, so you should only rely on the MivaScript remote IP variable, as that will always be either the actual IP or a substitution from a trusted forward header.

[alphabet]

@ILoveHostasaurus

What is s.apitype used for and does it have cardinals?

Is s.apitype used to issue a server header response?

Does uri.mvc issue s.apitype of 'Redirect' for cannonical urls?

And somewhat related:

I've sometime notice bot spam coming through proxys. Can s.HTTP_X_FORWARDED_FOR and s.HTTP_X_REAL_IP be used to investigate them further?

[ILoveHostasaurus]

What about other requests for page resources such as images, scripts, fonts, etc?

Those shouldn't matter because the page content should already be loading those with appropriate URI's, or relative. They'll also end up cached by Cloudflare as static content.

[alphabet]

Here is Cloudflare rule that may need to be removed:

http://*example-domain.com/*
Always Use HTTPS

Here is .htaccess code that may need to be removed:

# force https
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://www.example-domain.com/$1 [R=301,L]


# force www
RewriteCond %{HTTP_HOST} !^www.example-domain.com
RewriteRule ^(.*)$ https://www.example-domain.com/$1 [R=301,L]

Here is preprocessing code to add above html profile

Following code makes the assumption that g.domain:name contains 'www'.
s.documenturl has not been transcribed or redirected.
X-FORWARD headers are not always available.

<mvt:assign name="g.wrong_domain_name" value="'www.wrong_domain_name'" />
<mvt:assign name="g.documenturl" value="s.documenturl" />
<mvt:assign name="g.asciichar_58" value="asciichar(58)" />
<mvt:assign name="g.flags" value="'ltrim,insensitive'" />
<mvt:assign name="g.has_secure_protocol" value="miva_splitstring( g.documenturl, g.asciichar_58, g.proto, g.flags )" />
<mvt:if expr="('https' EQ tolower(g.proto[1])) OR (g.wrong_domain_name NE g.domain:name)">
<mvt:assign name="g.REQUEST_URI" value="s.REQUEST_URI" />
<mvt:assign name="g.href" value="'https://' $ g.domain:name $ g.REQUEST_URI" />
<mvt:assign name="g.ok" value="miva_output_header( 'Status', '301 Moved Permanently' )" />
<mvt:assign name="g.ok" value="miva_output_header( 'Location', g.href )" />
<mvt:assign name="l.void" value="miva_output_flush()" />
<mvt:exit>
</mvt:if>

ILoveHostasaurus

This means the only things that make it through to ever hit that template code will be valid targets, which the code ignores, while anything else URI Management will pick up first

What about other requests for page resources such as images, scripts, fonts, etc?

[jsdva]

Excellent! I'll sent the site owner to you. You are indeed the best!!

[ILoveHostasaurus]

Sure thing. I'm certainly no developer, but I know the three things the code needs to look at already exist as environment variables, so I suspect it would not be much of a programming job for an appropriate person to take on.

[jsdva]

Okay, I get it. Myself, I wouldn't attempt it, but I can send the site owner to you with reference to this post. The site does use URI Management, it seems like Option 2 is very doable.

Thank you, David for making the puzzle so much easier to understand! I am grateful!!

[ILoveHostasaurus]

Ah okay. It can be very complicated to accomplish only a single redirect for every possible scenario when you have so many variables:

• Old product/category/page code to new code, or discontinued prod/cat to replacement
• Adding or removing www if the site should only use one version
• Adding https if the request came in insecurely
• Shifting traffic to a new domain

Option 1 to fix all of this would be the more painful one from a labor cost perspective:

• You need the old domain to know about every out of date URL that had ever been used at the time the new domain took its place, because each one of those will need to be redirected to the proper new URL on the new domain. You can't just send it to the new domain because it would then the new domain would issue a second redirect to correct the out of date target. However, anything not meeting this criteria, i.e. up-to-date targets, can be sent to the new domain as-is with corrected (if necessary) www vs non-www and https-on, because the redirect will be a complete correct target not failing any of those other tests. Assembling such a list of requests and targets manually to load, and if it's past the Cloudflare bulk redirect limit, then higher costs or alternative solutions would be required.
  
  It sounds like you've performed a partial version of this if Cloudflare has a bulk redirect set up for known category codes to the new domain, correcting the other things in the process.
  
  If you do have all of these redirects already built, or plan to go this route regardless, we could potentially set up the old domain as a standalone website that Cloudflare simply proxies traffic to, and place all of those redirects in a redirect map. That would let you keep Cloudflare in front, but also not be subject to the bulk redirect quantity limit as we'd handle it on our side.
• Anything not caught by the above, the old domain can redirect to the new domain as-is because we'll assume it is either a correct up-to-date URL or a 404. Or if we're doing this on our side, same thing applies, anything not caught in the redirect map gets a pattern redirect.
• The new domain's copy of Miva Merchant uses normal URI management going forward, AND its htaccess or nginx rules can correct www vs non-www and https. The only time more than one redirect should occur is if someone were accessing a target that had become after the new domain launched, AND with the wrong www vs non-www or no https, but there should be no reason for that to ever happen unless they were doing so intentionally or typing the entire thing in manually, so I would not consider that as a concern.

Option 2; much easier, if the store had been using URI Management:

• Have both the old and new domain both point to the same "site", i.e. copy of Miva Merchant.
• Remove any correction for https and/or www vs non-www from any Cloudflare, nginx or htaccess configs, so all those requests hit Miva Merchant as-is.
• Add code to the Miva Merchant global header template that, before ANY content is served, it looks at if the request was insecure or secure, if the request is for the wrong domain, and if the request has the wrong www vs non-www. If it sees any of those being wrong, it issues a redirect where the target is the same URI which was originally requested, but on the new domain, with https, with the correct hostname. This lets a few lines of code solve the problem of a single redirect for every scenario except out of date targets, which it would not touch at all, and which come next.
• In a store, URI Management comes before any template code execution, so the first action will be to see if the inbound request needs to be redirected to a canonical URI. If it does, none of those other things matter because the store is going to redirect the visitor to the proper target, and the full URL sent will already correct https, www, domain name. This means the only things that make it through to ever hit that template code will be valid targets, which the code ignores, while anything else URI Management will pick up first.

[jsdva]

Yeah. We did a redirect for the www in htaccess, which cured the www, but the category pages had 2-3 redirects in cloudflare, so the SEO guy said the redirect should not be there, and a cloudflare support person fixed the category redirects.

Don't know if the multiple redirects on the category was because of mapping in some other way that had to be done for the switch to the new domain...I didn't have anything to do with that.

[ILoveHostasaurus]

Removing the www from a URL should be possible with a pattern-based redirect though; Cloudflare allows those too, including with variable substitution.

[jsdva]

I see. Well, we will consider all of this. It is looking like Cloudflare is the most economical and easier to deal with. Thanks for all of your input and patience, David!

[ILoveHostasaurus]

I believe it's also significantly more expensive. They and Akamai also require a more involved setup to ensure the wrong content does not end up cached. Their model differs from Cloudflare's fewer number of large data centers by having higher numbers of small points of presence, to get cached content as close as possible to visitors and reduce delivery times. That can often have benefit, but we've also found at times it can be problematic to troubleshoot, as we've had issues where one node on their network had origin connectivity issues, preventing it from loading uncached content. When you have only a small region of the country having issues, with multiple entities involved, it can produce a longer and more frustrating time to resolution.

[jsdva]

I'm going to pick your brain just this last time. What about Imperva...comparability in costs to Cloudflare?

[ILoveHostasaurus]

We aren't familiar with their feature set at this point in time, as what they were offering at the time was a solution to image caching, but Cloudflare then came along with a far greater feature breadth of features at a similar price point, and it did not make sense to continue. We typically only find customers using one of Cloudflare, Imperva, and Akamai, so most of our knowledge is around those.