Corrupted product descriptions with .csv product imports

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • becker
    Mini Mivite

    • Nov 2013
    • 47

    #1

    Corrupted product descriptions with .csv product imports

    After some searching I found this similar thread : http://extranet.mivamerchant.com/for...tions+changing

    Now, my issue is the exact same thing. When some (and only some, which makes the problem difficult to track down) products are imported, the descriptions are being cut off at a quotation mark. Usually (possibly always) it is the beginning that is cut off, and then a quotation mark following where it was cut off and the rest of the description is uploaded. I've been digging into this issue and examining the .csv files produced by excel in a simple text editor, but I can't notice any issues with the encoding. I could very easily have missed them though.

    Then it came to me that perhaps someone else uploading files is using a different .csv format, say .csv for MS-DOS or .csv for Mac. Could this possibly be causing issues due to the Miva import function expecting a specific encoding and receiving a different one?

    Any other ideas on what the issue could be? This is really starting to cause a problem as every day we find a few more descriptions that have been corrupted or changed. As an additional note for debugging purposes, I've noticed that quotation marks in the description are often getting doubled when the "bug" occurs. And I don't mean they are being doubled in the .csv text file; I understand that they are just being escaped when "" appears in the .csv text file. I mean that they are being doubled in the actual text uploaded to the product database. This indicates to me an escaping issue somewhere, but I can't seem to figure out why it would be happening.

    Also, if you see post seven in the other thread (http://extranet.mivamerchant.com/for...904#post385904), jsdva notes that descriptions were spilling over into multiple columns. The fields directly next to the descriptions are fine though, so this doesn't seem likely to me.
    Last edited by becker; 02-13-14, 04:39 PM.
  • Bruce - PhosphorMedia
    Developer Partner











    • Mar 2006
    • 11256

    #2
    Re: Corrupted product descriptions with .csv product imports

    If you are creating the spreed sheet yourself, then exporting them to a text file, you should probably use the Legacy Import tool. The new Import/Export tool uses "," as a delimiter and encloses all text (qualifier) with ". So, a " should actually be """.

    At least that's what I've seen...I always use the legacy cause that's what I grew up with :)
    Bruce Golub
    Phosphor Media - "Your Success is our Business"

    Improve Your Customer Service | Get MORE Customers | Edit CSS/Javascript/HTML Easily | Make Your Site Faster | Get Indexed by Google | Free Modules | Follow Us on Facebook
    phosphormedia.com

    Comment

    • becker
      Mini Mivite

      • Nov 2013
      • 47

      #3
      Re: Corrupted product descriptions with .csv product imports

      Generally I am exporting the spreadsheet and making changes to it, then importing it. Saving a .csv in excel automatically escapes all quotes, so I don't see how this could be the problem.
      Last edited by becker; 02-13-14, 05:00 PM.

      Comment

      • Bruce - PhosphorMedia
        Developer Partner











        • Mar 2006
        • 11256

        #4
        Re: Corrupted product descriptions with .csv product imports

        You might want to open up that .csv file in a text editor and double check that assumption. (Not an excel expert, but know enough not to trust it to do what you expect.)

        There might be some option in excel that got set incorrectly or maybe excel being excel.

        -Bruce
        Bruce Golub
        Phosphor Media - "Your Success is our Business"

        Improve Your Customer Service | Get MORE Customers | Edit CSS/Javascript/HTML Easily | Make Your Site Faster | Get Indexed by Google | Free Modules | Follow Us on Facebook
        phosphormedia.com

        Comment

        • becker
          Mini Mivite

          • Nov 2013
          • 47

          #5
          Re: Corrupted product descriptions with .csv product imports

          I've definitely spent a good amount of time looking over the .csv files in a text editor, looking for encoding and escaping mistakes, but I have been unable to find any mistakes. I still think it's very possible the issue might be something in excel's automatic changes, but I'm not sure what it is.

          Comment

          • Gordon - Phosphor Media
            Developer Partner


            • Mar 2006
            • 448

            #6
            Re: Corrupted product descriptions with .csv product imports

            If you haven't tried this already, you might try importing a small subset. Divide and conquer. When you get ONE record that reliably produces this behavior, then examine it with the microscope.

            (I've spent way too many hours going through large files looking for a tiny character difference)
            Gordon Currie
            Phosphor Media - "Your Success is our Business"

            Improve Your Customer Service | Get MORE Customers | Edit Any Document Easily | Free Modules | Follow Us on Facebook
            phosphormedia.com

            Comment

            • becker
              Mini Mivite

              • Nov 2013
              • 47

              #7
              Re: Corrupted product descriptions with .csv product imports

              Coming back to this.

              I'm still having this issue, and I decided to have another go at it today. I've done as Gordon said, and I attempted to narrow down to one product causing the issue. Here's where I'm seriously stumped though. I have a csv with 131 rows, one of which is a header row. The very last row has its description cut off every time I try this import (it is consistently at the same location). If I remove any row between the 1st and the 131st, the problem goes away. This is one of the strangest bugs I've ever encountered in any software or programming problem, and I'm seriously confused. I've tried various different ways to debug this issue, including opening the file in various csv browsers, which all work fine.

              I also have another spreadsheet that includes a large number of correct descriptions. Now, when I import this one, it generates four corrupted or cut off descriptions. The next time I import the same file, the number jumps to ten, then seventeen, and so fourth.

              One final bit of information, if I change the category code that I am importing the products to, the location in the description that it is cut off at changes. This may be true for other things other than the category code as well, but I'm not sure.

              As a CS student, I know this likely sounds incredibly ridiculous and impossible, but please bear with me here... I would appreciate any ideas on this if you have them.

              Comment

              • Rick Wilson
                Uber Mivite











                • Aug 2007
                • 10120

                #8
                Re: Corrupted product descriptions with .csv product imports

                I would bet money this has to do with either using the PR7 Imports along with Excel, or using PR8 Imports and stripping those oh so special Excel characters, ultimately causing unexpected breaks in the field.

                Have you sent a file to support?
                Thanks,

                Rick Wilson
                CEO
                Miva, Inc.
                [email protected]
                https://www.miva.com

                Comment

                • becker
                  Mini Mivite

                  • Nov 2013
                  • 47

                  #9
                  Re: Corrupted product descriptions with .csv product imports

                  Originally posted by Rick Wilson View Post
                  stripping those oh so special Excel characters, ultimately causing unexpected breaks in the field
                  I'm not sure it was Excel characters in particular, but it was definitely some set of special characters causing the problem. I couldn't figure out which they were, so instead I just removed anything special...
                  Code:
                  sed -i "s/[^[:print:]]//g" filename.csv
                  Seems to have taken care of the problem :D
                  Last edited by becker; 08-18-14, 02:01 PM.

                  Comment

                  Working...
                  X