Announcement

Collapse
No announcement yet.

mvimport and application timeout

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    mvimport and application timeout

    I have some large flat files. One supplies an id style field that gets used for a filter expression. The second is a complete set of data. An mvimport chugs through the second file without application timeout. When I read the 1st file, apply the filter on the second file I run into an application timeout sometime after about 830 lines. This doesn't make any sense and I can't figure where the resources are being exhausted. This script is not a module.

    Code:
    MvImport 'thefirstfile' field-is-filterrecord
         MvImport  'thesecondfile' ( filterrecord EQ secondfilefield
              cycle through loaded lines that match filter expression
         /MvImport
    /MvImport
    I have also loaded the first file into a single column array so the import isn't embedded, the import still needs to be embedded inside an mvwhile or foreach loop. Either technique results in an application timeout.

    I guess I am looking for a way to reset the timeout if that is possible? In my pseudo code I guess it would need to be before the out loop cycles to its next import?

    There is another idea(just thought of) to keep one import loop and use the single column array and use a counter++ in the index[counter++] to grab the value of that position. Will that work? I'm skeptical atm.

    FYI: Some real numbers from the current dataset: 1st file has about 9400 records, the second file has about 65000 records and a couple dozen columns.

    Thanks,

    Scott
    Need to offer Shipping Insurance?
    Interactive Design Solutions https://www.myids.net
    MivaMerchant Business Partner | Certified MivaMerchant Web Developer
    Competitive Rates, Custom Modules and Integrations, Store Integration
    AutoBaskets|Advanced Waitlist Integration|Ask about Shipping Insurance Integration
    My T-shirt Collection is mostly MivaCon T-shirts!!

    #2
    Don't nest the 2 imports?

    Comment


      #3
      Why not add a meta refresh and when the system timeout variable gets down to 20 seconds refresh the import? Keep a log of where you're at or something and then start from where you left off. I do that all the time when there's a process that times out and it works great.
      M.A.D.* since 1997

      http://www.scotsscripts.com

      *miva application developers

      Comment


        #4
        I use about the same solution as Scott. A module can detect an approaching timeout, and stop itself about 5-10 seconds before that. It saves its status in a DB field, and writes some extra Javascript onto the Module_XXXX_Content page. This Javascript, in effect, causes the page to click its own Update button. Then the module can read the saved status and resume where it left off.
        Kent Multer
        Magic Metal Productions
        http://TheMagicM.com
        * Web developer/designer
        * E-commerce and Miva
        * Author, The Official Miva Web Scripting Book -- available on-line:
        http://www.amazon.com/exec/obidos/IS...icmetalproducA

        Comment


          #5
          Apologies for not mentioning this before. I have used the meta refresh technique before with success. What I was neglect about is that I will need to incorporate the script into an Import module. didn't want to deal with the meta refresh in admin, if it were even possible. Thank for the idea.

          I did just finish with a possible solution which seems to be working. I'll need to test some more.

          I did understand what Jonathan confirmed -- not nesting the imports, and I also assumed nesting the import in a foreach is a similar problem. So the idea I mentioned which creates the array from the first file that stores the filter value to search the records in the second file. IOW, I load the array, and without looping through the array, I add a counter to the import loop which the counter is an index to the array position. I seem to be loading all the records that match the filter.

          Scott
          Need to offer Shipping Insurance?
          Interactive Design Solutions https://www.myids.net
          MivaMerchant Business Partner | Certified MivaMerchant Web Developer
          Competitive Rates, Custom Modules and Integrations, Store Integration
          AutoBaskets|Advanced Waitlist Integration|Ask about Shipping Insurance Integration
          My T-shirt Collection is mostly MivaCon T-shirts!!

          Comment


            #6
            Does the data need to be scrubbed for special characters.

            Seems unusual to timeout at line 830 no matter the file size.
            http://www.alphabetsigns.com/

            Comment


              #7
              I would remove all lines prior to 825 and all lines after 835 and see if the import still fails.
              David Carver
              Miva, Inc. | Software Developer

              Comment


                #8
                Hey David,

                Apologies if there is non-sequitur info, since I've worked on the script a little since posing the questions.

                I think what you're suggesting is that if the data in the flat file is clean, there should be no reason the script should have an application timeout? I will certainly check this again. There had been some issues that I caught prior to this post, but it's worth weeding this out if there is an issue.

                Here's the catch. When I don't have the mvimport nested in a loop, it cycles through the input flat file without an issue. The reason I am embedding in a loop is that I need to filter for the value of an index type of field and only want to cycle through matching records. The outside loop is storing that index value.

                Maybe another question is: how do I find out what operation, exactly, is timing out?

                Scott
                Need to offer Shipping Insurance?
                Interactive Design Solutions https://www.myids.net
                MivaMerchant Business Partner | Certified MivaMerchant Web Developer
                Competitive Rates, Custom Modules and Integrations, Store Integration
                AutoBaskets|Advanced Waitlist Integration|Ask about Shipping Insurance Integration
                My T-shirt Collection is mostly MivaCon T-shirts!!

                Comment

                Working...
                X