Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4

    Cfflush - sometimes stops flushing in middle of loop


    I have a page where a query resultset of various documents to be emailed or faxed to clients is being looped over. These can take up to 10 seconds to process, so I flush the progress to the browser so the user can see that document x of y total has been sent (or has failed). Y is usually 200, and never more than that.
    This is infrequent, but sometimes the flush just stops flushing to the browser during a random iteration in the loop, even though the process is still running in the background and runs all the way to completion. I'm using CF8, viewing on IE9 (I have also viewed on IE8; I have yet to see this happen using IE8. So far has only happened on IE9).
    I use divs to mimic table rows, as IE will not flush actual table rows one at a time; it waits until the whole table is done and only then renders it to the screen. I also set a very low interval so it doesn't wait too long before flushing the next bit of html. My code basically looks like this:

    Code:
    <cfset objDocSender = createObject(some cfc)>
    <cfflush interval=10>
    <cfoutput>
       Processing documents...#repeatString(" ", 100000)#  <!---this to force IE to render--->
    </cfoutput>
    <cfoutput>
       <div id="row">
          <div id="cell" class="header">Work Request ID</div><div id="cell" class="header">Delivery Method</div><div id="cell" class="header">Success</div><div id="cell" class="header">Progress</div>
       </div>
    </cfoutput>
    
    <cfloop query="qryDocsToSend">
       <cftry>
          <cfset blnSuccess['#qryDocsToSend.DocID[currentrow]#'] = objDocSender.sendDocument(#qryDocsToSend.DocID[currentrow]#, '#qryDocsToSend.Delivery_Method[currentrow]#')>
          <cfcatch>
             <cfset blnSuccess['#qryDocsToSend.DocID[currentrow]#'] = false>
          </cfcatch>
       </cftry>
     
       <cfoutput>
          <div id="row"<cfif NOT blnSuccess['#qryDocsToSend.DocID[currentrow]#']> class="red"</cfif>>
             <div id="cell">#qryDocsToSend.DocID[currentrow]#</div><div id="cell">#qryDocsToSend.Delivery_Method[currentrow]#</div><div id="cell"><cfif blnSuccess['#qryDocsToSend.DocID[currentrow]#']>Y<cfelse>N</cfif></div><div id="cell">#qryDocsToSend.currentrow# of #qryDocsToSend.recordcount# processed.</div>
          </div>
       </cfoutput>
    
       <cfif blnSuccess['#qryDocsToSend.DocID[currentrow]#'] NEQ true>
          <cfset intNumFailed = intNumFailed + 1>
          <cfset lstFailedIDs = listAppend(lstFailedIDs, '#qryDocsToSend.DocID[currentrow]#')>
       </cfif>
    </cfloop>
    
    <cfoutput>
       <br>
       <div style="clear:both;">
          <br>#qryWRsToSend.recordcount - IntNumFailed# record<cfif qryWRsToSend.recordcount - IntNumFailed NEQ 1>s were<cfelse> was</cfif> sent.<br>
          <cfif intNumFailed GT 0>
             #IntNumFailed# failed.<br>
          </cfif>
       </div>
    </cfoutput>

    Has anyone else experienced this kind of behavior with cfflush?

    Thanks,
    Christophe
  2. #2
  3. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    I haven't heard of this, but flushing every 10 bytes seems really, really low. That's a LOT of flushed updates to the browser. What if you change it to every 1k or 10k or something...any difference?
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    I haven't heard of this, but flushing every 10 bytes seems really, really low. That's a LOT of flushed updates to the browser. What if you change it to every 1k or 10k or something...any difference?
    Cfflush and IE behave oddly together. From IE8 onward, there's a buffer in the browser that will prevent data from being flushed until IE feels there's enough data to justify a flush. (That's why I have the #repeatString(" ", 100000)# at the end of the first line; this is a known hack to force IE to flush by creating a huge amount of whitespace.)

    With the interval at 10, it still doesn't flush every 10 bytes. It usually flushes out 2-3 rows and change at a time, which is well over 10 bytes of info. I arrived at "10" by experimenting until it was flushing frequently enough. With the interval at 1,000 or 10,000, the period between flushes would be way too long.
  6. #4
  7. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    If IE has a buffer on the client side, then that may be preventing the RENDERING of the incoming data. But the server has nothing to do with any client-side buffering. It doesn't care what browser is in use, it's just writing to the output stream. If you have the flush interval set to 10 bytes, then it is flushing to the output steam every 10 bytes. What IE does with that incoming data, I have no idea. So I'm not sure if it's the server-side flush interval being so low, or whatever client-side buffering IE is doing, or some combination of the two.
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    If IE has a buffer on the client side, then that may be preventing the RENDERING of the incoming data. But the server has nothing to do with any client-side buffering. It doesn't care what browser is in use, it's just writing to the output stream. If you have the flush interval set to 10 bytes, then it is flushing to the output steam every 10 bytes. What IE does with that incoming data, I have no idea. So I'm not sure if it's the server-side flush interval being so low, or whatever client-side buffering IE is doing, or some combination of the two.
    That's a good point. What I was doing even before this looked something like:

    Code:
    <cfflush interval=10>
    
    <cfoutput>Processing Documents...#repeatString(" ", 100000)#</cfoutput>
    
    <cfoutput>(div that serves as my "table" header)#repeatString(" ", 100000)#</cfoutput>
    
    <cfloop query="qryDocuments">
        (code to send document to fax queue)
       <cfoutput>(show a row for this document, and overall progress)#repeatString(" ", 100000)#
       </cfoutput>
    </cfloop>
    
    <cfoutput>All done. X successful, Y failed.</cfoutput>
    All still with a single cfflush tag with interval=10. So, what I'm trying now is getting rid of that single cfflush tag, and instead putting a cfflush tag after every section I want to flush to the browser. (I also don't use quite as much whitespace inside the loop, since this is where most of the whitespace will be generated. This causes about 3 rows to display at a time - not too much waiting time for the user).

    Code:
    <cfoutput>Processing documents...#repeatString(" ", 100000)#</cfoutput>
    <cfflush>
    
    <cfoutput>(div that serves as my "table" header row)#repeatString(" ", 100000)#</cfoutput>
    <cfflush>
    
    <cfloop query="qryDocuments">
        (code to send document to fax queue)
       <cfoutput>(show a row for this document, and overall progress)#repeatString(" ", 10000)#<!---using less whitespace here--->
       </cfoutput>
       <cfflush>
    </cfloop>
    
    <cfoutput>All done. X successful, Y failed.</cfoutput>
  10. #6
  11. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    Honestly, I don't think many people rely on output buffer flushing nowadays. You'll generally see this done with AJAX. Shorter tasks would use a simple loading spinner (and I'd say 10 seconds would be fine for a loading spinner). Longer tasks require a server queue with the client making AJAX calls at some interval to determine the status and update a progress bar.

    So if you keep having trouble using the flush approach, you may want to think about changing to an AJAX approach. Flushing was always goofy and not really an ideal solution, which is why it has faded from general use in the web development world.
    Last edited by kiteless; June 7th, 2013 at 08:26 AM.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    Honestly, I don't think many people rely on output buffer flushing nowadays. You'll generally see this done with AJAX. Shorter tasks would use a simple loading spinner (and I'd say 10 seconds would be fine for a loading spinner). Longer tasks require a server queue with the client making AJAX calls at some interval to determine the status and update a progress bar.

    So if you keep having trouble using the flush approach, you may want to think about changing to an AJAX approach. Flushing was always goofy and not really an ideal solution, which is why it has faded from general use in the web development world.
    I agree - as though the limitations of not being able to use cflocation and other cf tags with cfflush aren't bad enough. And even with my code change, this display "lockup" continues to happen now and then. I'd like to understand the mechanics behind why the browser suddenly decides to stop flushing info to the display for no obvious reason, especially when subsequent iterations keep telling it to flush. It's not like it just skips a row and then resumes flushing on the next iteration. It simply freezes up, and displays none of the rest of the page's output. I'm surprised I don't see more complaints of this exact thing on the web.

    Either way, I know you've expressed this suggestion before. I'm familiar with ajax, and it's a good excuse to become thoroughly acquainted with jQuery over the next couple weeks. If there's a solution to this problem, I've a feeling that's the only place it's going to be found.
  14. #8
  15. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    My guess is still that it is tied to the flush interval being so low and/or whatever IE is doing to buffer the incoming data stream. This may be why you don't see more people with the issue.

    To confirm, I'd try raising the interval to something like 1k. I know you said it takes longer to render, but the point here would be to confirm that it works correctly. If it does, then you know it has to be tied to the low flush interval.

    I'd also test in Chrome and FF, to determine if it is also related specifically to IE.
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    My guess is still that it is tied to the flush interval being so low and/or whatever IE is doing to buffer the incoming data stream. This may be why you don't see more people with the issue.

    To confirm, I'd try raising the interval to something like 1k. I know you said it takes longer to render, but the point here would be to confirm that it works correctly. If it does, then you know it has to be tied to the low flush interval.

    I'd also test in Chrome and FF, to determine if it is also related specifically to IE.
    But if you saw, with my new code I'm not using the interval attribute.
  18. #10
  19. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    In that case, what do you see in FF or Chrome? Also, with all the whitespace you're pushing, what is the actual total amount of data being sent to the client? 10K? 100K? 1000K?

    You've got 200,000 spaces, plus another 10,000 for every loop iteration. The browser may just be flipping out due to the sheer size of the page. At 200 iterations, that's 2,200,000 spaces getting pushed into the page.
    Last edited by kiteless; June 7th, 2013 at 04:06 PM.
  20. #11
  21. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    In that case, what do you see in FF or Chrome? Also, with all the whitespace you're pushing, what is the actual total amount of data being sent to the client? 10K? 100K? 1000K?

    You've got 200,000 spaces, plus another 10,000 for every loop iteration. The browser may just be flipping out due to the sheer size of the page. At 200 iterations, that's 2,200,000 spaces getting pushed into the page.
    Originally Posted by kiteless
    In that case, what do you see in FF or Chrome? Also, with all the whitespace you're pushing, what is the actual total amount of data being sent to the client? 10K? 100K? 1000K?

    You've got 200,000 spaces, plus another 10,000 for every loop iteration. The browser may just be flipping out due to the sheer size of the page. At 200 iterations, that's 2,200,000 spaces getting pushed into the page.
    No need to test in FF or Chrome -- it's an internal app and they only use IE here. With the browser, it seems only a couple of people are having the display problem, so I'm wondering if they need to clear their cache or the problem otherwise has to do with some aspect of the browser on their individual machines.

    AFA the whitespace - yes, it is a lot of data, and not ideal, but if using cfflush I have to either use the interval attribute or use the whitespace to force IE to render the data visible to the user at timely intervals.

    I'm aiming at using jQuery and Ajax, but I'm not sure how best to integrate it with the current display design. In pseudocode, it works this way:

    Code:
    html output:
       "Processing documents, please wait"
       (Header Row)
       "Document ID | fax or email | Successfully Sent? | Progress"
    
    start loop of documents (200 total)
       Look for and gather this document's attachments
       If delivery method is Email
          Send the document and its attachments via Email
          Count this document as successfully sent
       Else if delivery method is Fax
           Try to send via fax
            If successful
               Count this document as successfully sent
            Else
               Count this document as failed
    
       html output:
          (this document's ID) | (this doc's delivery method) | ("Y" if successfully sent, "N" if Failed) | (rownum) "of" (total count) "completed."
    
    end loop
    
    html output:
          "All done." (successcount) "successful", (failcount) "failed."
    Any suggestions? The thing is, in this particular case it's a lot simpler to keep track of the info using local coldfusion variables defined on this page than it is to query the database periodically using ajax.
  22. #12
  23. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Actually, I think I have the solution. Doesn't necessarily require jQuery, but I think it would make the code a bit cleaner.

    What I'm thinking is that I won't run the loop on this page. Instead, I'll run the whole loop on another cfm page, using a single ajax call. On this cfm page, it simply receives a list of the document IDs that need to be looped through, so I can pass that list on to the processing page via ajax, and now I don't have to wait for the looping to complete before the user can see something on the screen.

    On THIS page, I use setInterval() to call a second ajax function every few seconds that makes a call to another cfm page, where it queries the DB to check on the progress and returns that info in the responseText. (Writing the queries will be a little more involved, but nothing too difficult).

    For simplicity's sake, all I'll retrieve or show the user is the current progress -- x of totalcount processed (or show percent completed, or use a visual progress bar).

    When I see that totalcount of totalcount has been processed, then I'll use clearInterval() to stop the periodic ajax calls, and I'll do one final query to get all the details for each document that was processed and display them to the user in a table.
  24. #13
  25. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    Originally Posted by Xophe
    No need to test in FF or Chrome -- it's an internal app and they only use IE here.
    I meant just to try and determine if it is IE causing the problem. If you test it in Chrome and it always works as expected, you know it has to be an IE issue.
  26. #14
  27. No Profile Picture
    Moderator

    Join Date
    Jun 2002
    Location
    Raleigh, NC
    Posts
    5,275
    Rep Power
    968
    Yep, this is pretty much what I meant by using AJAX and some sort of server side queue.

    Originally Posted by Xophe
    On THIS page, I use setInterval() to call a second ajax function every few seconds that makes a call to another cfm page, where it queries the DB to check on the progress and returns that info in the responseText. (Writing the queries will be a little more involved, but nothing too difficult).

    For simplicity's sake, all I'll retrieve or show the user is the current progress -- x of totalcount processed (or show percent completed, or use a visual progress bar).

    When I see that totalcount of totalcount has been processed, then I'll use clearInterval() to stop the periodic ajax calls, and I'll do one final query to get all the details for each document that was processed and display them to the user in a table.
  28. #15
  29. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    43
    Rep Power
    4
    Originally Posted by kiteless
    Yep, this is pretty much what I meant by using AJAX and some sort of server side queue.
    Originally Posted by kiteless
    Yep, this is pretty much what I meant by using AJAX and some sort of server side queue.
    My dev on this seems to be going well so far -- what I'm wondering though is how to figure out when the loop is done, so I can stop the periodic checks and inform the user.

    Right now I'm passing all 200 document IDs via ajax to the page that does the looping. When the periodic query shows that the number of docs processed (and whether a doc has been processed or not is determined by values in a table) equals the number of docs in the loop, then it does a clearInterval() and displays the details to the user.

    However, there's always the chance that a doc is skipped for some reason, in which case the relevant values in the table are not updated. If that happens, then the page has no idea when the looping is done, so clearInterval never happens and the periodic checks keep going.

    I'd much rather have some way of telling when the looping on the executing page has completed, so there's no chance of the page getting stuck. Right now it seems the only way to do this is, rather than pass all 200 doc IDs in a single ajax call, to pass them one at a time, which I'd rather not do.

    I'm thinking I probably need to use a server-side, shared scope variable to act as a counter on the looping page, and use ajax to check its value periodically, rather than checking the DB periodically.
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo