 | | |
|
pfoley
User
Dec 4, 2002, 9:19 AM
Post #1 of 9
(2905 views)
Shortcut
|
|
Expose function for publish listing(s)
|
Can't Post
|
|
Hi folks,
The host that my client is using (hostway) has a fairly short timeout (60 seconds). This is causing my publish listings to choke at about 950 listings - I want to be able to take the number of listings quite a bit past that (about 1800).
As a worst case scenario, I can divide the database into chunks (thank you for providing rm-db.cgi) and process the listings from each chunk, but that's obviously a very clunky solution (I can also change hosts, but that's a huge pain). Is there any way you could expose the function that publishes listings - or better yet a single listing? That way, I can just publish the listings from a shell program (or use $| = 1 and show progress for each listing, which I suspect will work).
I know it's a lot to ask, but if that function is nicely partitioned inside rm.cgi anyway ...
Please let me know if that is possible.
Thanks,
Patrick Foley
Aptica Consulting
PS - if YOU want to try using $| = 1 to show some simple progress in a web page and then redirect back to rm.cgi, that would obviously be even cooler. If you give me the function, I may be able to demo a useful UI enhancement for you (although it might not solve my immediate problem).
(This post was edited by pfoley on Dec 4, 2002, 9:20 AM)
|
|
| |
|
pfoley
User
Dec 5, 2002, 12:52 PM
Post #2 of 9
(2882 views)
Shortcut
|
|
Re: [pfoley] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Here's an example of what I'm talking about with $| = 1:
Test (click here) EXAMPLE
This will run for 20 minutes or more without timing out, because $| = 1 causes each print to be flushed. Without that, this script would timeout within 60 seconds on hostway (frequently 3 minutes on other hosts) - and you would never see the output - just an error message after it timed out. This technique is useful for showing a "progress bar", because results are shown immediately and the web page just keeps growing. When you are done processing, just refresh or redirect the page.
---
The code:
---
#!/usr/bin/perl
use strict;
use CGI qw(:standard escapeHTML);
use CGI::Carp;
use LWP::UserAgent;
print header();
print start_html( -title => 'Test',
-bgcolor => '#FFFFFF',
);
$| = 1;
print "<h1 id='counter'>Working</h1>";
my $n = 1;
while (1) {
select(undef, undef, undef, 1); # the '1' can be a fraction, btw
print "<script>document.all.counter.innerHTML = $n</script>\n";
$n++;
}
print h1("Done");
print end_html();
1;
|
|
| |
|
Dave
Staff
/ Moderator
Dec 5, 2002, 12:55 PM
Post #3 of 9
(2880 views)
Shortcut
|
|
Re: [pfoley] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Hmm, I'm trying to think of the simplest solution for this. What if you could call publish from the shell, perhaps with a cron job or something. Would that work for you? Is there limits on how long you can run a perl process from the shell for?
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
Dave
Staff
/ Moderator
Dec 5, 2002, 12:57 PM
Post #4 of 9
(2879 views)
Shortcut
|
|
Re: [pfoley] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Is the problem a 'browser timeout' problem or a 'server kills your process if it runs longer than N seconds' problem? Or both? : ) We could probably have it output something to avoid browser timeouts if that's the problem.
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
pfoley
User
Dec 5, 2002, 3:13 PM
Post #5 of 9
(2875 views)
Shortcut
|
|
Re: [Dave] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Shell is great - I'm currently using LWP, simply because HTTP is the only thing exposed to me. This is approximately what interactivetools asked me to do:
my $ua = new LWP::UserAgent;
$ua->agent("My Agent/0.1 " . $ua->agent);
my $req = new HTTP::Request GET => 'http://mysite.com/rm/exec/rm.cgi?login=1&id=RM&pw=mypass&setup_publish_listing_index=1';
my $res = $ua->request($req);
Ultimately what I am trying to do is get a function that I can call from within perl - a separate command-line program is also fine. There is no (practical) time restriction on how long I can run a shell process. cron is a pain currently, but I can work around that for now.
|
|
| |
|
pfoley
User
Dec 5, 2002, 9:17 PM
Post #6 of 9
(2867 views)
Shortcut
|
|
Re: [Dave] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Dave wrote:
Is the problem a 'browser timeout' problem or a 'server kills your process if it runs longer than N seconds' problem? Or both? : ) We could probably have it output something to avoid browser timeouts if that's the problem.
---
I'm pretty sure it's the server (though it may be a combination of browser/server). Regardless, it only kills the process if it runs longer than N seconds AND DOESN'T OUTPUT ANYTHING (kills it after 60 minutes, regardless). Therefore, outputting something would solve the problem (and provides user feedback for a long running process). That's what I was ultimately suggesting above. With Perl and apache or IIS, nothing will be output unless you use $| = 1 at the top of the program, however.
Note that changing rm.cgi to provide feedback is the best solution for me, but it obviously has a certain release schedule. I have to solve my current problem within about 3 days, so an interim solution may be necessary. HOPEFULLY, you can help me with one, but if not, I'll have to do something kludgy.
Incidentally, if you want to add a simple progress indicator to rm.cgi, the simplest way would be to do something like this with a growing ....... and then just make it disappear:
$| = 1;
# http header stuff
print '<h1 id="progress">';
# loop {
print '.';
# process somewhere between 10-100 listings
# }
print '</h1><script>document.all.progress.innerHTML = "";</script>';
# do normal page display
|
|
| |
|
Dave
Staff
/ Moderator
Dec 6, 2002, 11:17 AM
Post #7 of 9
(2855 views)
Shortcut
|
|
Re: [pfoley] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
Ok, I'll ask Dlo (one of our programmers) to get in touch with you and figure something out. We'll could probably start by outputting something so the browser won't timeout and see if that solves the problem.
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
pfoley
User
Dec 10, 2002, 6:20 PM
Post #8 of 9
(2821 views)
Shortcut
|
|
Re: [Dave] Expose function for publish listing(s)
[In reply to]
|
Can't Post
|
|
I just gotta say WOW! That was really great service. Dave Lo put in some extra effort to come up with a creative solution to my problem. Thank you SO MUCH.
I am truly impressed - nay, AMAZED - at that kind of service. You guys rock!
Patrick
|
|
| |
|
dlo_itools
Staff
Dec 11, 2002, 8:39 AM
Post #9 of 9
(2817 views)
Shortcut
|
Thanks for the kind words!
For those with large numbers of listings who dread pressing the Publish All Listing button, the next upgrade will be a must-have -- we've significantly improved the performance of Publish All Listings (2x - 8x faster!)
/Dave Lo
|
|
| |
|
