 |
svsanchez
User
Jul 25, 2006, 4:05 PM
Post #1 of 23
(5605 views)
Shortcut
|
|
search.cgi eats all resources!
|
Can't Post
|
|
Hello guys, AM is great in almost every aspect, however I would like to know if there's a way of limiting the resources usage of search.cgi as it is causing my site major problems and V2 doesn't look as it will be ready any time soon. For some time it has been eating up resources, like today 86MB's of RAM and 99% usage constant for over 5 minutes. I had to chmod search.cgi to 644 as the site was horribly slow. Just after chmodding it it became very fast again, but of course if people click the "next" articles button they get an internal server error. I use google search for my site so users will still be able to make searches but not to navigate deeper into my categories pages with the "next" page links.
|
|
| |
|
ChetW
Staff
Jul 27, 2006, 4:40 PM
Post #2 of 23
(5582 views)
Shortcut
|
Hi,
Thanks for the post!
There are a few things that you can do to reduce memory usage:
- Create Archives this is one of the better ways to improve memory usage. You can create archives from inside the "Setup Options" under the "Archives" tab.
- You may also want to turn off "Auto Publish" to do this select "Setup Options" and then choose the "Server" tab. On the "Server" tab scroll down to the publish settings and set the Auto Publish setting off (uncheck the check box). This stops Article Manager from republishing its content every time one article's been changed/added.
- If you have archives or files that people no longer read, or you don't mind removing, you can remove them. That will also help.
Give these a try and just let us know is you have any other questions
Cheers,
Chet Woodside - Product Specialist
support@interactivetools.com
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
(This post was edited by Theo on Jul 31, 2006, 5:00 PM)
|
|
| |
|
svsanchez
User
Jul 27, 2006, 5:18 PM
Post #3 of 23
(5579 views)
Shortcut
|
Hello Chet, thx for your reply. Could you tell me how will search engines treat archived articles?
Thx!
(This post was edited by Theo on Jul 31, 2006, 5:00 PM)
|
|
| |
|
MikeB
Staff
/ Moderator
Jul 29, 2006, 2:57 PM
Post #4 of 23
(5557 views)
Shortcut
|
Hi,
Thanks for the post!
If you're using an external search like Google for your site then Archiving may not actually be the best option for you as these archived articles will be accessed through Article Manager's view.cgi script. The URL for an archived article would look something like this:
http://www.yoursite.com/cgi-bin/artman/exec/view.cgi?num=123
In this case "123" is the number of the article.
I'm a bit curious at this point, how many articles are you using within Article Manager? Generally Article Manager's search shouldn't run into anything like this until you have a higher amount of articles (depending on the speed of the server).
Also, another thing you may want to try is to setup Article Manager's category index pages to display more articles perpage. This would bring down (or completely eliminate) the amount of times people would need to click the "Next" button when viewing your content.
I hope this helps and if you have any other questions or comments, just let me know! 
Cheers,
Mike Briggs - Product Specialist
support@interactivetools.com
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
(This post was edited by Theo on Jul 31, 2006, 5:00 PM)
|
|
| |
|
svsanchez
User
Jul 31, 2006, 3:19 PM
Post #5 of 23
(5545 views)
Shortcut
|
Hello Mike, I currently have over 4,000 articles, and I am desperately waiting AM V.2 as I understand this new version will be much more efficient in resources usage. I had to chmodd the permissions on search.cgi as it drains the server resources incredibly. The solution you suggest about putting more articles per page is not so functional for us, as we are an Internet portal with thousands of visitors per day, and doing so would make the pages to appear slower!
(This post was edited by Theo on Jul 31, 2006, 5:00 PM)
|
|
| |
|
svsanchez
User
Aug 1, 2006, 9:10 PM
Post #6 of 23
(5527 views)
Shortcut
|
Hello again Mike, I am desperate, I tried what you said about more articles per page, but it seems to have the same effect as instead of looking only for 10 articles, now search.cgi takes a little longer to search 15 articles per page. It is dumb to have to chmod 000 search.cgi every day to prevent the abusive usage of resources of this script.
Is it possible to have a second server running search.cgi to prevent the main server from suffering the consequences of high traffic? When is V2 coming?!?!
|
|
| |
|
Donna
Staff
/ Moderator
Aug 2, 2006, 10:04 AM
Post #7 of 23
(5505 views)
Shortcut
|
Hi there,
4000 articles is quite a lot, so that could explain why your searches are taking longer. I'd recommend archiving some of the older articles so that you have fewer "live" articles.
When you archive an article, Article Manager creates a "pointer" for that file so that the old HTML link still works. This means that people can still go to the old link and get to the new location. Would that work for you?
Donna
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
svsanchez
User
Aug 2, 2006, 10:15 AM
Post #8 of 23
(5504 views)
Shortcut
|
Hello Donna, thank for your mail. I guess if I archive the older articles, they won't appear any more on my category index pages right? That would be a problem for me as when I bought your software in 2004 and I had a person publish over 2,500 pages which have very old dates! But if they won't disappear from the category index pages then it could be the solution.
However, I think the best solution would be to upload search.cgi on a second server and tell AM to use that search.cgi to handle all the "previous" and "next" category searches. How could we do this?
Thx
|
|
| |
|
LiquidQuest
User
Aug 2, 2006, 12:07 PM
Post #9 of 23
(5500 views)
Shortcut
|
you have 2500 articles in your category index pages? i guess your visitors have two choices of retrieving these articles:
1) use the search box and call for any article containing a keyword string or retrieve a specific date range etc..
2) continue to press "next" until they reach those articles (which can take a huge amount of effort on their part, and your server!)
my best advice to you is the following:
archive, Archive, ArChIvE <----- which ever way you want to read it 
thats whats killing your server resources, people over use your Next/Previous links which trigger the search.cgi script to retrieve information back to them. But then again, maybe your site is setup in away where they dont have any other choice?
Marssin
|
|
| |
|
svsanchez
User
Aug 2, 2006, 3:07 PM
Post #10 of 23
(5494 views)
Shortcut
|
Hello Marsin, over 80% of my users get to my articles through google, and another 10% through Yahoo and MSN searches. So that is why I am asking if archived articles will continue to appear on the category index pages or if they will disappear, as most of my site was uploaded in 2005 when I bought this AM and I can't have my sections "blank"! For example, there are over 500 recipes, of which at least 95% were published in 2004, so if I archive them, what will people see when they go to the recipes section? Only the 5% that has not been archived or will AM show at least the 10 last recipes uploaded on each subcategory? Please also note that I don't use the included search.cgi function to handle individual searches as that really would be too much, in place I use Google search.
If not, I believe the best solution would be to have another server handle all of the search.cgi calls. Could you please help me achieve this? My website is located at www.deguate.com in case looking at it can help you help me :)
Thank you.
(This post was edited by svsanchez on Aug 2, 2006, 3:08 PM)
|
|
| |
|
ross
Staff
/ Moderator
Aug 3, 2006, 11:33 AM
Post #11 of 23
(5459 views)
Shortcut
|
Hi there.
Thanks for posting!
When you archive articles they are actually taken off the index pages and stored dynamically in your database. Because of that part, the search engines like Google and Yahoo will most likely not be able to index them.
Visitors to your site will still be able to search for archived articles from the advanced search page though so they are still going to be available.
Now, since Article Manager’s search will need to be on the same server as its data is, I have an idea.
What you’ll do is make a copy of your entire publish directory and then start archiving the older articles. This actually removes the individual article pages from the publish directory but also relieves a lot of pressure from your search engine.
Once all the archiving is done, go back and put your copied publish folder’s contents in the actual publish folder.
What this is going to do is have all your article pages in there even though some are actually archived. Google/Yahoo should still be able to pick up on these articles because there are static html pages so visitors who find your site that way, are still going to find you. Visitors that come to your site by other means, can still search for these articles using the advanced search page.
One important thing is that you’ll need to tell Article Manager to not publish article redirect pages for archived articles because they will over right your articles. To do that, log into Article Manager and head to the Server tab of Setup Options. From there, scroll down to the Publish Settings section and find the checkbox labled ‘Publish "redirect" pages for external search engines.’ Uncheck that and you are all set.
Does that sound like it’s going to work for you? Let me know what you think .
-----------------------------------------------------------
Cheers,
Ross Fairbairn - Product Specialist
support@interactivetools.com
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
svsanchez
User
Aug 3, 2006, 12:40 PM
Post #12 of 23
(5456 views)
Shortcut
|
Hey Ross, that sound like a great idea, thx a lot for thinking about that solution!!!! I will give it a try and if I get stuck I will post you again :)
|
|
| |
|
trandel
User
Aug 24, 2006, 11:12 PM
Post #13 of 23
(5140 views)
Shortcut
|
Hi,
I have a similar need but having tested archiving I noticed that the index entries at the top level of all categories are removed. However, below that level the index entries remain?
I have a category structure:
All Publications
Industry 1
Magazine 1
News
Technical
etc
Magazine 2
Industry 2
So I archive Magazine 1 for example - entries under All publications are removed but they remain under Industry and Magazine 1 but area removed from the bottom level category (eg News). Ross states that all index entries are removed ..............Am I specifying something wrong in the Archive process?
Regards
Tony Randell
|
|
| |
|
MikeB
Staff
/ Moderator
Aug 26, 2006, 9:57 AM
Post #14 of 23
(5107 views)
Shortcut
|
Hi Tony,
Thanks for posting!
Once you've archived articles on your site there should no longer be any links to this (other than through the Advanced Search) this means that when people are looking at a category index page they shouldn't be able to see any of your archived content. If these articles are still coming up, they may not have been archived or the page may not have been re-published. I'd suggest clicking the "Publish All" button in the Setup Options and then double checking these category index pages.
If those archived articles are still appearing I'd be happy to have a closer look at this for you if you'd like to fill out a Support Request on our site with your FTP details:
https://www.interactivetools.com/support/email_support.cgi
This should allow me to get onto your server and have a closer look at this for you.
I hope this helps Tony!
Cheers,
Mike Briggs - Product Specialist
support@interactivetools.com
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
trandel
User
Aug 26, 2006, 10:59 AM
Post #15 of 23
(5106 views)
Shortcut
|
Hi Mike
 Sorry - forgot to republish!!
Regards
Tony Randell
|
|
| |
|
svsanchez
User
Aug 29, 2006, 10:37 PM
Post #16 of 23
(5037 views)
Shortcut
|
Hello Again Ross, guys...
I am sorry to keep bothering you with this issue, but the solution provided by Ross is not helping me. The reason is that many of my articles CAN NOT be taken off the site when you click on the "next page" links. The reason for this is that I migrated to AM from a less robust software and I had my site manager republish all my site in less than a month about 2 years ago (that's about 2,500 pages uploaded in about 30 days). Some of my categories are, for example:
TOURISM, GEOGRAPHY, PRESIDENTS, ETC...
So for example, in Geography I have info on each of the 23 states that comprise my country, then on presidents there's info on the 40 or so president's we've had so far, etc...
This means that if I go with the solution suggested by Ross, when I archive my categories people will access the presidents page, which was uploaded in september 2004, and it will appear empty, as the president we have entered office in january 2004 and he and all previous presidents were entered in september 2004 on my AM database! The same will happen with at least 50% of my categories which are not updated often.
So I have an idea, please tell me how we could implement this, perhaps it will help others as well. If I could copy the data file every week to another server, and have the search.cgi handle all searches on that server using the "backup" copy on that server, that way the main server will not be affected by the huge resources usage! My data file is now over 22 MB and since there is still no news about the release of the new AM version in mysql I think this could do the job while we wait for this much needed upgrade!
Could you please help me implement this!
Thank you...
|
|
| |
|
Theo
Project Manager
/ Moderator
Aug 30, 2006, 3:36 PM
Post #17 of 23
(5003 views)
Shortcut
|
Hey, SVS, thanks for posting!
If you take a look at the Archive tab (in Article Manager's Setup Options), you'll see that you can select which categories to archive. So you can limit your archiving to the categories that are "overflowing", and the categories that aren't updated often won't be affected.
If there's overlap, let me know and hopefully we can find another way to make this work. :)
Theo Wiersma
Project Manager
(This post was edited by Theo on Aug 30, 2006, 3:37 PM)
|
|
| |
|
svsanchez
User
Aug 30, 2006, 4:14 PM
Post #18 of 23
(4998 views)
Shortcut
|
Hello Theo, thx for your comments, the problem is I have more than 200 categories created on my site! Isn't it possible to have the search.cgi to work on another server?
|
|
| |
|
Dave
Staff
/ Moderator
Sep 1, 2006, 8:37 AM
Post #19 of 23
(4949 views)
Shortcut
|
SVS,
With 4000 articles and 200 categories your reaching the limits of what we recommend for the built in database. Your article data file needs to be loaded into memory and searched each time search.cgi is called and with many users running search.cgi that's what's using up your resources.
Our next version is designed to handle many many more records. If you're interested, I'd like to have you help us beta test it. I'm actually working on the data upgrade code right now so your site (or it's data) would be a great way to test everything.
You can sign up to the beta program here:
http://www.interactivetools.com/iforum/_P48404/
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
svsanchez
User
Sep 3, 2006, 10:20 AM
Post #20 of 23
(4918 views)
Shortcut
|
Hello, the next version of AM is what I've been waiting since a year ago when it was published on your forums that it would be available in december of last year. I was asking for a temporary solution to have my site working without problem while the new version arrived, so your offering of beta testing the new version is coming at a good moment :)
How do we do this!?!?!
|
|
| |
|
Donna
Staff
/ Moderator
Sep 4, 2006, 9:32 AM
Post #21 of 23
(4889 views)
Shortcut
|
Hi there,
Take a look at this post for more information on how to apply to the closed beta program Dave mentioned. :)
Donna
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
TitanJeff
User
Sep 18, 2006, 8:50 AM
Post #22 of 23
(4578 views)
Shortcut
|
I'm running into the same issue with search.cgi. I've archived about 1,200 articles and now only have 500 or so active.
I read where turning off the auto publish is another option but how does the site update if this is not selected?
|
|
| |
|
ross
Staff
/ Moderator
Sep 18, 2006, 10:46 AM
Post #23 of 23
(4575 views)
Shortcut
|
Hi Jeff.
Thanks for posting!
500 articles definitely shouldn’t be causing a problem so I am not exactly sure what’s happening there. Would you mind sending me a support request so I can do some testing on your server?
https://www.interactivetools.com/support/email_support.cgi
I look forward to hearing from you .
-----------------------------------------------------------
Cheers,
Ross Fairbairn - Product Specialist
support@interactivetools.com
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
