 | | |
|
Dave
Staff
/ Moderator
Aug 19, 2002, 2:52 PM
Post #1 of 27
(11687 views)
Shortcut
|
|
Creating a full article list for feeding search engines
|
Can't Post
|
|
Hello,
I've put together a quick example of how you can use a SSI and the search engine to generate a full list of all your articles for feeding to the search engines. Here's how to do it.
First, we create a new template called "search_results_minimal.html" and save it in /templates/search/. We'll tell the search engine to use this template for it's output and it will control how our full article list looks.
Next, we'll create a file called "all_articles.shtml" in the /publish/ folder that has nothing in it but a single SSI tag. The SSI tag needs to start with the full virtual path (everything after http://yourdomain.com) to your search engine and should look something like this:
<!--#include virtual="/cgi/artman/exec/search.cgi?search=1&perpage=999999&template=search/search_results_minimal.html" -->
The forum may wrap that, but make sure it's all on one line. It basically says: show the search results, show 999,999 results per page, and use the search_results_minimal.html template for output.
Now, when the search engines index "all_articles.shtml" they'll treat it as a regular static html file, but it will dynamically include all the latest article titles and links with search.cgi, making sure the search engines have a direct path to all your published articles so they can index them.
One last note, if you're on windows "include virtual" doesn't work properly so you'll need to replace that with "exec cgi". Interestingly enough, "exec cgi" doesn't work on unix (when using a query string). Weird, eh?
I've attached a sample file for "search_results_minimal.html" and "all_articles.shtml". We sure to edit "all_articles.shtml" first because it has the SSI's for windows and unix in it.
Dave Edis - Senior Developer
interactivetools.com
|
|
Attachments:
|
search_results_minimal.html
(2.78 KB)
|
|
|
all_articles.shtml
(0.49 KB)
|
|
| |
|
brianbankerus
User
Jul 27, 2003, 8:58 PM
Post #2 of 27
(11226 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Getting really close to a solution... I've tried juggling around my files and paths, and no matter what I do I come up with this message when I try to access that page...
#EXEC calls have been disabled for this virtual path
Any suggesions or ideas?
|
|
| |
|
Bob B
User
Jul 27, 2003, 9:48 PM
Post #3 of 27
(11225 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Apache config issue?
|
|
| |
|
brianbankerus
User
Jul 28, 2003, 1:36 PM
Post #5 of 27
(11194 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Seriously though, I don't have any understanding of what that might mean. Advice? Workaround?
|
|
| |
|
Dave
Staff
/ Moderator
Jul 28, 2003, 1:37 PM
Post #6 of 27
(11194 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Hi Brian,
It sounds like an issue with how the Server Side Include is working or how they are setup on your server.
Make sure you have the right path. For example, if your search script is located at website.com/scripts/search.cgi make sure the SSI path starts with everything after the domain. eg: /scripts/search.cgi
If that still doesn't work, post the URL of your search script and the SSI tag you're trying to use and I'll try and help you out.
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
brianbankerus
User
Jul 28, 2003, 5:29 PM
Post #7 of 27
(11181 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
I looked all through my server to find another place where a search script might be hanging out, and the only one I found was at http://glossynews.com/cgi-bin/artman/exec/search.pl Now that PL part mystified me a bit so I went and looked through my original zip file. There it's a .cgi file. The sizes still matched. What they hey, I got a wild hair, I uploaded the .cgi file, Re-Published all files, still to no avail.
Incidentally, the template is at http://glossynews.com/cgi-bin/artman/templates/search/search_results_minimal.html
I'm trying to make the super index page reside at; http://www.glossynews.com/artman/publish/all_articles.shtml , which is very similar to my standard publication address. In the all_articles file I did change cgi/ to cgi-bin/, though I tried it both ways. I've tried a bunch of things a bunch of ways.
Also, I went into my login and made sure all my search addresses and everything were right. Went and double checked my search utility is still functioning just fine.
Do I contact my host at this point?
|
|
| |
|
Dave
Staff
/ Moderator
Jul 28, 2003, 7:57 PM
Post #8 of 27
(11177 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Don't worry about the .pl extension, it's just a .cgi file that has been renamed. Most web hosts are fine with .cgi files, but some make you rename things .pl (which is suppose to stand for perl) to make your scripts run. Your web host is probably one of the ones that makes people rename files. : )
Have you tried the following SSI tag?
<!--#include virtual="/cgi-bin/artman/exec/search.pl?search=1&perpage=999999&template=search/search_results_minimal.html" -->
Note, make sure there is no space or enter after the ?, the forum may wrap that line.
If that doesn't work, then the next step would be to contact your ISP, point them to your all_articles.shtml page, and ask why you're getting that error message.
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
brianbankerus
User
Jul 28, 2003, 9:18 PM
Post #9 of 27
(11175 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Yep, that's about where we're at. I tried that tag (the unix one) and it gave me a very elaborate 404 the likes of which I've never seen.
HTTP/1.1 404 Object Not Found Server: Microsoft-IIS/5.0 Date: Tue, 29 Jul 2003 04:21:13 GMT Connection: close Content-Type: text/html 404 Object Not Found
Anyhow, I'll drop them a trouble ticket tonight.
Thanks!
|
|
| |
|
Dave
Staff
/ Moderator
Jul 28, 2003, 10:34 PM
Post #10 of 27
(11172 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Ok, I think I have it figured out. I didn't realize you were on a windows server. I did some tests locally and confirmed that IIS (Microsoft's Web Server) actually implements SSI's a little differently from the way the Apache does (Apache is the other common web server, mostly found on UNIX like servers).
Try this SSI tag, I'm running Microsoft IIS locally and seems to do the trick.
<!--#exec cgi="/cgi-bin/artman/exec/search.pl?search=1&perpage=999999&template=search/search_results_minimal.html" -->
Let me know if that works, I'm almost positive it will. It's good we found out about this too, I'm going to add a note to the original message some future IIS users won't have to go through as much trouble as you and me have! : )
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
brianbankerus
User
Jul 29, 2003, 12:30 AM
Post #11 of 27
(11171 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Nope... replaced my all_articles file with the exact (unwrapped) text from the post. Still gives me the #EXEC calls have been disabled for this virtual path error. Time to contact my ISP?
|
|
| |
|
Dave
Staff
/ Moderator
Jul 29, 2003, 11:00 AM
Post #12 of 27
(11159 views)
Shortcut
|
|
Re: [brianbankerus] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Yep, time to ask your ISP. : )
Dave Edis - Senior Developer
interactivetools.com
|
|
| |
|
travelify
New User
Aug 4, 2003, 7:42 PM
Post #14 of 27
(11051 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Dave, it's work great.
Thanks
http://www.travelify.com Thailand travel guide based on real experiences.
|
|
| |
|
Bob B
User
Aug 5, 2003, 11:13 AM
Post #15 of 27
(11039 views)
Shortcut
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Dave,
What is it that 'triggers' the search engines to index the all_articles.shtml page? Do we have a link to it from somewhere?
Have a nice day.
|
|
| |
|
Donna
Staff
/ Moderator
Aug 5, 2003, 11:27 AM
Post #16 of 27
(11038 views)
Shortcut
|
|
Re: [Bob B] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Bob,
Generally, for a search engine to index a page, it has to find it. So, you could link to it.
Or, you could submit the page directly to the search engine of your choice.
The best answer for that is to start reading what the various search engines require for submissions. :)
Donna
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
Bob B
User
Aug 5, 2003, 11:34 AM
Post #17 of 27
(11035 views)
Shortcut
|
|
Re: [Donna] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Thanks, Donna, for the reply.
It was the link that I was missing from Dave's first post in this thread.
Is using the one-pixel link just about the best way to accomplish the plan?
|
|
| |
|
Donna
Staff
/ Moderator
Aug 5, 2003, 11:39 AM
Post #18 of 27
(11034 views)
Shortcut
|
|
Re: [Bob B] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Hi Bob,
You'll want to check with the search engines on that -- I haven't kept up to date with what requirements they have for spidering. :)
Donna
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
Bob B
User
Aug 5, 2003, 11:52 AM
Post #19 of 27
(11033 views)
Shortcut
|
|
Re: [Donna] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Thanks for the reply, Donna.
I'll check that next month.
|
|
| |
|
brianbankerus
User
Aug 7, 2003, 7:15 PM
Post #20 of 27
(11010 views)
Shortcut
|
|
Re: [Bob B] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Dave, What is it that 'triggers' the search engines to index the all_articles.shtml page? Do we have a link to it from somewhere? Have a nice day.
|
|
| |
|
burnworld
User
Aug 11, 2003, 9:27 PM
Post #21 of 27
(10967 views)
Shortcut
|
|
Re: [Dave] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Got it working. Good stuff.
(This post was edited by burnworld on Aug 11, 2003, 9:40 PM)
|
|
| |
|
Bob B
User
Aug 11, 2003, 10:02 PM
Post #22 of 27
(10962 views)
Shortcut
|
|
Re: [burnworld] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
I'm a newbie. Could you elaborate a little.
TIA
|
|
| |
|
burnworld
User
Aug 12, 2003, 7:31 AM
Post #23 of 27
(10958 views)
Shortcut
|
|
Re: [Bob B] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Bob,
I originally posted an issue but I cleared it up by starting from scratch. Meaning downloaded the two files above and setting them up as instructed. Have you been able to get this to work? I noticed that you mentioned a 1 pixel link or something like that. I would discourage you from doing this. Search engines mainly Google are looking for little tricks like this and then penalizing your site if it finds things like this. Also don't use links the same color as the background.
|
|
| |
|
Donna
Staff
/ Moderator
Aug 12, 2003, 11:02 AM
Post #24 of 27
(10954 views)
Shortcut
|
|
Re: [burnworld] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
Good suggestions, burnworld. Thanks!
Donna
Hire me!
Save time by getting our experts to help with your project. Template changes, advanced features, full integration, whatever you need. Whether you need one hour or fifty, get it done fast with Priority Consulting.
|
|
| |
|
Bob B
User
Aug 13, 2003, 7:55 PM
Post #25 of 27
(10943 views)
Shortcut
|
|
Re: [Donna] Creating a full article list for feeding search engines
[In reply to]
|
Can't Post
|
|
burnworld,
If you or others are up to speed on preferred ways to make our AM stuff friendly to the search engines, it would be nice to have some kind of matrix of to-do's and to-do's NOT with the various search engines. I have some links to provide advice along this line but moving forward with AM is (happily) keeping me too busy to read them all.
|
|
| |
|
1
