Main
Index
Search
Posts
Who's
Online
Log
In

Home: Discontinued/Classic Products: Article Manager 1 Add-ons:
Creating a full article list for feeding search engines

 

First page Previous page 1 2 Next page Last page  View All


Dave
Staff / Moderator


Aug 19, 2002, 2:52 PM

Post #1 of 27 (15787 views)
Shortcut
Creating a full article list for feeding search engines Can't Post

Hello,

I've put together a quick example of how you can use a SSI and the search engine to generate a full list of all your articles for feeding to the search engines. Here's how to do it.

First, we create a new template called "search_results_minimal.html" and save it in /templates/search/. We'll tell the search engine to use this template for it's output and it will control how our full article list looks.

Next, we'll create a file called "all_articles.shtml" in the /publish/ folder that has nothing in it but a single SSI tag. The SSI tag needs to start with the full virtual path (everything after http://yourdomain.com) to your search engine and should look something like this:

<!--#include virtual="/cgi/artman/exec/search.cgi?search=1&perpage=999999&template=search/search_results_minimal.html" -->

The forum may wrap that, but make sure it's all on one line. It basically says: show the search results, show 999,999 results per page, and use the search_results_minimal.html template for output.

Now, when the search engines index "all_articles.shtml" they'll treat it as a regular static html file, but it will dynamically include all the latest article titles and links with search.cgi, making sure the search engines have a direct path to all your published articles so they can index them.

One last note, if you're on windows "include virtual" doesn't work properly so you'll need to replace that with "exec cgi". Interestingly enough, "exec cgi" doesn't work on unix (when using a query string). Weird, eh?

I've attached a sample file for "search_results_minimal.html" and "all_articles.shtml". We sure to edit "all_articles.shtml" first because it has the SSI's for windows and unix in it.

Dave Edis - Senior Developer
interactivetools.com
 
Attachments: search_results_minimal.html (2.78 KB)
  all_articles.shtml (0.49 KB)


brianbankerus
User

Jul 27, 2003, 8:58 PM

Post #2 of 27 (15326 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Getting really close to a solution... I've tried juggling around my files and paths, and no matter what I do I come up with this message when I try to access that page...

#EXEC calls have been disabled for this virtual path

Any suggesions or ideas?


Bob B
User

Jul 27, 2003, 9:48 PM

Post #3 of 27 (15325 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Apache config issue?


brianbankerus
User

Jul 27, 2003, 11:47 PM

Post #4 of 27 (15321 views)
Shortcut
Re: [Bob B] Creating a full article list for feeding search engines [In reply to] Can't Post


In Reply To
Apache config issue?

Wha' huh? Ya lost me...


brianbankerus
User

Jul 28, 2003, 1:36 PM

Post #5 of 27 (15294 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Seriously though, I don't have any understanding of what that might mean. Advice? Workaround?


Dave
Staff / Moderator


Jul 28, 2003, 1:37 PM

Post #6 of 27 (15294 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Hi Brian,

It sounds like an issue with how the Server Side Include is working or how they are setup on your server.

Make sure you have the right path. For example, if your search script is located at website.com/scripts/search.cgi make sure the SSI path starts with everything after the domain. eg: /scripts/search.cgi

If that still doesn't work, post the URL of your search script and the SSI tag you're trying to use and I'll try and help you out.

Dave Edis - Senior Developer
interactivetools.com
 


brianbankerus
User

Jul 28, 2003, 5:29 PM

Post #7 of 27 (15281 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

I looked all through my server to find another place where a search script might be hanging out, and the only one I found was at http://glossynews.com/cgi-bin/artman/exec/search.pl Now that PL part mystified me a bit so I went and looked through my original zip file. There it's a .cgi file. The sizes still matched. What they hey, I got a wild hair, I uploaded the .cgi file, Re-Published all files, still to no avail.

Incidentally, the template is at http://glossynews.com/cgi-bin/artman/templates/search/search_results_minimal.html

I'm trying to make the super index page reside at; http://www.glossynews.com/artman/publish/all_articles.shtml , which is very similar to my standard publication address. In the all_articles file I did change cgi/ to cgi-bin/, though I tried it both ways. I've tried a bunch of things a bunch of ways.

Also, I went into my login and made sure all my search addresses and everything were right. Went and double checked my search utility is still functioning just fine.

Do I contact my host at this point?


Dave
Staff / Moderator


Jul 28, 2003, 7:57 PM

Post #8 of 27 (15277 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Don't worry about the .pl extension, it's just a .cgi file that has been renamed. Most web hosts are fine with .cgi files, but some make you rename things .pl (which is suppose to stand for perl) to make your scripts run. Your web host is probably one of the ones that makes people rename files. : )

Have you tried the following SSI tag?

<!--#include virtual="/cgi-bin/artman/exec/search.pl?search=1&perpage=999999&template=search/search_results_minimal.html" -->

Note, make sure there is no space or enter after the ?, the forum may wrap that line.

If that doesn't work, then the next step would be to contact your ISP, point them to your all_articles.shtml page, and ask why you're getting that error message.

Dave Edis - Senior Developer
interactivetools.com
 


brianbankerus
User

Jul 28, 2003, 9:18 PM

Post #9 of 27 (15275 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Yep, that's about where we're at. I tried that tag (the unix one) and it gave me a very elaborate 404 the likes of which I've never seen.

HTTP/1.1 404 Object Not Found Server: Microsoft-IIS/5.0 Date: Tue, 29 Jul 2003 04:21:13 GMT Connection: close Content-Type: text/html 404 Object Not Found

Anyhow, I'll drop them a trouble ticket tonight.

Thanks!


Dave
Staff / Moderator


Jul 28, 2003, 10:34 PM

Post #10 of 27 (15272 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Ok, I think I have it figured out. I didn't realize you were on a windows server. I did some tests locally and confirmed that IIS (Microsoft's Web Server) actually implements SSI's a little differently from the way the Apache does (Apache is the other common web server, mostly found on UNIX like servers).

Try this SSI tag, I'm running Microsoft IIS locally and seems to do the trick.

<!--#exec cgi="/cgi-bin/artman/exec/search.pl?search=1&perpage=999999&template=search/search_results_minimal.html" -->

Let me know if that works, I'm almost positive it will. It's good we found out about this too, I'm going to add a note to the original message some future IIS users won't have to go through as much trouble as you and me have! : )

Dave Edis - Senior Developer
interactivetools.com
 


brianbankerus
User

Jul 29, 2003, 12:30 AM

Post #11 of 27 (15271 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Nope... replaced my all_articles file with the exact (unwrapped) text from the post. Still gives me the #EXEC calls have been disabled for this virtual path error. Time to contact my ISP?


Dave
Staff / Moderator


Jul 29, 2003, 11:00 AM

Post #12 of 27 (15259 views)
Shortcut
Re: [brianbankerus] Creating a full article list for feeding search engines [In reply to] Can't Post

Yep, time to ask your ISP. : )

Dave Edis - Senior Developer
interactivetools.com
 


brianbankerus
User

Jul 29, 2003, 12:39 PM

Post #13 of 27 (15254 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Found a short-term work-around... I went to see what the CGI link is for MORE HEADLINES, then I just altered it so it would show articles starting with #1, going through #999. I used this link, though clearly each person would need to adjust for themselves; http://www.glossynews.com/cgi-bin/artman/exec/search.pl?start=1&perpage=999&template=index/default.html

Then I just hit VIEW SOURCE, and saved it... That will be my solution until I can get things squared away with my ISP.


travelify
New User

Aug 4, 2003, 7:42 PM

Post #14 of 27 (15151 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Dave, it's work great.

Thanks
[url]http://www.travelify.com[/url] Thailand travel guide based on real experiences.


Bob B
User

Aug 5, 2003, 11:13 AM

Post #15 of 27 (15139 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Dave,

What is it that 'triggers' the search engines to index the all_articles.shtml page? Do we have a link to it from somewhere?

Have a nice day.Smile


Donna
Staff / Moderator


Aug 5, 2003, 11:27 AM

Post #16 of 27 (15138 views)
Shortcut
Re: [Bob B] Creating a full article list for feeding search engines [In reply to] Can't Post

Bob,

Generally, for a search engine to index a page, it has to find it. So, you could link to it.

Or, you could submit the page directly to the search engine of your choice.

The best answer for that is to start reading what the various search engines require for submissions. :)

Donna

--
support@interactivetools.com


Bob B
User

Aug 5, 2003, 11:34 AM

Post #17 of 27 (15135 views)
Shortcut
Re: [Donna] Creating a full article list for feeding search engines [In reply to] Can't Post

Thanks, Donna, for the reply.

It was the link that I was missing from Dave's first post in this thread.

Is using the one-pixel link just about the best way to accomplish the plan?Smile


Donna
Staff / Moderator


Aug 5, 2003, 11:39 AM

Post #18 of 27 (15134 views)
Shortcut
Re: [Bob B] Creating a full article list for feeding search engines [In reply to] Can't Post

Hi Bob,

You'll want to check with the search engines on that -- I haven't kept up to date with what requirements they have for spidering. :)

Donna

--
support@interactivetools.com


Bob B
User

Aug 5, 2003, 11:52 AM

Post #19 of 27 (15133 views)
Shortcut
Re: [Donna] Creating a full article list for feeding search engines [In reply to] Can't Post

Thanks for the reply, Donna.

I'll check that next month.

Smile


brianbankerus
User

Aug 7, 2003, 7:15 PM

Post #20 of 27 (15110 views)
Shortcut
Re: [Bob B] Creating a full article list for feeding search engines [In reply to] Can't Post


In Reply To
Dave,

What is it that 'triggers' the search engines to index the all_articles.shtml page? Do we have a link to it from somewhere?

Have a nice day.Smile

There are better ways than others. Search engines will rank that page they are looking at for it's relevance. The more links you have pointing to it, the better. What I did was change my copyright notice on the template of my articles. Now "(c) 2003 GlossyNews.com" has the site name as a link to the search_results_minimal page. Now, there are HUNDREDS of pages pointing to it. This makes search engines believe that it is highly relevant. And it's just built into my template. Done.


burnworld
User

Aug 11, 2003, 9:27 PM

Post #21 of 27 (15067 views)
Shortcut
Re: [Dave] Creating a full article list for feeding search engines [In reply to] Can't Post

Got it working. Good stuff.


(This post was edited by burnworld on Aug 11, 2003, 9:40 PM)


Bob B
User

Aug 11, 2003, 10:02 PM

Post #22 of 27 (15062 views)
Shortcut
Re: [burnworld] Creating a full article list for feeding search engines [In reply to] Can't Post

I'm a newbie. Could you elaborate a little.

TIASmile


burnworld
User

Aug 12, 2003, 7:31 AM

Post #23 of 27 (15058 views)
Shortcut
Re: [Bob B] Creating a full article list for feeding search engines [In reply to] Can't Post

Bob,

I originally posted an issue but I cleared it up by starting from scratch. Meaning downloaded the two files above and setting them up as instructed. Have you been able to get this to work? I noticed that you mentioned a 1 pixel link or something like that. I would discourage you from doing this. Search engines mainly Google are looking for little tricks like this and then penalizing your site if it finds things like this. Also don't use links the same color as the background.


Donna
Staff / Moderator


Aug 12, 2003, 11:02 AM

Post #24 of 27 (15054 views)
Shortcut
Re: [burnworld] Creating a full article list for feeding search engines [In reply to] Can't Post

Good suggestions, burnworld. Thanks!

Donna

--
support@interactivetools.com


Bob B
User

Aug 13, 2003, 7:55 PM

Post #25 of 27 (15043 views)
Shortcut
Re: [Donna] Creating a full article list for feeding search engines [In reply to] Can't Post

burnworld,

If you or others are up to speed on preferred ways to make our AM stuff friendly to the search engines, it would be nice to have some kind of matrix of to-do's and to-do's NOT with the various search engines. I have some links to provide advice along this line but moving forward with AM is (happily) keeping me too busy to read them allSmile.

First page Previous page 1 2 Next page Last page  View All