Main
Index 		Search
Posts 		Who's
Online 		Log
In

Home: Discontinued/Classic Products: Article Manager 1:
Search spider not seeing Articles

 
 


jmueller0823
User

Aug 14, 2002, 5:50 PM

Post #1 of 7 (3534 views)
Shortcut
Search spider not seeing Articles Can't Post
Using Article Manager for a few months now.

Just received a question today regarding article content "missing" from site searches.

[We're using Atomz search]

Since many of our articles are not linked on the site, we explicitly created a URL entry point as:

http://www.growthtrac.com/artman/publish/index.shtml

That change picked up a few more articles, but not all.

I suspect the problem is that the "More Articles" link at the bottom of each index/category page does not link to a static page. It goes out to a cgi search. Right?

Could that be the reason all our articles are not indexed?

Thanks. Love the product!


Jim

g r o w t h t r a c dot c o m
Life Changing Resources for Your Marriage


taylor
User

Aug 15, 2002, 6:59 AM

Post #2 of 7 (3523 views)
Shortcut
Re: [jmueller0823] Search spider not seeing Articles [In reply to] Can't Post
This is a pretty serious matter. Can someone from IT please answer this person's question?

If my AM-generated pages (haven't gone live yet) aren't going to get spidered, then there's no way I'm going to launch my site using AM.

Most of us here depend on search engine placement for people to find our sites.

Dave, Luke, dlo...somebody?

Cheers,

Taylor


Dave
Staff / Moderator


Aug 15, 2002, 12:30 PM

Post #3 of 7 (3500 views)
Shortcut
Re: [taylor] Search spider not seeing Articles [In reply to] Can't Post
Anytime you have dynamic content there is always the chance that some search engines will not index it. This is true of CGI, PHP or any other language. Some search engine will avoid any pages with a .cgi or similar extension for fear of getting caught in infinite loops.

Some search engines are starting to try and index more and more dynamic content, some likely still don't do any at all.

The way Article Manager works it's likely that the first time your site is crawled it will index all the articles linked to by the frontpage (including those linked in the top headlines of each category) and those will stay in the search engine's index. On every subsequent visit it will likely get more pages listed on the top headlines side bars of the pages it's indexing and get a lot of new articles as they come up in these lists. Looking at our logs we've had GoogleBot indexing our site every day this month. And if we ran Article Manager on this domain Google would likely have index many of the pages.

To be honest though, it's always possible that some of those pages might be missed by some search engines. In fact, pages that have no direct link path will likely not be found at all. This is a common problem for a lot of sites and many search engine sites recommend creating a sitemap or links page with links to all the pages on your site and letting the search engine follow all the links on that page.

So my recommendation, no matter what site or software you might be running, is to create a links page and either link to it or submit it to the search engine. There's lots of free ones out there and if requested we could probably even whip one up ourselves pretty quick.

That said, there's probably lots of other things you need to do to get good search engine placement, and you're probably best to refer to a good book or website about that topic as well.

Hope that helps.

Dave Edis - Senior Developer
interactivetools.com
 


jmueller0823
User

Aug 15, 2002, 7:44 PM

Post #4 of 7 (3467 views)
Shortcut
Re: [jmueller0823] Search spider not seeing Articles [In reply to] Can't Post
Actually, that was my "temporary" fix. I created a static page containing links to all our articles.


Jim

g r o w t h t r a c dot c o m
Life Changing Resources for Your Marriage


Dave
Staff / Moderator


Aug 15, 2002, 9:47 PM

Post #5 of 7 (3464 views)
Shortcut
Re: [jmueller0823] Search spider not seeing Articles [In reply to] Can't Post
jmueller0823,

We could look into some ideas to have this done automatically. Generally though, depending on the frequency you update your site, you may only need to do it once as long as the search engine indexes your site more often than the amount of time it takes an article to roll off the "top headlines" lists.

Another idea, I'm thinking we could create a simple page with a single SSI that would use the search engine to display links to all your articles. Since the page itself would be a static page it should be indexed by the search engines but we'd use an SSI to get a dynamic article list from the search engine.

Do you know if your server is Windows or Unix based? Let me know and I'll see what we can come up with so you don't have to recreate that page every time.

Dave Edis - Senior Developer
interactivetools.com
 


jmueller0823
User

Aug 16, 2002, 5:36 AM

Post #6 of 7 (3453 views)
Shortcut
Re: [jmueller0823] Search spider not seeing Articles [In reply to] Can't Post
<< you may only need to do it once as long as the search engine indexes your site more often than the amount of time it takes an article to roll off the "top headlines" lists.
>>

This is true.


<< Another idea, I'm thinking we could create a simple page with a single SSI that would use the search engine to display links to all your articles.>>

But, SSI content isn't static, is it?

Our server is Unix.


Jim

g r o w t h t r a c dot c o m
Life Changing Resources for Your Marriage


Dave
Staff / Moderator


Aug 19, 2002, 2:53 PM

Post #7 of 7 (3291 views)
Shortcut
Re: [jmueller0823] Search spider not seeing Articles [In reply to] Can't Post
>But, SSI content isn't static, is it?

Yes, but if you're including it in a static page the search engines won't know any better. : )

I created another thread on this with a solution:
http://www.interactivetools.com/forum/forum.cgi?post=1154

Dave Edis - Senior Developer
interactivetools.com