|
At one time or another you may have used a submission tool, or submitted by
hand and then wondered why you had not been indexed. Unfortunately, there are
many reasons that may delay or prevent you from being indexed by a search engine.
There's rarely one simple answer for why you're not being found. Fortunately,
there is generally an explanation and a way to correct the problem if you know
what to look for.
Below are the Top 21 reasons we've compiled over the years as to why you may
not be finding your Web site or Web page in one or more search engines:
1. INDEX TIME: First, make sure you've allowed enough time to become indexed.
The amount of time to allow is sometimes listed on the search engine's submission
page. Unfortunately, the engine's own advertised times are often inaccurate
or out of date. WebPosition's Submitter report and WebPosition's URL Verification
report will both tell you how much time you should allow before being concerned
about not being indexed. Average index times often range from one to eight weeks
depending on the engine. Some engines like AltaVista and Inktomi offer paid
options if you wish to be indexed more quickly.
2. ALREADY INDEXED: Be sure you're not already indexed but just don't know
it. Unfortunately, none of the major engines are kind enough to e-mail or notify
you as to if and when you've been indexed.
In addition, you cannot simply do a search on a keyword that applies to your
Web site and expect it to pop up at the top. In fact, you must take pro-active
steps to optimize your pages for each search engine. If you don't, it's very
unlikely you'll find your Web site except on the most specific of searches.
The method to determine if a page or domain has been indexed varies from one
engine to another, and in many cases, it's difficult to tell for sure whether
your pages are in fact in there. Never assume that you're not indexed just because
you searched for a bunch of keywords and you never came up in the first few
pages of results. You could be there (i.e., indexed) but be buried near the
bottom.
In addition, it's not very practical to check the status of a number of pages
on each major engine each week. Fortunately, WebPosition Gold has a URL verification
feature in the Reporter that makes this process much easier. Each time you run
a mission, it will report which URLs exist and do not exist in each engine.
If you're using WebPosition and are not finding your URLs after submitting,
be sure to see this page for common pitfalls to watch out for:
http://www.webposition.com/urlnotfoundhelp.htm
3. MISSING PAGE: Make sure you have uploaded the pages to your site before
submitting them. This one will seem obvious to many people, but submitting a
page that does not exist or submitting with a subtle typo in the URL is a goof
we might all make at one time or another. If you're using WebPosition's Submitter,
there's a checkbox on tab 2 labeled "Verify that each page exists on Web
site before submitting." This option defaults so that WebPosition will
verify that all your URLs are valid and actually exist before submitting them.
This is important since not all search engines will notify you if the URL does
not exist when you submit.
4. ROADMAP FROM HOME PAGE: Some engines have been known to drop pages that
cannot be traveled to from the home page. HotBot has been rumored to do this.
You may want to consider submitting your home page that links either directly
or indirectly to your doorway pages. Think of your Web site as a series of roads
(i.e., links) from one page to another. If there's no road from your home page
to the page you want indexed, a search engine may decide the page is unimportant
or of low-quality. You could submit the page directly, but the engine may reject
it or may drop it at a later date when it finds no "road."
5. EXTERNAL LINKS: Some search engines such as Google and HotBot have been
known to refuse to index Web sites that do not have any other Web sites linking
to them. Or, they may index your home page but refuse to index any other pages
until you achieve at least one or more links from another domain. Or, they may
index you for awhile but then "prune" their database later of all
Web sites that did not achieve any external links within a certain period of
time. However, do not worry! You simply need to establish some links and when
that's done, resubmit both your pages and the pages that link to you. Once you
have links to your Web site, it becomes much easier to get indexed, stay indexed,
and to achieve top rankings.
To check your link popularity and for tips on how to increase the links to
your Web site see our free service at: Link Popularity Tool
6. FRAMES: If you have content inside HTML frames, this can cause problems
with submissions. For example, the search engine may index the main content
of the page, but not the surrounding menu frame. Visitors to your site will
then find some information but may not see the associated menu! It's generally
best if you can create non-framed versions of your pages. You should then submit
the non-frames versions of your pages, which can of course link to your framed
Web site. Alternatively, you can enter your relevant text within the NOFRAMES
area of a framed page that most search engine spiders will read. However, don't
expect to achieve high rankings while optimizing the NOFRAMES area. Optimizing
a NON-framed page will often achieve better results.
7. SPIDER BLOCKS: Search engine spiders cannot index sites that require any
kind of registration or password. A spider cannot fill out a form of any kind.
The same rule applies regarding indexing of content from a searchable database.
That's because the spider cannot fill out a form to query that database. The
solution is to create static pages that the engines will be able to find and
index without performing a special action on your site. Depending on the database
system you have, there are utility programs out there that help you do this,
as well as companies that can assist you.
8. FREE SITES: Many engines no longer index pages from free web sites or they
limit the number of pages they will index from these hosts. Sometimes they will
get too many "junk" submissions from free web site domains such as
Geocities or others. Therefore, some engines choose not to index anyone from
some of these domains. Or, more commonly, they limit the number of pages they
will accept.
It's always best to buy your own domain name (very important) and place it
on a respected, paid service to avoid being discriminated against. The free
traffic you can generate from the search engines is just too valuable to be
sacrificed for the small savings a free hosting service provides. In addition,
free hosts are often unreliable or force you to display banners that send valuable
visitors away from your Web site soon after arriving. That can cost you sales.
9. GUILT THROUGH ASSOCIATION: If your Web site shares the same IP address as
many other Web sites on your host's Web server, then you may find your IP quietly
banned from something another Web site on the same server did! It's always best
to ask your hosting service if your domain name has its own unique IP assigned
to it. If not, ask them to move it to its own IP to avoid the potential of having
your submissions ignored because of something that a site sharing your IP did.
We've heard from many people who tried everything to be indexed only to find
it was a snap once they changed hosting services.
10. SUBMISSION LIMITS: Make sure you're submitting within the recommended limits.
Some engines do not like more than a certain number of submissions per day for
the same domain. If you exceed the limit, you may find that all your submissions
are ignored. Fortunately, WebPosition's submitter will warn you regarding current
limits and help keep you within them. Some submission consultants feel it is
dangerous to submit more than ONE page a day to an engine for a given Web site.
For those who wish to be ultra-conservative in their approach, the WebPosition
Submitter includes a checkbox to limit submissions to one URL per day per engine.
11. DYNAMIC PAGES: Dynamic pages are often ignored by the search engine spiders.
In fact, any URL containing special symbols like a question mark (?) or an ampersand
(&) will be ignored by many engines. Pages generated on the fly from a database
often contain these symbols. In this situation, it's important to generate "static"
versions of each page you wish to be indexed. In regard to the search engines,
the simpler the page is, the better. Does this mean, for example, having a javascript
to count visits to the page will prevent you from being indexed, or lower your
rankings? No. It simply means that the search engine will most likely ignore
the javascript and index the remaining areas of the page. There is evidence
that going too far with fancy scripts and code on a page can hurt your rankings
if the bulk of your page consists of java or VB scripts.
12. NON-INDEXABLE CONTENT: It's important to know the types of content that
the average search engine cannot index. Most engines cannot index text that
is embedded in images. Text that appears in multimedia files (audio and video)
will not be indexed. Most engines cannot index information that is generated
by Java applets or in XML coding.
13. LARGE PAGES: If your site has a slow connection or the pages are very complex
and take a long time to load, it might time out before the spider can index
all the text. For the benefit of your visitors and the search engines, limit
your page size to 50K or less. In fact, most Webmasters recommend that your
page size PLUS the size of all your images on the page should not exceed 50K-70K
total. If it does, many people on dial up connections will leave before the
page fully loads.
14. DEEP LINKS: If you submit just your home page, don't expect a search engine
to travel more than one or two links away from the home page or from the page
that you submitted. Over time they may venture deeper into your site, but don't
count on it happening quickly. You'll often need to submit pages individually
that appear further down into your site or create more direct links from the
home page (either visible links or hidden links). This way the search engines
can find them. Visible links are preferable, but when that's not practical,
there are methods to create links that won't be seen by the average visitor:
http://www.webposition.com/hiddenlinks.htm
The technique of submitting one page that then links to multiple other pages
you want found is called creating a "hallway page." In many cases,
this will not only get you indexed in cases where they are ignoring you, but
it will often improve your rankings. That's because many engines assign "bonus
points" to pages they find on their own versus pages that were specifically
submitted to them.
15. UNRELIABLE HOSTS: If your Web site fails to respond when the search engine
spider pays a visit, you will not be indexed. Even worse, if you are indexed
and they pay a visit when your site is down, you could be removed from their
database! Consequently, it pays to have a reliable hosting service that is up
99.5% of the time. However, at some point a spider is going to hit that other
0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a
close eye on your listings and resubmit when needed.
16. SPAM: If you have ever used any questionable techniques that might be considered
an overt attempt at spamming (i.e., excessive repetition of keywords, same color
text as background, or other things that the WebPosition Page Critic warns you
about), an engine may ignore or reject your submissions. If you're having trouble
getting indexed in the expected amount of time, make sure your site is spam-free.
17. REDIRECTS: If your site contains redirects or meta refresh tags these things
can sometimes cause the engines to have trouble indexing your site. Generally
they will index the page that it is redirecting TO, but if it thinks you are
trying to "trick" the engine by using "cloaking" or IP redirection
technology that it can detect, there is a chance that it may not index the site
at all.
18. PROPER DIRECTORY SUBMISSIONS: If you're submitting to a directory site
like Yahoo, Open Directory, Looksmart, or others, then a human being will review
your site. They must decide if the site is of sufficient "quality"
before they will list it. I recommend you read the submission guide on the directory
tab of the WebPosition Submitter. It contains tips on how to improve your chances
of obtaining a good listing in these directories. Getting listed in major directories
first can help you get listed elsewhere.
19. INDEX TIMES CAN FLUCTUATE: WebPosition will tell you the average index
time of each search engine. However, this is only an average. Sometimes engines
will index sites every 30 days fairly consistently and then suddenly stop indexing
most sites for several months. This can be frustrating, but it does happen.
Generally a major engine will not go more than three to four months without
refreshing its index. If you're wondering if others are experiencing trouble
getting indexed on a particular engine, try asking around. You can often find
valuable real-world feedback on a search engine discussion forum such as the
one at:
http://www.marketposition.com
20. PAGE LIMITS: If you have many pages indexed but are having a hard time
getting new ones recognized, be aware that there are limits. Each search engine
will only spider so many pages of your Web site. This may range from a few dozen
or three or four hundred depending on the engine. Some people have even been
successful in getting far more pages indexed depending on the engine. Google
is one engine that tends to crawl deeper into your site. However, how deep they
go may depend on factors like your link popularity. Sites with higher link popularity
are deemed "worthier" of more thorough indexing.
21. RANDOM ERRORS: Last but not least, sometimes the engines just lose submissions
at random through technical errors and bugs. After all, they are managing a
database of hundreds of millions of pages. Therefore, some people like to resubmit
once or twice a month for good merit in case they do lose a submission. Certainly
if you've followed all the "rules" and are still not listed, by all
means, re-submit! Sometimes a little persistence is all that's needed.
TIP: Once your page achieves a desirable ranking, it's best not to continue
submitting it. You risk the engine re-evaluating the page and possibly reducing
your ranking.
If any of the above scenarios apply to your submission, you should take the
appropriate actions and then re-submit. If that still does not work, you should
consider e-mailing or calling the search engine and asking them politely why
you have not been indexed yet. Sometimes they will reply back with "Sorry,
there was a problem with our system and I've now made sure you'll be indexed
within the next couple days." Or, sometimes they'll tell you why you were
not indexed so you might correct it. In other cases, they will ignore your e-mail
and you'll have to keep e-mailing or calling them until they respond. Still,
it's definitely worth the effort to get your site listed with the major engines.
This assumes you also take the time to optimize your pages so you'll achieve
top rankings.
About the author:
Robin Nobles is the Director of Training of the Academy of Web Specialists, which teaches online training in search engine marketing. She is also a trainer with Search Engine Workshops, which presents on location workshops in search engine marketing at various locations across the country. Please visit our site for more information about online training and other resources.
http://www.marketposition.com/
|
