As the head of an SEO agency, have you noticed that Google does not index pages as easily as they once were? Indexing has been tougher for a while, but the problem seems to have been exacerbated when ChatGPT was released to the public. There is a particularly good reason for that: Google’s limited indexing resources.
A simple explanation is that Google cannot index every page on the web. Theoretically, they would probably like to. But it is just not possible; Google is limited by its own resources. So if the number of pages on the internet continues to climb while Google’s resources remain static, indexing will have to become more difficult simply due to attrition.
Unlimited, But Not Really
A noticeable difficulty with indexing in recent years has led some people to question whether Google purposely limits the number of pages it indexes. One SEO agency would swear by hard limits while another would say no such limits exist. What does Google say? Unlimited, but not really.
Google’s John Mueller posted a brief video back in 2021 finally answering the question of whether Google has hard indexing limits. In short, they do not. Mueller explained in the video that, while there are no defined limits to indexing, Google’s limited resources force it to concentrate efforts “where we think they make sense.”
What does that mean practically? It means several things, according to Mueller:
- New Sites – Google bots will not automatically index every page on a brand-new site. It may do so if its algorithms have reason to believe the site is exceptional, but otherwise the bots will only index what they believe to be the main pages.
- Dynamic and Static Pages – Some SEO agencies assume that adding an HTML extension to a dynamic page can make it appear as static to Google’s bots, thereby increasing the chances it will be indexed. Google’s bots don’t care about it. Static pages are not any more likely to be indexed.
- Automated Indexing – Mueller also made the point that indexing is fully automated. So it’s more important than ever that new pages be built in such a way that algorithms understand them. Otherwise, the chances of being indexed go down.
This all boils down to the idea of competition. Google only has the resources to index so many pages. How many is a matter of debate. A common number is 400 billion documents. But it could be anything. Here is the point: Google has admitted to limited resources that force it to concentrate efforts on what their algorithms deem as being most important. So every SEO agency is competing against every other agency and website owner for limited exposure.
Make Your Site Awesome
How does an SEO agency compete? By making the sites it works on awesome. Of course, Google doesn’t define what ‘awesome’ means. At least not in a dictionary sense. But you can figure it out pretty easily if you go back and read up on Google’s helpful content update from 2022.
In a post explaining the update, Google clearly stated its desire that SEO producers create content for consumers first. They don’t want content creators obsessing over their algorithms to the point of creating content that real consumers aren’t interested in.
When you think about Google’s rationale, it makes complete sense. Who are their customers? Not algorithms, that’s for sure. Google’s customers are the people who use their search engine. Google needs to either give them what they want or stand by and watch them switch to other search engines.
What do they want? Quality content that meets their needs quickly and effectively. If I go online looking for information about Google ad management, I expect to find pages with plenty of detail. I am not interested in generic, nonspecific content that utilizes a lot of flowery buzzwords but offers no substance.
The Internet Is Content
Another way to look at awesomeness, in terms of a website, is through understanding what the internet actually is. The internet is content. It is a network of content that people access with mobile devices and computers. A single website acts as little more than an online repository of content. So to make a website awesome, you need to make its content awesome.
As SEO providers, we can concentrate on generating our own content. We can also hire freelancers and subcontractors to create it for us. One way or the other, truly awesome content doesn’t come easily or cheaply.
One of the things that makes generative AI so attractive is that you don’t have to pay it a salary. But as we have all learned over the last two years or so, generative AI doesn’t produce awesome results. Adequate? Maybe. But awesome? Absolutely not.
If an SEO agency wants awesome content, it needs to be willing to pay for it. Gone are the days of paying content mill prices and expecting the results to play well in organic searches. If we pay content mill prices, we will continue to get content mill results in a landscape that gets more competitive by the day.
There’s Only So Much Room
The fact that Google has limited resources for indexing proves that there is only so much room at the top, so to speak. And since there is no indication that Google’s resources are expanding at the same rate as the internet itself, the competition for indexing attention only gets stiffer.
The bottom line is that SEO agencies and producers need to focus more time and energy on content and not obsess as much over the technical aspects. Yes, mobile friendliness and page speed are still important. Keywords still rule the roost in search. But Google’s limits make appealing to consumers as important, if not more important, than appealing to algorithms.
SEO has changed dramatically in the last 5 years. Not every SEO agency has changed with at. Has yours? The answer lies in how well your pages index.
