.Google.com's John Mueller responded to a question concerning why Google indexes pages that are forbidden from crawling by robots.txt and why the it's secure to disregard the associated Look Console documents about those creeps.Robot Web Traffic To Question Criterion URLs.The individual talking to the inquiry chronicled that robots were generating hyperlinks to non-existent query guideline Links (? q= xyz) to pages with noindex meta tags that are actually likewise blocked out in robots.txt. What triggered the inquiry is actually that Google is crawling the links to those web pages, obtaining shut out by robots.txt (without noticing a noindex robots meta tag) at that point getting shown up in Google Search Console as "Indexed, though obstructed by robots.txt.".The person asked the following concern:." Yet right here is actually the huge inquiry: why will Google.com mark pages when they can't even view the material? What is actually the conveniences during that?".Google's John Mueller confirmed that if they can't creep the web page they can not see the noindex meta tag. He additionally creates an intriguing acknowledgment of the web site: hunt driver, encouraging to dismiss the outcomes considering that the "typical" consumers will not observe those results.He composed:." Yes, you're appropriate: if our experts can not crawl the web page, we can't find the noindex. That stated, if our company can not creep the web pages, at that point there is actually certainly not a great deal for our team to index. Thus while you could observe some of those pages with a targeted web site:- question, the ordinary consumer will not view all of them, so I would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it simply suggests the URLs will certainly find yourself being crept (and also end up in the Look Console file for crawled/not indexed-- neither of these standings lead to concerns to the remainder of the website). The vital part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the restrictions in operation the Site: search accelerated hunt operator for analysis causes. Among those main reasons is actually given that it's certainly not connected to the regular search mark, it is actually a different thing completely.Google.com's John Mueller discussed the internet site hunt operator in 2021:." The short answer is that a site: inquiry is certainly not indicated to become complete, neither made use of for diagnostics purposes.A web site concern is actually a details sort of hunt that limits the results to a particular internet site. It is actually basically just the word web site, a digestive tract, and then the web site's domain name.This question confines the results to a particular internet site. It is actually certainly not meant to become a complete compilation of all the webpages from that web site.".2. Noindex tag without utilizing a robots.txt is actually alright for these type of scenarios where a bot is actually connecting to non-existent web pages that are actually getting discovered through Googlebot.3. Links along with the noindex tag are going to create a "crawled/not indexed" entry in Explore Console and also those will not possess an unfavorable result on the rest of the web site.Check out the inquiry and respond to on LinkedIn:.Why will Google index web pages when they can not also find the material?Featured Graphic through Shutterstock/Krakenimages. com.