.Google's Gary Illyes uncovers Googlebot accumulates web links rather than "complying with" them, negating main documents.Googlebot picks up hyperlinks instead of following them in real-time.Gary Illyes shared this relevant information in Google's 'Browse Off The File' podcast.Google.com's records negates the actual crawler habits.