Photo Corners headlinesarchivemikepasini.com
A S C R A P B O O K O F S O L U T I O N S F O R T H E P H O T O G R A P H E R
Enhancing the enjoyment of taking pictures with news that matters, features that entertain and images that delight. Published frequently.
Update: The Google Attack
10 July 2023
Over the weekend, we took a rather extreme measure to protect your access to the site by blocking Google enirely from the site. We'd tried setting its crawl frequency back but that didn't stop it from continually downloading the entire 4.83-GB site.
Bytes Downloaded. In just the first week this month, over 22,774,859 bytes have been downloaded, threatening us with a $2,569.05 usage charge.
That seems to have stopped the bleeding. You can see in the fragment of our Webalizer chart above that, after deny Google, our normal level of activity was restored on July 7. July 8 had not yet been fully reported.
Of course, Google complained to us in an email about "new page indexing issues" at the site. With access denied, it can't index new pages.
Yeah, well, behave yourself and we'll see. Meanwhile, you're grounded until the end of August, Google. Your crawl behavior is ridiculous.
To properly index the site, all that has to be done is read the robots.txt file to see what files may be indexed, get the file modification dates for those files from the server, compare to the current index, download only the revised or new files and index those.
We know no legitimate reason to download the entire site every time a new file is posted. And typically that is not the behavior of Google or any other indexing bot we allow on the site (it's back to Bing for us).
As a consequence of our action, Google won't include any of our July or August articles in any search until September. But that's the price of being able to keep the site accessible to you through the end of the month. We have a 150-GB allowance and we've used 22-GB in a week.
But even under such a barrage as this, we want you to know that your use of the site does not really exacerbate the problem. In fact, despite the blizzard, we have continued to publish without restriction and welcome your visits.
They're legitimate and easily accommodated. Google's extreme battering of the site is not.