Photo Corners headlinesarchivemikepasini.com
A S C R A P B O O K O F S O L U T I O N S F O R T H E P H O T O G R A P H E R
Enhancing the enjoyment of taking pictures with news that matters, features that entertain and images that delight. Published frequently.
Site Note: Bandwidth Update
23 October 2019
We were distressed to see our bandwidth quota disappear in September, threatening to shut down the site near the end of the month. To keep the site live, we blocked all indexing by the big search engines. And we made it.
At the start of this month, we unblocked the major ones (Google, Bing, etc.) but not a couple of new ones. And we monitored bandwidth on a daily basis with a new shell script running as a cron job to tell us how close we were to the daily average for our bandwidth quota as well as historical highs.
A look at the site logs revealed Google has been visiting the site excessively lately.
So how is October going? we wondered.
We were alarmed to see our daily bandwidth figure vault over the maximum average we could sustain four times in the first few days of the month.
What's going on? we wondered.
A look at the site logs revealed Google has been visiting the site excessively lately. The entire site of 30,748 URLs is about 5-GB and it appears multiple Google bots were mindlessly grabbing the whole thing.
Blocking Google from indexing the site isn't really a wise option. So we set a less frequent crawl rate than what Google's crawler preferred.
We had done that before, of course, but it expires every 90 days. Usually we get a notice and reset it. This time was different.
Since we reset the crawl rate, our daily bandwidth numbers easily fit in our bandwidth allocation for the month. The feeling is like fitting comfortably into a favorite old suit. Elegant.
We want to thank those of you who subscribed or made contributions during the crisis. We try to live within our means here and to the extent we can control things, we do.
But as everyone knows, that isn't always possible. And we appreciate those who responded to our call for help.