It’s not enough to block just Semrush’s bot, SemrushBot, in one’s robots.txt, since they have a whole bunch of them. We’ve usually allowed all bots but with “AI” and Semrush’s terrible programming that wasted hours of my time every day for nine months, there are certain parties no longer welcome to crawl. To block Semrush […]
Month: October 2024
Quick test of index size on the occidental search engines
Google really doesn’t like this blog. When I do a site: search, it still puts HTML pages up top, often pre-2010, and as of yesterday, it trails Mojeek and Bing in terms of the number of pages in its index. Who knew? Google being last of the three. It got me wondering how […]
Read More… from Quick test of index size on the occidental search engines