It was disappointing to note that after switching to HTTPS, and signing on to Bing Webmaster Tools, the search engine results for those sites of ours that made the change are still severely compromised.

I’ve written about searches for my own name earlier, where my personal and company sites lost their first and second positions on all search engines that I knew of after we made the switch. Only Google has my personal site back up top, with the company site on the middle of the second page. Bing has my personal site at number two, and I’d love to tell you where the company site is, but their search engine results’ pages won’t let me advance beyond page 2 (clicking ‘next page’ lands you back on the same page; clicking ‘3’ and above still keeps you on p. 2). Duck Duck Go, which uses Bing results, has it well below that—I gave up looking. And this is after I signed up to Bing Webmaster Tools in the hope I could get the sites properly catalogued.

It’s a real shame because Duck Duck Go has been my default for 12 years this August.

However, it was the loss of search results for Lucire that really bothered me. Here’s a site that’s 25 years old, with plenty of inward links, and c. 5,000 pages. Before the switch to HTTPS, the popular search engines had thousands of pages from our site. These days, Bing and Duck Duck Go tell me they have dozens of pages from Lucire’s website. Again, only Google seems to have spidered everything.

When I check Bing Webmaster Tools, the spidering has been shockingly poor.

The received wisdom that you should have HTTPS instead of HTTP to do better in search engines is BS, and the belief that search engines will eventually catch up has also not been realized. We made the switch in March, and I’m to believe that Bing hasn’t completed the indexing of our sites.

Are they using the same computers New Zealand banks do? (Cheques used to clear overnight in the 1970s, and now banks tell us that even electronic payments can take days. When we last used cheques, they were telling us they would take five to seven days. Ergo, bank computers are slower today than in 1976.)

The real downer is that Lucire’s website search box is powered by Duck Duck Go, so our own site visitors can’t find the things they want to look for. If you believe some of the search engine marketing, over 40 per cent of site visitors use your search function.

What to do?

I began looking at having an internal search again. We used to have a WhatUSeek (later SiteLevel) internal site search, but that site’s search functions appear to be dead (the site is still live). A user on Mastodon recommended Sphinx Search, an open-source internal site search, but the instructions were too complex. I even saw real computer geeks having trouble. The only one that I could understand was called Sphider—I could follow the instructions and knew enough about PHP and MySql—but it was last updated many years ago, and successive projects also looked a bit complex.

A Google internal search was absolutely out of the question, as I have no desire to expose our readers to tracking—which is why so many other Big Tech gadgets have been removed from our site(s). Baidu and Yandex also have very limited indices for our sites.

I am very fortunate to have tried Mojeek again, a British search engine recommended to me by Matias on July 2. What I didn’t know then was Mojeek has its own spider and its own index, so it doesn’t have to license anything from Bing. And, happily, it claims to have 3,535 results from lucire.com, which might not be as good as Google’s 5,830, but it beats Bing’s 50 earlier today—in fact, at the time of writing, it showed a grand total of 10. That’s how bad it’s got. Duck Duck Go now has 48, also down from a few thousand before March.

Like Google, it seems to have coped with the switch to HTTPS without falling to pieces! And guess what? For a search of my own name, my personal site is number one, and our work site is number two. Presumably, Mojeek is the only search engine which coped and behaved exactly as the experts said!

You can imagine my next move. Mojeek has a site search, so now all Lucire searches are done through it. And readers can actually find stuff again instead of coming up nearly empty (or having very irrelevant results) as they have done for months.

Duck Duck Go’s lustre had been wearing off as there were recent allegations that its browser allowed Microsoft to track its users, something which Duck Duck Go boss Gabriel Weinberg personally denied on Reddit, saying that users were still anonymous when loading their search results.

I still have good memories of chatting to Gabriel in the early days and figuring out ways of spreading the word on Duck Duck Go. My contribution was going to hotels and changing the search defaults on business centre computers. Back then I had the impression Duck Duck Go did some of its own spidering, but these days, if Bing has a shitty index for your site, the Duck will follow suit. And with HTTPS not living up to its promise, that’s simply not good enough.

Tonight, Mojeek is very much the site of the day here, and I heartily recommend you try it out. I’ve switched the desktop to Mojeek as a default, and I’ll see how it all progresses. Right now I feel it deserves our support more than Duck Duck Go. Finally, we might truly have an alternative to Google, and it’s run from the UK’s greenest data centre. With our servers now being greener, too, running out of Finland, the technology is starting to match up to our beliefs.
 

Google, the biggest index of them all
 

Mojeek, a creditable second place
 

This is it on Bing: a 25-year-old history on the web, and it says it has 10 pages from lucire.com. Altavista, Excite and Hotbot had more in the 1990s
 

Duck Duck Go is slightly better, with 48 results—down from the thousands it once had
 
After switching to HTTPS
Number of results for lucire.com
Google: 5,830
Mojeek: 3,535 (containing the word Lucire, as term-less searches are not allowed)
Duck Duck Go: 48
Bing: 10
 
Number of results for jackyan.com
Google: 878
Mojeek: 437 (containing the term “Jack Yan”)
Duck Duck Go: 54
Bing: 24
 
Number of results for jyanet.com
Google: 635
Mojeek: 297 (containing the word jyanet)
Duck Duck Go: 46
Bing: 10
 

Presumably the only search engine that could handle a server going from HTTP to HTTPS and preserving the domains’ positions