I was chatting to another Tweeter recently about the Ford I-Max, and decided I’d have a hunt for its brochure online. After all, this car was in production from 2007 to 2009, the World Wide Web was around, so surely it wouldn’t be hard to find something on it?
I found one image, at a very low resolution. The web’s not a repository of everything: stuff gets removed, sites go down, search engines are not comprehensive—in fact, search engines favour the new over the old, so older posts that are still current—such as this post about the late George Kennedy—can’t even be found. This has been happening for over a decade, so it shouldn’t surprise us—but we should be concerned that we cannot get information based on merit or specificity, but on novelty. Not everything new is right, and if we’re only being exposed to what’s “in”, then we’re no better at our knowledge than our forebears. The World Wide Web, at least the way it’s indexed, is not a giant encyclopædia which brings up the best at your fingertips, but often a reflection of our bubble or what the prevailing orthodoxy is. More’s the pity.
I can’t let this post go without one gripe about Facebook. Good news: as far as I can tell, they fixed the bug about tagging another page on your own page, so you don’t have to start a new line in order to tag another party. Bad news, or maybe it’s to do with the way we’ve set up our own pages: the minute you do, the nice preview image that Facebook extracted vanishes in favour of something smaller. I’ll check out our code, but back when I was debugging Facebook pages, it was pretty good at finding the dominant image on a web page. Lesson: don’t tag anyone. It ruins the æsthetic on your page, and it increases everyone’s time on the site, and that can never be healthy. Time to fight the programming of Professor Fogg and his children (with apologies to Roger McNamee).
Top: The post Facebook picks up from an IFTTT script. Above: What happens to a post that once had a proper image preview after editing, and tags added.
My friend Richard MacManus wrote a great blog post in February on the passing of Clive James, and made this poignant observation: ‘Because far from preserving our culture, the Web is at best forgetting it and at worst erasing it. As it turns out, a website is much more vulnerable than an Egyptian pyramid.’
The problem: search engines are biased to show us the latest stuff, so older items are being forgotten.
There are dead domains, of course—each time I pop by to our links’ pages, I find I’m deleting more than I’m adding. I mean, who maintains links’ pages these days, anyway? (Ours look mega-dated.) But the items we added in the 1990s and 2000s are vanishing and other than the Internet Archive, Richard notes its Wayback Machine is ‘increasingly the only method of accessing past websites that have otherwise disappeared into the ether. Many old websites are now either 404 errors, or the domains have been snapped up by spammers searching for Google juice.’
His fear is that sites like Clive James’s will be forgotten rather than preserved, and he has a point. As a collective, humanity seems to desire novelty: the newest car, the newest cellphone, and the newest news. Searching for a topic tends to bring up the newest references, since the modern web operates on the basis that history is bunk.
That’s a real shame as it means we don’t get to understand our history as well as we should. Take this pandemic, for instance: are there lessons we could learn from MERS and SARS, or even the Great Plague of London in the 1660s? But a search is more likely to reveal stuff we already know or have recently come across in the media, like a sort of comfort blanket to assure us of our smartness. It’s not just political views and personal biases that are getting bubbled, it seems human knowledge is, too.
Even Duck Duck Go, my preferred search engine, can be guilty of this, though a search I just made of the word pandemic shows it is better in providing relevance over novelty.
Showing results founded on their novelty actually makes the web less interesting because search engines fail to make it a place of discovery. If page after page reveals the latest, and the latest is often commodified news, then there is no point going to the second or third pages to find out more. Google takes great pride in detailing the date in the description, or ‘2 days ago’ or ‘1 day ago’. But if search engines remained focused on relevance, then we may stumble on something we didn’t know, and be better educated in the process.
Therefore, it’s possibly another area that Big Tech is getting wrong: it’s not just endangering democracy, but human intelligence. The biases I accused Google News and Facebook of—viz.their preference for corporate media—build on the dumbing-down of the masses.
I may well be wrong: maybe people don’t want to get smarter: Facebook tells us that folks just want a dopamine hit from approval, and maybe confirmation of our own limited knowledge gives us the same. ‘Look at how smart I am!’ Or how about this collection?
Any expert will tell you that the best way to keep your traffic up is to generate more and more new content, and it’s easy to understand why: like a physical library, the old stuff is getting forgotten, buried, or even—if they can’t sell or give it away—pulped.
Again, there’s a massive opportunity here. A hypothetical new news aggregator can outdo Google News by spidering and rewarding independent media that break news, by giving them the best placement—as Google News used to do. That encourages independent media to do their job and opens the public up to new voices and viewpoints. And now a hypothetical new search engine could outdo Google by providing relevance over novelty, or at least getting the balance of the two right.
Don’t believe everything you read on the internet, e.g.:
Anyone alive during this period will be wondering, ‘Where’s Altavista?’ Just on visitor numbers, as opposed to visits per month, they were doing 19 million daily in 1996, 80 million daily in 1997. Goodness knows how many searches we were doing per day. Yet they are nowhere to be seen on this animation till December 1997, with 7 million monthly visits. If you were anything like me, you were using Altavista countless times a day—even conservatively, say you went on four times daily, and you were one of the 80 million per day, you would be putting Altavista at 9,600 million visits per month, dwarfing AOL and Yahoo. By 1997–8, we weren’t using directories like Yahoo for search, we were using these newfangled search engines. Google only overtook Altavista in 2001 in searches.
And I am old-fashioned enough to think this channel should be called Data Are Beautiful, not Data Is Beautiful.
I’d love to know the sources: not only could I remember clearly Altavista’s position (and Alexa had them in number one as well), it took me no time at all to confirm my suspicions through a web search.
Above: A re-created Altavista home page from 1999.
Keen to be seen as the establishment, and that means working with the military–industrial complex, Google is making software to help the Pentagon analyse drone footage, and not everyone’s happy with this development.
The World Economic Forum’s ‘This is the future of the internet’ makes for interesting reading. It’s not so much about the future, but what has happened till now, with concerns about digital content (“fake news”), privacy and antitrust.
Others have written a lot about search engines and social media keeping people in bubbles (or watch the video below, but especially from 5′14″), but the solution isn’t actually that complex. It’s probably time for search engines to return to delivering what people request, rather than anticipate their political views and feed them a hit of dopamine. They seem to have forgotten that they exist as tools, not websites that reinforce prejudices. Duck Duck Go has worked well for me because it has remained true to this; but others can do it, too.
However, there needs to be one more thing. Instead of Facebook’s botched suggestion of having everyday people rate news sources, which I believe will actually result in more “bubbling”, why not rank websites based on their longevity and consistency of delivering decent journalism? Yes, I realize both Fox News and MSNBC will pass this test. As will the BBC. But this weeds out splogs, content mills, and websites that steal content through RSS. It actually takes out the “fake news” (and I mean this in the proper sense, not the way President Trump uses it). The websites set up by fly-by-nighters to make a quick buck, or Macedonian teenagers to fool American voters, just disappear down the search-engine indices. Facebook can analyse the same data to check whether a source is credible and rank them the same way.
It could be done through an analysis of the age of the content, and whether the domain name had changed hands over the years. A website with a healthy archive going back many years would be ranked more highly; as would one where the domain had been owned by the same party for a long period.
Google’s Pagerank used to look at incoming links, and maybe this can still be a factor, even if link-exchanging is no longer one of the basic tenets of the web.
There’s so much good work being done by independent media all over the world, and they deserve to be promoted in a truly meritorious system, which the likes of Google used to deliver. Shame they do not today.
We do know that its claim that analysing the content on the page to determine rank hasn’t worked, if some of the results that pop up are any indication. Instead, we see Google News permit the most ridiculous content-mill sites and treat them as legitimate sources; in 2005 such behaviour would be unthinkable by the big G. As to Facebook, they’ll boost whomever gives them money, so ethics don’t really score big there.
Both these companies must realize they have a duty to do right by the public, but they should also know that it’s in their own interests to be honest to their users. If trust increases, so can usage. They might even ward off some of the antitrust forces looming on the horizon; fairness certainly will help Google’s future in Europe. But they seem to have forgotten they are providers of tools, perhaps reflecting their principals’ desires to be seen as tech celebrities or power-players.
Google already has the technology to deliver a fairer web, but I sense it doesn’t have the desire to. I miss the days when Google, in particular, was an enfant terrible, there to shake things up. Now it exists to boost its own properties or rub shoulders with the military–industrial complex. Everyone’s keeping an eye on Alphabet’s share price. Forget the people or ‘Don’t be evil.’
As I have said often on this blog, there lies a grand opportunity for others to fill the spaces that Google and Facebook have left. A new site can play a far more ethical game, maybe even combine what these two giants offer. If Altavista, once the world’s biggest website, and Myspace, once the king of social networks, can be toppled, then so can these two. Yet at their peak, neither appeared to be vulnerable. Who would have thought back in 1998 that Altavista would be toast? (The few that did, and you are out there, are visionaries.)
So who is best poised out there to deliver such tools? It would seem now is the time to start, and as people realize that this way is better, be prepared to scale, scale, scale. Remember, Google once did the same thing to oust Altavista, by figuratively building a better mousetrap. Someone just needs to take that first step.
The troubles with Google that I’ve faced—privacy breaches, Ads Preferences Manager not honouring its claims, fighting for six months on behalf of a friend over a deleted Blogger blog, Chrome being buggy (but not nearly as badly as IE9), phantom entries in my Google dashboard, unanswered messages—would suggest, to anyone studying business or a graduate from B-school, that there is something very, very rotten inside the company. It’s being evil.
Judging by an article I linked yesterday from Techcrunch, there probably is something rotten.
It’s sad to see that Techcrunch didn’t have the ethics to keep an off-the-record comment off the record—it even plays an answerphone message on its site, which I am sure its speaker never intended for broadcast—but it does make an interesting guess of the company’s internal problems.
I’ve heard of similar things second-hand and, in at least one case, first-hand, but this one illustrates that the problems could be at quite a senior level.
With all the internal politicking going on, a few people are doing their jobs correctly, and honouring Google’s commitment to its users. In 2010, I named Rick Klau at Blogger as being one of them. I reckon the other has to be Matt Cutts, whose initiative to cut down content mills and Google-spam I applauded some weeks ago as being one of the company’s right moves.
Matt has done his job so well that it has cut down even Google’s own content mill, the Google Places site.
He deserves even more applause because he’s not singling out his own employers for special treatment, which means, as far as the rest of us are concerned, we face a level playing field getting on the site.
He’s even stated, ‘Google absolutely takes action on sites that violate our quality guidelines regardless of whether they have ads powered by Google.’
What is interesting is that it has pissed off certain people inside Google, who have become accustomed to the search engine biasing results toward itself—something it has admitted on some occasions, contradicting its stated policy on other occasions. Élitism much?
Among the content mills Matt’s team has targeted includes the sites of Demand Media, who I had a run-in with as well over contradictory terms and conditions and the company’s refusal to respond. (In fact, it continued to pester me to integrate an account I had with a firm it had acquired even though, legally, under its own terms, I could not.)
Reading the Techcrunch piece, Matt Cutts is a hero for fairness and for running things exactly the way netizens expect. Some commenters agree. He might even be the guy who saves Google from being an élitist, unethical monster. He’s done exactly what he set out to do, and Google needs to realize that if it is to recover any mana for its misdeeds of the past few years, it has to clean its own doorstep first.
If the article is correct, other Google senior staff—Nikesh Arora, Marissa Mayer (who has already revealed that Google publishes biased results)—are part of the problem, and why Google is so desperate to violate its own stated policies repeatedly.
And if that off-the-record comment on Techcrunch is accurate, then Marissa Mayer probably believes that users are stupid. Way to earn that goodwill, Marissa.
I’m not enough of a bastard to only dis Google, because they have made a pretty good move today.
Google’s new algorithm, it is claimed, will weed out content farms, one type of site that has annoyed us here regularly.
These are sites that just pinch others’ content automatically. Because search engines pick them up, people visit their pages. Those pages are filled with ads—quite often supplied by Google. The content-pinchers make money, but the people who took the time to create the piece don’t.
I wrote, not a long time ago, that Google Blog Search had become entirely useless. That’s no exaggeration: head in there, and a lot of the blogs are scraped: they are duplicates of other sites.
In fact, when Vincent Wright’s blog was deleted, and I helped him to get it back, the Googlebot was trying to delete those scraped blogs. It’s just a shame that the Google machine was so damned useless at helping legitimate people get their blogs back, and intentionally stonewalled us to get some weird kick. If it were not for the Blogger product manager’s intervention, Vincent’s blog would still be in cyber-oblivion.
So the move, in principle, is a good one. Google claims, ‘If you take the top several dozen or so most-blocked domains from the Chrome [Personal Blocklist] extension, then this algorithmic change addresses 84 percent of them, which is strong independent confirmation of the user benefits.’
Let’s just hope that Google won’t mistakenly take out legit sites again (I have to ask what the other 16 per cent consists of!), though the fact that there has been some correlation with human editing (sites chosen by users for the Blocklist) helps.
I found out a day after many netizens: Google is now forcing all YouTube account holders to merge their accounts with their Google ones.
As part of my de-Googling, I won’t be following suit. Instead, I plan to stay logged out of YouTube: it makes very little difference to me.
So I won’t be able to comment or like a video—not the end of the world. In fact, I imagine I could very easily comment on videos on a blog and get any possible frustration I have out of my system that way. YouTube is still letting non-account holders embed. And I’m not really a YouTube video uploader: I can always go on Vimeo if I were that keen, or use SmugMug, which was in the digital photo-storage game long before Flickr, and which now hosts videos, too.
I felt very sympathetic when I found that there were people far more pissed off about this development than I am. The only news outlet to have reported on the compulsory linking that I could find, Brandchannel, has scores of unhappy users who are commenting that the move has even locked them out of YouTube. Others are concerned about their privacy, with good reason.
Looks like Google still hasn’t learned about user choices after the débâcles last year over Buzz and the other services it offers. If anything, it seems to be getting worse.
Remember, too, how Google has stated on numerous occasions that it would not bias search results? Consider this: I wanted to search for an old post of mine so I could link it from the above text. The term: Google Buzz “de-Googling”.
On Duck Duck Go, I found the post immediately:
Out of interest, on Google, it cannot be seen: only positive things are mentioned and Google Buzz itself is the first result.
I know I have done more obscure tests to show that Google’s results are getting less precise. But the above is interesting.
It backs up an earlier article I read online about how Google treats search results, and that there is actually some bias in the system now.
I don’t begrudge Google for doing this, but it needs to stop saying that it doesn’t. We all know that it was quite happy to engage in censorship when it had Google China, already making its brand less idealistic than it once was.
Having set this precedent and created this brand association, it’s easy to believe that it now does this quite selectively for a lot more countries.
You might say that my one search is not a sign of bias, merely one where Google has a less than comprehensive search index and it could not find three old blog entries that have been around for a while. And which it used to be able to find.
It’s quite a coincidence that three negative posts about Google are no longer easily found with the relevant search terms.
That’s not great news for Google, either. Duck Duck Go is looking better by the day as the Google search engine, the one service to which its brand is tied, gets less precise.
Aside from writing a branding report today (which I will share with you once all contributors have OKed it), I received some wonderful news from Gabriel Weinberg of Duck Duck Go.
Those who are used to the Duck will know that you can search using what he calls bangs—the exclamation mark. On Chrome, which has a minimalist design, I have set my default search engine to Duck Duck Go. But what if I wanted to search on Google?
I can either do what is built in, and what Google suggests, by beginning my typing in the search box with the word Google. Or, I can simply add !g to the end of the query, which, I might add, is something you can do from Duck Duck Go itself, too.
Of course, Google would prefer that I put all searches through them, but having Duck Duck Go as a default isn’t a bad idea. There’s a huge list of bangs that you can use at the Duck Duck Go website, which include !amazon (which will take you directly to an Amazon.com search), !gn (for Google News—this one is a godsend, especially for Chrome, which has made it much harder to search the news section, even if programmed into the search settings), !video (for YouTube), and !eb or !ebay (for Ebay).
I’m glad to announce that Gabriel has taken on board a few of my suggestions for motoring and fashion publications, such as !autocar, !vogueuk, !jalopnik, !randt (Road and Track) and !lucire (had to get that one in).
We’ll announce this on Lucire shortly, but readers already saw me announce, on Thursday, a nip–tuck of our Newsstand pages. That’s not really news, so I chose to complement it with a few other announcements.
Since we were already fidgeting with that part of the site, Gabriel’s announcement prompted me to do some changes to the search pages, which were woefully out of date. The community home page was last designed four years ago, and that time, we just shifted the content over. Never mind that that content was also out of date, and included some letters to the editor that are no longer relevant. It had a link to the old forum, which only results in a PHP error. So today, I had all the old stuff stripped out, leaving us with a fairly minimalistic page.
I didn’t plan on making a blog post out of it, so I never took screen shots of the process. But at left is one of the old page from Snap Shots, and long-time readers will recognize this as the website design we had many years ago. When we facelifted other parts of the site, this was left as is: it’s old-fashioned dynamic HTML and hasn’t been moved over to a content-management system.
The new one may be a bit sterile (below), but it takes out all the extra bits that very few used: the Swicki, the Flickr gadget (we haven’t added anything to it since 2008), and a complex sign-up form for the Lucire Updates’ service. It’s been stripped to basics, but it now includes the obligatory links to Twitter, Facebook, Vkontakte and the RSS feed.
Which brings us on to the search pages. These also have the updated look, but, importantly, I fixed a bug in the Perl script that kept showing the wrong month. Regardless of whether it was March or December, the script would show January:
This is probably nothing to the actual computer hackers out there, but for a guy who has used an ATM only three times in his entire life (mainly because I lacked the local currency), this is a momentous occasion.
It was one of those evening-tweaking cases where it was simpler for me to do it myself than to ask one of the programmers, and I managed to remedy something that had plagued our website for 10 years.
The searches revealed a few strange links from long-expired pages, and here is where we might get in to a bit of discussion about online publishing.
Once upon a time, it was considered bad form to have dead links, because people might point to them. Even more importantly, because a search engine might, and you could get penalized for having too many.
This is why we’ve kept some really odd filenames. The reason the Lucire Community page is at lucire.com/email is that the link to a free email service we provided at the turn of the century was linked from there. Similarly, we still kept pages called content.htm, contents.shtml and editorial.shtml, even though these pages had not been updated for half a decade.
There are now redirects from these pages, which were once also bad form as far as the search engines were concerned.
But, given that search engines update so much more quickly in 2010, do we still need to bother about these outdated links? Will we still be penalized for having them? Should we not just simply delete them?
If you look to the right of this blog at the RSS feed links, you’ll see some dead ones—there were more, but I have been doing online weeding here, too. It almost seems to be a given that people can remove things without warning and if you encounter a dead link, well, you know how to use a search engine.
To me, it still seems a little on the side of bad manners to do that to your readers, but one might theorize that few care about that any more as many sites revamp on to CMSs to make life easier for themselves.
A side note: earlier this week, when weeding through dead links at Lucire, we noticed that many people had moved to CMSs, with the result that their sites began to look the same. Some put in excellent customizations, but many didn’t. And what is it with all the big type on the news sites and blogs these days? Is this due to the higher resolutions of modern monitors, or do they represent a change in reading habits?
PS.: The search script bug was fixed by changing $month[$mon] to simply $mon. Told you it was nothing, though I noticed that the site that we got the script from still has the bug.—JY
For around a year, I’ve been at Google for its misbehaviours. And one thing I dislike about these tech companies—whether it’s Facebook or Google or any of their ilk—is how they are slaves to technology, rather than masters of it. Somewhere along the line, they have allowed algorithms to determine guilt, thereby offending that old-fashioned idea of the presumption of innocence. From Blogger blocks to false copyright-infringement accusations to, now, this:
While Duck Duck Go is my default now, occasionally I’ll still put a search through Google. There is no malware on this system, or on this network, and I certainly haven’t put through a single automated request (how could human typing be mistaken for this?!)—reasons Google gives for this message. It’s just another case of guilty till proved innocent that this northern California company, and others, are so good at creating.
Funny, isn’t it, that it has relied on an automated process to accuse a human process of being automated? It’s the Blogger fight all over again.
I hadn’t heard of Blekko, a search engine, till last week, so armed with a new entrant, I wanted to see how they all compared.
Blekko’s very pretty, and I’ve told Gabriel Weinberg, the man behind Duck Duck Go, just what it is that makes it attractive. Most of it is the modernist design approach it takes. But is it more functional?
I have a couple of tests. You may have heard me dis Google’s supplemental index, where pages it deems to be less important wind up. But who makes that determination? And what if there is a page in there that is actually relevant but Google fails to dig it up?
Google says the supplemental index doesn’t exist any more, but the fact remains that it fails to dig up some pages, especially older ones. So much for its comprehensive index.
The first test, therefore, is one I have subjected every search engine I encounter to: will it find a 2000 article on Lucire about Elle Macpherson Intimates’ 10th anniversary? It is probably the only article on the subject, and because of this test, I’ve even linked it this year so it can be spidered by the search engines. Last month, Google could not find it, though in 2000–1, it was very easily found.
If the search engines are as intelligent as their makers claim, it should be able to figure out these concepts and deliver the pages accordingly. The page itself is very basic with no trick HTML—just plain old meta data, as you would imagine for a ten-year-old file.
Will the search engines find it now, with a few more inward links?
Duck Duck Go: 1st Blekko: not found, though it locates a reference made on this blog and two others in Lucire, one going back to 2001, at positions 1, 2 and 12 Google: 73rd, with blog entries from here referring to it at 5 and 42, and another link in Lucire at 6 Bing: 1st with old frameset at 2nd Ask: 7th
Here’s the second test. In Wired, Google bragged about how its index could find a page about a certain lawyer in Michigan (mike siwek lawyer mi). Unfortunately for Mr Siwek, most of the top entries quickly became those about the Wired article and he was lost again in the index.
Mr Don Wearing, a friend of mine, is a partner in a shoe retail chain. If I typed “Don Wearing” shoes, which of the search engines will deliver me an entry referring to Don Wearing specifically and not some guy called Don who happens to be wearing shoes? (Not long ago, the best the search engines could do was around 12th.)
Duck Duck Go: 2nd
Blekko: says ‘No results found for: “Don Wearing” shoes’ but actually finds the article at 5th Google: 3rd Bing: 2nd Ask: 5th
Not bad: an improvement all round.
OK, how about speed of addition? Let’s see if the search engines will find the last entry in this blog, added a few hours ago. I’ll use the search term “Jack Yan” TPPA.
Duck Duck Go: not found Blekko: not found Google: found the main blog page Bing: found a link to it at MyBlogLog Ask: not found, but came up with seven irrelevant results
This is just a quick test based on three examples that might not reflect everyday use. However, the first two frustrated me earlier when I went to hunt for them on Google (and before I had heard of Duck Duck Go), which is why I remembered them, so admittedly Google was at a slight disadvantage in this test as a result. I never went to Bing or Ask regularly.
Therefore, I’m not going to draw any conclusions about who is best, but I will say that Google is quicker at finding new material. I would, however, encourage others to give these other search engines a go and see how effective they are. I’m very happy with Duck Duck Go, especially as it does not second-guess my queries with Google’s annoying ‘Showing results for [what Google thinks I typed]. Search instead for [what I actually typed]’. No, Google, I did not type my query wrong—so give me the results already!
I prefer Duck Duck Go’s approach, which is to treat the web more as a research medium. There is no hiding pages: it just delivers the most relevant result to what I typed, which is why I originally moved to Google at the end of the 1990s.
Judging by the above, I’m not convinced Blekko is ready for prime-time (which is why it still has a beta tag).
Of the five tested, it looks like it’s still the Duck for me, complemented with Google News. I’m way more impressed with Duck Duck Go’s privacy policy: no search leakage, no search history, and no collecting of personal information to hand over to law enforcement or, for that matter, the Chinese Politburo.
And in a year where people have shown that they care about privacy, Duck Duck Go seems to make more sense.
I was chatting to another Tweeter recently about the Ford I-Max, and decided I’d have a hunt for its brochure online. After all, this car was in production from 2007 to 2009, the World Wide Web was around, so surely it wouldn’t be hard to find something on it?
