Ancient holy wars Dead religions, holocausts New regimes, old ideas That's now myth, that's now real Original sin, genetic fate Revolutions, spinning plates It's important to stay informed The commentary to comment on
-Father John Misty, “Holy Shit”
This essay is in two parts. The first part contains instructions on how to defeat the (soft) paywalls used on many news websites. The second part contains philosophical reflections on the value of information, and how I hope using the techniques in the first part will lead you to consume less "news" overall.
Paywalls, Hard and Soft
I’m going to take it as as given that my sophisticated readership understands how major online platforms like Google and Facebook are able to provide their content for free at the point of access. If you need a refresher, see the footnote.1
The alternative to providing content for free, of course, is making people pay for it. Thus the paywall - a digital block on the consumer’s access to content, whether that is a news story, an entertaining video, or a game.2 These paywalls come in two varieties: hard and soft.
Hard Paywalls are absolute: the content is not readable or searchable by any outside entity except for a paid-up subscriber. Substack, Patreon, and other online platforms like XBox Live and Netflix use hard paywalls as a true gate around their content: if you do not pay, the web server will not serve you the content. A few newspapers and magazines, notably the Economist, the Financial Times, and the Wall Street Journal, operate in this matter.
Technically, the paywall runs through user authentication: only authenticated users, those with accounts and paid-up membership dues, can access the content. All other users meet a firm block, with only a few sentences of the content available to preview, if that.Soft Paywalls use web coding, mostly JavaScript, to block your access to content, rather than strictly server-side controls. These sites, like the New York Times, may allow you a certain number of ‘free’ reads in a month, but after you reach the limit you are cut off, and the server stops you from scrolling further down, or places a blocking popup over the text asking you to subscribe. But the whole text is available in the HTML.
The sites using soft paywalls are playing a double game: they want people to subscribe to access their content, but they still want it to be shareable over social media and indexed by search engines. For these two things to happen, the block cannot be total. Chances are, most people seeing the link to a New York Times story in a social media post will not have used up their quota for the month, and thus be able to read the story. This will draw more people to the story, improving the story’s user engagement metrics. This is to the Times’ benefit because it can drive subscriptions - if you’re finding yourself being linked to the www.nytimes.com a lot, you may think you might as well subscribe - and it lets the newspaper collect advertising revenue for ad views. If potential readers cannot read enough of the content to decide to share it, then these things cannot happen.
The linking through social media further serves the Times because it raises the visibility of the story in search engines. The New York Times story on a particular issue of public moment will almost always rank higher than that from a site that uses hard paywalls like the Economist. This in turn drives social media sharing in a positive feedback loop. Someone may not have read the New York Times story yet, but they hear through friends or some other media about the story, search for it, click on it, read it, and very likely decide to share it.
To be clear: I think soft paywalls are bullshit. They lead, consciously or not, to the shaping of news less for its information value than for its shareability and noticeability. This is an old issue with profit journalism in a free society: since the providing of news is a business, and needs to make money, and is mostly subsidized by advertising rather than people buying the newspapers/tuning in their televisions, editors are going to select for sensation rather than information. The problem is made worse - orders of magnitude worse - by the fact that news editors and their managers and corporate overseers can now see up to the minute engagement and tracking stats. And what can be easily measured will be optimized for.
But more about that later. You’re here to access sensational, titillating, shareable “news.” Here’s how:
The Hard Way - If you’re willing to do so, you can directly access the HTML of a web page through your browser. From the soft paywalled page, use your browser’s View Source or Inspect tools to see the text. Often, if you scroll down, you will be able to find the text of the article. To get a jumpstart, copy the first sentence or two of the article and search for them directly on the source page. Or search for <p> paragraph tags.
Internet Archive Wayback Machine - The Internet Archive regularly crawls major news sites. Because the Archive’s web crawlers hop between IP addresses, they are not affected by the soft paywalls, as the servers think this is their first visit.
To access a news story backed from a soft-paywalled site, type https://web.archive.org/web/*/ in your browser’s search bar, followed by the URL of the story. If the page has been crawled, you’ll be given some options about which crawl you wish to access. Often these are redundant, but depending on the site sometimes you can see interesting editorial changes, mostly in the headlines as they are changed based on A/B testing. You can also, if the page has never been crawled before, ask the Archive to preserve it and send you a link to the new archive.Google Cache - Google’s web crawlers, like the Internet Archive, regularly crawl and cache major news websites. Google’s crawlers look at the HTTP of the sites directly, following links, so they are not blocked by soft paywalls.
To access a cached site, enter the site’s URL into the search box on Google’s main page. If you are using Google as your default search engine, prefix the URL with site: to search Google rather than be taken directly to the web link. Once you get the search results, select the link to your original URL, and click the three dots at the end of the title. This will bring up a popup with information Google has about the site. One of the buttons will say Cached. Click this to see a (very plain, mostly text) version of the website, without the soft paywall.JavaScript Blocking - If you tried the Hard Way listed above, you’ll have noticed that vast swathes of major websites are taken up with JavaScript code. Some of this is UI configuration, detecting what device and browser you are using to deliver a better experience to you (no reading microdot-sized font on your smartphone, for example). But most of the rest of it, 5x to 50x more than the text of the actuql article, will be tracking code, browser cookies, and the code that runs the paywall.
To get around this, use a browser extension like uBlock Origin that allows you to disable the JavaScript on a site. Simply go to the blocked page, turn on the extension’s JavaScript blocker, then reload the page. Voila (most of the time)! There is the text of the website for you to read.
The Other Side of the Wall
Now for what I’m sure you, as loyal readers of this Substack, really came here for: recondite philosophical musings about information, news, and problem solving.
So now that you can read every soft paywalled article you want, what next? Now it’s time to take a serious look at the actual value of the information the purveyor tried to get you to pay for.
Giving something a price has an endowment effect - we tend to believe something is worth what we paid for it, even when no one else does. Witness people keeping their house on the market at the same price they bought it for, against the advice of their real estate agent, even when houses around them are selling for much less. In the same way, charging a price for something can make people think it is more valuable than they would if they were given it for free. Sites using hard paywalls are making a statement about the value of their information: it’s so valuable they’re not going to give away even a small part of it: if you want it, they implicitly say, pay for a subscription.
As I said above, a company using a soft paywall is attempting to have it both ways. They want to collect subscriber revenue, but they want to have their content be easy to search and share. This means that the value they place on their content is primarily its exhange value rather than its informational value.
Let’s take a step back and think about information. Practically, information you get should inform your decision making. Schematically, you have a problem, a gap between the way the world is and the way you desire it to be. Perhaps, you want a muffin. In that case, information on where muffins can be procured is valuable to you, reducing your set of possible solutions to your existential muffinlessness. Information about where muffins are not has negligible value, especially if it is far outside your context of action (“there are no muffins on Titan at the present hour”). If the potential set of solutions is smaller, the negative information can have a slightly greater value (“there are no muffins at the neighborhood McDonalds”). Positive information is more valuable, as it immediately reduces the potential solution set.
If the information were not available for free, and your desire for a muffin was very great, and time was short (not enough time to visit all the potential muffin vendors) then you would likely be willing to pay some amount for info on verified muffin sources. You would pay more for positive info, less for negative info, and nothing at all for unrelated information.
Now think hard about all the news stories you have consumed in the past twenty-four hours, the past week, and the past month. Chances are most of them have vanished from your memory. The reason they did so is because they contained no information worth knowing - nothing that helped you decide what to do to solve a problem. You spent time reading them, were momentarily diverted by them, perhaps expressed an opinion about them to your spouse or friend or coworker or your social media followers, but did not one thing in your life consequentially differently on account of the news.
Few of us live lives and face choices that even a year worth of daily newspapers could inform. Except, of course, for knowledge of what is the news.
The news that soft paywall sites deal in is oriented towards the maximally understandable and shareable - stories that are fit only to be news and not information. People being people, are going to primarily share stories in order to express themselves - see the kind of person I am because of the teardrop emoji I put on this article about something sad. See the angry face I put on this story about something our shared side of some political conflict perceive to be an injustice.
Hardly anyone shares the traffic or weather report, though these are the news that is most immediately and consequentially useful: do I take the highway to my destination, or should I work my way there through rural roads and the suburbs? Do I take a raincoat, or will shorts and a wide-brimmed hat be more appropriate?
When news does not serve decision-making, it serves for entertainment and as a token in social exchanges with others, a chance to signal, to be seen to be informed, and to appear to oneself as a responsible citizen. Bill Z is going to pass the legislature, and this will be Bad. I am against Bill Z, and (oddly) so are all of my friends. I post an angry Instagram about Bill Z, which gets many shares, likes, and positive comments supporting me in my anger.
Bill Z passes anyway, and we all find something else to get angry about.
Benjamin Graham has a timeless piece of investing wisdom: before purchasing a stock, ask yourself if you would be happy owning X% of this company if there was no ready market for your shares.p Consider the following question about the news you consume: imagine if the story is true, but you could not (perhaps via magic or a mental block) tell anyone the news you received. Would you pay for that piece of news? You could take some concrete action in the world in response to it, but it has zero social capital - you cannot signal anything about yourself by stating it.
How much would you pay for such information? And, conversely, what kind of information would you want to get for money?
This is the position the subscribers to the Economist, the Wall Street Journal, and the Financial Times are in: they subscribe not for the social capital, but because at least a portion of the news contained in those publications drives their decisions, reducing the uncertainty of how to act in the world. If you manage a pension fund with international holdings, you need good information about the stability of the nation's currency, its inflation rate, the growth rate of the GDP. What then is the position of the New York Times subscriber?
Simply put, one subscribes to the Times to feel oneself to be in the know, and to appear to others to be in the know. If your social set is full of people discussing the latest opinion columns by a Times opinion writer then you need (in an extremely artificial sense) to know, regardless of how little impact that opinion has on any real decision you will make or for which you need to take account of how others will decide.
I’m not saying that learning information just to share it is irrational - getting along with others, having a sense of the “ongoing conversation” can be important, just like pretending to know about professional sports in some circles - but there’s an opportunity cost issue here. You could have done something else with the time - if you think being an engaged citizen is important, reading reports by civil society nonprofits or government agencies would be massively more worthwhile than reading those same reports filtered through the biases, editorial dictates, and misunderstandings of some time-crunched journalist without a degree in the subject matter. You could even have spent the time more enjoyably playing a game, going for a walk, or learning some skill either for play or work.
I hope that the ability to consume more news for free will help you to understand the valueless nature of general news, and consume less of it. May you find better things to do with your time than knowing which “commentary to comment on.”
Google and Facebook are, like newspapers and television channels before them, attention merchants. That is, they offer attractive and potentially useful services for free to the consumer (search, social networking, video streaming, email, instant messaging, games, and a host of others) in order to draw consumers to them. Once they are there, the attention of those consumers, their time spent on the site, can be sold to advertising companies.
There is another, more sinister alternative available to game providers: microtransactions and exploiting ‘whales’, the tiny minority of players at the far end of the Pareto distribution who will spend ludicrous amounts of money to guarantee themselves wins in online games.