Do You Work For A Newspaper?


You may dislike this so.

Keith writes:

This might be of interest in relation to your post earlier regarding the National Newspapers of ireland taking issue with search engines ‘pirating’ their content.

All the major search engine crawlers will obey a file called robots.txt where you can set rules as to what they are allowed to index.

All of the major newspaper can make use of this file and you will be surprised to know  all the major Irish newspapers explicitly allow search engines to index their content.

If they don’t want links to their content showing up in search results all they have to do is add the following line to the file and all will be sorted,

User-agent: *
Disallow: /

But why would they do so as most of their traffic is from search engines?


Earlier: The Dead Tree Trolls

23 thoughts on “Do You Work For A Newspaper?

  1. Kevin

    I work in SEO and it baffles me that they wouldn’t want to be crawled by google, especially as they’re closing off a large revenue stream in the process.

  2. Clampers Outside!

    OK class…. typo test.

    How many typos in this sentence?

    “All of the major newspaper make use this file and you will be surprised to know that the all explicitly allow search engines to index their content.”

    1. Kath

      Should be:
      “All of the major newspaper[s] make use [of] this file and you will be surprised to know that the[y] all explicitly allow search engines to index their content.”

      1. Domquixote

        Clearly the original post was from an Irish journalist, see any of our papers for hundreds of further examples. Do we not have sub-editors in this country?

      2. Greg

        Should it not be, “surprised to learn” as we either know or don’t know something but we can learn things? Just sayin’

  3. Daniel Sullivan

    i’ve never quite understood why the print interst when publishing their content on-line haven’t bothered to fund a service along the files of but modifiied so that it would ensure that the original source creator was always the first to appear in search results meaning that no one could then ‘steal’ their content or the traffic associated with it. Or could it be because a considerable of the content in the press is lifted from elsewhere whether online or simply less prominent sources.

  4. ironcorona

    It’s not linking in the sense of creating a hyperlink to a site. The issue seems to be that the papers don’t want aggregators taking the text from their sites and using it elsewhere. Such as a page that doesn’t have their ads on it.

    1. No Fun

      In my head that’s the same as them wanting to charge people for reading the headline of a print newspaper in a queue at Tescos.

      If I’m interested, I’ll by it. If not, I’ll leave it.

      News sites seems to want to charge us for not being interested enough in their content to actually want to read it.

      1. Daniel Sullivan

        The printed press does have a legitimate issue in that with more and more people able to get free news online that unless they out their content out online for free then no one will read it, but if it is there for free then they have little or no revenue stream. So long as someone have decent willing to publish news content online for free then all the others have to follow suit and that’s killing them. It does cost money to generate the content, the question is do we really need as much coverage as there is when so much of it is so similar.

      2. Neil

        Google News doesnt have any ads on it so its not like they are stealing ad revenue from the papers.

        Exactly No Fun. I was in Easons earlier and browsed through lots of magazines while i was there. Next step, cctv cameras totting up what you have read and giving you a bill at the door.

  5. nickm

    I think it would be great if online newspapers stopped being linked to by google.

    Provided of course the newspapers in kind stopped using blog and forum posts as the inspiration and source quotes for their articles.

    And that f**king ‘best of youtube’ segment on sky news.

  6. Neil

    Great post Broadsheet. Murdock complains about Google all the time, similarily to most news orgs. Last year, News Int blocked their papers from appearing in Google news. Once they realised how much traffic visiting their sites fell they unblocked them all. It’s a stupid argument from the papers. Just more crying because they dont make the large profits of the boom years. Without the use of news aggregators, I would not know about, nor visit, the majority of news sites that are out there.

    1. lolly

      I would not know about, nor visit, the majority of news sites that are out there.
      this sentence is clearly a lie. do you spend your time looking at porn? most people use the internet as an information source and the best information is from news sites. if you don’t use news sites then what do you look at? Porn perhaps?

      1. Neil

        That’s only part of the sentence. You left out “without the use of news aggregators”. They help to quickly find news you want to read.

        You should read the comment properly first before commenting, Though thanks for letting us know what you look at on the net when you time on your hands!

Comments are closed.