What the WhiteHouse.gov website doesn't want Google to see...

[@_@ Artman]

http://www.whitehouse.gov/robots.txt

This is a robot text listing of what the White House wishes to disallow from Google searches.

I only posted this to show some of the interesting topics, names and other elements that has been listed. It probably won't be up long so take a gander.

A few nuggets:

Disallow: /911/911day/text
Disallow: /911/heroes/text
Disallow: /911/messages/text
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/text
Disallow: /911/progress/text
Disallow: /911/remembrance/text
Disallow: /911/response/text :wow:
Disallow: /911/sept112002/text
Disallow: /911/text
Disallow: /ConferenceAmericas/text
Disallow: /GOVERNMENT/text

Disallow: /bioshield/text :err:
Disallow: /birdflu/text
Disallow: /blackhistory/text

Disallow: /firstlady/newborn/backup/text :???:
Disallow: /firstlady/newborn/text
Disallow: /firstlady/news-speeches/releases/print/text
Disallow: /firstlady/news-speeches/releases/text
Disallow: /firstlady/news-speeches/speeches/print/text
Disallow: /firstlady/news-speeches/speeches/text
Disallow: /firstlady/news-speeches/text
Disallow: /firstlady/onemonth/backup/text
Disallow: /firstlady/onemonth/text



Disallow: /360pics/text
George doesn't want you to see his Xbox account :lol:

[Gon]

Are there any circumstances under which a government official should disallow a search engine from indexing material freely available on the Internet? I can't think of any. Sounds like a lose-lose to me.

[@_@ Artman]

Are there any circumstances under which a government official should disallow a search engine from indexing material freely available on the Internet? I can't think of any. Sounds like a lose-lose to me.

They're just making sure Google is directing newcomers to the first pages they want seen. Which, honestly, isn't more than slightly nefarious - many major websites do that.

[Gon]

But whatever they disallow in robots.txt, Google won't find at all, correct?

If the stuff they are hiding from indexing is something that needs to be seen in context (and is shown by some dynamic page, combined to other data) it would be far better if it could be indexed in Google and an effort to use the link offered by Google was redirected to where the data is in full context. Is this possible?

Also, is there a major search engine that does not honor robots.txt in case I want to find something Google can't?

[jimmac]

http://www.whitehouse.gov/robots.txt

This is a robot text listing of what the White House wishes to disallow from Google searches.

I only posted this to show some of the interesting topics, names and other elements that has been listed. It probably won't be up long so take a gander.

A few nuggets:

Disallow: /911/911day/text
Disallow: /911/heroes/text
Disallow: /911/messages/text
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/text
Disallow: /911/progress/text
Disallow: /911/remembrance/text
Disallow: /911/response/text :wow:
Disallow: /911/sept112002/text
Disallow: /911/text
Disallow: /ConferenceAmericas/text
Disallow: /GOVERNMENT/text

Disallow: /bioshield/text :err:
Disallow: /birdflu/text
Disallow: /blackhistory/text

Disallow: /firstlady/newborn/backup/text :???:
Disallow: /firstlady/newborn/text
Disallow: /firstlady/news-speeches/releases/print/text
Disallow: /firstlady/news-speeches/releases/text
Disallow: /firstlady/news-speeches/speeches/print/text
Disallow: /firstlady/news-speeches/speeches/text
Disallow: /firstlady/news-speeches/text
Disallow: /firstlady/onemonth/backup/text
Disallow: /firstlady/onemonth/text




George doesn't want you to see his Xbox account :lol:

Typical.:no::rolleyes:

[Jubelum]

Typical.:no::rolleyes:

Trying to direct people to the front page. Typical.
:lol:

Psst... there is a FUCKING SEARCH FEATURE on the TOP of the front page so all of you can go directly there and get whatever you want.

Mountain, meet molehill. Like anyone needs another reason here to hate the man.

With the myriad colossal fuckups of this Administration, the best we can toss over the fence is what parts of the Whitehouse web site are not indexed? Really?

[giant]

Trying to direct people to the front page. Typical.
:lol:
Interesting. No, they aren't doing it to direct people to the front page. Amazing that you don't actually have any idea why they are blocking most pages and yet you're still so quick to jump up and play defender. So much for your bullshit pretense of neutrality when it comes to this administration.

However, although your clearly unaware of it, what they are doing in most cases is indeed benign and actually pretty obvious.
Psst... there is a FUCKING SEARCH FEATURE on the TOP of the front page so all of you can go directly there and get whatever you want.
So what if there is a search feature? They could easily manipulate the index without anyone having any idea, which is precisely what every search tool does in one way or another for various reasons, intentional or not. And if it wasn't getting indexed by independent organizations, we could be able to have records when the white house does do shady things (http://lessig.org/blog/2003/12/speaking_of_new_speak_a_report.html).

[Jubelum]

Go and do a search for any of the (PANIC!) "blocked" terms and BAM- hundreds of documents.

The site's content is created and maintained by the White House. If they wanted something to not be there at all, they'd take it off- not put it in the robots list for PO posters to rub one out over. Rove and Cheney are smarter than to post "smoking gun" data on a website RUN BY THEIR OWN OFFICES. Duh. Serial detractors should head over to the national archives or somewhere besides the whitehouse website to find the crucifixion stones. There are sure to be plenty about. This Administration is much better at obfuscation than posting problematic stuff on their own website, and then doing some "cover up" with a robots file. Blocking a bunch of old events and photoessays seems to be rather benig.... oops... don't want to "abandon neutrality" by saying something in "defense." ;)

The indexing of a site (which Whitehouse staff produce documents for in the first place) is a non-issue.

As an aside, giant... one nice thing said, one simple "defense" in your holy opinion means that I'm "abandoning any pretense of neutrality." Big, big words. As far as neutrality goes, at least I post about good and bad... you can't find one damn nice thing to say in almost 7 years.

[giant]

Defender? Jeezus. Go and do a search for any of the (PANIC!) "blocked" terms and BAM- you'll see hundreds of documents. Just more of the same, giant. Just post "Fuck Bush," save yourself the keystrokes, me the boredom, and go on.
Is there anything of value somewhere in the midst of that torrent of nonsense? That you think the robots.txt exclusions are "to direct people to the front page" shows you have no idea what it's actually excluding in the vast majority of those cases. That you think that putting some words into a view's search field, hitting enter and seeing results means anything beyond there being sort of search functionality is just absolutely confounding. And "fuck bush?" No, that post was very clearly "fuck your reflexive defense of the admin when you don't even understanding what you are defending them about."
The site's content is creat...
It's really interesting that you've gone to such great lengths to come up a series of defenses from a variety of angles, all of which demonstrate that you still haven't bothered to spend a second to figure out what the robots.txt file is actually doing. Since you clearly don't know what it's blocking or why, not to mention anything about the whitehouse.gov search field, you have no idea what you are defending. It could be nefarious, it could be benign, but you don't know.

[Jubelum]

reflexive defense of the admin

Was there a defense? I missed that part. I stated that it was a non-issue... and in response to a "typcial" snark from another poster. Bueno?

I personally use robots files for exactly what I stated... to prevent listing of pages outside the front page. Will His Smugness please bless us with the almighty answer to this riddle? I mean, since you assume that mantle...

[giant]

I stated that it was a non-issue
Which you don't know, since you don't know what the exclusions are doing or why they are doing it, meaning you can't possibly know that it's a "non-issue," regardless of whether or not it is.

Furthermore, your argument that the white house's search engine (I'm confident you don't even know the software) is sufficient is completely meaningless without providing a technical account of its neutrality, accessibility and comprehensiveness. Accessibility would also have to include something about the fact that it's not integrated with traditional search services, meaning that information would indeed be obscured for the vast majority of researchers. There are a whole range of reasons why this could be desirable, but most of them are to limit accessibility in some way.

Your arguments also ignore the fact that the whitehouse.gov/robots.txt file did at one point exclude most of the Iraq information on the site until some journalists began asking about it. Whether it was intentional or not is unknown. Regardless, the file in question has already excluded important information without having a normal navigational reason for doing so. There have been other incidents of hiding or removing politically embarrassing information, most famously with the images, articles and videos of the "mission accomplished" speech. Thus, we are back to the importance of understanding what the exclusions are actually doing and why.
Will His Smugness please...
Ah, yes, smugness (http://forums.appleinsider.com/showthread.php?p=1147895#post1147895).
you can't find one damn nice thing to say in almost 7 years.
Silly me. Here I thought that this political ghetto of an apple rumors site's discussion board simply served as a little momentary distraction from work. Little did I know that we are each required to post a complete compendium of our political views so as to please Jubelum. Is this some sort of opinion extortion racket thing? Something like where you try to punish transgression by making increasingly inane and unsound arguments?

[jimmac]

Really glad I can't read most of Jubelum's posts anymore!:rolleyes:

[@_@ Artman]

Trying to direct people to the front page. Typical.
:lol:

Psst... there is a FUCKING SEARCH FEATURE on the TOP of the front page so all of you can go directly there and get whatever you want.

Mountain, meet molehill. Like anyone needs another reason here to hate the man.

With the myriad colossal fuckups of this Administration, the best we can toss over the fence is what parts of the Whitehouse web site are not indexed? Really?

I'll just cite one example: Right after last year's election of congress et al there was a photo on the New York Daily news of Bush crying at a medal ceremony for a fallen soldier in Iraq. After that story had run it's course I wanted to search the White House site for more context of the ceremony and all. I tried in vain to find anything concerning it and there was nothing there. I found the photo though somewhere else...

http://i58.photobucket.com/albums/g255/artman46/crying_bush.jpg

The same with the ceremony where Bush gave medals to a group of widows of fallen soldiers where he said, "Now don't sell it on EBay (http://www.npr.org/templates/story/story.php?storyId=10572647)" No surprise that there was anything even concerning this meeting or this quote.

Of course they are going to remove or not even reveal these gaffs on their White House website, so why not anything else? All propaganda, all the time. Don't expect transparency on a government run website.

[Jubelum]

Really glad I can't read most of Jubelum's posts anymore!:rolleyes:

How many times are you going to make this same post? We're all proud of you. Very.

[Jubelum]

Just answer me this- WHY would it not be simply removed, rather than blocked, hidden, etc by the impossible-for-anyone-but-you to understand search engine and robots file?

[Jubelum]

Don't expect transparency on a government run website.

Bingo.

Do all of you that are smelling blood in the water here think that our leaders' websites are not used for propaganda purposes? It's almost as if some of you think that these websites are part of the public record, that their content is not tightly controlled and should, once posted, always be accessible to the people. Well, it's not that way. Websites are there for communicative and propaganda purposes... their owners control their content and access to it. Squawking about a robots or index system misses the point that data that the owners want to disappear can do just that in a few keystrokes. No need for complicated attempts to "hide" what can be easily vaporized.

I don't see Biden's famous Dunkin Donuts or Clean Black videos on his website... does that mean that he qualifies for the same treatment? Does the Mother of All Transparency, Ms. Pelosi, also need to be excoriated for what is NOT on her website? :lol:

Of course not... the content and indexing of a website is meaningless. The content of these sites is created by individuals, and it appears at their pleasure. It is not part of the official record.

Want "Mission Accomplished"? Results 1 - 18 of about 47,800 for "mission accomplished" (http://images.google.com/images?client=safari&rls=en&q=%22mission+accomplished%22&ie=UTF-8&oe=UTF-8&um=1&sa=N&tab=wi)

Want the speech verbatim? CNN has it for you (http://www.cbsnews.com/stories/2003/05/01/iraq/main551946.shtml).

You see, once gaffe is spoken or picture taken, it's all over the world within minutes. The Internets work that way. There is no shortage of places to get anti-anything (http://nocirc.org/) content.

[giant]

impossible-for-anyone-but-you to understand search engine and robots file?
Just because you haven't bothered to actually take a second to see what it's blocking and why doesn't mean it's "impossible," particularly in a situation like this where it's actually really obvious.

An "WHY would it not be simply removed, rather than blocked...?" In this case or in others? There are a whole list of uses for a robots.txt file, some navigational, some to stop content from showing up in search engines while still having it publicly accessible, something potentially desirable for a whole range of possible reasons.

It really is interesting that with so many posts in this thread arguing that it's a "non-issue" you still have no idea what the file is actually blocking. Your argument is completely baseless and pure speculation. That you've provided such an energetic defense without any knowledge of whether or not the exclusions are actually benign demonstrates that you are being nothing more than a run-off-the-mill apologist.

[@_@ Artman]

You see, once gaffe is spoken or picture taken, it's all over the world within minutes. The Internets work that way.

This "conversation" comes to mind...

RUMSFELD (http://www.washingtonpost.com/ac2/wp-dyn/A8575-2004May7?language=printer): They were part of a criminal investigation. And they were in that Central Command -- I say no one in the Pentagon had seen them. And they were part of that investigative process.

It is the photographs that gives one the vivid realization of what actually took place. Words don't do it. The words that there were abuses, that it was cruel, that it was inhumane -- all of which is true -- that it was blatant, you read that and it's one thing. You see the photographs and you get a sense of it and you cannot help but be outraged.

Now, there are -- at any given time in the Department of Defense as I said, there are these 3,000 courts-martial under way, general courts-martial some 1,200, criminal investigations 18,000 a year last year. And the importance of protecting the people charged, protecting their rights, and the importance of seeing that if in fact they're guilty that they don't get off because of command influence. So there's a pattern of not reaching down into those things, bringing them up and looking at all the evidence before it ever arrives. And in this case, it was released to the press.

Now, we announced the problem to the press. We did not release the Taguba report to the press. That was done by someone to release against the law a secret document.

That's how it surprised everyone. It shocked the Congress. It shocked me. It shocked the president. It shocked the country.

But to suggest that they had not taken tough, swift, corrective actions in the Central Command, it seems to me is inconsistent with what took place.

COLLINS: Well, Mr. Secretary, that's not what I said. What I said -- and I have no doubt that the military is committed to swift corrective action. It's the disclosure of the abuse and the promise to take those actions -- that's where I feel the Pentagon fell short.

And I think that rather than calling CBS and asking for a delay in the airing of the pictures, it would have been far better if you, Mr. Secretary, with all respect, had come forward and told the world about these pictures and of your personal determination -- a determination I know you have -- to set matters right and to hold those responsible accountable.

RUMSFELD: Well, Senator Collins, I wish I had done that. I said that in my remarks.

I wish I knew -- and we've got to find a better way to do it. But I wish I knew how you reach down into a criminal investigation when it is not just a criminal investigation, but it turns out to be something that is radioactive, something that has strategic impact in the world. And we don't have those procedures. They've never been designed.

We're functioning in a -- with peacetime restraints, with legal requirements in a war-time situation, in the information age, where people are running around with digital cameras and taking these unbelievable photographs and then passing them off, against the law, to the media, to our surprise, when they had not even arrived in the Pentagon.

Still haven't found a YouTube link to Rumsfeld's testimony. I think it's the only time I've seen him with his guard down and steam coming out of his ears.

There is no shortage of places to get anti-anything (http://nocirc.org/) content.

:wow: OUCH! :wow:

[Jubelum]

OK, Cap'n Smug... tell all of us (who can read the robots file in plain engrish) about the "obvious."
You keep referencing it... go for it, your public is waiting. I'm anxious to see your interpretation.

:lol: I love ya, giant. Truly.

[Jubelum]

in the information age, where people are running around with digital cameras and taking these unbelievable photographs and then passing them off, against the law, to the media, to our surprise, when they had not even arrived in the Pentagon.[/B][/SIZE]


...Which begs the question... how BIG does a coverup have to be, how incredibly thorough, to prevent one enlisted guy with a digital camera and laptop, never mind the hordes of people who spend their entire lives looking for their own Pulitzer. We're talking incredible powers here. Godlike even.

The Internets have been good for our country. Once spoken (or even rumored to have been spoken) the message, regardless of truth, is passed through the blogosphere at light speed. Reproduced thousands of times. Cover up in this day in time takes a monumental effort and no one in the chain of command that will ever speak. These guys must be GOOD.

(BTW- Rumsfeld garnered a fool's portion of laughter for that exchange. But that way Rummy... defending the indefensible so much of the time...)

[giant]

What's next, Jube? Are you going to start smearing white-out on the monitor while yelling "Show me this magic 'delete key' if you're so smart, mr smuggypants?!?!" No, Jube, I'm not responsible for your ignorance of this subject and subsequent baseless defense of something you don't even understand, nor am I required to take what's explicitly stated in the exclusions in plain english and restate it in some way you can understand, particularly since you've had more than enough opportunities to put your defense on hold until you actually knew what was going on.

[Jubelum]

What's next, Jube? ... ... ... ...

Just as I thought... there is no mystery to the plain english in the robots file. It's right there in plain language, yet you continue to claim that I cannot read it. Stunning. You still will not state "what is going on" that you smugly claim I'm somehow unaware of. Show me the money.

Nothing then, or now, to see here. Case closed.

[@_@ Artman]

The Internets have been good for our country. Once spoken (or even rumored to have been spoken) the message, regardless of truth, is passed through the blogosphere at light speed. Reproduced thousands of times. Cover up in this day in time takes a monumental effort and no one in the chain of command that will ever speak. These guys must be GOOD.

NO ONE IS SAFE.

http://farm2.static.flickr.com/1403/1431892021_f2e9492b64.jpg

Stolen iMac turned in (http://www.canada.com/vancouversun/news/story.html?id=6db9bed3-87db-4640-b1fb-a00cffbb20ac&k=91553)

Man whose photo ended up online tells police he bought the computer from a friend
Jonathan Fowlie, Vancouver Sun2007
Published: Wednesday, September 26, 2007

A man who attained instant infamy by unwittingly using a stolen computer to upload a shirtless self-portrait to the Internet has turned the machine over to police in Victoria.
The tattooed man, whose photo has been viewed hundreds of thousands of times around the world and who suddenly found himself famous from Iceland to Brazil, walked into the Victoria police station with the stolen machine at about 4 p.m. Tuesday, said gt. Colin Brown.
The man also contacted Global BC on Tuesday, telling the station he did not know the computer was stolen and that he had bought it from a friend who had bought it from someone else.
At Flickr.com - a popular website where the photo surfaced on Monday - people were viewing the photo at a rate of about 90 hits per minute. By late Tuesday, it had been viewed more than 217,000 times.

:lol:

[giant]

It's right there in plain language, yet you continue to claim that I cannot read it. .
You claim the exclusions are "to direct people to the front page" and used that as the foundation for your claim that it's a "non-issue." The fact is that exclusions are actually doing something completely different, something unbelievably obvious had you ever bothered to actually look at what was being excluded. You're simply dead wrong.

Really, Jube, you made a claim, your claim is wrong, invalidating the predetermined conclusion you pretend to base on the erroneous claim.

[Jubelum]

The fact is that exclusions are actually doing something completely different, something unbelievably obvious had you ever bothered to actually look at what was being excluded.

Is there an echo in here? Or an answer to my request to produce some epiphany, wise one?
One last time, please, end the suspense. It's killing me.

Unless you got that for me, I'm done with discussing this with you.

[giant]

The robots.txt file contains a list of paths that the site owner doesn't want indexed. I wonder how we could find out what exactly is being blocked. Hmmm. What a complete fucking conundrum. If only there was some was to actually find out what each exclusion was blocking. I mean, how in the world could we ever find this information? Maybe we should call the geek squad! We wouldn't want to go break the internets by making typing on the computer intarweb program.

Maybe you are right, Jube. Expecting you to actually bother to know what's being excluded before declaring that they are "trying to direct people to the front page," claiming this is why it's a "non-issue" while you laugh at others is apparently just unrealistic on my part.

And, wow, you still haven't provided a technical explanation to back up your claims that the site's search engine is adequate and comprehensive. Maybe that's because you can't. In case you are wondering, such an explanation would include a description of the software, how it can be configured, what kind of configuration options we can tell this site is using and what options we don't have access to and examples of a variety of search cases specifically designed to demonstrate the neutrality and comprehensiveness of the index and result display.

[Jubelum]

The robots.txt file contains a list of paths that the site owner doesn't want indexed. I wonder how we could find out what exactly is being blocked. Hmmm. What a complete fucking conundrum. If only there was some was to actually find out what each exclusion was blocking. I mean, how in the world could we ever find this information? Maybe we should call the geek squad! We wouldn't want to go break the internets by making typing on the computer intarweb program.

You keep going back to your silly assumption that I cannot read the robots file.
I got it, Chief. Promise. You can even enter those sexy little excluded directories and read away.

actually bother to know what's being excluded

We have already read it. And been over that fact for the fourth time now.

still... nothing. We know how robots and excludes work. We can read what is on the list. We can even view what is in those locations. Now, the train is leaving the station, what exactly is it that you claim to understand that cannot be understood by the aforementioned pieces of information?

And, wow, you still haven't provided a technical explanation to back up your claims that the site's search engine is adequate and comprehensive. Maybe that's because you can't

:lol: I never claimed either of those things. Quite the contrary, actually.

[giant]

We can even view what is in those locations.
OK. And what's your justification for claiming that it's "blocking a bunch of old events and photoessays" "to prevent listing of pages outside the front page" "to direct people to the front page"? (quotes reversed for clarity)
And, wow, you still haven't provided a technical explanation to back up your claims that the site's search engine is adequate and comprehensive. Maybe that's because you can'tlol: I never claimed either of those things.
You actually did exactly that:
Psst... there is a FUCKING SEARCH FEATURE on the TOP of the front page so all of you can go directly there and get whatever you want.

[Jubelum]

|||

We're going in circles here... you will not tell us, with all of your superior skillz, what is going on that you claim I'm not aware of. Go ahead, look smart, wow us all... what do YOU think is happening on the White House website... Still I have heard nothing.

I've been over, twice, concerning content on govt sites being at the pleasure of the poster. Their search engine does not have to be thorough or exhaustive.

Now, if you are determined to have the absolute last word, please, go on and have it, such that this now-pointless train wreck of a thread can take its rightful place on page 2.

[AsLan^]

Come on now people.

I hate the neocons just as much as the next guy, but this list is obviosuly autogenerated.

It seems to cover every sub directory of the white house website. I'm pretty sure it's just to conserve bandwidth and minimize server load.

[@_@ Artman]

Come on now people.

I hate the neocons just as much as the next guy, but this list is obviosuly autogenerated.

It seems to cover every sub directory of the white house website. I'm pretty sure it's just to conserve bandwidth and minimize server load.

:lol: A government website (White House even) that is saving energy and money! Oh that's rich! :lol: ;)

[Fellowship]

There is NOTHING we can do about the website as it is configured today.

However,

We can have EVERYTHING to do with the change of this website in the future!!

http://www.youtube.com/watch?v=za78ZGmVErs

Fellows

RON PAUL 2008

[audiopollution]

RON PAUL 2008

Aaaaaaaaaaaaaargh.

[Fellowship]

Aaaaaaaaaaaaaargh.

Please forgive me... I will try to contain myself now :embarrass ;)

:D You know I love you audiopolution and respect this great forum!

Fellows

[giant]

what do YOU think is happening on the White House website... Still I have heard nothing
I've given you more than enough information. For more than a day you have continued to claim the file is "blocking a bunch of old events and photoessays" "to prevent listing of pages outside the front page" "to direct people to the front page." It's simply not doing this.

Really, jube, it's dead simple to verify from multiple angles that it's not actually doing what you claim.

Upon having your claim challenged you could have recognized that you lacked verification of your claim and reserved judgement until you could confirm the nature of the exclusions. Instead, you continued your reflexive defense and lashing out at other members, demonstrating that you are more concerned with defending the white house against criticism than actually understanding whether either that criticism or your defense are actually valid.

And that, jube, is the crux of the issue: that you are defending against a criticism without even looking into its validity, thereby demonstrating that the facts are inconsequential to whether or not you decide to launch a vigorous defense of the white house's actions.
Their search engine does not have to be thorough or exhaustive.
Psst... there is a FUCKING SEARCH FEATURE on the TOP of the front page so all of you can go directly there and get whatever you want.
These two statements are completely at odds with one another.