We’ve all seen the movies where the robots are coming for us, including classics such as The Terminator and The Matrix. What people may not know is that they already came for us – and got all of our information… In fact, the most widely used website in the world is built upon one of these robots: The GoogleBot.

Indeed, not all robots are here to subjugate humanity and turn us into subservient slaves. They are actually quite helpful, indexing web sites on popular search engines so that visitors may come and indulge in the pages of our websites. Without these robots, most of the information revolution we’ve seen in the past 20 years would not have been possible.

But what if you have some information on your website you’d rather not have the whole world take a look at? Perhaps a baby picture from when you were small that you only share with family friends, internal pages that you may want to keep out of the search results page from a business perspective.  There are many valid reasons for “banning” the bots from certain pages, and there are some good ways to do this.

One answer is a robots.txt file. Essentially this is a text file (which can be written in any text editor) that issues commands to robots to visit only the portions of a website that you allow. The basic syntax is fairly simple, and a good overview is available here. We want to be very careful when employing these files, however, and make absolutely sure that we know what effects our actions will have.  For this reasons, many webmasters are uncomfortable with editing this themselves, as one small mistake could render your site entirely invisible (or entirely visible) to any robot.

Luckily, Google now offers a tool that will automatically generate a robots.txt file for you, saving some time and perhaps avoiding an unintentional disaster.

Using this tool can help you control the pages of your website, and we can make sure our robots keep coming back on our terms, without terminating us.

---

Corey Koberg
Corey is a co-founder and principal consultant at WebShare...you can find out more about Corey here.

See more posts by Corey Koberg

The internet is the largest and most varied medium of information in all of human history, housing literally billions of web pages ranging in topics from the mundane to the esoteric. Many companies struggle with determining how to make their websites visible and heavily trafficked on the Internet, and Google is helping out website owners, search marketing managers and webmasters with tools that include Content Analysis, Sitemaps, and more.

These tools are incredibly useful vehicles to develop and manage sites that a search engine can navigate and use, which any Search Engine Marketer will tell you is fundamentally important for the success and visibility of a site on the web. One aspect of this topic includes the importance of page elements and how to effectively leverage rich content like flash, AJAX, video, and more.

One thing to remember is the importance of using textual alternatives for content that is primarily audio/visual for the benefit of searching technologies. While humans have the ability to comprehend the “message” behind this rich content, search engines cannot.  While we as humans can look at a picture of a car and understand and interpret what we’re looking at conceptually and even specifically, a search engine spider is left with nothing more than an array of pixels.  One way to “tell” a search engine what’s behind this content is to use alt attributes for images and noscript/alternative content for browsers without JavaScript/Flash.

This has twofold importance. For one, this makes a site much more accessible to the visually impaired (who may use programs such as JAWS, a text/speech tool for visualization), and second, it helps search engine spiders index a site. Failure to use such options effectively leaves large portions of the internet essentially blind to your content, resulting in fewer page visits and less overall user engagement.  In order to be more than just another fish in the sea, webmasters should leverage these tools and techniques to free their information, bringing it into the light for all to see.

If you’re not using Google’s Webmaster Central, get an account today and get instant visibility into the pages of your site.

---

Corey Koberg
Corey is a co-founder and principal consultant at WebShare...you can find out more about Corey here.

See more posts by Corey Koberg

The group that controls the distribution of IP addresses has passed a resolution that makes it virtually impossible to get more IPv4 addresses. In a strongly worded resolution, ARIN is taking “any and all measures necessary” to “encourage migration to IPv6 numbering”.

ARIN has been preparing for this for this moment for a long time, as their own root servers that control the DNS system across the internet have been upgraded to IPv6 since 2004. While this is a logical first step for the organization responsible for the internet architecture, end users with less sophistication may want to approach the upgrade with caution as the potential pitfalls are numerous, particularly with regard to stateful firewalls. In most IPv4 implementations there is a stateful firewall where external packets can only traverse the network in response to an originating packet from inside the network. Apple’s new Airport Extreme garnered much geek appreciation with it’s IPv6 implementation, but it has received it’s share of bad press from the mainstream publications over these security concerns. As much as the technologist and tester in me would love to implement IPv6 immediately on Webshare’s routers and servers, the responsible sysadmin will likely take a wait-and-see approach and wait for a few BKMs.

There may be a silver lining from this: the examination over the role of a firewall in modern computing. The problem in the above mentioned case arises because the computers behind the router are falsely assuming the all “bad” packets have already been filtered out by a secure firewall. This works fine in the case of corporate servers, but what about laptops on dialup connections, data cards, or open hotspots? Smartphones on cellular networks? If each device were to employ it’s own packet inspection and security measures, these firewall exploits would be far less devastating. Instead, our current reality is that an unpatched machine exposed with no firewall can be compromised in mere minutes.

---

Corey Koberg
Corey is a co-founder and principal consultant at WebShare...you can find out more about Corey here.

See more posts by Corey Koberg