Robots and your Website – Google’s robots.txt File Generator

We’ve all seen the movies where the robots are coming for us, including classics such as The Terminator and The Matrix. What people may not know is that they already came for us - and got all of our information… In fact, the most widely used website in the world is built upon one of these robots: The GoogleBot.

Indeed, not all robots are here to subjugate humanity and turn us into subservient slaves. They are actually quite helpful, indexing web sites on popular search engines so that visitors may come and indulge in the pages of our websites. Without these robots, most of the information revolution we’ve seen in the past 20 years would not have been possible.

But what if you have some information on your website you’d rather not have the whole world take a look at? Perhaps a baby picture from when you were small that you only share with family friends, internal pages that you may want to keep out of the search results page from a business perspective.  There are many valid reasons for “banning” the bots from certain pages, and there are some good ways to do this.

One answer is a robots.txt file. Essentially this is a text file (which can be written in any text editor) that issues commands to robots to visit only the portions of a website that you allow. The basic syntax is fairly simple, and a good overview is available here. We want to be very careful when employing these files, however, and make absolutely sure that we know what effects our actions will have.  For this reasons, many webmasters are uncomfortable with editing this themselves, as one small mistake could render your site entirely invisible (or entirely visible) to any robot.

Luckily, Google now offers a tool that will automatically generate a robots.txt file for you, saving some time and perhaps avoiding an unintentional disaster.

Using this tool can help you control the pages of your website, and we can make sure our robots keep coming back on our terms, without terminating us.

The internet is the largest and most varied medium of information in all of human history, housing literally billions of web pages ranging in topics from the mundane to the esoteric. Many companies struggle with determining how to make their websites visible and heavily trafficked on the Internet, and Google is helping out website owners, search marketing managers and webmasters with tools that include Content Analysis, Sitemaps, and more.

These tools are incredibly useful vehicles to develop and manage sites that a search engine can navigate and use, which any Search Engine Marketer will tell you is fundamentally important for the success and visibility of a site on the web. One aspect of this topic includes the importance of page elements and how to effectively leverage rich content like flash, AJAX, video, and more.

One thing to remember is the importance of using textual alternatives for content that is primarily audio/visual for the benefit of searching technologies. While humans have the ability to comprehend the “message” behind this rich content, search engines cannot.  While we as humans can look at a picture of a car and understand and interpret what we’re looking at conceptually and even specifically, a search engine spider is left with nothing more than an array of pixels.  One way to “tell” a search engine what’s behind this content is to use alt attributes for images and noscript/alternative content for browsers without JavaScript/Flash.

This has twofold importance. For one, this makes a site much more accessible to the visually impaired (who may use programs such as JAWS, a text/speech tool for visualization), and second, it helps search engine spiders index a site. Failure to use such options effectively leaves large portions of the internet essentially blind to your content, resulting in fewer page visits and less overall user engagement.  In order to be more than just another fish in the sea, webmasters should leverage these tools and techniques to free their information, bringing it into the light for all to see.

If you’re not using Google’s Webmaster Central, get an account today and get instant visibility into the pages of your site.

Google Analytics has just put in place a new feature that will be formally announced very soon – comparison graphing for your reports.

For those of you that have grown accustomed to the tireless examination of your key Analytics reports day after day, this new feature is a welcome addition to Google Analytics from a usability standpoint. Whereas we have been seeing key metrics compared to site averages for quite some time now (displayed in red or green, in parentheses next to key numbers), we now have the ability to graph two metrics or two date ranges.

Ever wanted to quickly see things like:

Has all of our work paid off and did we get more traffic this holiday season than last?
Just head to your favorite traffic report and drop down your date selection box and select the two date ranges you’d like to compare. Black Friday 2006 to 2007 or any period you’d care to see.

Are more visitors looking at our December Discounts page after we highlighted it on our major landing pages last week?
Find the page in question in your content reports, set your date ranges to compare and enjoy the view!

Did that string of newspaper ads in San Francisco and L.A. get us more traffic from the state of California? More e-commerce transactions? Better conversion rates?
Just navigate to the Map Overlay report for California, set your comparison date ranges and have a look, city by city. You can use the tabs at the top to compare traffic, conversion, and e-commerce numbers over your date range and any geographic location.

Are visitors from my banner ads staying on my site longer after I made my landing pages better a month ago? Are they converting better as a result?
Have a look at the time on site column of the campaign report where you track your banner ad visitors. Conversion and e-commerce numbers are just a click away!

Does my G1 conversion rate trend in the same way as my G2 goal over time? What does it look like for Chicago visitors that came from a geo-targeted Adwords campaign?
Head over to your All Traffic Sources report, drop down the graph options, select “Compare Two Metrics” and pick the goal conversion rates you’d like to compare. Get even more detailed by finding the cross segment of a specific Adwords (or any other) campaign while in the Map Overlay report. Now drop down the graph options and pick your two goal conversion rates!

The answers to all of these questions can be quickly graphed right in the Google Analytics console now, and there are myriad ways to use this new feature and quickly see a snapshot of date range comparisons.

Webshare is a Google Analytics Authorized consultancy and can help you set up, configure, and analyze this invaluable data. We offer customized analytics training as well as Google Analytics consulting for any project.

Everyone who has a website and a search marketing plan knows just how indispensable solid analytics data has become. While solid analytics packages were only available at a premium not long ago, the introduction of Google Analytics has given the power of information to any website owner for a very competitive price: free.

Google Analytics has continued to mature since its release, undergoing a large scale change to the enhanced version 2 earlier this year, and the Google Analytics team didn’t stop there. Recently a number of new features have been announced, and we caught up with Brett Crosby, Senior Manager at Google Analytics to talk about the latest and greatest:

Webshare is a Google Analytics Authorized consultancy, and can help you get the most out of your analytics. We offer a wide range of services, from Google Analytics training to Google Analytics consulting, and can work with projects of virtually any size.

Microsoft’s Project Gatineau is now in beta and being tested by the public. On October 29th they began inviting certain users to sign up for the beta, although you do need to be an adCenter user to be granted an invitation. At this time it is not known when Microsoft will release it for all customers to use.

Gatineau is Microsoft’s “answer” to Google’s Analytics, and they state that it won’t be exactly the same as Analytics (and of course they say it will be better). Ian Thomas of Microsoft’s Digital Advertising Solutions group has a blog specifically discussing Gatineau, and in it he states, “We think there’s room in the market for another service of this nature; plus, we have some stuff up our sleeves that we hope you’ll like and which will differentiate us from Google’s and others’ offerings.” He went on to note that “we have more resources than DeepMetrix did (development team has more than quadrupled since the acquisition, for example), so hopefully we won’t disappoint you.” Microsoft’s attainment of DeepMetrix tipped off the internet community that the software giant was looking to get into web analytics. Gatineau is the name of the Canadian city where DeepMetrix was based for a number of years.

Ian goes on in his more recent blogs to give us just a taste of something we may see from Gatineau, and I’ve tried to capture some differences between this and Google Analytics. Perhaps the most exciting aspect of Gatineau is that it has the ability to track some demographic data (such as usage statistics between men and women) for users that have a Microsoft Live ID and are logged in as they browse the web. Another feature is the ability to map the document hierarchy from your content management system into the tool and see this in Gatineau’s reports. The tool also includes similar things that Google Analytics already does but in a different way, including funnel reports, outbound link tracking, inbound referrals, ROI reports, goal analysis, and client system reports.

At this time, Gatineau is not getting much attention, but we’ll be watching to see how Microsoft’s entry into the Analytics game unfolds. It is also likely that the features introduced with Gatineau could be incorporated into the market’s current offering of web analytics tools, but we’ll let time tell. We at WebShare are evaluating Gatineau and its capabilities to understand where it shines and where it lacks, and will continue to communicate on this new service from Microsoft and let you know how it might (or might not) benefit you.

WebShare, LLC is a full service Internet Marketing firm specializing in Search and Conversion Marketing. We offer a variety of services to help you make the most of your search marketing efforts, and get an edge on your competitors.

It appears that the search market is alive and well. ComScore released the findings from their report showing the worldwide search activity in August of 2007. Google is still the worldwide leader with over 60% of the market share. It is interesting to note that Youtube searches (5 billion) account for a little over 13% of all Google searches. Yahoo comes in at a distant second while Chinese search engine, Baidu comes in third place. Microsoft continues to hold a minor presence in the search market in fourth place.

We should note that this data does not include searches conducted on public computers or mobile devices. This would indicate that there are actually even more people performing searches on the internet. As mobile device and wireless technologies continue to become more prevalent, the number of searches performed will continue to grow. A couple of interesting notes include the fact that the Asia-Pacific market has the highest search volume of all regions, while the Latin America region having the smallest internet population has the most searches per searcher. If these markets continue to grow at their current rates, they could open some great international search marketing opportunities.

To put this all in perspective, let’s take a look at the average number of searches in a little shorter time period. Breaking down the number of worldwide searches conducted in August of 2007, you can see that the average number of searches performed worldwide on a daily basis is nearly 2 billion! That’s over 22,000 searches conducted per second, which presents a whole lot of marketing opportunity!

With an audience that size the question isn’t, “Why would I start a search marketing campaign?” It’s “Why wouldn’t I start a search marketing campaign?” WebShare is a Google Website Optimizer and Analytics Authorized Consultancy as well as a Google Adwords Qualified Company. We offer a variety of services including search engine optimization and search and conversion marketing to help you make the most of your search marketing efforts, and get an edge on your competitors.

In a recent post from Google’s Webmaster Central Blog, Greg Grothaus and Shashi Thakur have shed some light on a few questions that are likely close to many web developer’s hearts.

1. The relationships between search results served to varying geographies, the geographic location of a website’s server, IP addresses and domain extensions are playing a part in which sites top the lists of search results pages.

Domain extensions can tell you something about a site before the first glimpse of a page actually loads. If you’re running a website in Australia and targeting Australian customers, then the .com.au extension will likely help you in Australian search results pages. “Because we attempt to serve geographically relevant content, we factor domains that have a regional significance,” states Shashi Thakur in the post. Additionally, IP addresses can be used to determine geographic location, and Google is not ignoring this information. Be smart about who your target audience is and where they’re located - you can use this information to your advantage.

2. What effects can be expected from cross linking a number of websites that you control, and what guidelines are there for linking strategies if you own and maintain multiple websites?

Shashi weighs in on this issue in a manner that is very consistent with any other linking guidelines that we’ve seen from Google – no surprises here. As with any other linking strategy, the general advice is that links between sites that “provide value” to a user are good, and links that are there simply for the sake of linking are bad. The interesting piece here is the explicit mention of links between topically or thematically related sites: “If the sites are related in business …then it could make sense — the links are organic and useful.”

This advice can be applied to any links, whether between two sites that you own or otherwise. The importance of links to your rankings cannot be understated, and making sure that your limited time and resources are applied to linkbuilding strategies in the most efficient manner is essential.

3. What suggestions are there to helping search engines “comprehend” websites that use DHTML, AJAX, Flash and other Web 2.0 technologies?

Greg Grothaus in his post reaffirms our belief that search engines are not human, and not yet capable of human comprehension. As human visitors to a website, we can look at a photo of a car, and in a fraction of a second draw the conclusion that we are looking at a car. We can watch animations and video and make sense of what we’re seeing. Search engine spiders, on the other hand, can “see” only a mess of pixels and compiled code that does not assist in comprehending what the images or animations are about. For this reason, when designing websites and pages, one must be conscious of the balance between what a search engine sees and a human user will see.

Google officially recommends using Flash “only where it is needed,” and warns of the dangerous ground you enter when your website displays different content to a search engine spider than it would to a human visitor. Without malicious intent, it would make sense to find a way to tell a visitor that does not have Flash installed (or a search engine spider) what they’re missing in a non-Flash manner. Techniques such as cloaking, doorway pages, scripted redirects, and the use of CSS to hide text CAN be used to accomplish this, but as these are the same techniques that are commonly abused in hopes of manipulating a search engine, using these on your site is begging to be penalized.

In a separate post, Mark Berghausen of Google’s Search Quality Team writes “the only hard and fast rule is to show Googlebot the exact same thing as your users.” We’d heed that warning.

The group that controls the distribution of IP addresses has passed a resolution that makes it virtually impossible to get more IPv4 addresses. In a strongly worded resolution, ARIN is taking “any and all measures necessary” to “encourage migration to IPv6 numbering”.

ARIN has been preparing for this for this moment for a long time, as their own root servers that control the DNS system across the internet have been upgraded to IPv6 since 2004. While this is a logical first step for the organization responsible for the internet architecture, end users with less sophistication may want to approach the upgrade with caution as the potential pitfalls are numerous, particularly with regard to stateful firewalls. In most IPv4 implementations there is a stateful firewall where external packets can only traverse the network in response to an originating packet from inside the network. Apple’s new Airport Extreme garnered much geek appreciation with it’s IPv6 implementation, but it has received it’s share of bad press from the mainstream publications over these security concerns. As much as the technologist and tester in me would love to implement IPv6 immediately on Webshare’s routers and servers, the responsible sysadmin will likely take a wait-and-see approach and wait for a few BKMs.

There may be a silver lining from this: the examination over the role of a firewall in modern computing. The problem in the above mentioned case arises because the computers behind the router are falsely assuming the all “bad” packets have already been filtered out by a secure firewall. This works fine in the case of corporate servers, but what about laptops on dialup connections, data cards, or open hotspots? Smartphones on cellular networks? If each device were to employ it’s own packet inspection and security measures, these firewall exploits would be far less devastating. Instead, our current reality is that an unpatched machine exposed with no firewall can be compromised in mere minutes.