Retail: Shaken Not Stirred by Kevin Ertell

Wanna be better with metrics? Watch more poker and less baseball.

Both baseball and poker have been televising their World Series championships, and announcers for both frequently describe strategies and tactics based on the statistics of the games. Poker announcers base their commentary and discussion on the probabilities associated with a small number of key metrics, while baseball announcers barrage us with numbers that sound meaningful but that are often pure nonsense.

Similarly, today's web analytics give us the capability to track and report data on just about anything, but just because we can generate a number doesn't mean that number is meaningful to our business. In fact, reading meaning into meaningless numbers can cause us to make very bad decisions.

Don't get me wrong, I am a huge believer in making data-based decisions, in baseball, poker, and on our websites. But making good decisions is heavily dependent on using the right data and seeing the data in the right light. I sometimes worry that constant exposure to sports announcers' misreading and misappropriation of numbers is actually contributing to a misreading and misunderstanding of numbers in our business settings. 

Let's consider a couple of examples of misreading and misappropriating numbers that have occurred in baseball over the last couple of weeks:

1. Selection bias
   This one is incredibly common in the world of sports and nearly as common in business. Recently, headlines here in Detroit focused on the Tigers "choking" and blowing a seven-game lead with only 16 games to go. In a recent email exchange on this topic, my friend Chris Eagle pointed out the problems with the sports announcers' hyperbole:
   

   "They’re picking the high-water mark for the Tigers in order to make their statement look good.  If you pick any other random time frame (say end-of-August, which I selected simply because it’s a logical break point), the Tigers were up 3.5 games.  But it doesn’t look like much of a choke if you say the Tigers lost a 3.5 game lead with a month and change to go."
   

   Unfortunately, this type of analysis error occurs far too often in business. We might find that our weekend promotions are driving huge sales over the last six months, which sounds really impressive until we notice that non-sale days have dropped significantly as we've just shifted our business to days when we are running promotions (which may ultimately mean we've reduced our margins overall by selling more discounted product and less full-price merchandise).
   
   In a different way, Dennis Mortensen addressed the topic in his excellent blog post "The Recency Bias in Web Analytics," where he points out the tendency to give undue weight to more recent numbers. He included a strong example about the problems of dashboards that lack context. Dashboards with gauges look really cool but are potentially dangerous as they are only showing metrics from a very short period of time. Which leads me to...
   
2. Inconsistency of averages over short terms
   Baseball announcers and reporters can't get enough of this one. Consider this article on the Phillies' Ryan Howard after Game 3 of the World Series that includes, "Ryan Howard's home run trot has been replaced by a trudge back to the dugout.The Phillies' big bopper has gone down swinging more than he's gone deep...He's still 13 for 44 overall in the postseason (.295) but only 2 for 13 (.154) in the World Series."
   
   Actually, during the length of the season, he had three times as many strike outs as home runs, so his trudges back to the dugout seem pretty normal. And the problem with the World Series batting average stat is the low sample size. A sample of thirteen at bats is simply too small to match against his season long average of .279. Do different pitchers or the pressures of the situation have an effect? Maybe, but there's nothing in the data to support such a conclusion. Segmenting by pitcher or "postseason" suffers from the same small sample size problems, where the margin of error expands significantly. Furthermore, and this is really key, knowing an average without knowing the variability of the original data set is incomplete and often misleading.
   
   This problems with variability and sample sizes arise frequently in retail analysis when we either run a test with too small a sample size and assume we can project it to the rest of the business, or we run a properly sized test but assume we'll automatically see those same results in the first day of a full application of the promotion. Essentially, the latter point is what is happening with Ryan Howard in the postseason. We often hear the former as well when a player is all of the sudden crowned a star when he outperforms his season averages over a few games in the postseason.
   
   In retail, we frequently see this type of issue when we're comparing something like average order value of two different promotions or two variations in an A/B test. Say we've run an A/B test of two promotions. Over 3,100 iterations of test A, we have an average order size of $31.68. And over 3,000 iterations of Test B, we have an average order size of $32.15. So, test B is the clear winner, right? Wrong. It turns our there is a lot more variability in test B, which has a standard deviation of 11.37 compared with test A's standard deviation of 7.29. As a result the margin of error on the comparison expands to +/- 48 cents, which means both averages are within the margin of error and we can say with 95% confidence that there really is no difference between the tests. Therefore, it would be a mistake to project an increase in transaction size if we went with test B.
   
   Check out that example using this simple calculator created by my fine colleagues at ForeSee Results and play around with your own scenarios. Download Test Difference Between Two Averages.xls

Poker announcers don't seem to fall into all these statistical traps. Instead, they focus on a few key metrics like the number of outs and the size of the pot to discuss strategies for each player based largely on the probability of success in light of the risks and rewards of a particular tactic. Sure, there are intangibles like "poker tells" that occur, but even those are considered in light of the statistical probabilities of a particular situation.

Retail is certainly more complicated than poker, and the number of potential variables to deal with is immense. However, we can be much more prepared to deal with the complexities of our situations if we take a little more time to view our metrics in the right light. Our data-driven decisions can be far more accurate if we ensure we're looking at the full data set, not a carefully selected subset, and we take the extra few minutes to understand the effects of variability on averages we report. A little extra critical thinking can go a long way.

What do you think? Are there better ways to analyze key metrics at your company? Do you consider variability in your analyses? Do you find the file to test two averages useful?

Related posts:

How retail sales forecasts are like baby due dates

Are web analytics like 24-hour news networks

True conversion - the on-base percentage of web analytics

How the US Open was like a retail promotion analysis

The Right Metrics: Why keeping it simple may not work for measuring e-retail performance (Internet Retailer article)

Inspiration (a life changing day)

As anyone familiar with my endless supply of analogies and metaphors knows, I'm capable of gaining inspiration from almost anything. But last Wednesday I was lucky enough to attend TEDxDetroit, where inspiration was the order of the day, and the event did not disappoint. Because the TED theme is "ideas worth spreading," I'm going to use this space to highlight my inspirations and learnings from the day. I hope you'll find my thoughts worthwhile.

For those of you who are unfamiliar, "TED is an annual event where the top minds in the world share, connect and inspire. TED stands for Technology, Entertainment, Design — three subjects that, collectively, shape our future. The event draws CEOs, scientists, creatives, philanthropists and extraordinary speakers." This year, the TED organizers decided to open up the concept to local organizers through the TEDx concept, and several Detroiters, led by Charlie Wollberg, produced the inaugural TEDxDetroit.

While there was much inspiration to be gained from each of the 15 speakers of the day, three had a particularly strong effect on me.

Chazz Miller is the founder and muralist at Public Art Workz in Detroit, an organization that creates "bold, innovative community redevelopment projects that use the arts, culture, creativity and innovation as a catalyst for reinventing and revitalizing the communities of Old Redford and Northwest Detroit, Michigan, into a multi-discipline, arts, education, entertainment and cultural community." Chazz described with passion and affection some of the community art he has developed in the form of wall murals, "mood swings" and "poet trees."

What struck me most during Chazz's talk was his specific involvement of the community in his art. He actively recruited members of the local community to help complete his park wall murals, and he found that the community then maintained those parks better than ever. As he said, if you truly and honestly involve more people in the effort, they'll care more about what's been accomplished.

I realized that there's strong business value in Chazz's philosophy. While a top-down command culture may be able to achieve results, a truly participatory culture breeds ownership. And with ownership comes the type of pride and attention to details that enhances the nuances that can make the difference between good and great. How can we as leaders bring an overall vision to our businesses but allow our teams the freedom to actively participate and bring their strengths and ideas to the details we cannot possibly see?

_______________________________

Lee Thomas, anchor and entertainment reporter on FOX Detroit and author of Turning White, discussed his story of living with the skin disorder vitiligo -- an affliction that is literally turning him white. He recalled a turning point in his life when the disease affected his face to the point that a young girl screamed in fright at the sight of him. The very thought of scaring young children confined him to his home for weeks. It wasn't until another young girl saw him the supermarket and, instead of screaming, asked him if he had a boo-boo and if it hurt. He realized the dichotomy of the two reactions were driven by perception. The girl who thought he had a boo-boo offered compassion, while the girl who didn't know what it was reacted with fear. A subsequent conversation with a 15-year-old boy also afflicted with vitiligo, who prodded him to go public because public knowledge and understanding could help all those afflicted with the disorder, finally prompted Thomas to appear live on TV without make-up covering the affects of the disease.

And his life took on new meaning. He talked of never knowing where your success will come from. You could, as he said, "find your weakness may be your greatest strength." Although, I personally don't see vitiligo as a weakness for him. It's simply skin color. The fact that he perceived it as a weakness sapped his confidence and that lack of confidence was truly the weakness he overcame. And his resulting public awareness campaign is helping turn fear to compassion.

From Lee Thomas I was inspired to find ways to turn my personal shortcomings into opportunities to help myself and to help others.

_______________________________

As powerful and inspiring as Chazz and Lee were, it may be the poetry of Blair that had the most immediate impact on me. Just before Blair's time slot, I received a call from the editor of a trade publication that was running an opinion piece I'd written. He informed me, minutes before their deadline, that they decided to remove a part of my article that they felt was too controversial for them to print. I was still stewing with outrage, feeling censored and violated, when I returned to my seat as Blair took the stage to perform his poem, "Detroit (While I Was Away)."

Before starting the poem, he briefly described the genesis of the piece. He was traveling in Texas and missed all of Detroit -- the good, the bad and the ugly. He then launched into his piece, and he almost instantly quelled my anger and stirred my passions in an entirely different direction. I was blown away by his ability to find beauty in the blight and the good in the bad and ugly.

Not only did he cause me to see Detroit in an entirely different light, but he made me realize the importance of finding the positives in our lives, not dwelling on the disappointments. I instantly became pleased with the two-thirds of my article that did make it to print in the trade magazine rather than angered by the one-third that didn't. And every day since then I've strived to find the positives in all situations. And you know what, finding positives is a lot more fun than stewing on negatives.

_______________________________

While no single speaker at TEDxDetroit instantly changed the whole of my life, I was able to gather a piece of inspiration from just about all of them. For me, that's living. I'm always in search of personal improvement wherever I can find it, and I'm grateful to the organizers and speakers at TEDxDetroit for bringing so much together in one convenient spot. While I certainly can't do these extraordinary people justice with my recounting of their tales, I hope I've done my small part to help spread the positivity and perseverance they bring to the world on a daily basis.

I'd love to hear your thoughts. Where do you find inspiration?

Conversion tip: Don't let bad error messages cost you sales

Writing error messages is not sexy. In fact, it's incredibly tedious and boring. But don't confuse tedious and boring with unimportant. Often, the quality of an error message can be the difference between a sale and an abandonment. And a poorly written error message is a needless and shameful way to lose a sale. The good news is that improving error messages has a high ROI as the cost of the investment is very low.

It's important to remember that our sites are really self service software applications, and they're very likely not as intuitive as we think they are. Referencing back to one of my previous posts, "Is elitism the source of poor usability," we have to remember that our customers probably aren't as tech savvy as we are, and they are definitely not anywhere near as familiar with our sites as we are. So, it's important that we're very clear in our messaging when something goes wrong.

So what does it take to write a quality error message?

1. Be specific
   It's so important that we tell our customers exactly what went wrong. Our developers have to write code for every possible instance, but all too often we resort to generic and vague language in our error messages. Here are a couple of examples:
   
   
   As a customer, I'm not sure what I'm not sure what happened or what I should do about it. I might try once again, but if I got this message a second time I would be gone.
   
   
   This either/or scenario is really an example of a lazy error message. Which is it? Is the address improperly formatted or does it contain invalid characters? We need to tell customers specifically what is wrong and tell them how to fix it.
   
   Here's a much better example:
   
   
   
2. Use clear language
   It's very important to avoid anything that even remotely resembles tech jargon. Try instead to use short words that are part of everyday language.
   
   First, a bad example:
   
   Huh? Customers understand "password" but "authentication credentials" are certainly unclear and sound kind of scary, frankly.
   
   This one is much better:
   
   
   This is both specific and written in clear and simple language.
   
   
3. Strong visibility
   Error messages need to be extremely prominent. Use color and other symbols, such as exclamation points, to help the error message stand out. It's also helpful to separate the error messages from the rest of the page with white space. Include the message prominently at the top of the page and also at the specific field, if it's a form error.
   
   Here's a good top of page error:
   
   I would like to see more white space around the error message, but otherwise this is really good.
    
   
   And I really like this way to highlight a particular field where the error has occurred. It may not be pretty, but then it probably shouldn't be. It should stand out, and this does. Even better, we get a very specific message telling us exactly what's wrong with the field.
   
   
   
4. Be polite
   Whenever an error occurs during our customer's experience with our site, we're in danger of losing her if we don't handle it well. So, let's be as courteous as possible. The cost of courtesy is zero, and it allows us to come across as friendly as possible.
   
   Here's one that is both specific and polite:
   
   
   Here's one that goes the extra mile to suggest calling Customer Service if there is still a problem. This is a very nice touch that will go a long way towards saving the sale.
   
   
   
5. Provide examples for how the information should be entered correctly
   It's very important they we're not only specific in defining the problem that occurred but also specific in explaining how to correct the problem. If the customer has entered his email incorrectly, we cannot assume that he knows what he did wrong or how to enter it correctly.
   
   Here's an error message that explains the format pretty well:
   However, the customer may not understand what "domain" means. It may be be better to also use a real example with a well-known domain like "name@aol.com." Even better, incorporate the information the customer entered, if possible.
   
   For example, the error might say something like:
   
   You entered "kevin" for your email address, which is not a complete address. Please enter an "@" symbol followed by an email provider after your email name. For example, "kevin@yahoo.com."

Even better, be proactive. Stop the error before it occurs.

I really love how Restaurant.com handles their form fields. Upon entry to a form field, a dialogue box dynamically appears next to the field with some helpful information. The movement that occurs upon entry really draws your attention to the helpful information, which I find considerably more effective than help text persistently present under or next to a field. It's far easier to ignore static text than something that appears when you enter the field.

Additionally, the folks at Restaurant.com have included some great help text that provides important information. In this example, they're letting us know the address must match the billing address on our credit card. Excellent!

And here, we get some specific information about the value of our password and the basic requirements for the password. And we get some nice politeness to close it out.

Save those sales. Give error messaging your full attention.

Error messages should get just as much attention as any other site functionality in the requirements processes for our sites. We should give error messaging as much attention as we give to marketing copy. It may not be sexy, but it's critically important if we want to avoid needlessly losing sales.

What do you think? How much time to you put into error messaging? Do you have examples of particularly good error messaging? Would you add anything to the list of quality error message attributes?

Sitting in the "Marketing Hot Seat"

My good buddy Adam Cohen, a Rosetta partner who heads up their Search, Online and Social Media businesses, issued a challenge called "The Marketing Hot 

You’re the CMO.  You have a marketing budget of $1M.  Your company is a consumer product company, relatively unknown / early stage.  Customers who know the product like it. CEO wants ROI within 12 months.  What do you do?

I thought this would be a fun exercise to take on, particularly because the scenario placed me in the seat of a manufacturer, publisher or product company. Would my retail oriented perspective provide a different line of thinking than would typically come from a manufacturer, and would that perspective be worthwhile? I'd certainly love to know your thoughts.

My take is actually the first one Adam posted on his blog, A Thousand Cuts. Check things out over there over the next few weeks to see perspectives from the other 12 bloggers.

Here's my answer to Adam's challenge:

OK. Setting aside all the caveats about the fact that I don’t know what the product is, what it costs to make and what our margins are, here’s generically how I would approach the situation:

Strategy

1. Thoroughly understand the customers who like our product
   The customers who know our product like it. We need to find out why, in their words, and determine what personality traits, hobbies, demographics, etc. in those customers are relevant to their liking our products so that we can speak to others like them.
2. Get our online destinations right
   With a relatively small marketing budget, we’re going to need to maximize our online strategy. (Actually, we should do that even if have a large marketing budget.) We need to make sure our website and our retailer websites are highly usable and highly effective in merchandising our product and providing the ability for customers to easily spread the word about us.
3. Drive traffic with whatever budget is left
   Only when we have ensured that we have solid destinations for our traffic will we start to actively search for traffic.

Tactics

1. Learn as much as we can about the customers who most love the product.
   Why do they like it? What are there personality types; let’s use the Myers-Briggs personality test and really get a thorough understanding of these folks. How do they describe our product? Let’s pay attention to the words they use as we’re going to reuse those words in our copy.
2. Hire ForeSee Results to measure our site’s effectiveness from our customers’ perspectives.
   I realize this may seem self-serving since it’s my company, but I was a client for seven years before joining the company three months ago, and I’ve see how well it works.  So, I want it in this role. So there! We’ll use measurements, analysis, Session Replay and usability audits to ensure we’re providing the best experience we can.
3. Hire Bryan Eisenberg to develop archetypes and to implement Persuasion Architecture on our site.
   We need to speak to customers in language that resonates, and Bryan understands how to do that. We’ll also use his language for product descriptions and other content we give to retailers for their sites.
4. Create a high quality product video.
   We’ll use this video on our own site and we’ll give it to retailers for their sites. We’ll focus on the key aspects customers love and use copy that includes words that resonate with those customers. We’ll also show real customer testimonials.
5. Launch customer reviews and customer forums on our site
   We need to make sure our customers can openly provide their thoughts about our product, even when they’re negative.
6. Launch several blogs on our site
   Since we only have one product, we need to provide some fresh and compelling content on our site to give people a reason to come back. The content doesn’t need to be about the product all the time. It can be able anything, as long as it’s compelling. I’ll focus on general marketing, our CEO can blog about leadership, and we’ll find some people to blog about topics our customers are interested in. All of this blog content will also be great for SEO.
7. Launch a marketing campaign to retailers informing them about key customers and teaching them how to sell the product
   Our initial marketing efforts will essentially be internal. Let’s get the sellers pumped up and doing their jobs well before we send customers their way.
8. Develop a widget for retailers that gives customers the ability to easily share information about the product
   We need to give our customers ways to share information about our product on their own in a way that is easy and positive. Let’s create a fun widget that people want to share on Facebook, Twitter, email, etc.
9. Get our SEO right, buy search terms, send emails, run re-marketing campaigns, etc.
   I don’t want to minimize the value of these techniques, but we really need to make sure our destinations are right before we add lots of traffic.
   
   So there you have it. My main point here is to focus on the customers first, the destination second and the traffic driving last.

What do you think? Does my strategy make sense? How would you have addressed the challenge? Do your manufacturer/publisher/product partners address your needs?

Conversion tip: Don't block the product with window signs

My friend Bryan Eisenberg is always telling retailers, "You don't have traffic problems. You have conversion problems." When 95+% of the people who come to a site don't purchase, it's a hard point to deny. Could giving customers quicker and better access to the product be one way to start to solve the conversion problem? My experience says Yes.

A quick story

Imagine walking into a store and smacking into a giant promotional banner that stretches from the ceiling to chest level. Below it and to its right hang a series of smaller promotional banners. A few feet behind the banners, you see a series of doors with signs above them that appear to represent different product categories. You push through the banners and open the door under the "Dresses" sign. There you step into a room where a flashy video projected on a large wall highlights stylish dresses and beautiful models and also runs copy about the same promotion you saw on the banner at the front of the store. The wall to the right features several smaller signs for various promotions. The wall to the left is littered with 20 or 30 doors, each with a sign above it for what appears to be a type of dress.

Nowhere in the room are there any dresses.

You pass through the door labeled "Casual dresses" and finally see actual merchandise.

Does that story seem ludicrous? Then why is that basically the experience on so many retail websites?

In brick and mortar retail, we use promotional signs in our windows to draw people into the store, where we expertly display lots and lots of product to customers the moment they walk in the door. We certainly reinforce our promotional messages with signage throughout the store, but we never block the product with the signs. On our sites, our promotions seem to be more important than our products. What message are we sending to our customers about the value of our products when promotions get more prominence than the merchandise?

When customers arrive at our home page, they've already effectively entered the store. So, why are our "window signs" blocking the product?

Apparel and department store sites seem to have almost uniformly adopted the experience described above, but most other retail sites that I've seen don't stray too far from "the window sign" experience. Consumer electronics and computer sites often feature a few specific deals or featured products, but otherwise they generally follow a similar approach. In fact, about half of the Internet Retailer Top 25 sites on my recent viewing didn't show any products on their home pages, and the remainder only displayed a very few select products.

Are we missing conversion opportunities by taking too many pages to get to the products?

Certain retail categories, like apparel, books, jewelry and flowers/gifts to name a few, seem to have large customer contingencies who are prone to browsing to see what's new. Physical stores in those categories absolutely cater to the desire of customers to check out the latest stuff, but the web sites seem to assume customers are only interested in promotions. Or are the promotions simply the result of our own self-interest? What percentage of customers click on the promo spots versus hitting the search box or clicking into a department or sub-department? If it's a fairly small percentage, perhaps a different approach might pay off.

A case study

When we launched the new Borders.com last year, we knew that about half our customers came to the store looking for something new to read without a specific book in mind. As a result, we created the Magic Shelf, a virtual and interactive book shelf that housed up to 120 books in an easily browsed application.  
And we placed the Magic Shelf in the most prominent position on our home page -- front and center. The decision to offer such valuable real estate to this new feature was hardly unanimous, but those of us who supported it won at least enough support to give it a try.

The result? Not only did customers say they loved it, those who interacted with the Magic Shelf converted at a rate 62% higher than those who didn't. As we dug deeper, we discovered that the reason they converted more was that they viewed about 41% more products than those who didn't interact with the Magic Shelf. (If you're interested in more detail, you can download the case study we did with Allurent, the vendor we used to develop the Magic Shelf). 

How might the shopping experience change on an apparel site if there was prominently placed virtual rack of some sort that allowed customers to easily browse, on one page, a wide selection of the latest styles? How about virtual jewelry cases or flower bins?

Ann Taylor trendsetting

The new Ann Taylor site design has made some strong strides towards a nice product browsing experience. While they still seem to feature window signs on the home page, their landing pages provide a very nice browsing experience where customers can easily peruse lots of merchandise. The product images are very clean and easy to see, and the page layout lends itself to the ability to occasionally replace one of the product spots with a visible but unobtrusive promo spot. Bravo to Matthew Seigel and the team at AnnTaylor.com!

I'm not sure it's necessary to replicate a physical fixture to achieve the benefits of great product browsing. To me, the key is giving customers easy access to our merchandise and letting them very easily view lots of different items. That basic concept is something we discovered long ago in the physical retail world. How did we lose sight of it online?

What do you think? What is the thinking at your company? What sites have you seen that do a good job giving access to the product? Or, are the current methods working for you?