This webcast hosted by MJ De Pallma has Bing’s Advertisers Science Team members Lars Hirch (Director of Advertiser Science at Bing Ads) and Frances Donegan-Ryan (SEM Pro Digital Marketing Manager) discussing about the value of Brand Bidding. This informative webinar shares useful insights on how to go about experimenting and building your brand bidding strategy to impact your SEM and SEO results.
You can view the webinar here.
The transcript of the webinar is as follows:
MJ DePalma: Welcome everyone to the Bing Ads Webinar on the Science of Brand Bidding. My name is MJ DePalma and I will be your host today. This is the first webinar with our Advertisers Science team and we’re excited to share their findings and methods, so you can achieve more with your advertising dollars and help you earn that bonus! Before we begin, let’s review some housekeeping items to enhance your experience with us today. Our webinar console that you see is customizable on your site, where you can click the content, what you see on your screen and move them around or size them for your screen size. You can expand your slide area by clicking on the ‘Maximize’ icon on the top right of your slide area or by dragging the bottom right corner of that slide area. If you have any technical difficulties, please click on that ‘Help’ widget. It has the ‘?’ icon and covers common technical issues.
We recommend, if you can, to leave all the widgets as they are, so that you have the ability to see all the great resources and Q & A at the same time. For Q & A, please feel free to input your questions as you have them, but for the most part we will be getting to that list of questions at the end of the webinar and we will try to answer as many as possible. Any questions, we don’t have time to answer today. We will post the blog. Actually Frances here, who I will be introducing in a moment, will post the blog with all the frequently asked questions and any unanswered questions and you can find that at: BingAds.com/blog. You will also see a ‘Resource’ widget in the upper right corner and we’ll refer to that during the webinar, so go ahead right now and there is an ‘Excel’ icon to download a ‘t-test’ spreadsheet, so you’ll be ready to follow one later in the presentation and also for your convenience, the deck is already available for download there as well.
Also there are links to stay in touch with us in the deck, so you’ll get that later and lastly, there is a Twitter widget that will enable you to tweet right from the webinar confo. with the #bingadswebinars and @BingAds so we can see those streaming live and those will appear right there on the right hand side. An on-demand version of the webcast will be available approximately 4 days after the webcast and can be accessed at the link at the bottom of all the slides in our deck. It’s our Bing Ads Youtube Channel’s Webinar Playlist and the last housekeeping item before we get started, we would love your participation in our survey at the end of the webinar, so that we can improve in any way possible.
Let’s begin by introducing who we have on the webcast today. As you’ve heard, my name is MJ DePalma and I’ll be your host today and I’ll be monitoring the Q & A and making sure as much as possible it’s being moved along smoothly and then we also have Lars Hirsch, our Director of Advertiser Science here at Bing Ads. Super-excited to have him today and been working with him and we have Frances Donegan-Ryan, our SEM Pro Marketing Manager, SEM guru, so really happy to have you both here today, talking about the science of brand bidding. Great! Excited to be here.
Let’s get into it, the agenda. Here is what you’re going to learn today. Super-cool stuff! The Bing Ads Advertisers Science team’s overall brand experiment results and we’ve done it in a couple of verticals, so we’ll take a look at that. It will be in Retail, Finance and Travel and also even more exciting, I think, is how to design your own brand term experiment to really finally put to rest, you know, that office today, to bid on your brand term or not to bid on your brand terms. So we’ll teach you how to actually look at your own data and decide that. So the most important part of it is to make sure that your experiment is statistically relevant and so in our brand…you know, like I had mentioned earlier, you will download that statistical relevance worksheet called the ‘t-test’ template for download and it has the formulas already built in, so that’s super-cool. We’re trying to make it as easy as possible for you guys to get the most our of your advertising dollars. So without further adieu, I want to pass it on to Lars, to tell us exactly what Advertiser Science is.
Advertiser Science Overview:
Lars Hirsch: Yeah, thank you, MJ. So my name is Lars Hirsch. I lead the group called ‘Advertiser Science’ and we are the data science team for the Bing Ads business. In the Advertiser Science group, we have three main paratheids. Number one is analyze and interpret advertiser behavior and the impact relating to the marketplace dynamics. Second, we identify, test and measure strategies that help our advertisers to take full advantage of the Bing Ads platform. And finally we enable sales channels and partners to leverage data insight, to prioritize, drive efforts against the most meaningful activities in the campaign changes.
So we’ve been working on this question about brand queries and brand ads for a long time now. And we started up thinking about, that there wasn’t really a consensus among our advertisers. Does it make sense to bid on brand terms? It does not make sense. Of course, the question is: Does brand terms give additional clicks or does it primarily cannibalize clicks that you would have gotten for free, if it doesn’t…went through organic search? So you really want to turn this kind of dynamic tier and really discover the truth through your scientific methods, so let’s dive into the brand analysis.
So we compared two scenarios. We compared brand queries that had brand ads from the brand owner present and we compared that to brand queries where there wasn’t the brand ads from the brand owner present and when we talk about brand ads, may be we should specify what we talk about. So when we talk about brand queries, it’s a query that contains either the name, like the brand name of the company, it could be a product, it could be that brandname.com, it could be that class of modifiers such as [Kayak Slides 0:06:27] or , [Expedio Slides 0:06:28], but we only include queries that have above a certain threshold numbers. We look at all brand queries in our platform that has atleast 1000 searches per month.
Frances: Great. So thanks Lars. We are spitting on stage, and so what we wanted to show you is the click reality, so what happened and what were the results that we saw? So we looked at, whenever advertisers use the Social Network and when they….we looked at what happened when someone searches for their brands or modifiers of their brand and they were not running a brand ad campaign. They would capture about 55% of the clicks, as resulting from that search. When they were running a brand ad campaign, they were able to not only increase the organic results, clicks, but also ensure that they were capturing a 100% of the clicks, when someone searched for their brand. So there is obviously a really impressive growth for them to be able to move from only capturing that 55% of the clicks to a 100%.
When we did the same test and we looked at another one of our advertisers with the wireless brand, what we saw is, when they did not run a brand ad campaign, again they were only capturing just over 60% of the clicks, but not only that, they were losing clicks to their competitors. So they were losing clicks to their competitor’s organic listing and the competitor with running a brand ad campaign of their own and they were able to really take a chunk of click from our advertiser’s own brand name and go…what we saw, when they started including their own brand ads, they were able to retain almost, looks like 99% of the clicks there, and only last, that sliver to the competitor’s ad.
We ran the same tests for one of our Telecom advertisers and again, saw the same types of results and when we started to look at even more competitive industries like Telecom and particularly Retail, you can really see the magnificent results they saw with the brand ad campaigns. So if you look at the Retail, they were only capturing, you know, under 40% of the clicks and their competitor, in fact, was getting more percent of the clicks when someone searched their brand between their own organic listing and their brand ad and ones of Retail, this is a women’s Retail brand – once they added their own brand ad campaign, they were able to really capture and hold on to what looks like their 90% of the clicks, when someone searched into the brand and really only losing that 10% rather than, you know, the almost 50% before.
Lars: Yeah, that’s a pretty dramatic increase there and I think it’s worth to may be reiterating that this is real data; this isn’t…this is for specific brand queries that we have seen on our networks. Now, sort of, to put on to Retail brand, you can see, they almost eliminate the competitor…the clicks to competitors and more than double the clicks with their own website, by advertising on their brand term and when you think about the different types of brand queries, it’s worthwhile to think about the degree of competitiveness in the business. Of course, Retail is very very competitive; Social Network is probably very much less competitive, right? So those are kind of like extremes.
It’s also worth thinking about, if the query reflects us, subscription or ongoing relationship or if it’s more transactional, if it’s somebody who is clearly looking for wanting particular location, like if somebody searched for the Netflix login, for example, that person is probably going to go to Netflix, even if competitors place some ad in front of that user. It’s also worth thinking about, if the brand query is sort of like a proxy for a category and that’s what we see with these Retail brands and there is somebody that searches for a specific brand because that’s very top of mind. Well, it’s not necessarily about…somebody is dedicated to actually buying from that particular brand; it’s more like that’s the top of mind brand …mind query.
MJ: And I just wanted to say this visual, when I look at it from, you know, I’m looking at a kind of far away, look at all the bars on the left compared to all the bars on the right, so it’s like the dark grey and the orange stacked on top of one another are much higher than the ones on the right. So the big key takeaway, like, you know, you’re going to capture a lot more if you do the brand on the left than if you don’t…just boiling it down there. [Laughter]
Frances: And so what we want to show you now is, so like Lars mentioned before, this is real data from individual brands advertisers on Bing Ads and so when we were doing this for these specific advertisers, Lars and his team decided to look at the verticals overall as well to see what we could find when we looked in a bigger picture. So Lars is now going to take you through what we saw when we analyzed bring, doing brands campaigns across the Travel industry.
Lars: Thanks Frances. So we included all travel advertisers in this study and we included all travel queries that had atleast a 1000 queries per month. So that’s…this is the threshold; about everything is included within those thresholds. And this slide shows you the click yield to travel advertisers, what they don’t base on their brand terms. This is average across all travel advertisers. When we talk about click yield, that is out of a hundred searches, how many clicks did the brand owner get in this example? So of a hundred searches on the brand query, the brand only would get an average 61 clicks. So 61 clicks brand results. And you can see, on the right hand side you see a typical screenshot, so this 61% number is most related to Kayak.
This is an average, but you see on the right side, on the example, the Kayak; we see somebody is searching for Kayak flights, there is a bunch of competitors on top who are advertising and then you see Kayak further down in the page and this might be something like category search where people are looking for flights in general, Kayak is top of mind and they might be more or less like typically count some of the competitors. And of course as the brand only gets 61% of the clicks, the competitors would get 39% of the clicks in this example.
Frances: So again, just to point out, it’s not only you are not capturing the full percentage of the people who are doing the search with your brand term in it; you are losing a significant chunk of those searches to your competitor.
Lars: Yeah. So let me correct something that I said before. So this isn’t the click yield. This is actually the percentage of the clicks, so out of a hundred clicks, the brand owner gets 61%, with 61 clicks and the competitor gets 39 clicks, so that’s the width of the total clicks on Network. And let’s….so those numbers were averages across travel and here we are selling some real examples with a actual brand name [inaudible 0:14:15], but these are real queries, where you can see that without a brand ad presence, competitors captured nearly half of the clicks. It’s 27% for www.[brand].com and in the 40’s for some of the other, what they call brand modifier queries, we have a brand plus another search term, but think about it. I put brand.com, right? There is somebody that’s clearly looking for a very specific destination, right, but still competitors are able to capture more than a quarter of those clicks on that search. It’s pretty astonishing.
Frances: Yeah, particularly, you know, we’ll mention this time and again during this webinar, but the amount of work and dollars and marketing that has gone into having a customer retain the www.[brand name].com and the effort that they have done to do that search, to lose over a quarter, in fact almost a third, is a huge loss after the investment that you’ve made for them to, you know, know your brand name and be interested in going to your actual website as well. So you really don’t want to lose that. And it’s not only the opportunity to increase your leads, increase your customers, increase your conversions, like we mentioned. It’s really a way to retain your customers and your leads that you’ve already invested in, if they are already using your brand terms, but don’t let your competitors, you know, rob them from you and take them away from you.
Lars: So let’s see what happens if the brand owner is bidding on it’s own brand terms. So, as you remember, if the brand owner…yeah, this is averagicals Travel vertical. It’s the brand owner who doesn’t advertise. They get 61% of the clicks. If they do advertise, then what we see is an average; the brand will now get 88% of the clicks. So that’s actually a 45% growth in the total number of clicks that the brand owner will see. It’s pretty significant, I should say.
Frances: Yeah, that’s very significant. So we break that 88% down.
Lars: Yeah. So the question remains like how much of the clicks you now have to pay for, like if you don’t advertise…of course, all your clicks are free, it’s all through organic. If you do advertise, you’re going to pay for some of those organic clicks, but…or the clicks were organic, now they are not organic; now they are paid clicks. What we see is that it doesn’t look all that’s bad. Yes, you are going to pay…you will cannibalize some of the organic clicks, but most of the clicks that you got for free, when you didn’t advertise, you still get for free when you do advertise, so it’s your…most of those go for free, and then you have a big chunk that is just incremental.
Frances: Yeah. So this seems to be…well, definitely, when I have had conversations with advertisers and when I meet customers at events, this tends to be, you know, the main argument, they are against splitting on your brands and my background is actually in SEO. I have been doing Paid Search now two and a half years, but before that it was SEO, so even when I was an SEO with all the debates, you know, the amount of money, energy and time you’re putting into, you know, getting that mainline number one, number two, number three spot for your organic listing, why would you pay then to, you know, perhaps cannibalize that? And really what we saw time and again was that it was such a small percentage and the fact that you were able to retain such a larger percentage of the overall clicks that were coming to that search results page, that really it was worth your time and in fact helped you maintain your organic listing and made it still relevant and so popular.
MJ: It’s like the old analogy – “Sometimes you have to take one step back to go two steps forward”?
Frances: Yeah, exactly. And I think what you will see here is when you start running these tests for yourself, this overlap and, you know, in this example the average within Travel was about 18%, isn’t that going to be high enough generally for you to shy away from bidding on your own brand because the results of having that total 88% of the clicks is much stronger than the 61 and then adding on top of that the fact that those then are not going to your competitors, making you the more valuable.
Lars: Yeah, let’s take a look, closer look at that competitor angle, as you mentioned…
Frances: So here is some data on that.
Lars: So what we saw is that if the brand owner does not advertise, competitors are getting 39% of the clicks, but if the brand owner do advertise, then the competitors will get 12% of the clicks, so that is less than one-third.
Frances: So, I mean, if you are, within that 88%, what we saw on the previous slide there, was where you will see a little bit of cannibalization into your organic clicks, but when you really start comparing the larger numbers against what your competitors are able to take from you, it shows that that sacrifice or that pain for those additional clicks is so much more valuable.
MJ: Well, we had a question come in about, may be you could just touch on real quick. Would you clarify how you calculate the overlap percentage in clicks? If we were to go back to that slide just real quick and that 18% is overlap…?
Lars: Yeah. So what we saw is that if you advertise in the brand terms, 43% of the clicks…if you look at out of that 88% chunk, the 43% chunk is now organic, not paid for, while the 35% is paid clicks.
Frances: The net difference between 35 and 43?
Lars: It’s the difference between….
Frances: 61, I’m sorry.
Lars: 61 and 43.
Frances: Correct, yeah. Perfect. Thank you. So I’ll jump back, so again, let’s talk about what this one looks like versus your competitors, when you are running a brand ad campaign?
Lars: Yeah. So let’s assume…this is some more real examples. If you remember we were looking at some real examples of what happens if the brand owner is not advertising, where we saw, for www.[brand].com, if I remember, 27% of clicks going to competitors, turned out like totally navigational, right? It’s pretty astonishing. And this is a similar Travel advertiser. These are both two very large major Travel advertisers that are comparable, so it’s not comparing apples and oranges here. All of a sudden on www.[brand].com, each of those 27% of the clicks going to competitors; only 1% of the clicks are going to competitors. I will say some of the other examples are around the back…the Travel advertisers that didn’t advertise have upto almost 50% of the clicks going to competitors. Here there is no more than 11% in this example going to competitors. There is a pretty big difference. I’m pretty astonished especially on the www.[brand].com…
Frances: Yeah.
Lars: Turned the ad result, 27% going to competitors in the other example and here we have only 1%.
Frances: Yeah. I mean the retaining those customers…you know, we have all heard, in any business course, any business articles you’ve ever read that keeping a customer is far less expensive than acquiring new customers and this is a strategy to make sure that you are doing that. You know, keeping those customers that are already interested in looking for you, specifically if they are doing a www.[yourname].com, only losing, you know, 1% or less to your competitors, is a phenomenal increase over the 27% that you are losing.
Lars: Yeah. That’s a really good point, Frances, and you can think about this in the context of remarketing too, because you definitely want to remarket on the brand terms, so competitors don’t steal your customers that have already visited your website, may be already purchased from you all of a sudden. If you’re not there to capture them with an ad in the search for you, they might be going to your competitors and become your competitor’s customers instead.
Frances: Yeah. So that was a quick look at Travel. We also, like we mentioned before, we did the study across the Retail vertical in industry and we also did it across Financial Services and the same sort of study; we’ve looked at 3 million impressions for Retail purchase and worked with Retail advertisers. In December…I was going to say last year, but I realize now it’s December. Now, last year, the year before, only thirteen months ago, not too long ago! But we have the results of that study in the Appendix of this presentation, so when you are downloading, you’ll see it there, but we saw very similar things than what we saw in Travel in terms of how they were able to increase their clicks.
Lars: Yeah. I do think the results were even a little bit stronger than what we saw in Travel. I think for Travel…I think there is a pretty compelling case for bidding on the brand terms. For Retail, that case was even stronger, so like Frances said, it’s right there in the Appendix.
Frances: Yeah. So the price is there, so you can take a look through them there and they are pretty much identical to the Travel slide; just the numbers are represented above brand advertisers and some examples of Retail advertisers. Please wanted to make sure we had enough time to show you how to run this experiment for yourself, so what I am going to do is just a quick recap on what we have learnt from the test we ran and and then Lars will take you through how you actually run this experiment for yourself. So I am a massive sports fan, pretty much any sports out there, I love watching anything from rugby and soccer and American football, all the way to golf; I love watching golf. And so, you know, the major thing that you learn in sports, generally team sports is, you know, you are a great offense, you are going to win games here in Seattle, obviously big Sea Hawks fan! And so we know that Russell and his Offense’s team are great winning teams, but we would not have won the Super Bowl without our phenomenal defense team and I think you see that across all sorts of sports. If you are looking for winning a championship, winning in the long term, you must have a stronger defense as you do at amazing offense.
MJ DePalma: I have to say, I have to just chime in Frances. This is my mantra. I was a soccer goal-keeper, so I held on to the fact that defense won the championships, that leads in my mind. I love…you know, of course I love the force that are rescoring, get all those, you know, glory…. but I held on to that belief that, you know, we really need a strong defense.
Frances: Yeah. And that’s true. And really you see the same thing in your brand terms bidding strategies. Bidding on your brand terms, you know, helps you deliver more clicks and helps you deliver more leads and improves your conversions certainly, but what it also does which frankly might even be more powerful for you, based on how competitive your industry is, is that it absolutely means fewer clicks to your competitors. And it reduces that opportunity for your competitors to capture either your existing customers or new customers and that mind-share that they are able to steal from you when you don’t have a brand ad present.
And, you know, we’ve talked about this. You know, the risk there of not having it is not only fewer clicks on your organic; even if you have a strong SEO presence, you are going to get fewer clicks on your organic listing, if you do not have a brand ad campaign. And like Lars and I mentioned earlier, you know, you put so much money, time and energy into TV, radio, prints, the in-person events, perhaps sales at your store, if you are present modus; all of this marketing activity and lead-generation only to leave it at the last moment.
Lars: Yeah. It’s definitely worth reiterating that a lot of people see search as the end of the marketing funnel, right, so you put all of this money into upstreams marketing channels, only to have your competitors capitalize on the search to your brand, so basically you’re paying to drive traffic to your competitors, if you’re not able to capture that through, as per the funnel clicks.
Frances: Yeah. So, you know, this also is a great strategy that allows you to sit down and connect with other teams, so if you have separate teams that are working on SEO, separate teams that are working on acquisition, separate teams that are working on paid search, it’s a really…I’m someone who has worked on all three of those, so really awesome opportunity to come together, work as a team and ensure that search and paid search in particular is supporting the work of your SEO work, is supporting the work of your acquisition team and not burning that money and not losing out on that customer that you have already put so much work into nurturing.
MJ DePalma: We just had that question, so stay tuned. We’re going to teach you how to do an experiment and possibly help your company. Besides that….?
Frances: Yeah. So just wanted to do a quick overview. So what we saw when we did these larger studies across…these larger tests and experiments across industries as a whole, we looked at Financial Services, Retail and Travel. Financial Services are average brands click-list with 33%; Retail, like Lars mentioned, was just astonishing at 51% increase in click-lift and Travel at 44%, so if you fall into any of these categories, these are the types of numbers you could see; these are the average across the industry and talking about our offense/defense, here is what we saw competitor click lost. Massive in Financial Services – 57%, Retail at 43% and Travel again that 44% really showing that not only are you increasing with the opportunity for your ads in your organic listing to be clicked on, but you are also really shutting down the opportunity for your competitor to take those away from you.
MJ: Yeah. A question to that point, Frances, coming from iCrossing: What is an average impression-share a competitor receives when brand ads are not being displayed? So I think in a way this slide is directional in each vertical and so if you don’t actually run your brand ads, this is what you can pretty much see your competitors getting.
Lars: Yeah, so we haven’t looked specifically at impressions within, focusing on clicks… We should have that data somewhere, so we might do that on our follow-up studies possibly, but it’s a very interesting question.
Frances: Well, we’ll take note of that question and see either in the blog posts or in another webinar in this series, if we can take a look at that.
MJ: Fantastic! So want to jump into how do you do that for yourself, so, you know, this web-series is all about the Science of Advertising, so how do you set up your experiments or your tests to really use your own data to look at this phenomena for yourself?
Frances: You know, one of the things that I, in working with Lars to produce this webinar, is that, you know, I came to understand very clearly and, you know, from my college days, of setting up actually psychological experiments….my major was in Psychology, is that you have to learn how to set up statistically significant experiments and to do that, make sure that you download the ‘t-test’ in the upper-right hand corner. It’s an Excel template that Lars built for you guys with the formulas already built-in, but you definitely want a statistically significant difference in your experiments so that it yields for liable results.
And so, in order to set up your ‘t-Test’, just a general idea, is you are going to want access to two data sets, so you’re going to want the organic clicks and the paid clicks and you want the amount of data for the organic clicks to be equal to the lengths that you are going to be running your experiments during the paid clicks. Is that right, Lars?
Lars: Yeah. So you need the organic click data as well as the paid click data for those lengths of the experiments and possibly also for a time period before the experiment equal to the length….
Frances: Perfect.
Lars: Well, we’ll get back to more details in the next couple of…
MJ: That means probably partnering with your SEO teams and…
Lars: Yeah, you need access to the SEOs clicks data, for sure.
MJ: So tell us about the first type. [Laughter]
Lars: So we have two, there is two different ways that you can set up experiments to find the effects within the brand terms. One is to run brand campaigns on alternating days of the week, so that means that your brand ads will be active every other day and you need to do this for atleast two weeks, possibly, you should probably do it for may be four weeks, but atleast two weeks because of the day-a-week effect. So you have one Monday with brands…Brand ad is active while the next Monday you would have an inactive, so you don’t skew the results because of different patterns of different days a week. So the first thing that you need to do is to organize your brand keywords into one or more separate campaigns that only have brand keywords and all the brand keywords are contained into those campaigns.
So it’s…you can apply targeting specifically to those brand campaigns. So once you have that organized, you should then go to Bing Ads’ Day-a-Week targeting to enable and disable brand campaigns every other day and that is a targeting and setting that lets you target individual week days. So you can, for example, for the first week, you enable the brand campaigns only on Monday, Wednesday, Friday and Sunday. Then at the end of the week, you will have to go and change that setting because it doesn’t let you alternate with alternating weeks, so the next week you would enable the brand campaigns on Tuesday, Thursday and Saturday. Did I get that right, Frances?
Frances: Yes.
Lars: I think so. And then you run this for atleast two, may be four weeks. It’s got to be a even number of weeks and then you compare each day’s total brand clicks for the brand-enabled days with the brand-disabled days using t-test and MJ mentioned there is already that template for this. You can just fill in numbers into the Excel template and it will tell you the calculation. Your calculations tells you the results, so obviously….
Frances: Yeah. So that was…so this is looking at your brand clicks as the days of the week, so once you collect that data, you dropped it into the Excel template, you have your results that can tell you, “Figure them out!”.
Lars: Yeah. So you want the t-value to be less than 0.1 to be statistically significant and what that means is that there is only a 10% chance that these two data sets come from the same distribution and 90% chance that there is actually a difference, which is what usual, a normal threshold for determining statistical significance. And I think MJ….
MJ: Oh, yeah, I alluded to the fact that if this isn’t statistically relevant, chances are that you don’t have a large enough sample set, so you’re going to want to run the campaign longer to get more data, but there is also the other option, that it’s telling you not to run brand terms. There is that also option.
Lars: Because there is no difference between the total clicks that you get with the brand terms, you will basically get the same clicks through organic, if there is no, if you don’t advertise [cross-talk 0:35:59]…The results that we have seen for the absolute most part support the second bullet here that, “Well, may be you didn’t run it for long enough,” but of course, if you have an experiment, you are set out to prove or disprove something, so there is also possibility that actually there isn’t that effect, right?
MJ: Exactly. You want to be fair!
Lars: Want to be fair!
MJ: Yeah. And also just to mention that the day of the week test is really good for SMB’s rather than this one, the Geo…
Lars: Well, specifically for local businesses, so that’s the other way that we…
MJ: Local, yes.
Lars: That you can set up a brand experiment, is to target different geographies and say, “Well, we’re only going to run brand campaigns in certain geographies,” and of course, if you are limited to a singular, few geographies, you can’t do this. You have to use the day-a-week approach.
MJ: Yeah, okay.
Lars: This is a good approach. The Geo approach is a good approach for national advertisers that advertise across all or many geographies and here, what you need to start off with is to identify the pairs of metro areas with similar customer profiles to your business. You can think about how do your customers differ and you want to look at metro areas that are similar across in those important attributes. It could be age, gender, interest, what have you… whatever is important for you. So you want pairs of metro areas with similar composition with regards to the attributes that are important for your customers. That makes sense, MJ?
MJ: Yeah, makes sense. If you have any questions, you know, please keep sending them. We will get to them at the end.
Lars: So start with as many as you can identify, I think, twenty pairs – that’s probably a good number. You have forty metro areas total. And then from each pair, you have to randomly select one of the metro areas in each pair into what we call the ‘Treatment Group’ and one metro area from …the other metro area from that areas then go to ‘Control’ and this has to be done randomly and what’s important is that you don’t have a pattern to how you select it, right? That’s why we say it’s got to be random. You shouldn’t select like “If one of my pair is a niche pair…” or “This is a little bit larger” or “SKUs is one particular way…” It’s important to have that person always go to the same group, either that be ‘Treatment’ or ‘Control’. It should be done in a random way. And there is a link to random numbers generated here that you can use. There is probably a million of them out there, in this search for [inaudible 0:38:25]… That shouldn’t be too hard.
Frances: Yeah.
Lars: Then what you do is to measure combined organic and paid clicks by metro area before the experiment and this has to be done in a time period that is the same as the duration of the experiment. So if you’re running a two weeks experiment and you measure the combined organic and paid clicks, also the two weeks before the experiment. So what you want to do is, you use that as a baseline, as you measure the total clicks during the experiment, I guess. So you have the…If you run a two weeks experiment, you look at the two weeks before the experiment, the total clicks, the two weeks before experiment.
If you have a 4 week experiment, then you look at 4 weeks before the experiment. I think two to four weeks is probably the ideal, good length to run the experiment. And the final step, you enable the geo targeting on brand campaigns, so you put all the DNAs, the geographies that are assigned to the ‘Treatment’ group. Those are enabled for the brand campaigns. So you only run your brand campaigns for the ‘Treatment’ geo areas.
Frances: So this is the same…you know, whenever you run an experiment, whether it was in high school or college class, you always have the control group, right? You always had the group that you aren’t modifying so you can see those results and then the groups that you are modifying or testing, so that you can compare them back and forth, and that’s what you mean by the ‘Treatment’.
Lars: Yeah, exactly. So the ‘Treatment’ group geo areas get brand campaigns enabled and the ‘Control’ do not get brand control and they will…and for all of the geo areas that are outside the experiment, doesn’t matter. You can just do what you usually do.
MJ: Brilliant. And just to remind you guys, in the upper-right hand corner, is the Resource list. This is what it looks like. You can just go, download that t-Test and there is also other resources there; few of you have apps. for soft copies of the presentation, there is already a PDF there for you to download, so go ahead and do that and there is also some pretty nerdy links there for more information on statistical relevance in t-test.
So wanted to show you quickly and we can have Lars chat about this as well because he created this Excel document. This is just a screenshot of what you’re going to see when you download that template and because we have talked about two different ways to run the test, we included how to calculate for both of those. So when you open the Excel doc., you will see tabs along the bottom that let you run either day of the week, if you are running that version of the test or geo-targeting. And I am going to little bit include such…and Lars can show you how to use it here.
Lars: Yeah. So for this tab, this is a geo-targeting example. You have, you see the yellow columns on the left side. That’s where you put the total number of clicks, in the pre-period, before the experiment started. As I mentioned, you have to look at the total number of clicks for each metro area in that time period that’s identical to the time period you want to run the experiments, but you look at the time period before the experiment starts and that’s what…to set the baseline and that’s what you put on the left hand side. You’ll see, like the first upper-left yellow cell, you put in the total number of clicks to Boston across paid…. I should say total number of clicks on brand queries to Boston on the upper-left cell.
Then upper-right cell, Boston is paired with Philadelphia, so that’s the first pair of metro areas. You have the total number of clicks on brand queries to Philadelphia, etc., down those two yellow columns and the two yellow columns further to the right, that is the total number of clicks on brand terms during the experiment period. So you put those numbers in there, you see out of those two columns, you can see the left column is the ‘Treatment’; there is the right column that’s ‘Control’ – that those are the metro areas that don’t get the brand campaigns enabled.
You just go through that column, everything will populate once you fill in those yellow numbers and then you see in the black rectangle underneath, you see the t-test results. In this example, there is a 0.04, the t-value is 0.04, which is a statistically significant value and you see the average click increase is 6%, which is, I should point out this is fake data; this is not real data. We expect the average click increase to actually be much much bigger than that. This is just basically random numbers, so…
Frances: Yeah. Just to be able to show you once you run your reports to whatever analytical tools you are using, whether it’s Google Analytics or something else, once you run your report, these are the columns that you drop your data into. You would obviously need the paid clicks reports from your Bing Ads campaign and then the calculations are already there and will set out the t-test results. Again, it’s showing you if it is statistically significant data and then will also set out the average clicks increase that you are likely to see, when you do run a Bing Ad brand campaign. So just another sort of recap, when you are running an experiment, my sister is a chemical engineer, so I went to her and chatted with her when she was in college…
Lars: I’m too!
Frances: You are too? Oh! So I was like, “Margaret, I need some experiments terms. When you are running experiments in your lab, what are the things that you have to keep an eye on to?” And she said, “Well, we always had safety considerations.” So this is for my sister. So safety considerations. Really it just means, “Hey, these are reminders for you.” Like MJ said, you do need to have patience when you are running an experiment, any type of experiment, so making sure that you give yourself the time to collect the data that’s going to be the strongest data set that you can use… This is really basic, but I have definitely done, you know, a ton of research on a topic to decide, ‘Hey, is this something I should be doing or not?” and then never really followed through with the implementation.
And so we really just wanted to remind you, after you’ve gone through all of this work, you’ve shown your patience, you’ve collected your data, go ahead and implement them. Make sure that you set up the opportunity. If it’s the fact that you might need more budget, for example, if you are an agency, you work with your customer to make sure that you can get that. If you need more budget, could be you have to take some other campaigns and move to this one; just make sure you that are set up to be able to implement, particularly if your results are showing a really great increase in your click rates.
And then wash, rinse and repeat. All of us who work in search know that this is not a static industry. It’s changing all the time, both on the SEO side and on the paid search side and so, this is probably not a test that you will do once and never revisit. It might be something that you want to do once a year, might be something that you want to do during a closed seasons versus a busy season. It’s certainly not something you need to be doing monthly, but make sure that it’s something that schedules into the auditing or the quality-checks that you do on your paid search campaigns overall. If you do kind of an audit yearly, this is probably one of the things you want to be looking at.
So make sure that it’s built into your process and that your teams and your customers are aware of that. It’s something you’re really taking advantage of. I’m going to hand back to MJ. We Want to make sure we’ve enough time for questions. Like MJ mentioned, whatever questions we don’t get to answer for the rest of this hour, I will be writing a blogpost. It will go live on Thursday and it will include the most recently asked questions as well as anything you don’t get to hear, and you could always tweet us: @bing ads or myself: @FrancesDR. Lars, you also….?
Lars: Yeah. @larshirsch.
Frances: Yeah, @larshirsch (Twitter), and so we’re always available on there to help you, either with these types of questions or really anything else about Bing Ads.
MJ: Thank you so much Lars and Frances. I appreciate that. So just want to recap real quick and make sure I heard what you guys have been saying this whole time and that there is really…you got to first decide what type of experiment is right for your business, so the geo versus the days of the week and then just have patience because, you know, you may not have enough data in the first time around, and just, you know, you will probably have to do the experiments several times to reach statistically relevant pool of data that you need. The other thing that I wanted to mention is someone actually asked a question that is going to take more time to answer, but, you know, we live in a very complex world and there is a lot of variables going on, so we’re not saying like this one single test is really going to be a real clear-cut, like a 100%, not considering, like may be there is news in about your brand that comes into play, you know. You have to really kind of think bout a lot of different things and that question is, you know, very interesting. We are going to have to dig into that a little bit more about, you know, all the variables, so thank you for that question.
But really, use the results to expand your brand keywords, if the experiment is telling you to. It’s playing offense and defense; as we know, in sports, really helps you in championships, not just games. So hopefully you have now the information to go, run your own experiment and to finally put the office debate to rest, with may be your SEO team, I am not sure – to bid or not to bid. So hope you guys have got some great information. Let’s get into the Q & A because we do have a ton of questions. We’ve got a nice good chunk of ten minutes. Let’s see. Our first question, let’s see. Here we go. You see one, Lars, that you would like to ….?
Lars: Yeah, so I saw a couple of questions around targeting your competitor’s brands. There were several of those sorts of question about what to do if you are a SDM and you don’t really have a known brand name, and I think all of those questions are just looking at the same data from a different angle. It’s obvious that targeting your competitor’s brand is something that works to drive traffic to your website especially, one – if the competitor is not bidding on his or her own brand terms for the brand name, and second, if there is a very competitive vertical.
If you are going to try to be at Social Network, may be it doesn’t pay to advertise on Facebook.com because people that search for Facebook will go to Facebook because that’s what they are looking for, but if you have done very competitive verticals, very competitive segments of Retail, Travel or Financial Services, you will probably be successful at driving traffic to your website by bidding on the competitor’s brands. Well, of course, follow the policies that are relevant in your country, like in the US, of course don’t…you shouldn’t put competitor’s brand names in your actual ad, but you can bid on competitor’s brand query.
MJ: Right, yeah. So you can bid and this, I thought this question…you can bid against competitor’s brand terms and brand names and the different brand modifiers that you spoke about, like brand name slides, brand name dresses, etc, but you cannot use their brand name in your ad copy. That’s…not sure if I should share this, but currently, you can use your competitor’s brand name in your custom URLs and text and that is a strategy I have seen very competitive brands use… So, you know, you put in your destination URL that’s hidden, and then you have your custom URL that’s visible. You can get creative there at how you use, get some brand terms. You have to be careful still and to be honest, I’m not sure how long that will continue. There is a case pending in URLs, actually [inaudible 0:51:44] is suing Google over that currently, so I’m not sure where that will end up, but if you really want to get savvy with your paid search versus your competitor, you know, you can and…
Frances: I don’t know. Do unto others and they haven’t do unto.. you would like… That’s my motto.
Lars: I mean, we should…URLs to have positive, some of this [inaudible 0:52:03] term in the US to… Our policies do not allow bidding on competitor’s brand terms in some countries in Europe.
MJ: Correct, yeah. So if you are a global advertiser, that you may have different strategies in different markets. If you are a national advertiser, you can definitely get competitor’s … here in the US. Some of the other questions we saw were, it touches upon the, against different verticals in industry, so I saw someone has asked about Education; someone else had asked about …I’m not sure if I saw another specific example, but Lars, if you have looked at some others, if you…?
Lars: We have data alone, but we haven’t still… This is sort of like the insight into three of the major verticals products. Not sure if we’re going to produce additional charts and slides and more verticals. That could come in the future; I don’t know, but right now we’ve focused on those three verticals.
MJ: Alright. There is a vertical that you would like to see them do another study on, you know, the ‘squeaky will get’s the oil’…
Lars: Yeah. Just…
MJ: Just tweet your request, for sure. There is…
Frances: Also, just to mention, if you have a Bing Ads account team, they have access to the data and so you can work directly with your Bing Ads account team to dig into some more data, if you are interested.
Lars: Oh, thanks. Yeah, that’s a really good point, Frances. Yes, you can get personalized data through your Bing Ads account team. They have access to all the data on every single major brand query. Again, same threshold – 1000 searches per month. So you could look at the individual queries and whatever you want.
MJ: Yeah. Do you have test results for incremental clicks? We have answer to that question on other verticals or industries, so we just answered that. Do you have examples of the impact on not bidding on branded…when the competition is often not bidding?
Lars: So, we do have. So that is also part of the study and the impact is smaller. You would still drive incremental clicks if you bid on your brand term. All of a sudden you will own both the ad space as well the organic space and you will still push some of that competitor’s organic results further down on the page, but the impact is definitely smaller than when there is a competitor ad on top.
Frances: Yeah. We did see an example, again, specifically in Retail, which tends to be fiercely competitive, that even if the other brand isn’t bidding and they don’t have paid ads for that results, they still will have organic results and like Lars said, you still will lose clicks to those competitor organic results with or without ads from either of you, but what we did for you is one such brand started their own brand bidding campaign. They were able to increase the number of clicks coming to them and decrease any risk of losing those customers to the competitors.
Lars: Yeah. It’s really intuitive. Think about just owning more of this space multi-fold, right, more of the paid space multi-fold and your competitor owning less of the multiple space.
Frances: Yeah, it’s all about the mind-share and I think, you know, then there is background in Psychology as well, you do see that and when I was in the SEO world, we always talked about having the magic 1-2-3 which was the ad, the top organic listing and then the referral listing, right of results seeking about you from a valued third-party. So having those 1-2-3 is always going to increase the mind-share and that is going to help you retain customers as well as capturing new ones.
MJ: So I’ve another question that’s kind of interesting and I was thinking a second version for this question. The question is: Does the data for year and when there is a new story about the company brand…and my second part of that question is: Besides our data, what if you are running an experiment and something happens and your brand is splashed over the news? Do you stop the experiment and reset?
Lars: So, it depends. So the assumption is that news story should affect the ‘Treatment’ and ‘Control’ groups equally, if you are running in ‘Treatment’ and ‘Control’ geographies and this has a huge effect on one particular geography, like one particular area. You won’t want to exclude the pair, that that area is a part of, so say, we have Boston and Philadelphia pair, that’s…”Control’. If something happens, that’s only relevant to Boston, you want to just exclude that, but if it affects across the board, then you are good, because the experimental design will take care of it. So you have a tour type ‘Treatment’ and ‘Control’ in a similar way across the board and they have results pulling out.
MJ: Great, love that, thanks. So kind of just follow up on that question. What the question is: Lars, is there ever a time when you should stop doing brand campaigns or do you reach a threshold which…no longer necessary?
Frances: My assumption is to say ‘No’ because as soon as you stopped, you’re going to start losing the advantage that you’ve gained.
Lars: I agree with that, Frances, so that generally, probably don’t want you stop doing brand campaigns, but if you want to test the waters, then you should run one of these experiments. You can think about this as a reverse set of things, whether trying to measure the effect of advertising or continuing brand… Instead of measuring the effect of introducing brand ads, you can think about this measuring effect of stopping your brand campaigns, right, so you can say the treatment is not having the brand ads active and that’s basically the same set up and we can again get the results of what happens, if you stop advertising on the brand.
MJ: Great. Great. Want to respect everyone’s time. We have about 45 more seconds and I wanted to remind everyone that if you have additional questions, just tweet us at: #AskBingAds and we will definitely follow up: #AskBingAds and it wouldn’t hurt to also throw in #BingAdsWebinars, so we know it’s connected to the webinar and also coming up, we have our February webinar that you can already sign up for, with Lars, Frances and myself and it’s ‘The Kevin Bacon Approach to Keyword Attribution’, totally fun in an interesting way to look at keywords that lead up to that last click conversion keyword. So feel free, sign up. We hope to see you on the next one and thank you so much everyone. As soon as this webinar is over, we would love your feedback. A survey will pop up. Thanks again and thanks for the feedback.
Frances: Thanks everyone. See you on Twitter.
Lars: Thanks everyone.
Related Articles:
Stop the wasted ad spend. Get more conversions from the same ad budget.
Our customers save over $16 Million per year on Google and Amazon Ads.