Apple’s iOS 15 gives its internal apps, products, and services an advantage over third parties. Are they intentionally confusing customers? Here’s what we learned:
California Court rules against Apple’s anticompetitive practices in Epic Games lawsuit
App Tracking Transparency (ATT) differences for iOS 14 third parties vs iOS 15 Apple’s apps
Insights into Apple’s anti-industry and anti-consumer practices
“…the Court finds that common threads run through Apple’s practices which unreasonably restrains competition and harm consumers…”
Epic Games, Inc. v. Apple Inc.
Is iOS 15 Intentionally Confusing Customers?
Consumer Acquisition has been reporting on Apple’s anti-developer, anti-advertiser restrictions for months now. While our reports have primarily focused on devastating advertising revenue loss and impacts, two recent Apple developments have shown they are also anticompetitive to the disservice of consumers and developers.
Personalized Ads or Tracking: Apple’s ATT Prompt
Apple released iOS 15 on September 20 and there are notable differences in the App Tracking Transparency (ATT) prompt for internal Apple products versus third-party ATT prompts. Their apps are custom-designed to harm the mobile app advertising ecosystem and confuse consumers. According to iMore, this change was visible in the public beta as well.
Any app distributed through the App Store that is now Apple’s is required to ask each consumer if they will allow having their activity tracked using the following language: Allow [App] to track your activity across other companies’ apps and websites? (See graphic below). Developers can add one customizable sentence to explain the value of tracking. Users must select “Ask App Not to Track” or “Allow” before continuing in the app.
Apple’s Inconsistent Privacy Language
For iOS 14.5, Apple automatically opted-in consumers to be tracked for Apple’s internal apps, which was troublesome considering their privacy rhetoric. With iOS 15, Apple is now showing an ATT prompt for their internal Apple apps; however, they provide substantial advantages to Apple by using specifically designed soft language for this purpose. In contrast to the terse, transactional prompt to permit tracking, for their own apps Apple asks you to “turn on personalized ads”. They also include a wordy explanation of all the benefits of said personalization, along with privacy protections and increased content relevance.
You cannot overstate the difference in language. One prompt is about empowering a user to do a positive thing while the other prompt is about preventing a negative thing. Personalization is delightful and sought out, while tracking is intrusive and objectionable. An empowered user can turn a feature on or off like a light. However, a disempowered user must ask for something negative not to happen to them.
Apple’s Unfair Standards
Apple’s behavior is custom-designed to give their products a strategic advantage over third parties like Facebook, Google, and TikTok. For all of Apple’s extensive privacy promises, the rhetorical spin-doctoring they employ to persuade users to be tracked for Apple’s benefit casts a new light on their commitment to privacy. But, the end result is the same. If a consumer opts out of ATT they will not see fewer ads. Instead, they will see non-personalized ads, essentially unsolicited spam.
Why is it okay for Apple to use advertising-friendly language with their own apps and then use aggressive anti-advertising language for third parties? This language is creating an inconsistent and confusing consumer experience. However, Apple seems to be building a walled garden of perceived consumer safety and security. At the same time, they are screaming like Chicken Little when third-party, AI-driven algorithms offer elegant, unbiased recommendations outside of Apple’s human-curated preferences. As app developers and advertisers are hobbled by IDFA loss, what resources do they have when the iOS rules are so blatantly different and unfair? Hopefully, Apple will create universal standards they apply consistently to their own apps. Apple’s recent announcement that native apps can now be reviewed and rated is a step in that direction.
Monetary Impact
In Q2, 2021 earnings calls, both Applovin and Playtika shared that iOS 14.5+ adoption was over 80% and the ATT opt-in rate was only 35-40%. Zynga warned IDFA removal would have a material impact and H2 2021 bookings could be down by $100M. Consumer Acquisition is seeing a -15% to -20% average revenue loss for iOS broad audience games and a -35% revenue loss for heavy IAP/niche audience games.
September data from Flurry shows that worldwide, users being shown the ATT prompt have been opting in at a rate of only 23% week-over-week since August. In the U.S., the opt-in rate is an abysmal 16%. Custom and lookalike audience size is down -77% worldwide (86% in the U.S.) with these low ATT consent rates. As a result, effectiveness has dropped. Many mobile app developers have paused spending on iOS or shifted heavily to Android, inflating CPMs.
Epic v. Apple Ruling
Last week, there was a monumental ruling in the industry-shaking lawsuit of Epic Games v. Apple. Epic sued Apple “alleging violations of federal and state antitrust laws and California’s unfair competition law based upon Apple’s operation of its App Store,” with Apple disputing the allegations. While the court did not conclude that Apple was a “monopolist” or engaging in “antitrust conduct,” the trial did show Apple’s “anticompetitive conduct” under California law.
The evidence in the case demonstrated that gaming apps make up nearly 70% of all App Store revenue. And only 10% of all app store users generate that revenue. The court found that Apple’s anti-steering restrictions—which prevented developers from communicating with users about lower prices on other platforms— “hide critical information from consumers and illegally stifle consumer choice” and called for “nationwide remedy.” You can read the full court findings here.
In the ruling, Judge Yvonne Gonzalez Rogers issued a permanent injunction declaring that Apple is “permanently restrained and enjoined from prohibiting developers from (i) including in their apps and their metadata buttons, external links, or other calls to action that direct customers to purchasing mechanisms, in addition to In-App Purchasing and (ii) communicating with customers through points of contact, obtained voluntarily from customers through account registration within the app.”
The ruling takes effect in 90 days unless delayed or reversed. Then, app developers can add metadata buttons and external links for alternative purchasing mechanisms outside of the App Store. By offering users these alternatives, app developers can decrease IAP costs for users. They can also reduce their own overall App Store commissions by 30 percent. Increasing the diversity of the monetization ecosystem will ultimately benefit the entire mobile game industry.
Here is My Take on Apple’s iOS 15 Anticompetitive Behavior
Only 10% of all App Store users, mainly from games, are generating 70% of App Store revenue. Apple appears to be trying to corner the market of high-value spenders by:
Preventing users from accessing less expensive and less flexible alternative purchasing methods in the App Store. This is to maintain their high margins (i.e., the Epic Case)
Luring users out of third-party advertising platforms with privacy scare tactics (i.e., anti-advertising ATT language)
Making it very difficult for third-party platforms to identify top spenders and effectively build lookalike models (i.e., obfuscation of IDFA)
Ultimately, this behavior is highly disruptive to the vitality of the iOS advertising market. It is having a negative impact on mobile app game developers, advertisers, and consumers.
For More Background On Our Perspectives Around Post-IDFA Loss
Mobile app advertisers agree that the full costs of internal UA teams are 6-8% of media spend. We work with your team to bring industry-wide expertise and insights across paid social providers. This will provide a broader understanding of the highly evolving IDFA impact.
We provide a creative studio and user acquisition services for Facebook, Google, TikTok, Snap, and Apple Search advertisers.
Our Hollywood-based storytellers can produce ongoing persona-driven creative concepts to stay ahead of creative fatigue.
Founded in 2013. Manage over $3 billion in creative and social ad spend for the world’s largest mobile apps and web-based performance advertisers.
Get creative tactics from high performing TikTok ads to use in your next UA campaign
Get the latest creative recommendations straight from TikTok for Business
TikTok says, “to captivate new audiences, the key is to continuously share fresh and diverse content.”
TikTok Creative Hot Tips
To offset the impacts of IDFA loss and automated media buying across networks, mobile game advertisers must be willing to try different platforms to unlock new audiences. To help UA and creative teams adjust to the evolving ad ecosystem, we have compiled expert recommendations for TikTok UA campaigns. Our market insights and creative expertise come from managing over $3 billion in creative and social ad spend for the world’s largest mobile apps and web-based performance advertisers.
Appeal to mission- and achievement-oriented gamers through a direct call-to-action in text or copy, with TikTok reporting an 11.3% lift in impressions for this tactic.
Showing features and gameplay is more effective than talking about them, with TikTok reporting a 12.6% lift in impressions.
Adding an end card to summarize key download information or gameplay tips resulted in a 47.3% lift in impressions.
83.2% of top-performing videos have a resolution of 720p or higher so shoot high-resolution high-quality videos.
Always include music or talk track because 93% of top-performing videos use audio.
1 in 4 top-performing videos are between 21 and 34 seconds long, but the length of your video should ultimately be informed by the message you want to get across.
Videos that use a full 9:16 aspect ratio see a 60.5% lift in impressions over those that include black space or fit poorly.
TikTok users expect to keep their devices in portrait orientation and videos shot in vertical format see a 40.1% lift in impressions compared to videos using a square or horizontal aspect ratio.
TikTok’s Creative Solutions
As we have reported, without deterministic tracking on iOS or effective lookalike audiences, relying on general broad-stroke creative won’t be enough to profitably expand your audience. However, persona-led creative developed around player motivations, preferences, and interests is the key to sustainable user acquisition. In TikTok’s Creative Solutions: The Ultimate How-to Guide their recommendation is clear: “in order to captivate new audiences, the key is to continuously share fresh and diverse content.” With the platform built on a content graph instead of a social graph, “diverse discovery” keeps users “inspired and energized.”
When we analyze high-performing ads through the lens of player motivations, such as those identified in Facebook’s 2021 Big Catch Playbook, we can break down why those ads appeal to different audiences.
TikTok’s List of High-Performing Game Ads
According to public reporting from TikTok For Business Creative Center, below are six examples of high-performing game ads in the U.S. from the past 30 days, based on impressions, CTR, and video view rate. We’ve identified what creative trends these ads leverage to be optimized for TikTok as well as player motivations they trigger.
Founded in 2013, we are a technology-enabled marketing services company and creative studio that has managed over $3 billion in creative and social ad spend for the world’s largest mobile apps and performance advertisers.
We provide end-to-end creative and user acquisition services for mobile app marketers via performance-oriented creative storytelling, integrated UA, and creative optimization.
We provide game-changing results driven by quantitative optimization and a relentless focus on increasing your net profit driven by client-specific, creative learning agendas.
Clients we’ve helped grow: Roblox, Glu Mobile, Disney, SuperHuman, Rovio, Jam City, Wooga, NBA, MLB, Ford, Sun Basket, Lion Studios, MobilityWare, and many others.
Want better results from TikTok creative UA?
Working with a knowledgeable marketing partner that can provide extensive quantitative creative testing ensures you get the best performance from your TikTok ads. Email us at sales@consumeracquisition.com to discuss your user acquisition goals.
Are you ready for NFL kickoff? Check out what’s working right now for NFL sports apps ad creative.
Discover user motivations for sports app users
Compare top creative trends
Get fresh creative trends
NFL & Sports Apps: High Performing Ad Creative
NFL starts this week and mobile app advertisers competing for the attention of fans will have to get really creative. As we have reported, the loss of IDFA has caused massive disruption to the mobile ad ecosystem, and the erosion of lookalike audience targeting alongside automated ad management means app advertisers must focus on persona-led creative designed for different motivations and interests to unlock new audiences and Facebook just published Creative Prototyping and The Big Catch Playbook full of recommendations. Krystel Bitar, Global Gaming Product Manager at Facebook emphasized in a recent article about Automated App Ads (AAA) how critical creative diversification based on motivation is: “it’s time to make different creatives inspired by these motivators. The more unique these are, the easier it will be to ultimately determine what has attracted its audience.”
With the loss of deterministic tracking in full effect, insights into player behavior continue to deteriorate and a one-size-fits-all ad creative limits advertisers to the players they’re already familiar with. Now is the opportunity for mobile app advertisers to adapt. Tailoring creative to different interests, motivations, and desires of players will attract high-quality top-of-funnel installs from new audiences that were missed by more broad advertising, making creative the most efficient lever for sustained profitable user acquisition. Our creative learning agenda incorporates intensive research into game genre, personas, and user motivations as well as understanding what ads our target users are responding to in other markets.
Here are competitor trends we’re seeing for sports apps ad creative:
Gameplay and game overview that showcase graphics and players (Home Run Clash, NBA 2K, Boxing Star)
Distance or performance challenges for viewers to propel an object to reach a target (Baseball Boy, Slap Master)
Competition videos dramatizing head-to-head competition between rival players or teams. (Darts of Fury, Home Run Clash)
Noob vs. Pro videos displaying bad and good gameplay (Darts of Fury, Mini Golf King)
Real player footage replacing game characters and gameplay (Tennis Clash, Sniper Arena, Draft Kings)
Cinematic techniques like slow-motion and camera pan utilizing game characters (Johnny Trigger, Mr. Bullet, Sniper Arena, Tennis Clash)
Augmented gameplay adding illustrations, emojis, talk bubbles, and voiceovers to gameplay or characters (Mr. Bullet, Draft Kings, Flip & Dive 3D)
Game controller overlay on gameplay to give it a game console effect (Tennis Clash, City Fighter Vs. Street Gang)
UGC styles leveraging influencer and player interviews and reviews (Underdog Fantasy)
Team creation scenarios and what-ifs
Mapping these creative trends to game player motivations, such as those identified by market researcher Quantic Foundry, we can break down why they work and find more opportunities for unique creative storytelling.
Real player footage and cinematic techniques provide immersion, fantasy, and excitement
Distance challenges and Noob v. Pro videos appeal to mastery, competition, power, and fantasy
Player rivalries appeal to competition, mayhem, action, challenge, power, and fantasy
Gameplay and controller overlays appeal to immersion, excitement, challenge, and competition
Augmented gameplay provides social appeal and story immersion
Team creation scenarios appeal to creativity, customization, strategy, and making decisions
Here’s what’s working in sports apps ad creative based on MobileAction data and Facebook Ads Library:
Head-to-Head Player Stats
DraftKings showcases NFL player’s pics with head-to-head stats between rival players, appealing to users who value achievement, challenge, and competition.
Team Creation
SuperDraft also showcases NFL player’s pictures and fantasy team stats, appealing to users who value strategy, decision making, creativity, completion, and challenge.
Head-to-Head Team Rivalry
WinView games showcase head-to-head team rivalry appealing to users that value community, achievement, challenge, competition.
Head-to-Head Players
FanDuel showcases individual NFL players to appeal to users who value strategy, making decisions, achievement, and competition.
Influencer/UGC
Underdog Fantasy uses side-by-side player discussion to appeal to players who value a connection to other players.
Dramatic Gameplay
Baseball Clash dramatic cinematic techniques applied to gameplay to appeal to players who value immersion, fantasy, and excitement.
Performance Challenge
Tennis Clash uses an overlay of a player’s hand to guide shots, appealing to players who value mastery, competition, power, and fantasy.
Simple Gameplay
Golf Rival uses simple progressive gameplay and mellow music to appeal to players who value immersion and mastery.
Noob vs. Pro
Pool Ball Night uses gameplay footage of cringe-worthy shots overlaid with a player-in-picture and request for help from the viewer, appealing to players who value achievement, mastery, challenge, competition, and community.
Sports Apps Creative Concepts
Create videos that incorporate statistics into the gameplay
Overlay graphics to show the distance of tape measure shots and challenge viewers to play
Overlay stats graphics on characters, challenging viewers to beat their opponent
Appeals to players motivated by competition, power, challenge, excitement
Create a countdown of featured players in the game
Showcases animation style, range of players, and stats
Style graphics like collector cards
Spark curiosity about who is (or is not) included
Appeals to players motivated by completion, surprises, competition, community, and power
Add trash talk over gameplay
Showcases gameplay and graphics alongside subtitles
Appeals to players motivated by competition, community, being on a team, mayhem, interacting
Feature gameplay with emoji overlays and talk bubbles to communicate emotions
Humanizes and adds humor
Showcases game graphics
Appeals to players motivated by community, being on a team, interacting, fantasy, and surprises
Intercut gameplay with real footage, so that it’s hard to tell what’s real and what’s the game
Showcases game graphics and simulation
Appeals to players motivated by immersion, fantasy, action, discovery
Re-create the feel of a famous announcer calling a legendary game with gameplay and voice over
Creates emotional “what if” scenarios
Engages viewer’s curiosity
Highlights the passion of sports fans
Appeals to players motivated by community, being on a team, immersion, fantasy, action, discovery
Create videos using cinematic techniques and voiceover to up the emotional ante of the game
Showcases game graphics
Utilizes nostalgic sports documentary feel
Appeals to players motivated by community, being on a team, immersion, fantasy, action, discovery
Create side-by-side videos of good (Pro) versus bad (Noob) gameplay
Shows the difficulty of the game
Highlights both “wins” and “fails”
Appeals to players motivated by competition, power, challenge, excitement
Create a sing-along music video of gameplay with a famous sports chant
Most chants are in the public domain
Showcases graphics and gameplay
Appeals to players motivated by community, being on a team, immersion, fantasy
Why trust our UA and creative recommendations?
Founded in 2013, we are a technology-enabled marketing services company and creative studio that has managed over $3 billion in creative and social ad spend for the world’s largest mobile apps and performance advertisers.
We provide end-to-end creative and user acquisition services for mobile app marketers via performance-oriented creative storytelling, integrated UA, and creative optimization.
We provide game-changing results driven by quantitative optimization and a relentless focus on increasing your net profit driven by client-specific, creative learning agendas.
Clients we’ve helped grow: Roblox, Glu Mobile, Disney, SuperHuman, Rovio, Jam City, Wooga, NBA, MLB, Ford, Sun Basket, Lion Studios, MobilityWare, and many others.
Facebook, TikTok & Google Creative Testing Best Practices: Post-IDFA Loss is our new Q4 2021 update to our wildly successful whitepaper. IDFA loss, SKAN requirements, and media buying automation through Facebook AAA or Google UAC are tough challenges facing user acquisition and creative teams right now. We have new best practices with updated recommendations, including the following:
Facebook, TikTok & Google creative testing best practices
iOS A/B creative testing without IDFA
Updated Android A/B creative testing best practices
Analysis of the evolving mobile app ad ecosystem
Creative testing recommendations for IAP and IAA apps
Automated ad buying benefits and deficits
Facebook, TikTok & Google Creative Testing Best Practices: Updated For IDFA Loss
Creative diversification based on motivation is critical, according to The Big Catch from Facebook: “it’s time to make different creatives inspired by these motivators. The more unique these are, the easier it will be to ultimately determine what has attracted its audience.” It seems obvious that ads should be tailored to appeal to different audiences’ motivations and interests. However, too many UA managers rely on the crutch of behavioral targeting to solve for one-size-fits-all, middle-of-the-pack creative.
Lookalike Targeting: Before and After
Prior to the loss of IDFA, lookalike audience targeting was reliable, effective, and efficient; it offered limitless opportunities to slice and dice revenue events to uncover new high-value users. A strong lookalike audience could scale and run for a month or more; but now, ROAS maybe 0.5% when it was previously 15%. As a result, advertisers are decreasing social ad spend for iOS by 40-50% each month since ATT enforcement and are consistently shifting more ad spend over to Android.
As reliance on upper-funnel campaigns increases, ad creative optimized to appeal to discreet personas is the most efficient lever for sustained profitable user acquisition. Without deterministic tracking, understanding user motivation is critical to attracting high-quality top-of-funnel installs. Beyond the install, the first 48 hours of app usage should identify consumers that indicate a propensity to monetize. We recommend streamlining and instrumenting your onboarding flows and events to capture these early monetization signals. With persona-led creative paired with onboarding events, the algorithms will learn what changes to make to deliver better audiences.
UA teams that understand their target personas can scale efficiently, even without IDFA. While much of the industry has been focused on user behavior, the deterioration of lookalike audiences and the black box of deeper funnel events means a user’s declared interests are critically important for insights into motivation and intent.
To maintain advertising efficiency, you need a steady pipeline of new creative ideas and content to test. When successful, a new creative concept can lift performance by 200% or more. This performance increase is worth the cost, time, and risk that’s inherent with testing new concepts.
STRONG PERFORMANCE COMES FROM SMART TESTING
We test a lot of ad creative. We produce and test more than 100,000 videos and images yearly for our clients, and we have performed over 25,000 A/B and multivariate tests on Facebook, Google, TikTok, and Snap. Our industry expertise comes from managing over $3 billion in creative and paid social spend for the world’s largest mobile apps and performance advertisers. We focus on gaming, e-commerce, entertainment, automotive, D2C, eSports, digital subscriptions, financial services, and lead generation.
Consumer Acquisition runs our tests using our software AdRules via Facebook, Google, and TikTok APIs. Our process is designed to save time and money by killing losing creatives quickly and to significantly reduce non-converting spend. This process also does not necessarily follow the published Facebook, Google, or TikTok guidance of running a split test to reach statistical significance before moving into the optimized phase. Our insights are specific to the above scenarios, not a representation of how all testing on all platforms operates.
THE COSTS OF CREATIVE TESTING
In classic testing, a 95% confidence rate is ideal to declare a winner, exit the learning phase, and reach statistical significance (StatSig). However, that ideal 95% confidence rate for in-app purchases may end up costing an advertiser $20,000 per creative variation.
Here is an example scenario: To reach a 95% confidence level, you’ll need about 100 purchases. With a 1% purchase rate (typical for gaming apps), and a $200 cost per purchase, you will end up spending $20,000 for each variation to accrue enough data for that 95% confidence rate. There are not a lot of advertisers who can afford to spend $20,000 per variation, especially if 95% of new creative fails to beat the control.
With a cost of $20,000 per variation and 20 variations to find a winner with a 95% failure rate, it would cost $400,000 just to find a new control.
THE SECRET TO COST-EFFECTIVE CREATIVE TESTING
To avoid such high testing costs, we move the conversion event we are targeting up, or towards the beginning of the sales funnel. For mobile apps, instead of optimizing for purchases, we optimize for impressions per install (IPM). For websites, we optimize for an impression-to-top funnel conversion rate. Again, this is not a Facebook recommended best practice. This is our own methodology, designed to allow advertisers to find new, top-performing creative in the most cost-efficient and reliable way.
This process does pose a risk that high CTRs and high conversion rates for top-funnel events may not be true winners for down-funnel conversions and ROI/ROAS. It is more efficient to optimize for IPMs over purchases, so we take that risk to save the time and expense of optimizing for StatSig bottom-funnel metrics. Perhaps most importantly, this method means you can run more tests for less money per variation. For many advertisers, that alone can make more testing financially viable. $200 testing cost per variation versus $20,000 is the difference between a couple of tests or an ongoing, robust testing program.
Because the outcomes of our tests have consequences, we also test our creative testing methodology. That might sound a little “meta,” but it is essential for us to validate and challenge our methodologies, assumptions, and results. The outcome of every test shapes our evolving creative strategy, so making the wrong call means incremental changes that could have significant consequences. When we choose a winning ad out of a pack of competing ads, we ensure it’s the right decision.
Section 2: Our Foundational Testing Approach
FACEBOOK TESTING BASICS
When testing creative on Facebook, we typically test three to six videos along with a control video using Facebook’s split test feature. We show these ads to broad or 5-10% lookalike audiences. We restrict distribution to the Facebook newsfeed only, Android only, and use mobile app install bidding (MAI) to get about 100-250 installs. If one of those new “challenger” ads beats the control video’s IPM or comes within 10%-15% of its performance, we launch those potential new winning videos into the ad sets with the control video and let them fight it out to generate ROAS.
In November and December 2019, we produced 60 new video concepts for a client. All of these creatives failed to beat the control video’s IPM, which was statistically impossible. We expected to generate a new winner 5% of the time, resulting in three fresh winners. Confident in our creative expertise and execution, we looked deeper into the testing methods.
Based on numerous ad accounts and confirmations from 7-figure spending advertisers, the following ad performance scenario has been common. We’ve anonymized the scenario to share critical performance insights.
A/A TESTING IN FOUR ACTS
To ensure a methodology is sound, you must test your testing system through an A/A test. Instead of testing multiple creatives as you would with an A/B test, A/A tests run the same creative in each “slot” of the test.
If your testing system is working as expected and you are close to statistical significance, all “variations” in an A/A test should produce similar results. If your testing system concludes that one creative variation significantly outperforms or underperforms compared to the others, there could be an issue with the testing method or data.
We set up an A/A test to validate our non-standard approach to Facebook testing and to understand if Facebook maintains a creative history for the control. If so, the control video would get a performance boost making it very difficult to beat.
A/A Test 1
We copied the control video four times and added one black pixel in different locations in each of the new “variations.” This allowed us to run what would look like the same video to humans but would be different videos in the eyes of the testing platform. The goal was to get Facebook to assign new hash IDs for each cloned video and then test them all together and observe their IPMs.
For anonymity purposes, we’ve replaced the actual ad creative with this dapper dachshund. The far-right ad in the blue square is the control creative and all others are clones of the control with one black pixel added. The IPMs for each ad are on the far right of the image.
The far-left ad/clone outperformed the control by 149%. As described earlier, a difference like that should not happen if the platform were truly variation agnostic.
We ran this test for 100 installs, our standard operating procedure for creative testing. However, we did not follow Facebook’s recommendation to allow the ad set(s) to exit the learning phase. We analyzed the results then scaled up to 500 installs to get closer to statistical significance. Our remaining question was if more data would result in IPM normalization (in other words, if the test results would settle back down to more even performance across the variations). However, the results of the second test remained the same. Note: the ad set(s) did not exit the learning phase and we did not follow Facebook’s best practice.
The results of this first test, while not statistically significant, were surprisingly enough to merit additional tests. So, we tested further!
A/A Test 2
Test 2 included six videos. Four videos were controls with different headers; two videos were new concepts that were very similar to the control. Again, the actual ad creative has been replaced with cute animals.
The IPMs for all ads ranged between 7-11 – even the new ads that did not share a thumbnail with the control. IPMs for each ad in the far right of the image.
A/A Test 3
Test 3 included six videos: one control, four visually similar variations to the control, and one very different for a human. IPMs ranged between 5-10. IPMs for each ad in the far right of the image.
A/A Test 4
This was when we had our “ah-ha!” moment. We tested six very different video concepts: the one control video and five brand new ideas, all of which were visually very different from the control video and did not share the same thumbnail.
The control’s IPM was consistent in the 8-9 range, but the IPMs for the new visual concepts ranged between 0-2. IPMs for each ad in the far right of the image.
A/A TEST RESULT ANALYSIS
Facebook’s split tests maintain creative history for the control video. This gives the control an advantage with our IPM testing methodology.
We are unclear if Facebook can group variations with a similar look and feel to the control. If it can, similar-looking ads could also start with a higher IPM based on influence from the control — or perhaps similar thumbnails influence non-statistically relevant IPM.
Creative concepts that are visually very different from the control appear to not share a creative history. IPMs for these variations are independent of the control.
It appears that new, “out of the box” visual concepts vs the control may require more impressions to quantify their performance.
Our IPM testing methodology appears to be valid if we do NOT use a control video as the benchmark for winning.
The line graphs below show the IPMS from the second, third, and fourth tests
FACEBOOK’S AAA ADVANTAGES
We do all our testing on Facebook because of the granular controls we have for bidding, creative reporting, and targeting. While Facebook’s Automated App Ads (AAA) reduce visibility into ad campaigns, there are some clear advantages to using the system.
You get creative performance reporting on images, videos, and text, and you can get that tied to standard events. This reporting persists whereas Google’s doesn’t.
You have a choice between an auto bid and a bid cap.
There are fewer restrictions on the text. You can use CAPS, exclamation points, emojis, etc.
Flexibility with creative assets. You can basically do whatever you want with 50 assets, and the reporting will persist.
You can assume there is a CPM discount and a CPI discount. Whenever you use Facebook solutions that they are encouraging, the platform will give you a discount.
Android and iOS performance can be comparable to standard campaigns.
4 x 5 and square aspect ratios tend to do best on Facebook, though in some placements, a full portrait (9 x 16) is better. Be careful about Facebook allowing you to run creative assets that don’t entirely fit on given placements. Facebook will just shave off the top and bottom of your ads in some cases, which could mean they are chopping off the ad’s CTA or other critical information.
Note: We also suspect that Facebook will soon give advertisers even more flexibility with how many creative assets they can use. They know that creative testing is the best competitive advantage left for advertisers, and they want to give us the tools to aggressively optimize our ads.
Our creative testing process was built leveraging deterministic tracking and 1:1 asset-level reporting of multi-stage creative’s lifecycle testing. This process spans from IPM to ROAS across the learning phase to the eventual optimized phase. A/B creative test reporting is coupled with client-provided revenue targets frequently provided by an MMP (Appsflyer, Adjust, Singular, Kochava).
Even prior to IDFA loss, we tested on Android because it was less expensive and easily translated to iOS. We saved iOS testing for the rare clients without an Android app or those who were solely targeting iOS users. Testing creative for both IAP and IAA (outlined below) continue to work effectively on Android. In that sense, IDFA loss has validated our cost-effective strategies.
Fortunately, advertisers with Android apps who are NOT using Facebook’s AAA algorithm can follow our best practices and maintain their deterministic efficiency. You’ll maintain your ability to A/B test and see results at the individual asset level. Once a winning creative is identified, move it to iOS or other paid social platforms.
OUR 3-STEP CREATIVE TESTING PROCESS FOR IAP (IN-APP-PURCHASES)
Phase 1: IPM Test
No control videos
Create a new split test campaign using 3~6 new creatives (no control).
Setup campaign structure for basic App Install (No event optimization or value optimization)
Spend an equal amount on each creative. Ex: One ad per ad set.
Budget for at least 100 installs per creative
$200~$400 spend per ad is recommended (based on a CPI of $2-$4) if T1 English-speaking country
$20~$40 spend per ad/adset testing in India (based on $0.20-$0.40 CPI)
US Phase 1 testing.
10-15% LAL with a seed audience like past 90-day installers, or past 90-day payers.
Non-US Phase 1 testing.
Use broad targeting & English speakers only
If not available in India, try other English-speaking countries with lower CPMs than the U.S. and similar results. Ex: ZA, CA, IE, AU, PH, etc.
Use the OS (iOS or Android) you intend to scale in production
Use one body text
Headline is optional
FB Newsfeed or Facebook Audience Networking placement only (not both and not auto placements)
Be sure the winner has 100+ installs (50 installs acceptable in high CPI scenarios)
100 installs: 70% confidence with 5% margin of error
160 installs: 80% confidence with 5% margin of error
270 installs: 90% confidence with 5% margin of error
IAP Titles: kill losers, top 1~3 winners go to phase 2
IAA Titles: kill losers, allow top 1~3 “possible winners” to exit the learning phase, and then put into “the Control’s” campaign
Which Creatives Move from Phase 1 > Phase 2?
How To Pick a Phase 1 IPM Winner?
IPMs may range broadly or be clumped together
Goal: kill obvious losers and test remaining ads in phase 2
Ads (blue) have IPMs 6.77 & 6.34, move to phase 2
If all ads are very close (e.g., within 5%), increase the budget
IAA (in-app ads titles) you may need more LTV data before scaling
Phase 2: Initial ROAS
No control videos
Create a new campaign with AEO or VO optimization
Place all creatives into a single ad set (Multi Ads Per Ad Set)
Use IPM winner(s) from Phase 1 (you can combine winners from multiple Phase 1 tests into a single Phase 2 test)
OS – Android or iOS. 5-10% LALs from top seeds (purchases, frequent users + purchase) + Auto Placements
Testing can be done at a lower cost if you wish to run this campaign in other countries where ROAS is similar or higher but CPMs are much lower compared to the US – ie., South Africa, Ireland, Canada, etc.
Lifetime budget $3,500-$4,900 or daily budgets of $500-$750 over 4-6 day (depending on your $/purchase).
WARNING! Skipping this step is highly likely to result in one of the following scenarios:
Challenger immediately kills the champion/control but hasn’t achieved enough statistical relevance or exited the learning phase and therefore the sustained ROAS/KPI may not be sustained.
Champion/control video has a lot more statistical history and relevance and most likely has exited the learning phases and may immediately kill the challenger before it has a chance to get enough data to properly fight for ROAS.
The latest change to our Phase 2 strategy: we have become more focused on retention than we were a few months ago. Before, we were just looking at pure ROAS. Now, we pay attention to retention information as it comes in on Day 1 to Day 3 or so. This is usually possible, because we’ll run Phase 2 ads for about 5 days, and that’s worth weighing in for which pieces of creative you take to Phase 3.
Phase 3: ROAS Scale
No control videos
Use strong CBO campaign
Choose winner(s) from Phase 2 with good/decent ROAS
You have proven the ad has great IPM and “can monetize”
To win this phase, it must hit KPIs (D7 ROAS, etc.)
Create a copy of an existing ad set
Delete old ads and replace them with your Phase 2 winner(s)
Allows new ads to spend in a competitive environment
Then, create a new ad set, roll it out towards target audiences with solid ROAS / KPIs
CBO controls budgets between ad sets with control creatives and ad sets with new creative winners.
Intervene with ad set min/max spend control only if new creatives do not receive spend from CBO.
Require challenger to exit the learning phase before moving to challenge the control “Gladiator” video.
Once the challenger has exited the learning phase, allow CBO to change budget distribution between challenger and champion.
Our 3-Step Creative Testing Process for IAA (In-App-Ads)
Phase 1: IPM Test
We use the same approach for IAA ads as we do for IAP ads in Phase 1.
Phase 2: Initial RPM
Look at your MMP. Find whichever countries/regions/geos are paying out a good RPM, and just go with those.
No control videos. Multiple ads per ad set. We call this “the gladiator battle.”
Create a new campaign with AEO (not VO), but instead of the event being a purchase, have it be an event that could only occur after someone has played the game for a long time like they’ve achieved a certain level.
For AEO, with a nonpurchase event, you can lower your budget down to even $250 per day.
Auto placements are okay if that’s what you typically use for the account. Generally, just use the placements you already know will work. No need to reinvent the wheel here. Just run your default best setup.
For audiences, just run your top two to three audiences. If you are concerned about budget, stack your audiences so you only have one ad set. This will get the ad set out of the learning phase faster and save you some money. (Keep in mind that you may not get out of the Learning Phase at all sometimes in Phase 2 testing.)
Phase 3: ROAS Scale
For IAA, there really is not a Phase 3, but our recommendations for what to do at this stage depend on your budget:
If your budget is small, you are not going to know performance for several weeks and so you might as well just roll your best-performing ads out into production.
If your budget is large, do a scaled-down Phase 3 structure as we suggest for IAP advertising. This is especially important if you have control ads that are still doing well. Roll strong Phase 2 performers whenever you need a win.
iOS CREATIVE TESTING WITHOUT IDFA
The loss of IDFA has profoundly impacted creative testing on iOS and requires a new strategy for creative optimization. Unfortunately, the trifecta of IDFA loss, account simplification required by SKAN, and media buying automation through Facebook’s AAA or Google UAC had an immediate impact on creative testing and creative strategy.
If you don’t have an Android app or you have embraced Facebook’s AAA algorithm or Google’s UAC, get ready for a different way to A/B test: ASSET FEEDS!
Most major platforms (Facebook, Google, Tik Tok, Snap) will have limited account configurations due to iOS14 SKAN tracking limitations. iOS14 accounts will be restricted to 9-11 campaigns with 5 ad sets per campaign, meaning you’ll have 45-60 permutations. It will be difficult to justify burning these ad sets on A/B testing. Therefore, new concepts will be the most important lever for your UA team.
Facebook (XML feed spec) and TikTok (XML feed spec) have published solutions for asset-level reporting data tied to dynamic asset-feeds for creative. The solutions attempt to:
Allow creative partners to tag, track and measure the performance of individual media.
Allow basic dynamic reporting (eg. CTR, spend/asset, clicks, impressions) for an individual asset in the Ad Set. However, they appear to not allow for multivariate-level reporting of the combination of ad copy, headline, and creative.
Prevent MMP data from being married to asset feeds based on the current platform’s specs. This may be a concern for companies leveraging their reporting to make creative or financial decisions from A/B testing.
Help with fatigue identification through Google’s introduction of asset performance labels (Best/Good/Poor…). This will aid in asset feed performance diagnosis and is a starting point to provide simple suggestions for what asset to optimize or replace.
From small changes to wholly new concepts
Due to limited testing slots and opaque creative-level reporting, creative strategy is shifting from optimizing with small changes to testing entirely new concepts. When creative optimization moves into asset feeds, the performance results of each creative are blended together into a kind of creative blob. Unfortunately, it will be difficult to know the contribution of each element is optimized. What we’re seeing:
Creative iterations and variations based on the most effective asset in a portfolio provide a higher likelihood for success but a much lower lift in performance (think < 5%).
Creative optimization is very likely to shift toward new concepts that take 5-20x as long to conceive and execute, but they offer a much higher potential for success (think 20% to 500%) and a correspondingly large risk of failure.
On average, we see a 5-15% success rate for new creative concepts, but when they succeed, the results can be a massive increase in KPI performance.
As creative becomes a targeting mechanism and user-intent filter, the new demand for fresh creative concepts vs iterations is certain to put a large strain on internal creative teams.
Google UAC Creative Testing
The loss of Apple’s IDFA has a cascade of consequences including how ad creative is used on Google. If you are running an asset feed ad like Google UAC, you need plenty of creative combinations within the feed to be truly effective.
With Google AC split testing feature not yet widely available, we recommend a phased Evergreen> Motivation ad group test plan.
Account Creative Cycling Structure
Week 1
Launch with one ad group (evergreen)
Fully loaded 20 video assets, landscape image, and 10 text assets
Week 2
Open 3-4 ad groups with distinct text assets only, centered on product motivations
Social, Exploration, Achievement, etc.
Week 3
match top motivation text group with corresponding video assets and iterate
Kill underperforming motivation groups
Repeat the process and iterate on the winning motivation concept
Tactics
Avoid relying solely on Google’s Best/Good/Low Creative Score for optimization; these scores are relative to other comparable sized assets in the ad group and do not always align with KPI
Instead, identify creatives with scale potential at KPI and consider creative metrics IPM/CTR along with CPM
Google UAC Advantages
Flexibility in campaign bidding structure.
You can have multiple campaigns running in any geo. You can have different CPAs, campaigns just running for installs, etc.
Better transparency for performance with 1:1 event reconciling via MMP data.
You can see the performance by traffic source.
Using multiple Ad Groups you can put different creative approaches into separate “buckets” and see how their performance compares. This also means you can run seasonal or ad hoc events without disturbing evergreen campaigns.
Being able to create multiple Ad Groups also means you can test a lot more creative than you can on Facebook. Google does cap your assets at 20 videos and 10 images, but if you need more you can just create more Ad Groups.
Portrait (9 x 16) and landscape (16 x 9) aspect ratios are the best bet right now for videos on UAC.
Google UAC CHALLENGES
In Google, you can have up to 20 videos and 10 images. Those are hard caps; you cannot swap out and have even 21 videos and 9 images. But if you swap those assets out to add new assets, you will not be able to see the stats for those creative assets. It is possible to get the stats back if you re-add the videos. So, the data isn’t lost. It’s just not shown.
Reporting on creative performance is harder to get.
You cannot control bid settings (like an auto bid and bid cap).
You cannot control where your traffic is coming from, or how it is allocated throughout your campaigns, Ad Groups, and ads. This means you cannot really do a proper split-test. You can have one creative in one Ad Group, and then create as many Ad Groups as creative variations, but that is not a proper split-test.
Google is more restrictive about the text. It will not allow CAPS, exclamation points, emojis, etc.
There are no standard campaign options on Google. There’s only UAC.
TIKTOK CREATIVE TESTING
TikTok’s A/B testing capabilities are still rapidly evolving so we recommend A/B testing on Facebook Android. If you can only test on TikTok, here are our suggested best practices:
Creative testing on TikTok is limited to running two creatives at a time, due to the limitation of two split audiences
Broad targeting is recommended for testing to keep CPMs low
Each ad set within a split test must spend at least $20/day in order to run
TikTok automates split tests to run for 7 days, but this can be shortened
Tests can be run with an optimization goal of Clicks, Installs, or In-App Events
Available bidding: Cost Cap Bidding or Lowest Cost Bidding. The lowest cost is recommended to maximize your number of results.
Both Standard Ads and Spark Ads (Organic Posts) can be split tested.
Musical iterations of ads can be easily created and tested using the TikTok Video Editor
OPTIMAL CREATIVE SIZES ACROSS PLATFORMS
To maximize creative testing, run optimal aspect ratio and media types relative to each platform across Facebook, Google App Campaigns, TikTok, and Snap. However, as the failure rate of new creative is 95% you will also want to minimize the number of sizes you produce until you have uncovered a fresh winning creative. Below is our recommendation to maximize distribution and while minimizing creative production.
Section 4: A Final Word On Creative Testing
IDFA loss, SKAN requirements, and media buying automation through Facebook’s AAA or Google UAC are tough challenges facing user acquisition and creative teams.
But don’t despair. Dean Takahashi from GamesBeat says, “Advertisers aren’t helpless. In a post-IDFA world, Facebook, Branch, and Consumer Acquisition recommend focusing on ‘persona-led creative’ (or marketing to a type of person) to regain efficiency by allowing paid social algorithms to cluster users based on behaviors and creative trends.”
Tactics we recommend throughout these pages may not be strictly aligned with publicly available platform best practices. But we know they work. Our market insights and creative expertise come from managing over $3 billion in creative and social ad spend for the world’s largest mobile apps and web-based performance advertisers. Since 2013 we’ve worked with Roblox, Glu Mobile, Disney, SuperHuman, Rovio, Jam City, Wooga, NBA, MLB, Ford, Sun Basket, Lion Studios, MobilityWare, and many others. We provide end-to-end creative and user acquisition services for mobile app marketers via performance-oriented creative storytelling, integrated UA, and creative optimization.
