Archive for the ‘DashO’ Category

Improving Ad performance: Correlating ad activity with feature usage and user behavior

Friday, June 24th, 2011 by Sebastian Holst

In this third installment on application analytics patterns and practices I’m going to focus on how Runtime Intelligence can be used to shed light on Ad activity within the context of one or more applications. While the use cases covered here are nowhere near exhaustive, I’m going to show how to answer the following questions (and hopefully give some indication as to why you may care about the answers):

  • What are ad impression volumes across multiple apps?
  • What are the click-through rates (the ratio of users clicking on ads to the volume of impressions) across various pivots?
  • What influence does culture (country of origin) have on click-through rates, e.g. are Germans more or less likely to click on ads versus Italians?
  • What carriers/ISP providers are giving me the most business, e.g. where are my users most likely to be found?
  • Where are users spending most of their time inside an app? Does that usage pattern correlate with a user’s likelihood of clicking on an ad?
  • Do returning users interact with ads differently than first time users or power users?
Many of these metrics are valuable in scenarios other than ad effectiveness of course (knowing where users spend their time and understanding how power users behave are two obvious examples), but for this installment, I am going to focus exclusively on how Ad interaction can be viewed across these metrics.

Implementation
I’m using the same trusty one line method WhatPoseWhen that I described in the first installment – this time, I call the method on the New Ad event (to count impressions) and the Ad Engaged event (to count clicks on ads). I could just as easily collect data on any other ad-related event and grab any data that is available to the program at that point in its execution as well. Here is the code for that method in its entirety:

private void WhatPoseWhen (string page, string selection)
{ return; }

The first parameter tells me from which page the method is being called and the second parameter tells me why I might care, e.g. was a new ad displayed, etc.)

I pass the page name and the ad event into WhatPoseWhen and Runtime Intelligence grabs these parameters and sends them up to the repository (no programming for this). I can then correlate the ad activity within the context of sessions, feature usage, and runtime stack data that I am getting as a part of runtime intelligence.

For these metrics, I export my CSV data into a regular excel spreadsheet and then generate the pivot tables shown below.

App background
I always like to use data from true production apps rather than fabricate data sets; I am using two apps that I wrote and launched on the marketplace that are both ad driven, Yoga-pedia and A Free WPC Yogi – the former is a free version of a yoga app that (hopefully) helps to drive sales of a for an upgrade to A Pose for That. A Free WPC Yogi plays a similar role for The WPC Yogi, a tailored version of A Pose for That targeting WPC 2011 attendees.

The following post uses their Ad activity over the same one week period.

Impression counts
The following pie chart shows the “new ad” event count by application. As you can see, Yoga-pedia has roughly 4X the number of ad impressions and given the fact that these apps are very similar (but not identical) in their behavior, this also roughly correlates to the volume of usage as well.

Click-through rates
However, when I divide the total number of “ad engaged” events by the total number of “new ads,” I see that A Free WPC Yogi has a 28% higher click-through rate (1.78% versus 1.37). In point of fact the demographics of the app users are quite different (randomized consumers versus MSFT partners who are attending WPC 2011).

Advantage: This intelligence helps to segment users by differences in their behavior and to do a better job of targeting those differences across apps.

Impressions by country (or culture)
Runtime Intelligence can grab the IP address of the sending tower – this is not personally identifiable and cannot be used to locate an individual with any precision – but it is more than adequate to identify country, state, and city. In the following graph, I simply count new ad events by country and show the top 10 countries by impression volume.

Advantage: If your app has a cultural bias that would benefit from localization, understanding where your users are can help prioritize those localization efforts.

Click-through rates by country (or culture)
The following bar chart calculates the click-through rates for the top 10 countries listed above. What is interesting here is that there appears to be a significant difference in click through rates by country (culture).

Advantage: Understanding when/if users from specific cultures are significantly more likely to respond to (click on) ads can further help to prioritize localization or marketing investments.

Impressions by ISP provider (top 25)
To produce the next graph, I used an application to tell me who owned the IP addresses that my mobile clients are using (I used IP2Location – but there are many of them out there).

This is a nice way to see who my users favor in terms of their carrier. Here I only show the top 25.

Advantage: Understanding carrier popularity will help focus business development/marketing efforts and better manage potential risks associated with how your users may be negatively impacted by upgrade schedules (delays). Will your next app be dependent upon Mango?

Sessions per app page
In the raw CSV files that can be exported from the Runtime Intelligence portal, there is a column, ApplicationGroupId. The value in this column is unique for all signals (messages) that are sent from within a single app session. In other words, I can use this field to organize all user activity into the relative user sessions using this field. This is helpful for plotting specific user patterns.

The following graph simply counts the unique occurrences of ApplicationGroupId values by page name value (recall that this is the first parameter of the WhatPoseWhen method). This avoids counting multiple views of a single page within a single session and tells me how popular specific pages are across my user base. For this posting and for illustration, I’m only showing data for five specific pages.

FindAPoseDetail and BrowseSelectPose are central to the user experience (browsing for yoga poses and then drilling into a specific pose for detailed imagery and instruction). TellMeMore is the page where I describe what comes with the paid version of the app (nice to see that 10% of my users deliberately choose to investigate the upgrade possibility) and AppGuide and TopicList are essentially app documentation and I can see that these pages are not hit very often – and that’s not a bad thing – users should not need to use the documentation after their first use.

So – this graph is telling me that

a) My users are spending their time using the app rather than trying to use the app
b) I am at least getting my user’s attention regarding a possible upgrade – perhaps my content is not compelling enough if my conversion rate does not correlate.

Advantage: broad user proofing can be used to validate developer assumptions about user experience and effectiveness of pages for their specific purpose.
Ads shown per page compared to volume of times viewed
Next I calculate the average number of ads shows per page by dividing the total count of New Ad messages by page (this combines the two parameters, page name and the even New Ad) by the total count of the times the page is shown. TOTAL ADS SHOWN PER PAGE / TOTAL TIMES PAGE VISITED

I use the same ad duration interval across all of my pages – so this is actually another means of calculating how much time my users are spending on each page (this can be done with Runtime Intelligence alone, but in this case, I don’t have to do that).

The graph below shows the average number of ads shown per page and maps them to where they rank in terms of how often the page is visited.

Happily, the two core pages of my app also get the most ads (and are also where my users are stopping to spend time). I can also see that users spend more time on detailed pose descriptions than they do browsing – even though the browse more often than they drill down (which makes perfect sense).

Sadly, my upsell page is getting the least love – I definitely have to work on making this page more engaging.

Advantage: Ad frequency by page provides insight into where users spend their time. Calculating click-through rates by page identifies where users stop to look around and may be most open to suggestion.

Returning users and sessions per user
Another column in the CSV extract is the ANID – this is either the result of hashing the true ANID from a user’s phone (it is not the actual ANID value), or, if they opt-out of that, it will contain a GUID generated by our software and written to isolated storage. In either case, this value acts as a unique user identifier.

The ANID can be used to identify new and returning users. Dividing session count (ApplicationGroupId) by ANID gives the average number of sessions per user. The following bar chart takes the 10 ANIDs with the highest session counts and compares the resulting sessions per user value to the rest of the user base (whose count is roughly 500 other users).

What I see is that there is a core group of users that are heavily using my apps (YAY!). Now that I know who they are, I can zero in on their specific behaviors, how they relate to my ads, what features they use most heavily, etc.

Advantage: Segmenting users into new, returning, and power categories dramatically improves a developer’s ability to target, prioritize, and validate development, marketing, and support activities.
Conclusion

I hope to have shown how using Runtime Intelligence, developers can materially improve their ability to build more effective applications and refine their advertising strategy while coordinating that strategy with complimentary upsell strategies as well.

Advantage: Development!

Increasing App sales with Analytics: Free apps versus trials

Tuesday, June 14th, 2011 by Sebastian Holst

In my previous entry I introduced my app, A Pose for That, explained how I had instrumented my app with Runtime Intelligence to better track user experience and behavior. As a case in point, I illustrated how strategically placing upgrade opportunities in various locations inside my trial version, I was able to increase my conversion rates – perhaps by as much as 50%!

HOWEVER, when I was at MIX11, a very experienced developer (let’s call him David because that’s actually his name) told me that he had already established the optimal app blend to maximize revenue – it was to have a free app (not a trial version) that also offered ways to upgrade to the premium app. He pointed out that trial apps do not show up as free in the marketplace and are therefore almost always overlooked by most casual marketplace browsers. A free app gets the eyeballs that a trial misses.

For those who know me, they know one of my core principles is that my ideas never have to be original, they only have to be good – so is David’s idea really a good one?

To test it out, I created Yoga-pedia, a free app that included the browsing capabilities of A Pose for That with good imagery and instruction, but did not include the pairing of poses to real-world situations (a feature I believe is valuable) or flows (the stringing together of multiple poses). On the welcome page (and one or two other places) I give users a chance to learn more about our software and upgrade; Here is the welcome page and the “tell me more about why I should upgrade page.”

I instrumented the various points where users can upgrade in both the trial and the free app so that I can compare BOTH the usage levels of the two apps AND the upgrade requests that stem from that usage. So… let’s go to the video tape – or better yet, Runtime Intelligence. (Note – these specific graphs are built by extracting the data from the runtime intelligence repository into a spreadsheet and then generating a simply pivot table).

By looking at application starts (not downloads in the marketplace sense of the word), the graph showing App Runs seems to support David’s logic; my free version, Yoga-Pedia, takes off like a rocket and within 24 hours eclipses trial activity in dramatic fashion. …but, it also seems to be cannibalizing trial activity too – Should I care? (NOTE – I am combining usage of multiple applications – not always easy to do with canned dashboards)

I probably should care IF users are more likely to upgrade from A Pose for That trials versus from Yoga-pedia. In other words, are my sales going up because of the free app even though it is depressing my trial volume? Let’s go to Runtime Intelligence one more time…

What the graph above shows is that upgrades from my trials also decreased dramatically with the launch of Yoga-pedia, BUT the volume of upgrade requests from within Yoga-pedia more than made up for that shortfall. (NOTE – I am combining feature usage across multiple applications – not always easy to do with canned dashboards)

In the one week where both the free and the trial versions lived side-by-side, the free version generated 86% of the upgrades.

More important is the bottom line: I saw an 85% increase in the total number of upgrades when I had the combination of both a free and trial version of my app available.

Coming up next (I promise this time) will be a discussion of the last leg of David’s magic formula for success – making your free version ad-driven. What will Runtime Intelligence be able to tell us about that?

Privacy and Security: if it’s your app, then it’s your @$$

Monday, March 14th, 2011 by Sebastian Holst

This is the first of a continuing series to help smartphone app developers make more informed design, development, and policy decisions. The objective is to raise material issues, options, and risks specific to mobile app development (but not to give legal advice).

Specific topics to come include;

  • PII: what definitions are out there and which ones can you not afford to ignore?
  • Opt-in Opt-out: what should your defaults be? How often must you ask? Do you need separate opt-ins per app? Per app version? For regular use AND exception reporting?
  • Data retention and reuse: do you own your own data? Do you need to care about partner data policies or only your own?
  • App hardening: what risks stem from app reverse engineering and/or tampering? How do you know if you should care?

This first entry will delve into the broader motivations behind this series and call out aspects of the mobile app development experience that set it apart from other platforms and markets.

It’s impossible to guarantee that an app will never do harm

Evolving technologies, emerging and divergent regulations, and evolving social and ethical mores have made it is simply impossible to define a concise, bullet-proof set of policies and development patterns that are guaranteed to do no harm to either user or developer. Effective risk management must, by necessity, be a practice governed, to a significant degree, by subjective guidelines.

When god wants to punish you, He answers your prayers

It’s no secret; the convergence of technical, social, economic, and market forces that we call “the smartphone” is a mega-opportunity for those who “get it right.” And what’s greasing the skids speeding this disruptive force of change? Apps of course. Apps are on the front-line hooking consumers and driving the smartphone revolution – pretty cool right? Most definitely.

Swimming in deep waters is always cool, but sometimes it can be deadly too.

The most powerful corporations, the smartest entrepreneurs, and the crème de la crème of investors are racing to meet exploding smartphone demand. And when that much money, information, and power are on the move, criminals (of all kinds), lawyers, regulators, law enforcement, and all genus of government will be right there with us.

These are deep waters indeed.

To serve an app: Is it a cookbook?
Do app developers need to care about this stuff? Consider that when a developer ships their first phone app, they will most likely have already entered into as many as five different (and almost certainly contradictory) binding legal agreements that include assignments of personal liability tied to the developer’s privacy policy and their app’s security. Each tool supplier, marketplace owner, platform provider, ad-server provider, and (last but not least) user demands this assurance.

As an app moves across platforms and international borders, the developer’s legal obligations multiply and, as such, so does their material risk.

To be clear, there is nothing inherently wrong or unreasonable here; but it IS unprecedented.


The list of potential concerns grows considerably when one stops to consider ethical considerations (what’s the right thing to do versus the legal thing) on one hand, and the technical/development security requirements on the other.

In order for a developer to ensure that both their users’ and their own interests are being addressed, application design and development priorities must account for operational and contractual requirements (in addition to market-driven feature functionality of course.)

What level of due diligence and development investment is appropriate?

The diversity of the interested parties and the number of governing agreements all covering essentially the same act (running a single app on a phone) make it especially important that the developer understand:

  • Definitions: For example, do all parties use the same definition for PII (personally identifiable information) and do they use it consistently (when defining the developer’s obligations versus their own)?
  • Obligations: For example, are you agreeing (and indemnifying the other party) to adhere to multi-national regulations – many of which you will have no knowledge of?
  • Rights and privileges: For example, what usage and commercial rights to application usage, user behavior, and other data are conveyed? Are these subject to change?
  • Notifications: Under what conditions, through what channels, and in what timeframes must which information be communicated? Does the developer have different special obligations?

Knowledge management versus risk management
Most people divide the world into things they know and things they don’t and then try to manage their risk within that “circle of knowledge.”


Knowledge management is not risk management and often misses a danger-zone where risk most often hides; in “the things we don’t know that we don’t know” and “the things we think we know, but we don’t.”

If you “don’t know you don’t know”, then you miss the chance to educate yourself or ask for expert advice.

Similarly, if you think you know something, but you’re wrong, you may find yourself exposed or missing an opportunity.

The single-minded objective of this series will be to shrink the “danger zone” for mobile app developers; to make sure that you don’t get bitten by security, regulatory, or social gotcha’s that you didn’t even know were out there.

The take away from today’s installment is a very simple one; if it’s your app that gets jammed up – rest assured; it’s going to be your @$$ on the line.

Build a Windows Phone 7 app and secure a $100 donation for your local user group

Friday, February 18th, 2011 by Sebastian Holst

Grow Your WP7 Dev Skills

The only thing better than doing something good is when it also helps you do well – and that is the spirit behind a joint initiative between PreEmptive Solutions, Telerik, and Microsoft.

What’s the deal? Simply put, any developer who develops their own Windows Phone app using both Telerik controls and Runtime Intelligence will:

  • Have $100 donated to their local user group (that’s doing good), AND
  • Have an app on the marketplace that combines a great user experience with the usage and adoption analytics to prove (and improve) it (that’s doing well).

Of course, nothing can be this simple right? WRONG! All you need to do is:

  • Develop and deploy any WP7 application using Telerik RadControls for Windows Phone AND PreEmptive Runtime Intelligence for Windows Phone, AND
  • Get this done and certified before May 31st 2011

By the way, there is no limit to how many apps you and your user group can submit, but we will only recognize the first 500 qualifying applications – so build early and build often!

One more important point - developers will also have free access to all of the required software. So go on – click through and do some good at http://www.telerik.com/windowsphoneFTW.

Survey sez!

Wednesday, February 16th, 2011 by Sebastian Holst

We are deep into two WP7 dev surveys, but I just love this stuff and so I’m going to leak some preliminary data out now – if you want me to send you a link to the final results, send me an email at sebastian at preemptive dot com.

The first survey we are doing goes back to the earliest group of WP7 developers, “first 300,” and asks how they have made out over the past two months.

The second survey targets roughly another 1,500 active wp7 devs who are using analytics and/or obfuscation but who began their work after the initial survey.

Here are a few tidbits that we see so far…

Analytics improves mobile development practices and app value

Going back to the original 300 and asking those who had deployed apps with analytics, we have found that:

  • 82% of the devs say that using Runtime Intelligence (RI) has “helped them to establish for themselves the value of analytics for app development overall”

With regards to their specific WP7 app in the marketplace, the devs directly credited their use of RI as:

  • Increasing the value of their app (45%)
  • Improving their app’s user experience (36%)
  • Improving their app’s quality (27%)

Mobile app devs migrating to WP7 4X’s faster than .NET devs

In the first 300, only 4% of the registered developers targeted two or more additional mobile platforms. This indicated to me at least that the very first developers to develop for WP7 were already MSFT devotees versus serious mobile app developers exploring WP7 as an alternative/incremental mobile platform.

However, in the latest wave of developers, that stat has more than quadrupled. There is no question that developers who identify themselves as mobile app developers first rather than iOS, .NET, or Android developers are building for WP7.

Platform share for those developers targeting multiple mobile platforms are (in addition to WP7):

  • iOS 69%
  • Android 67%
  • RIM 23%
  • Symbian 5%

There’s a lot more to come – so stay tuned (or shoot me an email). Cheers!