Archive for the ‘Runtime Intelligence’ Category

Security and privacy concerns identified as most common obstacle to implementing application analytics

Monday, October 8th, 2012 by Sebastian Holst

This is the first installment of a series posts on the state of application analytics and modern application development patterns and practices.

In a recent survey that includes responses from 100’s of development organizations, two thirds identified application analytics as either essential or important in one or more of the following categories: Product planning, Development prioritization, Test plan definition, Customer support, and/or Development ROI calculation.

Among this group where application analytics has the greatest impact, the following were identified as the most serious obstacles to implementation. (click to enlarge graphic)

Obstacles preventing the use of application analytics in my organization

Half of all respondents identified security and privacy a 20% higher response rate than the next two closest obstacles e.g. lack of expertise and general quality concerns).

The emphasis on security and privacy is even more pronounced inside larger development teams. Nearly 3 out of every 4 development organization with greater than 50 identified privacy and security as an impediment – 50% more likely than development teams between 5 and 15.

Correlating perceived obstacles to implementing analytics with development organization size

In fact, an organization’s size appears to have a significant influence on virtually every perceived obstacle; larger organizations appear to be more concerned with performance, quality and connectivity while smaller organizations struggle with awareness of analytics solutions, development best practices, and the required integration of their development and operations processes.

One might make the generalization that, due to the complexities that come with size, larger organizations have had to move to more tightly integrated platforms and practices – putting them in a better position to implement application analytics (and so they focus on potential risks stemming from an implementation) whereas smaller teams may not have as an entrenched “feedback-driven” integrated approach to development. As such, they are more likely to struggle with how to move forward (keep in mind that all respondents identified application analytics as either essential or important).

Privacy and Security and PreEmptive Analytics

Regardless of development team size, privacy and security is the number one perceived obstacle – and PreEmptive Analytics is unique in its approach to this critical requirement. PreEmptive Analytics includes the following:

  • Development teams own their own data. PreEmptive asks for no rights to aggregate, inspect or resell your data.
  • A two-level opt-in switch is included ensuring user opt-in to transmit runtime data from both regular usage AND application exceptions. The logic itself can be injected post-build for .Net and Java and can always be defined by the development organization.
  • All data is, by default, encrypted on the wire.
  • Tamper-detection and defense can be used to detect and defend against any attempt to alter or redirect runtime data transmission.
  • Obfuscation can be used to obscure inspection by third parties of what is being collected and transmitted.
  • Unique keys identify both the organization and the application source for data.

For more information on how PreEmptive Analytics addresses the number one obstacle for implementing application analytics (as ranked by those that need it the most), visit www.preemptive.com/pa.

The Forward-Looking Development Organization

Thursday, May 3rd, 2012 by Gabriel Torok

I am giving the keynote at the VSLive conference this month in New York City titled the “Developer of the Future”. As we all know, the software development industry is constantly evolving. While it may be difficult to see those changes day to day, the effects are very apparent from year to year, and what makes a developer or development project successful certainly changes as a result. So, making predictions is always risky and prone to a notoriously low rate of success.

I could focus on the near term and state obvious trends — the demand for developers will not subside. Or, I could stretch far into the future (where no one can prove me wrong anyway) and talk about how “Software will eventually write Software.” Instead, I’m going try to find middle ground – take some risks with my predictions and make those predictions near enough that someone might actually hold me accountable in the coming years.

Applications that just a couple of years ago were available only on a single desktop platform are now available in the cloud or as apps that run on one or more phones/tablets. This trend clearly indicates where the software development market is heading — make the same application available on a multitude of platforms (desktop, phone, tablet, web, etc.). For developers, this means that a new type of expert will soon emerge. Let’s call him or her the “general expert”. The general expert will be valued over the many “narrow experts” that we have today. In the past, being a narrow expert in a single stack such as Windows or SQL Server was seen as a positive. Going forward though, developers will need to navigate across multiple languages and technology platforms to launch a single project.

Moving from the developer to the development team, teams will no longer be able to afford to produce big bang roll-outs and will need to reply upon continuous deployment workflows. Today, services already lack the familiar concept of a version number. What version of gmail are you using? It won’t be long before most applications will also lose their “version” identity. Users will simply expect to be running the latest version of whatever may be on (or connected) to their device. If you use Chrome today, I bet you don’t know what version you have – Chrome is already silently updating itself.

Customer expectations are higher than ever before and their attention spans are shorter. Developers that don’t have an understanding of what drives customer behavior, especially those who are trying to sell apps via marketplaces, will be at a huge disadvantage. Customers eat with their eyes; applications must be visually appealing and dead simple to use.

In any design project you’ll make some good decisions and some bad ones, but you won’t know for certain which is which until you find out from your customers. Rather than try to speak with all of your customers (they won’t always tell you the truth anyway), the use of Application Analytics or Telemetry will be an increasingly important aspect of the software development process. Applications should report back and make “actionable” the production incidents that cause unhappy users. They should also make generic usage information actionable as well (adhering to all privacy concerns of course) to allow for continual improvement based on real world usage and split testing. This is important whether you are building a mobile app, a desktop client application or a mission critical server application.

In summary, in addition to domain expertise, it is more important now than ever to have a broad understanding of platforms and technologies. Also, while understanding of technology is critical, knowledge of your customers, how they use your applications, and what’s important to them about the apps are just as critical. There’s no time like the present (don’t wait for the world to change around you). Get a head start by implementing feedback mechanisms inside your apps today, and make feedback driven development a reality, rather than a vision of what could be.

For more information on how PreEmptive Solutions is helping development today, see: http://www.preemptive.com/pa

The Microsoft sponsored service for WP7 ends – the PreEmptive sponsored service debuts

Tuesday, November 29th, 2011 by Sebastian Holst

At 24:00 EST on December 9, 2011 the Microsoft sponsored protection and analytics service for Windows Phone 7 will be shut-off.

A different service fully and solely subsidized by PreEmptive Solutions will take its place. This service is materially different – please read the following notice carefully for information on how to continue to work with PreEmptive Solutions technology for Windows Phone.

Background

While Microsoft’s sponsorship expired on September 30, 2011, PreEmptive continued the service for an additional 60 days at our own expense while we explored a variety of options to continue our support for the Windows Phone development community. Microsoft’s sponsored service has ended, but our commitment and support for this community continues unabated.

PreEmptive’s challenge was to find an affordable means to support for the burgeoning WP7 development community without compromising the quality and capabilities unique to our protection and application analytics technologies; we believe the following provides both a valuable set of services at little and no cost for small WP7 development efforts with a smooth “on-ramp” for larger development projects and for organizations with more demanding service levels, governance or scalability requirements.

Summary

Obfuscation and Instrumentation continues at no cost through 12/31/2012.

Dotfuscator for Windows Phone, the post-compile tool that obfuscates and injects application instrumentation will continue to be offered to Windows Phone 7 developers at no cost through December 31, 2012.

Mobile analytics endpoint (wp7.runtimeintelligence.com) will be shut-off on 12/9/2011

The current analytics endpoint will be discontinued. However, developers have a number of options that they can consider;

· Subscribe to the PreEmptive Solutions commercial endpoint. This is a fee-based option that includes all of the features currently offered PLUS an advanced mobile portal, a RESTful API, and a higher service level (plus support beyond WP7). For more information, email sales@preemptive.com.

· License PreEmptive Analytics for TFS. This is also a fee-based option. This solution is an on-premises solution focused on exceptions rather than feature tracking. For more information, see Using Analytics for Windows Phone and Azure Exception Tracking - User Community Virtual Series and email sales@preemptive.com.

· Develop and host a homegrown endpoint. This is a no-fee option but development will be required. The CodePlex Runtime Intelligence Endpoint Starter Kit repository starter kit project may be of some help.

· Plan to migrate to the PreEmptive Analytics for TFS community edition to be included with Dev-11. This is a no-fee option but is NOT yet generally available from Microsoft. For more information, see the video A Lap Around PreEmptive Analytics for TFS with Justin Marks .

· Publish their app as a CodePlex project and utilize the CodePlex analytics endpoint (this is different than the option above). This is a no-fee option. For more information, see this tutorial (note that this assumes the developer is limited to the Community Edition of Dotfuscator – but the WP7 edition has full functionality).

The following feature summary table highlights the three principle options available to the Windows Phone 7 development community with a comparison to the discontinued Microsoft sponsored service. (click thumbnail to enlarge)

Click to enlarge

Click to enlarge

FAQ

Q: Will developers have to republish my WP7 app on or before December 9, 2011?

No

Q: Will users notice any difference in app behavior after December 9, 2011?

No

Q: Will I have to re-register my installation of Dotfuscator for Windows Phone 7?

No

Q: If I have only been using Dotfuscator for obfuscation, will I lose any functionality or will I have to do anything differently?

No

Q: Will developers have access to earlier runtime data generated by Runtime Intelligence for Windows Phone after December 9, 2011?

They will not.

Q: Where can developers ask additional questions regarding migration, upgrades or discontinuing use of PreEmptive Solutions technologies?

Post to the PreEmptive forum at http://www.preemptive.com/forum/index.php?f=26&sid=dfe90c2ba80de07692372dae962c58b2&rb_v=viewforum

PLEASE NOTE – this is NOT a moderated forum.

Q: Why would a development organization upgrade to a professional SKU of PreEmptive Analytics?

· Multi-platform (WP7, Android, JavaScript, all .NET and Java, native API…)

· Private endpoint (for large-scale enterprises with demanding scalability, governance, or other unique requirements).

· Analytics for TFS option (out-of-the-box integration with Microsoft Team Foundation Server).

· True application analytics such as custom data fields, development ownership of data, true SLA and support for developers, etc.

Conclusion

Of course we would have preferred that Microsoft had opted to extend their sponsorship for Runtime Intelligence for Windows Phone, but that was not the decision that they ultimately made. However, over the past 12 months, we had a front row seat watching a flood of innovative apps launch. We know that a good percentage of the WP7 development community relied upon PreEmptive Solutions for both analytics and protection (in a recent analysis of the marketplace, it was shown that 17% of all apps used either protection, analytics or both).

This experience combined with our confidence in the future of the Windows Phone platform has prompted us to extend free access to Dotfuscator for Windows Phone. We look forward to our continued support and participation in the growth and success of this exciting technology and marketplace.

60 days – déjà vu all over again?

Monday, October 10th, 2011 by Sebastian Holst

Hi all – today I sent out a notice to registered Runtime Intelligence for Windows Phone users letting them know that a change would be coming to our service on (or perhaps after) December 9th. I have already received messages from some rather annoyed developers who feel like we are playing some kind of Machiavellian pricing game; the basic flaw in this view is that it presumes we sit in the Prince’s chair – which we do not.

First, let me start by stating categorically that we remain excited and committed to the Windows Phone platform.

Our approach to analytics has always been squarely focused on development organizations rather than on marketing; this is not a “spin,” rather, this approach informs and influences our product’s architecture and business model.

I’ll drill into this distinction in a moment, but first, I want to point out that it is this distinction coupled with our belief in WP7 as a platform and the value of our approach to application analytics in general that led us to invest in building out the WP7-specific service. While I cannot speak to Microsoft’s motivations (in any context, let alone this one), it is public knowledge that they funded this service so that it could be offered at no charge to the WP7 dev community (but now that’s no longer the case).

Why was this even necessary? Why couldn’t PreEmptive just give away our analytics and protection service like Google does?

“Marketing-centric” analytic services make their money off of your data – but first they have to cover their costs; the “back end” or analytics portal is where the majority of that “cost” sits. Today, Google or Flurry (or any other similar service) each rely upon web and other mobile platforms to generate the data volume that ultimately funds their backend service. At this early stage, WP7 app data can fund client API’s (perhaps) but is nowhere near the volume-levels required to pay for an entire service.

But there’s more – focusing exclusively on resalable content (and by extension, divesting from any requirements that do not) have profound feature and architecture implications.

Given a marketing-centric focus, you can readily understand why these solutions all share the following traits:

· Custom data fields and data types are limited in scope and volume(unique/heterogeneous data points cannot be easily rolled-up, sold or used to better target users)

· No built-in enforcement for opt-out policies – this is left entirely to the developer (if you do not provide user data to these services, you are of no value)

· No investment in analytics of software away from the presentation layer(this data has no relevance to advertising, profiling etc.)

· Software-specific data is only marginally supported – unhandled exceptions can be captured, caught and thrown exceptions are out of scope *same as above – does not contribute to monetization model.

· They own your data (privacy rules notwithstanding – the have full rights to monetize your data in any way they see fit). This is the entire reason they are in business – if they don’t own the data, how can they make money?

· They can provide their analytics software for free – since they have built a monetization strategy based on your data and do not invest in any development that does not directly contribute to that strategy, they can afford to give you the picks and shovels for free in exchange for the gold you produce for them (that’s big of them don’t you think?)

Conversely, focusing on application stakeholders as we do, we invest in features that do not contribute to a roll-up across companies and apps to monetize user preferences – in fact, some of these features can materially impede that objective – yet they are equally as important for developers who want to build better, faster and more effective software…

For example:

· Supporting custom data (app-specific fields and states) and data types (other than strings)

· Built-in exception tracking (unhandled, caught and thrown) to not only send you stack traces but provide insight into user experience and runtime environments (hostile or otherwise)

· Built-in opt-out policy enforcement (local, regional, cultural, legal and regulatory rules are so complex – serious development organizations need to centrally manage and reliably enforce compliance – could you sell a gun without a safety?!)

· Analytics away from the presentation layer (want to know how your Azure or other distributed services are performing and behaving?)

· AND LAST BUT NOT LEAST, you own your data. (given the heterogeneity and specificity of each application’s data – it’s value is exponentially higher to development stakeholders and proportionately lower in aggregate)

So what’s the takeaway?

With Microsoft, we invested in this mobile platform because we believed (and now we know) that it has significant value.

Given our focus on development rather than advertising, we have to innovate on both the technical and business ends.

We had hoped that our previous relationship with MSFT Windows Phone division had lasted a bit longer, but it is what it is.

We have a number of very interesting scenarios in mind that we think will prove to be both exciting technically and innovative from a business perspective – but we are not ready to discuss that yet (as soon as we can – we will).

If you don’t care about these added value features and don’t mind the strings that come with web/mobile analytics services like Google – then you should certainly use them. They are not flawed – they are just fundamentally different.

If you value the services we have been offering or have ideas on how we can improve them even further – PLEASE LET US KNOW…

Registered users will be getting a survey in the next 24-48 hours – please let us know what you think and what’s working (and not) for you.

As I’ve already said, we remain excited and committed to the Windows Phone platform.

(now i have to break to prepare my wp7 training materials; i’ve got a crew from my son’s high school writing WP7 apps for independent study credit)

Runtime Intelligence and Dotfuscator for WP7 developers speak (Mikey likes it!)

Monday, September 26th, 2011 by Sebastian Holst

In my last post, I drew a correlation between apps that used Runtime Intelligence and their relative (positive) success as measured by user ratings and engagement. While it was fairly clear that developers who chose to use Runtime Intelligence built more successful apps than their counterparts, it really said nothing about a) Runtime Intelligence analytics’ contribution to their success or b) developer satisfaction with Runtime Intelligence overall.

Well, there’s really only one way to answer these questions …and that’s to ASK THEM.

I sent out an electronic survey starting on Monday of last week (September 19) and have received over 200 developer responses. Here is what they said…

Who is doing what?

  • 32% indicated that they are only using Dotfuscator to protect their application
  • 24.5% said they were only using Runtime Intelligence
  • And 43.4% indicated that they were using both.

Do these “smarter than the average bear” developers see the value?

In a word, YES.

Looking only at those developers who indicated that they already had their applications in the marketplace (representing over 100 development organizations):

  • 60% indicated that analytics set the wp7 platform apart from all other platforms or added significant value to the platform.
  • 68% indicated that protection set the wp7 platform apart from all other platforms or added significant value to the platform.

Analytics’ perceived value increases by 450% with developer experience

When looking at those developers that indicated that analytics and/or protection “set the WP7 development platform apart from all others,” analytics’ value actually increased by 450% (from 2% to 9%) as developers moved from no app, to less than four weeks to a ship date, to actually having an app in the marketplace (and getting analytics back). Interestingly, obfuscation (protection) peaked in value just prior to shipping.

So what’s the takeaway?

In my last post, we established the user of Runtime Intelligence were more successful than other WP7 developers. In this post, we see that these developers credit their success, to some material degree, to either Runtime Intelligence or Dotfuscator protection (or both).

In their own words
(selected - but unedited - responses to the open-ended question “what are you most excited about?”)

I like being able to get crash reports without much additional work.

It gives the developer ability to know about usage patterns in an application. Obviously code obfuscation is a necessity, especially for paid apps.

It offers a unique way to see how users interact with the application, and with the latest release it also has error reporting. Awesome!

I’m an excitable person.

Fabulous data provided by RIS to analyze the performance, usage and app demographics.

So I can know what is happening in my app and protect my code.

Used correctly, the analytics really let me see how and by whom my application is being used. I get more insight into this information than I could if I set up a usability lab or just did extensive user testing. There is no better way to observe than to do so in production.

The concept of attaching runtime analytics after the compilation process is very useful for us (standard software development, single application in various customer-specific configurations), since we are able to attach this on a per-customer basis and don’t have to manage it in code.

UI for parameterization

It gives detailed statistics about the usage of all parts of the software and helps to recognize the hot features of the software are and which parts are less used. This adds great value into the effort of making software better.

Really gives me insight into what my customers are doing with my applications. They help me to understand where I can enhance functionality and add value.

Quality of product

Analytics give me an idea on what I should work on next to improve my application

Runtime analytics is cool because there is no code to write.

I can collect the exact information i need.

It allows me to phase out or strengthen certain parts of my apps. I currently have seven apps and the instrumentation is crucial.

Because i can have a deep analysis of when and especially how my application is being used. If you add the fact that all these data are aggregated and presented in such a nice way by the portal, you end up with a great product

I produce libraries (DLL’s) that are handed to third parties, hence the need for obfuscation.

Kickass obfuscator.

It helps me keep track of any bugs. And it allows feature tracking. And it gives me the cool world map that shows where some of the users are.