Proposal: Simply mobile app analytics

[barbeau]

Current design of OBA iOS and Android analytics is discussed here:
https://github.com/OneBusAway/onebusaway/wiki/Tracking-and-Analytics

@cagryInside will post a proposal here to simply the "session" analytics that are logged when the apps start up, as discussed in OneBusAway/onebusaway-android#237 (comment).

Tagging @bbodenmiller and @aaronbrethorst so they can be part of the discussion.

[cagryInside]

Proposal for simplifying the session reporting:

Reporting the OBA session discussed in OneBusAway/onebusaway-android#237 (comment). Also, following table shows the currently reported session events when app starts.

Event Category	Event Action	Event Label	When?
app_settings	configured_region	API Region	App starts
app_settings	general	Set Region Automatically	App starts
app_settings	general	Show Experimental Regions	App starts
app_settings	configured_region_auto	Set Region Automatically: %@; Old Region: %@	Auto region config

However, these reported events cause huge data in the Google analytics admin as shown in the image below:

Since, we are already reporting the API region as a secondary dimension, we don't need to report the API Region itself. Hence, we can simplify these events as shown in below:

Event Category	Event Action	Event Label	When?
app_settings	startup	Conf: [Set Region Automatically: YES or NO, Show Experimental Regions: YES or NO, ...]	App starts

Furthermore, application preference change events logged under ui_action:

Event Category	Event Action	Event Label
ui_action	button_press	Turned on Experimental Regions
ui_action	button_press	Turned off Experimental Regions
ui_action	button_press	Set Region Automatically
ui_action	button_press	Set region manually

To better address and simplify these events, we might change the event category and rearrange these events as shown in below:

Event Category	Event Action	Event Label
app_settings	general	Set Experimental Regions: YES or NO
app_settings	general	Set Region Automatically: YES or NO

[bbodenmiller]

Thanks @cagryInside for the ideas.

If we drop region reporting at the startup of each session will we still be able to tell how many sessions each region had each day?

If we combine the current settings reporting I believe that will make it much harder to tell how many sessions are running with specific settings. For example, how would we build a report showing number of sessions today with experimental regions enabled.

Renaming the recording of set setting events will result in a lose of the current recording, is there enough benefit to warrent the change vs. just improving the documentation around what events mean what?

[cagryInside]

@bbodenmiller thanks, here is my comments:

If we drop region reporting at the startup of each session will we still be able to tell how many sessions each region had each day?

If we push single event, we still be able to tell how many sessions each region had each day:

Event Category	Event Action	Event Label	When?
app_settings	startup	Conf: [Set Region Automatically: YES or NO, Show Experimental Regions: YES or NO, ...]	App starts

If we combine the current settings reporting I believe that will make it much harder to tell how many sessions are running with specific settings. For example, how would we build a report showing number of sessions today with experimental regions enabled.

I don't think it's so hard. For example,
First, choose the event category and event action in the GA admin:

Then, basically search whatever attribute you want to see in the GA admin:

Renaming the recording of set setting events will result in a lose of the current recording, is there enough benefit to warrent the change vs. just improving the documentation around what events mean what?

Yes, I agree that after these possible changes, we will lose the current recording. However, the current implementation looks like these in the GA console:

Since all set setting events logged under ui_event, its hard to distinguish which events are changing the app settings and which events are regular ui event. I though it would be better to have separate events for settings.

[bbodenmiller]

It looks like your proposal will work if we are willing to do manually data collection. I think either way we can drop configured_region reporting since it is a custom dimension.

If we want to be able to use the Google Analytics Dashboards however I haven't found a way to make that work for the combined settings status event. Additionally the dashboard allows us to do some custom labeling so that, along with good documentation, could avoid confusion without resulting in loss of existing analytics data.

However we setup the analytics I think it'd be good to have some easy to use dashboards where we can pull relevant info similar to the following:

[barbeau]

+1 for @cagryInside's proposal, and +1 for good dashboards

Additionally the dashboard allows us to do some custom labeling so that, along with good documentation, could avoid confusion without resulting in loss of existing analytics data.

To my understanding losing this existing data mostly isn't an issue, because the data already captured for iOS for the region changing "session" variables is already heavily polluted with bad data due to these events being reported in applicationWillResignActive, which records every time the user starts the app, hits the home button, or gets a phone call (i.e., OneBusAway/onebusaway-iphone#376). So the current data really isn't meaningful, and starting over with new labels is really the only way to make sense of it (other than always running reports for the data after which the above issue is fixed on iOS).

@cagryInside am I right in this regard?

[cagryInside]

To my understanding losing this existing data mostly isn't an issue, because the data already captured for iOS for the region changing "session" variables is already heavily polluted with bad data due to these events being reported in applicationWillResignActive, which records every time the user starts the app, hits the home button, or gets a phone call (i.e., OneBusAway/onebusaway-iphone#376). So the current data really isn't meaningful, and starting over with new labels is really the only way to make sense of it (other than always running reports for the data after which the above issue is fixed on iOS).

I think, existing data is not an issue with session events. However, there is a tradeoff between old and proposed configurations. Old one enables better dashboard usage (#5 (comment)) and the proposed version reduces the size of the reported data (#5 (comment)) and simplifies the event labels.
PS: I also haven't found a way to make the proposed events work with better dashboards. Perhaps, there is a way to make it work.

[bbodenmiller]

From my understanding the only bad data is the region reporting which I agree we can remove since we have a custom dimension already with that data.

[barbeau]

@cagryInside Can you please do a little more digging to see if you can set up custom reports, or another method, that could enable better dashboards? IMO not being able to create good dashboards would be the only thing blocking this proposal.

[bbodenmiller]

I've done some further research and have some suggestions. I believe that rather than track users current settings using events at app startup (which results in over counting, etc.) we should be using Custom Dimensions. This allows us to see the number of users running specific settings over a period of time, allows us to see which settings are most popular by region, allows us to see if users in experimental regions are getting the most crashes, etc. "Dimensions allow you to break down a metric by a particular value, like screen views by screen name. Dimensions correspond to the rows in a report."

Custom Dimension	Index	Values
API Region	1	API Region or "Custom URL" if custom API set
Set Region Automatically	2	YES or NO
Show Experimental Regions	3	YES or NO

This change would result in a data loss of custom API counts and Experimental Region counts on iOS but it turns out we already are using a custom dimension for the API Region & Set Region Automatically setting so those counts would remain. Increased Contrast is another potential candidate to be a custom dimension. Note GA limits us to 20 custom dimensions.

I believe we should keep the following as is:

Event Category	Event Action	Event Label	When?
app_settings	startup	Startup View: view loaded at startup	App start
app_settings	configured_region_auto	Set Region Automatically: new region; Old Region: old region	Auto region config

On iOS the Startup View (which is saved so the app always resumes to the same tab) is only recorded when the app starts fresh not every time the app is resumed and I'd recommend similar implementation on Android.

In regards to the other suggested changes around logging when a user changes their settings I believe we can address that through better documentation and dashboards rather than requiring a change in the logging structure and data loss. If you two as well as @aaronbrethorst & possibly @caitbonnar agree that the benefits outweigh the data loss then I'm fine with us changing it. Here is an example dashboard which I have setup on the tracker under Dashboards > Shared > Setting Changes:

I also added some details to and simplified the Tracking and Analytics page. It'd be helpful if someone could start populating the events section with a table of the exact events currently being logged by platform so we can identify where gaps exist or tweaks need to be made as a result of logging slightly different events (e.g. iOS and Android have same event name but log different actions). One could start putting that together by exporting data from GA using the following and the export to CSV option.

[cagryInside]

Can you please do a little more digging to see if you can set up custom reports, or another method, that could enable better dashboards? IMO not being able to create good dashboards would be the only thing blocking this proposal.

@barbeau We can show also the proposed config as following:

[bbodenmiller]

@barbeau @cagryInside any further comments on my proposed changes in #5 (comment) or progress on tracking what events are tracked by platform?

/cc @aaronbrethorst

[bbodenmiller]

FYI @aaronbrethorst has added another custom dimension in the OBA iOS app per my suggestion in OneBusAway/onebusaway-iphone#405.

Custom Dimension	Index	Values
VoiceOver	4	VoiceOver: ON or not set

I might look in to changing it to the following just to make sure we know if any events are picked up without the dimension set. Update: PR at OneBusAway/onebusaway-iphone#407

Custom Dimension	Index	Values
VoiceOver	4	ON or OFF

If Android has a similar feature this custom dimension can be renamed (in the GA dashboard without an app update) then reused for Android.