Hello everyone (cc: @Sandsquad @DrMetaverso @LastYearTryer )
Thank you for your questions. I understand your concerns. I cannot speak with certainty on the previous Buildersā Challenges and causes for any data issues occurring in these as I was not directly managing the events. However, I am happy to provide more insights on the challenges we faced during Buildersā Challenge 3!
Analytics Updates #1 and #2
These addressed parts of the same issue where one of our data flows had an error, blocking the data from coming through. This affected Single Player experiences more, as we have more data streams in Multi Player. The fixes released the data. No data was lost as it was a simple matter of āremoving the blockage,ā causing all data to come through.
Update #3: The issue
This was an issue on product side. An undiscovered error in Game Client 0.11 resulted in an event not sending during a play session. This event is used to track playtime and thus caused many experiences to have this metric reduced or, in some cases, reaching 0. The impact depended on the amount other events and triggers we received. A GC fix was issued to restore the necessary event. This would only affect sessions occurring after the fix, however. This should not have happened, and we have made sure that these event triggers are tested and verified more effectively in future updates.
The Fix: This was more complex than removing a blockage. Apart from the GC fix, we needed to recover past data following an alternative method from our usual workflow. The data was recovered, but unfortunately we could not push this to your analytics since this page follows the same usual method across all experiences, and changing this would create more problems.
Update #4
A data quality error was discovered where session time was incorrectly recorded as 0 and our session-formation logic created multiple records of what was, in fact, only a single session. In many cases, this unfortunately inflated the session counts which, in turn, also inflated the playtime in some cases as multiple records of the same session occurred.
We realise how important it is for creators to rely on their analytics to analyse their performance and plan ahead. While I canāt speak on past BC issues, I know that weāve learned and will take measures to predict, prepare, and test (both product and data) to avoid this in future events to the best of our ability.
@DrMetaverso: āI canāt help but feel that data accuracy and rewards arenāt being prioritized enough ā¦ If we donāt know how our metrics are being collected and judged, how are we supposed to trust that the process is fairā
Your metrics arenāt judged, what you are seeing is just data. However, regretfully this data was subject to errors. I would like to note that the effects on rewards has been minimal. The most prominent effects have been positive for those who were heavily affected.
In regard to worries relating to rewards, this discussion would be most productive if left until after distribution so the results can be taken into account. We feel positively about the balance achieved using our new system.
I hope these insights are helpful in understanding the challenges we faced and the actions we mean to take to prevent future occurrences.
We will notify once rewards are sent!
