In the world of accounts receivables, digital collections and constant experimentation, data analytics is crucial to improving business processes and making informed decisions. Our organization faced challenges in managing numerous with varying KPIs across different channels.
Through this process we started by tackling experiments within our digital collection efforts—assessing KPIs which ranged from open rates, click through rates (CTR), hit rates, payment rates and more. We aimed to create a comprehensive and easy-to-understand system to track, visualize, and analyze all our experiments in one place.
Our goal was to take this same framework and replicate it across organizational experiments. This case study demonstrates how we leveraged ChatGPT to build a powerful and scalable tool for our experimentation efforts.
Our Problem:
Our organization conducts various experiments across multiple channels, each evaluating thousands of data points and utilizing different KPIs.
Additionally, the concepts of statistics and the ‘power’ of an experiment can be difficult for many to grasp. It’s critical to be able to manage experiments while communicating their results and consequences effectively completely.
Our Objectives:
- To seamlessly track experiments across multiple channels
- Provide visualization throughout the experimentation process, displaying various confidence levels and results in an easily digestible manner
- Create a framework which tracks the full experimentation process while providing key metrics (time to completion, estimated sample size, Bayesian vs. frequentist experimentation)
- Build a foundation for tracking dozens of experiments with numerous KPIs in one place. This enables relevant subject matter experts to fully analyze data and adjust inputs for confidence levels, expectations, and other metrics as needed. This also allows users without a statistical background to understand and monitor progress.
Our Process:
- We identified the limitations of our previous MySQL & Excel-based experiment monitoring framework, which was time-consuming, labor-intensive, and lacked advanced features.
- We leveraged our conceptual knowledge of both statistics and our business to iteratively feed coding requests into ChatGPT, starting with basic statistical techniques and gradually adding more complexity until we had a robust set of outputs that cleanly presented and interpreted our experiment status for a wide range of users.
Example of a simple exchange between a user and ChatGPT to codify a business process.
Our Results and Takeaways:
- Our new system provides clear visualizations, experiment progress tracking, and KPI monitoring.
Posterior distributions and 90% credible intervals for open rates in a Bayesian A/B test. The non-overlapping intervals indicate a significant difference between the test and control groups, suggesting the test group is superior. These distributions are obtained using a Bayesian simulation, which incorporates prior knowledge and observed data to update our beliefs about the effectiveness of each treatment.
- We created a centralized dashboard to manage existing experiments and easily accommodate new experimentation efforts.
Output from our experiment dashboard showing sample size calculations for frequentist A/B testing. Although Bayesian inference doesn’t strictly require sample size calculations, we include them as a reference to help guide our experimental design and decision-making process.
- The dashboard allows stakeholders and subject matter experts to assess performance on an ongoing basis, enabling data-driven decision-making.
- We achieved significant time savings——by streamlining the experimentation management process.
Our Conclusion:
By harnessing the power of ChatGPT, we developed a comprehensive and efficient tool to manage and analyze our organization-wide experimentation efforts in the accounts receivables space.
This innovative solution has not only saved us time but has also improved the overall quality of our decision-making, ultimately leading to better business outcomes.
