The analytics team used Jupyter notebooks and product analytics tool, to gain insights from the user interactions on their mobile application. They had to pre-define events between the application and users on the product analytics tool. This limited their ability to conduct exploratory analysis. Additionally, these tools would consume a lot of time from analysts. The data from these tools would sometimes need to be transformed and enriched, so a solution that could extract and transform the data was required.
In order to get data for analysis and reporting, the teams would directly query the production database. It was observed that many users across different teams would need similar data, only with slight modifications. The multiple requests would increase the load of the database. Thus, they wanted to build a separate database for analytics purposes.
The team wanted to house data from all the sources into a data warehouse and use it for their use cases. The motive to build the data pipeline
incorporating data warehouse was also to establish a single source of truth for all the data. On top of that the data pipelines, reports and dashboards needed to be automated such that it does not require any intervention. The data team at Brightmoney was seeking a solution to meet all these requirements.