A Homegrown Retail Giant Focused on Greater Efficiency

‍

Founded in Greenville, Michigan in 1934, Meijer has grown from a single, family-operated store to a chain of over 240 supercenters, and is known for pioneering the one-stop-shop concept. Over the course of its expansion, Meijer has consistently focused on providing high quality products, ensuring a family-like shopping experience, and evolving to prioritize customer needs. Becoming a leader in the retail industry, however, came with a substantial data processing footprint in order to support the vast network of stores and customers. Faced with escalating operational expenses, Meijer sought to transform its cost monitoring and optimization practices.

‍

Optimizing Costs and Increasing Data Observability

‍

The Meijer team was experiencing challenges managing and optimizing their reporting pipeline costs utilizing Azure Databricks. The overwhelming amount of data and difficulty in pinpointing cost drivers led to inefficiencies across Databricks clusters, jobs, and notebooks. Existing tools were insufficient for providing meaningful insights on cost analysis, proactive monitoring, and alerting capabilities. With over $4 million in annual Azure Databricks spend, Meijer needed a scalable, data-driven solution to gain better visibility and control over costs.

‍

Developing a Dashboard for Prediction and Optimization

‍

RightBrain Networks saw an opportunity to clarify Meijer’s costs and reduce inefficiencies through a comprehensive, four-part approach:

‍

• Enhanced Cost Monitoring Dashboard: The initial phase of the project was dedicated to building the foundation for in-depth cost analysis, using existing data from Databricks Overwatch to support initial recommendations. The ECMD was designed to provide a summarized view and breakdown of costs for clusters, jobs, and notebooks, offering teams improved visibility into their Databricks usage and spending patterns. 
  ‍
• Cost Optimization Recommendations: Armed with insights from the ECMD, RightBrain identified actionable steps for reducing costs, such as transitioning jobs to scheduled batch processing and optimizing cluster configurations. This helped RightBrain build the remaining tools, which were designed to help Meijer identify further possibilities for improvement as they grow.
  ‍
• Automated Alerts: Long-running jobs were causing a significant increase in costs and inefficiencies, so RightBrain implemented real-time alerting for cost anomalies and lengthy jobs. This system enables Meijer to monitor issues as they happen, ensuring optimal use of resources.
     
• Recommendations Dashboard: Designed to empower engineers with deeper insight into notebook metrics, RightBrain created the Recommendations Dashboard. This allows for pre-deployment analysis, equipping engineers with the knowledge needed to make cost-optimization decisions proactively.

‍

RightBrain Networks’ solutions delivered substantial benefits for Meijer’s Azure Databricks environment, resulting in annual cost savings estimated between $300,000 and $500,000. By utilizing the ECMD to optimize cost monitoring, implementing insights from the cost recommendations dashboard, and monitoring alerts, Meijer gained the ability to address inefficiencies and adjust job configurations before they led to significant expenses. This comprehensive approach not only reduced operational costs but also enhanced the overall efficiency of Meijer’s data processing workflows.
‍

Beyond the financial impact, the project also fostered more informed decision-making among Meijer’s engineering teams. The recommendations dashboard provided critical insights, enabling teams to evaluate both new and existing jobs for cost-effectiveness before deployment. Additionally, Meijer plans to democratize access to these tools to share insights across the organization, promoting transparency and empowering a wide variety of stakeholders. Through this partnership with RightBrain Networks, Meijer has enhanced their ability to achieve sustainable cost control and greater operational agility in their data environment not only for the immediate future but also for the company’s future growth.

‍