This case study highlights AIM Consulting’s recent work with a leading multinational research and development organization. The company sought to consolidate diverse on-premises data onto a unified cloud platform and improve data management, manipulation, sharing, and advanced analytics capabilities.
AIM Consulting was chosen due to our distinctive, holistic approach to technology and modern methodologies, as well as our deep expertise in development and data architecture. The project outcomes include enhanced data integration development, self-service reporting capabilities, and the establishment of a robust foundation for future state advanced analytics.
Case Study Contents:
- Business Challenge: Difficulty Creating Efficient, Organized, Secure Data Environment
- Approach: Implemented Robust, Secure Data Lakehouse/Data Mesh Solution
- Results: Seamless Access, Analysis, Search, and Integration of Datasets
What AIM Consulting did:
- Introduced a modern data architecture, incorporating cloud-based storage, ETL/ELT pipelines, and Databricks technology to improve data processing efficiency.
- Implemented automation frameworks for data ingress and egress, indexing/optimization, and data quality checks, enhancing operational efficiency.
- Developed self-service enablement features, allowing users to independently source, organize, and share data, reducing dependence on IT for routine tasks.
- Implemented data catalog, metadata integration, workflow orchestration, and computational governance for improved data management and security
- Supported the client in implementing change management practices, including creating go-to-market plans, facilitating communities of practice, and driving cultural change
- Designed a scalable architecture with decentralized sensibilities, addressing current functionality gaps and providing a foundation for future state advanced analytic capabilities.
Technologies used:
- Databricks
- AWS Simple Storage Service (S3)
- AWS Kubernetes Service (EKS)
- AWS Lambda
- AWS Athena AWS Neptune
- AWS GraphQL
- AWS Storage Gateway
- AWS Aurora MySQL
- AWS OpenSearch
- AWS Managed Kafka (MSK)
- Microsoft Azure Active Directory (AAD)
- Microsoft .Net Core
- Temporal
- Github
- Docker
- Splunk
Business Challenge: Difficulty Creating Efficient, Organized, Secure Data Environment
A prominent research and development organization faced a critical challenge in managing its R&D data scattered across on-premises file systems. Inefficient data consolidation, limited sharing capabilities, and challenges in targeted searches and metadata management hindered research optimization.
Previous attempts at implementing a data lake were unsuccessful due to the complexities of existing file systems and challenging security landscapes. Internal expertise was centered on data science, with a lack of proficiency in development or architecture, compelling the organization to operate within disorganized on-premises data file systems and raising concerns about data quality and security.
The client sought to consolidate diverse on-premises data onto a unified cloud platform, fostering collaboration, improving communication, and establishing a foundation for advanced analytics.
Ultimately, the goal was to create an efficient, organized, and secure environment for R&D data, overcoming previous challenges and ensuring the successful utilization of a modern data storage solution.
After unsuccessful engagements with previous vendors, AIM Consulting, with its holistic approach to technology and modern data architecture methodologies, was selected to develop a modern data lakehouse/data mesh solution.
Approach: Implemented Robust, Secure Data Lakehouse/Data Mesh Solution
AIM’s work on the OnePlace project entailed the development and implementation of a robust data platform and architecture adhering to data mesh best practices. Our experts’ role involved evaluating and enhancing the existing architecture and addressing data scalability, sharing, and quality concerns.
The project involves several key milestones and deliverables, spanning multiple contracts and change orders. AIM’s experts began by developing project management processes, which included conducting technical interviews to assess existing on-premises and future state cloud data sources and data sets.
Our team also evaluated data movement, tools, transformation processes, and management, ultimately forming and providing architecture recommendations and implementation plans.
Our technical experts then focused on development of the AWS and Databricks lakehouse architecture, which included:
- Creating ETL/ELT data pipelines with workflow orchestration and streaming.
- Performing indexing/optimization and data quality checks.
- Developing a data management framework leveraging Databricks Unity Catalog.
- Involvement in application architecture, orchestration engineering, data engineering, domain node design, pipeline sequencing, global catalog administration, and more.
- Transfering data from local and on-premises locations to the OnePlace cloud.
We designed and implemented data management and governance best practices through implementing a data catalog, metadata integration, workflow orchestration, and computational governance. This ensures compliance with legal and privacy requirements, enhances data quality, and fosters trust among stakeholders.
Our experts provided additional value to the client by implementing automation frameworks for data ingress and egress, indexing/optimization, and data quality checks, dramatically enhancing operational efficiency.
Furthermore, we supported the adoption of new technologies and methodologies through change management practices, driving cultural change, fostering a culture of collaboration, facilitating communication between teams, and ensuring user acceptance across various groups.
Results: Seamless Access, Analysis, Search, and Integration of Datasets
AIM Consulting’s ongoing project integrates seven data products in the OnePlace platform, attracting numerous data scientists and analysts engaged in intricate analytics.
Our collaboration with R&D stakeholders has contributed to the design of the front-end application, now undergoing Quality Assurance (QA) testing with stakeholders and Subject Matter Experts (SMEs). Upon deployment to production, this will significantly expand the platform’s user base.
Thanks to our experts’ work on this project, the client can now seamlessly transition data from on-premises storage to OnePlace using established pipelines. Through Databricks, they can access, analyze, and manipulate data in a cloud environment, facilitating seamless search and integration of datasets, a task that previously proved challenging, if not impossible.
OnePlace ensures data compliance, as all data products adhere to rigorous governance and security standards, enhancing data trustworthiness.
As the project progresses, the client is poised to extend its impact beyond existing R&D boundaries. Plans include incorporating data from additional sources, implementing expanded compliance rules, and onboarding more teams into OnePlace, leveraging its diverse features.
Simultaneously, our experts’ ongoing enhancements to the front-end platform aim to democratize data access, allowing individuals without technical expertise, but with data requirements, to easily access and utilize OnePlace-stored data.
Our technology experts’ work on this project is creating a more inclusive environment at the client, fostering data knowledge, understanding, transparency, and literacy for all users.
Insights By
Get in touch with our Data Experts.
AIM Consulting offers distinctive value with our hands-on experience in implementing Data Mesh and related concepts, our expertise in organizational change management, and our skilled resources ready to transform your data and analytical capabilities.
