Executive summary Executive summary
Client background
Our client is a Fortune 500 industrial supply company. It offers over 1.6 million quality in-stock products in such categories as safety, material handling, and metalworking. Also, the company provides inventory management and technical support to more than 3 million customers in North America.
Business challenge
The client needed to transform its existing on-premise data solution and move it to the cloud to ensure better scalability, improve reliability, and reduce costs.
N-iX approach
N-iX has helped the client develop the cloud-agnostic strategy towards cloud migration and built a unified data platform on AWS.
Value delivered
As a result, the client is able to access all data in one place, predict and optimize costs, easily switch a cloud provider, and deliver great service at scale.
Success story in detail
Being a large-scale industrial supply company, our client needed to efficiently manage large amounts of data, including the data on inventory-related costs. Therefore, the company decided to extend its data warehouse solution which collects data from multiple departments.
For the development and support of the solution, our client hired on-site consultants. However, that caused significant overhead, and the company was looking to reduce these operational costs. Also, our client wanted to migrate the solution to the cloud to make it more scalable and cost-efficient.
Therefore, to boost their engineering capabilities, further support the solution, and optimize costs, the company needed a reliable offshore development partner with long-standing big data expertise. As a result of the rigorous selection process, our client chose N-iX.
To migrate from on-premise Hadoop Hortonworks cluster to AWS and allow processing additional data in AWS, the N-iX team built an AWS-based big data platform from scratch. Also, we have been involved in extending and supporting the existing Teradata solution. Teradata is used to collect data from other systems and further generate reports with Business Object and Tableau. The data sources are MS SQL, Oracle, and SAP.
To choose the data warehouse design and the tech stack that fit our client’s business needs, our specialists created a proof of concept. We compared Amazon Redshift with Snowflake and preferred Snowflake as it met the client’s approach of cloud neutrality: it can easily scale up and down any amount of computing power for any number of workloads and across any combination of clouds.
The whole development process is cloud-agnostic and is designed to ensure that the client can easily change the cloud provider in the future. For example, we use Terraform as it is compatible with all cloud vendors - AWS, Azure, and Google Cloud.
The project is in the development phase. Currently, our specialists are in the process of building the environment that will be able to process large datasets. The use of Snowflake and Airflow technologies allows us to automate the data extraction process. Also, Snowflake minimizes data duplication by checking whether the ingested files have been already processed or not.
After half of the year of working on the data pipeline unification, we managed to integrate more than 100 different data sources into a unified data platform. This includes daily data loads, along with a backfill of historical data. We are working with TBs of data tables, and the size is growing.
- Saving infrastructure costs with cloud migration.
- Improved efficiency of data management thanks to the unified data platform that stores all the data in one place.
- Predictive analytics capabilities of the platform. For example, the finance department will be able to predict the inventory-related expenses.
- Using cloud-neutral technology to avoid a vendor lock-in and benefit from different cloud providers.
- Reducing software development overhead by replacing expensive on-site contractors.