The Cloud Migration Roadmap of Big Data Business PART III - Key Technical Transformation and Summary

Posted Dec 9, 2023

By Xinjie (Edward) HU 8 min read

Preface

In 2023 Q1, I participated in the cloud migration project for big data business in our department, from on-premises to Tencent Cloud. After the migration, we reached the goal of cost reduction and increasing operation & maintenance efficiency.

Some experience and works during this process has been summarized (not confidential things included). The whole content has been split into three techinical notes:

PART I: the big data business overview and on-premises architecture;

PART II: the work and experience learned during cloud migration;

PART III (this article): the key technological transformations and summary.

Key Technical Transformation

Storage and Computation Separation

Background

For nearly all enterprises switching from traditional on-premises IDC to cloud services, choosing the right data storage class is the biggest problem encountered in the process of cloud migration. Particularly, our cloud vendor provides three main options:

Using the EMR-native HDFS + on-premises disk. This solution can obtain higher local throughput and cheaper storage costs.
Using the EMR-native HDFS + cloud disk. This solution can obtain higher flexibility and can scale out the disk usage at any time as required.
Using object storage service (COS/OFS bucket). This solution can allow customers to save only a single copy to achieve high availability of data, thereby saving data storage costs. However, because data manipulation has been transformed from local I/O to network read process, this will sacrifice some I/O and throughput performance.

Before choosing a best solution, let us first look at the prices of these resources:

Analysis:

By using HDFS solely, this is undoubtedly the most expensive option. Suppose 1PB data is stored in HDFS D3 node (since the disk failure rate is 3‰ per year, any configurations less than 3 replications may cause data loss), at least 70 nodes are required, and the cost is about $64,064/month. On the contrary, using bucket standard storage only costs about $17,318 / month, not to mention that we can even use archive to further reduce the cost. Therefore, HDFS is 3.7 times more expensive than OFS.

In addition, heavily using HDFS may result in scaling difficulty since only two data nodes can be shut down at one time. Starting or shutting down nodes could also cause issues like rebalancing, which may significantly affect business.

For solely using object storage, each bucket is limited by network bandwidth, which is in the tens of Gb/s, the execution efficiency of data tasks will be negatively affected during executing a large number of concurrent tasks.

On the contrary, by using DataNode, the bandwidth of each machine node is in the 10Gb/s, and the accumulated bandwidth of dozens of machines is in the hundreds of Gb/s.

Our Choice

After fully discussing the current architecture and business logic with the cloud vendor, we followed the popular architecture prevailing in the industry and designed the Storage and Computation Separation architecture by mixing Hadoop and object storage.

Given that the time effect of offline data is obvious, the data generated closer is used more frequently, and many offline batch tasks at night are mainly for calculating data generated yesterday. Therefore, we set the standard of the cold and hot data by one week. The cold data (i.e. generated one week ago) is deposited into the bucket. The hot layer data of the last week is stored in the HDFS to ensure the efficiency of query tasks.

Under this design, since Hadoop cluster does not store a large volume of data, we are able to reduce the number of HDFS DataNode machine resources as much as possible and focus on deploying YARN computing resources on D3 and SA3 nodes. Particularly, SA3 scales out according to time schedule or computing requirements, which greatly reduced computing costs and ensured execution efficiency.

In addition to the data generated at the scheduled time of daily offline data tasks, data on the HDFS also contains historical data generated by supplementary data tasks, which may accumulate a large amount of data in a brief period of time. Therefore, cold data must be migrated to the OFS bucket in a timely and reliable manner, without influencing cluster’s performance.

Architecture

The High Availability data migration architecture based on Storage and Computation Separation idea is designed as follow:

the entry of migration task deployment is located on the offline data management platform mentioned above, which is based on the Quartz distributed task scheduling architecture.
run Distributer Job in Quartz to ensure that the Distributer Job is always running through the Quartz high availability architecture.
the Distributer Job obtains HIVE metadata in real time and checks whether the storage path of the partition one week before is in the storage bucket. If not, place the table meta information in the task scheduling queue.

In order to control the load impact of data migration tasks on the system, only 10 worker jobs are deployed to run data migration tasks. After a Worker Job is completed, a new worker job will be created in Quartz for data migration. These worker jobs are evenly scheduled to each node by Quartz. Within each worker job, data is migrated from HDFS to bucket, metadata information such as Hive and Impala is updated, and data in HDFS is eventually deleted.

Achievements

The space usage of the HDFS cluster can be controlled to about 65%, as shown below. This greatly reduces the HDFS storage overhead caused by the growth of data volume.

The offline EMR cluster is elastically auto-scaled by time schedule. Every day at 12 am, 2/3 of the total resources will be pulled up, and the resources will be released at 6 am. At this stage, Vcore usage is basically more than 90%, as shown below.

Throughout the day and night, only 1/3 of the total resources are retained, and the resource utilization rate of this part is maintained at about 60%, although there are some peaks in usage at a brief time period.

Cost management

At present, our cloud vendor can only provide the cost of the entire cluster, and the cost of a single task cannot be seen. However, we hope to be able to do the fine management of the cost, which means to collect a single user's running task resource information, and then analyzes on the resource usage statistics of individuals and teams.

To meet this requirement, we self collected relevant data and used StarRocks for statistical analysis. The data collection is divided into two parts:

Part of the data is collected from YARN, whose Cluster Applications API contains ID, AllocatedMB, AllocatedVCores and other fields. As shown in the following figure, data is collected every 5 seconds using a scheduled task to write to Kafka.
The other part of the data is based on the offline Data Management Platform (DMP). Once the Application in YARN is submitted from DMP, it corresponds to a Job. As shown in the following figure, the management platform collects logs generated by HIVE/Spark, obtains the Application ID, and writes the Application ID and associated Job ID to Kafka. Note that for HIVE tasks, one Job ID may be associated with multiple Application IDs.

In StarRocks, we create two Routine Load tasks to consume the data in Kafka and build a MySQL external table to get the ID and author information from the job. According to the idea of post calculation and the powerful computing performance of StarRocks, we execute JOIN and GROUP BY manipulations from the three data sources to perform various analysis.

For example, analyze which users use the most Vcore resources in a certain period of time.

Summary

Achievements after cloud migration

Cost reduction:

Compared with the total cost of Hadoop deployed on-premises, the cost of the entire big data in cloud has decreased for ~40%, which is a proud and satisfying result;
The cost sharing mechanism is clearer: daily cost can be partitioned to each team / each individual / even each single task, which significantly controls the unnecessary computation overhead;
More abundant methods for cost control: such as EMR elastic auto-scaling strategy, bucket archive storage tier.

Efficiency enhancement:

A single data task run on the cloud is faster than that under the cloud.

To make a sharp comparison, since the on-premises Hadoop cluster is multi-tenant, the resource utilization rate has been maintained at high level, and the tasks always have negative impacts on each other;

The SLA for successfully running daily data tasks has been significantly improved. The data tasks can be completed before 9 am;
More agile response to resource scaling requirements, especially when there are tasks which require a large amount of computing resources.

Future Plan

We will continue to explore the big data architecture best practice to further reduce cost and enhance efficiency, together with the cloud vendor. Particularly, here are the aspects we are going to optimize in the near future: