Proactive management of video transcoding services
Deneke, Tewodros (2017-05-04)
Turku Centre for Computer Science (TUCS)
04.05.2017
Julkaisu on tekijänoikeussäännösten alainen. Teosta voi lukea ja tulostaa henkilökohtaista käyttöä varten. Käyttö kaupallisiin tarkoituksiin on kielletty.
Julkaisun pysyvä osoite on
https://urn.fi/URN:ISBN:978-952-12-3536-8
https://urn.fi/URN:ISBN:978-952-12-3536-8
Tiivistelmä
The consumption of digital video has tremendously increased in the last decade due to advances in information technology, computational capability and communication networks. Following that, video processing applications such as video encoding, transmuxing and transcoding are increasingly being deployed as a service in cloud-based environments. However, these services exhibit high variations in required computational power depending on input video characteristics, processing parameters and request arrival rate patterns. Such differences critically affect the efficient management of computing resources and in turn the running cost of these services. Also, since cloud and distributed computing infrastructures are made up of modern multicore processors, multimedia processing services running on top of them need to be parallelized to enhance overall system utilization further. The primary focus of this thesis is thus the proactive management of multimedia processing services and their parallelization. A solution has been sought to these problems in the context of video transcoding services.
To obtain enough information on the characteristics of real world videos and their transcoding parameters needed to model transcoding workload, a video characteristics dataset was built using data collected from a large video-on-demand system, YouTube. The dataset contains a million randomly sampled video instances listing various fundamental video features. Analysis of the dataset provides insightful statistics on the main online video characteristics that can be further exploited to optimize or model components of a multimedia processing systems. Driven by the insight gained from our datasets an approach to predict transcoding workload of a video given transcoding parameters was proposed. The approach treats video transcoding time as a target variable and predicts it using a model learned from past observations. These observations are divided into training and validation sets while learning the prediction models. The method predicts the transcoding time as a function of several parameters of the input and output video streams. The effectiveness of the method is shown via comparing the resulting predictions with the actual transcoding times on unseen video streams. A squared correlation (predicted vs. measured) of 0.958 has been achieved by the best prediction model on 20-second video segments. Furthermore, simulation results show that the proposed prediction method enables significantly better resource management approaches. More specifically, a proactive load balancing method for transcoding jobs across virtual machines is presented and compared with standard load balancing methods in terms of total system utilization and quality of service. Up to 15% improvement has been achieved in terms of system throughput over the round robin and queue length approaches on a fixed server capacity. Also, a proactive provisioning method that enables provisioning of the right amount of transcoding servers for a given quality of service was presented and evaluated. An Experiment has shown that using proactive provisioning rather than fixed worst case based provisioning saves up to (86%) in terms of virtual machine (VM) hours over the course of a week. Also up to 8% in VM hours has been shown to be saved when augmenting proactive provisioning with proactive load balancing.
In addition to the proactive management of computing resources proper, parallelization of the video transcoding application is important in order to ensure efficient utilization of modern multicore processing units that constitute modern cloud and distributed computing infrastructures. To this end, this thesis provides a solution that allows convenient integration of finegrained parallel dataflow based implementations of video processing components into existing transcoding frameworks. The primary aim of this proposed solution was to facilitate fine-grained parallelization of video processing components better. Course grained parallelization approaches for video transcoding applications at a group of pictures level and based on a message passing programming model are also presented and evaluated in terms of scalability and throughput. Results show that the proposed parallelization approach scales almost linearly to the number of cores.
To obtain enough information on the characteristics of real world videos and their transcoding parameters needed to model transcoding workload, a video characteristics dataset was built using data collected from a large video-on-demand system, YouTube. The dataset contains a million randomly sampled video instances listing various fundamental video features. Analysis of the dataset provides insightful statistics on the main online video characteristics that can be further exploited to optimize or model components of a multimedia processing systems. Driven by the insight gained from our datasets an approach to predict transcoding workload of a video given transcoding parameters was proposed. The approach treats video transcoding time as a target variable and predicts it using a model learned from past observations. These observations are divided into training and validation sets while learning the prediction models. The method predicts the transcoding time as a function of several parameters of the input and output video streams. The effectiveness of the method is shown via comparing the resulting predictions with the actual transcoding times on unseen video streams. A squared correlation (predicted vs. measured) of 0.958 has been achieved by the best prediction model on 20-second video segments. Furthermore, simulation results show that the proposed prediction method enables significantly better resource management approaches. More specifically, a proactive load balancing method for transcoding jobs across virtual machines is presented and compared with standard load balancing methods in terms of total system utilization and quality of service. Up to 15% improvement has been achieved in terms of system throughput over the round robin and queue length approaches on a fixed server capacity. Also, a proactive provisioning method that enables provisioning of the right amount of transcoding servers for a given quality of service was presented and evaluated. An Experiment has shown that using proactive provisioning rather than fixed worst case based provisioning saves up to (86%) in terms of virtual machine (VM) hours over the course of a week. Also up to 8% in VM hours has been shown to be saved when augmenting proactive provisioning with proactive load balancing.
In addition to the proactive management of computing resources proper, parallelization of the video transcoding application is important in order to ensure efficient utilization of modern multicore processing units that constitute modern cloud and distributed computing infrastructures. To this end, this thesis provides a solution that allows convenient integration of finegrained parallel dataflow based implementations of video processing components into existing transcoding frameworks. The primary aim of this proposed solution was to facilitate fine-grained parallelization of video processing components better. Course grained parallelization approaches for video transcoding applications at a group of pictures level and based on a message passing programming model are also presented and evaluated in terms of scalability and throughput. Results show that the proposed parallelization approach scales almost linearly to the number of cores.