UDT: UDP-based Data Transfer for High-Speed Wide Area Networks

UDT：基于UDP的高速广域网数据传输

    Yunhong Gu and Robert L. Grossman                                        Yunhong Gu和Robert L. Grossman 
    National Center for Data Mining, University of Illinois at Chicago    芝加哥大学伊利诺伊国家数据挖掘中心
    851 S Morgan St, M/C 249, Chicago, IL 60607, USA         851 S Morgan St，M/C 249，芝加哥，IL 60607，USA 
    gu@lac.uic.edu, grossman@uic.edu                                        gu@lac.uic.edu，grossman@uic.edu

Abstract

In this paper, we summarize our work on the UDT high performance data transport protocol in the past four years. UDT was designed to effectively utilize the rapidly emerging high-speed wide area optical networks. It is built on the top of UDP with reliability control and congestion control, which makes it quite easy to get installed. The congestion control algorithm is the major internal functionality to enable UDT to effectively utilize the high bandwidth. Meanwhile, we also implemented a set of APIs to support easy application implementation, including both reliable data streaming and partial reliable messaging. The original UDT library has also been extended to Composable UDT, which can support various congestion control algorithms. We will describe in detail the design and implementation of UDT, the UDT congestion control algorithm, Composable UDT, and the performance evaluation.

摘要

在本文中，我们总结了过去四年我们对UDT高性能数据传输协议的工作。 UDT设计有效地利用快速新兴的高速广域光网络。它依赖UDP通信协议，也实现了可靠性控制和拥塞控制，这使得它的安装相当容易。拥塞控制算法是主要内部功能，此算法使UDT有效地利用高带宽。同时我们还实施了一套API来支持简单的应用程序实施，包括可靠的数据流和部分可靠的消息。原来的UDT库也扩展到可组合UDT，它可以支持各种拥塞控制算法。我们将详细描述UDT的设计和实施，UDT拥塞控制算法，可组合UDT和性能评估。

Keywords: Transport Protocol, Congestion Control, High Speed Networks, Design and Implementation
关键字：传输协议，拥塞控制，高速网络，设计与实现

1.INTRODUCTION

The rapid increase of network bandwidth and the emergence of new distributed applications are the two reciprocal motivations of networking research and development. On the one hand, network bandwidth today has been expanded to 10Gb/s with 100Gb/s emerging, which enables many data intensive applications that were impossible in the past. On the other hand, new applications, such as scientific data distribution, expedite the deployment of high-speed wide-area networks.

Today, national or international high-speed networks have connected most developed regions in the world with fibers [13,17]. Data can be moved at up to 10 Gb/s among these networks and often at a higher speed inside the networks themselves. For example, in the United States, there are national multi-10Gb/s networks, such as National Lambda Rail, Internet2/Abilene, Teragrid, ESNet, etc. They can connect to many international networks such as CA*Net 4 of Canada, SurfNet of the Netherlands, and JGN2 of Japan.

1.引言

网络带宽的快速增加和新的分布式应用程序的出现，对网络研究和开发而言是两个对等的驱动。一方面，现在的网络带宽已经扩展到10Gb / s并且随着 100Gb / s的兴起，许多过去不可能实现的数据密集型应用程序已成为可能。另一方面，新的应用程序，如科学数据分发，高速广域网的加快部署。

今天，国家或国际高速网络已经用纤维[13，17]连接全球大多数的发达地区。数据能够以高达10Gb / s的速度在这些网络之间移传输，并且在各个网络内部以更高的速率传输。例如，在美国，有国家的multi-10Gb/s网络，如国家铁路LAMBDA（National Lambda Rail），代互联网/阿比林（Internet2/Abilene）， TeraGrid，ESNet等，它们可以连接到许多国际网络，如加拿大的CA * 4荷兰的SURFNET和日本的JGN2网络。

Meanwhile, we are living in a world of exponentially increasing data. The old way of storing data in disk or tape storage and delivering them by transport vehicles is no longer efficient. In any situations, this old fashioned method of shipping disks with data on them makes it impossible to meet the applications'requirement (e.g., online data analysis and processing).

同时，我们生活在数据倍增的世界。将数据存储在磁盘或磁带上，通过运输车传发的老方法已经不再有效。很多情况下，运送磁盘传送数据的老式方法已经无法满足应用程序的需求(例如，在线数据分析处理)。

Researchers in high-energy physics, astronomy, earth science, and other high performance computing areas have started to use these high-speed wide area optical networks to transfer their terabytes of data. We expect that home Internet users will also be able to make use of the high-speed networks in the near future for applications with high-resolution streaming video, for example. In fact, an experiment between two ISPs in USA and Korea has demonstrated an effective 80Mb/s data transfer speed.

研究人员已经在高能物理，天文，地球科学和其他高性能计算领域开始使用这些高速广域光纤网络传输TB级的数据。我们预计，在不久的将来家庭互联网用户也将能通过高速网络使用具有高分辨率的视频流。例如，事实上，在美国和韩国两个ISP之间的实验，表现出了80MB / s的有效的数据传输速度。

Unfortunately, the high-speed networks have not been smoothly used by those applications. The Transmission Control Protocol (TCP), the de factotransport protocol of the Internet, substantially underutilizes network bandwidth over high-speed connections with long delays [13, 22, 39, 70]. For example, a single TCP flow with default parameter settings on Linux 2.4 can only reach about 5 Mb/s over a 1Gb/s link between Chicago and Amsterdam; with careful parameter tuning the throughput still only reaches about 70Mb/s. A new transport protocol as a timely solution is required to address this challenge. The new protocol is expected to be easily deployed and easily integrated with the applications, in addition to utilizing the bandwidth efficiently and fairly.

不幸的是，高速网络还没顺利被这些应用程序使用。传输控制协议（TCP），因特网的实际传输协议，因为长延时[13，22，39，70]基本上无法充分利用高速连接的网络带宽。例如，单个TCP流在Linux 2.4的默认参数设置下，在Chicago和Amsterdam之间的1Gb / s链路上只能达到约5 Mb / s;通过参数的精细调整，吞吐量仍然只达到约70MB /秒。需要一种新的传输协议及时解决应对这一挑战。新的协议，除了高效和公平使用带宽，预计将轻松地部署和集成应用程序。

Network researchers have proposed quite a few solutions to this problem, most of which are new TCP congestion control algorithms [8, 23, 25, 37, 40, 53, 55, 67] and application-level libraries using UDP [18, 26, 61, 65, 66, 71]. Parallel TCP [1, 56] and XCP [39] are two special cases: the former tries to start multiple concurrent TCP flows to obtain more bandwidth, whereas the latter is a radical change by introducing a new transport layer protocol involving changes in routers.

针对这个问题网络研究人员提出了不少解决方案，其中大部分是新的TCP拥塞控制算法[8，23，25，37，40，53，55，67]和使用UDP [18，26，61，65，66，71]应用级别库。并行TCP [1，56]和XCP [39]是两种特殊情况：前者试图启动多个并发TCP流，以获得更多的带宽，而后者则是从根本上改变通过引入新的传输层协议涉及在路由器改变。

In UDT we have a unique approach to address the problem of transfer large volumetric datasets over high bandwidth-delay product (BDP) networks. While UDT is a UDP-based approach, to our best knowledge, it is the only UDP-based protocol that employs a congestion control algorithm targeting shared networks. Furthermore, UDT is not only a new control algorithm, but also a new application level protocol with support for user configurable control algorithms and more powerful APIs.

在UDT中我们有一个独特的方法来解决在高带宽延迟产品（BDP）网络传输大容量数据集的问题。虽然UDT是基于UDP的方式，以我们所知，它是唯一基于UDP的协议，它采用了拥塞控制算法靶向共享网络。此外，UDT不仅是一个新的控制算法，也为用户可配置的控制算法和更强大的API支持新的应用程序级协议。

This paper summarizes our work of UDT in the past four years. Section 2 gives an overview of the UDT protocol and describes its design and implementation. Section 3 explains its congestion control algorithm. Section 4 introduces Composable UDT that supports configurability of congestion control algorithms. Section 5 gives an experimental evaluation of the UDT performance. Section 6 concludes the paper.

本文总结了过去四年我们UDT方面的工作。第2节给出了UDT协议的概述，并介绍了它的设计和实施。第3节说明了它的拥塞控制算法。第4节介绍了支持可配置可组合UDT的拥塞控制算法。部分5给出了UDT性能的实验评估。第6节总结全文。