7.2.1 ShuffleMaster与ShuffleEnvironment的设计与实现

ShufferService的设计与实现 #

在JobMaster中创建ShuffleMaster #

注册分区信息 #

在TaskManager中创建ShuffleEnvironment #

ResultPartition #

结构 #

创建ResultPartition #

向ResultPartition写入Buffer数据 #

ResultSubPartition #

结构 #

SingleInputGate #

结构 #

创建 #

初始化 #

请求分区数据 #

Netty #

NettyServer #

ChannelPipeline
public ChannelHandler[] getServerChannelHandlers() {
    PartitionRequestQueue queueOfPartitionQueues = new PartitionRequestQueue();
    PartitionRequestServerHandler serverHandler =
            new PartitionRequestServerHandler(
                    partitionProvider, taskEventPublisher, queueOfPartitionQueues);

    return new ChannelHandler[] {
        messageEncoder,
        new NettyMessage.NettyMessageDecoder(),
        serverHandler,
        queueOfPartitionQueues
    };
}

NettyClient #

ChannelPipeline
public ChannelHandler[] getClientChannelHandlers() {
    NetworkClientHandler networkClientHandler = new CreditBasedPartitionRequestClientHandler();

    return new ChannelHandler[] {
        messageEncoder,
        new NettyMessageClientDecoderDelegate(networkClientHandler),
        networkClientHandler
    };
}