# 2.5D 张量并行

## 引言​

$\left[\begin{matrix} X_{30} & X_{31} \\ X_{20} & X_{21} \\ X_{10} & X_{11} \\ X_{00} & X_{01}\end{matrix} \right],$

$\left[\begin{matrix} X_{10} & X_{11} \\ X_{00} & X_{01} \end{matrix} \right] \text{~and~}\left[\begin{matrix} X_{30} & X_{31} \\ X_{20} & X_{21} \end{matrix} \right].$

$\left[\begin{matrix} A_{10} & A_{11} \\ A_{00} & A_{01} \end{matrix} \right].$

$\left[\begin{matrix} Y_{10}=X_{10}A_{00}+X_{11}A_{10} & Y_{11}=X_{10}A_{01}+X_{11}A_{11} \\ Y_{00}=X_{00}A_{00}+X_{01}A_{10} & Y_{01}=X_{00}A_{01}+X_{01}A_{11} \end{matrix} \right] \text{~and~}$
$\left[\begin{matrix} Y_{30}=X_{30}A_{00}+X_{31}A_{10} & Y_{31}=X_{30}A_{01}+X_{31}A_{11} \\ Y_{20}=X_{20}A_{00}+X_{21}A_{10} & Y_{21}=X_{20}A_{01}+X_{21}A_{11} \end{matrix} \right].$

## 效率​

$O(1/dq^2)$$O(1/q^2)$$O(1/dq^2)$$\small O(3(q-1)(d+1)/dq)$$O(6(q-1))$

## 使用​

CONFIG = dict(parallel=dict(    data=1,    pipeline=1,    tensor=dict(size=8, mode='2.5d', depth=2),))

import colossalaiimport colossalai.nn as col_nnimport torchfrom colossalai.utils import print_rank_0class MLP(torch.nn.Module):    def __init__(self, dim: int = 256):        super().__init__()        intermediate_dim = dim * 4        self.dense_1 = col_nn.Linear(dim, intermediate_dim)        print_rank_0(f'Weight of the first linear layer: {self.dense_1.weight.shape}')        self.activation = torch.nn.GELU()        self.dense_2 = col_nn.Linear(intermediate_dim, dim)        print_rank_0(f'Weight of the second linear layer: {self.dense_2.weight.shape}')        self.dropout = col_nn.Dropout(0.1)    def forward(self, x):        x = self.dense_1(x)        print_rank_0(f'Output of the first linear layer: {x.shape}')        x = self.activation(x)        x = self.dense_2(x)        print_rank_0(f'Output of the second linear layer: {x.shape}')        x = self.dropout(x)        return x

parser = colossalai.get_default_parser()colossalai.launch(config=CONFIG,                  rank=args.rank,                  world_size=args.world_size,                  local_rank=args.local_rank,                  host=args.host,                  port=args.port)m = MLP()

Weight of the first linear layer: torch.Size([128, 512])Weight of the second linear layer: torch.Size([512, 128])

from colossalai.context import ParallelModefrom colossalai.core import global_context as gpcfrom colossalai.utils import get_current_devicex = torch.randn((16, 256), device=get_current_device())# partition inputtorch.distributed.broadcast(x, src=0)x = torch.chunk(x, 2, dim=0)[gpc.get_local_rank(ParallelMode.PARALLEL_2P5D_DEP)]x = torch.chunk(x, 2, dim=0)[gpc.get_local_rank(ParallelMode.PARALLEL_2P5D_COL)]x = torch.chunk(x, 2, dim=-1)[gpc.get_local_rank(ParallelMode.PARALLEL_2P5D_ROW)]print_rank_0(f'Input: {x.shape}')x = m(x)

Input: torch.Size([4, 128])Output of the first linear layer: torch.Size([4, 512])Output of the second linear layer: torch.Size([4, 128])

2.5D并行中的 activation 张量都是同时在$d \times q$行和$q$列分割的。例如，第一个线性层的输出是 [4, 512], 而第二层的输出为 [4, 128]。 注意，2.5D并行使用与2D并行相同的划分方法来处理权重，区别在于对输入的划分。