o H h2 @s6ddlmZddlmZddlZddlmmmZ ddl m Z ddl m Z ddlmZddlmZmZmZmZmZde d e d eed eeed feed fffd dZdejd e d eed eeeeeffddZdejjdeed e fddZde d e d eed eed ffddZ d ej!fddZ"dS))Sequence)castN) ShapeType) DeviceMesh) DTensorSpec) _StridedShardPartial Placement ReplicateShard global_shapemesh placementsreturn.c s>dur dSt|}dgt|}fddtt|D}dgt|}t|D]b\}}|} t|tr|j} dgt|} | t|ksWJd| dt||j || | |d d \} } | || <| | | <|| | | kr}| | || <n || | | 7<|| | 9<q-t d d |D}|rd gt|}d gt|}t|D]S\}}|} t|tr|j} || rt d|d|d|d|| rd || <t|t rd || <|| |j | || |<q|| | <|| || |<qfddt|D}ddt||D}t|t|fS)a Compute the local tensor shape and the global offsets into the original tensor of a DTensor on its current global rank. This is useful for checkpointing purpose. Example (2 host with 4GPUs each): # Below is a DeviceMesh with mesh_shape of (2, 4) mesh = DeviceMesh(device_type="cuda", mesh=[ [0, 1, 2, 3], [4, 5, 6, 7] ], ) Let's say we distribute a global_tensor of shape (8,4) over the above DeviceMesh with a placements of [Shard(0), Shard(0)]. The local shape and global offset will be as follows: rank0 -- local_shape:[1, 4], global_offset:[0, 0] rank1 -- local_shape:[1, 4], global_offset:[1, 0] rank2 -- local_shape:[1, 4], global_offset:[2, 0] rank5 -- local_shape:[1, 4], global_offset:[5, 0] rank3 -- local_shape:[1, 4], global_offset:[3, 0] rank4 -- local_shape:[1, 4], global_offset:[4, 0] rank6 -- local_shape:[1, 4], global_offset:[6, 0] rank7 -- local_shape:[1, 4], global_offset:[7, 0] Let's say we distribute a global_tensor of shape (2) over the above DeviceMesh with a placements of [Shard(0)]. We will not have non-empty local tensor for all the ranks. The local shape and global offset will be as follows: rank0 -- local_shape:[1,], global_offset:[0,] rank1 -- local_shape:[1,], global_offset:[1,] rank2 -- local_shape:[0,], global_offset:[2,] rank5 -- local_shape:[0,], global_offset:[2,] rank3 -- local_shape:[0,], global_offset:[2,] rank4 -- local_shape:[0,], global_offset:[2,] rank6 -- local_shape:[0,], global_offset:[2,] rank7 -- local_shape:[0,], global_offset:[2,] N)rrcsg|]}dgjqSr)ndim).0_)r rS/var/www/vscode/kcb/lib/python3.10/site-packages/torch/distributed/tensor/_utils.py Bs z9compute_local_shape_and_global_offset.. Sharding dim  greater than tensor ndim T) return_offsetcss|]}t|tVqdSN) isinstancer)rprrr {sz8compute_local_shape_and_global_offset..FzTStrided sharding does not allow Shard() to appear after the strided part has ended. z at idx z in z violates this assumption.cs(g|]\}}tddt|DqS)cSg|]\}}||qSrrrxyrrrrzDcompute_local_shape_and_global_offset...)sumzip)r shard_dimshard_idx_stride) my_coordinaterrrscSrrrr rrrrr#)get_coordinatelistlenrange enumeratesizerr dim_local_shard_size_on_dimanyNotImplementedErrorr split_factorr%tuple)r r r local_shape global_offsetshard_idx_stride_by_mesh_dimnum_shards_by_tensor_dimidx placement mesh_dim_sizer& local_offset shard_size shard_offsetstrided_shardingstrided_part_seenstrided_part_end shard_idxr)r r(r%compute_local_shape_and_global_offsets(           rCtensorc Cst|}t|}t|D]n\}}||}|rmtt|}|jdkr.td||j} | |j ksEJd| d|j d|d|| } | ||| <t t |D]} | | krk|| || krk|| ||| <qUqt |t tfs~tdt|dq||fS) aV Compute the global size and stride of a DTensor from the given local tensor. The local size is multiplited by `world_size` per Sharding dim. The local stride is multiplited by `world_size` per Sharding dim, as long as the dimension is outside sharding dim. For example, if we have a local tensor with size (4, 8, 2) and stride (16, 1, 8). If the DTensor placements are [Shard(2)] and world_size is 2; then the global size is (4, 8, 4) and stride is (16 * 2, 1, 8). Args: tensor (:class:`torch.Tensor`): Local tensor which DTensor will be constructed from. mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]]): The attribute of the DTensor that describes its layout on the mesh topology. Return: tensor_shape: A List of int which specifies the size of DTensor which build on top of the local tensor. tensor_stride: A List of int which specifies the stride of DTensor. rzOShard placements should have negative dims normalized in the user-facing APIs: rrz for placement number .zplacement type z not supported!)r*r.strider-is_shardrr r/AssertionErrorrr,r+rr r RuntimeErrortype) rDr r tensor_shape tensor_strider9r:r;shard_placementr&local_dim_sizeirrrcompute_global_tensor_infos6       rPop_callargscCsp|D]-}t|tjtfr|jSt|ttfr/t|dkr/t|dtjtfr/|djSqtd|d)z Find the device mesh object from args. It returns None if no mesh is found. NOTE: we can optimize this search if needed rz+Cannot find device mesh from args for op : rE) rdtensorDTensorr device_meshr*r4r+ ValueError)rQrRargrrrtry_find_mesh_from_argss   rX global_stridecsdgtt|D]*\}}|r5tt|j}ttD]}||kr4|||9<qq tfddttDS)z Compute the stride of a local tensor shard, given the global stride of the DTensor. NOTE: Currently this function is assuming the DTensor is evenly shardable. rc3s |] }||VqdSrr)rrOrYstride_divisorsrrrs z'compute_local_stride..) r+r-rGrr r/r,r.r4)rYr rmesh_idxrrOjrrZrcompute_local_strides  r^cCs\t|tjr|St|tr|g}nt|dkr%t|dtr%t|d}nt|}t|S)z Unify variable types of size argument to torch.Size Acceptable types include: int, Sequence[int], Tuple[int], Tuple[Sequence[int]], or torch.Size rr)rtorchSizeintr+rr*)r. torch_sizerrrnormalize_to_torch_size s   rc)#collections.abcrtypingrr_torch.distributed.tensor._api distributedrD_apirStorch._prims_commonrtorch.distributed.device_meshr&torch.distributed.tensor._dtensor_specr(torch.distributed.tensor.placement_typesrrr r r r4rarCTensorr*rP_ops OpOverloadobjectrXr^r`rcrrrrsX       ;