3j RSSKrSSKJr SSKJr SSKJr SSKrSSKJ r J r SSK J r J r JrJrJrJrJrJrJrJrJrJrJr /SQr\"SS 5r"S S \R65rS \R:4S \R6S\S\S\4Sjjr "SS\5r!"SS\5r"S q#Sr$S q%Sr&g)N) namedtuple)Callable)Any))sparse_semi_structured_from_dense_cutlass'sparse_semi_structured_to_dense_cutlass) fallback_dispatchersemi_sparse_addmmsemi_sparse_clonesemi_sparse_detachsemi_sparse_indicessemi_sparse_linearsemi_sparse_mmsemi_sparse_scaled_mm semi_sparse_tsemi_sparse_tosemi_sparse_to_copysemi_sparse_valuessemi_sparse_view)SparseSemiStructuredTensor!SparseSemiStructuredTensorCUTLASS$SparseSemiStructuredTensorCUSPARSELTto_sparse_semi_structured_SEMI_STRUCTURED_SPARSE_CONFIGz=sparse_min_rows sparse_min_cols dense_min_rows dense_min_colsc \rSrSr%SrSr\\S'\\ R\ 4\S'Sr \ \S'Sr\ \S'Sr\ \S '\\S '\\\4\S '\ R$S -\S '\ R$S -\S'\ R$S -\S'\ R$S -\S'\ R$S -\S'\ \S'\\S'/SQr\S)S\ R*S \ R$S -S\ R$S -S\ R$S -S\ R$S -S\ R$S -S\ S\S\ 4Sjj5rS\4SjrS\\\\\ R*\ \\ 444Sjr\S\\ R*\ \\ 4S\ R$4Sj5r\ R:R<r\S\ 4Sj5r!\S*S+Sjj5r"\S\ R$SS 4S j5r#S!r$\\4S\ R$S"\SS4S#jj5r%S S$.S%\ R$S&\ R$S -S\ R$4S'jjr&S(r'g ),r*a This class implements semi-structured sparsity as a Tensor subclass. Semi-structured sparsity describes a sparsity pattern where n in every 2n elements are sparse, depending on the datatype. It is also referred to as 2:4 sparsity or fine-grained structured sparsity. There are two backends available for semi_structred sparsity, either cuSPARSELt or CUTLASS. This class is meant to serve as a base class for both implementations. SparseSemiStructuredCUTLASS and SparseSemiStructuredCUSPARSELT both inherit from this class and define three backend-specific items. Note that as such, this class cannot be instantiated directly. -`_DTYPE_SHAPE_CONSTRAINTS` - A dictionary holding backend specific dense/sparse min shape constraints - `def from_dense()` - backend specific compression routines - `def _mm()` - backend specific mm op (either torch._cslt_sparse_mm or torch._sparse_semi_structured_(mm|addmm)) r_DEFAULT_ALG_ID_DTYPE_SHAPE_CONSTRAINTSF_FORCE_CUTLASS_FUSE_TRANSPOSE_PROTOTYPE_WARNING_SHOWNBACKENDSPARSE_DISPATCHNpackedmetapacked_tmeta_tcompressed_swizzled_bitmaskfuse_transpose_cusparseltalg_id_cusparselt)r#r$r%r&r'shape requires_gradc UR(dP[R"S[SS9 SUlUR 5 [ R RU5 UbUn OUbUn O [S5e[ RRUUU RU RU RU S9n X+lX;lXKlX[lXklX{lXlU $)a Create a new instance of the tensor subclass from the compressed sparse representation. We have the option to create the subclass with the compressed representations of both X and X', for training. For inference, we only need a single representation (either X or X'), while the corresponding other set will be None. Depending on the backend selected, certain fields will be set to None. (CUSPARSELT vs CUTLASS) Args: shape: The shape of the original dense tensor packed: The compressed representation of the original dense tensor meta: The metadata of the original dense tensor, if it is stored separately packed_t: The compressed representation of the transposed original dense tensor meta_t: The metadata of the transposed original dense tensor, if it is stored separately compressed_swizzled_bitmask: The masks used by the CUTLASS backend to determine which threads should participate in the computation. Used for pointwise ops. fuse_transpose_cusparselt: When running with cuSPARSELt, we have the option to fuse a transposition with a matmul, which is useful in the case of 2:4 sparse training. alg_id_cusparselt: The algorithm id to use when using cuSPARSELT, will have effect on performance Returns: torch.Tensor: A torch.Tensor wrapper subclass. Raises: ValueError: If all of the tensor arguments are None. zThe PyTorch API of SparseSemiStructuredTensor is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.sparse module for further information about the project. stacklevelTz3At least one of packed or packed_t must be provided)devicedtypelayoutr+)r warningswarn UserWarning_load_dispatch_tabletorch_dynamoallow_in_graph ValueErrorTensor_make_wrapper_subclassr0r1r2r#r$r%r&r'r(r)) clsr*r#r$r%r&r'r(r)r+previous_tensortensors V/home/wildlama/miniconda3/lib/python3.13/site-packages/torch/sparse/semi_structured.py__new__"SparseSemiStructuredTensor.__new__OsN++ MMH  ,0C (  $ $ & MM ( ( -  $O  !&ORS S44  "))!''"))' 5   " -H*+D(#4  returnc[US5(d [S5eURRSURS3$)Nr*ztensor has no shape attributez(shape=))hasattrAssertionError __class____name__r*)selfs r@__repr__#SparseSemiStructuredTensor.__repr__s=tW%% !@A A..))*'$**Q??rCc^[[U4SjTR55nTRTRTR TR 4nX4$)Nc >[TU5SL$N)getattr)xrKs r@?SparseSemiStructuredTensor.__tensor_flatten__..sWT1-T9rC)listfilter __slots__r*r(r)r+)rK inner_tensors tensor_metas` r@__tensor_flatten__-SparseSemiStructuredTensor.__tensor_flatten__sT 94>> J  JJ  * *  " "     ))rCrYc UupVpxU"UURSS5URSS5URSS5URSS5URSS5UUUS9 $)Nr#r$r%r&r' r*r#r$r%r&r'r(r)r+)get) r=rXrY outer_size outer_strider*r(r)r+s r@__tensor_unflatten__/SparseSemiStructuredTensor.__tensor_unflatten__sNYJ*; $$Xt4""640"&&z48 $$Xt4(5(9(9-t)'@/'  rCcURUR;a%[URSURS35eURUR"XX45$)NzI only supports a specific set of operations, can't perform requested op (rF)_overloadpacketr"NotImplementedErrorrJ)r=functypesargskwargss r@__torch_dispatch__-SparseSemiStructuredTensor.__torch_dispatch__sb   s':': :%<<.!//3}}oQ@ ""4#7#78dSSrCc [USS5Gc[RRR[ [RRR [[RRR[[RRR[[RRR[[RRR[[RRR[ [RRR"[$[RRR&[$[RRR([*[RRR,[.[RRR0[2[RRR4[6[RRR8[:[RRR<[>0Ul UbUR@RCU5 ggg)zD Loads the op overload sparse dispatch table for the current class. r"N)"rQr7opsatenvaluesrindicesr is_same_sizerdetach_detachr trviewrmmrmatmuladdmmr linearr _to_copyr _scaled_mmrcloner torr"update)r=custom_dispatch_tables r@r6/SparseSemiStructuredTensor._load_dispatch_tablesm 3)4 0 8 %%'9 &&(; ++-@ &&(; %%'9   - ##%5 !!> %%~ $$&7 %%'9 '')< ))+@ $$&7 !!>#C "%0##**+@A1% 9rCoriginal_tensorc UR(d[SURS35eUR5S:wa[SUR5S35eUR 5(d [S5eUR UR ;a[SUR SUS 35eURup#UR UR RnUR UR RnX$:dX$-(dX5:d X5-(a[S URS US US 35eg)zO Assert that the given tensor is valid for semi-structured sparse compression. zError original_tensor.device= z= is not supported! Only CUDA tensors are currently supported.r-zError original_tensor.dim = z; is not supported! Only 2d tensors are currently supported.zXError original_tensor is not contiguous!Only contiguous tensors are currently supported.zError original_tensor.dtype z is not a supported dtype for !zError original_tensor.shape zS is not supported! Both dimensions must be larger or equal than and a multiple of (z, rFN) is_cuda RuntimeErrorr0dim is_contiguousr1rr*sparse_min_rowssparse_min_cols)r=rmnmin_rowsmin_colss r@ _validate_device_dim_dtype_shape;SparseSemiStructuredTensor._validate_device_dim_dtype_shapesk &&01G1G0HI==     A %./B/B/D.EF;;  ,,..C   (D(D D./D/D.EEcdgchhij  $$//0E0EFVV//0E0EFVV <1<1<1<./D/D.EFSS[R\\^_g^hhik >> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> A = torch.Tensor([0, 0, 1, 1]).tile((128, 32)).half().cuda() tensor([[0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], ..., [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.], [0., 0., 1., ..., 0., 1., 1.]], device='cuda:0', dtype=torch.float16) >>> A_sparse = to_sparse_semi_structured(A) SparseSemiStructuredTensor(shape=torch.Size([128, 128])) >>> A_sparse.values() tensor([[1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], ..., [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.], [1., 1., 1., ..., 1., 1., 1.]], device='cuda:0', dtype=torch.float16), >>> A_sparse.indices() tensor([[-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], ..., [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370], [-4370, -4370, -4370, ..., -4370, -4370, -4370]], device='cuda:0', dtype=torch.int16)) zSetting transpose from `to_sparse_semi_structured` is deprecated and will be removed in a future release. `SparseSemiStructuredTensor` only support contiguous input tensors.r-r.)r) r3r4 FutureWarningrrr7sparserrr)rrrSPARSE_SUBCLASSs r@rr5sdh  R    & 4 4  66 \\ > >  % %o % EErCc ^\rSrSrSrSr\R\"SSSS5\R\"SSSS5\R\"SSSS5\R\"SSS S 50r \ \R4S \R S \S S4S jj5rU4Sjr\ SS \R S S4Sjj5rSSS.S\R S\R S-S\S \R 4SjjrSrU=r$)ri|a This class implements semi-structured sparsity for the CUTLASS backend. In this implementation, the specified elements and metadata are stored separately, in packed and meta respectively. When _FORCE_CUTLASS is set, or when cuSPARSELt is not available, this subclass calls into _sparse_semi_structured_(mm|addmm) and sparse_semi_structured_from_dense for conversion to the compressed format. cutlass @rrrDc URU5 [U5unnU"URUUSSSURS9$)Nr#r$r%r&r'r+)rrr*r+)r=rrsparse_tensor_cutlassmeta_tensor_cutlasss r@r,SparseSemiStructuredTensorCUTLASS.from_densesW ,,_= 6o F !   ! !($(,)77  rCc>URb URc [S5eURRS:Xa [ URUR5$[ TU]5$)Nz meta and packed must not be Noner-)r$r#rHndimrsuperr)rKrIs r@r*SparseSemiStructuredTensorCUTLASS.to_densesc 99  3 !CD D yy~~" 4    !#  rCrc d[R"XSS9unnnnnU"URUUUUUSS9$)aF This function takes in a unpruned dense tensor and runs a (branchless) static sort across a 4x4 tile. It greedily picks the largest values in the tile, upholding the 2:4 sparsity constraint across both rows and columns. The algorithm used to prune the matrix is implemented in `_sparse_semi_structured_tile`. Then it creates the packed and meta tensors for the compressed sparse representation of the pruned dense tensor. It also calculates the packed_t and meta_t tensors for the compressed sparse representation of the transposed pruned dense tensor. Since we cannot transpose the compressed representations, we store both for the fw/bw pass respectively. Finally, this function also computes a compressed swizzled bitmask that encodes the sparsity pattern This can be used in the backward pass to mask the gradients. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to CUTLASS semi-structured -> packed [1 2 6 2] [0 0 6 2] -> metadata -> pack to transposed CUTLASS -> packed_t semi-structured representation -> metadata_t -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same five outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUTLASS from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cutlass, meta_cutlass = sparse_semi_structured_from_dense_cutlass(pruned) packed_t_cutlass, meta_t_cutlass = sparse_semi_structured_from_dense_cutlass( pruned.t().contiguous() ) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUTLASS( dense.shape, packed_cutlass, meta_cutlass, packed_t_cutlass, meta_t_cutlass, bitmask, ) ``` T algorithm use_cutlassFr)r7_sparse_semi_structured_tiler*r=rrr#r$r%r&r's r@prune_dense_static_sort9SparseSemiStructuredTensorCUTLASS.prune_dense_static_sortsVz  . . d      '   ! !(C  rCNFrshould_transpose_denserrrc .[U[5(a [S5eURRnUR S:wdUR S:wa[ SUS35eURb URc[ SUS35e[5 URURn[RRRUURURUUR SUR"UR$U5$)NZ`SparseSemiStructuredTensor @ SparseSemiStructuredTensor` is not supported by the hardwarer-`)` matmul: Broadcasting is not implemented$` matmul: operation is not supportedr) isinstancerr:rIrJrrer#r$_ensure_cutlass_mm_registeredrr1r7rmsemi_structured cutlass_mmr*dense_min_rowsdense_min_cols)rKrrrricls_name constraintss r@r%SparseSemiStructuredTensorCUTLASS._mms a3 4 4l >>** 99>QVVq[%H:FG  ;; $))"3%H:AB  * +77@K99,,77   1 ****&  rCr)rJrrrrr!r7int8rfloat16bfloat16float32rrrrr;rrrrrrr __classcell__)rIs@r@rr|s( G 22sBC 5b"aC 6r2q!D 5b"aC  1@@    -   *  68I #llI %I I ^%)', ! <<!llT! ! !% ! !!rCrc \rSrSrSrSr\R\"SSSS5\R\"SSSS5\R\"SSSS5\R\"SSSS50r \ \R4S\R S\S S4S jj5r\ SS\R S S 4S jj5rS SS.S\R S\R S -S\S \R 4SjjrSrg )ri"ab The cuSPARSELt backend expects the specified elements and the metadata to be stored in a single tensor: packed = [ specified elements of original tensor | metadata ] For an original tensor of size (m, k) we expect the first m * k // 2 elements to be the kept elements The rest of the tensor is metadata. Since there is only one tensor, we only use the packed and packed_t attributes respectively. cuSPARSELt also supports transposition fusion, which is necessary for performant 2:4 sparse training, as well as specifying alg_id, a config that affects the performance of the matmul depending on matmul sizes. cusparseltrrrrrrDc URU5 U"UR[R"U5SSSS[R UUR S9 $)Nr])rr*r7_cslt_compressrrr+rs r@r/SparseSemiStructuredTensorCUSPARSELT.from_dense6sW ,,_=!''''8(,&@&P&P$)77  rCrc [R"XSS9unnnnnURURSS5nURURSS5nU"URUUUUUSS9$)am This function does the same thing as described in SparseSemiStructuredCUTLASS, but uses the cuSPARSELt metadata layout and sparse matmul. The only functional difference is that cuSPARSELt stores `metadata` and `packed` together into a single tensor. [9 1 7 4] [9 0 7 0] [1 2 3 0] [0 2 0 0] [8 3 5 4] -> prune 4x4 tile -> [8 0 0 4] -> pack to cuSPARSELT semi-structured -> packed [1 2 6 2] [0 0 6 2] -> pack to transposed cuSPARSELt -> packed_t semi-structured representation -> compute swizzled bitmask -> compressed_swizzled_bitmask The equivalent PyTorch code to create the same three outputs from the dense tensor can be found below: ``` from torch.sparse import SparseSemiStructuredTensorCUSPARSELT from torch.sparse._semi_structured_conversions import ( _sparse_semi_structured_tile, _compute_compressed_swizzled_bitmask, ) pruned = _sparse_semi_structured_tile(dense) packed_cusparselt = torch._cslt_compress(pruned) packed_t_cusparselt = torch._cslt_compress(pruned.t().contiguous()) bitmask = _compute_compressed_swizzled_bitmask(pruned) SparseSemiStructuredTensorCUSPARSELT( dense.shape, packed_cutlass, None, packed_t_cutlass, None, bitmask ) ``` Frrrr)r7rrur*rs r@r  ! !(C  rCNFrrrrc T[U[5(a [S5eURS:wdURS:wa#[ SUR R S35eURUR:wai[ SUR R S[UR5S[UR5SURSURS 3 5eUbURUR:wai[ SUR R S[UR5S[UR5S URS URS 3 5eUR[R:Xa\[ SUR R S[UR5S[UR5S URS3 5eURc#[ SUR R S35e[5 URURn[RR R#UURUURSUR$UR&UR(UR*U5 $)Nrr-rrz` matmul: trying to do `A=z @ B=z`, with A.dtype=z and B.dtype=zH. This operation is only supported when A and B have the same data type.z + C`, with A.dtype=B.dtype=z and C.dtype=zK. This operation is only supported when A, B and C have the same data type.z`, with A.dtype=B.dtype=zO. mm is not supported for float8_e4m3fn, please use `torch._scaled_mm` instead.rr)rrr:rrerIrJr1rr*r7 float8_e4m3fnr# _ensure_cusparselt_mm_registeredrrmr cusparselt_mmrrr(r))rKrrrrirs r@r(SparseSemiStructuredTensorCUSPARSELT._mmsz a3 4 4l  99>QVVq[%DNN++,,UV  77djj %DNN++,,FuTZZGXFYY^_defelel_m^no $ |= BYY    djj 8%DNN++,,FuTZZGXFYY^_defelel_m^no((, |= J\\  ::,, ,%DNN++,,FuTZZGXFYY^_defelel_m^no((, |4``  ;; %DNN++,,PQ  - .77@K99,,::  1 ****..&&&  rCrr)rJrrrrr!r7rrrrrrrrrr;rrrrrrrrCr@rr"s  G ;BBK 22r2rB 5b"aC 6r2q!D  1@@    0   &68> #ll> %> > H%)', 4 <<4llT! 4 !% 4 44rCrc[(agSqSSKJn U"SSS9S[RS [RS [RS [RS-S [ S [ S[ S[ S[R4Sj5nURS[RS [RS [RS [RS-S [ S [ S[ S[ S[R4Sj5ng)zLazily register the cutlass_mm custom op. Registration is deferred to avoid importing torch.library at module load time, since torch.sparse is imported early during ``import torch``. NTr custom_opzsemi_structured::cutlass_mmr mutates_argsdenser#r$r out_featuresrrrrDcURupU*U-n U *U-n U S:g=(d U S:gn Un U (a.[RRR USU SU 45n U(aU R 5OU nUc[R "XU5nO[R"X1X.5nU (a<U(aUOU nUSURSSU5R[RS9$UR5$)Nrr memory_format) r*r7nn functionalpadrt_sparse_semi_structured_mm_sparse_semi_structured_addmmnarrowr|contiguous_format contiguous)rr#r$rrrrrrrto_pad_mto_pad_nneed_pad dense_paddedmm_inputresout_colss r@r1_ensure_cutlass_mm_registered..cutlass_mms{{B(?B(?q=1HM  88..2251h8:TUL'=<>>#< <226JC55dDSC 2qHM\"1h'U%<%<=  ~~rCtranspose_densecU(aURSOURSn[R"UUURURS9$Nrrrr*r7emptyr1r0) rr#r$rrrrrrs r@_cutlass_mm_fake7_ensure_cutlass_mm_registered.._cutlass_mm_fakesB&55;;q>%++a.{{  ++<<   rC)_cutlass_mm_registered torch.libraryrr7r;rr register_fake)rrrs r@rrs(!',2> ||   ll llT!       !%   ? > ||   ll llT!           rCc[(agSqSSKJn U"SSS9SS[RS [RS [RS-S [ S [ S [ S[ S[ S[ S[R4Sjj5nURS[RS [RS [RS-S [ S [ S [ S[ S[ S[ S[R4Sj5ng)z,Lazily register the cusparselt_mm custom op.NTrrzsemi_structured::cusparselt_mmrrrr#rrrrfuse_transposerrrDc URupU *U-n U *U-n U S:g=(d U S:gn UnU (a.[RRR USU SU 45nU(aUR 5OUn[R "UUUUUS9nU(aUR 5nU (a9U(aU OU nURSSU5R[RS9$UR5$)Nr)rtranspose_resultrrr) r*r7rrrrt_cslt_sparse_mmrr|rr)rr#rrrrrrrrrrr r r r r rs r@r7_ensure_cusparselt_mm_registered..cusparselt_mms{{B(?B(?q=1HM  88..2251h8:TUL'=<>>#<##  +   %%'C 2qH::aH-33#554 ~~rCc U(aURSOURSn [R"UU URURS9$rr) rr#rrrrrrrrs r@_cusparselt_mm_fake=_ensure_cusparselt_mm_registered.._cusparselt_mm_fake6sB&<5;;q>Q{{  ++<<   rC)F)_cusparselt_mm_registeredrrr7r;rrr)rrr!s r@rr s;!  $'/bA(-! ||!  ! llT!!  !  !  ! ! ! !%!  ! B! F   ||   llT!         !%   ! rC)'r3 collectionsrcollections.abcrtypingrr7)torch.sparse._semi_structured_conversionsrr!torch.sparse._semi_structured_opsrr r r r r rrrrrrr__all__rr;rrrrrrrrrr#rrrCr@r*s"$ " ",$C" H"H"Z,<<DF\\DFDF DF DFNc(BcL]+E]@> B"? rC