3j] S SK JrJr S SKrS SKJr SSKJr SSKJ r SSK
J r SSK J
r
SS KJrJrJr SS
KJrJr SS KJr SS KJrJrJr SS
KJ r J r SSK J r SSK J!r! " S S\RD 5 r# " S S\RD 5 r$ S/S\RD S\RJ S\RJ S\RJ S\RJ S- S\&S- S\&S \\ 4S jjr' " S S \RD 5 r( " S S!\RD 5 r) " S" S#\
5 r* " S$ S%\RD 5 r+\ " S& S'\5 5 r,\ " S( S)\,5 5 r-\" S*S+9 " S, S-\,5 5 r./ S.Qr/g)0 )CallableIterableN)nn )initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs ) VivitConfigc n ^ \ rS rSrSrS\4U 4S jjrS\R S\R 4S jr Sr
U =r $ ) VivitTubeletEmbeddings' ae
This class turns `pixel_values` of shape `(batch_size, num_frames, num_channels, height, width)` into the initial
`hidden_states` (tubelet embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a
Transformer encoder.
The seq_length equals (num_frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]).
configc N > [ TU ] 5 UR nUR n[ U[
5 ( a UOX34nUR US - US US - - US US - - U l X0l [ R " UR UR X"S9U l g )Nr r ) kernel_sizestride)
super__init__ tubelet_size
image_size
isinstancer
num_frames num_patchesr Conv3d num_channels hidden_size
projection)selfr r# r$ __class__s b/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/vivit/modeling_vivit.pyr" VivitTubeletEmbeddings.__init__0 s
** &&
#-j(#C#CZ*Ia
,q/
1!} Q/
1!} Q/
1
% )) !3!3
pixel_valuesreturnc UR SS5 nU R U5 R S5 R SS5 $ )Nr r ) transposer+ flatten)r, r1 s r. forward VivitTubeletEmbeddings.forward@ s; #--a3 |,44Q7AA!QGGr0 )r$ r' r+ ) __name__
__module__ __qualname____firstlineno____doc__r r" torchTensorr6 __static_attributes__
__classcell__r- s @r. r r ' s8
{
HELL HU\\ H Hr0 r c ^ \ rS rSrSrS\4U 4S jjrS\R S\ S\ S\R 4S jr
SS
\R S \ S\R 4S jjr S
r
U =r$ )VivitEmbeddingsF zQ
Construct the CLS token, position and tubelet patch embeddings for video input.
r c > [ TU ] 5 [ R " [ R
" SSUR 5 5 U l [ U5 U l U R R n[ R " [ R
" SUS- UR 5 5 U l [ R " UR 5 U l UR SS U l U R R" U l g )Nr )r! r" r Parameterr= zerosr* cls_tokenr patch_embeddingsr' position_embeddingsDropouthidden_dropout_probdropoutr#
patch_sizer$ )r, r r' r- s r. r" VivitEmbeddings.__init__K s
ekk!Q8J8J&KL 6v > ++77 #%<< A{QPVPbPb0c#d zz&"<"<= --ab1 //::r0
embeddingsheightwidthr2 c U R R nU R R S S-
n[ R
R
5 ( d XE:X a X#:X a U R $ U R SS2SS24 nU R SS2SS24 nUR S nX R S - n X0R S - n
[ US- 5 n UR SXU5 nUR SSSS5 n[ R R UX4SS S
9nUR SSSS5 R SSU5 n[ R " Xg4SS 9$ ) a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r Nr g ?r r bicubicF)sizemode
align_cornersdim)rI r' rJ shaper= jit
is_tracingrN r reshapepermuter
functional interpolateviewcat) r, rP rQ rR r'
num_positionsclass_pos_embedpatch_pos_embedrZ
new_height new_widthsqrt_num_positionss r. interpolate_pos_encoding(VivitEmbeddings.interpolate_pos_encodingW sY ++77 0066q9A=
yy##%%+*F6?++ + 221bqb59 221ab59 r"q 11
__Q // &}c'9 :)11!5G]`a)11!Q1= --33 ( 4
*11!Q1=BB1b#Nyy/;CCr0 r1 rj c UR u p4pVnU R U5 nU R R USS5 n [ R
" X4SS9nU( a XR
XU5 - nOdX`R S :w d XpR S :w a2 [ SU SU SU R S SU R S S3 5 eXR - nU R U5 nU$ ) NrT r rY r zInput image size (*z) doesn't match model (z).) r[ rI rH expandr= rc rj r$
ValueErrorrJ rM )
r, r1 rj
batch_sizer& r) rQ rR rP
cls_tokenss
r. r6 VivitEmbeddings.forward~ s >J>P>P;
e**<8
^^**:r2>
YY
7Q?
##&C&CJX]&^^J ++u8J/J (% 9 +,Adooa.@-AE $&>&>>J\\*-
r0 )rH rM r$ rI rN rJ )F)r8 r9 r: r; r< r r" r= r> intrj boolr6 r? r@ rA s @r. rC rC F sq
;{
;%D5<< %D %DUX %D]b]i]i %DNELL D ]b]i]i r0 rC modulequerykeyvalueattention_maskscalingrM kwargsc Uc UR S5 S- n[ R " XR SS5 5 U- nUb X- n[ R
R
US[ R S9R UR 5 n[ R
R XU R S9n[ R " X5 n U R SS5 R 5 n X4$ )NrT r r )rZ dtype)ptrainingr )
rV r= matmulr4 r r` softmaxfloat32tor~ rM r
contiguous)
ru rv rw rx ry rz rM r{ attn_weights attn_outputs
r. eager_attention_forwardr s **R.D( <<}}Q':;gEL!#4 ==(( 2U]](SVVW\WbWbcL==(( 6??([L,,|3K ''1-88:K $$r0 c ^ \ rS rSrS\4U 4S jjr S S\R S\R S- S\\ S\
\R \R 4 4S jjr S
r U =r
$ ) VivitAttention r c > [ TU ] 5 Xl UR U l [ USUR
UR - 5 U l UR U l U R S- U l SU l
[ R " UR
UR U R - UR S9U l [ R " UR
UR U R - UR S9U l [ R " UR
UR U R - UR S9U l [ R " UR U R - UR
SS9U l g )Nhead_dimr} F)biasT)r! r" r num_attention_headsgetattrr* r attention_probs_dropout_probattention_dropoutrz is_causalr Linearqkv_biasq_projk_projv_projo_projr, r r- s r. r" VivitAttention.__init__ s*
#)#=#=
F4F4F&JdJd4de
!'!D!D }}d* ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii 2 2F4N4NQUQ^Q^4^eketetu ii : :T]] JFL^L^eij r0 N
hidden_statesry r{ r2 c UR S S n/ UQSPU R P7nU R U5 R U5 R SS5 nU R U5 R U5 R SS5 nU R
U5 R U5 R SS5 n[ R " U R R [ 5 n U " U UUUU4U R ( d SO U R U R S.UD6u pU
R " / UQSP76 R! 5 n
U R# U
5 n
X4$ )NrT r r )rM rz )r[ r r rb r4 r r r
get_interfacer _attn_implementationr r r rz r^ r r ) r, r ry r{ input_shape hidden_shape query_states
key_states value_statesattention_interfacer r s r. r6 VivitAttention.forward sE $))#2. 88b8$--8 {{=166|DNNqRST [[ /44\BLLQPQR
{{=166|DNNqRST (?(M(M KK , ,.E)
%8 %
$}}C$2H2HLL %
%
! "));;;;FFH kk+. ((r0 )
r r r r r r r r rz r N)r8 r9 r: r; r r" r= r> r r tupler6 r? r@ rA s @r. r r sk k{ k" /3 ) || ) t+ ) +, )
u||U\\) * ) )r0 r c j ^ \ rS rSrS\4U 4S jjrS\R S\R 4S jrSr U =r
$ )VivitMLP r c > [ TU ] 5 Xl [ UR U l [ R " UR UR 5 U l
[ R " UR UR 5 U l g r ) r! r" r r
hidden_act
activation_fnr r r* intermediate_sizefc1fc2r s r. r" VivitMLP.__init__ sb
#F$5$5 699V / /1I1IJ99V 5 5v7I7IJr0 r r2 c l U R U5 nU R U5 nU R U5 nU$ r )r r r )r, r s r. r6 VivitMLP.forward s4 /
**=9
/
r0 )r r r r r8 r9 r: r; r r" r= r> r6 r? r@ rA s @r. r r s1 K{ K U\\ ell r0 r c ^ \ rS rSrS\4U 4S jjr S S\R S\R S- S\\ S\R 4S jjr
S
r U =r $ )
VivitLayer r c d > [ TU ] 5 [ U5 U l [ R
" UR UR S9U l [ R
" UR UR S9U l [ U5 U l [ R " UR 5 U l g )Neps)r! r" r attentionr LayerNormr* layer_norm_epslayernorm_beforelayernorm_afterr mlprK rL rM r s r. r" VivitLayer.__init__ sz
'/ " V-?-?VEZEZ [ !||F,>,>FDYDY Z F#zz&"<"<= r0 Nr ry r{ r2 c UnU R U5 nU R " X40 UD6u pU R U5 nX- nUnU R U5 nU R U5 nU R U5 nX- nU$ r )r r rM r r )r, r ry r{ residual_s r. r6 VivitLayer.forward s ! --m<
>>-R6R
]3
%0
! ,,];
/
]3
%0
r0 )r rM r r r r )
r8 r9 r: r; r r" r= r> r r r6 r? r@ rA s @r. r r s[ >{ > /3 || t+ +,
r0 r c j ^ \ rS rSrS\4U 4S jjrS\R S\R 4S jrSr U =r
$ ) VivitPooleri r c > [ TU ] 5 [ R " UR UR
5 U l [ UR U l g r )
r! r" r r r* pooler_output_sizedenser
pooler_act
activationr s r. r" VivitPooler.__init__ s>
YYv 1 163L3LM
!2!23r0 r r2 c \ US S 2S4 nU R U5 nU R U5 nU$ )Nr )r r )r, r first_token_tensor
pooled_outputs r. r6 VivitPooler.forward s6 +1a4 0
#56
6
r0 )r r r rA s @r. r r s/ 4{ 4
U\\ ell r0 r c ^ \ rS rSr% \\S' SrSrSrSr SS/r
Sr Sr Sr
SrSr\\S .rS
r\R* " 5 U 4S j5 rS rU =r$ )
VivitPreTrainedModeli! r vivitr1 )videoTrC r )r
attentionsrI c > [ TU ] U5 [ U[ 5 ( aA [ R
" UR 5 [ R
" UR 5 gg)zInitialize the weightsN)r!
_init_weightsr% rC initzeros_rH rJ )r, ru r- s r. r "VivitPreTrainedModel._init_weights4 sH f% fo . . KK(( ) KK22 3 /r0 )r8 r9 r: r; r __annotations__base_model_prefixmain_input_nameinput_modalities supports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backend_can_compile_fullgraphr r _can_record_outputs_input_embed_layerr= no_gradr r? r@ rA s @r. r r ! sv $O!&*#*L9N "& !#$ ,
]]_4 4r0 r c ^ \ rS rSrSS\S\4U 4S jjjr\\" SS9\ SS\
R S- S \S
\
R S- S \
\ S \4
S
jj5 5 5 rSrU =r$ )
VivitModeli= r add_pooling_layerc > [ TU ] U5 Xl [ U5 U l [
R " [ UR 5 Vs/ s H n[ U5 PM sn5 U l
[
R " UR UR S9U l U( a [ U5 OSU l U R# 5 gs snf )z^
add_pooling_layer (bool, *optional*, defaults to `True`):
Whether to add a pooling layer
r N)r! r" r rC rP r
ModuleListrangenum_hidden_layersr layersr r* r layernormr pooler post_init)r, r r r r- s r. r" VivitModel.__init__? s
)&1mmvG_G_A`$aA`AZ%7A`$ab f&8&8f>S>ST->k&)D %bs CF)tie_last_hidden_statesNr1 rj ry r{ r2 c U R XS9n[ U R UUS9nUnU R H nU" Xc40 UD6nM U R U5 nU R
b U R U5 OSn [
XS9$ )a(
Examples:
```python
>>> import av
>>> import numpy as np
>>> from transformers import VivitImageProcessor, VivitModel
>>> from huggingface_hub import hf_hub_download
>>> np.random.seed(0)
>>> def read_video_pyav(container, indices):
... '''
... Decode the video with PyAV decoder.
... Args:
... container (`av.container.input.InputContainer`): PyAV container.
... indices (`list[int]`): List of frame indices to decode.
... Returns:
... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
... '''
... frames = []
... container.seek(0)
... start_index = indices[0]
... end_index = indices[-1]
... for i, frame in enumerate(container.decode(video=0)):
... if i > end_index:
... break
... if i >= start_index and i in indices:
... frames.append(frame)
... return np.stack([x.to_ndarray(format="rgb24") for x in frames])
>>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
... '''
... Sample a given number of frame indices from the video.
... Args:
... clip_len (`int`): Total number of frames to sample.
... frame_sample_rate (`int`): Sample every n-th frame.
... seg_len (`int`): Maximum allowed index of sample's last frame.
... Returns:
... indices (`list[int]`): List of sampled frame indices
... '''
... converted_len = int(clip_len * frame_sample_rate)
... end_idx = np.random.randint(converted_len, seg_len)
... start_idx = end_idx - converted_len
... indices = np.linspace(start_idx, end_idx, num=clip_len)
... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
... return indices
>>> # video clip consists of 300 frames (10 seconds at 30 FPS)
>>> file_path = hf_hub_download(
... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
... )
>>> container = av.open(file_path)
>>> # sample 32 frames
>>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames)
>>> video = read_video_pyav(container=container, indices=indices)
>>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400")
>>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400")
>>> # prepare video for the model
>>> inputs = image_processor(list(video), return_tensors="pt")
>>> # forward pass
>>> outputs = model(**inputs)
>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 3137, 768]
```)rj )r
inputs_embedsry N)last_hidden_state
pooler_output)rP r r r r r r )
r, r1 rj ry r{ embedding_outputr layersequence_outputr s
r. r6 VivitModel.forwardM s j ?? r r r r6 r? r@ rA s @r. r r = s { t E2 26)..2 ^j''$.^j #'^j t+ ^j
+, ^j
$
^j 3 ^jr0 r a
ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the
[CLS] token) e.g. for Kinetics-400.
Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by
setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained
position embeddings to the higher resolution.
) custom_introc
^ \ rS rSrS\4U 4S jjr\\ S S\R S- S\R S- S\ S\ \
S \4
S
jj5 5 rS rU =r$ )
VivitForVideoClassificationi r c . > [ TU ] U5 UR U l [ USS9U l UR S: a+ [
R " UR UR 5 O[
R " 5 U l U R 5 g )NF)r r ) r! r"
num_labelsr r r r r* Identity
classifierr r s r. r" $VivitForVideoClassification.__init__ ss
++ %@
OUN_N_bcNc"))F$6$68I8IJikititiv
r0 Nr1 labelsrj r{ r2 c U R " U4SU0UD6nUR nU R USS2SSS24 5 nSnUb U R " X'U R 40 UD6n[ UUUR UR S9$ )a
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the image classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
Examples:
```python
>>> import av
>>> import numpy as np
>>> import torch
>>> from transformers import VivitImageProcessor, VivitForVideoClassification
>>> from huggingface_hub import hf_hub_download
>>> np.random.seed(0)
>>> def read_video_pyav(container, indices):
... '''
... Decode the video with PyAV decoder.
... Args:
... container (`av.container.input.InputContainer`): PyAV container.
... indices (`list[int]`): List of frame indices to decode.
... Returns:
... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3).
... '''
... frames = []
... container.seek(0)
... start_index = indices[0]
... end_index = indices[-1]
... for i, frame in enumerate(container.decode(video=0)):
... if i > end_index:
... break
... if i >= start_index and i in indices:
... frames.append(frame)
... return np.stack([x.to_ndarray(format="rgb24") for x in frames])
>>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len):
... '''
... Sample a given number of frame indices from the video.
... Args:
... clip_len (`int`): Total number of frames to sample.
... frame_sample_rate (`int`): Sample every n-th frame.
... seg_len (`int`): Maximum allowed index of sample's last frame.
... Returns:
... indices (`list[int]`): List of sampled frame indices
... '''
... converted_len = int(clip_len * frame_sample_rate)
... end_idx = np.random.randint(converted_len, seg_len)
... start_idx = end_idx - converted_len
... indices = np.linspace(start_idx, end_idx, num=clip_len)
... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64)
... return indices
>>> # video clip consists of 300 frames (10 seconds at 30 FPS)
>>> file_path = hf_hub_download(
... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset"
... )
>>> container = av.open(file_path)
>>> # sample 32 frames
>>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames)
>>> video = read_video_pyav(container=container, indices=indices)
>>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400")
>>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400")
>>> inputs = image_processor(list(video), return_tensors="pt")
>>> with torch.no_grad():
... outputs = model(**inputs)
... logits = outputs.logits
>>> # model predicts one of the 400 Kinetics-400 classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
LABEL_116
```rj Nr )losslogitsr r )r r r
loss_functionr r
r r ) r, r1 r rj r{ outputsr r r
s r. r6 #VivitForVideoClassification.forward s x $(:: $
3K$
OU$
"33 Aq!9: %%fdkkLVLD$!// ))
r0 )r r r )NNF)r8 r9 r: r; r r" r r r= r
LongTensorrt r r r
r6 r? r@ rA s @r. r r s
{
26*.). i
''$.i
4'i
#' i
+, i
i
i
r0 r )r r r )Nr )0collections.abcr r r= r r r activationsr
masking_utilsr modeling_layersr
modeling_outputsr r r
modeling_utilsr r processing_utilsr utilsr r r
utils.genericr r utils.output_capturingr configuration_vivitr Moduler rC r> floatr r r r r r r r __all__r r0 r. r" s * / & ! 6 9 b b F & B B I 5 , HRYY H>Lbii Lj !% II% <<%
% << %
LL4' % T\
% % '(%8.)RYY .)b
ryy
+ @ ")) 4? 4 46 pj% pj pjf
x
"6 x
x
v Pr0