3j:SSKJr SSKrSSKJr SSKJr SSKJr SSKJ r SSK J r J r J r SS KJr SS KJrJrJrJr SS KJr SS KJr S SKJr S SKJrJrJrJr SSK J!r! "SS\5r""SS\5r#"SS\5r$\"SS9"SS\55r%"SS\RL5r'\"SS9\"SS \ 555r(\"S!S9"S"S#\$55r)\"S$S9"S%S&\$\ 55r*/S'Qr+g)() dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)merge_with_config_defaults)capture_outputs) AutoModel)Qwen2AudioAttentionQwen2AudioEncoderQwen2AudioEncoderLayerQwen2AudioPreTrainedModel) VoxtralConfigc\rSrSrSrg)VoxtralAttention+N__name__ __module__ __qualname____firstlineno____static_attributes__re/home/wildlama/miniconda3/lib/python3.13/site-packages/transformers/models/voxtral/modular_voxtral.pyrr+r%rc\rSrSrSrg)VoxtralEncoderLayer/rNrrr%r&r)r)/r'r%r)c(\rSrSrSrSrSrSrSrSr g)VoxtralPreTrainedModel3TNr) r r!r"r#_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraph_no_split_modulesr$rr%r&r,r,3s  "&!r%r,z: The Voxtral encoder, which is a Whisper encoder.  custom_introcV\rSrSr\\S.r\\SS\ \ S\ \ -4Sjj55r Srg) VoxtralEncoder<) attentions hidden_statesNkwargsreturnc URRURRS-URRS-nUR SU:wa"[ SUSUR SSUS35eURURRRURRRS9n[RRURU55n[RRUR U55nURSSS 5nURRnXV-RUR5n[RR!XpR UR"S 9n[%UR&5H upU "UUS 9nM UR)U5n[+US 9$) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rz7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .dtypedevicerr)ptraining)attention_mask)last_hidden_state)configmax_source_positionsconv1strideconv2shape ValueErrortoweightr@rAr functionalgelupermuteembed_positionsdropoutrC enumeratelayers layer_normr ) selfinput_featuresrDr:expected_seq_length inputs_embeds embed_posr9idx encoder_layers r&forwardVoxtralEncoder.forwardGs(#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :IJ]I^^jkykkAClDkEErsFrGGHI (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --m||VZVcVc-d "+DKK"8 C)-M#9  6 )+  r%rN)r r!r"r#rr)_can_record_outputsrrr r tupler r^r$rr%r&r6r6<sS',  + +, + + + +  + r%r6c6^\rSrSrS\4U4SjjrSrSrU=r$)VoxtralMultiModalProjectorwrFc^>[TU]5 [R"URR UR RSS9Ul[URUl [R"UR RUR RSS9Ul gNF)bias) super__init__rLinear audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rWrF __class__s r&rj#VoxtralMultiModalProjector.__init__xsz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r%clURU5nURU5nURU5nU$r`)rprrrs)rWaudio_featuresr9s r&r^"VoxtralMultiModalProjector.forward~s2 n5 /  m4 r%)rrrprs) r r!r"r#rrjr^r$ __classcell__rus@r&rdrdwsn}n r%rdzL Base class for Voxtral outputs, with hidden states and attentions. cB\rSrSr%SrSr\RS-\S'Sr g)VoxtralModelOutputWithPastz[ audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_statesr) r r!r"r#__doc__rtorch FloatTensor__annotations__r$rr%r&r}r}s 59**T18r%r}z The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model, without a language modeling head. c^\rSrSrU4Sjr\\"SS9S\RS\ \ S\ \ -4Sj55r S \RS \RS \R4S jr\\SS \RS -S\RS -S\R S -S\RS -S\S -S \RS -S\S -S\ \ S\ \-4Sjj55rSrU=r$) VoxtralModelc>[TU]U5 [R"UR5Ul[R"UR 5Ul[U5Ul UR5 gr`) rirjr from_configrl audio_towerrnlanguage_modelrdmulti_modal_projector post_initrts r&rjVoxtralModel.__init__sY  $001D1DE'33F4F4FG%?%G" r%zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r3rXr:r;c UR"U4SS0UD6nURnURSURRR 5nUR U5nXSlU$)a) input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr=)rrEreshaperFrlrmr pooler_output)rWrXr: audio_outputsr audio_embedss r&get_audio_featuresVoxtralModel.get_audio_featuressj((TTTVT +==199"dkk>V>V>h>hi112EF &2#r% input_idsrZrxc &UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnUR5nURSnURS5RU5RUR 5n[X$R5UR5:HSUSU35 U$)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. r?r=rz6Audio features and audio tokens do not match, tokens: z , features: )get_input_embeddingsrtensorrFaudio_token_idlongrAallsumrK unsqueeze expand_asrMrnumel)rWrrZrxspecial_audio_maskn_audio_tokensn_audio_featuress r&get_placeholder_mask!VoxtralModel.get_placeholder_masks  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H +//1)//2/99"=GG VYYZgZnZno  - 3 3 59M9M9O OD^DTT`aq`r s "!r%NrD position_idspast_key_values use_cachec UcUR5"U5nSn UbWUbTURUSS9Rn URXU S9n UR XR UR 55nUR"SUUUUUS.UD6n [U RU RU RU RU S9$)NT)r)rZrx)rDrrrZr)rErr9r8rr) rrrrmasked_scatterrMrArr}rErr9r8) rWrrXrDrrrZrr:rroutputss r&r^VoxtralModel.forwards   557 BM  %)*?22>t2TbbL"&!:!:|";" *889K__]j]q]qMrsM+/+>+>, )%+' ,  , *%77#33!//)) ,   r%)rrr)NNNNNNN)r r!r"r#rjrrrrr r rbr r LongTensorrTensorrboolr}r^r$rzr{s@r&rrs\w#//;ABT;U + +&"))":?:K:K"]b]n]n"0.237.204(,26!%' ##d*' ))D0'  t+ ' &&- '  ' ((4/' $;' +,'  + +' ' r%rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model. cn^\rSrSrS/rU4SjrSr\\SS\ RS-S\ RS-S\ RS-S \ RS-S \ S-S \ RS-S \ RS-S \S-S\\ R-S\\S\\-4Sjj55rU4SjrSrU=r$)VoxtralForConditionalGenerationrRc>[TU]U5 [U5Ul[R "UR RUR RSS9Ul UR5 grg) rirjrmodelrrkrnro vocab_sizelm_headrrts r&rj(VoxtralForConditionalGeneration.__init__sS  !&) yy!3!3!?!?ASASA^A^ejk  r%c:URR"U0UD6$r`)rr)rWargsr:s r&r2VoxtralForConditionalGeneration.get_audio_features szz,,d=f==r%NrrXrDrrrZlabelsrlogits_to_keepr:r;c UR"SUUUUUUUS.U D6n U Rn [U [5(a [ U *S5OU n UR U SS2U SS245nSnUb3UR "SXURRRS.U D6n[UUU RU RU RS9$)a Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)rrXrDrrrZrN)logitsrr)lossrrr9r8r)rrE isinstanceintslicer loss_functionrFrnrr rr9r8)rWrrXrDrrrZrrrr:rr9 slice_indicesrrs r&r^'VoxtralForConditionalGeneration.forwards`**  ))%+'     11 8B>SV8W8W~ot4]k mA}a,?@A  %%9P9P9[9[_eD&#33!//))   r%c>URSS5nURSS5n[TU] "U0UD6nU(dURSS5(dX5S'U$)NrXis_first_iterationFrT)popgetriprepare_inputs_for_generation)rWrr:rXr model_inputsrus r&r=VoxtralForConditionalGeneration.prepare_inputs_for_generation[s^ $4d;#ZZ(.237.204(,26*.!%-.I ##d*I ))D0I  t+ I &&- I  I ((4/I   4'I $;I ell*I +,I  ' 'I I V  r%r)r,r6rr), dataclassesrrr activationsr cache_utilsr generationrmodeling_outputsr r r processing_utilsr utilsr rrr utils.genericrutils.output_capturingrautor qwen2_audio.modeling_qwen2_audiorrrrconfiguration_voxtralrrr)r,r6Modulerdr}rr__all__rr%r&rs2 " ! ) 'aa75 1 *  0 6 3 &3  3 l    9!89  9 ` )`  ` F e&