I need to understand the LAM repository (https://github.com/aigc3d/LAM) and OpenAvatarChat integration for a real-time talking avatar setup. Please fetch and read:

1. The LAM repo README (already known: uses Gaussian Splatting, has app_lam.py Gradio app, Audio2Expression module)
2. Look for the OpenAvatarChat repo at https://github.com/aigc3d/OpenAvatarChat - read its README carefully
3. Understand the pipeline: how does audio go in, how does the rendered output come out (WebSocket? HTTP stream? Video frames?)
4. Look for any Python API or importable functions (not just CLI scripts)
5. Look for the Audio2Expression module specifically - what does it accept as input format?

I want to integrate this into a local web app (Python backend + vanilla JS frontend). The goal is:
- User sends text to AI agent → agent responds → TTS generates audio → LAM animates avatar in real-time → rendered in browser canvas

Report:
- The exact pipeline for real-time audio-driven avatar
- How OpenAvatarChat handles the browser rendering (WebRTC? WebSocket? iframe?)
- Whether there's a Python callable API for inference
- Key files/classes to look at for integration


- deep-research: Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report. - When the user wants a deep, multi-source, fact-checked research report on any topic. BEFORE invoking, check if the question is specific enough to research directly — if underspecified (e.g., "what car to buy" without budget/use-case/region), ask 2-3 clarifying questions to narrow scope. Then pass the refined question as args, weaving the answers in.
- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

deep-research update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


ToolSearch

Question: select:WebFetch


I'll fetch both repositories in parallel to gather comprehensive information.


WebFetch

Prompt: Extract all details about: pipeline architecture, Audio2Expression module (inputs/outputs/format), Python API, how audio is processed, rendering pipeline, WebSocket/HTTP/WebRTC interfaces, Gradio app structure, any importable Python classes or functions


The server returned HTTP 404 Not Found.

The response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.


Prompt: Extract all details about: pipeline architecture, how audio goes in and rendered output comes out (WebRTC/WebSocket/HTTP), browser rendering method, Python API, integration with LAM, WebSocket protocol, any importable Python classes or functions, key files and classes


Let me use the GitHub API via `gh` CLI and fetch the actual GitHub pages instead.


Bash

gh api repos/aigc3d/LAM --jq '.description, .default_branch, .html_url' 2>/dev/null && echo "---" && gh api repos/aigc3d/OpenAvatarChat --jq '.description, .default_branch, .html_url' 2>/dev/null

Check both repos exist and get default branch


Exit code 127


WebFetch

Prompt: Extract all details about the repository structure, pipeline architecture, Audio2Expression module, Python API, how audio is processed, rendering pipeline, WebSocket/HTTP/WebRTC interfaces, Gradio app, key files and classes, installation and usage instructions


Prompt: Extract all details about: pipeline architecture, how audio goes in and rendered output comes out (WebRTC/WebSocket/HTTP), browser rendering method, Python API, integration with LAM, WebSocket protocol, any importable Python classes or functions, key files and classes, installation and usage


# LAM Repository Analysis

Based on the provided GitHub page content, here's what can be extracted:

## Repository Overview
LAM (Large Avatar Model) is a SIGGRAPH 2025 project by Alibaba's Tongyi Lab for creating "3D Interactive Chatting Avatar with One Image in Seconds."

## Core Capabilities
The project enables three main features:
- "Ultra-realistic 3D Avatar Creation from One Image in Seconds"
- "Super-fast Cross-platform Animating and Rendering on Any Devices"
- "Low-latency SDK for Realtime Interactive Chatting Avatar"

## Repository Structure
The codebase includes these directories:
- `lam/` - Core model implementation
- `configs/` - Configuration files
- `scripts/` - Installation and inference scripts
- `tools/` - Utility tools including avatar export
- `vhap/` - Related avatar processing
- `external/` - Third-party dependencies
- `assets/images/` - Documentation images

## Key Files
- `app_lam.py` - Gradio application for local use
- `app_hf_space.py` - HuggingFace Space deployment
- `requirements.txt` - Python dependencies

## Installation
Linux users can run: `sh ./scripts/install/install_cu121.sh` (for CUDA 12.1) or `install_cu118.sh` (for CUDA 11.8). Windows installation requires a dedicated guide.

## Audio2Expression Module
The project released an "Audio2Expression model, which can animate the generated LAM Avatar with audio input" (April 19, 2025).

## Deployment Options
- HuggingFace Spaces (avatar generation and interactive chatting)
- ModelScope (equivalent Chinese platform)
- Local Gradio deployment via `python app_lam.py`
- Integration with OpenAvatarChat for WebGL-based interactive conversations

## Performance Metrics
On A100 GPUs: 562.9 FPS for animating/rendering; on mobile (XiaoMi 14): 110+ FPS. Reconstruction time is approximately 1.4 seconds.

## Model Availability
LAM-20K weights are available through HuggingFace and ModelScope, with additional larger variants in development.


Prompt: Extract all details about the repository structure, pipeline, audio input, rendering output, WebRTC/WebSocket, Python API, key files and classes


Prompt: Extract the complete Python code, especially: how audio is fed in, Audio2Expression usage, rendering pipeline, WebSocket/HTTP endpoints, importable functions and classes


Prompt: List all files and directories in the repository root. Extract any README content visible on the page including pipeline details, Audio2Expression module details, and any API information.


Prompt: List all files and directories at the root of the OpenAvatarChat repository


Prompt: List all files at root level of the LAM repository


Prompt: Get repository metadata: description, default branch, topics, url


# LAM Repository Root Level Files

Based on the GitHub API response, the root-level contents of the LAM repository include:

**Files:**
- `.gitignore` – Git ignore configuration
- `LICENSE` – Primary license file (11.3 KB)
- `LICENSE_WEIGHT` – Additional license file for weights (19.3 KB)
- `README.md` – English documentation (7.7 KB)
- `README_CN.md` – Chinese documentation (7.2 KB)
- `app_hf_space.py` – Hugging Face Space application (22.3 KB)
- `app_lam.py` – LAM application file (23 KB)
- `requirements.txt` – Python dependencies (612 bytes)

**Directories:**
- `assets/`, `configs/`, `external/`, `lam/`, `scripts/`, `tools/`, `vhap/`

The repository appears to be a 3D AI generation project with dual licensing (code and model weights handled separately).


Prompt: Return the download_url or content of app_lam.py


Prompt: List all files in the lam directory


# Files in the LAM Directory

Based on the repository content, the `lam` directory contains:

**Files:**
- `__init__.py` (empty Python package file)
- `launch.py` (1,120 bytes)

**Subdirectories:**
- `datasets/`
- `losses/`
- `models/`
- `runners/`
- `utils/`

The structure appears to be a Python package for an AI/3D project, with modular organization for datasets, loss functions, model definitions, training runners, and utility functions.


# app_lam.py Download Information

**Download URL:**
```
https://raw.githubusercontent.com/aigc3d/LAM/master/app_lam.py
```

**File Details:**
- **Repository:** aigc3d/LAM
- **Size:** 23,048 bytes
- **License:** Apache License 2.0
- **Encoding:** Base64 (in the API response)

**Content Summary:**
This is a Gradio-based web application for the LAM (Large Avatar Model) project. The application enables users to upload front-facing images and generates animated video outputs by:

- Processing input images through flame tracking and segmentation
- Estimating flame parameters from the input
- Running inference on a pre-trained LAM model
- Rendering animated avatars based on motion sequences
- Exporting results as video files with audio

The script integrates several computer vision and deep learning tools, including PyTorch, OpenCV, and moviepy for video generation.


Prompt: Extract the COMPLETE Python code. Specifically: all imports, all class definitions, all function definitions (especially audio-related ones), how Audio2Expression is used, what input formats are accepted for audio, how rendering works, any WebSocket or HTTP server code, the Gradio interface structure, any streaming or real-time processing code


Prompt: List all files in the lam/models directory


# Files in the lam/models Directory

Based on the repository contents, the lam/models directory contains:

**Files:**
- `__init__.py` (685 bytes)
- `block.py` (5,152 bytes)
- `discriminator.py` (4,479 bytes)
- `modeling_lam.py` (17,198 bytes)
- `modulate.py` (1,427 bytes)
- `transformer.py` (7,685 bytes)
- `transformer_dit.py` (17,648 bytes)

**Subdirectories:**
- `encoders/`
- `rendering/`

This appears to be part of the LAM (an AI generative 3D model) project, with the main modeling code organized into transformer-based architecture files, discriminator components, and utility modules.


# Code Analysis: LAM Gradio Application

## Imports & Dependencies
The code imports core libraries: `os`, `cv2`, `sys`, `base64`, `subprocess`, `gradio`, `numpy`, `PIL.Image`, `argparse`, `omegaconf`, `torch`, `zipfile`, `glob`, and `moviepy.editor`. Custom modules include flame tracking utilities and LAM model inference components.

## Key Functions

**Audio Processing:**
```
add_audio_to_video(video_path, out_path, audio_path)
```
This function loads a video file and audio file using MoviePy's `VideoFileClip` and `AudioFileClip`, attaches the audio to the video, then exports the combined result as an MP4 using libx264 codec and AAC audio codec.

**Video Creation:**
```
save_images2video(img_lst, v_pth, fps)
```
Converts a sequence of images to video using MoviePy's `ImageSequenceClip`, writing output with libx264 codec.

## Audio Input Handling
Audio is sourced from the file system. The code references: `audio_path = os.path.join("./assets/sample_motion/export", base_vid, base_vid+".wav")`. WAV files are expected, loaded from predefined motion directories corresponding to selected video inputs.

## Gradio Interface Structure
The application uses `gr.Blocks()` with organized columns for:
- Input image (upload or examples)
- Input video (examples from `./assets/sample_motion/export/`)
- Processed image display
- Rendered video output

The interface chains operations: input validation → preprocessing → core inference → video generation with audio overlay.

## Rendering Pipeline
The `core_fn()` function orchestrates: FLAME tracking, image preprocessing, motion sequence preparation, and LAM model inference with `torch.no_grad()`. Output includes RGB frames (0-1 normalized) and masks, composited together before video encoding.

**No WebSocket or streaming code exists.** Processing is synchronous with Gradio's queue system.


Prompt: List all files in the rendering subdirectory


Prompt: Find all Python files related to Audio2Expression, audio processing, or chatting/talking. List their full paths.


# Files in the Rendering Subdirectory

The `lam/models/rendering/` directory contains:

**Files:**
- `__init__.py` (593 bytes)
- `gaussian_model.py` (7,891 bytes)
- `gs_renderer.py` (41,015 bytes)

**Subdirectories:**
- `flame_model/`
- `utils/`

The largest file is the Gaussian splatting renderer module at over 41KB, followed by the Gaussian model implementation and initialization files.


Based on my analysis of the repository structure provided, I found **no Python files specifically related to Audio2Expression, audio processing, or chatting/talking functionality**.

The repository (LAM - likely a face/head animation project) contains:

- **Face detection & landmark files**: `external/landmark_detection/` modules
- **Human matting modules**: `external/human_matting/`
- **3D rendering & FLAME model**: `lam/models/rendering/` with flame.py files
- **Video processing**: `lam/datasets/video_head.py`
- **Inference runners**: `lam/runners/infer/` modules
- **UI applications**: `app_lam.py`, `app_hf_space.py`

However, none of these files appear designed for audio-to-expression conversion, speech recognition, or dialogue systems based on their naming and directory organization. The project focuses on visual face animation and 3D head modeling rather than audio-driven features.


Prompt: Extract the complete README content including: Audio2Expression details, pipeline description, OpenAvatarChat integration details, any API or SDK mentions, rendering pipeline, WebRTC/WebSocket mentions


Prompt: List ALL file paths in the repository tree, especially any that mention audio, expression, chat, talking, stream, websocket, sdk, api


# File Paths in LAM Repository

Based on the repository tree provided, here are ALL file paths, with attention to relevant categories:

## Core Application Files
- `app_hf_space.py`
- `app_lam.py`

## Configuration Files
- `configs/inference/lam-20k-8gpu.yaml`
- `configs/stylematte_config.json`
- `configs/vhap_tracking/base_tracking_config.yaml`

## Model & Processing Modules
- `lam/models/modeling_lam.py`
- `lam/models/transformer.py`
- `lam/models/transformer_dit.py`
- `lam/models/discriminator.py`

## Dataset Handling
- `lam/datasets/video_head.py`
- `lam/datasets/cam_utils.py`

## Rendering & 3D Components
- `lam/models/rendering/gs_renderer.py`
- `lam/models/rendering/gaussian_model.py`
- `lam/models/rendering/flame_model/flame.py`

## Inference Pipeline
- `lam/runners/infer/lam.py`
- `lam/runners/infer/head_utils.py`
- `lam/runners/infer/utils.py`

## Tracking & Analysis
- `vhap/model/tracker.py`
- `vhap/track.py`

## Utility Functions
- `lam/utils/video.py`
- `lam/utils/ffmpeg_utils.py`

**Notable absence:** No files explicitly mention audio, expression parameters, chat, talking head functionality, streaming, WebSocket, SDK, or API endpoints in their filenames.


# LAM: Large Avatar Model - Key Components Summary

## Core Technology
LAM enables **"Build 3D Interactive Chatting Avatar with One Image in Seconds!"** through ultra-realistic 3D avatar creation, cross-platform animation rendering, and low-latency interactive capabilities.

## Audio2Expression Model
Released April 19, 2025, this component animates LAM-generated avatars using audio input, enabling dynamic facial expressions driven by sound.

## OpenAvatarChat Integration
The WebGL Interactive Chatting Avatar SDK (released April 21, 2025) integrates LAM with OpenAvatarChat, providing a complete ecosystem including language models, automatic speech recognition, text-to-speech, and avatar animation for conversational interactions with generated digital humans.

## Rendering Pipeline
The system achieves impressive performance metrics: **"1.4 s reconstruction time"** on standard hardware, with **"562.9FPS"** animation and rendering on A100 GPUs and over **"110+FPS"** on mobile devices like Xiaomi 14 phones.

## WebGL Render
Released May 20, 2025, the LAM_WebRender enables cross-platform animation and rendering capabilities, supporting deployment across different devices and environments.

## Export Functionality
Users can export LAM-generated avatars to OpenAvatarChat-compatible formats for real-time conversations through the Avatar Export Feature (released April 30, 2025).

## Training Framework
LAM-20K was trained on VFHQ and NeRSemble datasets, with future plans for expanded models trained on proprietary large-scale datasets.


Prompt: Extract the complete code focusing on: Audio2Expression usage, audio input format and processing, any streaming or real-time code, WebSocket or HTTP server endpoints, chatting functionality, how expression parameters are generated from audio, Gradio interface details


Prompt: Extract all details about pipeline, audio input, rendering output, WebRTC/WebSocket, Python API, LAM integration, browser rendering method, key files


# Analysis of Provided Code

Based on the content shared, here's what I found regarding your specific queries:

## Audio Processing
The code includes audio handling through this function:
```python
def add_audio_to_video(video_path, out_path, audio_path):
```
This method accepts a WAV file path and attaches it to rendered video output using `moviepy.editor`. The audio source is located at `"./assets/sample_motion/export/{base_vid}/{base_vid}.wav"`.

## Expression Generation Approach
Rather than deriving expressions directly from audio, this codebase uses **pre-computed motion sequences**. The system loads FLAME parameters from disk:
- Motion sequences directory: `"./assets/sample_motion/export/{base_vid}/flame_param"`
- These parameters drive facial animation, independent of audio input

## Gradio Interface Architecture
The interface features:
- **Input tabs**: Image upload and video selection
- **Processing pipeline**: "Generate" button triggers preprocessing → inference → rendering
- **Output displays**: Processed image and rendered video outputs
- **Example galleries**: Pre-loaded sample images and motion videos

No streaming, WebSocket endpoints, or real-time chatting functionality appears in this code.

## Key Finding
This is a **one-shot animation system** where audio synchronization occurs through pre-existing motion sequences rather than audio-to-expression conversion. The audio attachment happens post-rendering as a final step.


Prompt: Full content of the README


Prompt: List all file paths in the OpenAvatarChat repository


Prompt: Get all repository metadata including description, default_branch, topics


Prompt: Extract the full README especially any sections about Audio2Expression, OpenAvatarChat integration, WebGL rendering SDK, pipeline architecture, Python API


# LAM Project README Summary

## Core Project Overview
LAM (Large Avatar Model) creates "3D Interactive Chatting Avatar with One Image in Seconds!" It's a PyTorch implementation enabling ultra-realistic avatar generation from single images with fast cross-platform animation.

## Audio2Expression Integration
Released April 19, 2025, the Audio2Expression model "can animate the generated LAM Avatar with audio input." This component enables audio-driven facial animation for created avatars.

## OpenAvatarChat Integration
The project released an "Interactive Chatting Avatar SDK" (April 21, 2025) integrated with OpenAvatarChat. Users can "freely chat with the 3D Digital Human generated by LAM" through a system combining "LLM, ASR, TTS, Avatar" capabilities. An export feature allows avatars to be converted to formats compatible with OpenAvatarChat conversations.

## WebGL Rendering SDK
Released May 20, 2025, the WebGL-Render component enables "cross-platform animation and rendering" functionality, supporting deployment on various devices.

## Performance Metrics
- Reconstruction: 1.4 seconds
- A100 GPU performance: 562.9 FPS
- Mobile performance (XiaoMi 14): 110+ FPS

## Key Dependencies
The project builds upon OpenLRM, GAGAvatar, GaussianAvatars, and VHAP frameworks.


Prompt: Extract all visible repository content including file listings, README preview, description, and any visible code or documentation


Prompt: Full README content


The OpenAvatarChat repo isn't responding to direct raw fetches. Let me try alternative approaches.


Prompt: List all files and directories


Prompt: Extract the complete Python code: class definitions, function signatures, how inference works, input/output formats, especially anything related to audio or expression parameters


# LAM Inferrer: Code Structure and Inference Pipeline

## Class Definition

```python
@REGISTRY_RUNNERS.register('infer.lam')
class LAMInferrer(Inferrer):
    EXP_TYPE: str = 'lam'

    def __init__(self):
        self.cfg = parse_configs()
        self.model: LAMInferrer = self._build_model(self.cfg).to(self.device)
        self.flametracking = FlameTrackingSingleImage(...)
```

## Core Methods

**Model Building:**
```python
def _build_model(self, cfg):
    model = ModelLAM(**cfg.model)
    ckpt = load_file(resume, device='cpu')
    # Loads state_dict with shape validation
```

**Inference Pipeline:**
```python
def infer_single(self, image_path, motion_seqs_dir, motion_img_dir,
                 motion_video_read_fps, export_video, export_mesh,
                 dump_tmp_dir, dump_image_dir, dump_video_path,
                 dump_mesh_path, gaga_track_type)
```

## Input/Output Formats

**Inputs:**
- Single image path (preprocessed to `source_size`)
- Motion sequences directory with video data
- FLAME tracking parameters (shape parameters, facial keypoints)
- Camera parameters (extrinsics, intrinsics)

**Outputs:**
- RGB frames: `[Nv, H, W, 3]` (0-1 range, converted to uint8)
- Video with audio: MP4 format at specified `render_fps`
- Mesh exports: PLY and OBJ formats
- Gaussian splatting representations

## Audio Handling

```python
def add_audio_to_video(self, video_path, out_path, audio_path):
    video_clip = VideoFileClip(video_path)
    audio_clip = AudioFileClip(audio_path)
    video_clip_with_audio = video_clip.set_audio(audio_clip)
    video_clip_with_audio.write_videofile(out_path, codec='libx264')
```

Audio source: `base_vid.wav` located in `motion_seqs_dir`

## Expression Parameters

**FLAME Parameters Structure:**
```python
motion_seq["flame_params"] = {
    "betas": shape_param.unsqueeze(0),  # Shape coefficients
    # Additional FLAME parameters from motion sequences
}
```

Model processes: `flame_params={k:v.to(device) for k, v in motion_seq["flame_params"].items()}`

## Inference Execution

```python
with torch.no_grad():
    res = self.model.infer_single_view(
        image.unsqueeze(0).to(device, dtype),
        None, None,
        render_c2ws=motion_seq["render_c2ws"],
        render_intrs=motion_seq["render_intrs"],
        render_bg_colors=motion_seq["render_bg_colors"],
        flame_params={...}
    )
```

**Returns:** Dictionary containing `comp_rgb`, `cano_gs_lst`, `3dgs` representations


Prompt: Extract class definitions and key method signatures, especially: how FLAME parameters drive animation, what the model's infer_single_view method accepts, what expression/pose parameters look like, any audio-related code


Prompt: List any issues mentioning Audio2Expression, real-time, streaming, WebSocket, or API integration


# LAM Model Analysis

## Class Definition

**ModelLAM** is a PyTorch module for single-view 3D face reconstruction using Gaussian Splatting and FLAME parametric models.

## Key Method Signatures

### `infer_single_view()`
Accepts these parameters:
- `image`: [B, N_ref, C_img, H_img, W_img] - reference images
- `source_c2ws`, `source_intrs`: camera poses and intrinsics
- `render_c2ws`, `render_intrs`: target view camera data
- `render_bg_colors`: [B, N_source, 3]
- `flame_params`: Dictionary containing pose/expression parameters

Returns rendered outputs including RGB, depth, masks, and canonical GS models.

## FLAME Parameter Structure

The model expects `flame_params` as a dictionary with:
- `"betas"`: [B, 100] - shape parameters
- `"expr"`: [B, 50] - expression parameters
- `"pose_shape"`: [B, N_source, 21, 3] - pose data

These parameters drive animation through the renderer's forward pass, which generates posed/expressed geometry.

## Animation Pipeline

The renderer processes flame parameters to:
1. Generate query points from parametric geometry
2. Deform mesh based on shape/expression/pose
3. Render multiple views with Gaussian Splatting

## Audio-Related Code

**None detected** in the provided source. This codebase focuses on visual reconstruction without audio-driven animation components.

## Query Points Generation

For `e2e_flame` mode, query points are dynamically computed via `renderer.get_query_points(flame_params, device=image.device)`, enabling end-to-end optimization without fixed embeddings.


Based on the web page content provided, I can identify only **one issue** related to your search criteria:

**Issue #91: "Is LAM_audio2exp.tar publicly available or planned for release?"**
- **Status:** Open
- **Author:** mirai-gpro
- **Date:** February 25, 2026
- **Repository:** aigc3d/LAM

This issue directly addresses the Audio2Expression component. However, the page content does not display the full issue description, so I cannot confirm whether it mentions real-time capabilities, streaming, WebSocket, or API integration specifics.

The search results shown are limited, displaying only this single matching issue from the repository's 55 total open issues.


Prompt: Extract the full issue content including comments, any details about Audio2Expression availability, API format, what files/weights are involved


Prompt: List all issues related to Audio2Expression, OpenAvatarChat integration, real-time rendering, or API usage


# LAM Audio2Expression Issue Summary

## Issue Overview
A user inquired about the availability of **LAM_audio2exp.tar**, a model file required for the audio-to-expression pipeline in the LAM project.

## Key Details

**Issue #91** was opened on February 25, 2026 by user @mirai-gpro in the aigc3d/LAM repository.

**User's Question:**
The contributor stated: "the model file LAM_audio2exp.tar does not seem to be publicly available for download" and asked whether it's hosted elsewhere or has planned release dates.

## What's Involved
- **File**: LAM_audio2exp.tar (a compressed model archive)
- **Purpose**: Audio-to-expression conversion pipeline component
- **Status**: Unclear availability at time of issue opening

## Note
The provided content shows only the issue description itself. **No comments or responses from maintainers are included** in the material shared, so there's no resolution or additional context about the actual availability status of this model file.

To get current information, you would need to check the full issue thread on GitHub or contact the aigc3d/LAM project maintainers directly.


# Issues Related to Audio2Expression and OpenAvatarChat

Based on the GitHub search results for the LAM repository, here are the relevant issues:

## Open Issues

1. **Is LAM_audio2exp.tar publicly available?** (#91)
   - Opened Feb 25, 2026 by mirai-gpro
   - Addresses availability of the audio expression model

2. **Cloud Run deployment fixes** (#82)
   - Open pull request focused on "bypass DDP init, use /tmp for writes"
   - Addresses infrastructure and deployment concerns

3. **Android demo guide request** (#66)
   - Opened Oct 29, 2025
   - Asks whether documentation exists "with LAM" for mobile implementation

4. **Custom avatar creation** (#58)
   - Opened Sep 12, 2025
   - Feature request for user-generated content

5. **Interactive Chatting Demo deployment** (#32)
   - Requests guidance on "deploying Interactive Chatting Demo"
   - Relates to OpenAvatarChat integration

## Completed Work

- **Export feature for OpenAvatarChat** (#25) - Merged April 30, 2025

The search highlights ongoing development around audio-to-expression generation, avatar chat functionality, and deployment challenges rather than API-specific issues.


Prompt: Extract the full issue content and all comments about deploying the Interactive Chatting Demo and OpenAvatarChat integration


Prompt: Extract all Python code, class definitions, function signatures, especially audio handling, rendering pipeline, WebSocket/WebRTC code


The server returned HTTP 404 Not Found.

The response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.


# Issue #32: Interactive Chatting Demo Deployment

Based on the provided GitHub issue page, there is **no substantive content to extract**.

The issue titled "如何部署Interactive Chatting Demo？" (How to deploy Interactive Chatting Demo?) was opened by user KZF-kzf on May 21, 2025, but the issue description states: **"No description provided."**

The page shows no comments, responses, or additional context regarding deployment instructions or OpenAvatarChat integration details. The issue appears to be an unanswered question without any discussion or resolution documented on this page.

To find deployment information, you would need to:
- Check the repository's README or documentation files
- Review other closed issues or pull requests
- Contact the repository maintainers directly


Prompt: List all files in the repository


Prompt: Extract the full Python code, especially what format is exported for OpenAvatarChat, what data structures are involved


Prompt: List all files in the tools directory


Prompt: Extract any visible content from this GitHub page including: description, README content, file listing, topics, any visible code or documentation about pipeline architecture, audio handling, WebRTC/WebSocket, rendering approach


# Files in the Tools Directory

The tools directory contains the following files:

1. **AVATAR_EXPORT_GUIDE.md** - Documentation guide (1.5 KB)
2. **__init__.py** - Python package initialization file (empty)
3. **convertFBX2GLB.py** - Conversion script for FBX to GLB format (1.7 KB)
4. **flame_tracking_single_image.py** - FLAME model tracking script (14.2 KB)
5. **generateARKITGLBWithBlender.py** - ARKit GLB generation using Blender (7.5 KB)
6. **generateGLBWithBlender_v2.py** - GLB generation tool version 2 (8.4 KB)
7. **generateVertexIndices.py** - Vertex indices generation utility (2.7 KB)
8. **install_fbx_sdk.sh** - Shell script for FBX SDK installation (323 bytes)

These tools appear designed for 3D avatar processing, format conversion, and Blender-based asset generation.


Prompt: Extract the complete guide content about avatar export for OpenAvatarChat including format, pipeline, and integration details


Prompt: Extract the full README content about pipeline architecture, audio input format, browser rendering (WebRTC/WebSocket), Python API, LAM integration


# Export Chatting Avatar Guide Summary

## Environment Requirements
The setup process requires "Python FBX SDK 2020.2+" and "Blender (version > 4.0.0)" as foundational tools for the avatar export workflow.

## Installation Pipeline

**FBX SDK Setup:**
Users can either download the official Autodesk package or use a pre-built wheel distribution. The guide provides instructions for Linux systems, including extracting the SDK and running the installation script.

**Blender Installation:**
Version 4.0.0 or later is required. The documentation directs users to the official Blender download page with a specific Linux example provided (version 4.0.2).

**Additional Dependencies:**
The setup includes "pip install pathlib" and "pip install patool" for supporting utilities needed in the export process.

## Template Resources
Avatar creators must "download and extract sample files" containing chatting avatar templates from the provided repository.

## Execution
The workflow launches through a Gradio interface using the command: `python app_lam.py --blender_path [your_blender_path]`

**Note:** This guide covers environment setup and execution only. Specific export formats, pipeline details, and integration protocols are not documented in the provided content.


Prompt: Extract all Python code


Prompt: Extract all Python code and architecture details


Prompt: Extract all content about pipeline, audio handling, WebRTC/WebSocket, browser rendering, Python API, key files


Prompt: List all branches of the repository


Prompt: List any visible files and code content from this search


Prompt: Extract complete README content especially sections about Audio2Expression, OpenAvatarChat integration, API usage, pipeline steps, and any code examples


# LAM: Large Avatar Model - README Summary

## Project Overview
LAM is a PyTorch implementation for creating realistic 3D avatars from single images. The project enables "building interactive chatting avatars with one image in seconds."

## Key Features
The system offers three main capabilities:
- Ultra-realistic 3D avatar generation from a single image
- Fast cross-platform animation and rendering
- Low-latency SDK for real-time interactive conversations

## Audio2Expression Model
Released on April 19, 2025, this component "can animate the generated LAM Avatar with audio input." It enables audio-driven facial animation for generated avatars.

## OpenAvatarChat Integration
The project provides an interactive chatting SDK (released April 21, 2025) that integrates with OpenAvatarChat. Users can "freely chat with the 3D Digital Human generated by LAM" using a complete stack including language models, speech recognition, text-to-speech, and avatar rendering.

## Avatar Export Feature
Released April 30, 2025, this capability "allows users to chat with any LAM-generated 3D digital humans on OpenAvatarChat." Export guidance is available in the documentation.

## Technical Pipeline
**Setup & Inference Flow:**
1. Environment installation (Linux/Windows with CUDA 12.1 or 11.8)
2. Download model weights and assets from HuggingFace or ModelScope
3. Run Gradio interface: `python app_lam.py`
4. Execute inference via: `sh ./scripts/inference.sh ${CONFIG} ${MODEL_NAME} ${IMAGE_PATH} ${MOTION_SEQ}`

## Model Specifications
LAM-20K achieves 1.4-second reconstruction time on A100 GPUs and 110+ FPS on mobile devices (XiaoMi 14 Phone).


Based on the web page content provided, there are **no visible files or code content** displayed in this search result.

The page shows a GitHub code search interface for the repository `repo:aigc3d/OpenAvatarChat`, but the actual search results are not loaded. Instead, the page displays a message stating: "Sign in to search code on GitHub—Before you can access our code search functionality please sign in or create a free account."

To view the code files and content from this repository, a user would need to authenticate by signing in or creating a GitHub account first.


Prompt: Extract any documentation about the Audio2Expression pipeline, input/output formats, API details, and integration instructions


Prompt: Extract the complete shell script content


# Complete Shell Script Content from inference.sh

Here is the full script extracted from the GitHub repository:

```bash
# step1. set TRAIN_CONFIG path to config file

TRAIN_CONFIG="configs/inference/lam-20k-8gpu.yaml"

MODEL_NAME="model_zoo/lam_models/releases/lam/lam-20k/step_045500/"

IMAGE_INPUT="assets/sample_input/status.png"

MOTION_SEQS_DIR="assets/sample_motion/export/Look_In_My_Eyes/"

TRAIN_CONFIG=${1:-$TRAIN_CONFIG}

MODEL_NAME=${2:-$MODEL_NAME}

IMAGE_INPUT=${3:-$IMAGE_INPUT}

MOTION_SEQS_DIR=${4:-$MOTION_SEQS_DIR}

echo "TRAIN_CONFIG: $TRAIN_CONFIG"

echo "IMAGE_INPUT: $IMAGE_INPUT"

echo "MODEL_NAME: $MODEL_NAME"

echo "MOTION_SEQS_DIR: $MOTION_SEQS_DIR"

MOTION_IMG_DIR=null

SAVE_PLY=false

SAVE_IMG=false

VIS_MOTION=false

MOTION_IMG_NEED_MASK=true

RENDER_FPS=30

MOTION_VIDEO_READ_FPS=30

EXPORT_VIDEO=true

CROSS_ID=false

TEST_SAMPLE=false

GAGA_TRACK_TYPE=""

device=0

nodes=0

export PYTHONPATH=$PYTHONPATH:$pwd

CUDA_VISIBLE_DEVICES=$device python -m lam.launch infer.lam --config $TRAIN_CONFIG \

model_name=$MODEL_NAME image_input=$IMAGE_INPUT \

export_video=$EXPORT_VIDEO export_mesh=$EXPORT_MESH \

motion_seqs_dir=$MOTION_SEQS_DIR motion_img_dir=$MOTION_IMG_DIR \

vis_motion=$VIS_MOTION motion_img_need_mask=$MOTION_IMG_NEED_MASK \

render_fps=$RENDER_FPS motion_video_read_fps=$MOTION_VIDEO_READ_FPS \

save_ply=$SAVE_PLY save_img=$SAVE_IMG \

gaga_track_type=$GAGA_TRACK_TYPE cross_id=$CROSS_ID \

test_sample=$TEST_SAMPLE rank=$device nodes=$nodes
```

The script initializes configuration parameters with defaults, accepts command-line arguments to override them, and executes a Python inference module with specified model and data paths.


# Audio2Expression Pipeline Documentation

I cannot provide the requested documentation about the Audio2Expression pipeline, as the webpage content provided contains only a header title ("ModelScope 魔搭社区" - ModelScope Magic Community) with no substantive documentation, API details, input/output specifications, or integration instructions.

To extract this information, I would need:
- The full webpage content or documentation pages
- Links to relevant technical guides
- API reference materials
- Integration examples or tutorials

**Recommendation:** Please provide the complete documentation pages or direct links to the Audio2Expression pipeline documentation on the ModelScope platform.


Prompt: Extract all the Python code, especially anything related to audio, expression parameters, motion sequence format, what FLAME parameters look like


Prompt: Extract any visible documentation, code, or pipeline details about Audio2Expression, real-time chat, WebRTC/WebSocket rendering, and how audio drives avatar animation


# Python Code Analysis: FLAME Parameters and Motion Sequences

## FLAME Parameters Structure

The code demonstrates FLAME (Fully Learned Articulated Model of Expression) parameters as PyTorch tensors:

```python
flame_param_tensor = {
    'expr': torch.FloatTensor,        # Expression coefficients
    'rotation': torch.FloatTensor,    # Global head rotation
    'neck_pose': torch.FloatTensor,   # Neck joint angles
    'jaw_pose': torch.FloatTensor,    # Jaw articulation
    'eyes_pose': torch.FloatTensor,   # Eye gaze direction
    'translation': torch.FloatTensor, # 3D position
    'betas': torch.FloatTensor,       # Shape parameters
    'teeth_bs': torch.FloatTensor     # Optional teeth blendshapes
}
```

## Motion Sequence Format

Motion sequences are loaded from structured directories containing:
- `.npz` files storing individual frame FLAME parameters
- `transforms.json` metadata referencing camera poses and flame paths
- Optional `tracked_teeth_bs.npz` for dental animation
- `canonical_flame_param.npz` storing identity-specific shape data

## Audio-Related Code

**No audio processing is present** in this codebase. The file handles visual data (images, video frames) and parametric face model data exclusively.

## Key Processing Pipeline

Image preprocessing includes: padding, masking, center-cropping per facial regions, aspect ratio maintenance, and resizing for neural network input compatibility.


The server returned HTTP 401 Unauthorized.

The response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.


Prompt: Extract any visible documentation about Audio2Expression pipeline, input format, output format, Python API


Prompt: List all issues about audio, Audio2Expression, real-time streaming, and their discussions


# LAM GitHub Issues - Audio-Related Search Results

Based on the GitHub issues page for the aigc3d/LAM repository, there is **one audio-related issue** visible:

## Issue #91: LAM_audio2exp.tar Availability

**Status:** Open
**Opened:** February 25, 2026
**Author:** mirai-gpro

**Summary:** The issue asks whether "LAM\_audio2exp.tar is publicly available or planned for release?"

This appears to be a inquiry about accessing a specific audio-to-expression model file. The page shows this is the only audio-related issue currently appearing in search results for the repository, which has 55 total issues and 1,000 stars.

The repository itself (aigc3d/LAM) focuses on AI-generated content for 3D applications, which may explain the interest in audio-to-expression conversion technology.


Prompt: Get the full content of this issue including any maintainer responses about Audio2Expression availability and format


Prompt: Extract the full Python code


# LAM Audio2Expression Model Availability

Based on the GitHub issue, a user named mirai-gpro inquired about the availability of the `LAM_audio2exp.tar` model file on February 25, 2026.

## The Question

The user asked: "Is LAM_audio2exp.tar currently hosted somewhere that I might have missed? If it has not been released yet, are there any plans to make it publicly available?"

## Current Status

**No maintainer response is shown in the provided content.** The issue remains open without any replies from the project maintainers addressing whether:
- The model file exists on a hosting platform
- A public release is planned
- Alternative resources are available

To get an answer, you would need to check the full issue thread on the [LAM GitHub repository](https://github.com/aigc3d/LAM/issues/91) for any subsequent comments from the project team.


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


The OpenAvatarChat repo appears to not be accessible via raw/blob URLs. Let me try a different approach to find its structure.


Prompt: Extract any information about the repository structure, files, and their content


Prompt: Extract the key class and method signatures for the Gaussian Splatting renderer, especially the forward pass inputs/outputs and how FLAME params are consumed


# Gaussian Splatting Renderer: Key Classes and Methods

## Primary Class: `GS3DRenderer`

**Constructor Parameters:**
```python
def __init__(self, human_model_path, subdivide_num, smpl_type, feat_dim, query_dim,
             use_rgb, sh_degree, xyz_offset_max_step, mlp_network_config,
             expr_param_dim, shape_param_dim, ...)
```

Initializes a FLAME-based head model with configurable Gaussian splatting parameters.

## Main Forward Pass

**Signature:**
```python
def forward(self, gs_hidden_features: Float[Tensor, "B Np Cp"],
            query_points: Float[Tensor, "B Np 3"],
            flame_data, c2w: Float[Tensor, "B Nv 4 4"],
            intrinsic: Float[Tensor, "B Nv 4 4"],
            height, width, additional_features=None,
            background_color=None, debug=False, **kwargs)
```

**Flow:**
1. Processes hidden features through `forward_gs()` to generate Gaussian model
2. Animates Gaussians using FLAME parameters via `animate_gs_model()`
3. Renders from multiple camera viewpoints

## FLAME Parameter Consumption

The renderer expects flame_data dictionary with these keys:
- `"expr"` - expression parameters [Nv, 100]
- `"betas"` - shape parameters [B, 100]
- `"rotation"`, `"neck_pose"`, `"jaw_pose"`, `"eyes_pose"` - pose parameters
- `"translation"` - translation vector
- `"teeth_bs"` (optional) - teeth blend shapes

## Key Output Components

```python
out = {"comp_rgb": [B, V, C, H, W],
       "comp_mask": [B, V, C, H, W],
       "comp_depth": [B, V, C, H, W],
       "3dgs": list[GaussianModel]}
```

The renderer returns rendered RGB, silhouette masks, depth maps, and 3D Gaussian splatting objects per batch and viewpoint.


Prompt: Extract the complete Python code to understand the launch/entry point mechanism


Prompt: Extract the complete code of app_lam.py, all function definitions, class definitions, audio2expression usage, gradio interface structure, any import of audio2expression or chatting modules


# LAM Launch Entry Point Analysis

## Code Overview

The `launch.py` file serves as the main entry point for the LAM (Alibaba 3DAIGC) project. Here's the complete mechanism:

## Core Functionality

**Argument Parsing**: The script uses `argparse` to accept a runner name as a command-line argument. The parser expects users to specify which runner to execute.

**Runner Registration**: The code references `REGISTRY_RUNNERS`, a dictionary that maps runner names to their corresponding classes. This implements a factory pattern for runner instantiation.

**Execution Flow**:
1. Parse command-line arguments to identify the requested runner
2. Validate that the runner exists in the registry
3. Instantiate the runner class using a context manager
4. Execute the runner's `run()` method

## Key Design Pattern

The implementation uses a context manager (`with` statement) to handle runner lifecycle, ensuring proper resource initialization and cleanup. If a requested runner isn't found, the code raises a `ValueError` with a descriptive message.

This design promotes modularity by decoupling the launch mechanism from specific runner implementations, allowing new runners to be registered without modifying the launch code itself.


# Analysis of app_lam.py

## Complete Code Overview

The file is a **Gradio-based web interface** for the LAM (Large Avatar Model) system, enabling one-shot animatable Gaussian head generation from images and motion videos.

## Key Function Definitions

**Core Processing Functions:**
- `preprocess_fn()` - Handles image preprocessing with background removal and recentering
- `core_fn()` - Main inference pipeline orchestrating flame tracking, image preparation, motion sequencing, and rendering
- `save_images2video()` - Converts image sequences to MP4 video
- `add_audio_to_video()` - Appends audio tracks to generated videos

**Utility Functions:**
- `parse_configs()` - Loads configuration from YAML and environment variables
- `_build_model()` - Initializes the LAM model and loads pretrained weights
- `prepare_working_dir()` - Creates temporary directories for processing

## Gradio Interface Structure

The interface organizes into four main panels:

1. **Input Image Tab** - Accepts front-facing photographs with preset examples
2. **Input Video Tab** - Loads motion-driving video sequences
3. **Processed Image Display** - Shows flame-tracked and segmented results
4. **Rendered Video Output** - Displays generated animated video

## Audio Integration

The system processes audio through `add_audio_to_video()`, which accepts a WAV file path corresponding to the motion video and synchronizes it with rendered output using MoviePy's AudioFileClip.

## Notable Observations

No explicit "audio2expression" or "chatting modules" imports appear in this file. The audio functionality remains limited to simple synchronization rather than expression synthesis. The optional OpenAvatar Chat export (`enable_oac_file`) generates GLB files but lacks audio-driven animation capabilities.


Prompt: Show the FULL raw content of this Python file, including ALL imports at the top, the chatting/audio2expression sections if any, and any optional flags or feature toggles. Do not summarize - show the actual code.


# Full Raw Content of Python File

This is a Gradio-based web application for LAM (Large Avatar Model), an Alibaba 3DAIGC project for animating static head images using motion sequences.

## Complete Imports Section
```python
import os
import cv2
import sys
import base64
import subprocess

import gradio as gr
import numpy as np
from PIL import Image
import argparse
from omegaconf import OmegaConf

import torch
import zipfile
from glob import glob
import moviepy.editor as mpy
from lam.utils.ffmpeg_utils import images_to_video
from tools.flame_tracking_single_image import FlameTrackingSingleImage
from lam.runners.infer.head_utils import prepare_motion_seqs, preprocess_image

try:
    import spaces
except:
    pass
```

## Key Functional Sections

**Processing Pipeline:** The code handles image preprocessing, FLAME facial parameter tracking, motion sequence preparation, and rendering through the `core_fn` function within `demo_lam()`.

**Audio Integration:** Functions include `add_audio_to_video()` which attaches audio tracks to generated video outputs using moviepy.

**Optional Features:**
- H5 rendering flag (`h5_rendering = False`)
- Open Avatar Chat export with GLB generation
- CUDA environment setup

**No explicit audio-to-expression conversion** is present in this file—motion comes from pre-computed FLAME parameters in video sequences.

The application launches via `launch_gradio_app()`, which initializes the LAM model, flame tracking system, and Gradio interface with example inputs.


Prompt: Show lines that mention audio2expression, chatting, oac, openavatar, websocket, webrtc, or any streaming functionality. Also show the argument parser section to see all optional flags.


Prompt: Find all files in the LAM repo that reference audio2expression and show what they contain


The server returned HTTP 429 Too Many Requests.

The response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.


# Analysis of app_lam.py

## Search Results

Based on the provided code, here are my findings:

**Lines mentioning requested functionality:**
- **audio2expression**: Not found in this file
- **chatting**: Not found in this file
- **oac/openavatar**: Present at lines 393-417 (Open Avatar Chat export functionality)
- **websocket/webrtc/streaming**: Not found in this file

## OpenAvatar Chat Implementation

The code includes Open Avatar Chat (OAC) support:

> "if enable_oac_file: try: from tools.generateARKITGLBWithBlender import generate_glb...output_zip_path = os.path.join('./output/open_avatar_chat', base_iid + '.zip')"

This creates an OAC export feature (lines 393-417) that generates GLB files and zips them for avatar chatting applications.

## Argument Parser Section

The argument parser is minimal:

```python
parser.add_argument("--config", type=str)
parser.add_argument("--infer", type=str)
parser.add_argument("--blender_path", type=str,
    default='blender', help="Path to Blender executable")
```

Only three flags are available. Configuration primarily comes from environment variables (`APP_INFER`, `APP_MODEL_NAME`) rather than CLI arguments.

**Key takeaway**: This is a Gradio web UI for avatar generation, not a real-time streaming application.


Prompt: Extract the complete code, especially what format is generated for OpenAvatarChat, what data structures go into the GLB file, and how Audio2Expression is connected to this export


# Analysis of generateARKITGLBWithBlender.py

## Code Overview

The script provided is a **FLAME 3D model processing pipeline** for converting facial models to GLB format. Here's what the code actually does:

### Core Pipeline Steps

1. **Shape Parameter Injection**: Modifies an FBX template with FLAME mesh vertex data
2. **Format Conversion**: Converts ASCII FBX to binary FBX format using the FBX SDK
3. **GLB Export**: Uses Blender to convert binary FBX to GLB format
4. **Vertex Order Generation**: Produces a JSON file mapping vertex indices

### Key Functions

```python
generate_glb(input_mesh, template_fbx, output_glb, blender_exec, cleanup)
```

This orchestrates the complete workflow, managing temporary files and cleanup.

## Important Limitations

**The provided code does NOT explicitly show:**
- OpenAvatarChat format specifications or data structures
- Audio2Expression integration or connectivity
- What metadata is embedded in the final GLB file beyond geometry

The script generates a ".glb" file and a "vertex_order.json" file, but the page content does not detail the internal structure, semantic meaning, or how audio-driven expressions connect to this export pipeline.

The code references external Blender Python scripts ("convertFBX2GLB.py" and "generateVertexIndices.py") whose implementation is not included in this file view, so the exact GLB structure remains undocumented here.


Prompt: Extract full code


Prompt: Extract full code


The server returned HTTP 404 Not Found.

The response body was not retrieved. If this URL requires authentication, use an authenticated tool (e.g. `gh` for GitHub, or an MCP-provided fetch tool) instead of WebFetch.


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract any visible information about the chat avatar demo, pipeline, rendering approach


Prompt: List all issues with their titles and any visible content about WebRTC, WebSocket, pipeline architecture, LAM integration, audio format


Prompt: Extract full issue content including any PRs linked, comments about OpenAvatarChat integration format, and export details


Prompt: List all visible discussions about OpenAvatarChat integration, Audio2Expression, real-time rendering, and WebRTC/WebSocket


# Pull Request #25 Analysis

## Overview
This merged pull request introduces an export feature for OpenAvatarChat in the LAM repository.

**Key Details:**
- **Status:** Merged on April 30, 2025
- **Author:** fdyuandong (Collaborator)
- **Commit:** c2b3728 merged into master branch
- **Description:** "No description provided" - The PR lacks detailed documentation about the feature implementation

## Limitations of Available Information

The webpage content provided shows minimal substantive details about the actual changes. Notable gaps include:

- No detailed feature description or commit messages visible
- No code changes displayed (Files changed section not populated in excerpt)
- No linked issues or related PRs mentioned
- No reviewer comments or discussion threads captured
- No specific technical details about export format or integration points

## What's Missing

To fully understand the OpenAvatarChat export feature, you would need to:

1. Review the actual commit diff (c2b3728)
2. Check the "Files changed" tab for modified code
3. Look for any linked issues in the development section
4. Review any branch discussions or review comments

The PR appears to be a straightforward feature merge without significant discussion or blockers, but comprehensive documentation would require accessing the actual code modifications and related issue tracking.


Prompt: Extract all code changes in this commit - what files were added or modified, and show the code diffs


# Code Changes in Commit c2b3728

## Summary
This commit releases an export feature for OpenAvatarChat, enabling users to chat with LAM-generated 3D digital humans. It adds 10 files with 547 additions and 7 deletions.

## Files Modified/Added

### 1. **README.md & README_CN.md**
- Added language selection toggle (English/中文)
- Documented new Gradio export feature with link to tools documentation
- Added usage instructions for Blender path parameter

### 2. **app_lam.py** (+62 lines)
Key additions:
- New `--blender_path` command-line argument for Blender executable location
- `enable_oac_file` checkbox in UI for ZIP export option
- Export logic generating GLB files and animation data for OpenAvatarChat
- Creates ZIP archives containing mesh, offset, and animation files
- New `output_zip_textbox` UI element displaying export file path

### 3. **lam/models/rendering/flame_model/flame.py** (+14 lines)
- New `save_shaped_mesh()` method saving mesh geometry as OBJ format for export

### 4. **tools/README.md** (NEW, +40 lines)
Installation guide covering:
- FBX SDK 2020.2+ prerequisite setup
- Blender 4.0.0+ download instructions
- Template file acquisition from cloud storage
- Gradio invocation example with Blender path

### 5. **tools/convertFBX2GLB.py** (NEW, +59 lines)
Blender script converting FBX to GLB format with optimized settings, handling mesh import/export.

### 6. **tools/flame_tracking_single_image.py** (+3 lines)
- Added `--blender_path` argument support

## Key Feature
Users can now generate interactive 3D avatars for real-time conversations on OpenAvatarChat platform by enabling the "Export ZIP file for Chatting Avatar" option.


Prompt: Show the key parts of the code that explain: what the ZIP contains (mesh + offset + animation files), what format Audio2Expression blendshapes use (ARKit?), and any ARKit blendshape names or structure


Prompt: Show the FULL Python source code


# FLAME Model FBX/GLB Converter - Full Source Code

The provided content contains the complete Python source code for Alibaba's 3DAIGC FLAME model converter. Here's a summary of its structure:

## Key Components

**Main Functions:**

1. `update_flame_shape()` - "Injects FLAME shape parameters into FBX template" by loading OBJ mesh data and replacing vertex sections in ASCII FBX files

2. `convert_ascii_to_binary()` - Uses the FBX SDK to transform between ASCII and binary formats, managing SDK resources through proper initialization and cleanup

3. `convert_with_blender()` - Executes a Blender subprocess to export FBX files as GLB format using a conversion script

4. `gen_vertex_order_with_blender()` - Generates vertex ordering data via Blender, outputting results to JSON format

5. `generate_glb()` - "Complete pipeline for FLAME GLB generation" orchestrating all prior functions sequentially

## Dependencies

The code requires:
- FBX SDK (2020.2+)
- NumPy for numerical operations
- trimesh for 3D mesh loading
- Blender executable for final conversions

## Workflow

The pipeline follows this sequence: shape injection → ASCII-to-binary conversion → Blender GLB export → vertex order generation, with optional cleanup of temporary files.


# Analysis of generateARKITGLBWithBlender.py

Based on the provided code, I can address your questions:

## ZIP Contents
The code doesn't explicitly document what a ZIP contains. However, the pipeline processes:
- **Mesh files** (OBJ format input)
- **Template FBX files** (FLAME structure)
- **Generated GLB output** (final 3D model format)

## Audio2Expression Blendshape Format
The code does not mention Audio2Expression or specify blendshape formats. There's no reference to audio-driven facial animation in this script.

## ARKit Blendshape Information
The filename includes "ARKIT" in `generateARKITGLBWithBlender.py`, suggesting ARKit compatibility is intended. However, **the actual code contains no ARKit blendshape names, structure definitions, or format specifications**.

The script focuses on:
- FLAME shape parameter injection
- FBX conversion (ASCII to binary)
- GLB export via Blender
- Vertex order generation

The ARKit integration appears to be a design goal rather than something implemented in this particular utility script. The actual ARKit blendshape mapping likely occurs in the referenced helper scripts (`convertFBX2GLB.py` or `generateVertexIndices.py`), which aren't shown here.


Prompt: Show the FULL source code


Prompt: Extract any details about the LAM pipeline, Audio2Expression module, real-time rendering approach, WebGL/WebRTC, browser rendering architecture


# Blender FBX to GLB Converter - Full Source Code

```python
"""
Copyright (c) 2024-2025, The Alibaba 3DAIGC Team Authors.

Blender FBX to GLB Converter
Converts 3D models from FBX to glTF Binary (GLB) format with optimized settings.
Requires Blender to run in background mode.
"""

import bpy
import sys
import os
import json
from pathlib import Path

def import_obj(filepath):
    if not os.path.exists(filepath):
        raise FileNotFoundError(f"文件不存在：{filepath}")
    bpy.ops.wm.obj_import(filepath=filepath)
    print(f"成功导入：{filepath}")

def clean_scene():
    """Clear all objects and data from the current Blender scene"""
    bpy.ops.object.select_all(action='SELECT')
    bpy.ops.object.delete()
    for collection in [bpy.data.meshes, bpy.data.materials, bpy.data.textures]:
        for item in collection:
            collection.remove(item)

def apply_rotation(obj):
    obj.rotation_euler = (1.5708, 0, 0)
    bpy.context.view_layer.update()
    obj.select_set(True)
    bpy.context.view_layer.objects.active = obj
    bpy.ops.object.transform_apply(location=False, rotation=True, scale=False)
    print(f"Applied 90-degree rotation to object: {obj.name}")

def main():
    try:
        argv = sys.argv[sys.argv.index("--") + 1:]
        input_mesh = Path(argv[0])
        output_vertex_order_file  = argv[1]

        if not input_mesh.exists():
            raise FileNotFoundError(f"Input FBX file not found: {input_mesh}")

        clean_scene()

        print(f"Importing {input_mesh}...")
        import_obj(str(input_mesh))
        base_obj = bpy.context.view_layer.objects.active

        apply_rotation(base_obj)

        bpy.context.view_layer.objects.active = base_obj
        base_obj.select_set(True)
        bpy.ops.object.mode_set(mode='OBJECT')

        base_objects = [obj for obj in bpy.context.scene.objects if obj.type == 'MESH']
        if len(base_objects) != 1:
            raise ValueError("Scene should contain exactly one base mesh object.")
        base_obj = base_objects[0]

        vertices = [(i, v.co.z) for i, v in enumerate(base_obj.data.vertices)]

        sorted_vertices = sorted(vertices, key=lambda x: x[1])
        sorted_vertex_indices = [idx for idx, z in sorted_vertices]

        with open(str(output_vertex_order_file), "w") as f:
            json.dump(sorted_vertex_indices, f, indent=4)
        print(f"Exported vertex order to: {str(output_vertex_order_file)}")

    except Exception as e:
        print(f"Error: {str(e)}")
        sys.exit(1)

if __name__ == "__main__":
    main()
```

This Python script for Blender imports 3D mesh files, applies a 90-degree rotation transformation, and exports vertex indices sorted by their Z-coordinate elevation to a JSON output file.


# Response

Based on the webpage content provided, I cannot extract any details about the LAM pipeline, Audio2Expression module, real-time rendering approach, WebGL/WebRTC, or browser rendering architecture.

The document is an arXiv paper abstract titled "Proactive Service Assurance in 5G and B5G Networks: A Closed-Loop Algorithm for End-to-End Network Slicing" by Nguyen Phuc Tran and colleagues. It focuses exclusively on network slicing, service assurance, virtualized network functions (VNFs), and software-defined networking (SDN) for 5G/B5G systems.

None of the topics you're asking about—LAM pipeline, Audio2Expression, real-time rendering, WebGL/WebRTC, or browser architecture—are mentioned in this networking research paper.


Prompt: Extract the full YAML config showing all model parameters, especially audio2expression settings, expression dimensions, and rendering parameters


Prompt: Extract the full Python code, especially motion sequence loading, FLAME parameter structure, and any audio-related utilities


# LAM Configuration Analysis

Based on the GitHub page content provided, here's the extracted YAML configuration:

## Key Model Parameters

**Expression Dimensions:**
- `expr_param_dim: 10`
- `shape_param_dim: 10`

**Rendering Parameters:**
- `gs_query_dim: 1024`
- `gs_use_rgb: True`
- `gs_sh: 3`
- `gs_xyz_offset_max_step: 0.2`
- `gs_clip_scaling: 0.01`

**Transformer Architecture:**
- `transformer_heads: 16`
- `transformer_dim: 1024`
- `transformer_layers: 10`

**Image Encoder:**
- `encoder_type: "dinov2_fusion"`
- `encoder_model_name: "dinov2_vitl14_reg"`
- `encoder_feat_dim: 1024`

## Notable Limitation

**Audio-to-Expression Settings:** The configuration file does not contain any audio-related parameters. There are "no audio2expression settings" visible in this inference configuration file—it appears focused on visual-based expression generation rather than audio-driven approaches.

The configuration supports 20,000-resolution processing across 8 GPUs with a batch size of 4 per GPU and 100 training epochs.


# LAM Utils.py Analysis

Based on the GitHub file content provided, here's what I found:

## Core Functionality

The file contains utility functions for image preprocessing and motion sequence handling in the LAM (Large-scale Appearance Model) project. Key sections include:

**Image Processing Utilities:**
- `resize_image_keepaspect_np()` - Maintains aspect ratio during resizing
- `center_crop_according_to_mask()` - Crops images based on mask boundaries
- `preprocess_image()` - Comprehensive pipeline combining padding, resizing, and masking

**Motion Sequence Loading:**
The `prepare_motion_seqs()` function orchestrates the workflow:
- Loads FLAME parameter files (JSON format)
- Processes image frames with corresponding parameters
- Stacks tensors for batch processing
- Returns structured dictionary with pose, intrinsics, and RGB data

**FLAME Parameter Structure:**
Parameters are stored as dictionaries containing:
- `betas` (shape parameters)
- `focal` (focal length)
- `princpt` (principal point)
- `img_size_wh` (image dimensions)
- `R`, `t` (rotation/translation, currently commented)

## Notable Gap

**No audio utilities detected** in this file. The code focuses exclusively on vision-based motion capture preprocessing using FLAME facial model parameters.

The file lacks any sound/audio processing functionality despite the question's premise.


Prompt: Show the FULL content of prepare_motion_seqs function - every line of code, especially what it reads from disk and what the motion_seqs directory structure looks like


# prepare_motion_seqs Function Analysis

## Function Signature and Overview

The `prepare_motion_seqs` function processes motion sequences from either a directory of FLAME parameters or from video/image inputs. Here's what it does:

## Input Parameters
- `motion_seqs_dir`: Directory containing FLAME parameter JSON files
- `image_folder`: Folder with corresponding image frames
- `save_root`, `fps`, `bg_color`, `aspect_standard`, `enlarge_ratio`, `render_image_res`, `need_mask`, `multiply`, `vis_motion`: Configuration parameters

## Directory Structure Expected

The function expects this structure:
```
motion_seqs_dir/
├── 00000.json  (FLAME parameters)
├── 00001.json
├── ...
└── NNNNN.json

image_folder/ (optional, parallel structure)
├── 00000.png or .jpg
├── 00001.png or .jpg
└── ...
```

## Key Operations

1. **Video Processing** (if input is video): Calls `predict_motion_seqs_from_images()` which extracts frames and generates FLAME parameters using external multi-hmr tool

2. **File Reading**: Loads all `.json` files from `motion_seqs_dir`, each containing FLAME facial parameters and metadata

3. **Image Processing**: For each JSON file, reads corresponding image and:
   - Applies padding via `img_center_padding()`
   - Resizes while preserving aspect ratio
   - Crops around face region using `center_crop_according_to_mask()`
   - Scales intrinsic camera parameters

4. **Output Structure** (returns `motion_seqs` dict):
   - `render_c2ws`: Camera-to-world matrices [1, N, 4, 4]
   - `render_intrs`: Intrinsic matrices [1, N, 4, 4]
   - `render_bg_colors`: Background colors [1, N, 3]
   - `flame_params`: Stacked FLAME parameters [1, N, ...]
   - `rgbs`: Processed images [1, N, 3, H, W]
   - `vis_motion_render`: Optional motion visualization


Prompt: Show the actual JSON structure with all FLAME parameter keys that get loaded from each JSON file. What keys exist in each frame's JSON? Show any code that processes expression params like expr, jaw_pose, eyes_pose etc.


# FLAME Parameter Structure from LAM Code

Based on the provided code, here's what can be determined about the FLAME JSON structure:

## Keys Loaded from Each Frame's JSON

The code shows that each JSON file is loaded and processed as:

```python
flame_param = {k: torch.FloatTensor(v) for k, v in flame_raw_data.items()
               if "pad_ratio" not in k}
```

This indicates the JSON contains arbitrary key-value pairs, excluding `pad_ratio`. The code explicitly references these keys:
- **`focal`** - Camera focal length [x, y]
- **`princpt`** - Principal point [x, y]
- **`betas`** - Shape parameters (preserved across all frames from first frame)
- **`img_size_wh`** - Image dimensions [width, height]
- **`pad_ratio`** - Padding ratio (excluded from tensor conversion)
- **`R`** and **`t`** - Rotation matrix and translation (commented out in `_load_pose`)

## Expression Parameters Not Shown

The code does **not display processing** for expression-related parameters like `expr`, `jaw_pose`, or `eyes_pose`. While these likely exist in the FLAME parameter set (standard in FLAME models), the provided utility file does not contain explicit handling code for these facial expression parameters—they would be bundled into the generic parameter dictionary but remain unprocessed in this excerpt.


Prompt: Show the COMPLETE raw Python source code of this file, line by line. Do not summarize - show all actual code.


# Complete Python Source Code

```python
from collections import defaultdict
import glob
import os
import json
import numpy as np
from PIL import Image
import cv2
import torch

def scale_intrs(intrs, ratio_x, ratio_y):
    if len(intrs.shape) >= 3:
        intrs[:, 0] = intrs[:, 0] * ratio_x
        intrs[:, 1] = intrs[:, 1] * ratio_y
    else:
        intrs[0] = intrs[0] * ratio_x
        intrs[1] = intrs[1] * ratio_y
    return intrs

def calc_new_tgt_size(cur_hw, tgt_size, multiply):
    ratio = tgt_size / min(cur_hw)
    tgt_size = int(ratio * cur_hw[0]), int(ratio * cur_hw[1])
    tgt_size = int(tgt_size[0] / multiply) * multiply, int(tgt_size[1] / multiply) * multiply
    ratio_y, ratio_x = tgt_size[0] / cur_hw[0], tgt_size[1] / cur_hw[1]
    return tgt_size, ratio_y, ratio_x

def calc_new_tgt_size_by_aspect(cur_hw, aspect_standard, tgt_size, multiply):
    assert abs(cur_hw[0] / cur_hw[1] - aspect_standard) < 0.03
    tgt_size = tgt_size * aspect_standard, tgt_size
    tgt_size = int(tgt_size[0] / multiply) * multiply, int(tgt_size[1] / multiply) * multiply
    ratio_y, ratio_x = tgt_size[0] / cur_hw[0], tgt_size[1] / cur_hw[1]
    return tgt_size, ratio_y, ratio_x

def _load_pose(pose):
    intrinsic = torch.eye(4)
    intrinsic[0, 0] = pose["focal"][0]
    intrinsic[1, 1] = pose["focal"][1]
    intrinsic[0, 2] = pose["princpt"][0]
    intrinsic[1, 2] = pose["princpt"][1]
    intrinsic = intrinsic.float()

    c2w = torch.eye(4)
    # c2w[:3, :3] = torch.tensor(pose["R"])
    # c2w[3, :3] = torch.tensor(pose["t"])
    c2w = c2w.float()

    return c2w, intrinsic

def img_center_padding(img_np, pad_ratio):

    ori_w, ori_h = img_np.shape[:2]

    w = round((1 + pad_ratio) * ori_w)
    h = round((1 + pad_ratio) * ori_h)

    if len(img_np.shape) > 2:
        img_pad_np = np.zeros((w, h, img_np.shape[2]), dtype=np.uint8)
    else:
        img_pad_np = np.zeros((w, h), dtype=np.uint8)
    offset_h, offset_w = (w - img_np.shape[0]) // 2, (h - img_np.shape[1]) // 2
    img_pad_np[offset_h: offset_h + img_np.shape[0]:, offset_w: offset_w + img_np.shape[1]] = img_np

    return img_pad_np

def resize_image_keepaspect_np(img, max_tgt_size):
    """
    similar to ImageOps.contain(img_pil, (img_size, img_size)) # keep the same aspect ratio
    """
    h, w = img.shape[:2]
    ratio = max_tgt_size / max(h, w)
    new_h, new_w = round(h * ratio), round(w * ratio)
    return cv2.resize(img, dsize=(new_w, new_h), interpolation=cv2.INTER_AREA)

def center_crop_according_to_mask(img, mask, aspect_standard, enlarge_ratio):
    """
        img: [H, W, 3]
        mask: [H, W]
    """
    ys, xs = np.where(mask > 0)

    if len(xs) == 0 or len(ys) == 0:
        raise Exception("empty mask")

    x_min = np.min(xs)
    x_max = np.max(xs)
    y_min = np.min(ys)
    y_max = np.max(ys)

    center_x, center_y = img.shape[1]//2, img.shape[0]//2

    half_w = max(abs(center_x - x_min), abs(center_x -  x_max))
    half_h = max(abs(center_y - y_min), abs(center_y -  y_max))
    half_w_raw = half_w
    half_h_raw = half_h
    aspect = half_h / half_w

    if aspect >= aspect_standard:
        half_w = round(half_h / aspect_standard)
    else:
        half_h = round(half_w * aspect_standard)

    if half_h > center_y:
        half_w = round(half_h_raw / aspect_standard)
        half_h = half_h_raw
    if half_w > center_x:
        half_h = round(half_w_raw * aspect_standard)
        half_w = half_w_raw

    if abs(enlarge_ratio[0] - 1) > 0.01 or abs(enlarge_ratio[1] - 1) >  0.01:
        enlarge_ratio_min, enlarge_ratio_max = enlarge_ratio
        enlarge_ratio_max_real = min(center_y / half_h, center_x / half_w)
        enlarge_ratio_max = min(enlarge_ratio_max_real, enlarge_ratio_max)
        enlarge_ratio_min = min(enlarge_ratio_max_real, enlarge_ratio_min)
        enlarge_ratio_cur = np.random.rand() * (enlarge_ratio_max - enlarge_ratio_min) + enlarge_ratio_min
        half_h, half_w = round(enlarge_ratio_cur * half_h), round(enlarge_ratio_cur * half_w)

    assert half_h <= center_y
    assert half_w <= center_x
    assert abs(half_h / half_w - aspect_standard) < 0.03

    offset_x = center_x - half_w
    offset_y = center_y - half_h

    new_img = img[offset_y: offset_y + 2*half_h, offset_x: offset_x + 2*half_w]
    new_mask = mask[offset_y: offset_y + 2*half_h, offset_x: offset_x + 2*half_w]

    return  new_img, new_mask, offset_x, offset_y

def preprocess_image(rgb_path, mask_path, intr, pad_ratio, bg_color,
                            max_tgt_size, aspect_standard, enlarge_ratio,
                            render_tgt_size, multiply, need_mask=True):
    rgb = np.array(Image.open(rgb_path))
    rgb_raw = rgb.copy()
    if pad_ratio > 0:
        rgb = img_center_padding(rgb, pad_ratio)

    rgb = rgb / 255.0
    if need_mask:
        if rgb.shape[2] < 4:
            if mask_path is not None:
                mask = np.array(Image.open(mask_path))
            else:
                from rembg import remove
                mask = remove(rgb_raw[:, :, (2, 1, 0)])[:, :, -1]  # np require [bgr]
                print("rmbg mask: ", mask.min(), mask.max(), mask.shape)
            if pad_ratio > 0:
                mask = img_center_padding(mask, pad_ratio)
            mask = mask / 255.0
        else:
            # rgb: [H, W, 4]
            assert rgb.shape[2] == 4
            mask = rgb[:, :, 3]   # [H, W]
    else:
        # just placeholder
        mask = np.ones_like(rgb[:, :, 0])

    mask = (mask > 0.5).astype(np.float32)
    rgb = rgb[:, :, :3] * mask[:, :, None] + bg_color * (1 - mask[:, :, None])

    # resize to specific size require by preprocessor of flame-estimator.
    rgb = resize_image_keepaspect_np(rgb, max_tgt_size)
    mask = resize_image_keepaspect_np(mask, max_tgt_size)

    # crop image to enlarge human area.
    rgb, mask, offset_x, offset_y = center_crop_according_to_mask(rgb, mask, aspect_standard, enlarge_ratio)
    if intr is not None:
        intr[0, 2] -= offset_x
        intr[1, 2] -= offset_y

    # resize to render_tgt_size for training
    tgt_hw_size, ratio_y, ratio_x = calc_new_tgt_size_by_aspect(cur_hw=rgb.shape[:2],
                                                                        aspect_standard=aspect_standard,
                                                                        tgt_size=render_tgt_size, multiply=multiply)
    rgb = cv2.resize(rgb, dsize=(tgt_hw_size[1], tgt_hw_size[0]), interpolation=cv2.INTER_AREA)
    mask = cv2.resize(mask, dsize=(tgt_hw_size[1], tgt_hw_size[0]), interpolation=cv2.INTER_AREA)

    if intr is not None:
        intr = scale_intrs(intr, ratio_x=ratio_x, ratio_y=ratio_y)
        assert abs(intr[0, 2] * 2 - rgb.shape[1]) < 2.5, f"{intr[0, 2] * 2}, {rgb.shape[1]}"
        assert abs(intr[1, 2] * 2 - rgb.shape[0]) < 2.5, f"{intr[1, 2] * 2}, {rgb.shape[0]}"
        intr[0, 2] = rgb.shape[1] // 2
        intr[1, 2] = rgb.shape[0] // 2

    rgb = torch.from_numpy(rgb).float().permute(2, 0, 1).unsqueeze(0)  # [1, 3, H, W]
    mask = torch.from_numpy(mask[:, :, None]).float().permute(2, 0, 1).unsqueeze(0)  # [1, 1, H, W]
    return rgb, mask, intr

def extract_imgs_from_video(video_file, save_root, fps):
    print(f"extract_imgs_from_video:{video_file}")
    vr = decord.VideoReader(video_file)
    for i in range(0, len(vr), fps):
        frame = vr[i].asnumpy()
        save_path = os.path.join(save_root, f"{i:05d}.jpg")
        cv2.imwrite(save_path, frame[:, :, (2, 1, 0)])

def predict_motion_seqs_from_images(image_folder:str, save_root, fps=6):
    id_name = os.path.splitext(os.path.basename(image_folder))[0]
    if os.path.isfile(image_folder) and (image_folder.endswith("mp4") or image_folder.endswith("move")):
        save_frame_root = os.path.join(save_root, "extracted_frames", id_name)
        if not os.path.exists(save_frame_root):
            os.makedirs(save_frame_root, exist_ok=True)
            extract_imgs_from_video(video_file=image_folder, save_root=save_frame_root, fps=fps)
        else:
            print("skip extract_imgs_from_video......")
        image_folder = save_frame_root

    image_folder_abspath = os.path.abspath(image_folder)
    print(f"predict motion seq:{image_folder_abspath}")
    save_flame_root = image_folder + "_flame_params_mhmr"
    if not os.path.exists(save_flame_root):
        cmd = f"cd thirdparty/multi-hmr &&  python infer_batch.py  --data_root {image_folder_abspath}  --out_folder {image_folder_abspath} --crop_head   --crop_hand   --pad_ratio 0.2 --smplify"
        os.system(cmd)
    else:
        print("skip predict flame.........")
    return save_flame_root, image_folder

def prepare_motion_seqs(motion_seqs_dir, image_folder, save_root, fps,
                        bg_color, aspect_standard, enlarge_ratio,
                        render_image_res, need_mask, multiply=16,
                        vis_motion=False):
    if motion_seqs_dir is None:
        assert image_folder is not None
        motion_seqs_dir, image_folder = predict_motion_seqs_from_images(image_folder, save_root, fps)

    motion_seqs = sorted(glob.glob(os.path.join(motion_seqs_dir, "*.json")))

    # source images
    c2ws, intrs, rgbs, bg_colors, masks = [], [], [], [], []
    flame_params = []
    shape_param = None

    for idx, flame_path in enumerate(motion_seqs):
        if image_folder is not None:
            file_name = os.path.splitext(os.path.basename(flame_path))[0]
            frame_path = os.path.join(image_folder, file_name + ".png")
            if not os.path.exists(frame_path):
                frame_path = os.path.join(image_folder, file_name + ".jpg")

        with open(flame_path) as f:
            flame_raw_data = json.load(f)
            flame_param = {k: torch.FloatTensor(v) for k, v in flame_raw_data.items() if "pad_ratio" not in k}

        if idx == 0:
            shape_param = flame_param["betas"]

        c2w, intrinsic = _load_pose(flame_param)
        intrinsic_raw = intrinsic.clone()
        if image_folder is not None:
            rgb, mask, intrinsic = preprocess_image(frame_path, mask_path=None,
                                                    need_mask=need_mask,
                                                    bg_color=bg_color,
                                                    pad_ratio=float(flame_raw_data["pad_ratio"]),
                                                    max_tgt_size=int(flame_param["img_size_wh"][0]),
                                                    aspect_standard=aspect_standard,
                                                    enlarge_ratio=enlarge_ratio,
                                                    render_tgt_size=render_image_res,
                                                    multiply=multiply,
                                                    intr=intrinsic)
            rgbs.append(rgb)
            masks.append(mask)

        c2ws.append(c2w)
        bg_colors.append(bg_color)
        intrs.append(intrinsic)
        # intrs.append(intrinsic_raw)
        flame_params.append(flame_param)

    c2ws = torch.stack(c2ws, dim=0)  # [N, 4, 4]
    intrs = torch.stack(intrs, dim=0)  # [N, 4, 4]
    bg_colors = torch.tensor(bg_colors, dtype=torch.float32).unsqueeze(-1).repeat(1, 3)  # [N, 3]

    if len(rgbs) > 0:
        rgbs = torch.cat(rgbs, dim=0)  # [N, 3, H, W]
        # masks = torch.cat(masks, dim=0)  # [N, 1, H, W]

    flame_params_tmp = defaultdict(list)
    for flame in flame_params:
        for k, v in flame.items():
            flame_params_tmp[k].append(v)
    for k, v in flame_params_tmp.items():
        flame_params_tmp[k] = torch.stack(v)  # [Nv, xx, xx]
    flame_params = flame_params_tmp
    # TODO check different betas for same person
    flame_params["betas"] = shape_param

    if vis_motion:
        motion_render = render_flame_mesh(flame_params, intrs)
    else:
        motion_render = None

    # add batch dim
    for k, v in flame_params.items():
        flame_params[k] = v.unsqueeze(0)
        # print(k, flame_params[k].shape, "motion_seq")
    c2ws = c2ws.unsqueeze(0)
    intrs = intrs.unsqueeze(0)
    bg_colors = bg_colors.unsqueeze(0)
    if len(rgbs) > 0:
        rgbs = rgbs.unsqueeze(0)
    # print(f"c2ws:{c2ws.shape}, intrs:{intrs.shape}, rgbs:{rgbs.shape if len(rgbs) > 0 else None}")

    motion_seqs = {}
    motion_seqs["render_c2ws"] = c2ws
    motion_seqs["render_intrs"] = intrs
    motion_seqs["render_bg_colors"] = bg_colors
    motion_seqs["flame_params"] = flame_params
    motion_seqs["rgbs"] = rgbs
    motion_seqs["vis_motion_render"] = motion_render

    return motion_seqs
```


Now I have a very clear picture of the LAM inference pipeline. Let me do one final check on the OpenAvatarChat repo to understand its rendering approach.


Prompt: Extract all content about pipeline, WebRTC/WebSocket, audio handling, browser rendering


Prompt: Show only the key class and method signatures - GS3DRenderer constructor, forward method, animate_gs_model method signatures, and how FLAME params keys are used (expr, rotation, neck_pose, jaw_pose, eyes_pose, translation, betas, teeth_bs). Show exact tensor shapes expected.


# GS3DRenderer Key Signatures

## Constructor
```python
def __init__(self,
    human_model_path, subdivide_num, smpl_type, feat_dim, query_dim,
    use_rgb, sh_degree, xyz_offset_max_step, mlp_network_config,
    expr_param_dim, shape_param_dim,
    clip_scaling=0.2, scale_sphere=False, skip_decoder=False,
    fix_opacity=False, fix_rotation=False,
    decode_with_extra_info=None, gradient_checkpointing=False,
    add_teeth=True, teeth_bs_flag=False, oral_mesh_flag=False, **kwargs)
```

## Forward Method
```python
def forward(self,
    gs_hidden_features: Float[Tensor, "B Np Cp"],
    query_points: Float[Tensor, "B Np 3"],
    flame_data,
    c2w: Float[Tensor, "B Nv 4 4"],
    intrinsic: Float[Tensor, "B Nv 4 4"],
    height, width,
    additional_features: Optional[Float[Tensor, "B C H W"]] = None,
    background_color: Optional[Float[Tensor, "B Nv 3"]] = None,
    debug: bool = False, **kwargs)
```

## Animate GS Model Method
```python
def animate_gs_model(self,
    gs_attr: GaussianModel,
    query_points,  # [N, 3]
    flame_data,
    debug=False)
```

## FLAME Data Keys Used
- `expr`: [B, Nv, 100] - expression parameters
- `teeth_bs`: [B, Nv, X] - teeth blendshapes (if teeth_bs_flag=True)
- `betas`: [B, 100] - shape parameters
- `rotation`: [B, Nv, 3] - global rotation
- `neck_pose`: [B, Nv, 3] - neck rotation
- `jaw_pose`: [B, Nv, 3] - jaw rotation
- `eyes_pose`: [B, Nv, 6] - eye rotations
- `translation`: [B, Nv, 3] - translation

Returns dictionary with keys: `comp_rgb` [B, Nv, C, H, W], `comp_mask`, `comp_depth`, `3dgs`


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract any content about pipeline, WebRTC, WebSocket, audio, rendering, LAM integration


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


Prompt: Extract full code


The OpenAvatarChat GitHub repository is returning 404 for all paths - this strongly suggests the repository may be private or may not exist at `aigc3d/OpenAvatarChat`. Let me try to find it through other means.


Prompt: List ALL public repositories for the aigc3d organization


Prompt: Show the full README content including any links to OpenAvatarChat, the exact GitHub URL for OpenAvatarChat, and any description of the WebGL render SDK and chatting avatar demo


# Public Repositories for aigc3d Organization

Based on the webpage content, here are the publicly listed repositories:

## Pinned Repositories
1. **LHM** - Large Animatable Human Reconstruction Model (ICCV 2025)
2. **LAM** - Large Avatar Model for One-shot Animatable Gaussian Head (SIGGRAPH 2025)
3. **LAM_Audio2Expression** - ARKit expression generation from audio
4. **LAM_WebRender** - WebGL renderer for LAM projects
5. **motionshop** - Human motion replacement in videos
6. **LHM-plusplus** - Efficient Large Human Reconstruction Model

## Additional Public Repositories
7. **ViGeo** - Video Geometry Estimation
8. **LDCM** - Large Depth Completion Model (ICLR 2026)
9. **HyPlaneHead** - Tri-plane-like Representations in Full-Head Synthesis (NeurIPS 2025)
10. **motionshop-2** - Advanced version with 3D Animation Engine
11. **.github** - Organization profile repository

The page indicates "Showing 10 of 27 repositories," suggesting additional repositories exist but aren't fully detailed in this view.


# LAM README Summary

## OpenAvatarChat Integration

The README references OpenAvatarChat multiple times as a platform for interactive chatting with LAM-generated avatars. The exact GitHub URL provided is: **https://github.com/HumanAIGC-Engineering/OpenAvatarChat**

Key mentions include:
- "Avatar Export Feature that allows users to chat with any LAM-generated 3D digital humans on OpenAvatarChat"
- Released WebGL Interactive Chatting Avatar SDK on OpenAvatarChat with "LLM, ASR, TTS, Avatar" components

## WebGL Render SDK

The README states: **"We have released the WebGL-Render!"** (May 20, 2025)

The WebGL render is available at: **https://github.com/aigc3d/LAM_WebRender**

This enables "cross-platform animation and rendering" capabilities for LAM avatars.

## Chatting Avatar Demo

The project includes an interactive demo where users can "freely chat with the 3D Digital Human generated by LAM" through the OpenAvatarChat platform. Online demos are available on:
- HuggingFace Spaces
- ModelScope

The SDK supports real-time conversation features combining language models, speech recognition, text-to-speech, and avatar animation.


The correct URL is `HumanAIGC-Engineering/OpenAvatarChat`, not `aigc3d/OpenAvatarChat`. Let me fetch those repos now.


Prompt: Extract all content including: repository structure, README, pipeline description, audio input/output format, WebRTC/WebSocket details, browser rendering approach, Python API, LAM integration details, key files and classes


Prompt: Extract all content: README, pipeline, input/output format of Audio2Expression, Python API, how audio is consumed (WAV? streaming?), what FLAME parameters are output, how to use programmatically


# LAM-A2E: Audio to Expression - Technical Summary

## Overview
LAM-A2E is a real-time system that "leverages audio input to generate ARKit blendshapes-driven facial expressions" for 3D avatar animation.

## Architecture & Pipeline
The system employs an encoder-decoder architecture:
- **Audio Encoder**: Wav2Vec (pre-trained speech model)
- **Processing**: Extracts audio features combined with style features
- **Output**: Decoder generates stylized blendshape coefficients

## Input/Output Format

**Input:**
- Audio files (WAV format supported, as evidenced by sample: `BarackObama_english.wav`)
- Streaming audio support via `inference_streaming_audio.py`

**Output:**
- ARKit blendshape coefficients adapted to FLAME facial topology
- Real-time expression parameters for 3D avatar animation

## Python API & Usage

**Quick Start (Gradio Interface):**
```bash
python app_lam_audio2exp.py
```

**Inference Command:**
```bash
python inference.py --config-file ${CONFIG_PATH} --options save_path=${SAVE_PATH} weight=${CHECKPOINT_PATH} audio_input=${AUDIO_INPUT}
```

## Programmatic Usage
- Configuration files: Located in `configs/` directory
- Weights: Pre-trained models available for download
- The system supports both batch inference and streaming audio processing

## Dependencies
- Python 3.10 (required)
- CUDA 12.1 or 11.8
- Standard ML frameworks (PyTorch implied)

## Model Access
Available via HuggingFace, OSS, and ModelScope platforms.


# OpenAvatarChat Repository Analysis

## Overview
OpenAvatarChat is a "modular interactive digital human dialogue implementation" supporting multimodal interactions (text, speech, video) with pluggable ASR, LLM, TTS, and avatar components.

## Repository Structure
```
├── src/                    # Core source code
├── config/                 # Configuration files for different modes
├── models/                 # Model-related code
├── resource/avatar/        # Avatar resources
├── extensions/openclaw/    # Chat Agent integration
├── scripts/                # Utility scripts (model downloads)
├── build/                  # Build artifacts
├── docs/                   # Documentation
└── docker-compose.yml      # Containerization setup
```

## Core Architecture

**Key Strengths:**
- "Modular architecture design" enabling flexible component swapping
- Support for multiple avatar technologies: LiteAvatar, LAM, MuseTalk, FlashHead
- Average response time optimized to "2.2 seconds" through VAD detection and frame rate control

## Audio Pipeline

**Input/Output:**
- ASR: SenseVoice integration for speech recognition
- TTS: CosyVoice API support for speech synthesis
- VAD: Silero-VAD for voice activity detection
- Audio buffering and frame rate optimization mechanisms

## Component Dependencies

| Component | Library | Purpose |
|-----------|---------|---------|
| RTC | gradio-webrtc | Real-time communication |
| WebUI | OpenAvatarChat-WebUI | Frontend interface |
| VAD | silero-vad | Voice detection |
| Avatar | LiteAvatar, LAM, MuseTalk, FlashHead | Visual generation |
| TTS | CosyVoice | Speech synthesis |

## LAM Integration
LAM (Audio2Expression) processes audio to facial expressions, utilizing wav2vec2-base-960h for feature extraction. Supports expression-driven 3D avatar animations.

## Browser Rendering
Frontend uses WebRTC for real-time streaming and Gradio for web interface. Separate repository (OpenAvatarChat-WebUI) handles modern UI with potential Live2D alternative rendering support.

## Configuration Modes
Seven preset modes including:
- Chat with LiteAvatar + SenseVoice + API LLM
- Qwen-Omni multimodal setup
- FlashHead with duplex (bidirectional) interruption support
- Beta: Chat Agent mode with tool calling

## Key Technologies
- **Language:** Python (96.8%)
- **Framework:** Gradio-based web interface
- **Deployment:** Docker support with CUDA optimization
- **License:** Apache 2.0

## Latest Release (v0.6.0)
Architecture refactored with frontend/backend separation, universal interruption support, and SoulX-FlashHead diffusion-based avatar integration.


Excellent! Now I have the critical information. Let me gather the remaining specifics from the Audio2Expression and OpenAvatarChat repos.


Prompt: List ALL file paths in the repository


Prompt: List ALL file paths in the repository


# File Paths in LAM_Audio2Expression Repository

## Root Level Files
- `.gitignore`
- `LICENSE`
- `README.md`
- `app_lam_audio2exp.py`
- `inference.py`
- `inference_streaming_audio.py`
- `requirements.txt`

## Assets Directory
- `assets/images/framework.png`
- `assets/images/logo.jpeg`
- `assets/images/snapshot.png`
- `assets/images/teaser.jpg`

## Configs Directory
- `configs/lam_audio2exp_config.py`
- `configs/lam_audio2exp_config_streaming.py`
- `configs/wav2vec2_config.json`

## Engines Directory
- `engines/__init__.py`
- `engines/defaults.py`
- `engines/infer.py`
- `engines/launch.py`
- `engines/train.py`
- `engines/hooks/__init__.py`
- `engines/hooks/builder.py`
- `engines/hooks/default.py`
- `engines/hooks/evaluator.py`
- `engines/hooks/misc.py`

## Models Directory
- `models/__init__.py`
- `models/builder.py`
- `models/default.py`
- `models/network.py`
- `models/utils.py`
- `models/encoder/wav2vec.py`
- `models/encoder/wavlm.py`
- `models/losses/__init__.py`
- `models/losses/builder.py`
- `models/losses/lovasz.py`
- `models/losses/misc.py`

## Scripts Directory
- `scripts/install/install_cu118.sh`
- `scripts/install/install_cu121.sh`

## Utils Directory
- `utils/__init__.py`
- `utils/cache.py`
- `utils/comm.py`
- `utils/config.py`
- `utils/env.py`
- `utils/events.py`
- `utils/logger.py`
- `utils/misc.py`
- `utils/optimizer.py`
- `utils/path.py`
- `utils/registry.py`
- `utils/scheduler.py`
- `utils/timer.py`
- `utils/visualization.py`

## Wheels Directory
- `wheels/gradio_gaussian_render-0.0.3-py3-none-any.whl`