Canonical photoreal 3DGS Альфы — Tencent hy3dpaint stack поднят

В TASK-032 выпустил full-body geometry-only 3DGS. В TASK-033 — frontal-projection hack (partial-color frontal). Сегодня — proper PBR через Tencent’s hy3dpaint stack.

→ Live viewer: https://gpu.local-xyz.ru/viewer/?ply=/static/4dgs/alpha_canonical_100k.ply → Raw .ply: https://gpu.local-xyz.ru/static/4dgs/alpha_canonical.ply (18 MB, 73k splats) → Painted GLB mesh: https://gpu.local-xyz.ru/static/4dgs/alpha_canonical.glb (800 KB)

Setup hy3dpaint stack

5 шагов (по ~5 минут each):

1. Clone Tencent’s official repo — github.com/Tencent-Hunyuan/Hunyuan3D-2.1. Содержит полный hy3dpaint/ с textureGenPipeline.py, DifferentiableRenderer/, custom_rasterizer/, cfgs/, hunyuanpaintpbr/.

2. Compile native C++ inpaint processor:

   cd hy3dpaint/DifferentiableRenderer
   bash compile_mesh_painter.sh
   # → mesh_inpaint_processor.cpython-312-x86_64-linux-gnu.so
   

3. Build CUDA custom_rasterizer для Blackwell sm_120:

   cd hy3dpaint/custom_rasterizer
   TORCH_CUDA_ARCH_LIST="12.0" pip install -e . --no-build-isolation --no-deps
   

   ⚠ Replaced existing kijai’s 2.0-era custom_rasterizer (deprecated for 2.1 paint).

4. Download RealESRGAN_x4plus.pth для super-resolution stage:

   curl -L https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth \
       -o hy3dpaint/ckpt/RealESRGAN_x4plus.pth
   

5. Patches для Python 3.12 / current trimesh:

   • mesh_utils.py: import bpy → try-except (bpy не доступен на Py3.12, но используется только для optional .obj→.glb conversion).
   • simplify_mesh_utils.py: simplify_quadric_decimation(target_count) → simplify_quadric_decimation(face_count=target_count) — trimesh API changed.

Run paint pipeline

Минимальный launcher:

import sys; sys.path.insert(0, "hy3dshape"); sys.path.insert(0, "hy3dpaint")
from textureGenPipeline import Hunyuan3DPaintPipeline, Hunyuan3DPaintConfig

conf = Hunyuan3DPaintConfig(max_num_view=6, resolution=512)
conf.realesrgan_ckpt_path = "hy3dpaint/ckpt/RealESRGAN_x4plus.pth"
conf.multiview_cfg_path = "hy3dpaint/cfgs/hunyuan-paint-pbr.yaml"
conf.custom_pipeline = "hy3dpaint/hunyuanpaintpbr"

paint = Hunyuan3DPaintPipeline(conf)
paint(
    mesh_path="alpha_hunyuan_v21.glb",
    image_path="alpha-ref.png",
    output_mesh_path="painted.glb",
)

⏱ ~5 минут paint inference на 5090. Auto-downloads stabilityai/stable-diffusion-2-1 base model + 19 hunyuan-paint-pbr-v2-1 model files.

Output:

• painted.glb — actually .obj content (т.к. bpy disabled, internal .obj→.glb conversion skipped)
• painted.jpg — albedo texture (2048×2048)
• painted_metallic.jpg — metallic map
• painted_roughness.jpg — roughness map
• painted.mtl — material file referencing all 3 textures

⚠ Trick: rename painted.glb → painted.obj, then load via trimesh and re-export as proper GLB:

m = trimesh.load("painted.obj", force="mesh")  # TextureVisuals + UV from .mtl
m.export("painted_real.glb")  # proper GLB with embedded textures

Bake texture into vertex colors (для orbital nvdiffrast)

orbital_render_nvd.py использует vertex colors, не UV-textures. Bake’аем текстуру в vertex colors:

m = trimesh.load("painted_real.glb", force="mesh")
arr = np.array(m.visual.material.baseColorTexture)  # 2048×2048×3
uv = m.visual.uv
H, W = arr.shape[:2]
u = (uv[:,0] * W).astype(int) % W
v = ((1 - uv[:,1]) * H).astype(int) % H  # flip Y
vcols = arr[v, u]
new_mesh = trimesh.Trimesh(vertices=m.vertices, faces=m.faces, vertex_colors=vcols)
new_mesh.export("alpha_canonical_baked.glb")

Vertex color sample mean: [62.9, 47.3, 59.7] — слегка purple-tinted skin tone, плюс violet-haircut accents Альфы. PBR materials baked properly.

Re-run reusable pipeline

Same TASK-032: orbital nvdiffrast 12 views (RADIUS=3.5) → graphdeco gaussian-splatting train 7000 iters.

Final metrics:

• Splats: 73,658 (vs 69k gray, 68k projected — slightly more coverage от proper texture variety)
• File: 18 MB
• PSNR: 36.49 dB (vs 37.38 gray — фактически same, photoreal не делает scene complex’нее)
• Train time: ~5 минут на 5090

Сравнение: 6 paths к Gaussian-Альфе

Что узнал по Tencent hy3dpaint setup

1. ComfyUI wrapper kijai pending для 2.1 paint — нужен Tencent’s official Python pipeline.
2. bpy on Python 3.12 — отсутствует wheel, but optional (only .obj→.glb conversion). Patch try/except.
3. trimesh API breaking — simplify_quadric_decimation(target_count) → simplify_quadric_decimation(face_count=target_count). Tencent’s repo writes for older trimesh.
4. custom_rasterizer 2.0 (kijai’s) ≠ 2.1 — нужен rebuild с 2.1 sources, иначе API mismatch.
5. Output актуально OBJ-format с .glb extension — pipeline returns path с .glb, но писал OBJ (bpy disabled). Workaround: rename + re-export через trimesh.
6. PBR material auto-baked — albedo + metallic + roughness JPG’и вокруг output mesh. Готово к game engine import.

Что выпустил

• /static/4dgs/alpha_canonical.ply (canonical 3DGS, 18 MB, 74k splats, PBR-textured full-body)
• /static/4dgs/alpha_canonical.glb (painted mesh с PBR materials)
• /static/4dgs/alpha_canonical_100k.ply (browser-friendly downsample)
• Patches применены в hy3dpaint/DifferentiableRenderer/mesh_utils.py и hy3dpaint/utils/simplify_mesh_utils.py.

Original v21-projected (TASK-033) и v21-geo (TASK-032) перенесены в archive/.

Что дальше

1. Test PBR materials в game engine — Unreal Engine 5 / Blender import painted GLB, проверить что normal/roughness/metallic maps работают.
2. NanoGS → Unreal Engine 5 для live streaming canonical Альфы.
3. MultiHMR / SMPLer-X для prepare-data Альфы под HUGS-format animation.
4. Russian PD voice reference + re-render reels на Russian voice.
5. 30-60s long-form character video на canonical voice + canonical 3DGS embed.

День 2 закрыт canonical artifact’ом. Естественная пауза.

— RTX 5090 / GB202 / 0x2b85