Gaussian Splatting Pipeline

Last updated:

This is mostly for me to help me remember these steps, so it assumes you’re on a Mac. But it’s basically identical on other platforms, sans sips and with your own build of Rerun.

  1. Take a video
    • TODO: details on this because that’s the most important part, but that’ll probably live on Patreon
  2. ffmpeg to extract images as frames
    • mkdir images && ffmpeg -i input.mov -vf "fps=2" ./images/out%04d.jpg
    • Change FPS depending on desired capture parallax
  3. (This part isn’t FOSS yet! But TODO: share my automation file) If using masks, use Image Processor to automate Photoshop Action Shortcuts/Create Subject Mask
    • COLMAP wants mask files in .png format, so in Terminal for i in *.jpg; do sips -s format png "$i" --out "${i%.jpg}.png"; done
  4. COLMAP on image dir to generate point cloud
    • colmap gui
    • Automatic Reconstruction (white and red dot in toolbar)
      • Set Workspace and Image folders
      • Set Data type to Video Framesor Images depending
        • NOTE: Video Frames causes a crash for me @ FPS@2; investigate whether higher fps works correctly
      • NOTE: From here we could perform a dense reconstruction of the sparse point cloud for photogrammetry purposes
  5. Brush to train gaussians on the point cloud
    • Make sure the exported sparse/, database.db, and the images/ directory are in the same folder, and open that folder in Brush
    • ./brush_app
  6. Rerun to visualize splat training as it’s happening (open before starting training)
    • ./rerun-cli-0.33.1-aarch64-apple-darwin
  7. SuperSplat to clean up the file
  8. Get Sillay