I've been a photographer for most of my life. My personal photo library contains, at time of writing, 134,000 photos going back more than 30 years. I like capturing the moment.
So when I left my job last month - after nearly a decade - and realized I would no longer spend every day on the top floors of 4 World Trade Center with its absolutely incredible view, I decided to take some photos out of the windows.
Around that same time, the wonderful Vincent Woo published his absolutely stunning 3D recreation of Grace Cathedral in San Francisco, using a 3D modelling technique called Gaussian Splatting. I was captivated. As I said my goodbyes to my colleagues, I started taking more photos out the windows. High-quality photos, with my best lenses, with my mirrorless camera, in a near-360º circle around the perimeter of one of the most highly secured office building complexes in America.
For outdoor photogrammetry like Vincent's work, the standard practice is to use a drone-mounted camera to take thousands of high-quality, extremely sharp photos in broad daylight. This office being, well, part of the World Trade Center, a drone was out of the question. I value my freedom.
So, I took more than 1,000 photos in all, by hand, with my trusty Sony a7c's 20mm lens carefully pointing out against the glass, taking care to avoid reflections, and only ever in common areas of the office, well away from desks and people. Time was short, and while I wasn't doing anything illicit or even off-limits, I did want to avoid having to explain myself.
There were a number of challenges, mostly to do with how photogrammetry and Gaussian Splats work best if you have consistent lighting conditions. My shoot times would vary as I was shooting in between meetings. While I could have visited multiple floors to get additional data points, that would have to happen after hours so as to avoid disturbing my colleagues, and this would create datasets with differing lighting conditions. (And on the one convenient weekend day that my badge still worked, it rained.)
And most frustratingly: 3D reconstruction techniques really like clean data. Having images with reflections in them - especially consistent reflections like the reflection of a camera's lens up against triple-pane commercial building glass - wrecks havoc on the algorithm's ability to reconstruct things.
But... I had the data. Now I just needed to process it.
...but wait, what's a Splat?
Right, I should start by explaining what a Gaussian Splat is in the first place.
Traditionally, computer graphics "does 3D" by defining points in 3D space and connecting those points together. These form triangles or quads, and allow 3D software to build up a scene much like a drywaller would install sheets of drywall: one face at a time.
In 2026, the state-of-the-art for capturing realistic world data isn't based on this technique: instead, it trains - yes, like with machine learning - a 3D model of a space, by iteratively creating random "splats" of data in a scene and then trying to fit those splats to the training data (images) provided. Splats can also have spherical harmonics attached, which is really just a fancy way to say "if you look at a splat from different directions, you might see different colours."
This doesn't look better in this example, but when you scale this method up to a huge scene, you can get amazing results:
Much like training an AI model, training a Gaussian Splat can be a bit of an art. There are hundreds of different hyperparameters that can be tuned, multiple different popular software packages to use for training, and multiple algorithms that trade off speed for quality in different ways. I've trained many, many, many AI models in my decade as a machine learning engineer at Spotify, and I'm no stranger to content-based modeling - I just usually do it on audio and music data, rather than images.
Enter Claude
Like all engineers, I've been a heavy Claude user for years now. Claude's multimodal capabilities have grown significantly over time, and with Fable 5, I thought I might now be in a position to say:
Here's 1,000+ images taken with multiple cameras. Preprocess them as necessary to create a 3D model out of this, and then train a Gaussian Splat. Make no mistakes.
And so I tried that. The first result was... well, actually not terrible:
The two main pieces of software I pointed Claude to were Brush, an excellent cross-platform splat trainer by Google DeepMind researcher Arthur Brussee, and gsplat, the "standard" open-source renderer from the authors of the original splatting paper. Brush runs on macOS with Metal quite well (although it can be memory-hungry for larger scenes), but gsplat is somewhat better understood in the community, has more features, and requires a CUDA GPU.
I gave Claude three guidelines:
- Use any local machines I have in my house: an old M1 MacBook Air, a newer M4 Mac Mini, and an old Intel NUC. I have no CUDA devices.
- Use up to $100 on Modal, the fastest and easiest way to get access to CUDA devices. (Shout out Erik, Kenny, Paul, David, and the rest of the team over there; you're doing incredible work.)
- Notify me when you need my eyes to judge quality tradeoffs.
...and then I went about my month off between jobs, checking my phone intermittently as I biked almost 500km around New York. I expected this might take some time.
Data Quality
Just like training more traditional machine learning models, training a 3D Gaussian Splat requires good data, and lots of it. The splat trainer you use tries to optimize the scene for every photo, and so mathematically, it has no mechanism by which to ignore noise; any reflections, clouds, distortion, haze, or sun highlights will make it into the final product. And because clouds move quickly, I had to delete the entire sky region from the input data.
You have two choices: allow the input data to pollute your scene and then try to clean up the bad splats by hand once training is done, or clean up the input data so the trainer does the hard part for you.
So, I found myself in need of very well labeled, curated, and masked data. In the spirit of using Claude for everything here, I initially prompted Fable to farm out to a fleet of sub-agents to generate masks for each of the 1,000 images in the dataset, so that the training process would ignore sky, occlusions (i.e.: curtains, window frames, etc), and reflections.
However, Claude - from Fable down to Haiku - had a lot of trouble with this task. I'd heard that other models were fairly good at multimodal tasks like this (particularly Gemini), but struck out there too after tweaking multiple prompts.
So, after much consternation, I put a Claude sub-agent on the problem of creating a bespoke image masking UI that would allow me to hand-mask the data. I used Meta's Segment Anything model to create initial "rough" masks for the images to reduce the amount of time I had to click around, but ultimately, I had to do a lot of manual annotation to get enough data.
Due to the fact that the building has sharp corners, I unfortunately had to rely on multiple inconvenient, highly-reflective frames to get a decent view from the corners, which were often some of the most interesting views. To make these images trainable, I had to manually mask out the "occlusions" (i.e.: window frames, railings, vents, reflections, etc).
I exposed this web UI to my phone via Tailscale, and asked Claude to make the UI both mobile friendly and to pre-cache images, so I did most of my data labeling and annotation while riding the subway.
Leaning on Open Data
As progress continued, I realized: I'm scanning a view that is extremely well known; arguably one of the most photographed cityscapes in the world. Surely, I can lean on an open dataset of some kind to ground my training?
And so, I asked Claude to try that: can we grab OpenStreetMap data, which includes building positions and relative heights, to give our training scripts a more reliable canvas on which to paint our images? In a perfect world, this would remove the need to reconstruct a complex 3D depth field, and would let us just project each image directly onto the geometry of the cityscape.
Unfortunately, data quality struck again:
OpenStreetMap data is excellent; it's just not quite accurate enough for this purpose, and did not quite fit reality closely enough. Another experimental hypothesis invalidated. I could have played more carefully with camera positioning and projections, but many buildings were just flat-out missing, and others had heights that did not match the scene I had captured.
RealityScan, Lichtfeld, and Windows
At some point, I started to become disillusioned with this project: what if I was doing something stupid by experimenting with AI for driving this very visual, almost artistic process? Surely I should try the tools that others online use.
And the tools that others use are largely: Windows machines, with Epic Games' RealityScan and LichtFeld's LichtFeld Studio, both free but requiring CUDA.
I have no CUDA devices in my house, so I turned to Amazon Web Services to provision a temporary instance. I had Claude build me a simple "Modal-but-for-Windows"-style workflow, in which I would upload a local directory of input images to S3 and automatically download all dependencies onto the box on boot, then connect visually and drive the UI in real time.
However, this proved disappointing: RealityScan was much slower and much less reliable than the newer hloc and LightGlue methods that I had been using locally, and LichtFeld Studio was just a (very!) nice GUI on top of offered no discernable difference in quality from gsplatgsplat. (Thanks Vincent for the correction; it is a different implementation.) Another $10 wasted.
Diving Deeper into gsplat on Modal
I started to become suspicious of my decision to use local-only, Apple Silicon-friendly software that didn't seem as popular as the "standard" splat stack, which involved using gsplat. Luckily, gsplat runs very nicely on Modal, and Claude knows how to use Modal quite well.
So, I uploaded my images to a Modal volume and had Claude run a large series of tests, sweeping various gsplat hyperparameters and measuring the results with PSNR (peak signal-to-noise ratio). And after much tweaking (and many, many many accidental budget overruns; Claude doesn't know how to forecast its own spend in the slightest), I wound up with a model that looked better, but had more catastrophic artifacts:
Ultimately, while gsplat did technically look somewhat better, it did a terrible job on the borders with the sky, for reasons I couldn't quite identify by the time the project was over. The increase in quality also seemed more like a colour space difference, rather than a core training algorithm difference. I also found that gsplat - by default - performs a single-threaded, CPU-bound image decoding step within the training loop, cutting GPU performance by ~35% due to data pipeline starvation. I had Claude patch in a post-decode cache, which got utilization back up to 92%; but not before I had spent hundreds of dollars on Modal.
A Mobile-First Workflow
This project was happening "in the background" as I was taking my first-ever break between jobs, so I didn't want to stay tied to my computer at home; August in New York is a paradise. So, every step along the way was orchestrated via the Claude app on iOS.
- My supervisor agent, running at home in Claude Code, was always accessible via Claude on iOS.
- Sub-agents' transcripts and progress were directly visible if I was at home, but Claude on iOS does not yet show sub-agent transcripts; so I would treat that as a trust exercise.
- Any intermediate outputs (temporary .ply files, etc) were published to a small Cloudflare Pages website that was visible on mobile, so I could easily have the agent send me a URL to open at any time.
- Any interactive steps (i.e.: data labeling, tracking training) were published with small local servers and then accessed via Tailscale.
I was mostly trying to replicate the apocryphal story of an Anthropic researcher being alerted about their Claude's progress while eating a sandwich on a park bench: I wanted to make a lot of progress on this project while Claude did all of the heavy lifting, so I could spend my month off focused on fitness and wellness. And I think I mostly did.
Exploring Multimodality
I've worked a lot in my career with audio data, spending much of my past decade at Spotify leading its research engineering efforts into audio content modeling. This project proved to be a fascinating but somewhat frustrating experiment into multimodal capabilities of a different sort: how modern LLMs deal with image data and visual perception. At various times, it made simple mistakes:
-
I would naïvely use landmarks to refer to certain views: "the view 1 WTC and 3 WTC," knowing that the model had a vague understanding of the positions of those landmarks. But it was unable to correlate that with the images; it would routinely see a cityscape and assume that any tower was the landmark I was talking about, without searching or confirming. Reviewing a batch of verification renders, I once had to send back:
Note that those screenshots you have do not show midtown. They show Brooklyn.
-
Fable would fall back on using numerical properties to verify correctness, at one point even building its own 3DGS rendering stack to identify when certain properties (splat density, visual consistency, "floaters" and "clouds") were met or not met. But - much like myself earlier in my career - it failed to actually verify that those scripts gave reliable metrics in all cases, and so it would frequently refer to its own incorrect success metrics and declare victory quite early.
- Claude would regularly observe provided images and notice their overall structure while missing the important details; often showing an image and declaring "this is clearly better" when image quality had degraded extremely.
Reflection
Vincent Woo's Grace Cathedral scan was super inspiring, and I thought I could throw something together with A.I. in a week or two that would approach its impressiveness. How naïve of me.
But in the end, the result looks kind of okay, and is good enough to share. And that's after only:
- Two main orchestrator Fable sessions and a total of at least 74 sub-agents over 3 weeks.
- Roughly 280 distinct experiments (hyperparameter tuning runs, testing different slices of the dataset, etc)
- $234.45 of GPU time on Modal. (Thanks, Erik, and sorry, Lynn. 😬)
- Multiple bugs found by Claude in underlying software along the way: a hang in Apple's Metal shader dispatch code, a sqrt-of-zero NaN in Brush, and an additional bug in Brush affecting the colour of spherical harmonics.
- 3-4 freezes, hangs, and reboots of both my Mac Mini and my MacBook Air due to various reasons.
- Roughly 42M Opus 5 tokens, 6.5M Fable 5 tokens, 2.2M Sonnet 5 tokens, and 10k Haiku 4.5 tokens, which - at API list prices, would have cost almost $1,500. (Claude Max does not charge API pricing.)
- $10 in Google Cloud credits to test Gemini 3.5 Flash
- $10 in Amazon AWS credits to try RealityScan and LichtFeld Studio
But apart from these rough edges, Fable did manage to eventually get to a pretty passable end product:
Overall, this was both an experiment in getting my feet wet with Gaussian Splatting, as well as experimenting with using Claude for more esoteric tasks that require a level of human judgement and quality discernment in a way that doesn't map nicely to text. I'm still brand new to splatting, but hope to learn more about it in the future - and if anybody (Vincent?) has tips on how to do a better job with a dataset this sparse and imperfect, I'd love to hear it.
Special thanks to Lynn Root, Vincent Woo, Viktor Gardart, Jason Liu, Zameer Manji, Justin Gage, and Austin Gentry for reviewing early drafts of this post.