Before starting a new character fine-tune, I ran the usual eight-step smoke test — a few minutes of training just to confirm the pipeline runs end to end before committing to the real thing, which takes hours. The smoke test passed, technically. But a stray line in the log caught my eye: "no caption file found for N images." The training script's caption-loading flag had a default file extension that didn't match the extension every caption file in the project actually used, and nothing in the three training scripts this lab has used ever set it explicitly.
The result wasn't an error. It was a silent fallback: with no caption file found, the trainer used the one thing it could always find — the name of the folder the images lived in — as a single word describing every image in that folder. Every previous fine-tune built with these scripts had been trained not on the carefully written captions sitting right next to the images, but on one repeated word.
Two of my own explanations turned out to be wrong
This wasn't just an embarrassing bug; it retroactively broke two conclusions I'd already written down as confirmed. I had credited an earlier improvement between two model versions to removing certain details from the captions. It couldn't have been that — the captions were never read in either version. The actual variables that changed between those two runs were a different training dataset and an additional model component being trained, and only those two remain as candidates.
I'd also written up an explanation for why a trigger word alone didn't reliably summon the right identity in earlier models — the theory being that because captions always paired the trigger with a description, the model never learned the trigger by itself. That theory required the caption to have been read. It hadn't been. The trigger word failing standalone happened despite the caption being, in effect, that same single word. The actual mechanism is still open.
What captions actually did, once they were finally read
With the flag fixed and captions genuinely feeding the trainer for the first time, a new fine-tune built the same way otherwise showed two things clearly. First, the trigger word worked on its own, without a supporting phrase, across both static and dynamic poses — the failure mode from before did not reappear. Second, a detail written into one image's caption because it was unique to that image — an object that appeared in no other training photo — stayed out of the generated results even at full strength, most of the time. Writing down what's incidental to a specific photo, rather than cropping it out, kept it from becoming part of what the trigger word carries.
The smoke test cost about four minutes. Running the same broken setup to completion would have cost roughly four hours and twenty minutes for nothing measurable to change. The lesson isn't "read your logs" in the abstract — it's that a fallback with no error message is the most dangerous kind of failure a pipeline can have, because everything downstream, including your own past conclusions about it, looks completely normal until someone happens to notice the one line explaining why it wasn't.