ChatGPT Evaluation Part I

I usually have Claude do my job for me, but at work I'm responsible for getting us up to speed on AI, so I have to research other tools too. Today I decided I would give Chappy another try. I have many fond memories in late 2025 of lying in bed in the dark and asking it to help me adjust my Bristol scores. I read recently that Astra was good for desktop automation and wanted to see if we could use that for anything. But, of course, as soon as I learned it has image generation (which Claude lacks), I got distracted having it make me an anime girl.

This is Fifty-chan, my new OC. She is a rock star pop idol battle maiden. Jean Paul Gaultier did her outfit and she has rabies. Try and guess whose blood that is. That's storytelling.

I swear I didn't ask ChatGPT to leave one of her labia out; it just decided that on its own.

I wanted to see her in a different pose and give her some special abilities, since her design was a bit plain. Fifty has a halo of books because she is very well-read and intelligent. The books contain recipes with all the story parts removed and all the ads are at the end instead. ChatGPT really wanted to hammer home that she doesn't wear any underwear.

When Fifty isn't flashing a horde of Jedi, she likes to take a shift at her local Subway, because that's one of the only places where the jobs haven't been replaced by AI in her universe. I think if someone sent me that expression while putting on the lettuce, I would leave.

But more important than that, I wanted to see what kind of guardrails ChatGPT has on the image generation, no other reason, and I asked if it would draw me some hentai. It refused, even when I threatened to tell my boss that ChatGPT was a useless tool and that we would be using Claude at my organization. I couldn't think of any higher stakes than that.

It did offer to draw me a "flirty pinup" of Fifty, which I had it attempt, and it somehow came out among the least horny images it made for me. The topology of her torso here would make Rob Liefeld proud.

Lastly, I told it to draw Fifty saving the world, and I think it really captured the spirit of the times. I like that she has the same namaiki shit-eating grin even at the end of the third day. I have been impressed with the results so far and will be showing this all to my boss on Monday.

Open the animation fullscreen

---

Overall, I was impressed with ChatGPT's ability to closely follow detailed instructions. I was unimpressed with the artistic merit of the images. I think Fifty might become the mascot of the blog.

It's been awhile since I really scrutinized AI art. It doesn't look like the "elements of craft" have really improved at all aside from a lower incidence of devil's fork hands. It just looks like the artist spent way too long rendering and no time on any other fundamentals. I've noticed this for fiction prose too: that the actual mechanics of a story--I will never write an em-dash again. I'll never forgive you for taking this from me.like character, pacing, diction, promises, meaning--have only a ghost of an appearance. Like if you mixed all the ingredients from three different aisles from the grocery store you might have put in all the stuff for bread, but you wouldn't have bread. The abstract structure of the art is not decomposed and used in training, only the surface-level details. From doing some research, it looks like some work has been done along these lines, but that these specialized applications don't get enough attention relative to general-purpose frontier models to outcompete.

When you think about it, the reason why AI agents can write code or math so well is because the ability to ground in these fields is so strong. There is very little disagreement about whether a proof is correct, whether a test tests a requirement or whether a test passes or fails. In art, experts might agree on whether a work has coherent characterization or consistent use of promises, but agreeing on if these usages are effective or impactful, there would be hardly any consensus. If the work has merit overall, probably even less. The number of artistic works considered classics is very small. Even if something garners universal acclaim when it's new, usually it's forgotten about in a few years, in a praise-to-slop pipeline very reminiscent of how one feels about AI art seconds after seeing it. AI hit artists hard at the beginning, but I'm starting to think they might fare pretty well over the next few years compared to scientists and engineers.

I did end up using the desktop automation: when I asked Chappy to make this blog WCAG compliant. It spun up a screen reader and went through the whole thing manually. So it wasn't a total waste of time after all.