Training a coding model to paint watercolours with TRL and OpenEnv

On 23 August, Surya Narreddi posted a video of watercolours painted by a language model writing JavaScript via p5.brush; the video surpassed 1.5M views. His earlier blog post covered only a narrower stage (close-up flowers) without open artifacts, and a full technical report is expected. The author of this post, coming from the engineering side, decided to reproduce the recipe with TRL and OpenEnv, publishing every piece: reference pool dataset, RL environment, training scripts, and trained models.

The pipeline runs fully on Hugging Face: training on Jobs, the RL environment and scorer model as Spaces, a pairwise judge through Inference Providers, and artifacts on the Hub in one collection. Once the Spaces are up, the recipe is a single command that invokes `hf jobs uv run train/watercolour_grpo.py` with the Qwen/Qwen3.5-35B-A3B model, LoRA, gradient checkpointing, and parameters such as `--subject 'a peach hibiscus'`, `--steps 110`, `--n-episodes 240`, and `--num-generations 8`. The author followed the original blog step-by-step, changing only when necessary; new material includes the open implementation, a hand-rated pool, and three reward mixes that were trained and compared in parallel, showing median paintings per step.

All artifacts and the full recipe are published so anyone can reproduce the pipeline. The author's own untested hypotheses were put into a "What I would try next" list, alongside the complete list of published artifacts. The author also suggests why the original video resonated: the paintings look loose, imperfect, and handmade at a time when image models produce perfect, statistically average pictures—this contrast is likely a large part of the viral appeal.