Spline-FRIDA: Towards Diverse, Humanlike Robot Painting Styles with a Sample-Efficient, Differentiable Brush Stroke Model
Lawrence Chen
Peter Schaldenbrand
Tanmay Shankar
Lia Coleman
Jean Oh
Abstract
A painting is more than just a picture on a wall; a painting is a process comprised of many intentional brush strokes, the shapes of which are an important component of a painting's overall style and message. Prior work in modeling brush stroke trajectories either does not work with real-world robotics or is not flexible enough to capture the complexity of human-made brush strokes. In this work, we introduce Spline-FRIDA which can model complex human brush stroke trajectories. This is achieved by recording artists drawing using motion capture, modeling the extracted trajectories with an autoencoder, and introducing a novel brush stroke dynamics model to the existing robotic painting platform FRIDA. We conducted a survey and found that our open-source Spline-FRIDA approach successfully captures the stroke styles in human drawings and that Spline-FRIDA's brush strokes are more human-like, improve semantic planning, and are more artistic compared to existing robot painting systems with restrictive Bézier curve strokes.
Methodology
Collecting Human Stroke Trajectories
We record artists drawing using motion capture, extract the stroke trajectories, and model them using a variational autoencoder (TrajVAE). The above image visualizes the latent space of one such TrajVAE by plotting the trajectories at their corresponding latent coordinates when they are projected down to 2 dimensions.
Rendering Pipeline The core of Spline-FRIDA is its rendering pipeline, which converts stroke parameters into rendered strokes. The algorithm takes in a latent vector and positional/rotational offsets, and outputs a bitmap of darkness values corresponding to the stroke. This pipeline is differentiable with respect to the latent vector and the offset. Internally, TrajVAE is used to convert the latent vector into a trajectory.
Painting Optimization We compose many strokes into a single drawing by alpha-blending them onto the same canvas, optionally adding some color as well. Since the entire rendering pipeline is differentiable, we can optimize the painting using gradient descent.
Results
Stroke Styles By fine-tuning TrajVAEs on distinct human stroke styles, we are able to replicate these distinctions in our own drawings. These are two pairs of artworks of made by our system. The left paintings use longer, zig-zagging strokes, while the right ones are composed of small circles and dots. While each pair depicts the same content, the stroke style vastly changes the appearance and vibe of each work.
More Examples Here, each column represents a distinct stroke style and each row represents a different input image to replicate. For these examples, we maximize the CLIP similarity score between the canvas and the objective. The top row contains original drawings made by human artists on our mocap system. One VAE was fine-tuned on each human drawing and used to plan the drawings in each column.
