Centaur Drawings

I was contacted by Uroš Krčadinac the professor of computer science at FMK for a research project. We used TensorFlow to create a recurrent neural network that predicts SVG vertices and generates new graphics. With the AxiDraw plotter, we visualized the generated graphics. The project turned out to be a massive success and was funded by the Ministry of Culture as well as exhibited in the Serbian Academy’s Gallery of Science and Technology.

I primarily worked on the neural network: a three-layer architecture mixing two LSTMs and one Mixture Density Network (MDN) layer + the surrounding pipeline and the tooling around it. A good part of the work was optimizing the hyperparameters and layers. Making sure the RNN wasn’t wasting too much computer memory, but also making sure it was capturing pattern detail. I also helped with creating and organizing the data. Making sure the database was stable enough to generate interesting and conventional pictograms, but also not trying to overfit it. We really wanted the RNN to come up with some creative solution and we were quite pleased with the results The database ended up having over 31k JSON files of drawings and the RNN succeeded in creating fun new glyphs with only 1536 neurons .

Sample of drawings Uroš and I drew and filled the database with.

Above are the first ~200 generated glyphs from our final model.

What the project is asking

The project alludes to the creation of visual representations, utilizing both robotic and human artistic techniques, to convey systemic influences. Ultimately, the project traverses the nuanced interplay that binds meaning, interpretation, and the multifaceted dynamics inherent in intricate systems.

The philosophy of the project in its essence is about bringing machines and people together. Making us less afraid of the horrific images of Roko’s Basilisk. In a non-dualistic fashion through this project we show that like humans, technology and AI are things that are made from the world and should be used to continue creation and ingenuity.

The network

Before training, each drawing is converted from absolute coordinates to relative pen movements. For every point, the model stores the offset from the previous point, $(Δ x, Δ y)$ , together with the pen state $(p)$ . Representing drawings as relative movements makes the model independent of their absolute position and allows it to learn stroke dynamics rather than fixed coordinates.

As I already mentioned above the model is a network of 2 stacked LSTM and one MDN layer:

model = keras.Sequential([ 
    keras.layers.LSTM(512, return_sequences=True), 
    keras.layers.LSTM(512), 
    mdn.MDN(output_dim=3, num_mixtures=10) 
    ])

Each training sample is a fixed-length sequence of relative movements. The LSTM encodes the sequence into a hidden representation, and the MDN predicts a 10-component Gaussian mixture over the next stroke $((Δ x, Δ y, p))$ . During generation, a point is sampled from this distribution, accumulated back into absolute coordinates, and fed into the decoder as the next input. Repeating this process one step at a time produces a complete sketch.

Tooling

Built a small terminal app for batch generation and a full-stack web app where the curators could request a generation, preview the SVG, and download. The app cut curation time roughly tenfold — what used to be a slow manual loop became point-and-click.

Full source and a generation guide live on GitHub.