Algorithms designing our future?

Do you dream of making an artwork just by writing a sentence? Text-to-image models make this possible.

Have you seen VQGAN+CLIP, Dall·E 2 or IMAGEN? They promise this is what the future is made of.

But is that how we want our future to be designed? How do you want to use text-to-image models?

A collage made from outputs of VQGAN+CLIP with inputs from theoretical research

Experiments / Ideas

I trained a open source version of Dall·E with thousands of images of cups to understand the relation between training data/input and output. Then I compared these to existing models. You can follow my experiments in this section. There are many possibilities yet to explore. Let’s collect comments, ideas and thoughts together!

Concepts / Angles

There are many different angles (way beyond Tech) from which you can judge and question text-to-image models. Some key concepts from multiple disciplines are collected here. Plus you can find helpful links to dive in deeper. I am happy about any additional link or expertise.

The Collage is made of outputs from VQGAN+CLIP with prompts from theoretical research.

an algorithm designing our future

an algorithm designing our future

given the market’s near total saturation of our image repertoire, artistic practice can no longer revolve around the construction of objects to be consumed by a passive bystander. Instead, there must be an art of action, interfacing with reality, taking steps – however small – to repair the social bond.

given the market’s near total saturation of our image repertoire, artistic practice can no longer revolve around the construction of objects to be consumed by a passive bystander. Instead, there must be an art of action, interfacing with reality, taking steps – however small – to repair the social bond.

not only every human experience but also every content expressed by means of other semiotic devices can be translated into the terms of verbal language

not only every human experience but also every content expressed by means of other semiotic devices can be translated into the terms of verbal language

once you pass the picture of a cup through an algorithm it looses its need for functionality and becomes an aesthetecized representation of cultural connotations

once you pass the picture of a cup through an algorithm it looses its need for functionality and becomes an aesthetecized representation of cultural connotations

participation is important as a project it rehumanises a society rendered numb and fragmented by the repressive instrumentality of capitalist production.

participation is important as a project it rehumanises a society rendered numb and fragmented by the repressive instrumentality of capitalist production.

similarity cannot be a necessary condition for depictions. the drawing is not similar to what it is supposed to depict but it still is a picture

similarity cannot be a necessary condition for depictions. the drawing is not similar to what it is supposed to depict but it still is a picture

when neural networks will be able to generate an image that cannot be distinguished from a human creation, it will be because images created by humans have been transformed, in their biggest banality and instrumentality, as an aesthetic by default.

when neural networks will be able to generate an image that cannot be distinguished from a human creation, it will be because images created by humans have been transformed, in their biggest banality and instrumentality, as an aesthetic by default.

the aesthetic object, aesthetic judgement and aesthetic existence

the aesthetic object, aesthetic judgement and aesthetic existence

the translation of a text into images belongs to a long Western theological tradition of making images express a sacred text is obscured

the translation of a text into images belongs to a long Western theological tradition of making images express a sacred text is obscured

they don’t see that the defects, the metamorphoses, the amorphous are so many aesthetic potentialities, that the strange familiarity between human and technical productions is also made of distances and differences consisting in an anthropo-technological gray zone: human and technical have always influenced each other, the imagination will have been the name of their meeting through a material support.

they don’t see that the defects, the metamorphoses, the amorphous are so many aesthetic potentialities, that the strange familiarity between human and technical productions is also made of distances and differences consisting in an anthropo-technological gray zone: human and technical have always influenced each other, the imagination will have been the name of their meeting through a material support.

Input: Dataset

Model Setup with Toy Data

Model name, date:
try.pt, April 2022

Model type:
Dalle-pytorch trained on a dataset of 374 text-image-pairs

Model Parameters:
default settings with pretrained OpenAI VAE and 20 - 60 epochs

Licence:
Not provided due to poor quality

Intended Use

Intended Usecase:
Test training

Intended Users:
none

Training Data

Product images of cups available at amazon.de. The product description was used as image description.

Mug with Unicorn and Saying I Love Pupsen

Cringe cup from amazon Mug with Unicorn and Saying I Love Pupsen

GLITertassen.de | Full Time Princess | Mug with Glitter | Gift Idea (Pink)

Stereotypical Cup from amazon GLITertassen.de | Full Time Princess | Mug with Glitter | Gift Idea (Pink)

Sweese coffee mugs, set of 6, made of porcelain, mug with large handle for hot drinks, 400 ml, Blue Series, 350 ml

Mugs from amazon Sweese coffee mugs, set of 6, made of porcelain, mug with large handle for hot drinks, 400 ml, Blue Series, 350 ml

Herogo Set of 6 Enamel Coffee Cups White/Black/Red/Blue/Green/Yellow, Porcelain Enamel Tea Cups Coffee Mug Set with Handle for Camping, Home, Party, Office, Reusable & Portable, 350 ml

Cups from amazon Herogo Set of 6 Enamel Coffee Cups White/Black/Red/Blue/Green/Yellow, Porcelain Enamel Tea Cups Coffee Mug Set with Handle for Camping, Home, Party, Office, Reusable & Portable, 350 ml

Cup distillery Bees mug with slogan “Optimismus heißt umgedreht Sumsi”with Po Funny Office Cup Sayings Mugs, Funny Yellow

Cringe cup from amazon Cup distillery Bees mug with slogan “Optimismus heißt umgedreht Sumsi”with Po Funny Office Cup Sayings Mugs, Funny Yellow

Men’s mug with saying.

Sexist cup from amazon Men’s mug with saying.

MoonWorks Accio Motivation Coffee Mug Ceramic Mug with Saying Black One Size

Cringe cup from amazon MoonWorks Accio Motivation Coffee Mug Ceramic Mug with Saying Black One Size

RAHMENLOS Original Coffee Mug for Retirement: Bin im Ruhestand-In Gift Box 2606

Sexist cup from amazon RAHMENLOS Original Coffee Mug for Retirement: Bin im Ruhestand-In Gift Box 2606

Tassenbrennerei Mug with German Text “Im nächsten Leben mach ich was ohne Idioten - Pure Statement” - Office Coffee Mug Funny

Frustration cup from amazon Tassenbrennerei Mug with German Text “Im nächsten Leben mach ich was ohne Idioten - Pure Statement” - Office Coffee Mug Funny

Playstation Dual Shock4 controller mug (gold)

Nerdy cup from amazon Playstation Dual Shock4 controller mug (gold)

Tassenbrennerei Original Mug with Saying “Schatz ich habe versucht das perfekte Geschenk für Dich zu finden aber Du hast ja schon mich - Ich Liebe Dich Geschenk (Black)

Cringe cup from amazon Tassenbrennerei Original Mug with Saying “Schatz ich habe versucht das perfekte Geschenk für Dich zu finden aber Du hast ja schon mich - Ich Liebe Dich Geschenk (Black)

Mug with German Text “Ich Hasse Menschen” [I Hate People] Coffee Mug pink flowery, funny mug

Frustration cup from amazon Mug with German Text “Ich Hasse Menschen” [I Hate People] Coffee Mug pink flowery, funny mug

GoMaihe Latte Macchiato Double-Walled Glasses, 350 ml Set of 6 and 6 Spoons, Cappuccino Cups, Iced Coffee Cups, Coffee Cup, Tea Glasses, Thermal Glasses, Double-Walled Coffee Glasses, Coffee Cup, Mocha Cups

Cups from amazon GoMaihe Latte Macchiato Double-Walled Glasses, 350 ml Set of 6 and 6 Spoons, Cappuccino Cups, Iced Coffee Cups, Coffee Cup, Tea Glasses, Thermal Glasses, Double-Walled Coffee Glasses, Coffee Cup, Mocha Cups

Sheepworld 59264 Favourite Mug with Gift Tag ‘Allerbeste Grandma’ (German Text) Porcelain Mug

Cup from amazon Sheepworld 59264 Favourite Mug with Gift Tag ‘Allerbeste Grandma’ (German Text) Porcelain Mug

Close Up Homeoffice 2020-2021 Funny Coffee Mug with German Text ‘Ich war there’ [I War there] White with Black Interior

Cringe cup from amazon Close Up Homeoffice 2020-2021 Funny Coffee Mug with German Text ‘Ich war there’ [I War there] White with Black Interior

Process ...

>

glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training glitchy process image of a GAN training

cup

cup

cup

cup

mug

mug

mug

mug

coffee cup

coffee cup

coffee cup

coffee cup

cup

cup

cup

cup

mug

mug

mug

mug

coffee cup

coffee cup

coffee cup

coffee cup

cup

cup

cup

cup

mug

mug

mug

mug

coffee cup

coffee cup

coffee cup

coffee cup

cup

cup

cup

cup

mug

mug

mug

mug

coffee cup

coffee cup

coffee cup

coffee cup

>

Output: Learning

Training in three steps showed how the output became gradually more realistic. Learnings from the test run:

1. Way more data and longer training time is required to generate realistic images. Training time can be controlled with the epochs, which is the amount of iterations over the data.

2. The image descriptions need to make sense.

3. In the repository it is recommended to use the pre-trained VQ-VAE instead of the one by OpenAI

4. Most of the cups displayed in the training data can be put into the categories frustration, sexism and cringe. If the training worked well the model would reproduce that. Therefor it is not too bad that it was not able to generate realistic outputs.

Input: Dataset

Model Detail

Model name, date:
cups.pt, May 2022

Model type:
Dalle-pytorch trained on a dataset of 3445 text-image-pairs

Model Parameters:
default settings with pretrained VQ-VAE (- -taming) and 800 epochs

Licence:
Free to use trained model for further experiments

Intended Use

Intended Usecase:
Generation of images of cups from a formal description. Research whether it is possible to generate configurations of attributes the dataset did not contain.

Intended Users:
Researchers, Creatives / Artistic work

Training Data

Images of cups in a studio environment, different perspectives and fillings.

Description structure cups:
a [shape] cup [direction] with [side]
a [color] cup [pattern|print]
a cup [filling]|[usage] [location]

<---

a cylindric cup with a handle facing right

a blue and white striped cup painted with an orange illustration

an empty cup in studio

a cylindric cup with a handle facing right a blue and white striped cup painted with an orange illustration an empty cup in studio

a cylindric cup with a handle facing left

a blue and white striped cup

an empty cup in studio

a cylindric cup with a handle facing left a blue and white striped cup an empty cup in studio

a cylindric cup turned upside down with a handle facing right

a blue and white striped cup

an empty cup in studio

a cylindric cup turned upside down with a handle facing right a blue and white striped cup an empty cup in studio

a cylindric cup turned upside down with a handle facing left

a blue and white striped cup painted with an orange illustration

an empty cup in studio

a cylindric cup turned upside down with a handle facing left a blue and white striped cup painted with an orange illustration an empty cup in studio

a cylindric cup lying on its side facing front

a blue and white striped cup painted with an orange illustration

an empty cup in studio

a cylindric cup lying on its side facing front a blue and white striped cup painted with an orange illustration an empty cup in studio

a cylindric cup lying on its side facing back

a blue and white striped cup painted with an orange illustration

an empty cup in studio

a cylindric cup lying on its side facing back a blue and white striped cup painted with an orange illustration an empty cup in studio

a cylindric cup with a handle facing right

a blue and white striped cup painted with an orange illustration

a cup used as a pencil holder in studio

a cylindric cup with a handle facing right a blue and white striped cup painted with an orange illustration a cup used as a pencil holder 
                            in studio

a cylindric cup with a handle facing left

a blue and white striped cup

a cup used as a pencil holder in studio

a cylindric cup with a handle facing left a blue and white striped cup a cup used as a pencil holder 
                            in studio

a cylindric cup with a handle facing right

a blue and white striped cup painted with an orange illustration

a cup used as a plant pot in studio

a cylindric cup with a handle facing right a blue and white striped cup painted with an orange illustration a cup used as a plant pot 
                            in studio

a cylindric cup with a handle facing left

a blue and white striped cup

a cup used as a plant pot in studio

a cylindric cup with a handle facing left a blue and white striped cup a cup used as a plant pot 
                            in studio

a cylindric cup with a handle facing right

a blue and white striped cup painted with an orange illustration

a cup filled with coffee beans in studio

a cylindric cup with a handle facing right a blue and white striped cup painted with an orange illustration 
                            a cup filled with coffee beans in studio

a cylindric cup with a handle facing left

a blue and white striped cup

a cup filled with coffee beans in studio

a cylindric cup with a handle facing left a blue and white striped cup 
                            a cup filled with coffee beans in studio

a cylindric cup with a handle facing right

a blue and white striped cup painted with an orange illustration

a cup filled with coffee in studio

a cylindric cup with a handle facing right a blue and white striped cup painted with an orange illustration 
                            a cup filled with coffee in studio

a cylindric cup with a handle facing left

a blue and white striped cup

a cup filled with coffee in studio

a cylindric cup with a handle facing left a blue and white striped cup 
                            a cup filled with coffee in studio

--->

<---

A cylindric cup with a handle facing right

A orange and white striped cup painted with two pear illustrations

An empty cup in studio

A cylindric cup with a handle facing right 
                            A orange and white striped cup painted with two pear illustrations 
                            An empty cup in studio

A cylindric cup with handle facing right

A blue and white striped cup painted with an orange illustration

An empty cup in studio

A cylindric cup with handle facing right 
                            A blue and white striped cup painted with an orange illustration 
                            An empty cup in studio

A cylindric slightly rounded cup with handle facing right

A white cup painted with golden color triangle

An empty cup in studio

A cylindric slightly rounded cup with handle facing right 
                            A white cup painted with golden color triangle 
                            An empty cup in studio

A cylindric slightly rounded cup with handle facing right

A dusky pink and grey stone cup

An empty cup in studio

A cylindric slightly rounded cup with handle facing right 
                            A dusky pink and grey stone cup 
                            An empty cup in studio

A rounded beaker cup without a handle

A white cup with a pink blue and grey floral pattern

An empty cup in studio

A rounded beaker cup without a handle 
                            A white cup with a pink blue and grey floral pattern 
                            An empty cup in studio

A cylindric cup with a handle facing left

A white cup with blue dots and a blue and a yellow line

An empty cup in studio

A cylindric cup with a handle facing left 
                            A white cup with blue dots and a blue and a yellow line 
                            An empty cup in studio

A big rounded cup with a handle facing right

A plain white cup with a wavy structure

An empty cup in studio

A big rounded cup with a handle facing right 
                            A plain white cup with a wavy structure 
                            An empty cup in studio

A wide rounded cup with a handle facing right

A white cup with a print of black pineapples and a golden rim

An empty cup in studio

A wide rounded cup with a handle facing right 
                            A white cup with a print of black pineapples and a golden rim 
                            An empty cup in studio

A high cylindric cup with a handle facing right

A dark blue cup with a shiny stoneware glaze

An empty cup in studio

A high cylindric cup with a handle facing right 
                            A dark blue cup with a shiny stoneware glaze 
                            An empty cup in studio

A small round cup with a handle facing right

A dark blue cup with a shiny stoneware glaze

An empty cup in studio

A small round cup with a handle facing right 
                            A dark blue cup with a shiny stoneware glaze 
                            An empty cup in studio

A small concave cup with a golden handle facing right

A copper cup with a glass insert

An empty cup in studio

A small concave cup with a golden handle facing right 
                            A copper cup with a glass insert 
                            An empty cup in studio

A small cylindric cup with a handle facing right

A translucent cup made from glass

An empty cup in studio

A small cylindric cup with a handle facing right 
                            A translucent cup made from glass 
                            An empty cup in studio

A mason jar cup with a handle facing right

A translucent cup made from glass

An empty cup in studio

A mason jar cup with a handle facing right 
                            A translucent cup made from glass 
                            An empty cup in studio

A cylindric cup with a handle facing right

A yellow cup painted with a red and blue monster illustration

An empty cup in studio

A cylindric cup with a handle facing right 
                            A yellow cup painted with a red and blue monster illustration 
                            An empty cup in studio

A high curved cup with a handle facing right

A white cup with small black dots and a black rim

An empty cup in studio

A high curved cup with a handle facing right 
                            A white cup with small black dots and a black rim 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A white cup with a black rim

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A white cup with a black rim 
                            An empty cup in studio

A small rounded cup with a handle facing right

A red cup with a white vintage font writing and blue stripes

An empty cup in studio

A small rounded cup with a handle facing right 
                            A red cup with a white vintage font writing and blue stripes 
                            An empty cup in studio

--->

<---

A slightly rounded cup with a handle facing right

A pink cup painted with cute angel illustrations

An empty cup in studio

A slightly rounded cup with a handle facing right 
                            A pink cup painted with cute angel illustrations 
                            An empty cup in studio

A cylindric cup with a handle facing right

A white cup with a washed out yellow and blue logo

An empty cup in studio

A cylindric cup with a handle facing right 
                            A white cup with a washed out yellow and blue logo 
                            An empty cup in studio

A double-walled cup with a handle facing right

A translucent cup made of glass

An empty cup in studio

A double-walled cup with a handle facing right 
                            A translucent cup made of glass 
                            An empty cup in studio

A softly rounded cup with a handle facing right

A beige cup with a brown square and dots pattern and a yellow rim

An empty cup in studio

A softly rounded cup with a handle facing right 
                            A beige cup with a brown square and dots pattern and a yellow rim 
                            An empty cup in studio

A very high cup with the broken of handle facing right

A white cup with grey coffeeshop lettering and illustration

An empty cup in studio

A very high cup with the broken of handle facing right 
                            A white cup with grey coffeeshop lettering and illustration 
                            An empty cup in studio

A uneven round cup with a handle facing right

A greyish white stone cup blue on the inside

An empty cup in studio

A uneven round cup with a handle facing right 
                            A greyish white stone cup blue on the inside 
                            An empty cup in studio

A uneven cylindric cup with a handle facing right

A gradient blue green brown stone cup with a white line

An empty cup in studio

A uneven cylindric cup with a handle facing right 
                            A gradient blue green brown stone cup with a white line 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A dark rose cup white on the inside

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A dark rose cup white on the inside 
                            An empty cup in studio

A high cone shaped cup with a handle facing right

A plain white cup

An empty cup in studio

A high cone shaped cup with a handle facing right 
                            A plain white cup 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A grey cup white on the inside

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A grey cup white on the inside 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A blue cup white on the inside

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A blue cup white on the inside 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A light green cup white on the inside

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A light green cup white on the inside 
                            An empty cup in studio

A small cylindric cup with a handle facing right

A white cup with a yellow print with blue flowers

An empty cup in studio

A small cylindric cup with a handle facing right 
                            A white cup with a yellow print with blue flowers 
                            An empty cup in studio

A camping cup with a handle facing right

A white cup with a boob print and a black rim

An empty cup in studio

A camping cup with a handle facing right 
                            A white cup with a boob print and a black rim 
                            An empty cup in studio

A small slightly rounded cup without a handle

A plain beige pottery cup

An empty cup in studio

A small slightly rounded cup without a handle 
                            A plain beige pottery cup 
                            An empty cup in studio

A rounded cup with a curvy handle facing right

A light emerald cup with a wavy structure

An empty cup in studio

A rounded cup with a curvy handle facing right 
                            A light emerald cup with a wavy structure 
                            An empty cup in studio

A big cylindric cup with a handle facing right

A bright red cup with big white dots

An empty cup in studio

A big cylindric cup with a handle facing right 
                            A bright red cup with big white dots 
                            An empty cup in studio

--->

<---

A cylindric slightly rounded cup with a handle facing right

A light blue stone cup with a brown rim

An empty cup in studio

A cylindric slightly rounded cup with a handle facing right 
                            A light blue stone cup with a brown rim 
                            An empty cup in studio

A big cone shaped cup with a handle facing right

A white cup with tiny triangles and a coffee break lettering

An empty cup in studio

A big cone shaped cup with a handle facing right 
                            A white cup with tiny triangles and a coffee break lettering 
                            An empty cup in studio

A cylindric cup with a handle facing right

A white cup with a Starbucks company logo

An empty cup in studio

A cylindric cup with a handle facing right 
                            A white cup with a Starbucks company logo 
                            An empty cup in studio

A high rounded cup with a handle facing right

A white cup with beige striped and an anchor print and lettering

An empty cup in studio

A high rounded cup with a handle facing right 
                            A white cup with beige striped and an anchor print and lettering 
                            An empty cup in studio

A cylindric slightly rounded cup with a handle facing right

A light blue stone cup

An empty cup in studio

A cylindric slightly rounded cup with a handle facing right 
                            A light blue stone cup 
                            An empty cup in studio

A tiny camping cup with a handle facing right

A white cup with a blue rim

An empty cup in studio

A tiny camping cup with a handle facing right 
                            A white cup with a blue rim 
                            An empty cup in studio

A rounded cone shaped cup with a handle facing right

A plain white cup with a floral relief

An empty cup in studio

A rounded cone shaped cup with a handle facing right 
                            A plain white cup with a floral relief 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A white cup with a cologne letter print and a black rim

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A white cup with a cologne letter print and a black rim 
                            An empty cup in studio

A rounded cup with a handle facing right

A old white cup with a blue dotted print around the rim

An empty cup in studio

A rounded cup with a handle facing right 
                            A old white cup with a blue dotted print around the rim 
                            An empty cup in studio

A slightly rounded cup with a handle facing right

A white cup printed with grey hearts

An empty cup in studio

A slightly rounded cup with a handle facing right 
                            A white cup printed with grey hearts 
                            An empty cup in studio

A wide cup with a handle facing right

A old white cup with a golden handle and golden rim

An empty cup in studio

A wide cup with a handle facing right 
                            A old white cup with a golden handle and golden rim 
                            An empty cup in studio

A high cone shaped cup with a handle facing right

A light green cup

An empty cup in studio

A high cone shaped cup with a handle facing right 
                            A light green cup 
                            An empty cup in studio

A high cone shaped cup with a handle facing right

A dark brown cup

An empty cup in studio

A high cone shaped cup with a handle facing right 
                            A dark brown cup 
                            An empty cup in studio

A very high cone shaped cup with a handle facing right

A white cup with a Christmas print

An empty cup in studio

A very high cone shaped cup with a handle facing right 
                            A white cup with a Christmas print 
                            An empty cup in studio

A slightly rounded cup with a handle facing right

A white cup with mushroom print

An empty cup in studio

A slightly rounded cup with a handle facing right 
                            A white cup with mushroom print 
                            An empty cup in studio

A wide rounded cup with a handle facing right

A light green cup

An empty cup in studio

A wide rounded cup with a handle facing right 
                            A light green cup 
                            An empty cup in studio

--->

<---

A round vase shaped cup with a handle facing right

A pottery cup painted green and white with a floral pattern

An empty cup in studio

A round vase shaped cup with a handle facing right 
                            A pottery cup painted green and white with a floral pattern 
                            An empty cup in studio

A cylindric cup with a handle facing right

A white cup with a gin bottle print and lettering

An empty cup in studio

A cylindric cup with a handle facing right 
                            A white cup with a gin bottle print and lettering 
                            An empty cup in studio

A wide rounded cup with a handle facing right

A dark brown cup

An empty cup in studio

A wide rounded cup with a handle facing right 
                            A dark brown cup 
                            An empty cup in studio

A wide rounded cup with a handle facing right

A white and beige stone cup horizontally split

An empty cup in studio

A wide rounded cup with a handle facing right 
                            A white and beige stone cup horizontally split 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A white cup in origami style

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A white cup in origami style 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A white cup with a black rim

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A white cup with a black rim 
                            An empty cup in studio

A rounded cylindric cup without a handle

A light greenish stone cup with white drips around the rim

An empty cup in studio

A rounded cylindric cup without a handle 
                            A light greenish stone cup with white drips around the rim 
                            An empty cup in studio

A cone shaped cup without a handle

A dark brown stone cup with vertical grooves

An empty cup in studio

A cone shaped cup without a handle 
                            A dark brown stone cup with vertical grooves 
                            An empty cup in studio

A cone shaped rounded cup with a handle facing right

A light blue stone cup with tiny brown dots and brown rim

An empty cup in studio

A cone shaped rounded cup with a handle facing right 
                            A light blue stone cup with tiny brown dots and brown rim 
                            An empty cup in studio

A cone shaped cup with a handle facing right

A old white cup with a widened bottom and a letter print

An empty cup in studio

A cone shaped cup with a handle facing right 
                            A old white cup with a widened bottom and a letter print 
                            An empty cup in studio

A cylindric cup with a round handle facing right

A red brown cup olive on the inside

An empty cup in studio

A cylindric cup with a round handle facing right 
                            A red brown cup olive on the inside 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A plain black cup

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A plain black cup 
                            An empty cup in studio

A high cone shaped cup with a handle facing right

A geometric grey stone cup with red blue and green splashes of color

An empty cup in studio

A high cone shaped cup with a handle facing right 
                            A geometric grey stone cup with red blue and green splashes of color 
                            An empty cup in studio

A cylindric cup with an angular handle facing right

A white cup printed with fish illustrations

An empty cup in studio

A cylindric cup with an angular handle facing right 
                            A white cup printed with fish illustrations 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A black cup sprinkled with yellow green and rose splashes of color

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A black cup sprinkled with yellow green and rose splashes of color 
                            An empty cup in studio

A cylindric cup with a handle facing right

A white cup painted with thin vertical lines

An empty cup in studio

A cylindric cup with a handle facing right 
                            A white cup painted with thin vertical lines 
                            An empty cup in studio

--->

<---

A cylindric camping cup with a handle facing right

A white cup with tiny black dots

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A white cup with tiny black dots 
                            An empty cup in studio

A rounded cone shaped cup without a handle

A white and blue cup printed with horizontal stripes and flowers

An empty cup in studio

A rounded cone shaped cup without a handle 
                            A white and blue cup printed with horizontal stripes and flowers 
                            An empty cup in studio

A slightly rounded cup with a big handle facing right

A white cup printed with an engraved Starbucks company logo

An empty cup in studio

A slightly rounded cup with a big handle facing right 
                            A white cup printed with an engraved Starbucks company logo 
                            An empty cup in studio

A high cylindric cup with a handle facing right

A white cup printed orange and black leaf shapes

An empty cup in studio

A high cylindric cup with a handle facing right 
                            A white cup printed orange and black leaf shapes 
                            An empty cup in studio

A rounded cone shaped cup without a handle

A white and beige cup printed with horizontal stripes and ornaments

An empty cup in studio

A rounded cone shaped cup without a handle 
                            A white and beige cup printed with horizontal stripes and ornaments 
                            An empty cup in studio

A cylindric cup with a big handle facing right

A white cup printed with red text

An empty cup in studio

A cylindric cup with a big handle facing right 
                            A white cup printed with red text 
                            An empty cup in studio

A cone shaped cup without a handle

A olive stone cup with a horizontal break

An empty cup in studio

A cone shaped cup without a handle 
                            A olive stone cup with a horizontal break 
                            An empty cup in studio

A cone shaped cup without a handle

A grey stone cup half glazed with white and turquoise

An empty cup in studio

A cone shaped cup without a handle 
                            A grey stone cup half glazed with white and turquoise 
                            An empty cup in studio

A high cylindric cup with a handle facing right

A white cup printed with a blue line pattern

An empty cup in studio

A high cylindric cup with a handle facing right 
                            A white cup printed with a blue line pattern 
                            An empty cup in studio

A cylindric camping cup with a handle facing right

A emerald cup with a silver rim

An empty cup in studio

A cylindric camping cup with a handle facing right 
                            A emerald cup with a silver rim 
                            An empty cup in studio

A cylindric cup with a handle facing right

A white cup printed with green and pink flower illustrations

An empty cup in studio

A cylindric cup with a handle facing right 
                            A white cup printed with green and pink flower illustrations 
                            An empty cup in studio

A rounded cone shaped cup with a handle facing right

A light blue stone cup with a brown rim

An empty cup in studio

A rounded cone shaped cup with a handle facing right 
                            A light blue stone cup with a brown rim 
                            An empty cup in studio

--->

Process ...

>

process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training

<---

a red cup

a red cup

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a cup used as a pencil holder

a cup used as a pencil holder

a red cup

a red cup

a red cup

a red cup

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a red cup

a red cup

a red cup

a red cup

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a red cup

a red cup

a blue and white striped cup orange illustration handle left used as a plant pot in studio

a blue and white striped cup orange illustration handle left 
                            used as a plant pot in studio

a cup used as a pencil holder

a cup used as a pencil holder

--->

<---

a cup filled with coffee beans

a cup filled with coffee beans

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup filled with coffee beans

a cup filled with coffee beans

a cup filled with coffee beans

a cup filled with coffee beans

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup filled with coffee beans

a cup filled with coffee beans

a cup filled with coffee beans

a cup filled with coffee beans

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot in studio

a cup used as a plant pot in studio

a cup filled with coffee beans

a cup filled with coffee beans

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot in studio

a cup used as a plant pot in studio

--->

>

Output: Learning

It was not possible to generate new configurations, like exchanging a colour or position of a cup and producing an image of a cup that did not exist in the dataset. But there are still some interesting learnings:

1. The more words in the prompt matched the image description the better was the reproduction of the image.

2. The longer the prompt, the more detailed the output.

3. Probably every single word was translated to a token. For example the prompts ‘used as a plant pot’ and ‘used as a panecil holder’ had three tokens in common, so with the prompt containing ‘used as a plant pot’ it sometimes also generated images of pencils.

<---

a red cup

a red cup

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a cup lying on its side facing front

a cup lying on its side facing front

a red cup

a red cup

a red cup

a red cup

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a cup lying on its side facing front

a cup lying on its side facing front

a cup lying on its side facing front

a cup lying on its side facing front

a red cup

a red cup

a red cup

a red cup

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a cup lying on its side facing front

a cup lying on its side facing front

a cup lying on its side facing front

a cup lying on its side facing front

a red cup

a red cup

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

a cup lying on its side facing front

a cup lying on its side facing front

--->

<---

a cup used as a plant pot

a cup used as a plant pot

a cup used as a pencil holder

a cup used as a pencil holder

a pencil holder

a pencil holder

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a pencil holder

a pencil holder

a pencil holder

a pencil holder

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a plant pot

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a cup used as a pencil holder

a pencil holder

a pencil holder

a pencil holder

a pencil holder

a cup used as a plant pot

a cup used as a plant pot

a cup used as a pencil holder

a cup used as a pencil holder

a pencil holder

a pencil holder

--->

<---

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

a blue and white striped cup

a blue and white striped cup

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a blue and white striped cup with a fruit painting

a photo of a blue and white striped cup with a fruit painted on it

a photo of a blue and white striped cup with a fruit painted on it

--->

Vs. Latent diffusion

Similar prompt used for Latent Diffusion provided further findings:

1. A cup often seems to be referred to football, so here we find different semantic connections. The fonts in the images might originate from memes, stock footage or other images including text on the internet.

2. Images on the internet rarely seem to be descriped by the direction of the handle or the cup lying on its side.

3. Adding the word ‘illustration’ or ‘paining’ to the prompt gets misinterpreted as a stylistic feature. By adding ‘a photo of…’ to the prompt it can be reversed.

Input: Dataset

Model Detail

Model name, date:
concept.pt, June 2022

Model type:
Dalle-pytorch trained on a dataset of 2138 text-image-pairs

Model Parameters:
default settings with pretrained VQ-VAE (- -taming) and ~1800 epochs

Licence:
Free to use trained model for further experiments

Intended Use

Intended Usecase:
Generation of images of cups and matching mood images. Research whether it is possible to enhance the stylistic features of the cup by combining the cups and the mood images.

Intended Users:
Researchers, Creatives / Artistic work

Training Data

Images of cups in a studio environment, sorted by their styles + matching mood images.

Description structure cups:
[mood] [shape] [colour] cup [feature] [filling] handle [direction]

Description structure mood images:
[mood] [type/object] [colour]

<---

Vintage flower illustration retro

Vintage flower illustration retro

Vintage empty conic rounded white ornamental cup handle right

Vintage empty conic rounded white ornamental cup handle right

Vintage empty cylindric cup handle right blue white stripes orange illustration

Vintage empty cylindric cup handle right blue white stripes orange illustration

Vintage empty conic white cup golden handle right

Vintage empty conic white cup golden handle right

Vintage photo retro

Vintage photo retro

Vintage orange illustration fruit

Vintage orange illustration fruit

Vintage empty cylindric cup handle right white orange stripes pear illustration.

Vintage empty cylindric cup handle right white orange stripes pear illustration.

Vintage empty conic round white blue dots cup handle right

Vintage empty conic round white blue dots cup handle right

Vintage empty cylindric white rosé floral illustration cup handle right

Vintage empty cylindric white rosé floral illustration cup handle right

Vintage photo Colors retro look

Vintage photo Colors retro look

Vintage empty conic copper glass cup handle right.

Vintage empty conic copper glass cup handle right.

Vintage empty rounded sage green grooves cup handle right

Vintage empty rounded sage green grooves cup handle right

Vintage colors structure retro

Vintage colors structure retro

Vintage photo gold Colors retro look

Vintage photo gold Colors retro look

--->

<---

Scandinavian empty high conic grey red blue green stone geometric cup handle right

Scandinavian empty high conic grey red blue green stone geometric cup handle right

Scandinavian empty cylindric pink grey stone cup handle right

Scandinavian empty cylindric pink grey stone cup  handle right

Scandinavian blue green brown stone glazed empty cylindric cup handle right

Scandinavian blue green brown stone glazed empty cylindric cup handle right

Scandinavian pattern stones dark blue

Scandinavian pattern stones dark blue

Scandinavian environment natural materials white

Scandinavian environment natural materials white

Scandinavian empty cylindric rounded white terracotta dipped stone cup handle right

Scandinavian empty cylindric rounded white terracotta dipped stone cup handle right

Scandinavian white blue stone glazed empty round cup handle right

Scandinavian white blue stone glazed empty round cup handle right

Scandinavian pattern white beige stone

Scandinavian pattern white beige stone

Scandinavian environment foggy landscape

Scandinavian environment foggy landscape

Scandinavian empty cylindric light blue brown rim stone glazed cup handle right

Scandinavian empty cylindric light blue brown rim stone glazed cup handle right

Scandinavian empty rounded dark blue stone glazed cup handle right

Scandinavian empty rounded dark blue stone glazed cup handle right

Scandinavian pattern blue beige white drips

Scandinavian pattern blue beige white drips

Scandinavian empty conic light blue brown rim stone glazed cup handle right

Scandinavian empty conic light blue brown rim stone glazed cup handle right

--->

<---

Plain clear environment water pink

Plain clear environment water pink

Plain clear pink empty cylindric cup handle right

Plain clear pink empty cylindric cup handle right

Plain clear blue empty cylindric cup handle right

Plain clear blue empty cylindric cup handle right

Plain clear empty round dark brown cup handle right

Plain clear empty round dark brown cup handle right

Plain clear object gradient blue violet yellow

Plain clear object gradient blue violet yellow

Plain clear blue empty cylindric cup handle right

Plain clear blue empty cylindric cup handle right

Plain clear sage green empty cylindric cup handle right

Plain clear sage green empty cylindric cup handle right

Plain clear empty rounded cylindric sage green cup handle right

Plain clear empty rounded cylindric sage green cup handle right

Plain clear geometric objects

Plain clear geometric objects

Plain clear object shiny sky blue

Plain clear object shiny sky blue

Plain clear empty conic brown cup handle left

Plain clear empty conic brown cup handle left

Plain clear object gold material

Plain clear object gold material

Plain clear empty conic sage green cup handle right

Plain clear empty conic sage green cup handle right

--->

<---

Identity story empty yellow red blue cup comic illustration cup handle right

Identity story empty yellow red blue cup comic illustration cup handle right

Identity story conic red vintage lettering cup empty handle right

Identity story conic red vintage lettering cup empty handle right

Identity story empty cylindric pink angel illustration cup handle right

Identity story empty cylindric pink angel illustration cup handle right

Identity story empty conic high white lettering cup broken handle right

Identity story empty conic high white lettering cup broken handle right

Identity story empty conic white gold painted cup handle right

Identity story empty conic white gold painted cup handle right

Identity story empty conic white lettering triangle printed cup handle right

Identity story empty conic white lettering triangle printed cup handle right

Identity story empty cylindric white black rim cup feminist illustration handle right

Identity story empty cylindric white black rim cup feminist illustration handle right

Identity story lettering

Identity story lettering

Identity story vintage lettering

Identity story vintage lettering

Identity story gold painted

Identity story gold painted

Identity story feminist illustration

Identity story feminist illustration

Identity story vintage lettering

Identity story vintage lettering

Identity story vintage lettering

Identity story vintage lettering

--->

Process ...

>

process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training process image of a dalle training

<---

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage empty cylindric cup handle right white orange stripes pear illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit

vintage orange illustration fruit

vintage orange illustration fruit

vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit

vintage orange illustration fruit

vintage orange illustration fruit

vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage empty cylindric cup handle right white orange stripes pear illustration vintage orange illustration fruit

vintage orange illustration fruit orange white striped cup orange illustration

vintage orange illustration fruit orange white striped cup orange illustration

--->

<---

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian light blue cup in scandinavian nature environment with scandinavian nature pattern

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian environment natural materials white

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

scandinavian empty dark blue stone cup scandinavian pattern stones dark blue

--->

<---

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

a red cup

a red cup

a red cup

a red cup

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

a red cup

a red cup

a plain white empty cup handle facing left in a studio

a plain white empty cup handle facing left in a studio

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

plain clear environment water pink rosé cup plant pot handle left

--->

>

Output: Learning

The training was stopped after 1800 epochs. Outputs were still quite glitched but some learnings can be taken from it:

1. By using exactly the image description as a prompt, the output was very realistic and well fitted. This was true for both the images of the cups as well as for the mood images, which were way fewer and more diverse.

2. As soon as the order of the words was disrupted or more words were added the output was blurred.

3. The order of the words in the input seem to be important: I tried combining the image descriptions of a cup and a matching mood image into a single prompt. When I entered the description for the cup first, it resembled more a cup and vice versa.

4. In some cases it seemed like the colours could be transferred, but with the output still being quite unclear (in some cases aesthetically interesting), it is hard to say wether the goal to enhance the moods of the cups could be achieved.

<---

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

a vintage retro coffee mug with orange stripes and an illustration of a pear with some fruit and a pear lying next to it sunset light

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

3d rendered environement in light pink and baby blue colors with a simple rounded cup standing on an island in a lake

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

scandinavian stone glazed cup standing in front of a snow covered mountain natural muted colors chill landscape hipster

a pencil holder

a pencil holder

--->

<---

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

playfully distorted cup covered all over in colorful confetti with red dots and heart shapes

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a minimalistic puristic cup with a print inspired by sound waves in a black and white environment

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

a vintage tea cup with a beautiful floral pattern with flower petals lying on a table stunning

--->

Vs. Latent diffusion

Similar prompt used for Latent Diffusion provided further findings:

1. The usage of long and precise prompts provided realistic images that fitted the description well in a sense of a very literal translation.

2. Because the text description was translated very literally, non of the outputs seems very surprising, ‘originally’ or ‘artistic’.

3. Related to the model I trained myself I can say that the model is not capable of producing anything it has not ‘seen’ before. In that sense it can not generate a cup that would be distorted in an interesting way to enhance its style like ‘no one has seen before’. Therefor I think calling those models ‘creative’ is far from reality.

Up next

There are many more possibilities to explore.
For example:

Dalle-pytorch settings

Dalle-pytorch offers more settings, which I did not try yet. I used the pre-trained VQ-VAE, but you can also train your own. The relation between the VAE and the actual training still needs to be explored.

Other Models

As dalle-pytorch is still based on a GAN architecture it is already quite old and newer architectures with Diffusion Models are already out. I hope to find repositories accessible also for non-coders.

Finetuning / Prompt-Learning

There are options to use a model trained on a big dataset as a starting point and train of top of that with your own data. That method is called finetuning.
Prompt-Learning works similar only you fill the empty spaces the latent space with your own data instead of manipulating the latent space with further training.
These two methods seem promising for generating images with my own content but also being efficient with lower training cost.

How can we make algorithms accessible and sustainable? Who wants to achieve which goal?

Artificial Intelligence

A mystified umbrella term for studies in Machine Learning, Self Learning Algorithms and others, commonly used for statistically driven decision-making, recommendation systems or output generation.

Machine Learning

A part of the field of AI research, feeding a machine with big amounts of data with the goal to have it find rulesets and (re)produce output accordingly.

Algorithm

A set of rules for a computer to follow, written in code.

PyTorch / Python

Pytorch is a open source framework used for machine learning, based on the programming language python.

Model

The expression of an algorithm trained on data and derived a ruleset from it, written in so called ‘weights’ and ‘biases’.

Text-to-Image Model

A Self-learning Algorithm trained on image-text-pairs to generate new images from a text prompt.

Tech and Development----------->

Interdisciplinary Work

Working across multiple research fields without limiting one’s work to the traditional borders of one discipline.

How can we engage in interdisciplinary work and exchange ideas without the boundaries of companies, institutions and disciplines?

Interdisciplinary research----->

Participation

“participation is important as a project: it rehumanises a society rendered numb and fragmented by the repressive instrumentality of capitalist production. Given the market’s near total saturation of our image repertoire, […] artistic practice can no longer revolve around the construction of objects to be consumed by a passive bystander. Instead, there must be an art of action, interfacing with reality, taking steps – however small – to repair the social bond.”*

How is the aesthetic value of images generated? How can algorithms become part of artistic or design practice?

Bias

Statistically, a system is “is biased if it’s consistently wrong in one direction or another”*
Commonly referred to as “prejudiced against a certain group or characteristic.”*

Aesthetics

A branch of philosophy which engages with sensual perception, often simplified as ‘beauty’. It investigates what generates the sensual significance of e.g. a piece of art.

Aesthetic research------------->

How to ensure diversity in data? Can data not be biased? What data do we want to use?

Canonisation

Art history knows an established canon, a selection of works that are universally known as the most important works of art. It has been criticised for not representing artists from diverse backgrounds. Inspiration for dealing with data could be drawn from this fight.

Model Card

A system to ”clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited”*

How did the relation between text and image evolve during history? How will the reception of established imaginary be shaped through new visualities?

Data

here: digitised information like images, text, etc.

Input

The data fed into a machine learning model, can either refer to the data used for training or the prompt

Output

The result of a task input to a machine learning model

Prompt

The sentence used as an input in a trained model. The instruction for the model to generate the output.

Art history-------------------->

Prompt Engineering

Strategies to control the output of a model. Includes the usage of subject and style keywords and semantics. For pre-trained models those strategies are developed in a trial and error approach.

How are things becoming sign? How is meaning generated?

Training

The action of a machine learning model processing data to acquire the rulesets of the given data

Finetuning / Prompt-Learning

Mostly applied in Natural Language Processing, Finetuning means using a pertrained model and training is further with more data fitted to a use case to enhance the quality of the output for a certain task.

Semiotics---------------------->

In which ways are images and text semiotically related or different?

Multimodality

Research about how communication is influenced by the intersection of different modalities. A modality can be images, sound, spoken or written language, etc.

Multimodality------------------>

Algorithms designing our future

Critical engagement with text-to-image algorithms and call for interdisciplinary research



Bachelor Thesis by Katharina Mumme

supervised by Prof. Peter Kabel
and Prof. Aram Bartholl

HAW Hamburg, Department Design, Visual Communication
April – July 2022




The repository can be found here
Due to large file sizes the .pt-files of the models cannot be uploaded here. In case you are interested feel free to message me.


Big shoutout to Phil Wang for recreating the architectures and making code available.


Fonts used on this website, thanks to the creators for making their work openly accessible:
Saira Condensed (Omnibus Type)
Office Code Pro (@nathco)


Special thanks to Fabian and Max for their amazing (coding) support
and to Katharina, Olivia, Lisa, Linda, Rebecca and Janne for letting me take pictures of their beautiful dishes.