Alternative Techniques

Transferring Hopper’s Nighthawks to the Marnie sequence led to a significant increase in noticeable artefacts in the approach described previously. In addition, Hopper’s realistic painting style cannot be reproduced adequately during upscaling. This is why an image sequence with 24 fps and a size of 1280 x 720 pixels is calculated using the NIN-Imagenet model. The images are then upscaled to 1920 x 1080 pixels and the colours are superimposed with the original images.

Marnie: Original and colour transferMarnie: Original and colour transfer

The clip is rotoscoped in After Effects using the RotoToon plug-in (Bristle). The result is superimposed with the source material to reduce any posterization that occurs. However, the end result is still unsatisfactory. Computation using multiple GPUs at a higher resolution seems to be a promising alternative for future test series.

Marnie: RotoToon and composite imageMarnie: RotoToon and composite image

Transferring the Lichtenstein painting (1) onto the Pulp Fiction sequence is also very problematic. The original, which has been reduced to a few colours and rasterized, leads to strong colour shifts using the conventional approach; the dot pattern itself cannot be transferred at all.

Pulp Fiction: Working stepsPulp Fiction: Working steps

The original material is therefore first processed in Photoshop and prepared using the gallery filters smudge stick and poster edges as well as the surface blur filter in such a way that significant tonal separation arises in combination with washed-out colour areas (2). The end result, on the other hand, serves as a template for computation using the neural-style algorithm (3). Next, the original colours are layered on top in After Effects (4) and the dot pattern is applied (5). Once again, however, the result is not ideal. It seems reasonable to train a model with corresponding reference images for further attempts.

Installation of neural-style
Working with neural-style
Style Transfer for Video Clips
Alternative Techniques