Using the Torch implementation by Justin Johnson, an algorithm was developed by Manuel Ruder, Alexey Dosovitskiy and Thomas Brox which facilitates a consistent, low-intensity stylistic transfer of an image to a video sequence. The artistic-videos algorithm requires the full installation of neural-style (Torch, loadcaffe, CUDA backend, cuDNN, neural-style). The painting style is transferred to an image sequence using DeepFlow, an algorithm for calculating the optical flow of images.
Installation is easy to perform. DeepFlow and DeepMatching can be downloaded as CPU versions here. Next, deepmatching-static and deepflow2-static are copied to the neural-style or artistic-videos main directory.
The calculation of the optical flow and the processing of the video clip are launched with the following command in a terminal window:
|
1 |
th artistic_video.lua |
For details on this, see the corresponding GitHub page by Manuel Ruder. Alternatively, the stylizeVideo.sh file can be edited and launched. The script takes the following form for transferring the Caligari sequence:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
set -e # Get a carriage return into `cr` cr=`echo $'\n.'` cr=${cr%.} # Find out whether ffmpeg or avconv is installed on the system FFMPEG=ffmpeg command -v $FFMPEG >/dev/null 2>&1 || { FFMPEG=avconv command -v $FFMPEG >/dev/null 2>&1 || { echo >&2 "This script requires either ffmpeg or avconv installed. Aborting."; exit 1; } } if [ "$#" -le 1 ]; then echo "Usage: ./stylizeVideo exit 1 fi # Parse arguments filename=$(basename "$1") extension="${filename##*.}" filename="${filename%.*}" filename=${filename//[%]/x} style_image=$2 # Create output folder mkdir -p $filename echo "" read -p "resolution w:h $cr > " resolution # Save frames of the video as individual image files if [ -z $resolution ]; then $FFMPEG -i $1 ${filename}/frame_%04d.ppm resolution= else $FFMPEG -i $1 -vf scale=$resolution ${filename}/frame_%04d.ppm fi echo "" echo "Computing optical flow. This may take a while..." bash makeOptFlow.sh ./${filename}/frame_%04d.ppm ./${filename}/flow_$resolution # Perform style transfer th artistic_video.lua \ -content_pattern ${filename}/frame_%04d.ppm \ -flow_pattern ${filename}/flow_${resolution}/backward_[%d]_{%d}.flo \ -flowWeight_pattern ${filename}/flow_${resolution}/reliable_[%d]_{%d}.pgm \ -output_folder ${filename}/ \ -style_image $style_image \ -seed 125 \ -content_weight 10 \ -style_weight 1000 \ -style_scale 1 \ -init image \ -backend cudnn \ -cudnn_autotune \ -number_format %04d \ |
Installation of neural-style
Working with neural-style
Style Transfer for Video Clips
Post-production
Alternative Techniques