Image and video pencil sketch with OpenCV 3.X and CUDA

This prototype tests implementations of the pencil sketch operation for images and videos using C++, CUDA, OpenCV 3.X.

Sketching is a natural way of expressing some types of ideas. It conveys information that can be really hard to explain using text, and at the same time it does not require a tremendous amount of effort. It is also a suitable communication tool for children or illiterate people. As human-computer interaction moves towards easier and more high level languages, sketching will certainly continue to have its place in all sorts of applications, including image and shape retrieval, shape modeling and character animation. In a different perspective, sketching is possibly the most high-level and sparse type of visual media that can be understood by humans, which makes it an interesting object of study in computer vision. Why we can understand sketches so well and whether we can teach computers to do the same are research questions still in need of an answer.

Sketch classification is closely connected to image classification. Research in sketching goes back to the beginning of Computer Graphics as a field, with the development of SketchPad [Sutherland 1964]. Since then, much work has been devoted to the understanding of sketches, first constrained to specific low-level types, such as arcs and circles, and more recently extended to general objects.

Several sketch classifications exist and the most popular are Domain specific sketch recognitionGeneral sketch classification using Bag of Visual WordGeneral sketch classification using Structure-based Descriptors and Sketch classification and classification-driven analysis using Fisher vectors. 

sketch-logo

A sketch example

The sketching process usually has the following steps:

  1. Read in an image or a video frame by frame. The image can be either color or grayscale.
  2. Apply a blur filter to remove noise. In this prototype the Gaussian blur is used.
  3. Convert the RGB color image to grayscale (if it is the case).
  4. Extract the gradient image using the Sobel or Scharr operator.
  5. Optionally, apply a threshold operation to remove details.
  6. Invert the gradient image.
  7. Save the inverted gradient image or video frames.

OpenCV CPU

This first implementation is using only CPU only methods of OpenCV 3.X.

OpenCV GPU

This second implementation is using GPU methods of OpenCV 3.X.

Execution times of the 2 implementations

Original image and sketch images

sketch_color_1

sketch_color_1

sketch_out_color_1_sketching

sketch_out_color_1_sketching

sketch_color_2

sketch_color_2

sketch_out_color_2_sketching

sketch_out_color_2_sketching

sketch_color_3

sketch_color_3

sketch_out_color_3_sketching

sketch_out_color_3_sketching

sketch_color_4

sketch_color_4

sketch_out_color_4_sketching

sketch_out_color_4_sketching

sketch_color_5

sketch_color_5

sketch_out_color_5_sketching

sketch_out_color_5_sketching

sketch_ir_1

sketch_ir_1

sketch_out_ir_1_sketching

sketch_out_ir_1_sketching

sketch_ir_2

sketch_ir_2

sketch_out_ir_2_sketching

sketch_out_ir_2_sketching

sketch_ir_3

sketch_ir_3

sketch_out_ir_3_sketching

sketch_out_ir_3_sketching

sketch_ir_4

sketch_ir_4

sketch_out_ir_4_sketching

sketch_out_ir_4_sketching

sketch_ir_5

sketch_ir_5

sketch_out_ir_5_sketching

sketch_out_ir_5_sketching

Original video

Sketch video

Conclusion:

Sketch has been used as a basic visual communication media since pre-historic times and is becoming ubiquitous in recent years due to the popularity of touch sensitive devices such as smart phones and tablets. Neuroscience studies also suggest that sketching is the elementary way for human to conceptualize visual information, and is more expressive when retrieving images than raw text. This has resulted in a growing interest in automatic sketch recognition systems and sketch-based image retrieval (SBIR) systems.

Sources:

https://bitbucket.org/coldvisionio/coldvision-library/src/5cffa17a869dd190af29bcb9429c21272661244e/samples/2_imaging/sketching/

Resources:

https://en.wikipedia.org/wiki/Sketch_%28drawing%29

http://www.askaswiss.com/2016/01/how-to-create-pencil-sketch-opencv-python.html

http://www.learnopencv.com/non-photorealistic-rendering-using-opencv-python-c/

https://en.wikipedia.org/wiki/Sobel_operator

http://docs.opencv.org/2.4/doc/tutorials/imgproc/threshold/threshold.html

https://arxiv.org/ftp/arxiv/papers/1211/1211.2742.pdf

http://de.mathworks.com/help/vision/ug/image-classification-with-bag-of-visual-words.html

https://kops.uni-konstanz.de/bitstream/handle/123456789/15228/icmr11.pdf

http://www.inf.ufrgs.br/~rgschneider/research/2013a/paper.pdf

http://dl.acm.org/citation.cfm?id=2661231

http://www.eecs.qmul.ac.uk/~sgg/papers/LiEtAl_BMVC2013.pdf

http://docs.opencv.org/master/d8/dfe/classcv_1_1VideoCapture.html#gsc.tab=0

http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/histogram_equalization/histogram_equalization.html

http://stackoverflow.com/questions/5906693/how-to-reduce-the-number-of-colors-in-an-image-with-opencv

Leave a Reply

Your email address will not be published.