Blender and OpenCV
Hi!
I managed to have some good working interaction between OpenCV and Blender to do face tracking, using a simple webcam, in order to control a camera in the BGE. 
For those who don’t know what is OpenCV, they will find the answers here. OpenCV is a really powerful library for real-time image processing, so I strongly recommend you people interested in that to have a look at it!
Anyway how does it work this system I developed? There are basically two components:
- The face tracker
- The real-time BGE camera tracking
The face tracker gets frames from the webcam and for each frame checks if there are faces in the image using the pattern file “haarcascade_frontalface_alt.xml”, where the shape is defined by a set of coordinates (a lot as you can see yourself if you open the file!) which defines face’s pattern. Beware that multiple faces can be detected, but if you are using my example they won’t work, you must use just one face. Once the bounding box of the face is found, its center and area is calculated. CenterX and CenterY are used to control the camera position on the X and Z axis, while the area is the Y, so the bigger the closer the camera gets to the center of the scene (a cube). In the .blend file the area is not used because I’m still doing some experiments to get the best usage of that value, but you can try it yourself, just play with it.
The face tracking works really well in almost every light condition, but I recommend to have a plain light and a good color difference between you and the background. I haven’t made yet any screen recording to show you how it works, if somebody wants to do that in the meanwhile and send me the link of the video, I’ll put it in this post.
Note: this example works on Linux/OSX/Windows platforms as long as the webcam is supported by OpenCV. Mine wasn’t so I had to use VideoCapture module for windows. If you are going to run it in OSX/Linux, change the part of code regarding the webcam device initialization and image capture using the OpenCV methods instead of VideoCapture ones and you are ready to go! Basically you just need a PIL image format value that can be passed computed by OpenCV.
You can find both sources and a win32 build that doesn’t need anything installed on your machine, just run it! The python script needs OpenCV installed. Once you install it you will find in the folder a Python26 subfolder with libs and DLLs. Copy them in your Python folder and that’s all, those are the python wrappers for OpenCV.
- Download BlendCV win32 binary
Akta
December 16th, 2009 at 6:01 pm
whoa! thanks for posting the sources. is the .blend file in there too?
December 16th, 2009 at 9:35 pm
Yes of course, it’s in the sources package
January 11th, 2010 at 10:55 pm
awesome work! thanks for posting the complete package. this is a great help.
January 22nd, 2010 at 2:02 pm
fantastic sample, thank you
May 24th, 2010 at 7:45 am
Very impressive! I however cannot seem to get opencv to talk to blender. I used your blender (blendcv source) files and although I can get Facedetect to work and blender to work, the gameengine does not respond to any head movements. Icopied the python26 files from opencv to the Python directory and also directly into the directory where blendcv.blend is but still no joy. I also installed VideoCapture but that made no difference.
As I am a complete novice and do not know how to program/compile, can you please give me a more detailed guide on how to marry up the two programs?
Alternatively I have a mac. I do not understand the instructions on compiling face tracker in xcode. Is there an easy way to explain this?
Thank you in advance and keep up the amazing stuff
Cheers
Ad
May 24th, 2010 at 7:56 am
Hey Ad thanks!
Basically the communication between the Face detector and Blender happensa via local socket. So it might be that some antivirus or firewall is stopping the communication between them. Try to have a look at the source code and you will better understand what I mean
Cheers,
Marco
May 24th, 2010 at 10:34 pm
Thanks for the reply. I’m really showing my ignorance here… I turned off the firewalls but the socket is still not recognised in blender. Do I need to add something to get them to talk? The sample program Face detect works.
May 25th, 2010 at 6:30 pm
I think I found the source of the problem. When i run blendcv from the blendcv folder I get the error message:
File “blendcv.py”, line 59, in
File “C:\Python26\lib\VideoCapture.py:’ line 49, in_init_
self.dev = vidcap.new_Dev(devnum, showVideoWindow)
vidcap.error: Capture Graph could not be created.
What does that mean?
ad