In this article we will discuss how to create a Java based LBP face recognizer using JavaCV. JavaCV is a wrapper to the Open CV libraries for image processing. The workflow that I am trying to implement is as below
- The calling program will supply a bunch of images and a person name to this class ( learnNewFace(String personName, IplImage[] images)
- The class will store the images in a folder and map the person name to an integer id
- The class will then read all the images and person names and train the model
- Once training is complete client can invoke identifyFace method giving an image to find out the matching personName (identifyFace(IplImage image))
Step -1)
Install JDK and Eclipse ( I used JDK 7 on 64 bit Windows 7 PC)
Step-2)
Install OpenCV and JavaCV(details can be found at http://code.google.com/p/javacv/). One thing we need to keep in mind is that the bit-ness of JDK and JavaCV and OpenCV must match. That means for a 64 bit JDK, you have to use the javacv-linux-x86_64 jar along with the other jar files. Also, please make sure you extract the opencv to C:\ drive. Extracting anywhere else is a nightmare, to setup the correct path variables.
Step -3)
Create a new Java project in Eclipse and add the following jar files to the class path
javacv.jar, javacpp.jar and one of the javacv-linux-x86.jar,javacv-linux-x86_64.jar,javacv-macosx-x86_64.jar,javacv-windows-x86.jar,javacv-windows-x86_64.jar according to your operating system
The workspace should look like this
Step-4)
Create a new class LBPFaceRecognizer in a package and copy the below code
The code is given below
package com.test.face;
import static com.googlecode.javacv.cpp.opencv_contrib.createLBPHFaceRecognizer;
import static com.googlecode.javacv.cpp.opencv_core.CV_32SC1;
import static com.googlecode.javacv.cpp.opencv_core.IPL_DEPTH_8U;
import static com.googlecode.javacv.cpp.opencv_core.cvCreateImage;
import static com.googlecode.javacv.cpp.opencv_core.cvGetSize;
import static com.googlecode.javacv.cpp.opencv_core.cvLoad;
import static com.googlecode.javacv.cpp.opencv_core.cvSetImageROI;
import static com.googlecode.javacv.cpp.opencv_highgui.cvLoadImage;
import static com.googlecode.javacv.cpp.opencv_highgui.cvSaveImage;
import static com.googlecode.javacv.cpp.opencv_imgproc.CV_BGR2GRAY;
import static com.googlecode.javacv.cpp.opencv_imgproc.CV_INTER_LINEAR;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvCvtColor;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvEqualizeHist;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvResize;
import static com.googlecode.javacv.cpp.opencv_objdetect.cvHaarDetectObjects;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.Iterator;
import java.util.Properties;
import java.util.Set;
import com.googlecode.javacpp.Loader;
import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizer;
import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;
import com.googlecode.javacv.cpp.opencv_core.CvMat;
import com.googlecode.javacv.cpp.opencv_core.CvMemStorage;
import com.googlecode.javacv.cpp.opencv_core.CvRect;
import com.googlecode.javacv.cpp.opencv_core.CvSeq;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
import com.googlecode.javacv.cpp.opencv_core.MatVector;
import com.googlecode.javacv.cpp.opencv_objdetect;
import com.googlecode.javacv.cpp.opencv_objdetect.CvHaarClassifierCascade;
public class LBPFaceRecognizer {
private static String faceDataFolder = "C:\\test\\data\\";
public static String imageDataFolder = faceDataFolder + "images\\";
private static final String CASCADE_FILE = "C:\\opencv\\data\\haarcascades\\haarcascade_frontalface_alt.xml";
private static final String frBinary_DataFile = faceDataFolder + "frBinary.dat";
public static final String personNameMappingFileName = faceDataFolder + "personNumberMap.properties";
final CvHaarClassifierCascade cascade = new CvHaarClassifierCascade(cvLoad(CASCADE_FILE));
private Properties dataMap = new Properties();
private static LBPFaceRecognizer instance = new LBPFaceRecognizer();
public static final int NUM_IMAGES_PER_PERSON =10;
double binaryTreshold = 100;
int highConfidenceLevel = 70;
FaceRecognizerPtr ptr_binary = null;
private FaceRecognizer fr_binary = null;
private LBPFaceRecognizer() {
createModels();
loadTrainingData();
}
public static LBPFaceRecognizer getInstance() {
return instance;
}
private void createModels() {
ptr_binary = createLBPHFaceRecognizer(1, 8, 8, 8, binaryTreshold);
fr_binary = ptr_binary.get();
}
protected CvSeq detectFace(IplImage originalImage) {
CvSeq faces = null;
Loader.load(opencv_objdetect.class);
try {
IplImage grayImage = IplImage.create(originalImage.width(), originalImage.height(), IPL_DEPTH_8U, 1);
cvCvtColor(originalImage, grayImage, CV_BGR2GRAY);
CvMemStorage storage = CvMemStorage.create();
faces = cvHaarDetectObjects(grayImage, cascade, storage, 1.1, 1, 0);
} catch (Exception e) {
e.printStackTrace();
}
return faces;
}
public String identifyFace(IplImage image) {
System.err.println("==========================================================");
String personName = "";
Set keys = dataMap.keySet();
if (keys.size() > 0) {
int[] ids = new int[1];
double[] distance = new double[1];
int result = -1;
fr_binary.predict(image, ids, distance);
//just deriving a confidence number against treshold
result = ids[0];
if (result > -1 && distance[0]<highConfidenceLevel) {
personName = (String) dataMap.get("" + result);
}
}
return personName;
}
//The logic to learn a new face is to store the recorded images to a folder and retrain the model
//will be replaced once update feature is available
public boolean learnNewFace(String personName, IplImage[] images) throws Exception {
int memberCounter = dataMap.size();
if(dataMap.containsValue(personName)){
Set keys = dataMap.keySet();
Iterator ite = keys.iterator();
while (ite.hasNext()) {
String personKeyForTraining = (String) ite.next();
String personNameForTraining = (String) dataMap.getProperty(personKeyForTraining);
if(personNameForTraining.equals(personName)){
memberCounter = Integer.parseInt(personKeyForTraining);
System.err.println("Person already exist.. re-learning..");
}
}
}
dataMap.put("" + memberCounter, personName);
storeTrainingImages(personName, images);
retrainAll();
return true;
}
public IplImage preprocessImage(IplImage image, CvRect r){
IplImage gray = cvCreateImage(cvGetSize(image), IPL_DEPTH_8U, 1);
IplImage roi = cvCreateImage(cvGetSize(image), IPL_DEPTH_8U, 1);
CvRect r1 = new CvRect(r.x()-10, r.y()-10, r.width()+10, r.height()+10);
cvCvtColor(image, gray, CV_BGR2GRAY);
cvSetImageROI(gray, r1);
cvResize(gray, roi, CV_INTER_LINEAR);
cvEqualizeHist(roi, roi);
return roi;
}
private void retrainAll() throws Exception {
Set keys = dataMap.keySet();
if (keys.size() > 0) {
MatVector trainImages = new MatVector(keys.size() * NUM_IMAGES_PER_PERSON);
CvMat trainLabels = CvMat.create(keys.size() * NUM_IMAGES_PER_PERSON, 1, CV_32SC1);
Iterator ite = keys.iterator();
int count = 0;
System.err.print("Loading images for training...");
while (ite.hasNext()) {
String personKeyForTraining = (String) ite.next();
String personNameForTraining = (String) dataMap.getProperty(personKeyForTraining);
IplImage[] imagesForTraining = readImages(personNameForTraining);
IplImage grayImage = IplImage.create(imagesForTraining[0].width(), imagesForTraining[0].height(), IPL_DEPTH_8U, 1);
for (int i = 0; i < imagesForTraining.length; i++) {
trainLabels.put(count, 0, Integer.parseInt(personKeyForTraining));
cvCvtColor(imagesForTraining[i], grayImage, CV_BGR2GRAY);
trainImages.put(count,grayImage);
count++;
}
//storeNormalizedImages(personNameForTraining, imagesForTraining);
}
System.err.println("done.");
System.err.print("Training Binary model ....");
fr_binary.train(trainImages, trainLabels);
System.err.println("done.");
storeTrainingData();
}
}
private void loadTrainingData() {
try {
File personNameMapFile = new File(personNameMappingFileName);
if (personNameMapFile.exists()) {
FileInputStream fis = new FileInputStream(personNameMappingFileName);
dataMap.load(fis);
fis.close();
}
File binaryDataFile = new File(frBinary_DataFile);
System.err.print("Loading Binary model ....");
fr_binary.load(frBinary_DataFile);
System.err.println("done");
} catch (Exception e) {
e.printStackTrace();
}
}
private void storeTrainingData() throws Exception {
System.err.print("Storing training models ....");
File binaryDataFile = new File(frBinary_DataFile);
if (binaryDataFile.exists()) {
binaryDataFile.delete();
}
fr_binary.save(frBinary_DataFile);
File personNameMapFile = new File(personNameMappingFileName);
if (personNameMapFile.exists()) {
personNameMapFile.delete();
}
FileOutputStream fos = new FileOutputStream(personNameMapFile, false);
dataMap.store(fos, "");
fos.close();
System.err.println("done.");
}
public void storeTrainingImages(String personName, IplImage[] images) {
for (int i = 0; i < images.length; i++) {
String imageFileName = imageDataFolder + "training\\" + personName + "_" + i + ".bmp";
File imgFile = new File(imageFileName);
if (imgFile.exists()) {
imgFile.delete();
}
cvSaveImage(imageFileName, images[i]);
}
}
private IplImage[] readImages(String personName) {
File imgFolder = new File(imageDataFolder);
IplImage[] images = null;
if (imgFolder.isDirectory() && imgFolder.exists()) {
images = new IplImage[NUM_IMAGES_PER_PERSON];
for (int i = 0; i < NUM_IMAGES_PER_PERSON; i++) {
String imageFileName = imageDataFolder + "training\\" + personName + "_" + i + ".bmp";
IplImage img = cvLoadImage(imageFileName);
images[i] = img;
}
}
return images;
}
}
Step-5)
Write a client code. The client code must do the following
1) Get an instance of the face recognizer
LBPFaceRecognizer fr = LBPFaceRecognizer.getInstance();
2) Capture image and grab the face coordinates using the code described here
FrameGrabber grabber = new OpenCVFrameGrabber(0);
grabber.start();
IplImage img = grabber.grab();
IplImage snapshot = cvCreateImage(cvGetSize(img), img.depth(), img.nChannels());
cvFlip(img, snapshot, 1);
CvSeq faces = fr.detectFace(img);
CvRect r = new CvRect(cvGetSeqElem(faces, 0));
3) Preprocess the image using fr.preprocess
trainImages[imageCounter] = fr.preprocessImage(img, r);
4) Invoke the learn method
fr.learnNewFace("User1", trainImages);
5) Once the learning is complete you can use fr.identiofyFace to identify a face
fr.identifyFace(fr.preprocessImage(img, r));
NOTE: The convenience method preprocessImage is used to clip face, resize and normalize image to improve accuracy.
Happy coding :)
Cool article. Now I know where to point people asking me for the FaceRecognizer with JavaCV! :) Thanks for writing it.
ReplyDeleteSure..Thanks to Samuel for creating such a great framework
DeleteHow do you call the FaceRecognizerPtr using the updated JavaCV 0.3? Thanks :)
ReplyDeleteSorry, I did not get a chance to play with JavaCV.03. But I am assuming it wont be much different.
DeleteHello there, did you find a solution for this problem I can not find the FaceRecognizerPtr class!
DeleteI'm having the same problem with FaceRecognizerPtr class that cant be found
Deleteimport com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;
Deletecan someone give client class to me
Deleteanyone got the solution on FaceRecognizerPtr class? IDE is unable to fine FaceRecognizerPtr class in java.
Deletepls upload the solution.
I cant found this class too...
Deletei have tested your implementation and trained the system successfully but the predict function always returns the last label trained in the databased .... Did you encounter this problem?
ReplyDeleteYou have to adjust the threshold variable in the first few lines of the code to fine tune the performance of the face recognizer. The logic computes confidence based on the threshold value.
Deletedouble binaryTreshold = 100;
Its a painful trial and error procedure.. but after some fine tuning you will get the correct response.
i have encounter the same problem as Iman Firouzian had.
ReplyDeletethis trainImages[]and imagecounter on the client code,where they came from ?
ReplyDeletesorry if i missed sometihing...
trainImages[] should be supplied by the applciation using the face recognizer. I had a small application that capture images using the computer camera and collect the images into an array and send it to the Face recognizer
DeleteWhat kind of file is frBinary.dat ??
ReplyDeleteWhat content does it hold???
frBinary.dat is a file that will be created by the program. The only input to the program is the face images and haarcascade.
DeletefrBinary.dat will contain the binary model saved into a file. The program will reload the frBinary.dat next time you run, so that so dont have to retrain the model again. I
Exelente trabajo, estoy trabajando en netbeans 7.1.2 pero a la hora de compilar el proyecto me manda
ReplyDelete"Cargando archivo binario ....OpenCV Error: Unspecified error (File can't be opened for writing!) in unknown function, file ..\..\..\src\opencv\modules\contrib\src\facerec.cpp, line 304"
Alguien sabe cómo solucionarlo?line 304
Excellent work, I am working in netbeans 7.1.2 but when I compile the project send
ReplyDelete"Loading binary file .... OpenCV Error: Unspecified Error (File can not be opened for writing!) In unknown function, file .. \ .. \ .. \ src \ opencv \ modules \ contrib \ src \ facerec. cpp, line 304 "
Anyone know how to fix it?
Hi, I am kind of stuck at step 2, as I have some trouble with javacv as most of the classes from com.googlecode.javacv.cpp package are annotated with com.googlecode.javacpp.annotation.Properties and that annotation uses a property inherit which is not defined in Properties annotation. Using javacpp-0.5.jar. I'm thinking to remove that inherit annotation property in order to compile, but I am not sure if that is alright.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteAlso, where can you find specific platform jar files? (javacv-linux-x86.jar,javacv-linux-x86_64.jar,javacv-macosx-x86_64.jar,javacv-windows-x86.jar,javacv-windows-x86_64.jar). As these are not shipped with javacv. Thanks!
ReplyDeletehi, I'm not able to locate FaceRecognizerPtr class in opencv ..can any one help out with the issue.
ReplyDeleteAdd this to the imports
Deleteimport com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;
The import is not working even after importing all the mentioned .jar files. So can you suggest some better solution to this?
DeleteCause JavaCV 0.3 changed a lot.
DeleteYou can download and use v0.2
Another way, just replace FaceRecognizerPtr by FaceRecognizer ( remove .get() or sample code above)
Hi!!
ReplyDeletei m facing an error in this method
private void createModels() {
ptr_binary = createLBPHFaceRecognizer(1, 8, 8, 8, binaryTreshold);
---->>fr_binary = ptr_binary.get();( Error over here)
}
help me with the solution for my error.
Thank You.
remove .get() if you changed from FaceRecognizerPtr to FaceRecognizer
DeleteEntire steo2 is failing in my case, any method call like cvFlip or cvGetSeq... anything releated to CV is failed or crashing
ReplyDeleteI didn't get the last part about the client code. What I did is:
ReplyDelete1) I created a new java class and the following-
package com.test.face;
import static com.googlecode.javacv.cpp.opencv_core.cvCreateImage;
import static com.googlecode.javacv.cpp.opencv_core.cvFlip;
import static com.googlecode.javacv.cpp.opencv_core.cvGetSeqElem;
import static com.googlecode.javacv.cpp.opencv_core.cvGetSize;
import com.googlecode.javacv.FrameGrabber;
import com.googlecode.javacv.OpenCVFrameGrabber;
import com.googlecode.javacv.cpp.opencv_core.*;
public class Client {
public static void main(String s){
LBPFaceRecognizer fr = LBPFaceRecognizer.getInstance();
FrameGrabber grabber = new OpenCVFrameGrabber(0);
try{
grabber.start();
IplImage img = grabber.grab();
IplImage snapshot = cvCreateImage(cvGetSize(img), img.depth(), img.nChannels());
cvFlip(img, snapshot, 1);
CvSeq faces = fr.detectFace(img);
CvRect r = new CvRect(cvGetSeqElem(faces, 0));
IplImage[] trainImages = null;
int imageCounter = 0;
trainImages[imageCounter] = fr.preprocessImage(img, r);
fr.learnNewFace("User1", trainImages);
fr.identifyFace(fr.preprocessImage(img, r));
}
catch(Exception e){
System.out.println("Getting error:::"+e.getMessage());
System.out.println("+++++++++++++++++++++++++++++");
}
}
}
But this is not executing. Should I specify a list of images to trainImages object?
Please tell the part about the client code..
ReplyDeleteHi,thanks for this code.
ReplyDeleteBut you plz write the client code,because i'm having difficulties writng it.
And thank you.
ReplyDeleteException in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.googlecode.javacpp.Loader.load(Loader.java:553)
at com.googlecode.javacpp.Loader.load(Loader.java:532)
at com.googlecode.javacv.cpp.opencv_objdetect$CvHaarClassifierCascade.(opencv_objdetect.java:170)
at test2.LBPFaceRecognizer.(LBPFaceRecognizer.java:58)
at test2.LBPFaceRecognizer.(LBPFaceRecognizer.java:60)
at test2.test.main(test.java:30)
Caused by: java.lang.IllegalStateException: Can't overwrite cause
at java.lang.Throwable.initCause(Throwable.java:456)
at com.googlecode.javacpp.Loader.load(Loader.java:581)
at com.googlecode.javacpp.Loader.load(Loader.java:532)
at com.googlecode.javacv.cpp.opencv_objdetect.(opencv_objdetect.java:91)
... 8 more
Java Result: 1
FaceRecognizerPtr ptr_binary = null; (Can not find symbol )
ReplyDeletegetting trouble with above variable.
Any one help for above problem.
I got error like this : Could not find method com.googlecode.javacv.cpp.opencv_contrib.createLBPHFaceRecognizer, referenced from method org.opencv.samples.facedetect.FdActivity.
ReplyDeletePlease help me...
I don't understand. Why people post a half done solution here. Is it a challenge to debug it, finish it and wrote a client for it?
ReplyDeleteNeed full source code
ReplyDeleteI am getting error at FaceRecognizerPtr.... What should I replace it with..? I didn't get the solution of remove mentioned above.. I am using the import org.bytedeco.javacpp...... because com.googlecode is not available I guess ..pls reply
ReplyDeleteAnd Thanks for the code forgot to mention it above..! :)
ReplyDeleteCan I use android camera to capture a image instead off 'FrameGrabber?' If so then how can I convert the captured image to IplImage?
ReplyDeleteHi, thank you for sharing this code, but unfortunately I'm having this issue:
ReplyDeleteLoading Binary model ....OpenCV Error: Unspecified error (File can't be opened for writing!) in unknown function, file ..\..\..\src\opencv\modules\contrib\src\facerec.cpp, line 325
java.lang.RuntimeException: ..\..\..\src\opencv\modules\contrib\src\facerec.cpp:325: error: (-2) File can't be opened for writing!
Can someone help me?
need full client code
ReplyDeleteUnspecified error (File can't be opened for writing!) in cv::FaceRecognizer::load, file ..\..\..\..\opencv\modules\contrib\src\facerec.cpp, line 325
ReplyDeletejava.lang.RuntimeException: ..\..\..\..\opencv\modules\contrib\src\facerec.cpp:325: error: (-2) File can't be opened for writing! in function cv::FaceRecognizer::load
i have implemented client code and also run if fine upto some extend.
ReplyDeletebut my pc gets slow down whenever i run it.
also face detection and recognition have less accuracy.
could you help me? plzzz..
"Loading binary file .... OpenCV Error: Unspecified Error (File can not be opened for writing!) In unknown function, file .. \ .. \ .. \ src \ opencv \ modules \ contrib \ src \ facerec. cpp, line 304 "
ReplyDeleteAnyone know how to fix it?
anyone have a full working code yet? i have so many problems with this one
ReplyDeleteytttt
ReplyDelete