Saturday, October 27, 2012

Creating an LBPH (Local Binary Pattern Histogram) face recognizer using FaceRecognizer class in JavaCV


In this article we will discuss how to create a Java based LBP face recognizer using JavaCV. JavaCV is a wrapper to the Open CV libraries for image processing. The workflow that I am trying to implement is as below

  • The calling program will supply a bunch of images and a person name to this class ( learnNewFace(String personName, IplImage[] images)
  • The class will store the images in a folder and map the person name to an integer id
  • The class will then read all the images and person names and train the model
  • Once training is complete client can invoke identifyFace method giving an image to find out the matching personName (identifyFace(IplImage image))



Step -1)
Install JDK and Eclipse ( I used JDK 7 on 64 bit Windows 7 PC)

Step-2)
Install OpenCV and JavaCV(details can be found at http://code.google.com/p/javacv/). One thing we need to keep in mind is that the bit-ness of JDK and JavaCV  and OpenCV must match. That means for a 64 bit JDK, you have to use the javacv-linux-x86_64 jar along with the other jar files. Also, please make sure you extract the opencv to C:\ drive. Extracting anywhere else is a nightmare, to setup the correct path variables.


Step -3)
Create a new Java project in Eclipse and add the following jar files to the class path
javacv.jar, javacpp.jar  and one of the javacv-linux-x86.jar,javacv-linux-x86_64.jar,javacv-macosx-x86_64.jar,javacv-windows-x86.jar,javacv-windows-x86_64.jar according to your operating system
The workspace should look like this


Step-4)
Create a new class LBPFaceRecognizer in a package and copy the below code
The code is given below
package com.test.face;

import static com.googlecode.javacv.cpp.opencv_contrib.createLBPHFaceRecognizer;
import static com.googlecode.javacv.cpp.opencv_core.CV_32SC1;
import static com.googlecode.javacv.cpp.opencv_core.IPL_DEPTH_8U;
import static com.googlecode.javacv.cpp.opencv_core.cvCreateImage;
import static com.googlecode.javacv.cpp.opencv_core.cvGetSize;
import static com.googlecode.javacv.cpp.opencv_core.cvLoad;
import static com.googlecode.javacv.cpp.opencv_core.cvSetImageROI;
import static com.googlecode.javacv.cpp.opencv_highgui.cvLoadImage;
import static com.googlecode.javacv.cpp.opencv_highgui.cvSaveImage;
import static com.googlecode.javacv.cpp.opencv_imgproc.CV_BGR2GRAY;
import static com.googlecode.javacv.cpp.opencv_imgproc.CV_INTER_LINEAR;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvCvtColor;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvEqualizeHist;
import static com.googlecode.javacv.cpp.opencv_imgproc.cvResize;
import static com.googlecode.javacv.cpp.opencv_objdetect.cvHaarDetectObjects;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.Iterator;
import java.util.Properties;
import java.util.Set;

import com.googlecode.javacpp.Loader;
import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizer;
import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;
import com.googlecode.javacv.cpp.opencv_core.CvMat;
import com.googlecode.javacv.cpp.opencv_core.CvMemStorage;
import com.googlecode.javacv.cpp.opencv_core.CvRect;
import com.googlecode.javacv.cpp.opencv_core.CvSeq;
import com.googlecode.javacv.cpp.opencv_core.IplImage;
import com.googlecode.javacv.cpp.opencv_core.MatVector;
import com.googlecode.javacv.cpp.opencv_objdetect;
import com.googlecode.javacv.cpp.opencv_objdetect.CvHaarClassifierCascade;

public class LBPFaceRecognizer {



        private static String faceDataFolder = "C:\\test\\data\\";
        public static String imageDataFolder = faceDataFolder + "images\\";
        private static final String CASCADE_FILE = "C:\\opencv\\data\\haarcascades\\haarcascade_frontalface_alt.xml";
        private static final String frBinary_DataFile = faceDataFolder + "frBinary.dat";
        public static final String personNameMappingFileName = faceDataFolder + "personNumberMap.properties";

        final CvHaarClassifierCascade cascade = new CvHaarClassifierCascade(cvLoad(CASCADE_FILE));
        private Properties dataMap = new Properties();
        private static LBPFaceRecognizer instance = new LBPFaceRecognizer();

        public static final int NUM_IMAGES_PER_PERSON =10;
        double binaryTreshold = 100;
        int highConfidenceLevel = 70;

        FaceRecognizerPtr ptr_binary = null;
        private FaceRecognizer fr_binary = null;

        private LBPFaceRecognizer() {
                createModels();
                loadTrainingData();
        }

        public static LBPFaceRecognizer getInstance() {
                return instance;
        }

        private void createModels() {
                ptr_binary = createLBPHFaceRecognizer(1, 8, 8, 8, binaryTreshold);
                fr_binary = ptr_binary.get();
        }

        protected CvSeq detectFace(IplImage originalImage) {
                CvSeq faces = null;
                Loader.load(opencv_objdetect.class);
                try {
                        IplImage grayImage = IplImage.create(originalImage.width(), originalImage.height(), IPL_DEPTH_8U, 1);
                        cvCvtColor(originalImage, grayImage, CV_BGR2GRAY);
                        CvMemStorage storage = CvMemStorage.create();
                        faces = cvHaarDetectObjects(grayImage, cascade, storage, 1.1, 1, 0);

                } catch (Exception e) {
                        e.printStackTrace();
                }
                return faces;
        }

        public String identifyFace(IplImage image) {
                System.err.println("==========================================================");
                String personName = "";

                Set keys = dataMap.keySet();

                if (keys.size() > 0) {
                        int[] ids = new int[1];
                        double[] distance = new double[1];
                        int result = -1;

                                fr_binary.predict(image, ids, distance);
                                //just deriving a confidence number against treshold
                                result = ids[0];

                                if (result > -1 && distance[0]<highConfidenceLevel) {
                                        personName = (String) dataMap.get("" + result);
                                }
                }

                return personName;
        }


        //The logic to learn a new face is to store the recorded images to a folder and retrain the model
        //will be replaced once update feature is available
        public boolean learnNewFace(String personName, IplImage[] images) throws Exception {
                int memberCounter = dataMap.size();
                if(dataMap.containsValue(personName)){
                        Set keys = dataMap.keySet();
                        Iterator ite = keys.iterator();
                        while (ite.hasNext()) {
                                String personKeyForTraining = (String) ite.next();
                                String personNameForTraining = (String) dataMap.getProperty(personKeyForTraining);
                                if(personNameForTraining.equals(personName)){
                                        memberCounter = Integer.parseInt(personKeyForTraining);
                                        System.err.println("Person already exist.. re-learning..");
                                }
                        }
                }
                dataMap.put("" + memberCounter, personName);
                storeTrainingImages(personName, images);
                retrainAll();

                return true;
        }


        public IplImage preprocessImage(IplImage image, CvRect r){
                IplImage gray = cvCreateImage(cvGetSize(image), IPL_DEPTH_8U, 1);
                IplImage roi = cvCreateImage(cvGetSize(image), IPL_DEPTH_8U, 1);
                CvRect r1 = new CvRect(r.x()-10, r.y()-10, r.width()+10, r.height()+10);
                cvCvtColor(image, gray, CV_BGR2GRAY);
                cvSetImageROI(gray, r1);
                cvResize(gray, roi, CV_INTER_LINEAR);
                cvEqualizeHist(roi, roi);
                return roi;
        }

        private void retrainAll() throws Exception {
                Set keys = dataMap.keySet();
                if (keys.size() > 0) {
                        MatVector trainImages = new MatVector(keys.size() * NUM_IMAGES_PER_PERSON);
                        CvMat trainLabels = CvMat.create(keys.size() * NUM_IMAGES_PER_PERSON, 1, CV_32SC1);
                        Iterator ite = keys.iterator();
                        int count = 0;

                        System.err.print("Loading images for training...");
                        while (ite.hasNext()) {
                                String personKeyForTraining = (String) ite.next();
                                String personNameForTraining = (String) dataMap.getProperty(personKeyForTraining);
                                IplImage[] imagesForTraining = readImages(personNameForTraining);
                                IplImage grayImage = IplImage.create(imagesForTraining[0].width(), imagesForTraining[0].height(), IPL_DEPTH_8U, 1);

                                for (int i = 0; i < imagesForTraining.length; i++) {
                                        trainLabels.put(count, 0, Integer.parseInt(personKeyForTraining));
                                        cvCvtColor(imagesForTraining[i], grayImage, CV_BGR2GRAY);
                                        trainImages.put(count,grayImage);
                                        count++;
                                }
                                //storeNormalizedImages(personNameForTraining, imagesForTraining);
                        }

                        System.err.println("done.");

                        System.err.print("Training Binary model ....");
                        fr_binary.train(trainImages, trainLabels);
                        System.err.println("done.");
                        storeTrainingData();
                }

        }

        private void loadTrainingData() {

                try {
                        File personNameMapFile = new File(personNameMappingFileName);
                        if (personNameMapFile.exists()) {
                                FileInputStream fis = new FileInputStream(personNameMappingFileName);
                                dataMap.load(fis);
                                fis.close();
                        }

                        File binaryDataFile = new File(frBinary_DataFile);
                        System.err.print("Loading Binary model ....");
                        fr_binary.load(frBinary_DataFile);
                        System.err.println("done");


                } catch (Exception e) {
                        e.printStackTrace();
                }
        }

        private void storeTrainingData() throws Exception {
                System.err.print("Storing training models ....");

                File binaryDataFile = new File(frBinary_DataFile);
                if (binaryDataFile.exists()) {
                        binaryDataFile.delete();
                }
                fr_binary.save(frBinary_DataFile);

                File personNameMapFile = new File(personNameMappingFileName);
                if (personNameMapFile.exists()) {
                        personNameMapFile.delete();
                }
                FileOutputStream fos = new FileOutputStream(personNameMapFile, false);
                dataMap.store(fos, "");
                fos.close();

                System.err.println("done.");
        }


        public void storeTrainingImages(String personName, IplImage[] images) {
                for (int i = 0; i < images.length; i++) {
                        String imageFileName = imageDataFolder + "training\\" + personName + "_" + i + ".bmp";
                        File imgFile = new File(imageFileName);
                        if (imgFile.exists()) {
                                imgFile.delete();
                        }
                        cvSaveImage(imageFileName, images[i]);
                }
        }

        private IplImage[] readImages(String personName) {
                File imgFolder = new File(imageDataFolder);
                IplImage[] images = null;
                if (imgFolder.isDirectory() && imgFolder.exists()) {
                        images = new IplImage[NUM_IMAGES_PER_PERSON];
                        for (int i = 0; i < NUM_IMAGES_PER_PERSON; i++) {
                                String imageFileName = imageDataFolder + "training\\" + personName + "_" + i + ".bmp";
                                IplImage img = cvLoadImage(imageFileName);
                                images[i] = img;
                        }

                }
                return images;
        }


}

Step-5)
Write a client code. The client code must do the following

1)      Get an instance of the face recognizer
                        LBPFaceRecognizer fr = LBPFaceRecognizer.getInstance();
2)      Capture image and grab the face coordinates using the code described here
                        FrameGrabber grabber = new OpenCVFrameGrabber(0);
            grabber.start();
            IplImage img = grabber.grab();
            IplImage snapshot = cvCreateImage(cvGetSize(img), img.depth(), img.nChannels());
            cvFlip(img, snapshot, 1);
            CvSeq faces = fr.detectFace(img);
            CvRect r = new CvRect(cvGetSeqElem(faces, 0));

3)      Preprocess the image using fr.preprocess
                     trainImages[imageCounter] = fr.preprocessImage(img, r);
4)      Invoke the learn method
                     fr.learnNewFace("User1", trainImages);
5)      Once the learning is complete you can use fr.identiofyFace to identify a face
                    fr.identifyFace(fr.preprocessImage(img, r));

NOTE: The convenience method preprocessImage is used to clip face, resize and normalize image to improve accuracy.
Happy coding :)

47 comments:

  1. Cool article. Now I know where to point people asking me for the FaceRecognizer with JavaCV! :) Thanks for writing it.

    ReplyDelete
    Replies
    1. Sure..Thanks to Samuel for creating such a great framework

      Delete
  2. How do you call the FaceRecognizerPtr using the updated JavaCV 0.3? Thanks :)

    ReplyDelete
    Replies
    1. Sorry, I did not get a chance to play with JavaCV.03. But I am assuming it wont be much different.

      Delete
    2. Hello there, did you find a solution for this problem I can not find the FaceRecognizerPtr class!

      Delete
    3. I'm having the same problem with FaceRecognizerPtr class that cant be found

      Delete
    4. import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;

      Delete
    5. can someone give client class to me

      Delete
    6. anyone got the solution on FaceRecognizerPtr class? IDE is unable to fine FaceRecognizerPtr class in java.
      pls upload the solution.

      Delete
    7. I cant found this class too...

      Delete
  3. i have tested your implementation and trained the system successfully but the predict function always returns the last label trained in the databased .... Did you encounter this problem?

    ReplyDelete
    Replies
    1. You have to adjust the threshold variable in the first few lines of the code to fine tune the performance of the face recognizer. The logic computes confidence based on the threshold value.

      double binaryTreshold = 100;

      Its a painful trial and error procedure.. but after some fine tuning you will get the correct response.

      Delete
  4. i have encounter the same problem as Iman Firouzian had.

    ReplyDelete
  5. this trainImages[]and imagecounter on the client code,where they came from ?
    sorry if i missed sometihing...

    ReplyDelete
    Replies
    1. trainImages[] should be supplied by the applciation using the face recognizer. I had a small application that capture images using the computer camera and collect the images into an array and send it to the Face recognizer

      Delete
  6. What kind of file is frBinary.dat ??
    What content does it hold???

    ReplyDelete
    Replies
    1. frBinary.dat is a file that will be created by the program. The only input to the program is the face images and haarcascade.
      frBinary.dat will contain the binary model saved into a file. The program will reload the frBinary.dat next time you run, so that so dont have to retrain the model again. I

      Delete
  7. Exelente trabajo, estoy trabajando en netbeans 7.1.2 pero a la hora de compilar el proyecto me manda
    "Cargando archivo binario ....OpenCV Error: Unspecified error (File can't be opened for writing!) in unknown function, file ..\..\..\src\opencv\modules\contrib\src\facerec.cpp, line 304"
    Alguien sabe cómo solucionarlo?line 304

    ReplyDelete
  8. Excellent work, I am working in netbeans 7.1.2 but when I compile the project send
    "Loading binary file .... OpenCV Error: Unspecified Error (File can not be opened for writing!) In unknown function, file .. \ .. \ .. \ src \ opencv \ modules \ contrib \ src \ facerec. cpp, line 304 "
    Anyone know how to fix it?

    ReplyDelete
  9. Hi, I am kind of stuck at step 2, as I have some trouble with javacv as most of the classes from com.googlecode.javacv.cpp package are annotated with com.googlecode.javacpp.annotation.Properties and that annotation uses a property inherit which is not defined in Properties annotation. Using javacpp-0.5.jar. I'm thinking to remove that inherit annotation property in order to compile, but I am not sure if that is alright.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Also, where can you find specific platform jar files? (javacv-linux-x86.jar,javacv-linux-x86_64.jar,javacv-macosx-x86_64.jar,javacv-windows-x86.jar,javacv-windows-x86_64.jar). As these are not shipped with javacv. Thanks!

    ReplyDelete
  12. hi, I'm not able to locate FaceRecognizerPtr class in opencv ..can any one help out with the issue.

    ReplyDelete
    Replies
    1. Add this to the imports
      import com.googlecode.javacv.cpp.opencv_contrib.FaceRecognizerPtr;

      Delete
    2. The import is not working even after importing all the mentioned .jar files. So can you suggest some better solution to this?

      Delete
    3. Cause JavaCV 0.3 changed a lot.
      You can download and use v0.2
      Another way, just replace FaceRecognizerPtr by FaceRecognizer ( remove .get() or sample code above)

      Delete
  13. Hi!!
    i m facing an error in this method
    private void createModels() {
    ptr_binary = createLBPHFaceRecognizer(1, 8, 8, 8, binaryTreshold);
    ---->>fr_binary = ptr_binary.get();( Error over here)
    }
    help me with the solution for my error.
    Thank You.

    ReplyDelete
    Replies
    1. remove .get() if you changed from FaceRecognizerPtr to FaceRecognizer

      Delete
  14. Entire steo2 is failing in my case, any method call like cvFlip or cvGetSeq... anything releated to CV is failed or crashing

    ReplyDelete
  15. I didn't get the last part about the client code. What I did is:
    1) I created a new java class and the following-

    package com.test.face;
    import static com.googlecode.javacv.cpp.opencv_core.cvCreateImage;
    import static com.googlecode.javacv.cpp.opencv_core.cvFlip;
    import static com.googlecode.javacv.cpp.opencv_core.cvGetSeqElem;
    import static com.googlecode.javacv.cpp.opencv_core.cvGetSize;

    import com.googlecode.javacv.FrameGrabber;
    import com.googlecode.javacv.OpenCVFrameGrabber;
    import com.googlecode.javacv.cpp.opencv_core.*;
    public class Client {
    public static void main(String s){
    LBPFaceRecognizer fr = LBPFaceRecognizer.getInstance();
    FrameGrabber grabber = new OpenCVFrameGrabber(0);
    try{
    grabber.start();
    IplImage img = grabber.grab();
    IplImage snapshot = cvCreateImage(cvGetSize(img), img.depth(), img.nChannels());
    cvFlip(img, snapshot, 1);
    CvSeq faces = fr.detectFace(img);
    CvRect r = new CvRect(cvGetSeqElem(faces, 0));

    IplImage[] trainImages = null;
    int imageCounter = 0;
    trainImages[imageCounter] = fr.preprocessImage(img, r);

    fr.learnNewFace("User1", trainImages);

    fr.identifyFace(fr.preprocessImage(img, r));



    }
    catch(Exception e){
    System.out.println("Getting error:::"+e.getMessage());
    System.out.println("+++++++++++++++++++++++++++++");
    }
    }
    }
    But this is not executing. Should I specify a list of images to trainImages object?

    ReplyDelete
  16. Please tell the part about the client code..

    ReplyDelete
  17. Hi,thanks for this code.
    But you plz write the client code,because i'm having difficulties writng it.
    And thank you.

    ReplyDelete

  18. Exception in thread "main" java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at com.googlecode.javacpp.Loader.load(Loader.java:553)
    at com.googlecode.javacpp.Loader.load(Loader.java:532)
    at com.googlecode.javacv.cpp.opencv_objdetect$CvHaarClassifierCascade.(opencv_objdetect.java:170)
    at test2.LBPFaceRecognizer.(LBPFaceRecognizer.java:58)
    at test2.LBPFaceRecognizer.(LBPFaceRecognizer.java:60)
    at test2.test.main(test.java:30)
    Caused by: java.lang.IllegalStateException: Can't overwrite cause
    at java.lang.Throwable.initCause(Throwable.java:456)
    at com.googlecode.javacpp.Loader.load(Loader.java:581)
    at com.googlecode.javacpp.Loader.load(Loader.java:532)
    at com.googlecode.javacv.cpp.opencv_objdetect.(opencv_objdetect.java:91)
    ... 8 more
    Java Result: 1

    ReplyDelete
  19. FaceRecognizerPtr ptr_binary = null; (Can not find symbol )

    getting trouble with above variable.

    Any one help for above problem.

    ReplyDelete
  20. I got error like this : Could not find method com.googlecode.javacv.cpp.opencv_contrib.createLBPHFaceRecognizer, referenced from method org.opencv.samples.facedetect.FdActivity.

    Please help me...

    ReplyDelete
  21. I don't understand. Why people post a half done solution here. Is it a challenge to debug it, finish it and wrote a client for it?

    ReplyDelete
  22. Need full source code

    ReplyDelete
  23. I am getting error at FaceRecognizerPtr.... What should I replace it with..? I didn't get the solution of remove mentioned above.. I am using the import org.bytedeco.javacpp...... because com.googlecode is not available I guess ..pls reply

    ReplyDelete
  24. And Thanks for the code forgot to mention it above..! :)

    ReplyDelete
  25. Can I use android camera to capture a image instead off 'FrameGrabber?' If so then how can I convert the captured image to IplImage?

    ReplyDelete
  26. Hi, thank you for sharing this code, but unfortunately I'm having this issue:

    Loading Binary model ....OpenCV Error: Unspecified error (File can't be opened for writing!) in unknown function, file ..\..\..\src\opencv\modules\contrib\src\facerec.cpp, line 325
    java.lang.RuntimeException: ..\..\..\src\opencv\modules\contrib\src\facerec.cpp:325: error: (-2) File can't be opened for writing!

    Can someone help me?

    ReplyDelete
  27. need full client code

    ReplyDelete
  28. Unspecified error (File can't be opened for writing!) in cv::FaceRecognizer::load, file ..\..\..\..\opencv\modules\contrib\src\facerec.cpp, line 325
    java.lang.RuntimeException: ..\..\..\..\opencv\modules\contrib\src\facerec.cpp:325: error: (-2) File can't be opened for writing! in function cv::FaceRecognizer::load

    ReplyDelete
  29. i have implemented client code and also run if fine upto some extend.
    but my pc gets slow down whenever i run it.
    also face detection and recognition have less accuracy.
    could you help me? plzzz..

    ReplyDelete
  30. "Loading binary file .... OpenCV Error: Unspecified Error (File can not be opened for writing!) In unknown function, file .. \ .. \ .. \ src \ opencv \ modules \ contrib \ src \ facerec. cpp, line 304 "
    Anyone know how to fix it?

    ReplyDelete
  31. anyone have a full working code yet? i have so many problems with this one

    ReplyDelete

Blog Archive