Wednesday 30 July 2014

How to detect fingers?

Its always such a fun, to think at the beginning of some code-work, things like, "Yeah I'm going to write a blog on this, when I'm done....", but at the end, we're so overwhelmed with what we have achieved, (and thus what we potentially could), the task of writing about it, seems pretty absurd...
For people who don't write much, (like me), this seems like one of the most time-consuming processes on Earth!
But then, I've getting quite a few requests for code, and I didn't feel like to end up with mailing code to every one of them personally, or explain in details.
And since it happens to be one of the earliest days I've woken up one (8 AM fellas!), I've made up my mind on writing!

Objective: Detecting (separating) the fingers which are up, in a open palm. (And getting it done by your Computer!)

Prerequisites:
  • MATLAB (I used Matlab 13a)
  • Familiarity with Matlab. (I'm not explaining Matlab in this blog.)
  • A Webcam
Here's how it looks like..


















Disclaimer: Yes, this one's a Matlab one. The project is rather demonstrative, and by itself is of little use. You're free to use the article or the code. But mind the recognition of original work ;) But I hope you'll build up something stellar, and when you do so,  don't forget to share with everyone else :)

Confession: 
The code is rather slow. I was new to Matlab then, and I haven't optimized it since. Also, it uses plain skin detection in YUV color range. So, things get messed up quick when there are skin like elements in background. Also, lightning should be uniform. 

Also, the work here is based upon work of others. Some reference codes from here and there, and a few research papers. Yes, I got a lot of help from freely available knowledge all over the Internet!

Basic Idea:
  1. First skin is detected from image using filtering in YUV color space and largest blob is retained. This is new image [1].
  2.  Then image is cleared using Morphological Operations. 
  3.  Finally:
    • Find the nearest point from centroid of extracted blob (which we might safely assume to be palm)
    • The distance from the point can safely be assumed to approximate palm radius.
    • With a disc element of a nearly this distance, erode the image.
    • Subtract the resultant image[2] from image[1]
    • Do a few more morphological operations, to make fingers significant in appearance.
That's all!
This gives countable number of connected components, giving the number of fingers. The Video shows the Regions of Interest, i.e. the palm with fingers (which are up) only. Counting fingers, need some extra code (Which is being left as an exercise :p).

Screenshots:


   







Code:
1. bwFingers.m


%%
%Finger Tracking
%Author: Arindam Sarkar
%Copyright?
%You can use this code for whatever you want! I'm not responsible for
%whatever it brings upon you, your system or your pet!
%Try to attribute the effort when you use part of this code in your own
%projects ;)
%Happy Coding!
%(c)2014 & beyond
%%
%boundary of hand
%background substraction with skin marking
close all;
clc;
clear all;
vid=imaq.VideoDevice('winvideo',1,'RGB24_640x480');
%set(vid,'ReturnedColorSpace','grayscale');
%preview(vid);
while 1
dntshw=false;
pause(.001);
%acquireimage from webcam
%capture frame
frame = step(vid);
%Read the image, and capture the dimensions
img=uint8(255*frame);
out=skinDetect2Func(img);
stats=regionprops(out,'Centroid');
if length(stats)
    cx=stats.Centroid(1);
    cy=stats.Centroid(2);
    %find the nearest countour point
    boundary=bwboundaries(out);
    minDist=2*640*640;
    mx=cx;
    my=cy;
    bImg=uint8(zeros(480,640));
    for i=1:length(boundary)
        cell=boundary{i,1};
        for j=1:length(cell)
            y=cell(j,1);
            x=cell(j,2);
            sqrDist=(cx-x)*(cx-x)+(cy-y)*(cy-y);
            bImg(x,y)=255;
            if(sqrDist<minDist)
                minDist=sqrDist;
                mx=x;
                my=y;
            end
        end   
    end
   
    sed=strel('disk',round(sqrt(minDist)/2));
    final=imerode(out,sed);
    final=imdilate(final,sed);
    final=out-final;
   
    final=bwareaopen(final,200);
    final=imerode(final,strel('disk',10));
    final=bwareaopen(final,400);
   
    [L,num]=bwlabel(final,8);
    final=imclearborder(final,8);

    imshow(final);
   
else
    imshow(out)
end
% imshow(bImg);
end


2. skinDetect2Func.m


%skin detect return pixels
function out=skinDetect2func(img)

imshow(img);
sz=size(img);
r=1;g=2;b=3;y=1;u=2;v=3;
yuv=img;
region=yuv;

for i=1:sz(1)
    for j=1:sz(2)
        yuv(i,j,y)=(img(i,j,r)+2*img(i,j,g)+img(i,j,b))/4;
        yuv(i,j,u)=img(i,j,r)-img(i,j,g);
        yuv(i,j,v)=img(i,j,b)-img(i,j,g);
    end
end

for i=1:sz(1)
    for j=1:sz(2)
        if yuv(i,j,u)>20 && yuv(i,j,u)<74
            region(i,j,r)=255;
            region(i,j,g)=255;
            region(i,j,b)=255;
     
        else
            region(i,j,r)=0;
            region(i,j,g)=0;
            region(i,j,b)=0;
        end
    end
end

out=region;
%filtering
out=im2bw(out);
out=bwareaopen(out,100);
out=imdilate(out,strel('diamond',4));

%retain largest only
res=out;
cc=bwconncomp(res);
arr=(cellfun('length',cc.PixelIdxList));

newLabel=res;
if ~isempty(round(arr))
    msz=0;
    for i=1:length(arr)
        if msz<arr(i:i)
            msz=arr(i:i);
            index=i;
        end
    end

    labels=labelmatrix(cc);
    newLabel=(labels==index);
    out=newLabel;
end
out=imfill(out,'holes');
end



Full code: detectFinger.zip

Happy Coding! Share knowledge. Make the planet wiser, everyday!

Feedback expected & Welcome :)

Edit 1:
When you post a comment regarding problem while running the code, include sufficient details.
Lets say you're getting an error "some random error!", then instead of just writing
"some random error!", writing -
I'm getting "some random error! *Complete Error String* " *Matlab version, OS * *Personal Note :D* 
is much more useful ;)

Edit 2:
Try the code against a constant plain background, possibly white, and in uniform white light illumination, when in doubt if the code works. This code is pretty basic, and has limitations.

Edit 3:
Make two separate files, for each piece of code, with the given names ;)

Below are some links you could find useful.


 


Edit: 12/7/2016

The code above might seem a bit slow. One of the readers had complained. It can be speeded up by as much as 10x by a simple improvement.

In code below, I've used vectorization to speed up the earlier code. This might speed up the code by upto 10x.

skinDetect2Func.m
 %skin detect return pixels
function out=skinDetect2func(img)
imshow(img);
sz=size(img);
r=1;g=2;b=3;y=1;u=2;v=3;
yuv=img;
region=yuv;

yuv(:,:,y) = (img(:,:,r)+2.*img(:,:,g)+img(:,:,b))/4;
yuv(:,:,u) = img(:,:,r)-img(:,:,g);
yuv(:,:,v)=img(:,:,b)-img(:,:,g);

region = (yuv(:,:,u)>20 & yuv(:,:,v)<74) .* 255;

toc

out=region;
%filtering
out=im2bw(out);
out=bwareaopen(out,100);
out=imdilate(out,strel('diamond',4));
%retain largest only
res=out;
cc=bwconncomp(res);
arr=(cellfun('length',cc.PixelIdxList));
newLabel=res;
if ~isempty(round(arr))
    msz=0;
    for i=1:length(arr)
        if msz<arr(i:i)
            msz=arr(i:i);
            index=i;
        end
    end
    labels=labelmatrix(cc);
    newLabel=(labels==index);
    out=newLabel;
end
out=imfill(out,'holes');
end


Now this doesn't improve accuracy any way, but speeds up the code! 

2 comments:

  1. hii could please send me the full code

    ReplyDelete
  2. Hi Arindam Sarkar,
    Thank you for your interesting project.
    I'm currently try to understand the theory behind your code.
    Can you tell me any references or documents related to the techinices you use in your code.
    Thank you and have a nice day.

    ReplyDelete