How can I move my mouse according to a centroid value?

2 views (last 30 days)
I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.
The gestures I have are number-based:
1 - the mouse will move according to the centroid of my hand.
2 - left click
3 - right click
4 - double click
5- scroll
I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?
I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!
close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
%% Create video input object.
cam = videoinput('macvideo'); %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000; %change to Inf when needed
cam.FrameGrabInterval = 1;
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
%% Count Number of Fingers
runloop = true;
while runloop
img1 = getsnapshot(cam);
img1 = rgb2gray(img1);
img2 = imcomplement(imbinarize(img1));
img3 = imfill(img2, 'holes'); %filling all holes
img4 = bwareaopen(img3, 10000); %removing objects lesser than 10K size
SE = strel('disk', 10); %defining structural elements
SE1 = strel('disk', 50);
SE2 = strel('disk', 60);
img4e = imerode(img4, SE1); %image erosion
img4d = imdilate(img4e, SE2); %image dilation
imgfo = img4-img4d; %getting fingers
imgfo = logical(imgfo);
imgfo = bwareaopen(imgfo,5000);
CC = bwconncomp(imgfo); %connected component analysis
nof = CC.NumObjects; %getting number of fingers
imgfog = uint8(255.*imgfo);
nofs = num2str(nof);
imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
step(videoplayer, imgfogrgb);
runloop = isOpen(videoplayer); %checking if videoplayer is on or off
set(gcf, 'Position', [100, 100, 500, 400])
% preview(cam);
% movegui('northeast');
%% Begin Move Movements
while nof==1 %move
st = regionprops(imgfo, 'BoundingBox', 'Centroid');
hold on
for k = 1 : length(st)
bbox = st(k).BoundingBox;
cbox = st(k).Centroid;
rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
plot(cbox(1),cbox(2), '-m+')
a = text(cbox(1)+15, cbox(2), strcat('X: ', num2str(round(cbox(1))), ' Y: ', num2str(round(cbox(2)))));
set(a, 'FontSize', 12, 'Color', 'white');
%hold on;
X = cbox(1);
Y = cbox(2);
set(0,'PointerLocation',[X Y]);
centroids = cat(1,st.Centroid);

Answers (0)




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!