How can I move my mouse according to a centroid value?

1 view (last 30 days)
I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.
The gestures I have are number-based:
1 - the mouse will move according to the centroid of my hand.
2 - left click
3 - right click
4 - double click
5- scroll
I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?
I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!
close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
%% Create video input object.
cam = videoinput('macvideo'); %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000; %change to Inf when needed
cam.FrameGrabInterval = 1;
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
set(cam,'ReturnedColorspace','rgb')
%% Count Number of Fingers
runloop = true;
while runloop
img1 = getsnapshot(cam);
img1 = rgb2gray(img1);
img2 = imcomplement(imbinarize(img1));
img3 = imfill(img2, 'holes'); %filling all holes
img4 = bwareaopen(img3, 10000); %removing objects lesser than 10K size
SE = strel('disk', 10); %defining structural elements
SE1 = strel('disk', 50);
SE2 = strel('disk', 60);
img4e = imerode(img4, SE1); %image erosion
img4d = imdilate(img4e, SE2); %image dilation
imgfo = img4-img4d; %getting fingers
imgfo(imgfo==1)=0;
imgfo = logical(imgfo);
imgfo = bwareaopen(imgfo,5000);
CC = bwconncomp(imgfo); %connected component analysis
nof = CC.NumObjects; %getting number of fingers
imgfog = uint8(255.*imgfo);
nofs = num2str(nof);
imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
step(videoplayer, imgfogrgb);
runloop = isOpen(videoplayer); %checking if videoplayer is on or off
set(gcf, 'Position', [100, 100, 500, 400])
movegui('northeast')
pause(2);
% preview(cam);
% movegui('northeast');
%% Begin Move Movements
while nof==1 %move
st = regionprops(imgfo, 'BoundingBox', 'Centroid');
imshow(imgfo)
hold on
for k = 1 : length(st)
bbox = st(k).BoundingBox;
cbox = st(k).Centroid;
rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
plot(cbox(1),cbox(2), '-m+')
a = text(cbox(1)+15, cbox(2), strcat('X: ', num2str(round(cbox(1))), ' Y: ', num2str(round(cbox(2)))));
set(a, 'FontSize', 12, 'Color', 'white');
%hold on;
end
X = cbox(1);
Y = cbox(2);
set(0,'PointerLocation',[X Y]);
centroids = cat(1,st.Centroid);
%mouse.mouseMove(1600-(centroids(1,1)*(5/2)),(centroids(1,2)*(5/2)-180));

Answers (0)

Categories

Find more on Image Preview and Device Configuration in Help Center and File Exchange

Products


Release

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!