ECE482 Presentation2

8/3/2019 ECE482 Presentation2

1/25

Video Based 3D HumanActivities Analysis

By:Jacob RandallAutumn SmithMuhammad Farooq


2/25

Outline

Background and motivation

Goals and flow map

Methods and Algorithms

Results and Discussions

Service Learning Project

Questions and comments


3/25

Object Implantation, Removal, and 3D Modeling in films

Human Machine Interface (HMI) & Virtual Reality

o Methods

Green screen

Body suits

o Cost Efficient

o Sometimes unrealistic.

Background and Motivation


4/25

Background & Motivation

Pressure sensor Motion capture suit

Virtual RealityMotion reconstruction

Gesture Reconstruction

Green screens


5/25

Track a person on film

Convert information to 3D walker

Trajectory, follow movements

Match 3D model as closely as possible

Goals and Objectives


6/25

Method & Algorithm

Motion reconstruction (feature vector) Gesture reconstruction (correlation based)


7/25

Feature Vector Algorithm


8/25

Background Subtraction

o This difference between two images gives the foreground objects.

If Ii is the current frame and Ib is the background image the

foreground image Ifi is Ifi = Ii Ib

o Converted to Binary Image

o Otsus Method for finding optimum threshold for binary

conversion

Results and Discussion


9/25

Motion Segmentation

Noise Removalo Median Filtering

Smoothing operation

o Erosion

Completely remove objects smaller that the structuring element

Structuring Element = 15


10/25

Skeleton Image Extraction

o Needed to extract features

o Uses MatLabs built-in function

Stick Model

o Angle information gathered from silhouette of the image is used to form

the stick model

Skeleton Image

Stick Model Stick model over video


11/25

Feature Extraction

o

Using skeleton image find the mean location of all pixels with value of 1 above the

first line.

o Draw a vertical line passing through the head dividing image into 4 parts.

o In each part find the point which is at the maximum distance from the reference.

o Draw lines from these points to the points of intersection of the two horizontal lines

o Find the angles of these lines giving us 4 angles (legs and arms). These angles create the feature

vectors.

End points of silhouette image


12/25

Feature vector

Get the feature vector

[ang1 ang2 ang3 ang3]

Input these to the 3D

model

Get the feature vector

[ang1 ang2 ang3 ang3]

Input these to the 3Dmodelang1 ang2

ang3

ang4


13/25

Some Challenges

Direction of motion Check for the change in the part

of the image

Left or right leg/arm? Look for patterns in motion

Model is rotating its legs and

arms at the same location? Translate the model in each

frame

Change the axis for each frame

In whichdirection to

move?

Left or rightarm/leg?


14/25

Stick model superimposed on the original video

Skeleton Model

Original Video

2D Walker to 3D Model Videos

Motion reconstruction using 3D model

Alternative view of 3D model
http://www.youtube.com/watch?v=ZIAr16fkQ88&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=9Ry49jt6ke0&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=Hg7eWaTkKu8&context=C2361fADOEgsToPDskJIsaiBlezw9RdzW-jSTawGhttp://www.youtube.com/watch?v=Hg7eWaTkKu8&context=C2361fADOEgsToPDskJIsaiBlezw9RdzW-jSTawGhttp://www.youtube.com/watch?v=YRuMwyz_SjU&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=qahJ9iWOAe8&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=qahJ9iWOAe8&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=YRuMwyz_SjU&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=YRuMwyz_SjU&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=Hg7eWaTkKu8&context=C2361fADOEgsToPDskJIsaiBlezw9RdzW-jSTawGhttp://www.youtube.com/watch?v=Hg7eWaTkKu8&context=C2361fADOEgsToPDskJIsaiBlezw9RdzW-jSTawGhttp://www.youtube.com/watch?v=9Ry49jt6ke0&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkChttp://www.youtube.com/watch?v=ZIAr16fkQ88&context=C2353bADOEgsToPDskKFpQYvWflA24P8u2tStCkC


15/25

Gesture Recognition Algorithm

Correlation

3DModelLibrary


16/25

Background subtraction

Image is then converted to a binary image

Otsus Algorithm was used to find optimum threshold

Normalization

o Manually done using correlation coefficients computation.

o Normalized images from the input human video are compared to

each image in library consisting of 3D model image silhouettes

o Multiply each matrix together to generate a correlation coefficiento Best Image is chosen

Gesture Recognition


17/25

3D Motion Reconstruction

o Each 3D image silhouette corresponded to the 3D image it was derived

from.

o Once all the 3D image silhouette images were determined, these images

were used to construct the 3D video based on the corresponding 3D

images.


18/25

Gesture Reconstruction Videos

Binary Input

Binary 3D

3D Model
http://www.youtube.com/watch?v=3AnpvtKjpMI&context=C20999ADOEgsToPDskIqJXv69xyb_hJQuNZRsXhMhttp://www.youtube.com/watch?v=3AnpvtKjpMI&context=C20999ADOEgsToPDskIqJXv69xyb_hJQuNZRsXhMhttp://www.youtube.com/watch?v=KS72fJbpIxY&context=C2f892ADOEgsToPDskL2rbjJeodxoK6U_1LWCfJbhttp://www.youtube.com/watch?v=UBtY_vP8s_Q&context=C2cf35ADOEgsToPDskJXpomrahDWfX9xqtkQxXSkhttp://www.youtube.com/watch?v=UBtY_vP8s_Q&context=C2cf35ADOEgsToPDskJXpomrahDWfX9xqtkQxXSkhttp://www.youtube.com/watch?v=KS72fJbpIxY&context=C2f892ADOEgsToPDskL2rbjJeodxoK6U_1LWCfJbhttp://www.youtube.com/watch?v=3AnpvtKjpMI&context=C20999ADOEgsToPDskIqJXv69xyb_hJQuNZRsXhMhttp://www.youtube.com/watch?v=3AnpvtKjpMI&context=C20999ADOEgsToPDskIqJXv69xyb_hJQuNZRsXhM


19/25

Camera Installation

o Decided camera location

o Camera Software

Issues

o Limited on cameras (6 total)

o Limited outlets

Service Learning Project

Layout of Kiddie World


20/25

Camera Software

Camera was turned on and the discovered by the program.

Main Screen on Camera Software List of Cameras in Area


21/25

IP address was then set to the local address

All cameras were set to the same IP

Connected using HTTP

General settings for cameraSet IP address


22/25

We can reconstruct human movements using a 3D model using the

algorithm

Algorithm can be improved in future work

o The performance will increase if some Machine learning algorithm is

applied to differentiate between the left and right arms and legs

o Clustering techniques (unsupervised algorithm) can be used to find

cluster of data and to calculate angles between these clusters.

Conclusion


23/25

All group members contributed to this project.

o Muhammads main contribution was towards the implementation of

feature vector based motion reconstruction.

o Jacob was mainly responsible for the implementation of the correlation

based gesture reconstruction.

o Autumn was responsible for videos and 3D model based reconstruction

implementation for both methods.

Staff Contribution


24/25

http://www.nbb.cornell.edu/neurobio/land/projects/hierarchy/

http://qh.eng.ua.edu/classes/fall2010/ece582/index_files/projects/group5

/index.html

qh.eng.ua.edu/research/iSMART/codes/simulation/3dReconstruction/wal

ker_test.rar

Reference


25/25

ECE482 Presentation2

Documents

Transcript of ECE482 Presentation2