The basic orientation input system for the CheapVR system is pretty much ready. Im using a combination of UDP and web-socket. Originally, I had the local server running a simple UDP server that received the orientation data from the phone, and then re-transmit it via HTTP POST queries from the browser. However this method had allot of latency.
Currently the Browser connects via a web-socket to the server and continually polls it for orientation data.
I have started working on a small VR web-app. I think 3D headsets will only get better and better as tech progresses and people start to get into it. I have owned the 1st gen Sony HMD unit HMZ-T1
This thing is great, but it lacks any kind of tracking like the Oculus Rift and the new Google Cardboard(which really got me thinking about this whole thing again). I have in the past used old Android phones to get tracking data. There is a great little app called Sensor UDP it lets you send UDP messages to a pc/tablet/phone etc.. with the sensor data from any android phone! This has been done before, Im working on a different angle!
A phone will be stuck to the HMD unit and track your head movements. It will then send this data via WIFI to the PC connected to the HMD via HDMI. I can then make really cool 3D web apps in Chrome or any HTML5 enabled browser.
Thanks to the Intel Galileo board, I can remove the wired part of the system. No more serial port, no more special buffering functions, a huge load off the implementation.
The Galileo board now sits on the same network as the client pc. It runs an simple web server at port 80. It reads requests from an incoming client connection. The client PC, connected to the Leap Motion, continuously polls the Galileo board by sending a GET request with the calculated PWM values. The Galileo board received the request, parses out the PWM values and send that data to the assigned PWM pins that control the feedback to each finger.
I also put in a lot of special feedback functions. Instead of simple strength values, i can now code it to accept specif patters of vibration.
There is no More Serial driver, so the NodeJS app i wrote that accepts GET requests and translates them into serial data is no more.
I have put together a small stress test for the Galileo. Its not much of a stress test, just a test of how quickly I can update data. The sliders control a PWM value from 0 to 255 for each finger, the screen on the left displays the Serial terminal of the Galileo.
With the sample size set to 50, motion is super fluid…but ridiculously laggy :)
Here is a really hacky function i put together to test out the lag i was experiencing with the Z axis, but i thought i would apply it to all the currently tracked fingers.
There are 3 public variables:
previosFrame
Has all the data from the leap motion controller
rawFrameData
Is a FILO Stack of finger data from each previous frame
stableFrameData
Is an array of [x,y,z] data for each finger. This is used as the stabilized tracking data instead of fingers[xx].stabilizedTipPosition[x,y,z]
By setting the
numberOfSamples
Variable you can set the super sampling size. 5-8 is pretty smooth. 10-20 is laggy but very smooth. 50 is ridiculously fun! as demonstrated in the video.
function stabilize()
{
if(previousFrame.hands.length)
{
var numberOfSamples = 8;
if(rawFrameData.length<numberOfSamples)
{
rawFrameData.push( previousFrame.hands[0].fingers );
}
else
{
rawFrameData.splice(0,1);
rawFrameData.push( previousFrame.hands[0].fingers );
}
stableFrameData = [[0,0,0],[0,0,0],[0,0,0],[0,0,0],[0,0,0]];
for(rawFrameCnt=0; rawFrameCnt<rawFrameData.length; rawFrameCnt++)
{
stableFrameData[0][0]+=rawFrameData[rawFrameCnt][0].tipPosition[0];
stableFrameData[0][1]+=rawFrameData[rawFrameCnt][0].tipPosition[1];
stableFrameData[0][2]+=rawFrameData[rawFrameCnt][0].tipPosition[2];
stableFrameData[1][0]+=rawFrameData[rawFrameCnt][1].tipPosition[0];
stableFrameData[1][1]+=rawFrameData[rawFrameCnt][1].tipPosition[1];
stableFrameData[1][2]+=rawFrameData[rawFrameCnt][1].tipPosition[2];
stableFrameData[2][0]+=rawFrameData[rawFrameCnt][2].tipPosition[0];
stableFrameData[2][1]+=rawFrameData[rawFrameCnt][2].tipPosition[1];
stableFrameData[2][2]+=rawFrameData[rawFrameCnt][2].tipPosition[2];
stableFrameData[3][0]+=rawFrameData[rawFrameCnt][3].tipPosition[0];
stableFrameData[3][1]+=rawFrameData[rawFrameCnt][3].tipPosition[1];
stableFrameData[3][2]+=rawFrameData[rawFrameCnt][3].tipPosition[2];
stableFrameData[4][0]+=rawFrameData[rawFrameCnt][4].tipPosition[0];
stableFrameData[4][1]+=rawFrameData[rawFrameCnt][4].tipPosition[1];
stableFrameData[4][2]+=rawFrameData[rawFrameCnt][4].tipPosition[2];
}
for(rawFrameCnt=0; rawFrameCnt<stableFrameData.length; rawFrameCnt++)
{
stableFrameData[rawFrameCnt][0]=stableFrameData[rawFrameCnt][0]/rawFrameData.length;
stableFrameData[rawFrameCnt][1]=stableFrameData[rawFrameCnt][1]/rawFrameData.length;
stableFrameData[rawFrameCnt][2]=stableFrameData[rawFrameCnt][2]/rawFrameData.length;
}
}
}
I spent a few hours trying to get smooth operation of some kind using both hands on the leap motion. However i was not able to get reliable results. My left hand would get tired of holding still and move off spot. I decided to at least focus on the detailed control and manipulation with 1 hand and have the left hand use the keyboard. For example. . In the video above, I’m using the keys:
A = Width S = Height D = Depth F = Location
If one is being held down, i can use the right hand to select the object by hovering over it and adjust its values by moving the thumb and index finger together or further apart.
I have started working with dual hand interactions. Left hand is for control selection, right hand is for manipulation. The biggest difficulty is the precise tracking of both hands.. hopefully that will change as the leap team puts work into the SDK, as they have been recently. Light interference is also a huge issue
X coordinates of Index-Thumb control the width, Y coordinates of Index-Thumb control the height and the Ring Finger+Middle finger X control the depth. I need to test the large version of this on a machine that’s not 5 years old.. I’m running into serious performance issues when rendering more than 10 SVGs but that is totally expected.
Currently adjusting how the cubes are layout out and what controls them. Here is a short video showing how the Z, depth, coordinate of my hand control the height in real time.
Learning from the last set of SVG polys, I created a new JS Class that renders 3D Rectangles/cubes. You need to create a SVG type HTML object and send this object in as a constructor for each new cube you insert. You create it via:
virPoly(htmlIDString,HTML_SVG_Object,Blank_CVG_Object,DimmensionArray[width,height,depth],XYPosition[x,y],DefaultLineColour[r,g,b],defaultFillColour[r,g,b]],toggledLineColour[r,g,b],toggledFillColour[r,g,b],LineWidthInPixels)My intention is to create a depth map of rectangles and control their height based on the x,y coordinates of the fingers and the depth control via the z coordinates