Computer Vision with Javascript

John Keogh | August 31, 2015

I've been working on several robot platforms, and have been using Javascript as a scripting language for run-time and end-user programmability of the robot. One of the interesting things that an end user may need to do is to get the pixels from the current image sent by the robot to the browser, and use them for computer vision applications. This brief article describes how to get the pixels from an image, and how to iterate through them for a simple computer vision application that sees what number of pixels match a color.

The following video shows a demonstration of controlling a robot using Javascript that examines the images that are fed to the front end. The code that provides that functionality is provided and discussed below the video.


The images are initially loaded into a img tag (in the robot that occurs when a javascript request an image from the iOS device, for the example code, it occurs when the attribute of the img tag is changed to point to a different image).


An example project is available which contains working code. The code is also shown below.

//get the rgba matrix from a canvas function getCurrentImagePixels(){ var currentImage=document.getElementById("image"); var canvas=document.getElementById("canvas"); canvas.width = currentImage.width; canvas.height = currentImage.height; var context = canvas.getContext("2d") context.drawImage(currentImage, 0, 0, currentImage.width, currentImage.height); var imgData=context.getImageData(0, 0, canvas.width, canvas.height); return imgData; } //this iterates through the pixels to find the number of red //and yellow pixels function examinePixels(imgData){ var redPixels = 0; var yellowPixels = 0; var totalPixels = 0; for (var i=0;igreen*2)&&(red>blue*2)&&(red>100)){ redPixels++; } if((red>blue*2)&&(green>blue*2)&&(red>100)){ yellowPixels++; } totalPixels++; } if(redPixels/totalPixels>0.3){ return "Red object found"; } else if(yellowPixels/totalPixels>0.3){ return "Yellow object found"; } return "No objects found"; }

Further Reading

If you are interested in using Javascript for computer vision and want to use the the most mature computer vision library available, you'll need to set up OpenCV and expose it via a web service. You may want also want to look at node-opencv.

Eyesbot Company

Computer vision

Artificial intelligence

Effecting the physical world