My purpose is to write a website (html/js/wasm) in which the user provides two jpg which are "compared" using OpenCV in wasm. By "compared", I mean : compute the key points, the descriptors, make a BF match and print the distances.
What I'm able to do is a pure c++ program reading two files on the disk and compare them. It's okay. With my two test image, the best matches have distances around 100 and the 10 best are bellow 200.
It turns out that, when trying in wasm, the keypoints are correctly computed, but the matches are bad. The best ones have distance around 200.
Here is my (minimal?) example.
The html part creates two canvas in which the images are stored.
<!doctype html>
<html>
<body>
<head>
<meta charset="utf-8">
<title>c++ generated</title>
</head>
<canvas id="viewport1"></canvas>
<input type='file' id='inputimage1' accept='image/*'>
<canvas id="viewport2"></canvas>
<input type='file' id='inputimage2' accept='image/*'>
<script>
var Module = {
onRuntimeInitialized: function() {
createQuery();
}
};
</script>
<script src="josef.js"></script>
<script src="main.js"></script>
</body>
</html>
The JS part contains canvas manipulations and calls to c++. Two calls :
- for the first image we call the function
recordImageto store thecv::Matin a global variable - for the second image, we call the function
comparewhich compares the two images.
/* global Module */
function putOnHeap(imgData, wasmModule) {
const uint8ArrData = new Uint8Array(imgData.data);
const numBytes = uint8ArrData.length * uint8ArrData.BYTES_PER_ELEMENT;
const dataPtr = wasmModule._malloc(numBytes);
const dataOnHeap = new Uint8Array(wasmModule.HEAPU8.buffer, dataPtr, numBytes);
dataOnHeap.set(uint8ArrData);
answer = {"byteOffset": dataOnHeap.byteOffset,
"length":uint8ArrData.length,
"dataPtr": dataPtr,
"dataOnHeap": dataOnHeap,
"wasmModule":wasmModule,
};
return answer;
}
/*
Call the c++ function on the given image.
*/
async function toCpp(canvas, wasmModule, functionName)
{
const context = canvas.getContext('2d');
const imgData = context.getImageData(0, 0,
canvas.width, canvas.height);
const heapInfo = putOnHeap(imgData, wasmModule);
const func = wasmModule[functionName];
func(heapInfo.byteOffset, heapInfo.length,
canvas.width, canvas.height);
wasmModule._free(heapInfo.dataPtr);
return answer;
}
/*
Record the image on the wasm side.
*/
async function recordImage(canvas, wasmModule) {
await toCpp(canvas, wasmModule, "recordImage");
}
/*
Compare the image with the recorded one.
*/
async function compare(canvas, wasmModule) {
await toCpp(canvas, wasmModule, "compare");
}
/* When the user provides the first image, wasm stores it
* in a cv::Mat.
*/
async function doFirstCanvas() {
const canvas = document.getElementById('viewport1');
const width = this.width;
const height = this.height;
canvas.width = width;
canvas.height = height;
const context = canvas.getContext('2d');
let imageData = context.getImageData(0, 0, width, height);
context.putImageData(imageData, 0, 0);
context.drawImage(this, 0, 0);
await recordImage(canvas, Module);
}
/*
* When the user provides the second image, wasm compares
* that image with the first one.
*
*/
async function doSecondCanvas() {
const canvas = document.getElementById('viewport2');
const width = this.width;
const height = this.height;
canvas.width = width;
canvas.height = height;
const context = canvas.getContext('2d');
let imageData = context.getImageData(0, 0, width, height);
context.putImageData(imageData, 0, 0);
context.drawImage(this, 0, 0);
compare(canvas, Module);
}
function failed() { }
function createQuery() {
document.getElementById('inputimage1').onchange = function () {
console.log('First image received');
const img = new Image();
img.onload = doFirstCanvas;
img.onerror = failed;
img.src = URL.createObjectURL(this.files[0]);
};
document.getElementById('inputimage2').onchange = function () {
console.log('Second image received');
const img = new Image();
img.onload = doSecondCanvas;
img.onerror = failed;
img.src = URL.createObjectURL(this.files[0]);
};
}
The c++ part contains in particular the function get_image which reads the heap and create the corresponding cv::Mat.
#include <emscripten/bind.h>
#include <emscripten/val.h>
#include <opencv2/core/core.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/opencv.hpp>
// The first image is recorded in `recordedImage`
cv::Mat recordedImage;
/*
* `get_image` reads the part of the heap indicated by JS.
* Then it interprets it as a cv::Mat.
* */
cv::Mat get_image(int offset, size_t size, int width, int height) {
uint8_t* pos;
pos = reinterpret_cast<uint8_t*>(offset);
auto gotMatrix = cv::Mat(width, height, CV_8UC4, pos);
return gotMatrix.clone();
}
/*The first image is recorded in `recordedImage`.*/
void recordImage(int offset, size_t size, int width, int height) {
recordedImage = get_image(offset, size, width, height);
}
/*The second image is compared to the first one.*/
void compare(int offset, size_t size, int width, int height) {
std::cout << "===== c++ comparing the images ======" << std::endl;
auto image1 = recordedImage;
auto image2 = get_image(offset, size, width, height);
auto orbDetector = cv::ORB::create(500);
cv::BFMatcher matcher(cv::NORM_L2);
std::vector<cv::KeyPoint> kpts1;
std::vector<cv::KeyPoint> kpts2;
cv::Mat descr1;
cv::Mat descr2;
orbDetector -> detectAndCompute(image1, cv::noArray(), kpts1, descr1);
orbDetector -> detectAndCompute(image2, cv::noArray(), kpts2, descr2);
std::vector<cv::DMatch> matches;
matcher.match(descr1, descr2, matches);
for (auto & match : matches) {
std::cout << match.distance << std::endl;
}
}
int main() {
std::cout << "Run main in c++." << std::endl;
return 0;
}
// Export the functions.
EMSCRIPTEN_BINDINGS(my_module) {
emscripten::function("main", &main);
emscripten::function("compare", &compare);
emscripten::function("recordImage", &recordImage);
}
I already tried to print the cv::Mat and, indeed, the images I have in this version are different than the ones in pure c++. The difference is that canvas adds a "alpha" channel and OpenCV permutes the R and B channels. I already made some pre-manipulations in JS to fix these differences, and checked that the matrices are the exact same matrices in the html/js/wasm version as in the c++ version (as far as I can check).
I did not included these manipulations in the example here.
My question : the values of match.distance are much larger than the values obtained with the same image using a pure c++ program reading on the disk (cv::imread).
Why ?
