Query Images

WARNING auKsys/5 is still under development, and subject to changes. This documentation has not been fully updated with changes since auKsys/4.

In this tutorial, we are going to look how to query for an image at a given location, convert it to an OpenCV image and display it. We will require a dataset store with the kDBSensing extension (this will also enable the kDBDatasets extension). It can be started with the following command:

kdb store --path tuto_kdb_store --extension kDBSensing

Or it can be started programmatically, refer to the Getting Started tutorial for more information.

Running examples from this tutorial can be found at the query images examples.

Query for Image

We will use the dataset of images acquired in the water near the Gränsö castle. It can be downloaded and uncompress with the following command.

curl -o images_sea.kdb_dataset.xz "https://gitlab.liu.se/lks/tutorials_data/-/raw/main/images_sea.kdb_dataset.xz?ref_type=heads&inline=false"
xz -d images_sea.kdb_dataset.xz

The dataset can be imported into the database using the following command:

kdb datasets import --path tuto_kdb_store --filename images_sea.kdb_dataset

To query for an image at a given location, we will use a SQL query. kDB uses three tables for storing images:

cameraframes contains the image data and associated metadata
agentpositions contains the agent positions
frametransformations contains the frame transformations used by the platform

For this tutorial we will ignore the frame transformations, and assume that we are interested in the 5 images acquired near a given location, we will use 16.681958, 57.760420 in this example:

SELECT cf.data, cf.width, cf.height, cf.encoding, ap.longitude, ap.latitude FROM cameraframes cf
  JOIN agentpositions AS ap
  ON cf.timestamp >= ap.timestamp AND cf.timestamp < ap.nexttimestamp
  ORDER BY ST_Distance(ST_SetSRID(ST_MakePoint(16.681958, 57.760420),4326), ST_SetSRID(ST_MakePoint(ap.longitude, ap.latitude),4326))
  LIMIT 5

This will return as result the rawdata of the images (cf.data), the width (cf.width), the height (cv.height), its encoding (cf.encoding, for instance rgb8) and the longitude (ap.longitude) and latitude (ap.latitude).

Display in OpenCV

The examples bellow show how to create a knowCore::Image from the previous SQL Query, and how to convert that to a C++ cv::Mat or a Python numpy.array, and to display the resulting image in OpenCV.

C++
Python

#include <clog_print>

#include <opencv2/opencv.hpp>

#include <knowCore/Image.h>
#include <knowCore/TypeDefinitions.h>

#include <knowDBC/Query.h>
#include <knowDBC/Result.h>

#include <kDB/Repository/Connection.h>

kDB::Repository::Connection connection = kDB::Repository::Connection::create("tuto_kdb_store", 1242);

connection.connect();

knowDBC::Query query = connection.createSQLQuery(R"V0G0N(SELECT cf.data, cf.width, cf.height, cf.encoding, ap.longitude, ap.latitude FROM cameraframes cf
  JOIN agentpositions AS ap
  ON cf.timestamp >= ap.timestamp AND cf.timestamp < ap.nexttimestamp
  ORDER BY ST_Distance(ST_SetSRID(ST_MakePoint(16.681958, 57.760420),4326), ST_SetSRID(ST_MakePoint(ap.longitude, ap.latitude),4326))
  LIMIT 5)V0G0N");

knowDBC::Result result = query.execute();

for(int i = 0; i < result.tuples(); ++i)
{
  clog_print("At coordinates ({}, {}) image ({}x{}, {})",
    result.value(i, 4), result.value(i, 5), result.value(i, 1), result.value(i, 2), result.value(i, 3));
  knowCore::Image img = knowCore::Image::fromRawData(result.value<QByteArray>(i,0).expectSuccess(),
          result.value<quint64>(i, 1).expectSuccess(), result.value<quint64>(i, 2).expectSuccess(),
          result.value<QString>(i, 3).expectSuccess()).expectSuccess();
  cv::Mat cv_img(cv::Size(img.width(), img.height()), CV_8UC3, img.dataPtr(), cv::Mat::AUTO_STEP);
  cv::imshow("image", cv_img);
  cv::waitKey(0);
}

from kDB.Repository import Connection
import knowCore
import numpy as np
import cv2

connection = Connection.create("tuto_kdb_store")
connection.connect()

query = connection.createSQLQuery("""SELECT cf.data, cf.width, cf.height, cf.encoding, ap.longitude, ap.latitude FROM cameraframes cf
  JOIN agentpositions AS ap
  ON cf.timestamp >= ap.timestamp AND cf.timestamp < ap.nexttimestamp
  ORDER BY ST_Distance(ST_SetSRID(ST_MakePoint(16.681958, 57.760420),4326), ST_SetSRID(ST_MakePoint(ap.longitude, ap.latitude),4326))
  LIMIT 5""")

result = query.execute();

for i in range(0, result.tuples()):
    print(f"At coordinates ({result.value(i, 4)}, {result.value(i, 5)}) image ({result.value(i, 1)}x{result.value(i, 2)}, {result.value(i, 3)})")
    img = knowCore.Image.fromRawData(result.value(i,0), result.value(i, 1), result.value(i, 2), result.value(i, 3))
    img_np = np.array(img, copy=True)
    cv2.imshow('image',img_np)
    cv2.waitKey(0)

The first image to be displayed is at coordinates (16.681914783967493, 57.76033842696137):

Image from the sea near Gränsö.