Tracking people’s movements and their behavior is a complex task with many applications in a variety of business areas. Prominent use-cases include people counting, path optimization in retail environments and analyzing in-store consumer behavior. More recently, the COVID-19 pandemic introduces new highly demanded solutions such as distance monitoring in public or work spaces.

Our new HemiStereo® NX depth camera with integrated AI is ideally suited for realizing this task. Its ultra-wide field of view of 180 degrees eliminates the need of a multiple sensor system reducing acquisition and data fusion costs in the process. Additionally, HemiStereo® comes with a powerful integrated GPU for processing of deep neural networks (DNNs) that lets you run the person tracking application directly on the sensor without external computing. Last but not least HemiStereo® provides extremely detailed 3D information due to its 12 mega-pixel resolution — crucial for the success of the application.

Setting up the person tracker
But enough with the introduction. Let us setup a person tracker on the HemiStereo® NX. Plug your sensor into your network or connect it locally to your PC. The DNN model used, requires the images in an upright position. The easiest way to achieve this is to put HemiStereo NX in a vertical orientation as show in the image. Find out the sensors IP address with HemiStereo Viewer. After that open a terminal and connect to the sensor via SSH:

$ ssh hemistereo@IP-ADDRESS-OF-THE-SENSOR

Enter your password (the default password is hemistereo). Now we need to run the following commands.

$ sudo apt update
$ sudo apt install 3dvl-jupyter
$ sudo hemistereo_app_run jupyter-notebook
$ sudo apt install 3dvl-jupyter-persontracker

We prepared those packages to include everything you need to get started. The 3dvl-jupyter package installs all dependencies you need (this may take some time) and enables you to comfortably access the application via a web browser. Along with that the 3dvl-jupyter-persontracker package provides the AI models and the program to track people. After the installation is finished open a browser of your choice and enter

https://IP-ADDRESS-OF-THE-SENSOR:8880/tree/samples
(make sure to start with https not http)

into the address bar. The notebook app uses a self-signed SSL certificate, which causes a warning for all browsers on first access. Type in the password “3dvisionlabs” and log in to the jupyter notebook. Click on “person_tracker.ipynb” to open the interactive person tracking script.

You can now run the application via the upper toolbar: Select “Cell” and “Run all” to run the entire notebook. The first execution of the application may take a bit longer since the program compiles computational-heavy functions to machine code to improve the performance on the subsequent executions. If everything is set up correctly, you will see the detected people on the camera image and the visualization of the tracking in a top-view perspective at the end of the notebook.

You can now track people with HemiStereo®. The application utilizes a YOLOv4 DNN model to detect people in the camera image. Those detections are tracked based on their position and movement with the help of a kalman filter.
The 3D positions of the detected people are calculated with the help of the 3D information from the sensor. Finally, the tracks of the people are displayed in the top-view image relative to the sensor.

HemiStereo® also gives you the flexibility to adjust and develop your own custom applications based on your needs. For example tracking other objects than persons. In our next post we will give more information on how to adjust the application and extend it to keep count of how many people are inside a monitoring area e.g. a building.