Smart cameras – the eyes of logistics

It is intelligent, uncomplicated and has awesome eyes. And, best of all: It really exists – the smart camera. In conjunction with the corresponding software for teaching an artificial intelligence, Fraunhofer IML has created a “starter kit” that is as simple as it is ingenious, paving the way for self-sufficient and data protection-compliant image processing in the logistics sector.

But let’s start from the beginning: Image processing or computer vision (CV) is no longer a mere buzzword; it has become established as a promising digitalization approach for logistics. CV makes it possible to glean meaningful information from digital images and videos as the basis for efficient process optimizations and cost savings.  

The smart camera is a promising example of the implementation of computer vision in an industrial environment. It consists of a 3D-printed housing, a camera sensor and a lens – all modular purchased parts available on the market. This allows the camera to be adapted to the respective application and the corresponding requirements for the images to be processed.

The camera gets its intelligence from a so-called NVIDIA Jetson board, a minicomputer the size of a credit card with a graphics processor. “As a result of the increasing calculation and graphics power of such embedded boards, smart cameras can evaluate the recorded images directly on the device without the images having to be transferred to a central server,” explains Julian Hinxlage, project manager at Fraunhofer IML. “The advantages are obvious: data efficiency and data protection,” adds Hinxlage.

Since the camera only transmits relevant information and not complete image data, the load on the networks is considerably reduced. This leads to a more efficient data transfer. In addition, the camera only needs a current source and can be connected to LAN, WLAN or mobile communications as needed. This makes it optimally suited for use in locations with a limited infrastructure. Since no personal image data is shared, the smart camera is also a data protection-friendly solution. It is able to analyze images and extract relevant information without revealing sensitive data. This is especially advantageous when people are visible in an image. “The works council may have certain reservations or concerns about it, but if an employee is screwing up, to put it that way, or is taking a break that is too long, the camera cannot recognize it. The image data does not leave the camera; it only passes on certain information, ultimately like an entry in a database,” Hinxlage points out. 

In addition to the smart camera, the developed software is the heart of the project. This allows AI models to be trained for image processing. It comprises various components and tools for data acquisition and management as well as for training the models.

The process starts with a data recording, followed by the annotation of the image data, in which certain objects or features are marked in the image. “If I want to identify people in a picture, for example, or maybe I simply want to hide people, then the AI has to recognize them as such first. For this purpose, I sort of draw a small box around all people in the image and then assign them to the class “Persons.” After training, the algorithm therefore knows what a person looks like and what features it has to pay attention to,” explains Hinxlage. 

After the annotation, the actual magic takes place: the model training. In this process, an AI model is developed on the basis of the annotated data and transferred to the smart camera. This model subsequently allows new data to be interpreted in real time. “We have trained with existing data, and now the AI has to apply what has been learned to new data. For example, if a person now walks through the image who has never been recorded before, the camera still has to recognize this object as a person. Depending on the application and complexity, the training can take from only a few minutes to several weeks.

The software has many standard components and tools for AI training. What would otherwise have to be laboriously developed through individual components can all be found through the training that is carried out. “I have everything in one place and can generate my own AI in a few minutes. I am actually even faster because I can train new objects at any time,” explains Hinxlage. “And this smart camera actually supplements it as well, because I have the software and this individual device that allows me to get started without having to assemble or wire everything. This should basically be the starter set, the start-up package for image processing. Especially when you think of small and medium-sized enterprises,” says Hinxlage. For as many companies as possible to benefit from this, both the camera housing and the software are available as open source in the framework of the “Silicon Economy.”

The software can be used to realize a variety of applications in image processing, for example, counting containers or recognizing pallet types during loading checks of incoming and outgoing goods. Quality checks and optimizing storage place occupancy are also possible scenarios. The camera can also be used outside to monitor the position of trucks in the yard. If a truck is at the wrong gate and risks being incorrectly loaded, the camera can report the mistake, especially if the camera is directly connected with the yard management system. This can prevent the truck from having to be unloaded and reloaded or, in the worst case, arriving at the destination with the wrong goods.

The response to the developed software and the smart camera is positive. The simplified image processing and the user-friendly interface allow even non-specialists to create their own AI models and thereby better realize the potential of artificial intelligence in logistics. “It is actually a hurdle for many companies to deal with (AI-based) image processing because they don’t have access to it and simply don’t know: How do I start with it? What do I have to do for this? I didn’t know that before either,” says Julian Hinxlage with a laugh. The smart camera and the innovative software are thus the ideal solution for all companies that are looking for efficient and data-minimizing image processing solutions, especially given the increasing data quantities and data protection concerns. With its self-sufficiency, data efficiency and data protection friendliness, the smart camera is a promising innovation for the logistics of tomorrow.