Home Articles Cybersecurity threats in the Endpoints of Artificial Intelligence (AI) systems

Cybersecurity threats in the Endpoints of Artificial Intelligence (AI) systems

cybersecurity threats

With AI Endpoints (or TinyML) being in the early stages and slow to be adopted by the industry, more companies are incorporating AI into their systems, such as for predictive maintenance in factories or keyword detection in consumer electronics. . But with the addition of an AI component to your IoT system, new security measures must be considered. IoT has matured to the point where you can reliably launch products in the field with peace of mind, with certifications such as PSA Certified™ providing assurance that your IP can be protected through a variety of techniques such as isolated security engines, cryptographic key safe storage and use of Arm® TrustZone®. Such guarantees can be found in microcontrollers (MCUs) designed with security in mind with scalable hardware-based security features, such as the Renesas RA family. However, the addition of AI leads to the introduction of new threats infesting secure areas, specifically in the form of adversarial attacks.

Adversarial attacks target the complexity of deep learning models and the underlying statistical mathematics to create weaknesses and exploit them in the field, leading to parts of the model being leaked, training data being leaked, or generating unexpected results. This is due to the black box nature of Deep Neural Networks (DNNs), where decision making in DNNs is not transparent, i.e. "hidden layers" and customers are not willing to risk their systems by adding a AI function, which slows down the proliferation of Endpoints. Adversarial attacks are different from conventional cyber attacks in that in traditional cyber security threats, security analysts can fix the bug in the source code and document it extensively. Since in DNNs there is no specific line of codes that you can address, it becomes understandably difficult.

Adversary attacks in pop culture can be seen in Star Wars. During the clone wars, Order 66 can be seen as an adversarial attack where the clones behaved as expected during the war, but once the order was given they changed, causing a tide in the war. .

Notable examples can be found in many applications, such as a team of researchers sticking stickers on stop signs, causing the AI ​​to predict it as a speed sign [1]. Such misclassification can lead to traffic accidents and increased public distrust in the use of AI in systems. The researchers were able to get 100% misclassification in a lab setting and 84,8% in field tests, showing that the stickers were quite effective. The tricked algorithms were based on convolutional neural networks (CNNs), so it can be extended to other use cases using CNN as a base, such as object detection and keyword detection.

endpoints
Figure 1. Stickers affixed to the STOP sign to trick the AI ​​into thinking it is a speed sign, the stickers (disturbances) are used to mimic graffiti to hide in plain sight.

Another example from researchers at the University of California, Berkley, showed that adding noise or disturbance to any music or speech would cause the AI ​​model to misinterpret it as something other than the music played or transcribe something else entirely, but that disturbance is inaudible. for the human ear [2]. This can be maliciously used in smart assistants or AI transcription services. The researchers have reproduced the audio waveform that is more than 99,9% similar to the original audio file, but can transcribe any audio file of their choice with a 100% success rate using Mozilla's DeepSpeech algorithm.

security attacks
Figure 2. By adding a small perturbation, the model can be tricked into transcribing any desired phrase.

Types of adversary attacks

To understand the many types of adversarial attacks, one should look at the conventional TinyML development pipeline, as shown in Figure 3. Where training is initially done offline, typically in the cloud, followed by the polished binary executable. final displayed on the MCU and used via API calls. The workflow requires a machine learning engineer and an embedded engineer. Since these engineers tend to work in separate teams, the new security landscape can lead to confusion about the division of responsibilities among the various stakeholders.

tinyml
Figure 3. End-to-end TinyML flow

Adversary attacks can occur in the training or inference phases. During training, a malicious attacker could attempt "model poisoning", which can be targeted or untargeted. In targeted model poisoning, an attacker would pollute the AI ​​base model/training dataset, resulting in a "backdoor" that can be triggered by arbitrary input to get a particular output, but works fine with the expected entries. Contamination could be a small disturbance that does not affect the expected operation (such as model accuracy, inference rates, etc.) of the model and would give the impression that there are no problems. This also does not require the attacker to take and deploy a clone of the training system to verify the operation, as the system itself was contaminated and would pervasively affect any system using the poisoned model/dataset. This was the attack used against the clones in Star Wars.

Targetless model poisoning or Byzantine attacks occur when the attacker intends to reduce the performance (accuracy) of the model and stalls the training. This would require going back to a point before the model/dataset was compromised (potentially from the beginning).

In addition to offline training, federated learning, a technique in which data collected from endpoints is used to retrain/improve the cloud model, is inherently vulnerable due to its decentralized nature of processing, allowing attackers to engage with compromised endpoint devices leading to the cloud. the model is compromised. This could have big implications, as that same cloud model could be used across millions of devices.

During the inference phase, a hacker may go for the "model evasion" technique, where they interactively query the model (for example, an image) and add some noise to the input to understand how the model behaves. . In such a way, the hacker could potentially get a specific/required result, i.e. a logical decision after adjusting his input enough times without using the expected input. Such a query could also be used for "model inversion", where information about the model or training data is extracted in a similar way.

Risk analysis during development AI TinyML

For the inference phase, adversarial attacks on AI models are an active field of research, where academia and industry have aligned to work on these issues and developed "ATLAS – Adversarial Threat Landscape for Artificial-Intelligence Systems!", which is a matrix that allows cybersecurity analysts to assess the risk of their models. It also consists of use cases across the industry, including edge AI. Learning from the provided case studies will provide developers/product owners with an understanding of how it would affect their use case, assess risks, and take additional precautionary security measures to alleviate customer concerns. AI models must be seen as prone to such attacks and a careful risk assessment must be carried out by various stakeholders.

For the training phase, ensuring data sets and models come from trusted sources would mitigate the risk of data/model poisoning. Such models/data should generally be provided by trusted software vendors. An ML model can also be trained with security in mind, making the model more robust, much like a brute force approach to adversarial training where the model is trained on many adversarial examples and learns to defend against them. Cleverhans, an open source training library used to build such examples to attack, defend, and benchmark a model for adversarial attacks, has been developed and used in academia. Defense distillation is another method where a model is trained from a larger model to generate probabilities of different classes rather than hard decisions that make it difficult for the adversary to exploit the model. However, both methods can be broken down with enough computational power.

Keep your AI intellectual property safe

Sometimes companies are concerned about the malicious intentions of competitors to steal the feature/model IP that is stored in a device where the company has spent R&D budget. Once the model is trained and polished, it is converted into a binary executable stored on the MCU and can be protected with conventional IoT security measures, such as protecting the physical interfaces of the chip, software encryption, and the use of TrustZone. One important thing to note though, even if the binary executable were to be stolen, it is only the final polished model that is designed for a specific use case that can be easily identified as copyright infringement, and reverse engineering. it would require more effort than starting with a base model from scratch.

Also, in TinyML development, AI models tend to be well known and open source, such as MobileNet, which can then be optimized via a variety of hyperparameters. Data sets, on the other hand, are kept secure as they are valuable treasures that companies spend resources to acquire and are specific to a given use case. For example, adding bounding boxes to regions of interest in images. Generalized data sets are also available as open source, such as CIFAR, ImageNet, etc. They are good for comparing different models, but custom data sets should be used for specific use case development. For example, for a visual wake word in an office environment, an isolated data set in an office environment would give the optimal result.

Summary

As TinyML continues to grow, it's good to be aware of various attacks that can occur on AI models, how they might affect your TinyML development, and specific use cases. ML models have now achieved high accuracy, but for implementation, it's important to make sure your models are robust as well. During development, both parties (ML engineers and embedded engineers) share responsibility for cybersecurity issues, including AI. Where ML engineers would focus on attacks during training and embedded engineers would ensure protection against inference attacks.

For the intellectual property (IP) model, a crucial element is ensuring that training data sets are kept secure to prevent competitors from being able to develop similar models for similar use cases. As for the executable binary, i.e. the IP model on the device, it can be protected with the best-in-class IoT security measures that the Renesas RA family is renowned for, making it very difficult to access. to secure information in a malicious way.