The adoption of artificial intelligence and machine learning is expected to deepen as organizations seek to increase efficiencies. But compromised models could cause financial losses or reputational damage to an organization.
As CISOs, then, it is our mandate to protect the integrity of AI/ML models by creating a security testing and validation program.
To make sure models don’t become unpredictable, it’s essential to test for susceptibility to two main types of attacks — data poisoning and manipulation.
Many models continue to learn once they’re put into use. With data poisoning, an attacker introduces malicious data into the training set to compromise the model’s performance after it is already deployed to get it to act in an unintended manner or to produce results it normally wouldn’t.
With data manipulation, an attacker injects inputs to deceive the model into making a wrong prediction or classification. Imagine an autonomous vehicle mistaking a stop sign for a speed limit sign. The effect could be deadly.
Poisoned or manipulated model can also introduce biases that can lead to unfair or discriminatory outcomes. Security testing will help to maintain that the model hasn’t been tampered with.
Security testing also has regulatory and compliance implications. We are already seeing more and more governments publishing requirements or guidelines around the use of AI and ML. As we see more AI and ML vulnerabilities, we can expect more regulations to follow, perhaps in the form of more specific security standards. If you’ve already set up a security testing and validation program, that should put you in front of some of these evolving standards.
Just as we do threat modeling in security in general, we need to do threat modeling for AI and ML models. Start by understanding what the potential threats are. Who might want to attack the model? Is it someone seeking financial gain? Is it someone looking to do reputation harm to an organization?
By knowing what the threat landscape is, you can put more effective testing and security around your model.
Once the landscape is established, you need to test for data poisoning. One way is to validate the data sources. You want to make sure that all the data sources being used for that model come from reliable verified sources, and that you have controls around who can put that data into production.
The same holds for anomaly detection. You want to have ways to monitor for any anomalies in the training data. Have there been unexpected changes to the data that could indicate poisoning attempts? Aside from monitoring, you want to make sure the model isn’t going to act oddly if it gets a string of code or an inject that wasn’t predicted.
As part of the testing, you want to build adversarial examples. Once you do the threat model, create manipulated inputs to test the model’s robustness against them. Did they compromise the model, or was the model able to successfully reject them?
You also want to make sure you have a means to recognize drift, and do regular updates to help the model defend against it.
In some cases, models need bounds in terms of expected inputs or outputs to limit the potential for harm.
While general testing focusing on performance, operation, ethics and bias avoidance bias needs to be taken into consideration for AI and ML, we also need to focus on preventing bad things from happening with the model. Security needs to be proactive in this space, as an essential component of a good AI and ML program.