Researchers have unveiled a new method to compromise artificial intelligence (AI) computer vision systems, enabling control over what the AI perceives. The technique, known as RisingAttacK, has proven effective in manipulating widely used AI computer vision systems.
The research focuses on “adversarial attacks,” where data fed into an AI system is manipulated to alter its perception of images. This could impact autonomous vehicles by interfering with their ability to detect traffic signals or pedestrians. Similarly, hackers could exploit medical devices like X-ray machines to produce inaccurate diagnoses.
“We wanted to find an effective way of hacking AI vision systems because these vision systems are often used in contexts that can affect human health and safety – from autonomous vehicles to health technologies to security applications,” said Tianfu Wu, co-corresponding author of the study and associate professor at North Carolina State University. “That means it is very important for these AI systems to be secure. Identifying vulnerabilities is an important step in making these systems secure since you must identify a vulnerability in order to defend against it.”
RisingAttacK operates by making minimal changes to an image, allowing users to manipulate what the AI perceives. Initially, it identifies visual features within the image and determines which are crucial for achieving the attack’s goal.
“For example,” Wu explained, “if the goal of the attack is to stop the AI from identifying a car, what features in the image are most important for the AI to be able to identify a car in the image?”
The technique then assesses how sensitive the AI system is to changes in key features’ data. “This requires some computational power but allows us to make very small, targeted changes that make the attack successful,” Wu stated. “The end result is that two images may look identical to human eyes… But due to RisingAttacK, the AI would see a car in the first image but would not see a car in the second image.”
The researchers tested RisingAttacK on four common vision AI programs: ResNet-50, DenseNet-121, ViTB, and DEiT-B. It successfully manipulated all four.
“While we demonstrated RisingAttacK’s ability to manipulate vision models, we are now in the process of determining how effective the technique is at attacking other AI systems, such as large language models,” Wu noted.
The paper titled “Adversarial Perturbations Are Formed by Iteratively Learning Linear Combinations of the Right Singular Vectors of the Adversarial Jacobian” will be presented at the International Conference on Machine Learning in Vancouver. Co-authors include Thomas Paniagua and Chinmay Savadikar from NC State University.
This research was supported by grants from both the National Science Foundation and Army Research Office. The team has made RisingAttacK publicly available for testing neural networks’ vulnerabilities at https://github.com/ivmcl/ordered-topk-attack.
