Texas Attorney General Resolves Investigation of GenAI Healthcare Technology Firm
Texas Attorney General Ken Paxton has announced that an agreement has been reached with a Texas-based artificial intelligence healthcare technology firm to resolve allegations the company violated the Texas Deceptive Trade Practices – Consumer Protection Act by making false, misleading, or deceptive statements about the accuracy of its artificial intelligence (AI)-based solution.
In June 2024, Attorney General Paxton announced that he had launched a new data privacy and security initiative to protect Texans’ sensitive data from illegal exploitation by technology, AI, and other firms. The initiative was housed within the Consumer Protection Division of the Office of Inspector General and was focused on enforcing compliance with Texas laws such as the Data Privacy and Security Act, Identify Theft Enforcement and Protection Act, Data Broker Law, Biometric Identifier Act, Deceptive Trade Practices Act and federal laws such as the Children’s Online Privacy Protection Act (COPPA) and the Health Insurance Portability and Accountability Act (HIPAA).
“Companies that collect and sell data in an unauthorized manner, harm consumers financially, or use artificial intelligence irresponsibly present risks to our citizens that we take very seriously,” said Attorney General Paxton when launching the initiative. “As many companies seek more and more ways to exploit data they collect about consumers, I am doubling down to protect privacy rights.”
Under this initiative, the Dallas, TX-based healthcare technology firm Pieces Technologies was investigated over alleged false and misleading statements about the accuracy and safety of its AI-based products for in-patient healthcare facilities. The company’s products include AI-based tools that can be used by physicians to help them treat patients, such as for summarizing, charting, and drafting clinical notes to be entered into electronic health records.
Pieces Technologies had developed a series of metrics and benchmarks that showed that its generative AI solution outputs were highly accurate. According to the company, which used the term “hallucination” to describe outputs that were incorrect or misleading, its generative AI had an error rate of less than 1 per 100,000.
According to Attorney General Paxton, at least four major Texas hospitals had been providing a real-time feed of patient data to Pieces Technologies to allow its generative AI solution to summarize patient conditions and treatments for clinicians. The investigation found the company’s claims that its generative AI solution was highly accurate and had an extremely low error rate “were likely inaccurate and may have deceived hospitals about the accuracy and safety of the company’s products.”
The agreement does not include any punitive actions or a financial penalty but does require Pieces Technologies to ensure compliance with state and federal laws by making clear and conspicuous disclosures in its marketing and advertising and avoiding making misrepresentations about the accuracy of its solutions.
These requirements include clearly and conspicuously disclosing the meaning or definition of any metric, benchmark, or similar measurement and the method or procedure used to calculate that metric, benchmark, or other measurement. Documentation must also be maintained and provided to customers about the type(s) of data and/or models used to train its products and services, their intended purpose and uses any known or reasonably knowable limitations and risks to patients and healthcare providers, and any misuses of a product or service that can increase the risk of inaccurate outputs or increase the risk of harm to individuals. The agreement covers a period of 5 years.
“This is an Assurance of Voluntary Compliance [AVC] into which Pieces entered with the Texas OAG to demonstrate leadership, transparency, and collaboration. It sunsets in five years, which is not the norm, and can be rescinded by the OAG as early as one year from its effective date upon request,” Pieces Technologies told the HIPAA Journal. “Pieces vigorously denies any wrongdoing and believes strongly that it has accurately set forth its hallucination rate, which was the sole focus of the AVC.”
“We are disappointed in the Texas Office of the Attorney General’s press release that misrepresents the Assurance of Voluntary Compliance (AVC) into which Pieces entered,” explained Pieces. “The AVC makes no mention of the safety of Pieces products, nor is there evidence indicating that the public interest has ever been at risk. The AVC focuses solely on the company’s reporting of hallucination rates in the context of an independently developed risk-classification system that is based on severity. Importantly, there is no industry-wide risk classification system for generative AI hallucinations for inpatient clinical summarization that exists today.”
This article has been updated to include comments from Pieces Technologies.
