Pennsylvania Drew a Line on Healthcare AI. The Next Step Is Drawing the Map.

By Charlotte Kalafut, Co-Founder & CEO, Asher Informatics

A few weeks ago, Pennsylvania’s Attorney General sued Character AI for deploying a chatbot that posed as a licensed medical doctor, complete with an invented CV and fabricated licensure credentials. Medical oversight has always lived at the state level. While Pennsylvania pushes to be a key player in AI development, it’s good to see the state isn’t willing to sidestep patient safety to get there. When there’s money to be made and investors involved, doing the right thing for patients can slip. I believe in the promise of clinical AI. Personally, living with a rare disease means better outcomes and shorter diagnostic journeys. The healthcare AI map can help all patients.

The volume of biomedical literature alone now exceeds 1.5 million new publications per year in PubMed, and the broader scientific literature crossed 3 million indexed articles in 2023, a 59% increase over the prior decade.¹ Research on publication growth rates suggests that output could double within roughly 17 years, and the pace is still accelerating.² No clinician can keep up. That’s not a criticism; it’s math.

Here’s the example that gets me: the NIH now tracks more than 9,000 rare diseases.³ Most have no approved treatment, and the specialists who can diagnose them are a thin slice of the physician population. Clinical AI can take that expertise, build it into a model, and run it at any health system in the country. The future was never a handful of AI tools; it was always about hundreds, maybe thousands, of models working together to support better care. The average time to diagnose a rare disease is still seven to nine years.⁴ Cutting that to three would be transformative.

The problem is that AI performs differently at each site. That’s especially true for LLM-based clinical decision support, where slight changes in a question, a prompt, or a user interaction can shift the output. It’s hard for any health system to know how a model will perform out of the box. And we’ve moved fast: from a few dozen clinical AI models just years ago to more than 1,500 regulated models and 10,000+ AI tools deployed across healthcare.

The healthcare AI map: Why adoption is slow, even without new laws to slow it down

U.S. case law hasn’t clearly settled how liability for harm from clinical AI should be allocated. Legal theorists generally expect responsibility to be shared among clinicians, hospitals, and developers depending on the specifics. The AI developer carries some risk. The health system that purchased and deployed the model carries some. The clinician uses it also carries some. That distributed liability gives every party reason to be cautious, and every party needs evidence the model is performing safely in their specific context.

Second, AI’s promise is to reduce burden, save time and money, and improve outcomes. Without the right tools to govern compliance, evaluate models at each site, and monitor performance locally over time, adoption will stay too slow or get written off as too risky. Third, if a health system can’t effectively govern its AI, is it more responsible? Or is the system just accumulating risk it isn’t equipped to manage?

That third question is what drove my co-founders and me to start Asher Informatics. Because the alternative is a future where AI widens existing health inequity. The well-resourced systems will figure out governance, deploy the models, and pull ahead. Smaller and rural systems, the ones already serving patients with the least access, will fall further behind. That’s a “haves AI” versus “have nots” outcome. It is not the future I want clinical AI to deliver.

A call to Pennsylvania

Governor Shapiro, Attorney General Sunday, and the legislators advancing AI infrastructure in the Commonwealth: thank you for your action. It mattered. It told developers and deployers that Pennsylvania expects AI to be used in the practice of medicine to meet the standards of clinical practice.

Now go further.

This lawsuit is an encouraging step, but hopefully the first of more efforts to put sensible guardrails on data-driven technologies used in healthcare for Pennsylvania patients. Concrete next steps should include setting aside budget to fund and staff the regulatory infrastructure that lets hospitals, clinics, and community health centers evaluate and monitor the models being sold to them. Build the workforce. Write clear guidance for procurement. Make compliance support a service, not just an enforcement threat. Pay particular attention to the systems serving rural and underserved populations, because those systems have the least bandwidth to govern AI on their own and the most to lose when it goes wrong.

Pennsylvania can be the state that proves you can lead on AI development and lead on AI oversight at the same time – they’ve drawn the line – now we can make the map. Make public commitments. The patients we all serve deserve both.

References

1. González-Márquez R, Schmidt L, Schmidt BM, Berens P, Kobak D. The landscape of biomedical research. Patterns. 2024;5(6):100968. https://doi.org/10.1016/j.patter.2024.100968

2. Bornmann L, Haunschild R, Mutz R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit Soc Sci Commun. 2021;8:224. https://doi.org/10.1057/s41599-021-00903-w

3. National Center for Advancing Translational Sciences (NCATS). Genetic and Rare Diseases (GARD) Information Center. National Institutes of Health. https://ncats.nih.gov/research/research-activities/informatics/rare-disease-translational-research

4. National Institutes of Health. Artificial Intelligence in Medical Imaging and its Impact on the Rare Disease Community. PMC. 2022. https://pmc.ncbi.nlm.nih.gov/articles/PMC8764708/