A new California law will bar hospitals and doctors’ offices from using artificial intelligence to independently perform medical tasks that require a professional license, such as prescribing medications or ordering tests.
Read more Barabak: Hilton makes history in California governor’s race. But not the kind he had in mind.
Assembly Bill 1979, signed last week by Gov. Gavin Newsom and taking effect Jan. 1, requires healthcare facilities to take reasonable steps to ensure licensed professionals can exercise independent judgment when AI tools help guide patient care. Clinicians can use those tools to inform their decisions.
“AI is rapidly integrating into our health care system and reshaping our personal experience with it,” Assemblymember Mia Bonta, an Oakland Democrat who carried the bill sponsored by the California Nurses Association, said in a statement. “Protecting patient safety, keeping our professional workforce engaged, and preserving the integrity of health care requires that we not rely exclusively on AI to do things quicker and cheaper.”
Michelle Gutierrez Vo, a registered nurse at Kaiser Permanente in Fremont and a president of the California Nurses Association labor union, said nurses are widely concerned that hospital administrators are replacing their functions with generative AI tools or algorithm-based devices, to the detriment of patients.
“It’s happening right now. And the employers are going to deny it,” she said. “Employers are inserting AI technology into workflows that were usually performed, rightfully so in health care, by nurses and physicians.”
A separate union, the National Union of Healthcare Workers, filed a against Kaiser with state and federal authorities in July. It alleged the nonprofit healthcare giant had implemented an algorithm to assess patients seeking mental healthcare, which produced recommendations for “specific levels of care without the oversight of licensed clinicians.”
And last year, the Wall Street Journal reported Kaiser had implemented an AI-based algorithm to flag patients at risk of developing sepsis, a dangerous physiological response to an infection that can lead to death, and another algorithm recommending care pathways for “advice nurses” who field calls from ill patients.
In emails, Elissa Harrington, a spokesperson for Kaiser Permanente, did not respond to requests for comment on the labor union’s complaint, the underlying technology of the mental health screening tool, or the tool for advice nurses.
“At Kaiser Permanente, technology, AI, and other advanced tools do not make care decisions,” she wrote. “Our clinicians do. AI and predictive tools are designed to support clinicians and care teams, not replace their judgment.”
She said the predictive algorithm that alerts care providers when patients are at heightened risk of deterioration had saved patient lives. Internal Kaiser research published in 2020, before recent leaps in AI technology, found the tool had reduced deaths by 16%
It’s unclear if the law prohibits any current uses of artificial intelligence in hospitals or clinics. A spokesperson for Bonta, Daniel McGreevy, did not provide any examples. Neither did Lucy Diavolo, a spokesperson for the California Nurses Association.
David Simon, a spokesperson for the California Hospital Association, which represents administrators and executives, said the law preserves the ability of physicians and nurses to use AI tools that benefit patients.
“Clinicians should retain ultimate authority in care decisions about patients,” said spokesperson David Simon, “and AI should be a supportive tool for those clinicians as we continue to make thoughtful, measured advancements that improve health outcomes for Californians.”
The hospital association and the California Medical Association, representing 50,000 doctors, were neutral on the law. The physicians group had been opposed to earlier versions of the bill, which it said would have broadly prohibited the use of AI tools in medicine, until Bonta narrowed the language.
While nurses have criticized algorithmic screening and alert systems implemented at Kaiser in particular, they have largely withheld their criticism of AI-rooted medical technologies. For example, the UC Davis Health system is applying recent advances to guide surgeons during spine surgeries and quickly spot diseases in diagnostic scans.
Read more Asking Eric: She thinks her mother was a saint. Maybe these photos would convince her otherwise.
Meanwhile, physicians are themselves increasingly asking generative AI tools for suggestions.
Especially popular are clinical decision support tools, such as the product Open Evidence, which summarizes vast amounts of research and recommends possible diagnoses. Another, Abridge, listens in on patient appointments with providers and generates notes. Physicians can also use it like a chatbot to ask questions about patient care.
Dr. Donaldo Hernandez, who manages care of hospitalized patients at two health centers in Santa Cruz, said Open Evidence is a “fast librarian,” generating care suggestions in seconds that may have taken him hours – with citations from medical journal articles and medical associations. Recently, he used the tool to develop guidelines for treating patients with an iron deficiency, he said.
Open Evidence only offers suggestions, he said, and physicians should always view those recommendations with a discerning eye.
With the new law, Henry Norwood, a San Francisco-based healthcare attorney with the defense firm Kaufman Dolowich, expects to see patients file more malpractice suits against doctors claiming they over-relied on AI guidance tools – especially when a doctor may have improperly delivered care.
“That could be a really big problem, particularly if there is a mistake,” Norwood said. “AI, we’ve seen, can be incorrect.”
The law still allows AI tools for documentation and communication, including in some automated messages to patients, and hospitals must take steps to ensure humans retain their power to make care decisions for patients.
State lawmakers also grappled with another question top of mind for California nurses this year: What consequences should result when a health professional consults an AI tool to make a clinical decision, and then the patient is harmed?
State legislators took on that question this year after nurses at Kaiser Permanente said they felt pressured by administrators to follow guidance from algorithmic tools, but have faced consequences in the workplace for doing so.
In one instance reported by the Wall Street Journal, an advice nurse at a Kaiser call center in Vallejo followed the suggestion of an algorithm used by the health giant to screen patients, which recommended that a Marin County man with a cough, chest pains and fever did not need to see a provider. The man died several days later, and an arbitrator later ruled the nurse was responsible because she had not overruled the algorithm’s advice, the news outlet reported.
Harrington, the Kaiser spokesperson, did not respond to an email seeking information about the case by press deadline.
The California Nurses Association sponsored a separate bill this year that would have granted special workplace protections to healthcare professionals in these scenarios.
But Newsom vetoed the bill, AB 2575, on technical grounds, writing in a letter that it set too high a bar compared to other workplace retaliation protections. The California Hospital Association — which often battles the nurses association in policy debates — also argued the bill would have applied too broadly to a plethora of AI devices, such as those managing electronic records systems.
Read more Harriette Cole: My husband has let running take over his entire weekend
About the Author
