Artificial intelligence predicts the outcomes of cancer treatments.

The melding of visual information (microscopic and X-ray images, CT and MRI scans, for example) with text (exam notes, communications between physicians of varying specialties) is a key component of cancer care. But while artificial intelligence helps doctors review images and home in on disease-associated anomalies like abnormally shaped cells, it’s been difficult to develop computerized models that can incorporate multiple types of data.

Researchers at Stanford Medicine have now developed an AI model able to incorporate visual and language-based information. After training on 50 million medical images of standard pathology slides and more than 1 billion pathology-related texts, the model outperformed standard methods in its ability to predict the prognoses of thousands of people with diverse types of cancer, to identify which people with lung or gastroesophageal cancers are likely to benefit from immunotherapy, and to pinpoint people with melanoma who are most likely to experience a recurrence of their cancer.