QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery

Abstract

Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions.

This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question–answer pairs designed to teach Multimodal Large Language Models (MLLMs) to understand the appearance and thickness of flakes.

Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a multimodal architecture that incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.

Reasoning with Physics-Aware Instructions

Physics-Informed Attention (PIA)

Physics-Informed Attention (PIA) adopts a simple, physics-aware proxy to approximate the optical thin-film contrast of two-dimensional materials using CIELAB distances. The perceptual color contrast (∆E) between a flake region and its surrounding substrate provides a robust and illumination-invariant measure of reflectance differences caused by thin-film interference, producing an attention map that highlights physically meaningful optical cues without requiring full optical simulation.

TEST INPUTS

SELECT A SAMPLE TO SEE PIA...

RAW MICROSCOPY

PIA ATTENTION MAP

Team

Xuan Bac Nguyen

University of Arkansas

Hoang-Quan Nguyen

University of Arkansas

Sankalp Pandey

University of Arkansas

Tim Faltermeier

University of Utah

Nicholas Borys

University of Utah

Hugh Churchill

University of Arkansas

Khoa Luu

University of Arkansas

Publications

Patents

Khoa Luu, Xuan Bac Nguyen, Hugh Churchill. Automatically detecting false negative objects in 2d material detection data sets. US Patent App. 18/443,058, 2024.

Papers

Hoang-Quan Nguyen, Xuan-Bac Nguyen, Hugh Churchill, Arabinda Kumar Choudhary, Pawan Sinha, Samee U. Khan, Khoa Luu. Quantum-Brain: Quantum-Inspired Neural Network Approach to Vision-Brain Understanding. IEEE Transactions on Image Processing (Under Review), 2025.

Hoang-Quan Nguyen, Xuan Bac Nguyen, Sankalp Pandey, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu. ϕ-Adapt: A Physics-Informed Adaptation Learning Approach to 2D Quantum Material Discovery. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (Under Review), 2025.

James B. Holliday, Eneko Osaba, Khoa Luu. An Advanced Hybrid Quantum Tabu Search Approach to Vehicle Routing Problems. Journal of Quantum Machine Intelligence (Under Review), 2025.

James B. Holliday, Darren Blount, Eneko Osaba, Khoa Luu. Advanced Quantum Annealing Approach to Vehicle Routing Problems with Time Windows. arXiv preprint, 2025.

Hoang-Quan Nguyen, Xuan-Bac Nguyen, Sankalp Pandey, Samee U. Khan, Ilya Safro, Khoa Luu. QMoE: A Quantum Mixture of Experts Framework for Scalable Quantum Neural Networks. IEEE Quantum Week Workshop, 2025.

Xuan Bac Nguyen, Benjamin Thompson, Hugh Churchill, Khoa Luu, Samee U. Khan. Quantum Vision Clustering. Discover Internet of Things, Springer, 2025.

James B. Holliday, Darren Blount, Hoang-Quan Nguyen, Samee U. Khan, Khoa Luu. QUADRO: A Hybrid Quantum Optimization Framework for Drone Delivery. IEEE Quantum Week Conference, 2025.

Sankalp Pandey, Xuan Bac Nguyen, Nicholas Borys, Hugh Churchill, Khoa Luu. CLIFF: Continual Learning for Incremental Flake Features in 2D Material Identification. NeurIPS Workshop, 2025.

Hoang-Quan Nguyen, Xuan Bac Nguyen, Samuel Yen-Chi Chen, Hugh Churchill, Nicholas Borys, Samee U. Khan, Khoa Luu. Diffusion-Inspired Quantum Noise Mitigation in Parameterized Quantum Circuits. Journal of Quantum Machine Intelligence, Springer, 2025.

Xuan Bac Nguyen, Hoang-Quan Nguyen, Hugh Churchill, Samee U. Khan, Khoa Luu. Hierarchical Quantum Control Gates for Functional MRI Understanding. IEEE Workshop on Signal Processing Systems (SiPS), 2024.

Xuan Bac Nguyen, Hoang-Quan Nguyen, Hugh Churchill, Samee U. Khan, Khoa Luu. Quantum Visual Feature Encoding Revisited. Journal of Quantum Machine Intelligence, 2024.

Xuan Bac Nguyen, Hoang-Quan Nguyen, Samuel Yen-Chi Chen, Samee U. Khan, Hugh Churchill, Khoa Luu. QClusformer: A Quantum Transformer-based Framework for Unsupervised Visual Clustering. IEEE Quantum Week Workshop, 2024.

James B. Holliday, B. Morgan, Hugh Churchill, Khoa Luu. Hybrid Quantum Tabu Search for Solving the Vehicle Routing Problem. IEEE Quantum Week Workshop, 2024.

Xuan Bac Nguyen, A. Bisht, B. Thompson, Hugh Churchill, Khoa Luu, Samee U. Khan. Two-dimensional quantum material identification via self-attention and soft-labeling in deep learning. IEEE Access, Vol. 12, 2024.

Acknowledgements

This work is partly supported by MonArk NSF Quantum Foundry, supported by the National Science Foundation QAMASE- i program under NSF award No. DMR-1906383. It acknowledges the Arkansas High-Performance Computing Center for providing GPUs.