Harsh Raj

I am an M.S. student in Computer Science at Northeastern University, Boston, and currently a Machine Learning Research Intern at Scale AI in New York.

My research centers on language agents, language model evaluation, and software engineering: building systems that are not just capable, but reliable and reproducible.

Previously, I led the MixtureVitae project on permissively licensed pretraining corpora. I am a core author of Terminal-Bench and Harbor, and collaborate with Ludwig Schmidt's lab.

Email CV Google Scholar GitHub Twitter

News

Sep 2026Released Root-Cause Attribution Is a Search Problem.
Sep 2026Three papers accepted to NeurIPS 2026, and one to the NeurIPS 2026 AI4Good Workshop.
Sep 2026Model or Harness? was featured by DAIR.AI and alphaXiv.
Jun 2026Reviewer for the NeurIPS 2026 Datasets & Benchmarks (Evaluations) Track.
Jun 2026Released OpenThoughts-Agent, an open data recipe for agentic post-training.
May 2026Started as a Machine Learning Research Intern at Scale AI, New York.
Mar 2026MixtureVitae accepted at TMLR with Featured and Journal-to-Conference Certifications; presented at ICML 2026, Seoul.
Jan 2026Research Intern at Bespoke Labs for Spring 2026.

Older news

Nov 2025Released Harbor, a framework for parallel agent-sandbox execution for training and evaluating agents.
Aug 2025Released MixtureVitae, a permissively licensed pretraining corpus (paper).
Jun 2025Led a Swiss AI Large Grant proposal under Robert West, securing CHF 61K and 200K GPU hours on the Alps supercomputer.
May 2025Released Terminal-Bench.
Apr 2025Research Assistant at the Interpretable Neural Networks Lab, advised by David Bau.
Jan 2025Chain of Guidance accepted at TMLR.
Dec 2024Mitigating Unsafe Feedback with Learning Constraints accepted at the AAAI-25 Workshop on AI for Cyber Security.
Sep 2024Wrote Interpreting the effects of jailbreak prompts in LLMs on LessWrong.
Jan 2024Joined Vijil as an Applied Scientist.
Jan 2024Ranked 3rd on the R2R vision-and-language navigation leaderboard.
Dec 2023Presented work on robustness transfer at NeurIPS 2023, New Orleans.
Nov 2022Our work on consistency evaluation won the Best Paper Award at the NeurIPS ML Safety Workshop.

Publications

* denotes equal contribution. Highlighted papers are ones I led.

	Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures Harsh Raj, David Lee, Anas Mahmoud, Renxiong Wang, Razvan-Gabriel Dumitru, Chenguang Wang, Tong Zhao, Yunzhong He, Darvin Yi, Vipul Gupta arXiv, 2026 paper
	Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures Harsh Raj, Vipul Gupta, Anas Mahmoud, Razvan-Gabriel Dumitru, Darvin Yi, Aakash Sabharwal, Yunzhong He arXiv, 2026 paper alphaXiv
	Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation Lin Shi, Haowei Lin, Zixuan Zhu, ..., Harsh Raj, ..., Mike Merrill, Ludwig Schmidt, Alex Shaw NeurIPS 2026 paper
	Strong Post-Training from Permissive, Reasoning-Dominant, Web-Scale Pretraining Harsh Raj, Ali Elganzory, Marianna Nezhurina, Victor May, Van Khue Nguyen, David Salinas, Huu Nguyen, Jenia Jitsev NeurIPS 2026
	Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability Harsh Raj, Niranjan Orkat, Suvrorup Mukherjee, Aritra Guha, Cheryl Flynn, Subhabrata Majumdar NeurIPS 2026 paper
	Synthetic Persona Pretraining: Alignment from Token Zero Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Andy Arditi, Shaobo Cui, Yiderigun Borjigin, Kartik Bali, Stefan Krsteski, Harsh Raj, Huu Nguyen, Jannik Brinkmann, Ashton Anderson, Roland Aydin, Robert West NeurIPS 2026 AI4Good Workshop paper
	OpenThoughts-Agent: Data Recipes for Agentic Models Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, ..., Harsh Raj, et al. arXiv, 2026 paper
	MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources Huu Nguyen, Victor May, Harsh Raj*, Marianna Nezhurina, Yishan Wang, ..., Jenia Jitsev TMLR, 2026 Featured Certification; presented at ICML 2026 paper code
	Improving Consistency in Large Language Models through Chain of Guidance Harsh Raj, Vipul Gupta, Domenic Rosati, Subho Majumdar TMLR, 2025 paper code
	Mitigating Unsafe Feedback with Learning Constraints Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad AAAI-25 Workshop on AI for Cyber Security paper
	On transfer of adversarial robustness from pretraining to downstream tasks Laura F. Nern, Harsh Raj, Maurice Georgi, Yash Sharma NeurIPS 2023 paper code
	Evaluating the robustness of biomedical concept normalization Sinchani Chakraborty, Harsh Raj, Srishti Gureja, Tanmay Jain, Atif Hassan, Sayantan Basu NeurIPS 2023 paper code
	Measuring reliability of large language models through semantic consistency Harsh Raj, Domenic Rosati, Subho Majumdar NeurIPS 2023 Best Paper Award, NeurIPS ML Safety Workshop paper code
	GANDALF: Gated Adaptive Network for Deep Automated Learning of Features Manu Joseph, Harsh Raj arXiv, 2022 paper code
	AskYourDB: An end-to-end system for querying and visualizing relational databases using natural language Manu Joseph, Harsh Raj, Anubhav Yadav, Aaryamann Sharma arXiv, 2022 paper