I am Harsh Raj, a M.S. student in Computer Science at Northeastern University, Boston, and currently a Machine Learning Research Intern at Scale AI in NYC.

My research centers on language agents, language model evaluation, and software engineering — building systems that are not just capable, but reliable and reproducible. Over the past few years I have been fortunate to grow as a researcher with the support of generous collaborators and mentors.

Most recently, I led the MixtureVitae project, developing state-of-the-art pretraining corpora for LLMs. Our work produced the first permissible dataset achieving results comparable to or even surpassing non-permissible sources (paper).

In addition, I am the core author of Terminal-Bench and Harbor and collaborate with Professor Ludwig Schmidt’s lab.

News and Timeline

2026

• June: Serving as a reviewer for the NeurIPS 2026 Datasets & Benchmarks (Evaluations) Track.
• June: Released OpenThoughts-Agent: Data Recipes for Agentic Models, an open data recipe for agentic post-training.
• May: Started as a Machine Learning Research Intern at Scale AI, NYC.
• May: Released Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability.
• January: Started as a Research Intern for Spring 2026 at Bespoke Labs.

2025

• November: Released Harbor for massive parallelization in agent-sandbox execution used for training and evaluating AI agents.
• August: Released MixtureVitae, the first permissibly licensed pretraining corpus to achieve performance comparable to or even surpassing some non-permissive sources (paper).
• June: Led the Swiss AI Large Grant proposal under Prof. Robert West. Secured CHF 61K and 200K GPU hours on the Alps Supercomputer.
• May: Released terminal-bench.
• April: Started as a Research Assistant at the Interpretable Neural Networks Lab advised by Prof. David Bau.
• January: Our work on Improving Consistency in Large Language Models through Chain of Guidance was accepted at Transactions of Machine Learning Research (TMLR).

2024

• December: Our work on Mitigating Unsafe Feedback with Learning Constraints got accepted for poster presentation at AAAI-25 Workshop on Artificial Intelligence for Cyber Security.
• September: Published my first Lesswrong blog on Interpreting the effects of Jailbreaking in LLMs.
• June: Released the preprint of our work on Reverse Preference Attack, led by Domenic.
• January: Joined VijilAI as an Applied Scientist.
• January: Our work on Vision and Language Navigation ranked 3rd on the R2R leaderboard. Team Name: MLR_Lab_DTU.

2023

• December: Presented our work on robustness transfer at NeurIPS 2023 in New Orleans.
• May: Our work on robustness transfer accepted to NeurIPS 2023. Led by Laura and mentored by Yash.

2022

• November: Our work on consistency evaluation won the Best Paper Award with a cash prize of $5000.
• April: Two papers accepted to NeurIPS Workshop 2023: one on consistency evaluation and another on evaluating the robustness of biomedical concept normalization.