Knowledge arbitrage in a pre-AGI world
Paving the potholes in human research
The frontier of human knowledge isn’t smooth. I see it more as a jagged boundary, full of gaps and missed connections. When something new is discovered, it doesn’t instantly ripple through every field. In medicine, for example, it takes about 17 years for research to become routine clinical practice. And that includes a 6-year wait just to make it into textbooks and reviews. Other fields move faster, but not by much. Most researchers stick to a narrow set of journals and conferences, so anything outside that bubble might as well not exist. There simply isn’t enough time for them to sift through the thousands of new papers to find what’s actually relevant.
Right now, the way knowledge spreads is flawed. Journals have long, messy review processes. The papers that do get published are often locked behind paywalls. So some researchers turn to social media to share their work, turning a meritocratic process into a popularity contest. Those with the most eye-catching titles or the largest followings tend to attract the most attention. And that means quieter, potentially more important work might not get seen. It's hard to know how many problems could already be solved if ideas flowed more freely between disciplines.
Still, the value of interdisciplinary work is obvious. Some of the biggest scientific breakthroughs happened when someone brought an idea from one field into another. Physics gave us nuclear magnetic resonance, which later became the MRI machine. Brownian motion, another physics concept, helped shape the Black-Scholes model in finance. Even recently, computer scientists have won Nobel Prizes in chemistry and physics for their contributions, which sparked controversy because it blurred traditional academic lines. But should progress be limited by how many biophysicists or computational chemists we happen to have?
In a perfect world, knowledge would spread at light speed, like information in financial markets. In finance, arbitrage is when traders spot something that's undervalued in one market and overvalued in another, and capitalize on that difference. By quickly buying in one place and selling in another, they help prices align, keeping global markets in sync (and making lots of money in the process). Now imagine researchers doing something similar with ideas. When they notice that one field is missing something another field has already figured out, they can bring that knowledge over and close the gap. That’s a form of knowledge arbitrage, and it’s incredibly powerful—but far too rare.
Most interdisciplinary research doesn’t make headlines. It’s often labeled as “incremental” and quietly dismissed. But that label has real consequences: projects deemed not groundbreaking can struggle to get published or secure funding. Faced with those odds, many researchers opt not to pursue cross-disciplinary work at all. And when someone does manage to make a major leap across fields, it’s more likely to be seen as a fluke than a model worth following.
But what happens when we build tools that don’t care about prestige? What happens when an AI can read thousands of research papers at once, spanning every field? That kind of system could connect dots humans would never think to link. A discovery in physics might spark new ideas in chip design, drug development, or even astronomy, instantly. Even if today’s AI systems are largely rehashing patterns from their training data, a behavior some critics refer to as "stochastic parrots," they can still provide meaningful value. By interpolating across vast amounts of human knowledge, they help fill in the blanks, reinforce what we already know, and surface connections we might have otherwise missed.
We’re already seeing signs of this. Google is working on an AI “co-scientist” that comes up with and tests new research ideas. Other tools, like Deep Research, let regular people synthesize findings from hundreds of sources in minutes. Remember, these are just the tools that have been enabled by today’s models. As context lengths and intelligence benchmark scores increase, AI will become one of the most powerful polymaths we’ve ever known.
Long before we get to expert-level AGI, we’ll start seeing AI make a real difference. By doing the grunt work of reading thousands of papers every day, it’ll keep surfacing useful insights from one field to another, helping connect areas of research that used to stand completely apart. In the process, it’ll begin to smooth the jagged frontier of human knowledge, paving paths between disciplines that once stood isolated.
Update: Turns out there’s way more existing work in this field than I originally thought. I detailed my additional findings on literature-based discovery in this thread.
If you’re also excited by, working on or want to work on AI for knowledge discovery, shoot me an email at hi@jinay.dev—I’d love to chat!