We Ingested All of Wikipedia. Here's What We Learned About Ideas.

At aiidea, we believe something radical: ideas can be inventoried. Not just documents. Not keywords. Actual, human-level ideas—grouped, tracked, and understood at scale.

To test that belief, we took on a challenge few attempt and even fewer finish:

We ingested the entirety of Wikipedia—169 million sentences—into our semantic pipeline.

But this wasn't about search. It was about structure. We wanted to know:

Can language be organized into reusable, traceable units of meaning—the way dictionaries organize words?

The answer? Yes. But it took Zipf, entropy, and a few million paraphrases to get there.

Sentences Obey Zipf's Law, Too

Most NLP systems are designed around words, because they're easy to tokenize and embed. But we deal in sentences, because that's where ideas live.

When we processed Wikipedia, we found something striking:

Sentences behave just like words in large corpora. They follow Zipf's Law: a small number of sentences appear frequently, while the long tail contains millions of once-seen fragments.

And like words, many of those rare sentences are actually just paraphrases of more common ones.

For example:

• "Photosynthesis converts sunlight into energy."
• "Plants use sunlight to produce food through photosynthesis."
• "Sunlight powers photosynthesis in plants."

Same idea. Different phrasing. Different frequencies. To a search engine, they're three strings. To aiidea, they're one node.

Paraphrase Detection at Scale: Not a Model, an Ensemble

To resolve this flood of sentences into idea nodes, we couldn't rely on one model. Every model has biases—some over-cluster, others under-cluster.

So we built an ensemble pipeline, powered by:

• Multiple sentence embedding models (different architectures, different lenses)
• Entropy and variance analysis to detect semantic drift
• Multi-model voting to decide paraphrase groupings
• Dynamic clustering (HDBSCAN, FISHDBC) to form meaning-stable nodes

Every decision was logged, every edge traceable. Because if we were going to claim that ideas could be inventoried, we had to prove it—not guess it.

From Sentences to Structure: Inventorying Ideas

Once the clusters were formed, something magical happened.

Each node wasn't just a data point—it was a semantic object. It had:

• A centroid vector
• A summary sentence
• Source attribution
• Popularity metrics
• Connected objects and ontology tags

This meant we could store ideas—not just search them. We could:

• Count them
• Rank them
• Visualize their growth
• Watch them evolve across topics, time, and documents

We had built an idea inventory system, where every new sentence could either reinforce an existing node or create a new one—just like new words enter a dictionary.

The Graph is the Context Window

Throughout the process, we obsessed over one thing: context. In language models, context is constrained by tokens. In aiidea, context is structural. It's relational. It's graph-native.

Our context window isn't a few thousand tokens. It's the diameter of the graph.

When you ask a question, we don't just match it semantically—we trace its implications through a web of connected ideas, sources, and object references. That's how we preserve nuance, relevance, and truth.

The graph isn't just data. It's memory. It's meaning. It's context that scales.

Why It Matters

This wasn't just a tech stunt. It was a proof of principle.

We did it to show that:

• Unstructured data isn't chaotic—it's latent structure waiting to be revealed
• Ideas are reusable units, not one-off strings
• Understanding isn't about better models—it's about better architecture
• Context isn't a sliding window—it's a semantic web

Ingesting Wikipedia gave us the training ground. Now we're scaling it to new domains, new languages, and new frontiers of meaning.

Because if ideas can be inventoried, then AI doesn't have to be a black box. It can be a semantic engine—and we just built the ignition system.