From Documents to Knowledge Objects: Rethinking AI Data Ingestion
In the previous posts, I introduced why I started building AIKOQL, what a Knowledge Object is, and where I believe AIKOQL fits alongside relational databases, document stores, vector databases, and graphs.
But there is a practical question behind all of this:
Where do Knowledge Objects come from?
For most AI systems today, the answer starts with ingestion.
Documents are uploaded.
Text is extracted.
The content is split into chunks.
Embeddings are generated.
The chunks are stored in a vector database.
And when a user asks a question, the system retrieves the most similar chunks.
This approach has made Retrieval-Augmented Generation possible at scale.
It is also useful.
But I believe there is an important problem hiding inside this pipeline.
A document is not just text.
And if we destroy its structure during ingestion, we may also destroy some of the meaning required to turn information into knowledge.
That is one of the problems AIKOQL is exploring.
The traditional AI ingestion pipeline
A simplified ingestion pipeline often looks like this:
Document
↓
Extract Text
↓
Split Into Chunks
↓
Generate Embeddings
↓
Vector Database
↓
Retrieve Top K Chunks
↓
LLM
This approach works surprisingly well for many use cases.
But consider what can happen during the process.
Imagine a technical document containing:
Headings
Paragraphs
Tables
Diagrams
Charts
Images
Formulas
Code
References
Footnotes
After text extraction and chunking, much of that structure may disappear.
A table becomes text.
A diagram becomes a caption.
A chart becomes an image with little or no semantic representation.
A formula becomes an isolated string.
A heading may become separated from the content it describes.
A relationship between two sections may disappear.
The result is still searchable.
But is it still the same knowledge?
That is the question.
Documents are more than text
Consider a simple architecture document.
It contains the statement:
The Payment Service depends on the Transaction Database.
That statement might appear inside a diagram.
The diagram might show:
Payment Service
│
│ DEPENDS ON
▼
Transaction Database
If we flatten the document into plain text, we might end up with something like:
Payment Service Transaction Database
The entities survive.
The relationship may not.
Now imagine a table:
| Service | Owner | SLA |
|---|---|---|
| Payment Service | Payments Team | 99.9% |
If the table is split across chunks, the relationship between:
Payment Service
and:
99.9%
can become ambiguous.
The value still exists.
But its meaning depends on its position inside the table.
This is one of the fundamental problems with treating all documents as flat text.
Meaning often comes from structure.
The problem with arbitrary chunking
Chunking is one of the most common techniques used in AI data ingestion.
But chunk boundaries are often determined by:
Character limits
Token limits
Fixed sizes
Sliding windows
For example:
Chunk 1
──────────────
The Payment Service handles customer transactions.
It communicates with the Transaction Database.
Chunk 2
──────────────
Database writes are replicated across regions.
The service must retry failed transactions.
This may work.
But now consider:
Chunk 1
──────────────
The Payment Service handles customer transactions.
It communicates with the Transaction...
Chunk 2
──────────────
Database. Writes are replicated across regions.
The semantic boundary has been broken.
The chunking algorithm successfully created two chunks.
But it may have failed to preserve knowledge.
This is the distinction I think is important:
Chunking optimizes for retrieval size. Knowledge extraction should optimize for meaning.
Those are not always the same thing.
AIKOQL starts with structure
The direction I am exploring in AIKOQL is to treat ingestion as a process of progressively understanding the source.
Instead of:
Document
↓
Text
↓
Chunks
The conceptual pipeline becomes:
Source
↓
Structured Parsing
↓
Knowledge Fragments
↓
Semantic Analysis
↓
Knowledge Candidates
↓
Knowledge Objects
The important difference is that the system should try to preserve structure before deciding how information should be represented for retrieval or reasoning.
Step 1: Understand the source
AI systems do not receive information from only one format.
Knowledge may exist in:
PDF documents
DOCX files
Markdown
HTML pages
Tables
Images
Diagrams
Charts
Formulas
APIs
Databases
Events
Logs
External systems
These sources have different structures.
A table is not a paragraph.
A chart is not an image.
A diagram is not simply decorative content.
An API response is not the same as a document.
The first question during ingestion should therefore be:
What kind of information are we looking at?
Before extracting meaning, the system should understand the boundaries of the source.
Conceptually:
Source
↓
Identify Format
├── Document
├── Table
├── Diagram
├── Chart
├── Image
├── Formula
├── API
└── Event
This is where structured parsing becomes important.
Step 2: Build a structural representation
A document has an internal structure.
For example:
Document
│
├── Title
│
├── Section: Architecture
│ │
│ ├── Paragraph
│ │
│ ├── Diagram
│ │
│ └── Table
│
├── Section: Deployment
│ │
│ ├── Paragraph
│ │
│ └── Configuration
│
└── References
Representing this structure allows the ingestion system to understand boundaries.
This can be thought of as a document tree or an abstract syntax representation of the source.
Conceptually:
Document AST
│
├── Section
│ ├── Paragraph
│ ├── Table
│ └── Diagram
│
├── Section
│ ├── Paragraph
│ └── Formula
│
└── Metadata
The exact implementation can evolve.
But the principle is important:
Do not flatten structure before you understand it.
Once structure is destroyed, it is difficult to reconstruct reliably.
Step 3: Create Knowledge Fragments
After understanding the structure of the source, AIKOQL can treat meaningful parts of the source as different kinds of fragments.
For example:
Knowledge Fragments
│
├── Text Fragment
│
├── Table Fragment
│
├── Diagram Fragment
│
├── Chart Fragment
│
├── Image Fragment
│
└── Formula Fragment
These fragments are not necessarily final Knowledge Objects.
They are meaningful units extracted from the original source.
That distinction is important.
A fragment answers:
What meaningful part of the source did we find?
A Knowledge Object answers:
What knowledge does this information represent?
There may be a transformation between the two.
Text Fragments
A text fragment might contain:
The Payment Service processes customer transactions.
But the fragment should ideally retain context.
For example:
Document:
Payments Architecture
Section:
Transaction Processing
Heading:
Payment Service
Position:
Paragraph 3
Now the system has more than text.
It has context.
The same sentence can mean different things depending on where it appeared.
Table Fragments
Tables contain relationships that are often lost when converted into plain text.
Consider:
| Service | Owner | SLA |
|---|---|---|
| Payment Service | Payments Team | 99.9% |
A table fragment should preserve:
Table
│
├── Columns
│ ├── Service
│ ├── Owner
│ └── SLA
│
└── Rows
└── Payment Service
├── Payments Team
└── 99.9%
The relationship between the values is part of the knowledge.
If we only store:
Payment Service
Payments Team
99.9%
the information survives.
But the structure that explains the information may not.
Diagram Fragments
Diagrams are particularly interesting for AI systems.
A diagram can represent:
Dependencies
Relationships
Architecture
Processes
Data flows
Hierarchies
Consider:
Application
│
▼
API Gateway
│
▼
Payment Service
│
▼
Transaction Database
The diagram does not only contain entity names.
It contains relationships.
The knowledge is closer to:
Application
│
└── communicates through
│
▼
API Gateway
│
└── routes to
│
▼
Payment Service
│
└── stores data in
│
▼
Transaction Database
This is a fundamentally richer representation.
For AIKOQL, diagrams are therefore interesting not simply as images to describe, but as potential sources of entities and relationships.
Chart Fragments
Charts contain information that often disappears during traditional text extraction.
A chart may represent:
Trends
Changes
Comparisons
Thresholds
Events
Correlations
Imagine a chart showing:
CPU Usage
90% ────────────────▲
│ Deployment
│
40% ────────────────┘
The raw image contains information.
But the knowledge may be:
CPU utilization increased significantly after the deployment.
That is not necessarily written anywhere in the document.
It is represented visually.
A knowledge ingestion system should eventually be able to distinguish between:
The image exists.
and:
The image represents information.
This is why multimodal ingestion becomes important.
Formula Fragments
Technical and scientific knowledge often exists inside formulas.
For example:
Revenue = Price × Quantity
A formula is not simply a sequence of characters.
It represents:
Variables
Relationships
Operations
Constraints
Flattening the formula into plain text may make it searchable.
But semantic understanding requires more.
The same principle applies to:
Mathematical models
Engineering formulas
Financial equations
Scientific expressions
For AI systems that work with technical knowledge, preserving these structures may become increasingly important.
Step 4: Preserve provenance
One of the principles I consider essential for AIKOQL is provenance.
If an AI system retrieves a piece of knowledge, it should ideally be possible to answer:
Where did this come from?
For a document, provenance might look like:
Knowledge Object
│
▼
Source Document
│
▼
Section
│
▼
Element
│
├── Paragraph
├── Table
├── Diagram
└── Chart
│
▼
Knowledge Fragment
│
▼
Knowledge Object
This creates a chain.
Conceptually:
Knowledge Object
│
▼
Knowledge Fragment
│
▼
Source Element
│
▼
Document Section
│
▼
Original Source
This means a piece of knowledge does not become detached from its origin during ingestion.
That is important for several reasons.
Trust
If an AI says:
"The Payment Service has a 99.9% SLA."
We should be able to ask:
Where did that information come from?
Verification
A user should be able to inspect the original source.
Updates
If the original document changes, the system should be able to understand which knowledge may be affected.
Explainability
An AI agent should eventually be able to explain:
I reached this conclusion based on these sources.
Knowledge lineage
Derived knowledge should not become disconnected from the information that produced it.
Step 5: Semantic analysis
After the structure has been preserved, the next challenge is understanding meaning.
This is where semantic analysis begins.
The system can look for:
Entities
Concepts
Relationships
Events
Attributes
Measurements
Changes
Dependencies
For example, consider:
The Payment Service depends on the Transaction Database.
Semantic analysis might identify:
Entity:
Payment Service
Relationship:
DEPENDS_ON
Entity:
Transaction Database
Now consider:
The database migration was completed on June 12.
The system may identify:
Event:
Database Migration
Status:
Completed
Time:
June 12
The important point is that semantic analysis should operate on meaningful fragments that still retain their original context.
Otherwise, we risk extracting knowledge from text while losing the structure that explains what the text actually means.
From fragments to Knowledge Objects
At this point, the ingestion pipeline can start producing knowledge candidates.
For example:
Knowledge Candidate
Subject:
Payment Service
Relationship:
DEPENDS_ON
Object:
Transaction Database
Source:
Architecture Document
Evidence:
Diagram
Confidence:
0.87
This is still not necessarily final knowledge.
The system may need to:
Validate the extraction
Compare it with other knowledge
Detect contradictions
Attach evidence
Determine authority
Assign an epistemic state
Eventually, the candidate may become a Knowledge Object.
Conceptually:
Knowledge Fragment
│
▼
Semantic Analysis
│
▼
Knowledge Candidate
│
├── Validate
├── Enrich
├── Connect
├── Compare
└── Verify
│
▼
Knowledge Object
This distinction is important because AI-generated extraction should not automatically become trusted knowledge.
An LLM may extract something incorrectly.
A source may be outdated.
Two documents may contradict each other.
The ingestion pipeline should preserve uncertainty rather than pretending that extraction is always correct.
Knowledge creation is not the same as text extraction
This is perhaps the most important distinction in the entire pipeline.
Traditional ingestion often asks:
Can we extract the content?
AIKOQL is exploring a different question:
Can we preserve enough structure and context to understand the content as knowledge?
These are not the same problem.
Consider:
Text Extraction
Document
↓
Text
Compared with:
Knowledge Extraction
Document
↓
Structure
↓
Meaningful Fragments
↓
Context
↓
Semantic Analysis
↓
Evidence
↓
Knowledge
The second pipeline is more complex.
But knowledge is more complex than text.
Embeddings still matter
This does not mean embeddings or vector databases become irrelevant.
They remain extremely useful.
Embeddings can help with:
Semantic search
Similarity
Clustering
Retrieval
Knowledge discovery
But in this model, embeddings become one representation of knowledge rather than the knowledge itself.
Conceptually:
Knowledge Object
│
├── Content
│
├── Provenance
│
├── Relationships
│
├── Temporal Context
│
├── Knowledge State
│
└── Semantic Representation
│
▼
Embedding
This is an important architectural distinction.
A vector can help us find knowledge.
But the Knowledge Object provides the context required to understand what was found.
The ingestion pipeline should preserve meaning
The direction I am exploring can be summarized in one principle:
Preserve meaning before optimizing retrieval.
That means:
Preserve structure
Do not flatten tables, diagrams, and documents too early.
Preserve boundaries
A meaningful section should not necessarily be split because it exceeds an arbitrary token count.
Preserve provenance
Every piece of extracted knowledge should remain connected to its source.
Preserve context
Headings, sections, relationships, and surrounding information may affect meaning.
Preserve modality
Text, tables, diagrams, images, and formulas should not all be treated as identical inputs.
Preserve uncertainty
Extracted information should not automatically be considered verified knowledge.
Why this matters for AI agents
AI agents increasingly need more than search results.
They need context.
Imagine an agent receives:
The Transaction Database is PostgreSQL.
Without context, the agent may assume it is true.
But what if the Knowledge Object tells the agent:
Knowledge:
The Transaction Database is PostgreSQL.
Status:
Superseded
Valid Until:
June 2026
Source:
Architecture Document v1
Replaced By:
Knowledge Object #247
Now the agent can make a better decision.
Similarly:
Knowledge:
The Payment Service depends on the Transaction Database.
Source:
Architecture Diagram
Evidence:
System Design Document
Confidence:
High
Current State:
Verified
The agent has more than text.
It has knowledge with context.
That is the outcome I am exploring with AIKOQL.
The real challenge: knowing what not to flatten
There is no universal rule that says:
Every table should become a Knowledge Object.
Or:
Every paragraph should become a Knowledge Object.
Or:
Every diagram should become a graph.
The ingestion pipeline needs to make decisions.
Some information may become:
A Knowledge Object
A Knowledge Fragment
Evidence
Metadata
A semantic representation
A relationship
A historical record
This is where the problem becomes difficult.
And interesting.
The ingestion system is not simply converting one format into another.
It is deciding:
What parts of the source represent meaningful knowledge?
That is one of the core engineering challenges behind AIKOQL.
From raw data to knowledge
The complete conceptual pipeline currently looks something like this:
RAW DATA
┌─────────────────────────────────────┐
│ │
│ Documents Tables Diagrams │
│ │
│ Charts Images Formulas │
│ │
│ APIs Databases Events │
│ │
└──────────────────┬──────────────────┘
│
▼
STRUCTURED PARSING
│
▼
KNOWLEDGE FRAGMENTS
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Text Tables Diagrams
│ │ │
└─────────────┼─────────────┘
│
▼
SEMANTIC ANALYSIS
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
Entities Relationships Events
│
▼
KNOWLEDGE CANDIDATES
│
▼
VALIDATION & ENRICHMENT
│
▼
KNOWLEDGE OBJECTS
┌───────────────┼────────────────┐
│ │ │
▼ ▼ ▼
Provenance Relationships Evidence
│ │ │
└───────────────┼────────────────┘
│
▼
AI-NATIVE KNOWLEDGE
│
┌───────────┼───────────┐
│ │ │
▼ ▼ ▼
Retrieval Reasoning Agents
This is the bridge I believe AIKOQL is trying to build.
From raw data.
To structured information.
From information.
To contextual knowledge.
And from knowledge.
To AI systems that can reason, retrieve, remember, and explain.
The challenge ahead
Of course, this is not easy.
A pipeline like this introduces difficult questions.
How accurately can we extract relationships from diagrams?
How do we represent ambiguous information?
How should conflicting sources be handled?
When should a Knowledge Candidate become trusted knowledge?
How should confidence be calculated?
How do we preserve provenance without creating excessive storage overhead?
How do we process large documents efficiently?
How do we update knowledge when the original source changes?
How do we prevent an incorrect AI extraction from contaminating the knowledge base?
These are not problems I believe have a single perfect answer.
They are engineering problems that need experimentation, testing, and evidence.
And that is exactly why I am building AIKOQL in the open.
The bigger idea
The traditional AI pipeline often focuses on this:
How quickly can we retrieve relevant text?
AIKOQL is exploring a broader question:
How do we preserve and operate on knowledge without losing the context that makes information meaningful?
That means thinking beyond:
Chunk size
Embedding models
Top K retrieval
And starting to think about:
Structure
Meaning
Provenance
Evidence
Context
Relationships
Time
Knowledge state
Because an AI system should not simply retrieve information.
Ideally, it should understand what that information represents.
What comes next
In the next post, I want to go deeper into the infrastructure behind these ideas:
Building AIKOQL in Rust: The Architecture Behind an AI-Native Knowledge Engine
Because a Knowledge Object model and ingestion pipeline are only useful if the underlying system can:
Store knowledge efficiently
Preserve history
Manage relationships
Support transactions
Handle concurrent workloads
Execute queries
Maintain provenance
Evolve without losing existing knowledge
That brings us to the systems engineering side of AIKOQL.
The abstractions are important.
But eventually, every abstraction has to survive contact with storage engines, memory, concurrency, transactions, and real workloads.
And that is where the next part of the journey begins.
AIKOQL is an evolving open project. The ingestion architecture and concepts described here are actively being developed and tested. The implementation may evolve as the system encounters real documents, multimodal data, performance constraints, and production workloads.
If you work with RAG, document processing, vector databases, multimodal AI, knowledge graphs, or AI agents, I would love your perspective.
Do you think traditional chunking is sufficient for AI knowledge systems, or do we need ingestion pipelines that preserve structure and provenance as first-class concepts?
