Distributed systems is simply the study of interactions between processes. Every two interacting processes form a distributed system, whether they are on the same host or not. Distributed systems create new challenges (compared to single-process systems) in terms of correctness (i.e. consistency), reliability, and performance (i.e. latency and throughput).

The best way to learn about the principles and fundamentals of distributed systems is to 1) read Designing Data Intensive Applications and 2) read through the papers and follow the notes in the MIT Distributed Systems course.

For Designing Data Intensive Applications (DDIA), I strongly encourage you to find buddies at work or online who will read it through with you. You can also always join the Software Internals Discord's #distsys channel to ask questions as you go. But it's still best if you have some partners to go through the book with, even if they are as new to it as you.

I also used to think that you might want to wait a few years into your career before reading DDIA but when you have friends to read it with I think you need not wait.

If you have only skimmed the book you should definitely go back and give it a thorough read. I have read it three times already and I will read it again as part of the Software Internals Book Club next year after the 2nd Edition is published.

Keep in mind that every chapter of DDIA provides references to papers you can keep reading should you end up memorizing DDIA itself.

When you've read parts of DDIA or the MIT Distributed Systems course and you want practice, the Fly.io x Jepsen Distributed Systems Challenge is one guided option. Other options might include simply implementing (in somewhat ascending complexity):

• two-phase commit
• three-phase commit
• single-decree Paxos
• chain replication (or CRAQ), using a 3rd-party consensus library
• Raft
• epaxos

And if you get bored there you can see Alex Miller's Data Replication Design Spectrum for more ideas and variants.

If these projects and papers sound arcane or intimidating, know that you will see the problems these projects/papers solve whether or not you know and understand these solutions. Developers often end up reinventing hacky versions of these which are more likely to have subtle bugs, while instead you can recognize and use one of these well-known building blocks. Or at least have the background to better reason about correctness should you be in a situation where you must work with a novel distributed system or you end up designing a new one yourself.

And again, if you want folks to bounce ideas off of or ask questions to, I strongly encourage you to join the Software Internals Discord and ask there!

Working with nested data in MongoDB simplifies mapping between application objects and database structures. However, challenges can arise when grouping or joining values within sub-document arrays, particularly for developers shifting from SQL databases with normalized data where the result is always flattened to tabular result.

I'll go through an example with the following collection, and link to MongoDB Playground for each explanation.

[
  {
    "projectName": "Troubleshooting PostgreSQL issues",
    "team": [
      { "memberId": "cyclops", "name": "Cyclops", "role": "Postgres Expert" },
      { "memberId": "wolverine", "name": "Wolverine", "role": "Consultant" },
      { "memberId": "storm", "name": "Storm", "role": "DBA" },
      { "memberId": "beast", "name": "Beast", "role": "Developer" },
      { "memberId": "tony", "name": "Tony", "role": "Architect" }
    ],
    "status": "active"
  },
  {
    "projectName": "Build new apps with MongoDB",
    "team": [
      { "memberId": "tony", "name": "Tony", "role": "Developer" }
    ],
    "status": "planned"
  }
]

Suppose you want a report of which projects each person was involved in, and their role(s) for each.

In PostgreSQL, you’d have normalized to three tables because of the Many-to-Many relationship. The join returns one row per project and team member, with information about projects and members duplicated. You can then aggregate them per team member to get the projects per person as an array:

SELECT    
  tm.member_id,    
  tm.name,    
  ARRAY_AGG(p.project_name ORDER BY p.project_id) AS projects    
FROM    
  team_members tm    
  JOIN project_team pt ON tm.member_id = pt.member_id    
  JOIN projects p ON pt.project_id = p.project_id    
GROUP BY    
  tm.member_id, tm.name    
ORDER BY    
  tm.member_id;    

Here's the example on db<>fiddle: https://dbfiddle.uk/FhsA9DpU

In MongoDB, relationships are embedded. I created a "team" array for project members. Let's explore how to aggregate it, starting with some common mistakes.

The Wrong Way: Grouping Directly on an Array

Suppose you try to group directly on the array field:

db.projects.aggregate([  
  { $group: { _id: "$team.memberId", projects: { $push: "$projectName" } } }  
])

What you get:

[
  {
    "_id": [
      "cyclops", "wolverine", "storm", "beast", "tony"
    ],
    "projects": ["Troubleshooting PostgreSQL issues"]
  },
  {
    "_id": [
      "tony"
    ],
    "projects": ["Build new apps with MongoDB"]
  }
]

Here's the example on MongoDB Playground: https://mongoplayground.net/p/bn7xsRMQhu2

What went wrong?

MongoDB used the whole array as the grouping key. Instead of grouping by each individual member, you get one group for each unique full-team combination.

Tip for SQL users:

In SQL, GROUP BY splits rows by scalar values, but in MongoDB, grouping on an array field groups by the entire array as a single value.

The Wrong Way: Lookup Directly on an Array

Similarly, you may try to enrich team data with $lookup on the array field:

{  
  $lookup: {  
    from: "members",  
    localField: "team.memberId",  
    foreignField: "memberId",  
    as: "memberInfo"  
  }  
}

Here's the example on MongoDB Playground:

https://mongoplayground.net/p/RVXoY5z7ke1

What you get:

• The entire array is used as the lookup key, so memberInfo will usually just be an empty array (no matches), unless a member’s memberId somehow matches an array, which it never will.

Key for SQL folks:

Unlike SQL, where the JOIN is applied per row value, MongoDB $lookup expects scalar fields, not arrays. If your join keys are inside an array, you need to flatten that array first.

The Right Way: Flatten with $unwind, Then $group

First, flatten out your team array so each person/project combination gets its own pipeline document:

{ $unwind: "$team" }

Here's how it looks like on MongoDB Playground:

https://mongoplayground.net/p/yGvvM-FZM5p

Then group by the member ID:

Don’t forget to include the role for each project. Here, we also collect all roles for completeness.

{
  $group: {
    _id: "$team.memberId",
    memberName: { $first: "$team.name" },
    roles: { $addToSet: "$team.role" },
    projects: {
      $push: {
        name: "$projectName",
        id: "$_id",
        role: "$team.role"
      }
    }
  }
}

The full working pipeline is like this:

db.projects.aggregate([
  { $unwind: "$team" },
  {
    $group: {
      _id: "$team.memberId",
      memberName: { $first: "$team.name" },
      roles: { $addToSet: "$team.role" },
      projects: {
        $push: {
          name: "$projectName",
          id: "$_id",
          role: "$team.role"
        }
      }
    }
  }
])

Here's the result on MongoDB Playground:

https://mongoplayground.net/p/oMOfvrZXa2a

This is the expected output, with one document per member and their project or array of projects:

[
  {
    "_id": "cyclops",
    "memberName": "Cyclops",
    "roles": ["Postgres Expert"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Postgres Expert" }
    ]
  },
  {
    "_id": "wolverine",
    "memberName": "Wolverine",
    "roles": ["Consultant"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Consultant" }
    ]
  },
  {
    "_id": "storm",
    "memberName": "Storm",
    "roles": ["DBA"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "DBA" }
    ]
  },
  {
    "_id": "beast",
    "memberName": "Beast",
    "roles": ["Developer"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Developer" }
    ]
  },
  {
    "_id": "tony",
    "memberName": "Tony",
    "roles": ["Architect", "Developer"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Architect" },
      { "name": "Build new apps with MongoDB", "id": ObjectId("60f...a2"), "role": "Developer" }
    ]
  }
]

Now you get the right result: Tony appears in both projects, and each is listed with the correct role.

Practical reminder:

MongoDB’s $unwind makes array data behave more like SQL’s flattened rows, so aggregation and joins work as SQL users expect.

Here is an example with $lookup: https://mongoplayground.net/p/I3__p2EE9ps

What About Filtering Instead of Grouping?

If you just want to filter out all but the developers in each project, no unwinding is needed:

db.projects.aggregate([
  {
    $addFields: {
      team: {
        $filter: {
          input: "$team",
          as: "member",
          cond: { $eq: ["$$member.role", "Developer"] }
        }
      }
    }
  }
])

Here's the MongoDB playground:

https://mongoplayground.net/p/bhDQJVKu7Rp

[
  {
    "_id": ObjectId("60f...a1"),
    "projectName": "Troubleshooting PostgreSQL issues",
    "team": [
      { "memberId": "beast", "name": "Beast", "role": "Developer" }
    ],
    "status": "active"
  },
  {
    "_id": ObjectId("60f...a2"),
    "projectName": "Build new apps with MongoDB",
    "team": [
      { "memberId": "tony", "name": "Tony", "role": "Developer" }
    ],
    "status": "planned"
  }
]

I used the following aggregation operators:

• $unwind: Flattens array fields, creating a separate document for each element in the array.
• $group: Similar to SQL’s GROUP BY, especially effective after arrays have been unwound.
• $lookup: Similar to a SQL LEFT OUTER JOIN, but by default MongoDB does not flatten joined results into multiple documents with repeated data. Instead, matched documents are returned normalized (0NF) as an array field.

SQL databases store their data normalized, often in third normal form, with multiple tables for one-to-many and many-to-many relationships to avoid duplicated data. The query result requires a join and is always in first normal form (1NF), as the result of a SQL query is a single flat table. Since 1NF cannot have arrays, SQL databases have to unnest groups, flattening relationships to multiple rows and introducing redundancy in a denormalized tabular result set.

MongoDB is not constrained by normal forms and supports rich document models, with arrays for repeating groups and nested objects directly in each document. When you use the aggregation pipeline, results can keep this nested structure. But if you want to group or join on values nested inside arrays, you’ll need to flatten the array to multiple documents using $unwind, so further aggregation stages work as expected. In practice, $lookup in MongoDB is often compared to JOINs in SQL, but if your fields live inside arrays, a join operation is really $unwind followed by $lookup.

Key takeaways for SQL users moving to MongoDB:

• Arrays are not automatically expanded or unnested.
• Whenever your “join key” or “group by value” is inside an array, always unwind first.
• In MongoDB, aggregation pipeline results can be nested and contain arrays, unlike the flat results of SQL queries.
• $unwind is the document-model equivalent of SQL’s UNNEST: it’s your bridge from nested arrays to flat, row-like documents for further aggregation.
• When joining with $lookup, always check whether your localField or foreignField are arrays and flatten as needed.

Working with nested data in MongoDB simplifies mapping between application objects and database structures. However, challenges can arise when grouping or joining values within sub-document arrays, particularly for developers shifting from SQL databases with normalized data where the result is always flattened to tabular result.

I'll go through an example with the following collection, and link to MongoDB Playground for each explanation.

[
  {
    "projectName": "Troubleshooting PostgreSQL issues",
    "team": [
      { "memberId": "cyclops", "name": "Cyclops", "role": "Postgres Expert" },
      { "memberId": "wolverine", "name": "Wolverine", "role": "Consultant" },
      { "memberId": "storm", "name": "Storm", "role": "DBA" },
      { "memberId": "beast", "name": "Beast", "role": "Developer" },
      { "memberId": "tony", "name": "Tony", "role": "Architect" }
    ],
    "status": "active"
  },
  {
    "projectName": "Build new apps with MongoDB",
    "team": [
      { "memberId": "tony", "name": "Tony", "role": "Developer" }
    ],
    "status": "planned"
  }
]

Suppose you want a report of which projects each person was involved in, and their role(s) for each.

In PostgreSQL, you’d have normalized to three tables because of the Many-to-Many relationship. The join returns one row per project and team member, with information about projects and members duplicated. You can then aggregate them per team member to get the projects per person as an array:

SELECT    
  tm.member_id,    
  tm.name,    
  ARRAY_AGG(p.project_name ORDER BY p.project_id) AS projects    
FROM    
  team_members tm    
  JOIN project_team pt ON tm.member_id = pt.member_id    
  JOIN projects p ON pt.project_id = p.project_id    
GROUP BY    
  tm.member_id, tm.name    
ORDER BY    
  tm.member_id;    

Here's the example on db<>fiddle: https://dbfiddle.uk/FhsA9DpU

In MongoDB, relationships are embedded. I created a "team" array for project members. Let's explore how to aggregate it, starting with some common mistakes.

The Wrong Way: Grouping Directly on an Array

Suppose you try to group directly on the array field:

db.projects.aggregate([  
  { $group: { _id: "$team.memberId", projects: { $push: "$projectName" } } }  
])

What you get:

[
  {
    "_id": [
      "cyclops", "wolverine", "storm", "beast", "tony"
    ],
    "projects": ["Troubleshooting PostgreSQL issues"]
  },
  {
    "_id": [
      "tony"
    ],
    "projects": ["Build new apps with MongoDB"]
  }
]

Here's the example on MongoDB Playground: https://mongoplayground.net/p/bn7xsRMQhu2

What went wrong?

MongoDB used the whole array as the grouping key. Instead of grouping by each individual member, you get one group for each unique full-team combination.

Tip for SQL users:

In SQL, GROUP BY splits rows by scalar values, but in MongoDB, grouping on an array field groups by the entire array as a single value.

The Wrong Way: Lookup Directly on an Array

Similarly, you may try to enrich team data with $lookup on the array field:

{  
  $lookup: {  
    from: "members",  
    localField: "team.memberId",  
    foreignField: "memberId",  
    as: "memberInfo"  
  }  
}

Here's the example on MongoDB Playground:

https://mongoplayground.net/p/RVXoY5z7ke1

What you get:

• The entire array is used as the lookup key, so memberInfo will usually just be an empty array (no matches), unless a member’s memberId somehow matches an array, which it never will.

Key for SQL folks:

Unlike SQL, where the JOIN is applied per row value, MongoDB $lookup expects scalar fields, not arrays. If your join keys are inside an array, you need to flatten that array first.

The Right Way: Flatten with $unwind, Then $group

First, flatten out your team array so each person/project combination gets its own pipeline document:

{ $unwind: "$team" }

Here's how it looks like on MongoDB Playground:

https://mongoplayground.net/p/yGvvM-FZM5p

Then group by the member ID:

Don’t forget to include the role for each project. Here, we also collect all roles for completeness.

{
  $group: {
    _id: "$team.memberId",
    memberName: { $first: "$team.name" },
    roles: { $addToSet: "$team.role" },
    projects: {
      $push: {
        name: "$projectName",
        id: "$_id",
        role: "$team.role"
      }
    }
  }
}

The full working pipeline is like this:

db.projects.aggregate([
  { $unwind: "$team" },
  {
    $group: {
      _id: "$team.memberId",
      memberName: { $first: "$team.name" },
      roles: { $addToSet: "$team.role" },
      projects: {
        $push: {
          name: "$projectName",
          id: "$_id",
          role: "$team.role"
        }
      }
    }
  }
])

Here's the result on MongoDB Playground:

https://mongoplayground.net/p/oMOfvrZXa2a

This is the expected output, with one document per member and their project or array of projects:

[
  {
    "_id": "cyclops",
    "memberName": "Cyclops",
    "roles": ["Postgres Expert"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Postgres Expert" }
    ]
  },
  {
    "_id": "wolverine",
    "memberName": "Wolverine",
    "roles": ["Consultant"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Consultant" }
    ]
  },
  {
    "_id": "storm",
    "memberName": "Storm",
    "roles": ["DBA"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "DBA" }
    ]
  },
  {
    "_id": "beast",
    "memberName": "Beast",
    "roles": ["Developer"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Developer" }
    ]
  },
  {
    "_id": "tony",
    "memberName": "Tony",
    "roles": ["Architect", "Developer"],
    "projects": [
      { "name": "Troubleshooting PostgreSQL issues", "id": ObjectId("60f...a1"), "role": "Architect" },
      { "name": "Build new apps with MongoDB", "id": ObjectId("60f...a2"), "role": "Developer" }
    ]
  }
]

Now you get the right result: Tony appears in both projects, and each is listed with the correct role.

Practical reminder:

MongoDB’s $unwind makes array data behave more like SQL’s flattened rows, so aggregation and joins work as SQL users expect.

Here is an example with $lookup: https://mongoplayground.net/p/I3__p2EE9ps

What About Filtering Instead of Grouping?

If you just want to filter out all but the developers in each project, no unwinding is needed:

db.projects.aggregate([
  {
    $addFields: {
      team: {
        $filter: {
          input: "$team",
          as: "member",
          cond: { $eq: ["$$member.role", "Developer"] }
        }
      }
    }
  }
])

Here's the MongoDB playground:

https://mongoplayground.net/p/bhDQJVKu7Rp

[
  {
    "_id": ObjectId("60f...a1"),
    "projectName": "Troubleshooting PostgreSQL issues",
    "team": [
      { "memberId": "beast", "name": "Beast", "role": "Developer" }
    ],
    "status": "active"
  },
  {
    "_id": ObjectId("60f...a2"),
    "projectName": "Build new apps with MongoDB",
    "team": [
      { "memberId": "tony", "name": "Tony", "role": "Developer" }
    ],
    "status": "planned"
  }
]

I used the following aggregation operators:

• $unwind: Flattens array fields, creating a separate document for each element in the array.
• $group: Similar to SQL’s GROUP BY, especially effective after arrays have been unwound.
• $lookup: Similar to a SQL LEFT OUTER JOIN, but by default MongoDB does not flatten joined results into multiple documents with repeated data. Instead, matched documents are returned normalized (0NF) as an array field.

SQL databases store their data normalized, often in third normal form, with multiple tables for one-to-many and many-to-many relationships to avoid duplicated data. The query result requires a join and is always in first normal form (1NF), as the result of a SQL query is a single flat table. Since 1NF cannot have arrays, SQL databases have to unnest groups, flattening relationships to multiple rows and introducing redundancy in a denormalized tabular result set.

MongoDB is not constrained by normal forms and supports rich document models, with arrays for repeating groups and nested objects directly in each document. When you use the aggregation pipeline, results can keep this nested structure. But if you want to group or join on values nested inside arrays, you’ll need to flatten the array to multiple documents using $unwind, so further aggregation stages work as expected. In practice, $lookup in MongoDB is often compared to JOINs in SQL, but if your fields live inside arrays, a join operation is really $unwind followed by $lookup.

Key takeaways for SQL users moving to MongoDB:

• Arrays are not automatically expanded or unnested.
• Whenever your “join key” or “group by value” is inside an array, always unwind first.
• In MongoDB, aggregation pipeline results can be nested and contain arrays, unlike the flat results of SQL queries.
• $unwind is the document-model equivalent of SQL’s UNNEST: it’s your bridge from nested arrays to flat, row-like documents for further aggregation.
• When joining with $lookup, always check whether your localField or foreignField is an array and flatten as needed.

The paper (arXiv 2020, also AI review 2023) opens up with discussing recent high-profile AI debates: the Montréal AI Debate and the AAAI 2020 fireside chat with Kahneman, Hinton, LeCun, and Bengio. A consensus seems to be emerging: for AI to be robust and trustworthy, it must combine learning with reasoning. Kahneman's "System 1 vs. System 2" dual framing of cognition maps well to deep learning and symbolic reasoning. And AI needs both.

Neurosymbolic AI promises to  combine data-driven learning with structured reasoning, and provide modularity, interpretability, and measurable explanations. The paper moves from philosophical context to representation, then to system design and technical challenges in neurosymbolic AI. 

Neurons and Symbols: Context and Current Debate

This section lays out the historic divide within symbolic AI and neural AI. Symbolic approach supports logic, reasoning, and explanation. Neural approach excels at perception and learning from data. Symbolic systems are good at thinking, but not learning. Deep learning is good at learning, but not thinking. Despite the great progress recently, deep learning still lacks transparency and remains energy-hungry. Critics like Gary Marcus argue that symbolic manipulation is needed for generalization and commonsense.

The authors here appeal to Valiant's call for a "semantics of knowledge" and say that neural-symbolic computing aims to answer this call. Symbolic logic can be embedded in neural systems, and neural representations can be interpreted in symbolic terms. Logic Tensor Networks (LTNs) are presented as a concrete solution. They embed first order logic formulas into tensors, and sneak logic into the loss function to help learn not just from data, but from rules. For this, logical formulas are relaxed into differentiable constraints. These are then used during training, guiding the model to satisfy logical relationships while learning from data. I was surprised to see some concrete work and software for LTNs on github. There is also a paper explaining the principles.

Distributed and Localist Representation

This section reframes the debate around representation. Neural networks use distributed representations: knowledge is encoded in continuous vectors, over which the concepts are smeared. This works well for learning and optimization. Symbolic systems use localist representations: discrete identifiers for concepts. These are better for reasoning and abstraction.

The challenge is to bridge the two. LTNs do this by grounding symbolic logic into tensor-based representations. Logic formulas are mapped to constraints over continuous embeddings. This enables symbolic querying over learned neural structures, while preserving the strengths of gradient-based learning. LTNs also allow symbolic structure to emerge during learning.

There is an interesting contrast here with the Neurosymbolic AI paper we reviewed yesterday. That paper favored Option2 approaches, which begins with the neural representation and lifts it into symbolic form. In other words, it advocates extracting structured symbolic patterns, explanations, or logical chains of reasoning from the output of neural systems. This paper, through advocating for LTNs, seems to favor Option1: embedding symbolic structures into neural vector spaces.

Neurosymbolic Computing Systems: Technical Aspects

Symbolic models use rules: decision trees, logic programs, structured knowledge. They are interpretable but brittle to change and new information. Deep nets, on the other hand, learn vector patterns using gradient descent. They are great with fuzz, but awful with generalizing rules. They speak linear algebra, not logic.

The first approach to combine them is to bake logic into the network's structure. The second approach is to encode logic in the loss function, but otherwise keep it separate from the network's architecture. The authors seem to lean toward the second approach for its flexibility, modularity, and scalability.

LTNs also seem to fall into the second approach. LTNs represent logical formulas as differentiable constraints, which are added to the loss function during training. The network learns to satisfy logic, but the logic is not hardwired into its structure. So the logic guides learning, but it is not embedded in the weights.

Challenges for the Principled Combination of Reasoning and Learning

Combining reasoning and learning introduces new challenges. One is how to handle quantifiers. Symbolic systems handle universal quantifiers (\forall) well. Neural networks are better at spotting existential patterns (\exists). This asymmetry makes hybrid systems attractive: let each side do what it does best.

Restricted Boltzmann Machines (RBMs) are discussed as early examples of hybrid models. They learn probability distributions over visible and hidden variables. With modular design, rules can be extracted from trained RBMs. But as models grow deeper, they lose modularity and interpretability. Autoencoders, GANs, and model-based reinforcement learning may offer ways to address this.

The paper (2023) argues for integrating two historically divergent traditions in artificial intelligence (neural networks and symbolic reasoning) into a unified paradigm called Neurosymbolic AI. It argues that the path to capable, explainable, and trustworthy artificial intelligence lies in marrying perception-driven neural systems with structure-aware symbolic models. 

The authors lean on Daniel Kahneman’s story of two systems in the mind (Thinking Fast and Slow). Neural networks are the fast ones: pattern-hungry, intuitive, good with unstructured mess. Symbolic methods are the slow ones: careful, logical, good with rules and plans. Neural networks, especially in their modern incarnation as large language models (LLMs), excel at pattern recognition, but fall short in tasks demanding multi-step reasoning, abstraction, constraint satisfaction, or explanation. Conversely, symbolic systems offer interpretability, formal correctness, and composability, but tend to be brittle (not incremental/monotonic), difficult to scale, and poorly suited to noisy or incomplete inputs.

The paper argues that true AI systems must integrate both paradigms, leveraging the adaptability of neural systems while grounding them in symbolic structure. This argument is compelling, particularly in safety-critical domains like healthcare or law, where transparency and adherence to rules are essential.

The paper divides Neurosymbolic AI into two complementary approaches: compressing symbolic knowledge into neural models, and lifting neural outputs into symbolic structures.

The first approach is to embed symbolic structures such as ontologies or knowledge graphs (KGs) into neural vector spaces suitable for integration with neural networks. This can be done via embedding techniques that convert symbolic structures into high-dimensional vectors, or through more direct inductive bias mechanisms that influence a model's architecture. While this enables neural systems to make use of background knowledge, it often loses semantic richness in the process. The neural model benefits from the knowledge, but the end-user gains little transparency, and the symbolic constraints are difficult to trace or modify. Nevertheless, this approach scales well and offers modest improvements in cognitive tasks like abstraction and planning.

The second approach works in the opposite direction. It begins with the neural representation and lifts it into symbolic form. This involves extracting structured symbolic patterns, explanations, or logical chains of reasoning from the output of neural systems. One common approach is through federated pipelines where a large language model decomposes a query into subtasks and delegates those to domain-specific symbolic solvers, such as math engines or search APIs. Another strategy involves building fully differentiable pipelines where symbolic constraints, expert-defined rules, and domain concepts are embedded directly into the neural training process. This allows for true end-to-end learning while preserving explainability and control. These lifting-based systems show the greatest potential: they not only maintain large-scale perception but also achieve high marks in abstraction, analogy, and planning, along with excellent explainability and adaptability.

The case study in mental health application shows promise. The system's ability to map raw social media text to clinical ontologies and generate responses constrained by medical knowledge illustrates the potential of well-integrated symbolic and neural components. However, these examples also hint at the limitations of current implementations: it is not always clear how the symbolic reasoning is embedded or whether the system guarantees consistency under update or multi-agent interaction.

Knowledge graphs versus symbolic solvers

The paper claims that knowledge graphs (KGs) are especially well-suited for this integration—serving as the symbolic scaffolding that supports neural learning. KGs are graph-structured representations of facts, typically in the form of triples: (subject, predicate, object). KGs are praised for their flexibility, updateability, and ability to represent dynamic real-world entities. But the paper then waves off formal logic, especially first-order logic (FOL) as static and brittle. That's not fair. Knowledge graphs are great for facts: "Marie Curie discovered radium". But when it comes to constraint satisfaction or verifying safety, you'll need real logic. The kind with proofs. 

First-order logic is only brittle when you try to do too much with it all at once. Modern logic systems (SMT solvers, expressive type systems, modular specs) can be quite robust. The paper misses a chance here. It doesn't mention the rich and growing field where LLMs and symbolic solvers already collaborate (e.g., GPT writes a function and Z3 checks if it's wrong, and logic engines validate that generated plans do not violate physics or safety).

Knowledge graphs and symbolic logic don’t need to fight, as they don't compete like Coke and Pepsi. They are more like peanut-butter and jelly. You can use a knowledge graph to instantiate a logic problem. You can generate FOL rules from ontologies. You can use SMT to enforce constraints (e.g., cardinality, ontological coherence). You can even use a theorem prover to validate new triples before inserting them into the graph. You can also run inference rules to expand a knowledge graph deductively.

But the paper doesn't explore how lifted neural outputs could feed into symbolic solvers for planning or synthesis or reasoning. It misses the current push to combine neural generation with symbolic checking, where LLMs propose, and the verifiers dispose in a feedback loop. 

You can understand how MongoDB stores documents internally with simple queries that rely on the physical storage ordering. Some databases store records (called rows or tuples) in heap tables, using their physical location in the data files, such as ROWID in Oracle or CTID in PostgreSQL, to reference those records from index entries. In contrast, databases like MySQL's InnoDB or YugabyteDB store records in the primary key index ("clustered index", or "index organized table"), storing them by the logical order of their primary key values, so that secondary indexes point to these logical locations with the primary key, or an encoded version of it.
MongoDB default collections are similar to heap tables because their documents are stored independently of the primary key ("_id") exposed to the application. Internally, the WiredTiger storage engine organizes collection documents using a B+Tree structure, with an internal RecordId as the key, assigned by MongoDB. This structure resembles a clustered index, but it is clustered on an internal key rather than the primary key.

MongoDB’s approach improves on traditional heap tables, especially for storing variable-size documents, because WiredTiger uses B+Tree nodes for efficient space management, reusing space and splitting pages as needed, rather than relying on settings like PCTFREE or FILLFACTOR to reserve space for updates, or SHRINK/VACUUM operations to defragment after deletes.
To cover all cases, with clustered collections, MongoDB can generate the RecordId from the "_id", the primary key exposed to the application, making storage similar to how some databases organize tables in clustered indexes, as "_id" can be a generated ObjectId or defined by the application at insert time. So, when looking at storage internals levels, there are two keys:

• "_id" is the application's primary key, a generated surrogate key, or a natural key. It is always indexed, and this unique index, like other secondary indexes, references the document with a RecordId
• RecordId is the internal key. It can be generated from "_id" (in clustered collections), but is more generally generated as a monotonically increasing 64-bit integer during inserts. It can be considered a physical address by the query layer, but it is not directly mapped to an address in the filesystem because files are B+Tree structures. This offers physical data independence since the primary key generation pattern does not affect the storage organization. However, it is helpful to understand how it functions when reviewing execution plans.

Another perspective is that, aside from clustered collections, all indexes in MongoDB, including the primary key index on "_id", are essentially secondary indexes, similar to those found in heap table databases (such as Db2, Oracle, and PostgreSQL). However, instead of a heap table with a fixed block size and row identification tied to their physical location, MongoDB documents are stored within an internal B+Tree index using the WiredTiger engine. Both approaches have their rationale:

• SQL databases are primarily optimized for fixed, normalized schemas with small, uniform row lengths, where a PCTFREE or FILLFACTOR can be set according to the expected updates. Storing larger types involves row chaining or slicing (like Oracle LOB chunks or PostgreSQL TOAST)
• MongoDB is designed for flexible schemas, and collections can contain documents of any size up to the BSON limit. Typically, documents are tens to hundreds of kilobytes, with some larger ones reaching a few megabytes. This flexibility requires adaptable storage management and efforts to minimize fragmentation beyond a small, fixed page size. A B+Tree with a flexible leaf block size is a suitable structure for this purpose.

The document size is flexible, thanks to the storage described above, but the ideal document size is a frequent question. Until I write a blog post on this, here’s a slide: the green area indicates where the most efficient access is, the red side is acceptable for outliers if they don't grow further, but may be a sign of embedding too much, and the blue side works for small documents inserted and queried together, but it may also be a sign that you did unnecessary normalization and should embed more to avoid runtime scattered lookups.

An example to understand the internal ordering

To demonstrate how it works, I generate ten documents and insert them asynchronously, so they may be written to the database in a random order:

db.collection.drop();
Array.from({ length: 10 }, (_, i) => {
  db.collection.insertOne({
    _id: `#${String(i).padStart(5, '0')}` ,
      val: Math.random()
  });
});

If I query without any filter or sort, the query planner chooses a COLLSCAN, which reads the records in the order of their RecordID, in the order they were inserted:

test> db.collection.find();

[
  { _id: '#00002', val: 0.07658988613973294 },
  { _id: '#00008', val: 0.39893981577036675 },
  { _id: '#00009', val: 0.5279631881196858 },
  { _id: '#00007', val: 0.8445363162277748 },
  { _id: '#00006', val: 0.01935050813731909 },
  { _id: '#00004', val: 0.0732484258238264 },
  { _id: '#00005', val: 0.7733464850237388 },
  { _id: '#00003', val: 0.3356001641172073 },
  { _id: '#00000', val: 0.8956753135566624 },
  { _id: '#00001', val: 0.4952318922619017 }
]

Keep in mind that I'm working with just a single node here. Sharding and parallel processing might retrieve rows in a different order than how they're stored. You should not rely on any "natural" order. Instead, unless you're conducting this type of investigation, where you're guessing the physical ordering from the query layer, ensure that you use an explicit sort operation to specify the expected order of results.

I can display the RecordId with .showRecordId(), which adds it to the cursor projection:

test> db.collection.find().showRecordId();

[
  { _id: '#00002', val: 0.07658988613973294, '$recordId': Long('1') },
  { _id: '#00008', val: 0.39893981577036675, '$recordId': Long('2') },
  { _id: '#00009', val: 0.5279631881196858, '$recordId': Long('3') },
  { _id: '#00007', val: 0.8445363162277748, '$recordId': Long('4') },
  { _id: '#00006', val: 0.01935050813731909, '$recordId': Long('5') },
  { _id: '#00004', val: 0.0732484258238264, '$recordId': Long('6') },
  { _id: '#00005', val: 0.7733464850237388, '$recordId': Long('7') },
  { _id: '#00003', val: 0.3356001641172073, '$recordId': Long('8') },
  { _id: '#00000', val: 0.8956753135566624, '$recordId': Long('9') },
  { _id: '#00001', val: 0.4952318922619017, '$recordId': Long('10') }
]

Documentation: showRecordId()

Forcing an index with a hint

I can force an index with a hint, for example the index on "_id" which was created automatically:

test> db.collection.find().hint( { _id: 1} ).showRecordId();

[
  { _id: '#00000', val: 0.8956753135566624, '$recordId': Long('9') },
  { _id: '#00001', val: 0.4952318922619017, '$recordId': Long('10') },
  { _id: '#00002', val: 0.07658988613973294, '$recordId': Long('1') },
  { _id: '#00003', val: 0.3356001641172073, '$recordId': Long('8') },
  { _id: '#00004', val: 0.0732484258238264, '$recordId': Long('6') },
  { _id: '#00005', val: 0.7733464850237388, '$recordId': Long('7') },
  { _id: '#00006', val: 0.01935050813731909, '$recordId': Long('5') },
  { _id: '#00007', val: 0.8445363162277748, '$recordId': Long('4') },
  { _id: '#00008', val: 0.39893981577036675, '$recordId': Long('2') },
  { _id: '#00009', val: 0.5279631881196858, '$recordId': Long('3') }
]

This runs a IXSCAN instead of a COLLSCAN and returns the documents in the order of the index. You can verify it with .explain(), but it is also perceptible from the order of the document fetched, which follows the order of "_id" rather than the order of insertion as before (also called "natural" order).

Rather than using a hint, I can add a filter, and the query planner chooses the index. A filter like {$gt:MinKey} or {$lt:MaxKey} does not change the result, but changes the execution plan to an IXSCAN:

test> db.collection.find({_id:{$gt:MinKey}}).showRecordId();

[
  { _id: '#00000', val: 0.8956753135566624, '$recordId': Long('9') },
  { _id: '#00001', val: 0.4952318922619017, '$recordId': Long('10') },
  { _id: '#00002', val: 0.07658988613973294, '$recordId': Long('1') },
  { _id: '#00003', val: 0.3356001641172073, '$recordId': Long('8') },
  { _id: '#00004', val: 0.0732484258238264, '$recordId': Long('6') },
  { _id: '#00005', val: 0.7733464850237388, '$recordId': Long('7') },
  { _id: '#00006', val: 0.01935050813731909, '$recordId': Long('5') },
  { _id: '#00007', val: 0.8445363162277748, '$recordId': Long('4') },
  { _id: '#00008', val: 0.39893981577036675, '$recordId': Long('2') },
  { _id: '#00009', val: 0.5279631881196858, '$recordId': Long('3') }
]

An equality filter will also run an IXSCAN, and we observe the result fetched in that order:

test> db.collection.find({_id:{$ne:null}}).showRecordId();

[
  { _id: '#00000', val: 0.8956753135566624, '$recordId': Long('9') },
  { _id: '#00001', val: 0.4952318922619017, '$recordId': Long('10') },
  { _id: '#00002', val: 0.07658988613973294, '$recordId': Long('1') },
  { _id: '#00003', val: 0.3356001641172073, '$recordId': Long('8') },
  { _id: '#00004', val: 0.0732484258238264, '$recordId': Long('6') },
  { _id: '#00005', val: 0.7733464850237388, '$recordId': Long('7') },
  { _id: '#00006', val: 0.01935050813731909, '$recordId': Long('5') },
  { _id: '#00007', val: 0.8445363162277748, '$recordId': Long('4') },
  { _id: '#00008', val: 0.39893981577036675, '$recordId': Long('2') },
  { _id: '#00009', val: 0.5279631881196858, '$recordId': Long('3') }
]

This technique is used to add an unbounded range predicate on the indexed sort field to get the index used for the sort in the absence of an equality predicate: MongoDB Equality, Sort, Range (ESR) without Equality (SR)

Forcing a full scan with a hint for natural order

Hints specify the index definition, and you may wonder how to force a full scan instead of the index scan chosen by the query planner. Remember that it's an index on RecordId that stores the documents. So you can hint this internal index using the $natural operator - asking for natural order of the collection documents:

test> db.collection.find({_id:{$ne:null}}).hint({$natural:1}).showRecordId();

[
  { _id: '#00002', val: 0.07658988613973294, '$recordId': Long('1') },
  { _id: '#00008', val: 0.39893981577036675, '$recordId': Long('2') },
  { _id: '#00009', val: 0.5279631881196858, '$recordId': Long('3') },
  { _id: '#00007', val: 0.8445363162277748, '$recordId': Long('4') },
  { _id: '#00006', val: 0.01935050813731909, '$recordId': Long('5') },
  { _id: '#00004', val: 0.0732484258238264, '$recordId': Long('6') },
  { _id: '#00005', val: 0.7733464850237388, '$recordId': Long('7') },
  { _id: '#00003', val: 0.3356001641172073, '$recordId': Long('8') },
  { _id: '#00000', val: 0.8956753135566624, '$recordId': Long('9') },
  { _id: '#00001', val: 0.4952318922619017, '$recordId': Long('10') }
]

The documents are fetched in order of RecordId from a COLLSCAN. The hint syntax allows an ascending or descending option to start at the beginning or end of the collection. I'm showing this to explain how records are stored internally. However, if you need a specific order, you should use sort() and let the query planner decide whether to use the index to avoid a sort operation.

MongoDB is more than a NoSQL database:

• Like many NoSQL databases, it allows you to query the indexes directly with .hint(), forcing the access path
• Like all SQL databases, it has a query planner offering data independence, allowing you to declare the collection and expected order with .sort() and let the database optimize the access path.

Avoid combining storage-level instructions, such as .hint(), .min(), or .max(), with declarative query filters in find() or $match, as this can undermine the query planner's guarantees that results match the query predicates. For example, hinting at a partial index might lead to incomplete results.

Covering indexes and "_id" projection

Understanding what is stored in the index entries helps optimize queries to use an index-only scan (covering index).

For example, the following query reads the index on "_id" and projects only "_id" (which is by default) and "val":

test> db.collection.find(
 { _id: { $ne: null } },
 { val: 1 }
).explain().queryPlanner.winningPlan;

{
  isCached: false,
  stage: 'PROJECTION_SIMPLE',
  transformBy: { val: 1 },
  inputStage: {
    stage: 'FETCH',
    inputStage: {
      stage: 'IXSCAN',
      keyPattern: { _id: 1 },
      indexName: '_id_',
      isMultiKey: false,
      isUnique: true,
      isSparse: false,
      isPartial: false,
      indexVersion: 2,
      direction: 'forward',
      indexBounds: { _id: [ '[MinKey, null)', '(null, MaxKey]' ] }
    }
  }
}

Because the index on "_id" holds only the key ("_id") and RecordId, it must fetch the document (FETCH) before the projection (PROJECTION_SIMPLE). Even if it is a primary index from the application's point of view, it is physically equivalent to a secondary index.

I can see the same with another secondary index:

test> db.collection.createIndex( { val: 1 } );

val_1

test> db.collection.find(
 { val:{$gt: 0} },
 { val: 1 }
).explain().queryPlanner.winningPlan;

{
  isCached: false,
  stage: 'PROJECTION_SIMPLE',
  transformBy: { val: 1 },
  inputStage: {
    stage: 'FETCH',
    inputStage: {
      stage: 'IXSCAN',
      keyPattern: { val: 1 },
      indexName: 'val_1',
      isMultiKey: false,
      multiKeyPaths: { val: [] },
      isUnique: false,
      isSparse: false,
      isPartial: false,
      indexVersion: 2,
      direction: 'forward',
      indexBounds: { val: [ '(0, inf.0]' ] }
    }
  }
}

Such query projects "_id" because it is there by default, and then the index on "val" is not covering all fields. To avoid the FETCH, I need to remove "_id" from the projection explicitly:

test> db.collection.find(
 { val:{$gt: 0} },
 { val: 1 , _id: 0 }
).explain().queryPlanner.winningPlan;

{
  isCached: false,
  stage: 'PROJECTION_COVERED',
  transformBy: { val: 1, _id: 0 },
  inputStage: {
    stage: 'IXSCAN',
    keyPattern: { val: 1 },
    indexName: 'val_1',
    isMultiKey: false,
    multiKeyPaths: { val: [] },
    isUnique: false,
    isSparse: false,
    isPartial: false,
    indexVersion: 2,
    direction: 'forward',
    indexBounds: { val: [ '(0, inf.0]' ] }
  }
}

Another possibility: if I need to project "_id", I can add it to the index definition, making it a covering index for my query:

test> db.collection.createIndex( { val: 1 , _id: 1 } );

val_1__id_1

test> db.collection.find(
 {
                                    
                                    
                                    

                                        by Franck Pachot