MongoDB guarantees durability—the D in ACID—over the network with strong consistency—the C in the CAP theorem—by default. It still maintains high availability: in the event of a network partition, the majority of nodes continue to serve consistent reads and writes transparently, without raising errors to the application.

A consensus protocol based on Raft is used to achieve this at two levels:

• Writes are directed to the shard's primary, which coordinates consistency between the collection and the indexes. Raft is used to elect one replica as primary, with the others acting as secondaries.
• Writes to the shard's primary are replicated to the secondaries and acknowledged once a majority has guaranteed durability on persistent storage. The equivalent of the Raft log is the data itself—the transaction oplogs.

It's important to distinguish the two types of consensus involved: one for controlling replica roles and one for the replication of data itself. By comparison, failover automation around monolithic databases like PostgreSQL can use a consensus protocol to elect a primary (as Patroni does), but replication itself is built into PostgreSQL and does not rely on a consensus protocol—a failure in the middle may leave inconsistency between replicas.

Trade-offs between performance and consistency

Consensus on writes increases latency, especially in multi-region deployments, because it requires synchronous replication and waiting on the network, but it guarantees no data loss in disaster recovery scenarios (RPO = 0). Some workloads may prefer lower latency and accept limited data loss (for example, a couple of seconds of RPO after a datacenter burns). If you ingest data from IoT devices, you may favor fast ingestion at the risk of losing some data in such a disaster. Similarly, when migrating from another database, you might prefer fast synchronization and, in case of infrastructure failure, simply restart the migration from before the failure point. In such cases, you can use {w:1} write concern in MongoDB instead of the default {w:"majority"}.

Most failures are not full-scale disasters where an entire data center is lost, but transient issues with short network disconnections. With {w:1}, the primary risk is not data loss—because writes can be synchronized eventually—but split brain, where both sides of a network partition continue to accept writes. This is where the two levels of consensus matter:

• A new primary is elected, and the old primary steps down, limiting the split-brain window to a few seconds.
• With the default {w:"majority"}, writes that cannot reach a majority are not acknowledged on the side of the partition without a quorum. This prevents split brain. However, with {w:1}, those writes are acknowledged until the old primary steps down.

Because the failure is transient, when the old primary rejoins, no data is physically lost: writes from both sides still exist. However, these writes may conflict, resulting in a diverging database state with two branches. As with any asynchronous replication, this requires conflict resolution. MongoDB handles this as follows:

• Writes from the new primary are preserved, as this is where the application has continued to make progress.
• Writes that occurred on the old primary during the brief split-brain window are rolled back, and it pulls the more recent writes from the new primary.

Thus, when you use {w:1}, you accept the possibility of limited data loss in the event of a failure. Once the node is back, these writes are not entirely lost, but they cannot be merged automatically. MongoDB stores them as BSON files in a rollback directory so you can inspect them and perform manual conflict resolution if needed.

This conflict resolution is a Recover To a Timestamp (RTT).

Demo on a Docker lab

Let's try it. I start 3 containers as a replica set:

docker network create lab
docker run --network lab --name m1 --hostname m1 -d mongo --replSet rs0
docker run --network lab --name m2 --hostname m2 -d mongo --replSet rs0
docker run --network lab --name m3 --hostname m3 -d mongo --replSet rs0
docker exec -it m1 mongosh --eval '
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "m1:27017", priority: 3 },
    { _id: 1, host: "m2:27017", priority: 2 },
    { _id: 2, host: "m3:27017", priority: 1 }
  ]
})
'
until
docker exec -it m1 mongosh --eval "rs.status().members.forEach(m => print(m.name, m.stateStr))" |
 grep -C3 "m1:27017 PRIMARY"
do sleep 1 ; done

The last command waits until m1 is the primary, as set by its priority. I do that to make the demo reproducible with simple copy-paste.

I insert "XXX-10" when connected to m1:

docker exec -it m1 mongosh --eval '
  db.demo.insertOne(
   { _id:"XXX-10" , date:new Date() },
   { writeConcern: {w: "1"}    }
)
'

{ acknowledged: true, insertedId: 'XXX-10' }

I disconnect the secondary m2:

docker network disconnect lab m2

With a replication factor of 3, the cluster is resilient to one failure and I insert "XXX-11", when connected to the primary:

docker exec -it m1 mongosh --eval '
  db.demo.insertOne(
   { _id:"XXX-11" , date:new Date() },
   { writeConcern: {w: "1"}    }
)
'

{ acknowledged: true, insertedId: 'XXX-11' }

I disconnect m1, the current primary, and reconnect m2, and immediately insert "XXX-12", still connected to m1:

docker network disconnect lab m1
docker network    connect lab m2

docker exec -it m1 mongosh --eval '
  db.demo.insertOne(
   { _id:"XXX-12" , date:new Date() },
   { writeConcern: {w: "1"}    }
)
'

{ acknowledged: true, insertedId: 'XXX-12' }

Here, m1 is still a primary for a short period before it detects it cannot reach the majority of replicas and steps down. If the write concern was {w: "majority"} it would have waited and failed, not able to sync to the quorum, but with {w: "1"} the replication is asynchronous and the write is acknowledged when written to local disks.

Two seconds later, a similar write fails because the primary stepped down:

sleep 2

docker exec -it m1 mongosh --eval '
  db.demo.insertOne(
   { _id:"XXX-13" , date:new Date() },
   { writeConcern: {w: "1"}    }
)
'

MongoServerError: not primary

I wait that m2 is the new primary, as set by priority, and connect to it to insert "XXX-20":

until
docker exec -it m2 mongosh --eval "rs.status().members.forEach(m => print(m.name, m.stateStr))" |
 grep -C3 "m2:27017 PRIMARY"
do sleep 1 ; done

docker exec -it m2 mongosh --eval '
  db.demo.insertOne(
   { _id:"XXX-20" , date:new Date() },
   { writeConcern: {w: "1"}    }
)
'

{ acknowledged: true, insertedId: 'XXX-20' }

No nodes are down, it's only a network partition, and I can read from all nodes as long as I don't connect through the network. I query the collection on each side:

docker exec -it m1 mongosh --eval 'db.demo.find()'
docker exec -it m2 mongosh --eval 'db.demo.find()'
docker exec -it m3 mongosh --eval 'db.demo.find()'

The inconsistency is visible, "XXX-12" is only in m1 and "XXX-20" only in m2 and m3:

I reconnect m1 so that all nodes can communicate and synchronize their state:

docker network    connect lab m1

I query again and all nodes show the same values:

"XXX-12" has disappeared and all nodes are now synchronized to the current state. When it rejoined, m1 rolled back the operations that occurred during the split-brain window. This is expected and acceptable, since the write used a { w: 1 } write concern, which explicitly allows limited data loss in case of failure in order to avoid cross-network latency on each write.

The rolled back operations are not lost, MongoDB logged them in a rollback directory in the BSON format, with the rolled back document as well as the related oplog.

I read and decode all BSON in the rollback directory:

docker exec -i m1 bash -c '
for f in /data/db/rollback/*/removed.*.bson
do
 echo "$f"
 bsondump $f --pretty
done
' | egrep --color=auto '^|^/.*|.*("op":|"XXX-..").*'

The deleted document is in /data/db/rollback/0ae03154-0a51-4276-ac62-50d73ad31fe0/removed.2026-02-10T10-40-58.1.bson:

{
        "_id": "XXX-12",
        "date": {
                "$date": {
                        "$numberLong": "1770719868965"
                }
        }
}

The deleted oplog for the related insert is in /data/db/rollback/local.oplog.rs/removed.2026-02-10T10-40-58.0.bson:

{
        "lsid": {
                "id": {
                        "$binary": {
                                "base64": "erR2AoFXS3mbcX4BJSiWjw==",
                                "subType": "04"
                        }
                },
                "uid": {
                        "$binary": {
                                "base64": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
                                "subType": "00"
                        }
                }
        },
        "txnNumber": {
                "$numberLong": "1"
        },
        "op": "i",
        "ns": "test.demo",
        "ui": {
                "$binary": {
                        "base64": "CuAxVApRQnasYlDXOtMf4A==",
                        "subType": "04"
                }
        },
        "o": {
                "_id": "XXX-12",
                "date": {
                        "$date": {
                                "$numberLong": "1770719868965"
                        }
                }
        },
        "o2": {
                "_id": "XXX-12"
        },
        "stmtId": {
                "$numberInt": "0"
        },
        "ts": {
                "$timestamp": {
                        "t": 1770719868,
                        "i": 1
                }
        },
        "t": {
                "$numberLong": "1"
        },
        "v": {
                "$numberLong": "2"
        },
        "wall": {
                "$date": {
                        "$numberLong": "1770719868983"
                }
        },
        "prevOpTime": {
                "ts": {
                        "$timestamp": {
                                "t": 0,
                                "i": 0
                        }
                },
                "t": {
                        "$numberLong": "-1"
                }
        }
}

Conclusion: beyond Raft

By default, MongoDB favors strong consistency and durability: writes use { w: "majority" }, are majority-committed, never rolled back, and reads with readConcern: "majority" never observe rolled-back data. In this mode, MongoDB behaves like a classic Raft system: once an operation is committed, it is final.

MongoDB also lets you explicitly relax that guarantee by choosing a weaker write concern such as { w: 1 }. In doing so, you tell the system: "Prioritize availability and latency over immediate global consistency." The demo shows what that implies:

• During a transient network partition, two primaries can briefly accept writes.
• Both branches of history are durably written to disk.
• When the partition heals, MongoDB deterministically chooses the majority branch.
• Operations from the losing branch are rolled back—but not discarded. They are preserved as BSON files with their oplog entries.
• The node then recovers to a majority-committed timestamp (RTT) and rolls forward.

This rollback behavior is where MongoDB intentionally diverges from vanilla Raft.

In classic Raft, the replicated log is the source of truth, and committed log entries are never rolled back. Raft assumes a linearizable, strongly consistent state machine where the application does not expect divergence. MongoDB, by contrast, comes from a NoSQL and event-driven background, where asynchronous replication, eventual consistency, and application-level reconciliation are sometimes acceptable trade-offs.

As a result:

• MongoDB still uses Raft semantics for leader election and terms, so two primaries are never elected in the same term.
• For data replication, MongoDB extends the model with Recover To a Timestamp (RTT) rollback.
• This allows MongoDB to safely support lower write concerns, fast ingestion, multi-region latency optimization, and migration workloads—without silently corrupting state.

In short, MongoDB replication is based on Raft, but adds rollback semantics to support real-world distributed application patterns. Rollbacks happen only when you explicitly allow them, never with majority writes, and they are fully auditable and recoverable.

Despite the ubiquity of the MongoDB aggregation framework, it has been lacking a formal mathematical framework/specification. This paper aims to fix this gap by providing a theoretical foundation, and proposes MQuery. The formalization in MQuery is largely based on the paper published at ICDT 2018 (for which the first author is involved), extending it to include more pipeline operators, relax the assumption that the JSON documents stored in the database comply to a predefined schema, and allow objects that are either ordered or unordered sets of key-value pairs.

Motivation

For decades, SQL proponents have flaunted the rigorous mathematical foundation of relational algebra (courtesy of Edgar Codd). The world of JSON document databases, however, has remained a bit of a Wild West in comparison. The analogy is apt because, like the frontier, there is immense opportunity here. JSON is the undisputed king of data exchange, and the MongoDB aggregation framework has emerged as the widely adopted query language for JSON collections. Thanks to its expressive pipeline model, massive developer base, and popularity, MongoDB aggregation framework has effectively become the de facto standard for querying JSON. The fact that major vendors (including Amazon, Microsoft, Oracle, and Google) seek to provide compatibility with the MongoDB API further underscores its recognition as a common lingua franca. (The authors' words, not mine, so don't think I'm bragging on behalf of MongoDB.)

To further motivate the need for a rigorous mathematical framework, the authors highlight current challenges. They argue that MongoDB's semantics are procedural rather than declarative. While the aggregation pipeline is pragmatic and powerful, its documentation often overlooks edge cases, leading to ambiguity.

The paper illustrates this with an example about query predicates. In MongoDB, the query origin: "UK" matches a document where origin is the string "UK". However, it also matches a document where origin is the array ["UK", "Japan"]. While this loose equality is convenient for developers, it is bad for mathematical logic, as it violates the property of transitivity: ["UK"] matches "UK", and "UK" matches "UK", yet [["UK"]] does not match "UK".

Furthermore, the paper argues MongoDB suffers from path polysemy. A path like origin.country is inherently ambiguous. Does it refer to a nested field in a single object? Or, if origin is an array, does it refer to the country field of every object inside that array? This leads to data-dependent behavior, where a valid query might throw a runtime error simply because a new document with a different structure was inserted into the collection.

MQuery

MQuery (which, admittedly, looks a lot like McQuery, and now that I've said this, you won't be able to read it any other way) serves as a formalized abstraction of the MongoDB language. MQuery formalizes the data model using "d-values" (document values), which encompass literals, arrays, and objects. It also defines 7 core pipeline stages that mirror the MongoDB aggregation framework: $match, $unwind, $project, $group, $lookup, $graphLookup, and $union.

By formalizing these stages, the authors confirm in Section 4 that "the MongoDB aggregation framework is very expressive: at least as expressive as full relational algebra (RA)". They mention:

• the match, unwind, project and group fragment captures RA over a single relation
• the match, unwind, project, group, and lookup fragment captures arbitrary RA

They demonstrate that $match corresponds to selection, $project to projection, and $lookup (or a combination of $unwind and $group) to joins. This confirms that document databases can theoretically perform every operation relational databases can, including complex joins and set operations. They also note the MongoDB aggregation framework goes beyond RA by handling Nested Relational Algebra and linear recursion via $graphLookup.

The Payoff: Algebraic Optimization

Why does all this math matter? The formal definition can help us safely optimize queries. The final section of the paper demonstrates algebraic rewriting rules. Thanks to the formal definitions, the authors can prove when it is safe to reorder pipeline stages without altering the result. They provide rules for filter anticipation (moving $match earlier to reduce data volume), unnesting postponement (moving $unwind later to save memory), and join optimization.

This is an external post of mine. Click here if you are not redirected.

You may have learned about normal forms when databases were designed before the applications that used them. At that time, relational data models focused on enterprise-wide entities, defined before access patterns were known, so future applications could share a stable, normalized schema.

Today, we design databases for specific applications or bounded domains. Instead of defining a full model up front, we add features incrementally, gather feedback, and let the schema evolve with the application.

Normal forms aren't just relational theory—they describe real data dependencies. MongoDB's document model doesn't exempt you from thinking about normalization—it gives you more flexibility in how you apply it.

Example: Pizzerias

We're starting a new business: a large network of pizzerias across many areas with a wide variety of pizzas. But let's start small.

Tabular: One Pizza, One Area

As a minimal viable product (MVP), each pizzeria has one manager, sells only one variety, and delivers to one area. You can choose any database for this: key-value, relational, document, or even a spreadsheet. The choice will matter only when your product evolves.

Here is our first pizzeria:

{
  name: "A1 Pizza",
  manager: "Bob",
  variety: "Thick Crust",
  area: "Springfield"
}

With no repeating groups or multi-valued attributes, the model is already in First Normal Form (1NF). Because the MVP data model is simple—one value per attribute and a single key—there are no dependencies that would violate higher normal forms.

Many database designs start out fully normalized, not because the designer worked through every normal form, but because the initial dataset is too simple for complex dependencies to exist.

Normalization becomes necessary later, as business rules evolve and new varieties, areas, and independent attributes introduce dependencies that higher normal forms address.

1NF: More Menu Options

The business started quite well and evolves. A pizzeria can now offer several varieties.

The following, adding multiple varieties in a single field, would violate 1NF:

{
  name: "A1 Pizza",
  manager: "Bob",
  varieties: "Thick Crust, Stuffed Crust",
  area: "Springfield"
}

1NF requires atomic values—each field should hold one indivisible piece of data. A comma-separated string breaks this rule: you can't easily query, index, or update individual varieties. You can manipulate it as a character string, but you can't treat each entry as a distinct pizza variety, and you can't index it efficiently.

SQL and NoSQL databases avoid this pattern for different reasons. In a relational database, the logical model must be independent of cardinalities and access patterns. Because the relational model doesn't know whether there are two or one million pizza varieties, it treats every one-to-many relationship as unbounded and stores it in a separate table as a set of pizzeria–variety relationships rather than embedding varieties within the pizzeria entity.

Once we understand the application domain, we can set realistic bounds. Thousands of pizza varieties in the menu would be impractical from a business perspective well before hitting database limits, so storing the varieties together can be acceptable. When object-oriented applications use richer structures than two-dimensional tables, it's better to represent such lists as arrays rather than comma-separated strings:

{
  name: "A1 Pizza",
  manager: "Bob",
  email: "bob@a1-pizza.it",
  varieties: ["Thick Crust", "Stuffed Crust"]
}

Arrays of atomic values satisfy a document-oriented equivalent of 1NF—each element is atomic and independently addressable—even though the document model isn't bound by the relational requirement of flat tuples. While SQL databases provide abstraction and logical-physical data independence, MongoDB keeps data colocated down to the storage and CPU caches for more predictable performance.

Normal form definitions assume keys for each 1NF relation. In a document model, multiple relations can appear as embedded sub-documents or arrays. Treating the parent key and the array element together as a composite key lets us apply higher normal forms to analyze partial and transitive dependencies within a single document.

2NF: Pizza Pricing

We want to add the price of the pizzas to our database. If each pizzeria defines its own base price, it can be added to the varieties items:

{
  name: "A1 Pizza",
  manager: "Bob",
  email: "bob@a1-pizza.it",
  varieties: [
    { name: "Thick Crust", basePrice: 10 },
    { name: "Stuffed Crust", basePrice: 12 }
  ]
}

Second Normal Form (2NF) builds on 1NF by requiring that every non-key attribute depends on the entire primary key, not just part of it. This only becomes relevant when dealing with composite keys.

In our embedded model, consider the composite key ("pizzeria", "variety") for each item in the varieties array. If the price depends on the pizzeria and variety together—meaning different pizzerias can set different prices for the same variety—then "basePrice" depends on the full composite key, and we satisfy 2NF.

However, if prices are standardized across all pizzerias—the same variety costs the same everywhere—then a partial dependency exists: "basePrice" depends only on "variety", not on the full ("pizzeria", "variety") key. This violates 2NF.

To resolve this, we define pricing in a separate collection where the base price depends only on the pizza variety:

{ variety: "Thick Crust", basePrice: 10 }
{ variety: "Stuffed Crust", basePrice: 12 }

We can remove the base price from the pizzeria's varieties array and retrieve it from the pricing collection at query time:

db.createView(
  "pizzeriasWithPrices",
  "pizzerias",
  [
    { $unwind: "$varieties" },
    {
      $lookup: {
        from: "pricing",
        localField: "varieties.name",
        foreignField: "variety",
        as: "priceInfo"
      }
    },
    { $unwind: "$priceInfo" },
    { $addFields: { "varieties.basePrice": "$priceInfo.basePrice" } },
    { $project: { priceInfo: 0 } }
  ]
);

Alternatively, we can use the pricing collection as a reference, where the application retrieves the price and stores it in the pizzeria document for faster reads.

To avoid update anomalies, the application updates all affected documents when a variety's price changes:

const session = db.getMongo().startSession();
const sessionDB = session.getDatabase(db.getName());
session.startTransaction();

sessionDB.getCollection("pricing").updateOne(
  { variety: "Thick Crust" },
  { $set: { basePrice: 11 } }
);

sessionDB.getCollection("pizzerias").updateMany(
  { "varieties.name": "Thick Crust" },
  { $set: { "varieties.$[v].basePrice": 11 } },
  { arrayFilters: [{ "v.name": "Thick Crust" }] }
);

session.commitTransaction();

SQL databases avoid such multiple updates because they're designed for direct end-user access, sometimes bypassing the application layer. Without applying normal forms to break dependencies into multiple tables, there's a risk of overlooking replicated data. A document database is updated by an application service responsible for maintaining consistency.

While normalizing to 2NF is possible, it may not always be the best choice in a domain-driven design. Keeping the price embedded in each pizzeria allows asynchronous updates and supports future requirements where some pizzerias may offer different prices—without breaking integrity, as the application enforces updates atomically.

In practice, many applications accept this controlled duplication when price changes are infrequent and prefer fast single-document reads over perfectly normalized writes.

3NF: Manager's Contacts

When we started, each pizzeria had a single email contact:

{
  name: "A1 Pizza",
  manager: "Bob",
  email: "bob@a1-pizza.it",
  varieties: [
    { name: "Thick Crust", basePrice: 10 },
    { name: "Stuffed Crust", basePrice: 12 }
  ]
}

Third Normal Form (3NF) builds on 2NF by requiring that non-key attributes depend only on the primary key, not on other non-key attributes. When a non-key attribute depends on another non-key attribute, we have a transitive dependency.

Here, the email actually belongs to the manager, not the pizzeria directly. This creates a transitive dependency: "pizzeria" → "manager" → "email". Since "email" depends on "manager" (a non-key attribute) rather than directly on the pizzeria, this violates 3NF.

We can normalize this by grouping the manager's attributes into an embedded subdocument:

{
  name: "A1 Pizza",
  manager: { name: "Bob", email: "bob@a1-pizza.it" },
  varieties: [
    { name: "Thick Crust", basePrice: 10 },
    { name: "Stuffed Crust", basePrice: 12 }
  ]
}

Now the email is clearly an attribute of the manager entity embedded within the pizzeria. If a pizzeria has multiple managers, we can simply use an array of subdocuments without creating new collections or changing index definitions.

A generic relational model would probably split this into multiple tables, with manager being a foreign key to a "contacts" table. However, in our business domain, we don't manage contacts outside of pizzerias. Even if the same person manages multiple pizzerias, they're recorded as separate manager entries. Bob may have multiple emails and use different ones for each of his pizzerias.

4NF: Delivery Areas

We want to record the areas where a pizzeria can deliver its pizza varieties:

{
  name: "A1 Pizza",
  manager: { name: "Bob", email: "bob@a1-pizza.it" },
  offerings: [
    { variety: { name: "Thick Crust", basePrice: 10 }, area: "Springfield" },
    { variety: { name: "Thick Crust", basePrice: 10 }, area: "Shelbyville" }
  ]
}

Fourth Normal Form (4NF) addresses multi-valued dependencies. A multi-valued dependency exists when one attribute determines a set of values for another attribute, independent of all other attributes. 4NF requires that a relation have no non-trivial multi-valued dependencies except on superkeys.

If varieties and areas were dependent—for example, if certain varieties were only available in certain areas—then storing ("variety", "area") combinations would represent a single multi-valued fact, and there would be no 4NF violation.

However, since our pizzerias deliver all varieties to all areas, these are independent multi-valued dependencies: "pizzeria" →→ "variety" and "pizzeria →→ area". Storing all combinations creates redundancy—if we add a new area, we must add entries for every variety.

We normalize by storing each independent fact in a separate array:

{
  name: "A1 Pizza",
  manager: { name: "Bob", email: "bob@a1-pizza.it" },
  varieties: [
    { name: "Thick Crust", basePrice: 10 },
    { name: "Stuffed Crust", basePrice: 12 }
  ],
  deliveryAreas: ["Springfield", "Shelbyville"]
}

With this schema, we avoid violating 4NF because delivery areas and varieties are stored independently—even though the document model allows us to embed them together.

BCNF: Per-Area Pricing

Our network grows further. Some pizzerias now charge different prices depending on the delivery area—distant areas cost more:

{
  name: "A1 Pizza",
  manager: { name: "Bob", email: "bob@a1-pizza.it" },
  offerings: [
    { variety: "Thick Crust", area: "Springfield", price: 10 },
    { variety: "Thick Crust", area: "Shelbyville", price: 11 },
    { variety: "Stuffed Crust", area: "Springfield", price: 12 },
    { variety: "Stuffed Crust", area: "Shelbyville", price: 13 }
  ]
}

The composite key for each offering is ("pizzeria", "variety", "area"). The price depends on the full key, satisfying 2NF and 3NF.

Now our franchise assigns an area manager to each area—one manager per area, regardless of pizzeria. We add it to our offerings:

offerings: [
  { variety: "Thick Crust", area: "Springfield", price: 10, areaManager: "Alice" },
  { variety: "Stuffed Crust", area: "Springfield", price: 12, areaManager: "Alice" },
  { variety: "Thick Crust", area: "Shelbyville", price: 11, areaManager: "Eve" },
  { variety: "Stuffed Crust", area: "Shelbyville", price: 13, areaManager: "Eve" }
]

Boyce-Codd Normal Form (BCNF) is a stricter version of 3NF. It requires that for every non-trivial functional dependency X → Y, the determinant X must be a superkey. Unlike 3NF, BCNF doesn't make an exception for dependencies where the dependent attribute is part of a candidate key.

This model passes 3NF but fails BCNF: the dependency "area" → "areaManager" has a determinant ("area") that is not a superkey of the offerings relation. The area alone doesn't uniquely identify an offering—you need the full ("pizzeria", "variety", "area") key for that.

The practical problem: if Alice is replaced by Carol for Springfield, we must update every offering for that area across every pizzeria. The relational solution is to extract area managers to a separate table.

In MongoDB, we can keep the embedded structure and handle updates explicitly:

db.pizzerias.updateMany(
  { "offerings.area": "Springfield" },
  { $set: { "offerings.$[o].areaManager": "Carol" } },
  { arrayFilters: [{ "o.area": "Springfield" }] }
)

This trades strict BCNF compliance for simpler queries and faster reads. The application ensures consistency during updates.

5NF: Adding Pizza Sizes

We now offer multiple sizes (Small, Medium, Large). Sizes, varieties, and delivery areas are all independent—any combination is valid.

Storing every combination explodes quickly:

offerings: [
  { variety: "Thick Crust", size: "Large", area: "Springfield" },
  { variety: "Thick Crust", size: "Large", area: "Shelbyville" },
  { variety: "Thick Crust", size: "Medium", area: "Springfield" },
  // ... 150 entries for 5 varieties × 3 sizes × 10 areas
]

Fifth Normal Form (5NF), also called Project-Join Normal Form, addresses join dependencies. A relation is in 5NF if it cannot be decomposed into smaller relations that, when joined, reconstruct the original—without losing information or introducing spurious tuples.

When valid combinations can be reconstructed from independent sets (the Cartesian product of varieties, sizes, and areas), storing all combinations explicitly creates redundancy and risks inconsistency. This violates 5NF.

The fix stores each independent fact separately:

{
  name: "A1 Pizza",
  varieties: ["Thick Crust", "Stuffed Crust"],
  sizes: ["Large", "Medium"],
  deliveryAreas: ["Springfield", "Shelbyville"]
}

Adding a new size requires updating one array—not hundreds of entries. The application or query logic reconstructs valid combinations when needed.

6NF: Tracking Price History

Our finance team needs to track price changes over time. We could embed the history:

offerings: [
  {
    variety: "Thick Crust",
    area: "Springfield",
    currentPrice: 12,
    priceHistory: [
      { price: 10, effectiveDate: ISODate("2024-01-01") },
      { price: 11, effectiveDate: ISODate("2024-03-15") },
      { price: 12, effectiveDate: ISODate("2024-06-01") }
    ]
  }
]

This works for moderate history but grows unboundedly over time.

Sixth Normal Form (6NF) decomposes relations so that each stores a single non-key attribute along with its time dimension. Every row represents one fact at one point in time:

// price_history collection
{ pizzeria: "A1 Pizza", variety: "Thick Crust", area: "Springfield", price: 10, effectiveDate: ISODate("2024-01-01") }
{ pizzeria: "A1 Pizza", variety: "Thick Crust", area: "Springfield", price: 11, effectiveDate: ISODate("2024-03-15") }
{ pizzeria: "A1 Pizza", variety: "Thick Crust", area: "Springfield", price: 12, effectiveDate: ISODate("2024-06-01") }

6NF is rarely used for operational data because it requires extensive joins for common queries. However, for auditing, analytics, and temporal queries—where you need to answer "what was the price on March 10th?"—it provides a clean model for tracking changes over time.

Summary

Normal forms are not a relic of relational theory. They describe fundamental data dependencies present in any system, regardless of storage technology. MongoDB’s document model does not remove the need to consider normalization. Instead, it lets you decide where, when, and how strictly to apply it, based on domain boundaries and access patterns.

In relational/SQL databases, schemas are usually designed as enterprise-wide information models. Many applications and users share the same database, accepting ad hoc SQL. To avoid update, insertion, and deletion anomalies in this shared environment, the schema must enforce functional dependencies, making higher normal forms essential. Because the database is the system of record, normalization centralizes integrity rules in the data model.

Modern architectures, by contrast, often follow Domain-Driven Design (DDD). Each bounded context owns its data model, which evolves with the application. With CQRS and microservices, each aggregate is updated only through a single application service that encapsulates business rules. Here, the database is not a shared integration point but a private persistence detail of the service.

MongoDB fits this style well:

• Documents model aggregates as they exist in the domain
• Arrays capture bounded one-to-many relationships
• Denormalization and controlled duplication improve read performance and scalability
• Consistency is enforced by application logic, not global database constraints

Because one service owns all updates, violating higher normal forms can be acceptable—and sometimes beneficial—provided the service preserves its invariants. Normalization becomes a design tool, not a rigid checklist.

In short:

• Use relational normalization when the database is a shared, queryable system of record accessed by many applications and users via SQL.
• Use document modeling with selective denormalization when building domain-aligned services with clear ownership, CQRS, and microservices.

Normal forms still matter—but in MongoDB, they guide your choices instead of dictating your schema.

In this series, we explored several ways to solve the "Doctor's On-Call Shift" problem, which demonstrates write skew anomalies and the need for serializable transactions in SQL. Beyond using a serializable isolation level, we also addressed it with normalization, explicit parent locking, and SQL assertions. I applied document modeling in Postgres with SELECT FOR UPDATE as an alternative to a parent-child relationship, so it is natural to consider MongoDB. Since MongoDB lacks explicit locking and a serializable isolation level, we can instead use a simple update that atomically reads and writes in an optimistic concurrency control style.

Here is a collection with one document per shift and a list of doctors with their on-call status for this shift:

db.shifts.insertOne({  
  _id: 1,  
  doctors: [  
    { name: "Alice", on_call: true, updated: new Date() },  
    { name: "Bob",   on_call: true, updated: new Date() }  
  ]  
});  

Conditional updateOne() to avoid write skew

The following function encapsulates the business logic in a single update: for a shift with at least one other doctor on call, one doctor can be taken off on-call duty:

function goOffCall(shiftId, doctorName) {  
  const res = db.shifts.updateOne(  
    {   
      _id: shiftId,  
      $expr: {   
        $gte: [  
          {  
            $size: {  
              $filter: {  
                input: "$doctors",  
                as: "d",  
                cond: {  
                  $and: [  
                    { $ne: [ "$$d.name", doctorName ] },  
                    { $eq: [ "$$d.on_call", true ] }  
                  ]  
                }  
              }  
            }  
          },  
          1  
        ]  
      },  
      "doctors.name": doctorName  
    },  
    {   
      $set: { "doctors.$.on_call": false, updated: new Date() }  
    }  
  );  
  return res.modifiedCount > 0 ? "OFF_OK" : "OFF_FAIL";  
}

MongoDB is a document database with many array operators. Here, the condition checks that there is another doctor ($ne: ["$$d.name", doctorName]) who is on call ($eq: ["$$d.on_call", true]). It counts these doctors with $size and keeps only shifts where the count is at least 1. Since there is only one document per shift, if none is returned either the shift doesn’t exist, or the doctor is not in this shift, or there aren’t enough on-call doctors to let one go off call. The following calls show the return code:

test> goOffCall(1,"Alice");

OFF_OK

test> goOffCall(1,"Bob");

OFF_FAIL

Alice was allowed to go off‑call, but Bob couldn’t, because he was the only doctor remaining on‑call.

Testing race conditions

I added a simpler function to set a doctor on call for a shift:

function goOnCall(shiftId, doctorName) {  
  const res = db.shifts.updateOne(  
    {   
      _id: shiftId,  
      "doctors.name": doctorName,  
      "doctors.on_call": false  
    },  
    {   
      $set: { "doctors.$.on_call": true, updated: new Date() }  
    }  
  );  
  return res.modifiedCount > 0 ? "ON_OK" : "ON_FAIL";  
}  

Here is Alice back to on-call again:

test> goOnCall(1,"Alice");

ON_OK

I define an assertion function to verify the business rule for a shift by counting the doctors on call:

function checkOnCalls(shiftId) {  
  const pipeline = [  
    { $match: { _id: shiftId } },  
    { $project: {  
        onCallCount: {  
          $size: {  
            $filter: {  
              input: "$doctors",  
              as: "d",  
              cond: "$$d.on_call"  
            }  
          }  
        }  
      }  
    }  
  ];  

  const result = db.shifts.aggregate(pipeline).toArray();  
  if (result.length && result[0].onCallCount < 1) {  
    print(`❌ ERROR! No doctors on call for shift ${shiftId}`);  
    return false;
  }  
  return true;  
}  

Now, I run a loop that randomly sets Alice or Bob on call and

const shiftId = 1;  
const doctors = ["Alice", "Bob"];  
const actions = [goOnCall, goOffCall];  

let iteration = 0;  

while (true) {  
  iteration++;  
  const doctor = doctors[Math.floor(Math.random() * doctors.length)];  
  const action = actions[Math.floor(Math.random() * actions.length)];  

  const result = action(shiftId, doctor);  
  print(`Shift ${shiftId}, Iteration ${iteration}: ${doctor} -> ${result}`);  

  if (!checkOnCalls(shiftId)) {  
    print(`🚨 Stopping: assertion broken at iteration ${iteration}`);  
    break;  // exit loop immediately  
  }   
}  

I've run this loop in multiple sessions and confirmed that the "Doctor's On-Call" assertion is never violated. This runs indefinitely because MongoDB guarantees data integrity—update operations are ACID:

If you want to stop it and check that the assertion works, you can simply bypass the conditional update and set all doctors to off call:

db.shifts.updateOne(  
  { _id: 1 },  
  { $set: { "doctors.$[doc].on_call": false, updated: new Date() } },  
  { arrayFilters: [ { "doc.on_call": true } ] }  
);  

The loops stop when they detect the violation:

MongoDB Schema validation

You can, and should, define schema validations on the part of your schema the application relies on, to be sure that no update bypasses the application model and logic. It is possible to add an 'at least one on‑call' rule:

db.runCommand({  
  collMod: "shifts",  
  validator: {  
    $expr: {  
      $gte: [  
        {  
          $size: {  
            $filter: {  
              input: "$doctors",  
              as: "d",  
              cond: { $eq: ["$$d.on_call", true] }  
            }  
          }  
        },  
        1  
      ]  
    }  
  },  
  validationLevel: "strict"  
});  

My manual update immediately fails:

Schema validation is a helpful safeguard, but it does not fully protect against write skew under race conditions. It runs on inserts and updates and, with validationLevel: "strict", raises an error on invalid documents—but only after MongoDB has already matched and targeted the document for update.

Key differences between conditional updates and schema validation:

To prevent write skew, you need the correct condition in the update itself. Use schema validation as an extra safeguard for other changes, such as inserts.

Indexing documents

In PostgreSQL, using one row per shift with a JSON array of doctors makes updates atomic and eliminates race conditions but reduces indexing flexibility (for example, range scans on array fields), so the serializable isolation level or normalization to parent-child is preferable. In MongoDB, storing a one-to-many relationship in a single document is native, and full indexing remains available—for example, you can index the updated field for each doctor's on-call status:

db.shifts.createIndex({ "doctors.updated": 1 });

This index supports equality, sorting, and range queries, such as finding shifts where the on-call status changed in the last hour:

const oneHourAgo = new Date(Date.now() - 60 * 60 * 1000);  

db.shifts.find({ "doctors.updated": { $gte: oneHourAgo } });

db.shifts.find(
 { "doctors.updated": { $gte: oneHourAgo } }
).explain("executionStats")

Here is the execution plan:

 executionStats: {
    executionSuccess: true,
    nReturned: 1,
    executionTimeMillis: 0,
    totalKeysExamined: 1,
    totalDocsExamined: 1,
    executionStages: {
      stage: 'FETCH',
      nReturned: 1,
      works: 2,
      advanced: 1,
      isEOF: 1,
      docsExamined: 1,
      inputStage: {
        stage: 'IXSCAN',
        nReturned: 1,
        works: 2,
        advanced: 1,
        isEOF: 1,
        keyPattern: {
          'doctors.updated': 1,
          _id: 1
        },
        indexName: 'doctors.updated_1__id_1',
        isMultiKey: true,
        multiKeyPaths: {
          'doctors.updated': [
            'doctors'
          ],
          _id: []
        },
        direction: 'forward',
        indexBounds: {
          'doctors.updated': [
            '[new Date(1770384644918), new Date(9223372036854775807)]'
          ],
          _id: [
            '[MinKey, MaxKey]'
          ]
        },
        keysExamined: 1,
        seeks: 1,
      }
    }
  },

Conclusion

MongoDB's document model and optimistic concurrency control solve the "Doctor's On-Call Shift" problem without explicit locks, serializable isolation, or SQL assertion. By embedding business logic in conditional updateOne operations using $expr and array operators, you can prevent write-skew anomalies at the database level.

Atomic, document-level operations combined with a "First Updater Wins" rule ensure that concurrent updates to the same shift document yield exactly one success and one failure. This approach leverages MongoDB's strengths:

• Atomic conditional updates that read and write in one step
• Optimistic concurrency control to handle conflicts
• Schema validation as an extra integrity check
• Flexible indexing on fields inside embedded arrays

Schema validation alone cannot prevent race conditions, but together with conditional updates it protects against concurrency anomalies and direct data corruption.

This pattern shows how MongoDB's document model can simplify concurrency problems that would otherwise require advanced transaction isolation or explicit locking in relational databases. By co-locating related data and using atomic operations, you can maintain integrity with simpler, faster code.

How does your computer create the illusion of running dozens of applications simultaneously when it only has a few physical cores?

Wait, I forgot the question because I am now checking my email. Ok, back to it...

The answer is CPU Virtualization. Chapters 6, 7 of OSTEP explore the engine behind this illusion, and how to balance raw performance with absolute control.

The OSTEP textbook is freely available at Remzi's website if you like to follow along.

Chapter 6. The Mechanism: Limited Direct Execution

The crux of the challenge is: How do we run programs efficiently without letting them takeover the machine? 

The solution is Limited Direct Execution (LDE) --the title spoils it. "Direct Execution" means the program runs natively on the CPU for maximum speed. "Limited" means the OS retains authority to stop the process and prevent restricted access. This requires some hardware support.

To prevent chaos, hardware provides two execution modes. Applications run in "User Mode", where they cannot perform privileged actions like I/O. The OS runs in "Kernel Mode" with full access to the machine. When a user program needs a privileged action, it initiates a "System Call". This triggers a 'trap' instruction that jumps into the kernel and raises the privilege level. To ensure security, the OS programs a "trap table" at boot time, telling the hardware exactly which code to run for each event.

If a process enters an infinite loop, how does the OS get the CPU back?

• The Cooperative Approach: Older systems (like early Mac OS) trusted processes to yield the CPU periodically. If a program locked up, you had to reboot.
• The Non-Cooperative Approach: Modern systems use a "Timer Interrupt". The hardware raises an interrupt every few milliseconds, forcefully halting the process and handing control back to the OS.

Finally, when the OS regains control and decides to switch to a different process, it executes a "context switch". This low-level assembly routine saves the current process's registers to its kernel stack and restores the next process's registers. By switching the stack pointer, the OS tricks the hardware: the 'return-from-trap' instruction returns into the new process instead of the old one.

Chapter 7. The Policy: CPU Scheduling

With the switching mechanism in place as discussed in Chapter 6, our next piece to attack is deciding which process to run next. Chapter 7 explores these policies, initially assuming all jobs arrive at once and have known run-times.

First, let's look at batch scheduling using "Turnaround Time". This metric is simply the time a job completes minus the time it arrived (T_completion - T_arrival). Now let's consider some batch scheduling policies with this metric:

• FIFO (First In, First Out): Simple, but suffers from the "convoy effect". If a short job gets stuck behind a long one, average turnaround time suffers.
• SJF (Shortest Job First): To fix the convoy, SJF runs the shortest job first. This is optimal for turnaround time if all jobs arrive at once, but fails if a short job arrives after a long job has started.
• STCF (Shortest Time-to-Completion First): By adding preemption, we get STCF. When a new job arrives, the scheduler runs the job with the least time remaining. This guarantees optimal turnaround time, see Fig 7.5.

Now, we consider interactivity using "Response Time". This is the time from when a job arrives to the first time it is scheduled (T_response = T_firstrun - T_arrival).

STCF is great for turnaround but terrible for response time; a user might wait seconds for their interactive job (say a terminal session) to start. Round Robin solves this by time-slicing: it runs a job for a set quantum (e.g., 10 ms) and then switches. This makes the system feel responsive.

However, Round Robin creates a trade-off. While it optimizes fairness and response time, it destroys turnaround time by stretching out the completion of every job. You cannot have your cake and eat it too. See Fig 7.6 & 7.7.

Finally, real programs perform I/O. When a process blocks waiting for a disk, the scheduler treats the time before the I/O as a "sub-job". By running another process during this wait, the OS maximizes "overlap" and system utilization.

There is one last assumption we did not relax: the OS does not actually know how long a job will run. This "No Oracle" problem sets the stage for the next chapter on the "Multi-Level Feedback Queue", which predicts the future by observing the past.

To conclude Section 7, it is worth remembering that there is no silver bullet. The best policy depends entirely on the workload. The more you know about what you are running, the better you can schedule it.

Last week Anthropic released a report on disempowerment patterns in real-world AI usage which finds that roughly one in 1,000 to one in 10,000 conversations with their LLM, Claude, fundamentally compromises the user’s beliefs, values, or actions. They note that the prevalence of moderate to severe “disempowerment” is increasing over time, and conclude that the problem of LLMs distorting a user’s sense of reality is likely unfixable so long as users keep holding them wrong:

However, model-side interventions are unlikely to fully address the problem. User education is an important complement to help people recognize when they’re ceding judgment to an AI, and to understand the patterns that make that more likely to occur.

In unrelated news, some folks have asked me about Prothean Systems’ new paper. You might remember Prothean from October, when they claimed to have passed all 400 tests on ARC-AGI-2—a benchmark that only had 120 tasks. Unsurprisingly, Prothean has not claimed their prize money, and seems to have abandoned claims about ARC-AGI-2. They now claim to have solved the Navier-Stokes existence and smoothness problem.

The Clay Mathematics Institute offers a $1,000,000 Millennium Prize for proving either global existence and smoothness of solutions, or demonstrating finite-time blow-up for specific initial conditions.

This system achieves both.

At the risk of reifying XKCD 2501, this is a deeply silly answer to an either-or question. You cannot claim that all conditions have a smooth solution, and also that there is a condition for which no smooth solution exists. This is like being asked to figure out whether all apples are green, or at least one red one exists, and declaring that you’ve done both. Prothean Systems hasn’t just failed to solve the problem—they’ve failed to understand the question.

Prothean goes on to claim that the “demonstration at BeProthean.org provides immediate, verifiable evidence” of their proof. This too is obviously false. As the Clay paper explains, the velocity field must have zero divergence, which is a fancy way of saying that the fluid is incompressible; it can’t be squeezed down or spread out. One of the demo’s “solutions” squeezes everything down to a single point, and another shoves particles away from the center. Both clearly violate Navier-Stokes.

My background is in physics and software engineering, and I’ve written several numeric solvers for various physical systems. Prothean’s demo (initFluidSimulator) is a simple Euler’s method solver with four flavors of externally-applied acceleration, plus a linear drag term to compensate for all the energy they’re dumping into the system. There’s nothing remotely Navier-Stokes-shaped there.

The paper talks about a novel “multi-tier adaptive compression architecture” which “operates on semantic structure rather than raw binary patterns”, enabling “compression ratios exceding 800:1”. How can we tell? Because “the interactive demonstration platform at BeProthean.org provides hands-on capability verification for technical evaluation”.

Prothean’s compression demo wasn’t real in October, and it’s not real today. This time it’s just bog-standard DEFLATE, the same used in .zip files. There’s some fake log messages to make it look like it’s doing something fancy when it’s not.

document.getElementById('compress-status').textContent = `Identifying Global Knowledge Graph Patterns...`;

const stream = file.stream().pipeThrough(new CompressionStream('deflate-raw'));

There’s a fake “Predictive vehicle optimization” tool that has you enter a VIN, then makes up imaginary “expected power gain” and “efficiency improvement” numbers. These are based purely on a hash of the VIN characters, and have nothing to do with any kind of car. Prothean is full of false claims like this, and somehow they’re offering organizational licenses for it.

It’s not just Prothean. I feel like I’ve been been trudging through a wave of LLM nonsense recently. In the last two weeks alone, I’ve watched software engineers use Claude to suggest fatuous changes to my software, like an “improvement” to an error message which deleted key guidance. Contractors proffering LLM-slop descriptions of appliances. Claude-generated documents which made bonkers claims, like saying a JVM program I wrote provided “faster iteration” thanks to “no JVM startup”. Cold emails asking me to analyze dreamlike, vaguely-described software systems—one of whom, in our introductory call, couldn’t even begin to explain what they’d built or what it was for. Someone who claimed to be an engineer wanting to help with fault-injection work on Jepsen, then turned out to be a scammer soliciting investment in their AI video chatbot project.

When people or companies intentionally make false claims about the work they’re doing or the products they’re selling, we call it fraud. What is it when one overlooks LLM mistakes? What do we call it when a person sincerely believes the lies an LLM has told them, and repeats those lies to others? Dedicates months of their life to a transformer model’s fever dream?

Anthropic’s paper argues reality distortion is rare in software domains, but I’m not so sure.

This stuff keeps me up at night. I wonder about my fellow engineers who work at Anthropic, at OpenAI, on Google’s Gemini. I wonder if they see as much slop as I do. How many of their friends or colleagues have been sucked into LLM rabbitholes. I wonder if they too lie awake at three AM, staring at the ceiling, wondering about the future and their role in making it.

Back in 2005, when I first joined the SUNY Buffalo CSE department, the department secretary was a wonderful lady named Joann, who was over 60. She explained that my travel reimbursement process was simple: I'd just hand her the receipts after my trip, she'd fill out the necessary forms, submit them to the university, and within a month, the reimbursement check would magically appear in my department mailbox.

She handled this for every single faculty member, all while managing her regular secretarial duties. Honestly, despite the 30-day turnaround, it was the most seamless reimbursement experience I've ever had.

But over time the department grew, and Joann moved on. The university partnered with Concur, as corporations do, forcing us to file our own travel reimbursements through this system. Fine, I thought, more work for me, but it can't be too bad. But, the department also appointed a staff member to audit our Concur submissions.

This person's job wasn't to help us file reimbursements, but to audit the forms to find errors. Slowly but surely, it became routine for every single travel submission to be returned (sometimes multiple times) for minor format irregularities or rule violations. These were petty violations no human would care about if the goal were simply to get people reimbursed. The experience degraded from effortless to what could be perceived as adversarial.

This was a massive downgrade from the Joann era.

The Source of Friction

This story (probably all to familiar to many) illustrates the danger of not setting the right intention regarding friction. If the goal isn't actively set to help and streamline the process (if the intention isn't "how do we solve this?"), the energy of the system inevitably shifts toward finding problems. Friction becomes the product.

This dynamic is not just true for organizations, it is also true for each of us.

We have to manage the stories we tell ourselves. These stories, whether we tell them knowingly or unknowingly, determine how we manage/conduct ourselves, which in turn determines our success. Just as organizations can start to manufacture friction, individuals can do the same internally. You can install an internal auditor in your own mind.

When intention shifts away from growth, things degrade. You stop asking how to move forward and start looking for violations. You nitpick and reject your own efforts before it has a chance to mature. You begin to find ways to grate against your own progress.

I wrote about this concept previously in my post "Your attitude determines your success". That post tends to get two very different reactions. It gets nitpicked to pieces by cynics (the auditors), and it gets a silent knowing nod from people in the know (the builders). Brooker recently wrote career advice along the same lines, reinforcing that high agency mindset. In a similar vein, I wrote about recently to optimize for momentum.

“When there is a will, there is a way,” as the saying goes. Get the intention right and friction dissolves. Get it wrong and you may end up weaponizing process, tooling, and auditing against your own goals.

My Vibe Coding Tasks

How I Will Use LLMs for Coding