Build a Social Media Comment System: Kafka, MongoDB and Redis

This hands-on build applies the system design template to a real mini app: the comment section under an Instagram post or a YouTube video. Along the way you'll see three ideas you'll reuse constantly: a message queue as a buffer between services that run at different speeds, eventual consistency, and designs with separate read and write paths.

Two loads shape the design. A viral post brings a burst of comments, more than the database can write at once, so a queue absorbs the burst. And millions of readers put even more load on the database, so user-facing reads never hit it directly: they're served from a cache.

Architecture

• Write path: the user sends a comment to the Uploader → it's enqueued in Kafka → the Database Updater consumes it and inserts it into MongoDB → after the insert succeeds, it writes the same record into Redis.
• Read path: the page calls the Comment Fetcher, which reads the post's comments from Redis and returns them.

Data model

MongoDB: database socialnetwork, collection comments, documents { _id: ObjectId, post_id: string, comment: string }:

socialnetwork> db.comments.find()
[
  { _id: ObjectId("6466a681bec0bd23bec4ce12"), post_id: '1', comment: 'this is a great post thank you' }
]

Redis: one hash per post, key post:<post_id>, with each field a comment ID and each value the comment as JSON:

127.0.0.1:6379> HGETALL "post:1"
1) "6466a29c6fb786d66e584e5a"
2) "{"post_id": "1", "comment": "this is a great post thank you", "_id": "6466a29c6fb786d66e584e5a"}"
3) "6466a681bec0bd23bec4ce12"
4) "{"post_id": "1", "comment": "this is another great post thank you", "_id": "6466a681bec0bd23bec4ce12"}"

Kafka: the Uploader publishes each comment to a topic named comments.

Infrastructure: docker-compose.yml

services:
  kafka:
    image: apache/kafka:3.8.0        # single-node KRaft broker, no ZooKeeper needed
    ports: ["9092:9092"]
  redis:
    image: redis:7
    ports: ["6379:6379"]
  mongo:
    image: mongo:7
    ports: ["27017:27017"]

docker compose up starts Kafka on 9092, Redis on 6379 and MongoDB on 27017 (older Kafka setups also ran ZooKeeper for coordination; current Kafka uses its built-in KRaft consensus). Then pip install flask kafka-python pymongo redis. Optional CLIs for poking around: Kafka's console tools, redis-cli and mongosh.

The three services

Comment Uploader (write API)

# comment_uploader.py
import json
from flask import Flask, request
from kafka import KafkaProducer

app = Flask(__name__)
producer = KafkaProducer(bootstrap_servers="localhost:9092")

@app.route("/comments", methods=["POST"])
def upload_comment():
    data = request.get_json()
    message = {"post_id": data.get("post_id"), "comment": data.get("comment")}
    producer.send("comments", value=json.dumps(message).encode())
    return "Comment uploaded", 200        # respond immediately; storage happens asynchronously

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5001)

Database Updater (Kafka consumer)

# db_updater.py
import json
import redis
from bson import ObjectId
from kafka import KafkaConsumer
from pymongo import MongoClient

class JSONEncoder(json.JSONEncoder):          # make MongoDB's ObjectId JSON-serializable
    def default(self, o):
        return str(o) if isinstance(o, ObjectId) else super().default(o)

consumer = KafkaConsumer("comments", bootstrap_servers=["localhost:9092"], group_id="db-updater",
                         value_deserializer=lambda x: json.loads(x.decode("utf-8")))
collection = MongoClient("mongodb://localhost:27017/")["socialnetwork"]["comments"]
cache = redis.Redis(host="localhost", port=6379, db=0)

for message in consumer:
    comment = message.value
    result = collection.insert_one(comment)          # 1. permanent storage (adds _id)
    if result.acknowledged:                          # 2. then update the cache
        comment_id = str(result.inserted_id)
        cache.hset(f"post:{comment['post_id']}", comment_id, json.dumps(comment, cls=JSONEncoder))
        print(comment)
    else:
        print("Error: failed to insert comment into MongoDB")

Comment Fetcher (read API)

# comment_fetcher.py
import json
import redis
from flask import Flask, request

app = Flask(__name__)
cache = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)

@app.route("/comments", methods=["GET"])
def get_comments():
    post_id = request.args.get("post_id")
    comments = cache.hgetall(f"post:{post_id}")     # all comments for this post
    return {"comments": [json.loads(c) for c in comments.values()]}

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5002)

Run each in its own terminal: python comment_uploader.py (port 5001), python db_updater.py (waits for messages), python comment_fetcher.py (port 5002). Optionally watch the topic with kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic comments --from-beginning.

Testing it

curl -X POST http://localhost:5001/comments \
     -H "Content-Type: application/json" \
     -d '{"post_id":"1", "comment":"this is a great post thank you"}'

1. The Uploader logs "POST /comments HTTP/1.1" 200.
2. The Kafka console consumer (if running) shows {"post_id": "1", "comment": "this is a great post thank you"}.
3. db_updater.py prints the comment with its new _id.
4. db.comments.find() in mongosh shows the document.
5. HGETALL "post:1" in redis-cli shows the cached record.
6. http://localhost:5002/comments?post_id=1 returns {"comments": [{"_id": "6466a681…", "comment": "this is a great post thank you", "post_id": "1"}]}.

Load test: buffering and eventual consistency

Put the request body in post_data.json, then fire 10,000 requests with 100 concurrent connections using Apache Bench:

echo '{"post_id":"1", "comment":"this is a great post thank you"}' > post_data.json
ab -n 10000 -c 100 -p post_data.json -T 'application/json' http://127.0.0.1:5001/comments

While it runs, HGETALL "post:1" (or the read API) returns fewer comments than were sent: messages pile up in Kafka faster than the single consumer drains them. A few seconds later the consumer catches up, and Redis, the read API and MongoDB all agree. Two lessons:

• Eventual consistency: user A gets 200 immediately, yet user B may not see A's comment for a few seconds. The inconsistency disappears on its own, which is fine for a non-critical feature like comments. See consistency models.
• A message queue as a buffer: without Kafka, the service couldn't accept requests faster than the database can write. The queue holds messages in flight and lets you scale consumers separately. See message queues.

Taking it to production

• Order and paginate: store comment IDs in a sorted set per post (score = timestamp) and fetch pages with ZREVRANGE, instead of returning every comment from a hash. See pagination.
• Bound the cache: keep only recent or top comments per post in Redis with a TTL; on a miss, read MongoDB (indexed on post_id, created_at) and refill. See cache strategies.
• Make consumers idempotent: Kafka delivers at least once, so give each comment a client-generated ID and upsert, or a retry creates duplicates. See delivery semantics.
• Scale out: partition the topic by post_id (keeps per-post order) and run several updater instances in one consumer group. See Kafka.
• Read-your-writes: show a user's own new comment immediately from the client (optimistic UI) so the async lag isn't confusing.
• Moderation and counts: spam filtering as another consumer of the same topic; comment counts as a counter updated by the updater.

This template (separate write and read paths, a queue buffering writes, a cache serving reads) comes up again and again: feeds, likes, notifications, analytics events.