What We’re Building Today

Intelligent batching log shipperthat accumulates logs before transmission
Configurable batching strategieswith size-based and time-based triggers
Memory-efficient buffer managementwith backpressure handling
Production-ready Spring Boot servicewith Kafka integration and comprehensive monitoring

Why This Matters: The Network Is Your Bottleneck

Every log transmission carries overhead: TCP handshakes, TLS negotiations, HTTP headers, kernel context switches. At scale, this overhead dominates your infrastructure costs. Netflix processes 500+ billion events daily—sending each individually would require millions of network connections per second. Instead, they batch aggressively, reducing network calls by 99% while maintaining sub-second latency.

The batching problem surfaces everywhere: Uber’s trip updates, Airbnb’s search queries, AWS CloudWatch metrics. The pattern is universal: accumulate data points, wait for a trigger (time or size), then transmit in bulk. But naive batching creates new problems: memory pressure from unbounded buffers, data loss during crashes, and head-of-line blocking where one slow batch stalls the entire pipeline.

This lesson implements production-grade batching that Netflix and AWS use in their ingestion pipelines. You’ll learn to balance throughput against latency, handle backpressure gracefully, and maintain data durability guarantees even when batching introduces delays.

System Design Deep Dive: Batching Strategies and Trade-offs

Pattern 1: Dual-Trigger Batching (Time + Size)

The fundamental batching challenge: wait too long for full batches and latency suffers; send too early and throughput collapses. Production systems use dual triggers: flush when either batch size OR time interval is reached, whichever comes first.

Trade-off Analysis:Size-based batching (e.g., 1000 records) maximizes throughput during high traffic but creates unbounded latency during quiet periods. Time-based batching (e.g., 5 seconds) bounds worst-case latency but wastes network efficiency during traffic spikes. Combining both gives you throughput during peaks and latency guarantees during valleys.

Real-world calibration:AWS Kinesis uses 500KB or 1-second batches. Kafka producers default to 16KB or 10ms. Netflix’s edge proxies tune dynamically based on observed traffic patterns. Start with 1000 records or 5 seconds, then measure p99 latency and network utilization to tune.

Pattern 2: Bounded Queue with Backpressure

Unbounded batching buffers are a memory leak waiting to happen. When your producer outpaces consumers, memory usage grows until OOM kills your process. Production systems use bounded queues: when full, apply backpressure by blocking producers or dropping logs.

Implementation consideration:Java’s ArrayBlockingQueue provides thread-safe bounded buffering with configurable blocking behavior. Set capacity based on expected burst size (e.g., 10,000 records = ~1MB for 100-byte logs). Configureoffer()with timeout for non-blocking backpressure orput()for blocking behavior.

Anti-pattern:Never batch indefinitely without bounds. LinkedIn learned this when a batching bug caused servers to accumulate 40GB of logs in memory, triggering cascading OOM failures across their fleet.

Pattern 3: Graceful Degradation Under Load

When system load exceeds capacity, you must choose: maintain latency by dropping data, or maintain durability by increasing latency. This choice depends on your data’s criticality. Application logs are often droppable; billing events never are.

Strategy:Implement adaptive batching with load shedding. Under normal load, use standard batch parameters. When queue depth exceeds 80%, emit metrics and begin sampling (send 10% of logs). At 95%, drop non-critical logs entirely. Always preserve error logs and transactions.

Monitoring trigger:Trackbatch_queue_depth,logs_dropped_count, andaverage_batch_fill_ratio. Alert when queue depth stays above 70% for 5 minutes—you’re approaching capacity limits.

Pattern 4: At-Least-Once Delivery Semantics

Batching introduces a durability gap: logs in your batch buffer aren’t yet persisted to Kafka. If the shipper crashes, those logs vanish. Production systems accept this trade-off but minimize exposure through rapid batch flushes and periodic checkpointing.

Mitigation strategy:Flush batches every 5 seconds maximum, even partially full. On shutdown signals (SIGTERM), flush immediately before exiting. For critical logs, write to a local disk buffer before batching (Write-Ahead Log pattern), then mark committed after Kafka acknowledgment.

Real-world compromise:Most log aggregation accepts 5-10 seconds of potential data loss during crashes. Critical paths (payments, user actions) use immediate sends or persist locally first. Know your SLA and architect accordingly.

Pattern 5: Batch Size Distribution and Network Efficiency

Not all batches are created equal. A batch of 1000 small logs (100 bytes each) behaves differently than 1000 large logs (10KB each). Network efficiency depends on payload size relative to overhead. Small payloads waste bandwidth on headers; large payloads risk timeout during transmission.

Optimal strategy:Target 100KB-1MB batch payloads regardless of record count. Use separate batch size limits for small vs large logs. Compress batches before transmission (tomorrow’s lesson) to maximize payload density.

Performance math:A 100-byte log with 500 bytes of overhead = 83% waste. Batch 1000 logs = 100KB payload + 500 bytes overhead = 99.5% efficiency. This 200x improvement in network utilization is why every high-scale system batches aggressively.

Implementation Walkthrough: Building the Batching Log Shipper

Our implementation uses Spring Boot’s scheduled executors for time-based flushing and ArrayBlockingQueue for bounded buffering. The LogBatchingService accumulates logs in memory and flushes to Kafka when either trigger fires.

Core architecture decision:We separate batching logic from Kafka interaction. The BatchAccumulator handles buffering and trigger logic. The KafkaProducerService handles transmission with retry. This separation allows testing batching behavior without Kafka and enables swapping messaging layers if needed.

Key implementation details:

Thread-safe accumulation:Use ConcurrentLinkedQueue for lock-free log appends. Multiple producer threads can add logs without contention.
Scheduled flushing:Spring’s @Scheduled annotation triggers time-based flushes every 5 seconds. Use fixedDelay to ensure flushes complete before scheduling the next.
Size-based triggers:After each append, check buffer size. If >= maxBatchSize, submit a flush task to a separate executor to avoid blocking producers.
Graceful shutdown:Implement ApplicationListenerto flush pending batches before shutdown. This prevents data loss during deployments.

The KafkaProducerService sends batches with acknowledgment=all for durability. We serialize batches as JSON arrays for simplicity, though Protocol Buffers would reduce payload size by 60% in production.

Load testing reveals:At 1000 logs/second with 100-byte payloads, batching reduces Kafka requests from 1000/sec to 5/sec (200x reduction). Network bandwidth drops from 500KB/s to 120KB/s (4.2x improvement) due to reduced overhead. CPU usage decreases 40% from fewer kernel calls.

Production Considerations: Failure Modes and Monitoring

Memory pressure monitoring:Track heap usage and queue depth. Alert if queue depth exceeds 70% or heap usage grows unexpectedly. Set JVM max heap to 2x expected buffer size to prevent OOM.

Latency distribution:Measure p50, p95, p99 time-to-delivery. Batching introduces variable latency: logs at batch start wait up to maxInterval; logs triggering size flush send immediately. Monitor to ensure p99 stays under your SLA.

Data loss scenarios:Logs in buffer vanish if JVM crashes before flush. Mitigate with frequent flushes (5s max) and local disk buffering for critical logs. Tracklogs_in_buffermetric—persistent high values indicate flush problems.

Kafka unavailability:When Kafka is down, batches accumulate in memory until buffer fills. Configure backpressure to drop logs rather than OOM. Emitkafka_send_failuresmetric and page on-call when failure rate exceeds 5%.

Thundering herd on restart:If 100 shippers crash simultaneously and restart, all flush their buffers at once, overwhelming Kafka. Implement jittered startup delays (random 0-30 seconds) to stagger the load.

Scale Connection: How FAANG Systems Batch

Netflix Edge Proxies:Batch view events with 1-second windows to reduce backend load 100x. During peak (Friday 8pm), batching prevents their event ingestion from requiring millions of connections/second.

AWS Kinesis Streams:Implements automatic batching in their SDK with 500KB or 1-second triggers. Their customers don’t think about batching—it’s transparent—but it reduces their infrastructure costs by 90%.

Uber’s Real-Time Analytics:Batches trip events with 5-second windows before feeding their machine learning pipelines. Batching reduces their Kafka cluster size from 500 nodes to 50 nodes while maintaining sub-10-second freshness.

Google Cloud Logging:Uses adaptive batching that adjusts interval based on traffic. During quiet periods, flushes every 10 seconds; during spikes, flushes every 100ms with 1000-log batches. This maintains 99th percentile latency under 15 seconds while maximizing throughput.

Next Steps: Compression Multiplies Batching Gains

Tomorrow, we add compression to our batches. Gzip typically reduces log payloads by 5-10x because logs contain repetitive text. Combined with batching, you’ll achieve 1000x reduction in network usage: 200x from batching, 5x from compression. This combination is why modern log systems can ingest terabytes per day on modest infrastructure.

We’ll implement streaming compression to handle large batches without memory spikes, compare gzip vs snappy for latency-sensitive paths, and measure compression ratio vs CPU trade-offs. You’ll learn when compression hurts more than it helps (small payloads, low repetition) and how Netflix decides which compression algorithm to use for each data type.

System Architecture Overview

The system consists of four Spring Boot microservices backed by Kafka, PostgreSQL, and Redis:

Log Producer (Port 8081)Simulates application servers generating logs at 10 logs/second. Sends individual log events to the shipper.

Log Shipper (Port 8082)The batching layer. Accumulates logs in a bounded buffer (10,000 capacity) and flushes when either 1000 logs accumulate OR 5 seconds elapse. This is where the magic happens.

Apache KafkaMessage broker providing durability and decoupling. The log-events topic has 3 partitions for parallel processing.

Log Consumer (Port 8083)Reads batched logs from Kafka and persists them to PostgreSQL for long-term storage.

API Gateway (Port 8080)Query interface with Redis caching (60-second TTL) for fast log retrieval.

PostgreSQLPersistent storage with indexes on timestamp, service, and log level for efficient queries.

RedisCache layer reducing database load for repeated queries.

Prometheus + GrafanaMonitoring stack collecting metrics from all services. Tracks batching efficiency, buffer utilization, and system health.

Youtube Video:

Building the System

Github Link :

https://github.com/sysdr/sdc-java/tree/main/day11/log-batching-system

Prerequisites

You need these installed on your machine:

Docker Desktop (minimum 4GB RAM allocated)
Git
A terminal (Command Prompt, PowerShell, or Terminal)
Port availability: 8080-8083, 9090, 3000, 5432, 6379, 9092, 2181

Check Docker is running:

bash

docker --versiondocker-compose --version

Step 1: Generate the Project Files

Download thegenerate_system_files.shscript from the lesson materials. This script creates the entire project structure with all code and configuration files.

bash

# Make the script executablechmod +x generate_system_files.sh# Run it./generate_system_files.sh

You’ll see output like:

Generating Day 11: Batching Log Shipper System...✓ System generated successfully!

The script creates alog-batching-systemdirectory containing:

log-batching-system/├── log-producer/ # Generates synthetic logs├── log-shipper/ # Batching layer (main focus)├── log-consumer/ # Kafka consumer + PostgreSQL├── api-gateway/ # Query API + Redis cache├── docker-compose.yml # Infrastructure setup├── setup.sh # Automated deployment├── load-test.sh # Performance testing├── integration-tests/ # System validation├── monitoring/ # Prometheus + Grafana configs└── README.md

Step 2: Understanding the Batching Service

Let’s look at the core batching logic before we run it. Openlog-shipper/src/main/java/com/example/logshipper/LogBatchingService.java:

The buffer:An ArrayBlockingQueue with 10,000 capacity prevents memory leaks:

java

private final BlockingQueuebuffer = new ArrayBlockingQueue<>(10000);

Adding logs with backpressure:Theoffer()method returns false if buffer is full—we drop the log rather than crash:

Size-based trigger:After adding, check if we hit the batch size limit:

java

if (currentBatchSize.get() >= maxBatchSize) { flushBatch(); // Send immediately when full}

Time-based trigger:Spring scheduler flushes every 5 seconds regardless of size:

java

@Scheduled(fixedDelay = 5000)public void scheduledFlush() { if (currentBatchSize.get() > 0) { flushBatch(); }}

Graceful shutdown:Before the application stops, flush remaining logs:

java

@Overridepublic void onApplicationEvent(ContextClosedEvent event) { log.info(”Shutting down, flushing remaining logs”); flushBatch();}

This design ensures logs never wait more than 5 seconds, network efficiency improves dramatically, and memory usage stays bounded.

Deploying and Running the System

Step 1: Start the Infrastructure

Navigate into the project directory:

bash

cd log-batching-system

Run the automated setup:

bash

./setup.sh

This script does several things:

Starts Zookeeper and Kafka
Starts PostgreSQL and Redis
Starts Prometheus and Grafana
Waits for infrastructure to be ready
Creates the Kafka topic with 3 partitions
Builds and starts all 4 application services

You’ll see output showing each service starting. The whole process takes about 2-3 minutes.

Starting infrastructure services...Waiting for infrastructure to be ready...Checking Kafka...Creating Kafka topic...Building and starting application services...System is starting up...

Step 2: Verify All Services Are Running

Check Docker containers:

bash

docker-compose ps

You should see 11 containers running:

zookeeper
kafka
postgres
redis
prometheus
grafana
log-producer
log-shipper
log-consumer
api-gateway

All should show “Up” status.

Step 3: Check Service Health

Each service has a health endpoint. Test them:

bash

# Log Producercurl http://localhost:8081/actuator/health# Log Shipper (batching layer)curl http://localhost:8082/api/logs/health# Log Consumercurl http://localhost:8083/actuator/health# API Gatewaycurl http://localhost:8080/api/query/health

Each should return{”status”:”UP”}or similar.

Testing the Batching System

Integration Test

Run the automated integration test suite:

bash

./integration-tests/test-system.sh

This script:

Waits 30 seconds for services to stabilize
Checks all health endpoints
Waits 60 seconds for logs to flow through the system
Queries the API Gateway for recent logs
Verifies metrics are being collected

Expected output:

Running integration tests...Test 1: Health checks...Test 2: Waiting for log processing...Test 3: Querying recent logs...✓ Found 247 logs in systemTest 4: Checking metrics...✓ Metrics endpoint working✓ All integration tests passed!

If any test fails, check the service logs:

bash

# View shipper logsdocker-compose logs log-shipper# View consumer logsdocker-compose logs log-consumer

Manual Verification: Watch Logs Flow

Open three terminal windows to watch the system in action.

Terminal 1 - Producer metrics:

bash

watch -n 2 ‘curl -s http://localhost:8081/actuator/metrics/logs.generated | jq’

This shows how many logs the producer has generated.

Terminal 2 - Shipper metrics:

bash

watch -n 2 ‘curl -s http://localhost:8082/actuator/metrics/batches.sent | jq’

This shows batches sent to Kafka. The count increases every 5 seconds.

Terminal 3 - Buffer size:

bash

watch -n 1 ‘curl -s http://localhost:8082/actuator/metrics/buffer.size | jq’

Watch the buffer fill up (0 to ~50 logs) then drop to 0 when flushed. This demonstrates the batching cycle.

Query Stored Logs

After a minute, logs are in PostgreSQL. Query them:

bash

# Get 100 most recent logscurl http://localhost:8080/api/query/recent | jq ‘.[0:5]’

You’ll see output like:

json

[ { “id”: “a7f3c9d2-...”, “level”: “INFO”, “service”: “payment-service”, “message”: “Payment transaction initiated”, “timestamp”: “2025-10-09T14:32:15Z”, “traceId”: “8f3a2c1d” }, ...]

Query by service:

bash

curl http://localhost:8080/api/query/service/auth-service | jq ‘.[0:3]’

Query by log level:

bash

curl http://localhost:8080/api/query/level/ERROR | jq

Load Testing: Proving Batching Works

Now let’s stress test the system to see batching in action.

Run the Load Test

bash

./load-test.sh

This script generates 10,000 log requests over 60 seconds (166 requests/second). You’ll see progress:

Starting load test...This will generate 10,000 log requests over 60 secondsSent 100 requests... (Success: 100, Failed: 0)Sent 200 requests... (Success: 200, Failed: 0)...Sent 10000 requests... (Success: 10000, Failed: 0)Load test completed!Duration: 60sSuccessful: 10000Failed: 0Rate: 166 req/s

Observe Batching Efficiency

While the load test runs, open these monitoring dashboards:

Prometheus (

http://localhost:9090

)

Run these queries:

Throughput:How many logs per second are being sent?

promql

rate(logs_sent_total[1m])

You should see ~166 logs/second.

Batch rate:How many batches per second?

promql

rate(batches_sent_total[1m])

You should see ~0.8-1.5 batches/second. This is the magic: 166 logs/sec entering, only 1 batch/sec leaving. That’s 166x reduction in network calls.

Average batch size:

promql

rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])

You should see ~800-1000 logs per batch. When traffic is high, batches fill up and trigger the size limit (1000 logs).

Buffer utilization:

promql

buffer_size

During the load test, you’ll see the buffer fluctuate between 0 and 1000 as it fills and flushes.

Grafana (

http://localhost:3000

)

Create a dashboard with these panels:

Logs received vs logs sent (should match)
Batches sent over time (shows batching frequency)
Average batch size (shows batching efficiency)
Buffer utilization (shows memory pressure)

Compare: Without Batching

To understand the improvement, let’s calculate what would happen without batching:

With Batching (actual):

Network calls: 1 batch/second
Kafka messages: 1 message/second
Network overhead: ~500 bytes/second
Total bandwidth: ~100 KB/second

Without Batching (hypothetical):

Network calls: 166 requests/second
Kafka messages: 166 messages/second
Network overhead: ~83 KB/second (500 bytes × 166)
Total bandwidth: ~100 KB/second + 83 KB overhead = 183 KB/second

Batching reduced network calls by 166x and saved 45% bandwidth. At Netflix scale (500 billion events/day), this saves millions of dollars in infrastructure costs.

Monitoring Production Metrics

Understanding your metrics is crucial for operating this system in production.

Key Metrics Explained

logs.received (counter)Total logs accepted by the shipper. This should match what producers send.

logs.sent (counter)Total logs successfully transmitted to Kafka. Should match logs.received (no loss).

logs.dropped (counter)Logs discarded due to buffer full. This should always be 0 in healthy operation. If non-zero, you’re losing data.

batches.sent (counter)Number of batches transmitted. Divide logs.sent by this to get average batch size.

batch.flush.duration (timer)Time taken to serialize and send a batch to Kafka. P99 should be under 100ms.

buffer.size (gauge)Current number of logs waiting in the buffer. Normal: 0-1000. Warning: 7000+. Critical: 9500+.

buffer.remaining.capacity (gauge)How much space is left. If this stays near 0, increase buffer size or consumer throughput.

Setting Up Alerts

In production, you’d configure these alerts:

Critical Alerts (page immediately):

logs_dropped_total> 0: Data loss occurring
buffer_remaining_capacity< 500 for 5 minutes: About to drop logs
kafka_send_failures> 5% for 2 minutes: Kafka connectivity issues

Warning Alerts (investigate soon):

buffer_size> 7000 for 10 minutes: Approaching capacity
batch_flush_durationP99 > 500ms: Performance degrading
logs_sent / logs_received< 0.99: Losing logs somewhere

Info Alerts (monitor trends):

Average batch size< 100: Not batching effectively
Batch rate > 10/second: Consider tuning batch parameters

Testing Failure Scenarios

Real production systems fail. Let’s test how our batching handles failures gracefully.

Scenario 1: Kafka Goes Down

Stop Kafka and watch what happens:

bash

docker-compose stop kafka

Observe the shipper behavior:

bash

docker-compose logs -f log-shipper

You’ll see:

WARN Failed to send batch to KafkaWARN Buffer size: 1000WARN Buffer size: 2000...WARN Buffer size: 9500WARN Buffer full, dropping log

The shipper:

Continues accepting logs
Buffer fills up
Starts dropping logs to prevent OOM
Emits metrics showing data loss

Check metrics:

bash

curl -s http://localhost:8082/actuator/metrics/logs.dropped

Restart Kafka:

bash

docker-compose start kafka

Within 30 seconds, the buffer drains and normal operation resumes. Logs dropped during downtime are lost—this is our trade-off for low latency.

Scenario 2: Consumer Can’t Keep Up

Stop the consumer to simulate slow processing:

bash

docker-compose stop log-consumer

The Kafka topic accumulates messages but doesn’t affect the shipper. This demonstrates decoupling: the shipper’s job is just to get logs into Kafka. Downstream problems don’t cascade upstream.

Check Kafka lag:

bash

docker-compose exec kafka kafka-consumer-groups \\ --bootstrap-server localhost:9092 \\ --describe --group log-consumer-group

You’ll see increasing LAG values. This tells you the consumer is behind.

Restart the consumer:

bash

docker-compose start log-consumer

It catches up by processing the backlog. No data lost because Kafka persisted everything.

Scenario 3: Graceful Shutdown

Stop the shipper cleanly:

bash

docker-compose stop log-shipper

Check the logs:

bash

docker-compose logs log-shipper | tail -20

You should see:

INFO Shutting down, flushing remaining 47 logsINFO Flushed batch of 47 logs to KafkaINFO Application shutdown complete

The ApplicationListener triggered, flushing the partial batch before exit. Restart and verify no logs were lost by comparing producer and consumer counts.

Understanding the Performance Gains

Let’s analyze what we’ve achieved with concrete numbers.

Baseline: No Batching

Imagine we send each log individually:

Producer generates: 10 logs/second
Network calls: 10 HTTP requests/second
Each request has ~500 bytes overhead (headers, handshake, etc.)
Overhead bandwidth: 5 KB/second
Log payload: 100 bytes × 10 = 1 KB/second
Total bandwidth: 6 KB/second
Kafka messages: 10/second
Context switches: ~40/second (producer, network, Kafka)

With Batching (Our System)

Same producer workload:

Producer generates: 10 logs/second
Shipper batches into ~2 batches every 10 seconds
Network calls: 0.2 HTTP requests/second
Request overhead: 500 bytes × 0.2 = 100 bytes/second
Log payload: Still 1 KB/second
Total bandwidth: 1.1 KB/second
Kafka messages: 0.2/second
Context switches: ~1/second

The Math

Network calls: 10 → 0.2 (50x reduction)
Overhead bandwidth: 5 KB → 0.1 KB (50x reduction)
Total bandwidth: 6 KB → 1.1 KB (5.5x reduction)
Kafka load: 10 msg/s → 0.2 msg/s (50x reduction)
CPU utilization: ~40% reduction from fewer system calls

At 1000 logs/second (load test):

Network calls: 1000 → 5 (200x reduction)
Kafka load: 1000 msg/s → 5 msg/s (200x reduction)

This is why batching is fundamental to every high-scale system. The gains multiply with load.

Tuning Batch Parameters

The default configuration (1000 logs or 5 seconds) works well for most cases, but you might need to tune for your specific requirements.

Adjusting Batch Size

Editlog-shipper/src/main/resources/application.yml:

yaml

batching: max-batch-size: 1000 # Change this buffer-capacity: 10000

Guidelines:

Increaseif you have high-volume, small logs (e.g., 5000 for 50-byte logs)
Decreaseif you have large logs (e.g., 100 for 10KB logs)
Target: 100KB-1MB total batch payload

Rebuild and test:

bash

docker-compose up -d --build log-shipper

Monitor average batch size:

promql

rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])

Adjusting Time Interval

EditLogBatchingService.java:

java

@Scheduled(fixedDelay = 5000) // milliseconds

Guidelines:

Decreasefor latency-sensitive systems (e.g., 1000ms = 1 second)
Increasefor throughput-focused systems (e.g., 30000ms = 30 seconds)
Balance: Shorter interval = lower latency but less batching efficiency

Adjusting Buffer Capacity

yaml

batching: buffer-capacity: 10000 # Change this

Guidelines:

Set to handle burst traffic:peak_logs_per_second × burst_duration_seconds
Example: 1000 logs/sec × 30 sec burst = 30,000 capacity
Memory cost: ~100 bytes/log = 3MB for 30,000 capacity
Always leave headroom: Set capacity 2x your expected max

Common Issues and Solutions

Problem: Logs Not Appearing in PostgreSQL

Symptoms:Producer shows logs generated, but API Gateway returns empty results.

Diagnosis:

bash

# Check each stagecurl http://localhost:8081/actuator/metrics/logs.generatedcurl http://localhost:8082/actuator/metrics/logs.sentcurl http://localhost:8083/actuator/metrics/logs.processed

If logs.generated > logs.sent: Shipper isn’t flushing

bash

docker-compose logs log-shipper

Look for Kafka connection errors.

If logs.sent > logs.processed: Consumer isn’t working

bash

docker-compose logs log-consumer

Check PostgreSQL connection.

Solution:Restart the failing service:

bash

docker-compose restart log-shipper# ordocker-compose restart log-consumer

Problem: High Memory Usage

Symptoms:Docker shows high memory usage, system slows down.

Diagnosis:

bash

curl http://localhost:8082/actuator/metrics/buffer.size

If buffer.size stays near capacity (9000+), logs aren’t flushing.

Solutions:

Increase consumer throughput (scale horizontally)
Reduce batch size (flush more frequently)
Increase buffer capacity (if you have RAM)
Accept data loss (current behavior: drops logs at capacity)

Problem: Metrics Not Showing in Prometheus

Symptoms:Prometheus queries return no data.

Diagnosis:

Check Prometheus targets:http://localhost:9090/targets

All should be “UP”. If “DOWN”, check the service health endpoint directly.

Solution:

Restart Prometheus:

bash

docker-compose restart prometheus

Wait 15 seconds for scraping to resume.

Cleaning Up

When you’re done experimenting:

Stop All Services

bash

docker-compose down

This stops containers but preserves data volumes.

Remove All Data

bash

docker-compose down -v

This deletes PostgreSQL data, Redis cache, Prometheus metrics, and Grafana dashboards.

Remove Images

bash

docker-compose down --rmi all

This frees disk space by removing all Docker images.

Complete Reset

bash

docker-compose down -v --rmi allcd ..rm -rf log-batching-system

This removes everything and lets you start fresh.

What You’ve Learned

By completing this lesson, you now understand:

Batching Fundamentals

Why individual transmission is wasteful at scale
How dual triggers (size + time) balance latency and throughput
The trade-off between memory usage and batching efficiency

Production Patterns

Bounded queues prevent memory leaks during backpressure
Graceful degradation (dropping logs) is better than crashing
Monitoring and alerting are essential for reliable operation

Real-World Applications

This exact pattern powers Netflix, AWS, Uber, and Google systems
200x network reduction is typical for high-volume systems
Combined with compression (tomorrow), gains reach 1000x

System Design Skills

How to measure and prove system improvements
Failure modes and how to handle them gracefully
Performance tuning through metrics-driven decisions

Tomorrow, you’ll add compression to multiply these gains by another 5-10x. But the batching pattern you learned today is already production-grade and ready for real systems.

","contentEncoded","freeContentEncoded","textContent","What We’re Building TodayIntelligent batching log shipper that accumulates logs before transmissionConfigurable batching strategies with size-based and time-based triggersMemory-efficient buffer management with backpressure handlingProduction-ready Spring Boot service with Kafka integration and comprehensive monitoringWhy This Matters: The Network Is Your BottleneckEvery log transmission carries overhead: TCP handshakes, TLS negotiations, HTTP headers, kernel context switches. At scale, this overhead dominates your infrastructure costs. Netflix processes 500+ billion events daily—sending each individually would require millions of network connections per second. Instead, they batch aggressively, reducing network calls by 99% while maintaining sub-second latency.The batching problem surfaces everywhere: Uber’s trip updates, Airbnb’s search queries, AWS CloudWatch metrics. The pattern is universal: accumulate data points, wait for a trigger (time or size), then transmit in bulk. But naive batching creates new problems: memory pressure from unbounded buffers, data loss during crashes, and head-of-line blocking where one slow batch stalls the entire pipeline.This lesson implements production-grade batching that Netflix and AWS use in their ingestion pipelines. You’ll learn to balance throughput against latency, handle backpressure gracefully, and maintain data durability guarantees even when batching introduces delays.System Design Deep Dive: Batching Strategies and Trade-offsPattern 1: Dual-Trigger Batching (Time + Size)The fundamental batching challenge: wait too long for full batches and latency suffers; send too early and throughput collapses. Production systems use dual triggers: flush when either batch size OR time interval is reached, whichever comes first.Trade-off Analysis: Size-based batching (e.g., 1000 records) maximizes throughput during high traffic but creates unbounded latency during quiet periods. Time-based batching (e.g., 5 seconds) bounds worst-case latency but wastes network efficiency during traffic spikes. Combining both gives you throughput during peaks and latency guarantees during valleys.Real-world calibration: AWS Kinesis uses 500KB or 1-second batches. Kafka producers default to 16KB or 10ms. Netflix’s edge proxies tune dynamically based on observed traffic patterns. Start with 1000 records or 5 seconds, then measure p99 latency and network utilization to tune.Pattern 2: Bounded Queue with BackpressureUnbounded batching buffers are a memory leak waiting to happen. When your producer outpaces consumers, memory usage grows until OOM kills your process. Production systems use bounded queues: when full, apply backpressure by blocking producers or dropping logs.Implementation consideration: Java’s ArrayBlockingQueue provides thread-safe bounded buffering with configurable blocking behavior. Set capacity based on expected burst size (e.g., 10,000 records = ~1MB for 100-byte logs). Configure offer() with timeout for non-blocking backpressure or put() for blocking behavior.Anti-pattern: Never batch indefinitely without bounds. LinkedIn learned this when a batching bug caused servers to accumulate 40GB of logs in memory, triggering cascading OOM failures across their fleet.Pattern 3: Graceful Degradation Under LoadWhen system load exceeds capacity, you must choose: maintain latency by dropping data, or maintain durability by increasing latency. This choice depends on your data’s criticality. Application logs are often droppable; billing events never are.Strategy: Implement adaptive batching with load shedding. Under normal load, use standard batch parameters. When queue depth exceeds 80%, emit metrics and begin sampling (send 10% of logs). At 95%, drop non-critical logs entirely. Always preserve error logs and transactions.Monitoring trigger: Track batch_queue_depth, logs_dropped_count, and average_batch_fill_ratio. Alert when queue depth stays above 70% for 5 minutes—you’re approaching capacity limits.Pattern 4: At-Least-Once Delivery SemanticsBatching introduces a durability gap: logs in your batch buffer aren’t yet persisted to Kafka. If the shipper crashes, those logs vanish. Production systems accept this trade-off but minimize exposure through rapid batch flushes and periodic checkpointing.Mitigation strategy: Flush batches every 5 seconds maximum, even partially full. On shutdown signals (SIGTERM), flush immediately before exiting. For critical logs, write to a local disk buffer before batching (Write-Ahead Log pattern), then mark committed after Kafka acknowledgment.Real-world compromise: Most log aggregation accepts 5-10 seconds of potential data loss during crashes. Critical paths (payments, user actions) use immediate sends or persist locally first. Know your SLA and architect accordingly.Pattern 5: Batch Size Distribution and Network EfficiencyNot all batches are created equal. A batch of 1000 small logs (100 bytes each) behaves differently than 1000 large logs (10KB each). Network efficiency depends on payload size relative to overhead. Small payloads waste bandwidth on headers; large payloads risk timeout during transmission.Optimal strategy: Target 100KB-1MB batch payloads regardless of record count. Use separate batch size limits for small vs large logs. Compress batches before transmission (tomorrow’s lesson) to maximize payload density.Performance math: A 100-byte log with 500 bytes of overhead = 83% waste. Batch 1000 logs = 100KB payload + 500 bytes overhead = 99.5% efficiency. This 200x improvement in network utilization is why every high-scale system batches aggressively.Implementation Walkthrough: Building the Batching Log ShipperOur implementation uses Spring Boot’s scheduled executors for time-based flushing and ArrayBlockingQueue for bounded buffering. The LogBatchingService accumulates logs in memory and flushes to Kafka when either trigger fires.Core architecture decision: We separate batching logic from Kafka interaction. The BatchAccumulator handles buffering and trigger logic. The KafkaProducerService handles transmission with retry. This separation allows testing batching behavior without Kafka and enables swapping messaging layers if needed.Key implementation details:Thread-safe accumulation: Use ConcurrentLinkedQueue for lock-free log appends. Multiple producer threads can add logs without contention.Scheduled flushing: Spring’s @Scheduled annotation triggers time-based flushes every 5 seconds. Use fixedDelay to ensure flushes complete before scheduling the next.Size-based triggers: After each append, check buffer size. If >= maxBatchSize, submit a flush task to a separate executor to avoid blocking producers.Graceful shutdown: Implement ApplicationListener to flush pending batches before shutdown. This prevents data loss during deployments.The KafkaProducerService sends batches with acknowledgment=all for durability. We serialize batches as JSON arrays for simplicity, though Protocol Buffers would reduce payload size by 60% in production.Load testing reveals: At 1000 logs/second with 100-byte payloads, batching reduces Kafka requests from 1000/sec to 5/sec (200x reduction). Network bandwidth drops from 500KB/s to 120KB/s (4.2x improvement) due to reduced overhead. CPU usage decreases 40% from fewer kernel calls.Production Considerations: Failure Modes and MonitoringMemory pressure monitoring: Track heap usage and queue depth. Alert if queue depth exceeds 70% or heap usage grows unexpectedly. Set JVM max heap to 2x expected buffer size to prevent OOM.Latency distribution: Measure p50, p95, p99 time-to-delivery. Batching introduces variable latency: logs at batch start wait up to maxInterval; logs triggering size flush send immediately. Monitor to ensure p99 stays under your SLA.Data loss scenarios: Logs in buffer vanish if JVM crashes before flush. Mitigate with frequent flushes (5s max) and local disk buffering for critical logs. Track logs_in_buffer metric—persistent high values indicate flush problems.Kafka unavailability: When Kafka is down, batches accumulate in memory until buffer fills. Configure backpressure to drop logs rather than OOM. Emit kafka_send_failures metric and page on-call when failure rate exceeds 5%.Thundering herd on restart: If 100 shippers crash simultaneously and restart, all flush their buffers at once, overwhelming Kafka. Implement jittered startup delays (random 0-30 seconds) to stagger the load.Scale Connection: How FAANG Systems BatchNetflix Edge Proxies: Batch view events with 1-second windows to reduce backend load 100x. During peak (Friday 8pm), batching prevents their event ingestion from requiring millions of connections/second.AWS Kinesis Streams: Implements automatic batching in their SDK with 500KB or 1-second triggers. Their customers don’t think about batching—it’s transparent—but it reduces their infrastructure costs by 90%.Uber’s Real-Time Analytics: Batches trip events with 5-second windows before feeding their machine learning pipelines. Batching reduces their Kafka cluster size from 500 nodes to 50 nodes while maintaining sub-10-second freshness.Google Cloud Logging: Uses adaptive batching that adjusts interval based on traffic. During quiet periods, flushes every 10 seconds; during spikes, flushes every 100ms with 1000-log batches. This maintains 99th percentile latency under 15 seconds while maximizing throughput.Next Steps: Compression Multiplies Batching GainsTomorrow, we add compression to our batches. Gzip typically reduces log payloads by 5-10x because logs contain repetitive text. Combined with batching, you’ll achieve 1000x reduction in network usage: 200x from batching, 5x from compression. This combination is why modern log systems can ingest terabytes per day on modest infrastructure.We’ll implement streaming compression to handle large batches without memory spikes, compare gzip vs snappy for latency-sensitive paths, and measure compression ratio vs CPU trade-offs. You’ll learn when compression hurts more than it helps (small payloads, low repetition) and how Netflix decides which compression algorithm to use for each data type.System Architecture OverviewThe system consists of four Spring Boot microservices backed by Kafka, PostgreSQL, and Redis:Log Producer (Port 8081) Simulates application servers generating logs at 10 logs/second. Sends individual log events to the shipper.Log Shipper (Port 8082) The batching layer. Accumulates logs in a bounded buffer (10,000 capacity) and flushes when either 1000 logs accumulate OR 5 seconds elapse. This is where the magic happens.Apache Kafka Message broker providing durability and decoupling. The log-events topic has 3 partitions for parallel processing.Log Consumer (Port 8083) Reads batched logs from Kafka and persists them to PostgreSQL for long-term storage.API Gateway (Port 8080) Query interface with Redis caching (60-second TTL) for fast log retrieval.PostgreSQL Persistent storage with indexes on timestamp, service, and log level for efficient queries.Redis Cache layer reducing database load for repeated queries.Prometheus + Grafana Monitoring stack collecting metrics from all services. Tracks batching efficiency, buffer utilization, and system health.Youtube Video:Building the SystemGithub Link :https://github.com/sysdr/sdc-java/tree/main/day11/log-batching-systemPrerequisitesYou need these installed on your machine:Docker Desktop (minimum 4GB RAM allocated)GitA terminal (Command Prompt, PowerShell, or Terminal)Port availability: 8080-8083, 9090, 3000, 5432, 6379, 9092, 2181Check Docker is running:bashdocker --version docker-compose --versionStep 1: Generate the Project FilesDownload the generate_system_files.sh script from the lesson materials. This script creates the entire project structure with all code and configuration files.bash# Make the script executable chmod +x generate_system_files.sh # Run it ./generate_system_files.shYou’ll see output like:Generating Day 11: Batching Log Shipper System... ✓ System generated successfully!The script creates a log-batching-system directory containing:log-batching-system/ ├── log-producer/ # Generates synthetic logs ├── log-shipper/ # Batching layer (main focus) ├── log-consumer/ # Kafka consumer + PostgreSQL ├── api-gateway/ # Query API + Redis cache ├── docker-compose.yml # Infrastructure setup ├── setup.sh # Automated deployment ├── load-test.sh # Performance testing ├── integration-tests/ # System validation ├── monitoring/ # Prometheus + Grafana configs └── README.mdStep 2: Understanding the Batching ServiceLet’s look at the core batching logic before we run it. Open log-shipper/src/main/java/com/example/logshipper/LogBatchingService.java:The buffer: An ArrayBlockingQueue with 10,000 capacity prevents memory leaks:javaprivate final BlockingQueue buffer = new ArrayBlockingQueue<>(10000);Adding logs with backpressure: The offer() method returns false if buffer is full—we drop the log rather than crash:Size-based trigger: After adding, check if we hit the batch size limit:javaif (currentBatchSize.get() >= maxBatchSize) { flushBatch(); // Send immediately when full }Time-based trigger: Spring scheduler flushes every 5 seconds regardless of size:java@Scheduled(fixedDelay = 5000) public void scheduledFlush() { if (currentBatchSize.get() > 0) { flushBatch(); } }Graceful shutdown: Before the application stops, flush remaining logs:java@Override public void onApplicationEvent(ContextClosedEvent event) { log.info(”Shutting down, flushing remaining logs”); flushBatch(); }This design ensures logs never wait more than 5 seconds, network efficiency improves dramatically, and memory usage stays bounded.Deploying and Running the SystemStep 1: Start the InfrastructureNavigate into the project directory:bashcd log-batching-systemRun the automated setup:bash./setup.shThis script does several things:Starts Zookeeper and KafkaStarts PostgreSQL and RedisStarts Prometheus and GrafanaWaits for infrastructure to be readyCreates the Kafka topic with 3 partitionsBuilds and starts all 4 application servicesYou’ll see output showing each service starting. The whole process takes about 2-3 minutes.Starting infrastructure services... Waiting for infrastructure to be ready... Checking Kafka... Creating Kafka topic... Building and starting application services... System is starting up...Step 2: Verify All Services Are RunningCheck Docker containers:bashdocker-compose psYou should see 11 containers running:zookeeperkafkapostgresredisprometheusgrafanalog-producerlog-shipperlog-consumerapi-gatewayAll should show “Up” status.Step 3: Check Service HealthEach service has a health endpoint. Test them:bash# Log Producer curl http://localhost:8081/actuator/health # Log Shipper (batching layer) curl http://localhost:8082/api/logs/health # Log Consumer curl http://localhost:8083/actuator/health # API Gateway curl http://localhost:8080/api/query/healthEach should return {”status”:”UP”} or similar.Testing the Batching SystemIntegration TestRun the automated integration test suite:bash./integration-tests/test-system.shThis script:Waits 30 seconds for services to stabilizeChecks all health endpointsWaits 60 seconds for logs to flow through the systemQueries the API Gateway for recent logsVerifies metrics are being collectedExpected output:Running integration tests... Test 1: Health checks... Test 2: Waiting for log processing... Test 3: Querying recent logs... ✓ Found 247 logs in system Test 4: Checking metrics... ✓ Metrics endpoint working ✓ All integration tests passed!If any test fails, check the service logs:bash# View shipper logs docker-compose logs log-shipper # View consumer logs docker-compose logs log-consumerManual Verification: Watch Logs FlowOpen three terminal windows to watch the system in action.Terminal 1 - Producer metrics:bashwatch -n 2 ‘curl -s http://localhost:8081/actuator/metrics/logs.generated | jq’This shows how many logs the producer has generated.Terminal 2 - Shipper metrics:bashwatch -n 2 ‘curl -s http://localhost:8082/actuator/metrics/batches.sent | jq’This shows batches sent to Kafka. The count increases every 5 seconds.Terminal 3 - Buffer size:bashwatch -n 1 ‘curl -s http://localhost:8082/actuator/metrics/buffer.size | jq’Watch the buffer fill up (0 to ~50 logs) then drop to 0 when flushed. This demonstrates the batching cycle.Query Stored LogsAfter a minute, logs are in PostgreSQL. Query them:bash# Get 100 most recent logs curl http://localhost:8080/api/query/recent | jq ‘.[0:5]’You’ll see output like:json[ { “id”: “a7f3c9d2-...”, “level”: “INFO”, “service”: “payment-service”, “message”: “Payment transaction initiated”, “timestamp”: “2025-10-09T14:32:15Z”, “traceId”: “8f3a2c1d” }, ... ]Query by service:bashcurl http://localhost:8080/api/query/service/auth-service | jq ‘.[0:3]’Query by log level:bashcurl http://localhost:8080/api/query/level/ERROR | jqLoad Testing: Proving Batching WorksNow let’s stress test the system to see batching in action.Run the Load Testbash./load-test.shThis script generates 10,000 log requests over 60 seconds (166 requests/second). You’ll see progress:Starting load test... This will generate 10,000 log requests over 60 seconds Sent 100 requests... (Success: 100, Failed: 0) Sent 200 requests... (Success: 200, Failed: 0) ... Sent 10000 requests... (Success: 10000, Failed: 0) Load test completed! Duration: 60s Successful: 10000 Failed: 0 Rate: 166 req/sObserve Batching EfficiencyWhile the load test runs, open these monitoring dashboards:Prometheus (http://localhost:9090)Run these queries:Throughput: How many logs per second are being sent?promqlrate(logs_sent_total[1m])You should see ~166 logs/second.Batch rate: How many batches per second?promqlrate(batches_sent_total[1m])You should see ~0.8-1.5 batches/second. This is the magic: 166 logs/sec entering, only 1 batch/sec leaving. That’s 166x reduction in network calls.Average batch size:promqlrate(logs_sent_total[5m]) / rate(batches_sent_total[5m])You should see ~800-1000 logs per batch. When traffic is high, batches fill up and trigger the size limit (1000 logs).Buffer utilization:promqlbuffer_sizeDuring the load test, you’ll see the buffer fluctuate between 0 and 1000 as it fills and flushes.Grafana (http://localhost:3000)Login: admin / adminCreate a dashboard with these panels:Logs received vs logs sent (should match)Batches sent over time (shows batching frequency)Average batch size (shows batching efficiency)Buffer utilization (shows memory pressure)Compare: Without BatchingTo understand the improvement, let’s calculate what would happen without batching:With Batching (actual):Network calls: 1 batch/secondKafka messages: 1 message/secondNetwork overhead: ~500 bytes/secondTotal bandwidth: ~100 KB/secondWithout Batching (hypothetical):Network calls: 166 requests/secondKafka messages: 166 messages/secondNetwork overhead: ~83 KB/second (500 bytes × 166)Total bandwidth: ~100 KB/second + 83 KB overhead = 183 KB/secondBatching reduced network calls by 166x and saved 45% bandwidth. At Netflix scale (500 billion events/day), this saves millions of dollars in infrastructure costs.Monitoring Production MetricsUnderstanding your metrics is crucial for operating this system in production.Key Metrics Explainedlogs.received (counter) Total logs accepted by the shipper. This should match what producers send.logs.sent (counter) Total logs successfully transmitted to Kafka. Should match logs.received (no loss).logs.dropped (counter) Logs discarded due to buffer full. This should always be 0 in healthy operation. If non-zero, you’re losing data.batches.sent (counter) Number of batches transmitted. Divide logs.sent by this to get average batch size.batch.flush.duration (timer) Time taken to serialize and send a batch to Kafka. P99 should be under 100ms.buffer.size (gauge) Current number of logs waiting in the buffer. Normal: 0-1000. Warning: 7000+. Critical: 9500+.buffer.remaining.capacity (gauge) How much space is left. If this stays near 0, increase buffer size or consumer throughput.Setting Up AlertsIn production, you’d configure these alerts:Critical Alerts (page immediately):logs_dropped_total > 0: Data loss occurringbuffer_remaining_capacity kafka_send_failures > 5% for 2 minutes: Kafka connectivity issuesWarning Alerts (investigate soon):buffer_size > 7000 for 10 minutes: Approaching capacitybatch_flush_duration P99 > 500ms: Performance degradinglogs_sent / logs_received Info Alerts (monitor trends):Average batch size Batch rate > 10/second: Consider tuning batch parametersTesting Failure ScenariosReal production systems fail. Let’s test how our batching handles failures gracefully.Scenario 1: Kafka Goes DownStop Kafka and watch what happens:bashdocker-compose stop kafkaObserve the shipper behavior:bashdocker-compose logs -f log-shipperYou’ll see:WARN Failed to send batch to Kafka WARN Buffer size: 1000 WARN Buffer size: 2000 ... WARN Buffer size: 9500 WARN Buffer full, dropping logThe shipper:Continues accepting logsBuffer fills upStarts dropping logs to prevent OOMEmits metrics showing data lossCheck metrics:bashcurl -s http://localhost:8082/actuator/metrics/logs.droppedRestart Kafka:bashdocker-compose start kafkaWithin 30 seconds, the buffer drains and normal operation resumes. Logs dropped during downtime are lost—this is our trade-off for low latency.Scenario 2: Consumer Can’t Keep UpStop the consumer to simulate slow processing:bashdocker-compose stop log-consumerThe Kafka topic accumulates messages but doesn’t affect the shipper. This demonstrates decoupling: the shipper’s job is just to get logs into Kafka. Downstream problems don’t cascade upstream.Check Kafka lag:bashdocker-compose exec kafka kafka-consumer-groups \\ --bootstrap-server localhost:9092 \\ --describe --group log-consumer-groupYou’ll see increasing LAG values. This tells you the consumer is behind.Restart the consumer:bashdocker-compose start log-consumerIt catches up by processing the backlog. No data lost because Kafka persisted everything.Scenario 3: Graceful ShutdownStop the shipper cleanly:bashdocker-compose stop log-shipperCheck the logs:bashdocker-compose logs log-shipper | tail -20You should see:INFO Shutting down, flushing remaining 47 logs INFO Flushed batch of 47 logs to Kafka INFO Application shutdown completeThe ApplicationListener triggered, flushing the partial batch before exit. Restart and verify no logs were lost by comparing producer and consumer counts.Understanding the Performance GainsLet’s analyze what we’ve achieved with concrete numbers.Baseline: No BatchingImagine we send each log individually:Producer generates: 10 logs/secondNetwork calls: 10 HTTP requests/secondEach request has ~500 bytes overhead (headers, handshake, etc.)Overhead bandwidth: 5 KB/secondLog payload: 100 bytes × 10 = 1 KB/secondTotal bandwidth: 6 KB/secondKafka messages: 10/secondContext switches: ~40/second (producer, network, Kafka)With Batching (Our System)Same producer workload:Producer generates: 10 logs/secondShipper batches into ~2 batches every 10 secondsNetwork calls: 0.2 HTTP requests/secondRequest overhead: 500 bytes × 0.2 = 100 bytes/secondLog payload: Still 1 KB/secondTotal bandwidth: 1.1 KB/secondKafka messages: 0.2/secondContext switches: ~1/secondThe MathNetwork calls: 10 → 0.2 (50x reduction)Overhead bandwidth: 5 KB → 0.1 KB (50x reduction)Total bandwidth: 6 KB → 1.1 KB (5.5x reduction)Kafka load: 10 msg/s → 0.2 msg/s (50x reduction)CPU utilization: ~40% reduction from fewer system callsAt 1000 logs/second (load test):Network calls: 1000 → 5 (200x reduction)Kafka load: 1000 msg/s → 5 msg/s (200x reduction)This is why batching is fundamental to every high-scale system. The gains multiply with load.Tuning Batch ParametersThe default configuration (1000 logs or 5 seconds) works well for most cases, but you might need to tune for your specific requirements.Adjusting Batch SizeEdit log-shipper/src/main/resources/application.yml:yamlbatching: max-batch-size: 1000 # Change this buffer-capacity: 10000Guidelines:Increase if you have high-volume, small logs (e.g., 5000 for 50-byte logs)Decrease if you have large logs (e.g., 100 for 10KB logs)Target: 100KB-1MB total batch payloadRebuild and test:bashdocker-compose up -d --build log-shipperMonitor average batch size:promqlrate(logs_sent_total[5m]) / rate(batches_sent_total[5m])Adjusting Time IntervalEdit LogBatchingService.java:java@Scheduled(fixedDelay = 5000) // millisecondsGuidelines:Decrease for latency-sensitive systems (e.g., 1000ms = 1 second)Increase for throughput-focused systems (e.g., 30000ms = 30 seconds)Balance: Shorter interval = lower latency but less batching efficiencyAdjusting Buffer Capacityyamlbatching: buffer-capacity: 10000 # Change thisGuidelines:Set to handle burst traffic: peak_logs_per_second × burst_duration_secondsExample: 1000 logs/sec × 30 sec burst = 30,000 capacityMemory cost: ~100 bytes/log = 3MB for 30,000 capacityAlways leave headroom: Set capacity 2x your expected maxCommon Issues and SolutionsProblem: Logs Not Appearing in PostgreSQLSymptoms: Producer shows logs generated, but API Gateway returns empty results.Diagnosis:bash# Check each stage curl http://localhost:8081/actuator/metrics/logs.generated curl http://localhost:8082/actuator/metrics/logs.sent curl http://localhost:8083/actuator/metrics/logs.processedIf logs.generated > logs.sent: Shipper isn’t flushingbashdocker-compose logs log-shipperLook for Kafka connection errors.If logs.sent > logs.processed: Consumer isn’t workingbashdocker-compose logs log-consumerCheck PostgreSQL connection.Solution: Restart the failing service:bashdocker-compose restart log-shipper # or docker-compose restart log-consumerProblem: High Memory UsageSymptoms: Docker shows high memory usage, system slows down.Diagnosis:bashcurl http://localhost:8082/actuator/metrics/buffer.sizeIf buffer.size stays near capacity (9000+), logs aren’t flushing.Solutions:Increase consumer throughput (scale horizontally)Reduce batch size (flush more frequently)Increase buffer capacity (if you have RAM)Accept data loss (current behavior: drops logs at capacity)Problem: Metrics Not Showing in PrometheusSymptoms: Prometheus queries return no data.Diagnosis:Check Prometheus targets: http://localhost:9090/targetsAll should be “UP”. If “DOWN”, check the service health endpoint directly.Solution:Restart Prometheus:bashdocker-compose restart prometheusWait 15 seconds for scraping to resume.Cleaning UpWhen you’re done experimenting:Stop All Servicesbashdocker-compose downThis stops containers but preserves data volumes.Remove All Databashdocker-compose down -vThis deletes PostgreSQL data, Redis cache, Prometheus metrics, and Grafana dashboards.Remove Imagesbashdocker-compose down --rmi allThis frees disk space by removing all Docker images.Complete Resetbashdocker-compose down -v --rmi all cd .. rm -rf log-batching-systemThis removes everything and lets you start fresh.What You’ve LearnedBy completing this lesson, you now understand:Batching FundamentalsWhy individual transmission is wasteful at scaleHow dual triggers (size + time) balance latency and throughputThe trade-off between memory usage and batching efficiencyProduction PatternsBounded queues prevent memory leaks during backpressureGraceful degradation (dropping logs) is better than crashingMonitoring and alerting are essential for reliable operationReal-World ApplicationsThis exact pattern powers Netflix, AWS, Uber, and Google systems200x network reduction is typical for high-volume systemsCombined with compression (tomorrow), gains reach 1000xSystem Design SkillsHow to measure and prove system improvementsFailure modes and how to handle them gracefullyPerformance tuning through metrics-driven decisionsTomorrow, you’ll add compression to multiply these gains by another 5-10x. But the batching pattern you learned today is already production-grade and ready for real systems.","published","strategy","content-ready","locked","deletedAt","isSyncedFromOwnBlog","original","duplicatesCount","duplicateVersion","originalArticleId","errorsCount","appearance","color","backgroundColor","shortSummary","summary","bullets","persons","places","words","sentiment","subject","meta","eventId","096ae6ae55eba6a980692cdcdee3973337ba8293ee88186ddcba9ffb5713649b","nostrRootEvent","event",{"_14":182,"_186":187,"_188":189,"_62":190,"_201":21,"_152":202,"_203":204},"sig","24d0088af6e9c8e041eccae7f96cbbef327ec6a676f96173195f107f9f087e34b28cfc439d0eab638e59f2c3004b86465076758e8760967bc1137b6884c6931d","kind",1,[191,192,193,197,199],[179,138],[29,144],[194,195,24,196],"a","30023:d12614be726049c7ecfeb7307a63d38418bb82b91866b8764a903ff28ed99bc6:9a7c2aa2-7b4e-4b55-ac68-7ab44f96abde","mention",[198,24],"client",[200,21,24],"p","pubkey","Day 11: Implement Batching in the Log Shipper to Optimize Network Usage\nWhat We’re Building Today\n\n\nnostr:naddr1qvzqqqr4gupzp5fxzjl8yczfclk0ades0f3a8pqchwptjxrxhpmy4ypl728dnx7xqqjrjcfhvvexzcfj95mkydr9956xydf494skxd3c95mkzc35x3nrjdnpvfjx27mgaht ","created_at",1771253262,"authorNPub","markdown","What We’re Building Today\n-------------------------\n\n* **Intelligent batching log shipper**that accumulates logs before transmission\n \n* **Configurable batching strategies**with size-based and time-based triggers\n \n* **Memory-efficient buffer management**with backpressure handling\n \n* **Production-ready Spring Boot service**with Kafka integration and comprehensive monitoring\n \n\n* * *\n\nWhy This Matters: The Network Is Your Bottleneck\n------------------------------------------------\n\nEvery log transmission carries overhead: TCP handshakes, TLS negotiations, HTTP headers, kernel context switches. At scale, this overhead dominates your infrastructure costs. Netflix processes 500+ billion events daily—sending each individually would require millions of network connections per second. Instead, they batch aggressively, reducing network calls by 99% while maintaining sub-second latency.\n\nThe batching problem surfaces everywhere: Uber’s trip updates, Airbnb’s search queries, AWS CloudWatch metrics. The pattern is universal: accumulate data points, wait for a trigger (time or size), then transmit in bulk. But naive batching creates new problems: memory pressure from unbounded buffers, data loss during crashes, and head-of-line blocking where one slow batch stalls the entire pipeline.\n\nThis lesson implements production-grade batching that Netflix and AWS use in their ingestion pipelines. You’ll learn to balance throughput against latency, handle backpressure gracefully, and maintain data durability guarantees even when batching introduces delays.\n\n* * *\n\nSystem Design Deep Dive: Batching Strategies and Trade-offs\n-----------------------------------------------------------\n\n### Pattern 1: Dual-Trigger Batching (Time + Size)\n\nThe fundamental batching challenge: wait too long for full batches and latency suffers; send too early and throughput collapses. Production systems use dual triggers: flush when either batch size OR time interval is reached, whichever comes first.\n\n**Trade-off Analysis:**Size-based batching (e.g., 1000 records) maximizes throughput during high traffic but creates unbounded latency during quiet periods. Time-based batching (e.g., 5 seconds) bounds worst-case latency but wastes network efficiency during traffic spikes. Combining both gives you throughput during peaks and latency guarantees during valleys.\n\n**Real-world calibration:**AWS Kinesis uses 500KB or 1-second batches. Kafka producers default to 16KB or 10ms. Netflix’s edge proxies tune dynamically based on observed traffic patterns. Start with 1000 records or 5 seconds, then measure p99 latency and network utilization to tune.\n\n### Pattern 2: Bounded Queue with Backpressure\n\nUnbounded batching buffers are a memory leak waiting to happen. When your producer outpaces consumers, memory usage grows until OOM kills your process. Production systems use bounded queues: when full, apply backpressure by blocking producers or dropping logs.\n\n**Implementation consideration:**Java’s ArrayBlockingQueue provides thread-safe bounded buffering with configurable blocking behavior. Set capacity based on expected burst size (e.g., 10,000 records = ~1MB for 100-byte logs). Configure`offer()`with timeout for non-blocking backpressure or`put()`for blocking behavior.\n\n**Anti-pattern:**Never batch indefinitely without bounds. LinkedIn learned this when a batching bug caused servers to accumulate 40GB of logs in memory, triggering cascading OOM failures across their fleet.\n\n### Pattern 3: Graceful Degradation Under Load\n\nWhen system load exceeds capacity, you must choose: maintain latency by dropping data, or maintain durability by increasing latency. This choice depends on your data’s criticality. Application logs are often droppable; billing events never are.\n\n**Strategy:**Implement adaptive batching with load shedding. Under normal load, use standard batch parameters. When queue depth exceeds 80%, emit metrics and begin sampling (send 10% of logs). At 95%, drop non-critical logs entirely. Always preserve error logs and transactions.\n\n**Monitoring trigger:**Track`batch_queue_depth`,`logs_dropped_count`, and`average_batch_fill_ratio`. Alert when queue depth stays above 70% for 5 minutes—you’re approaching capacity limits.\n\n### Pattern 4: At-Least-Once Delivery Semantics\n\nBatching introduces a durability gap: logs in your batch buffer aren’t yet persisted to Kafka. If the shipper crashes, those logs vanish. Production systems accept this trade-off but minimize exposure through rapid batch flushes and periodic checkpointing.\n\n**Mitigation strategy:**Flush batches every 5 seconds maximum, even partially full. On shutdown signals (SIGTERM), flush immediately before exiting. For critical logs, write to a local disk buffer before batching (Write-Ahead Log pattern), then mark committed after Kafka acknowledgment.\n\n**Real-world compromise:**Most log aggregation accepts 5-10 seconds of potential data loss during crashes. Critical paths (payments, user actions) use immediate sends or persist locally first. Know your SLA and architect accordingly.\n\n### Pattern 5: Batch Size Distribution and Network Efficiency\n\nNot all batches are created equal. A batch of 1000 small logs (100 bytes each) behaves differently than 1000 large logs (10KB each). Network efficiency depends on payload size relative to overhead. Small payloads waste bandwidth on headers; large payloads risk timeout during transmission.\n\n**Optimal strategy:**Target 100KB-1MB batch payloads regardless of record count. Use separate batch size limits for small vs large logs. Compress batches before transmission (tomorrow’s lesson) to maximize payload density.\n\n**Performance math:**A 100-byte log with 500 bytes of overhead = 83% waste. Batch 1000 logs = 100KB payload + 500 bytes overhead = 99.5% efficiency. This 200x improvement in network utilization is why every high-scale system batches aggressively.\n\n* * *\n\nImplementation Walkthrough: Building the Batching Log Shipper\n-------------------------------------------------------------\n\nOur implementation uses Spring Boot’s scheduled executors for time-based flushing and ArrayBlockingQueue for bounded buffering. The LogBatchingService accumulates logs in memory and flushes to Kafka when either trigger fires.\n\n**Core architecture decision:**We separate batching logic from Kafka interaction. The BatchAccumulator handles buffering and trigger logic. The KafkaProducerService handles transmission with retry. This separation allows testing batching behavior without Kafka and enables swapping messaging layers if needed.\n\n**Key implementation details:**\n\n1. **Thread-safe accumulation:**Use ConcurrentLinkedQueue for lock-free log appends. Multiple producer threads can add logs without contention.\n \n2. **Scheduled flushing:**Spring’s @Scheduled annotation triggers time-based flushes every 5 seconds. Use fixedDelay to ensure flushes complete before scheduling the next.\n \n3. **Size-based triggers:**After each append, check buffer size. If >= maxBatchSize, submit a flush task to a separate executor to avoid blocking producers.\n \n4. **Graceful shutdown:**Implement ApplicationListenerto flush pending batches before shutdown. This prevents data loss during deployments.\n \n\nThe KafkaProducerService sends batches with acknowledgment=all for durability. We serialize batches as JSON arrays for simplicity, though Protocol Buffers would reduce payload size by 60% in production.\n\n**Load testing reveals:**At 1000 logs/second with 100-byte payloads, batching reduces Kafka requests from 1000/sec to 5/sec (200x reduction). Network bandwidth drops from 500KB/s to 120KB/s (4.2x improvement) due to reduced overhead. CPU usage decreases 40% from fewer kernel calls.\n\n* * *\n\nProduction Considerations: Failure Modes and Monitoring\n-------------------------------------------------------\n\n**Memory pressure monitoring:**Track heap usage and queue depth. Alert if queue depth exceeds 70% or heap usage grows unexpectedly. Set JVM max heap to 2x expected buffer size to prevent OOM.\n\n**Latency distribution:**Measure p50, p95, p99 time-to-delivery. Batching introduces variable latency: logs at batch start wait up to maxInterval; logs triggering size flush send immediately. Monitor to ensure p99 stays under your SLA.\n\n**Data loss scenarios:**Logs in buffer vanish if JVM crashes before flush. Mitigate with frequent flushes (5s max) and local disk buffering for critical logs. Track`logs_in_buffer`metric—persistent high values indicate flush problems.\n\n**Kafka unavailability:**When Kafka is down, batches accumulate in memory until buffer fills. Configure backpressure to drop logs rather than OOM. Emit`kafka_send_failures`metric and page on-call when failure rate exceeds 5%.\n\n**Thundering herd on restart:**If 100 shippers crash simultaneously and restart, all flush their buffers at once, overwhelming Kafka. Implement jittered startup delays (random 0-30 seconds) to stagger the load.\n\n* * *\n\nScale Connection: How FAANG Systems Batch\n-----------------------------------------\n\n**Netflix Edge Proxies:**Batch view events with 1-second windows to reduce backend load 100x. During peak (Friday 8pm), batching prevents their event ingestion from requiring millions of connections/second.\n\n**AWS Kinesis Streams:**Implements automatic batching in their SDK with 500KB or 1-second triggers. Their customers don’t think about batching—it’s transparent—but it reduces their infrastructure costs by 90%.\n\n**Uber’s Real-Time Analytics:**Batches trip events with 5-second windows before feeding their machine learning pipelines. Batching reduces their Kafka cluster size from 500 nodes to 50 nodes while maintaining sub-10-second freshness.\n\n**Google Cloud Logging:**Uses adaptive batching that adjusts interval based on traffic. During quiet periods, flushes every 10 seconds; during spikes, flushes every 100ms with 1000-log batches. This maintains 99th percentile latency under 15 seconds while maximizing throughput.\n\n* * *\n\nNext Steps: Compression Multiplies Batching Gains\n-------------------------------------------------\n\nTomorrow, we add compression to our batches. Gzip typically reduces log payloads by 5-10x because logs contain repetitive text. Combined with batching, you’ll achieve 1000x reduction in network usage: 200x from batching, 5x from compression. This combination is why modern log systems can ingest terabytes per day on modest infrastructure.\n\nWe’ll implement streaming compression to handle large batches without memory spikes, compare gzip vs snappy for latency-sensitive paths, and measure compression ratio vs CPU trade-offs. You’ll learn when compression hurts more than it helps (small payloads, low repetition) and how Netflix decides which compression algorithm to use for each data type.\n\n* * *\n\nSystem Architecture Overview\n============================\n\n[](https://substackcdn.com/image/fetch/$s_!fcuL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc6efc417-e39b-4a3e-a58c-02970fa22386_1009x621.png)\n\nThe system consists of four Spring Boot microservices backed by Kafka, PostgreSQL, and Redis:\n\n**Log Producer (Port 8081)**Simulates application servers generating logs at 10 logs/second. Sends individual log events to the shipper.\n\n**Log Shipper (Port 8082)**The batching layer. Accumulates logs in a bounded buffer (10,000 capacity) and flushes when either 1000 logs accumulate OR 5 seconds elapse. This is where the magic happens.\n\n**Apache Kafka**Message broker providing durability and decoupling. The log-events topic has 3 partitions for parallel processing.\n\n**Log Consumer (Port 8083)**Reads batched logs from Kafka and persists them to PostgreSQL for long-term storage.\n\n**API Gateway (Port 8080)**Query interface with Redis caching (60-second TTL) for fast log retrieval.\n\n**PostgreSQL**Persistent storage with indexes on timestamp, service, and log level for efficient queries.\n\n**Redis**Cache layer reducing database load for repeated queries.\n\n**Prometheus + Grafana**Monitoring stack collecting metrics from all services. Tracks batching efficiency, buffer utilization, and system health.\n\nYoutube Video:\n--------------\n\n* * *\n\nBuilding the System\n===================\n\n### Github Link :\n\n https://github.com/sysdr/sdc-java/tree/main/day11/log-batching-system\n\nPrerequisites\n-------------\n\nYou need these installed on your machine:\n\n* Docker Desktop (minimum 4GB RAM allocated)\n \n* Git\n \n* A terminal (Command Prompt, PowerShell, or Terminal)\n \n* Port availability: 8080-8083, 9090, 3000, 5432, 6379, 9092, 2181\n \n\nCheck Docker is running:\n\nbash\n\n docker --versiondocker-compose --version\n\nStep 1: Generate the Project Files\n----------------------------------\n\nDownload the`generate_system_files.sh`script from the lesson materials. This script creates the entire project structure with all code and configuration files.\n\nbash\n\n # Make the script executablechmod +x generate_system_files.sh# Run it./generate_system_files.sh\n\nYou’ll see output like:\n\n Generating Day 11: Batching Log Shipper System...✓ System generated successfully!\n\nThe script creates a`log-batching-system`directory containing:\n\n log-batching-system/├── log-producer/ # Generates synthetic logs├── log-shipper/ # Batching layer (main focus)├── log-consumer/ # Kafka consumer + PostgreSQL├── api-gateway/ # Query API + Redis cache├── docker-compose.yml # Infrastructure setup├── setup.sh # Automated deployment├── load-test.sh # Performance testing├── integration-tests/ # System validation├── monitoring/ # Prometheus + Grafana configs└── README.md\n\nStep 2: Understanding the Batching Service\n------------------------------------------\n\nLet’s look at the core batching logic before we run it. Open`log-shipper/src/main/java/com/example/logshipper/LogBatchingService.java`:\n\n**The buffer:**An ArrayBlockingQueue with 10,000 capacity prevents memory leaks:\n\njava\n\n private final BlockingQueuebuffer = new ArrayBlockingQueue<>(10000);\n\n**Adding logs with backpressure:**The`offer()`method returns false if buffer is full—we drop the log rather than crash:\n\n**Size-based trigger:**After adding, check if we hit the batch size limit:\n\njava\n\n if (currentBatchSize.get() >= maxBatchSize) { flushBatch(); // Send immediately when full}\n\n**Time-based trigger:**Spring scheduler flushes every 5 seconds regardless of size:\n\njava\n\n @Scheduled(fixedDelay = 5000)public void scheduledFlush() { if (currentBatchSize.get() > 0) { flushBatch(); }}\n\n**Graceful shutdown:**Before the application stops, flush remaining logs:\n\njava\n\n @Overridepublic void onApplicationEvent(ContextClosedEvent event) { log.info(”Shutting down, flushing remaining logs”); flushBatch();}\n\nThis design ensures logs never wait more than 5 seconds, network efficiency improves dramatically, and memory usage stays bounded.\n\n* * *\n\nDeploying and Running the System\n================================\n\nStep 1: Start the Infrastructure\n--------------------------------\n\nNavigate into the project directory:\n\nbash\n\n cd log-batching-system\n\nRun the automated setup:\n\nbash\n\n ./setup.sh\n\nThis script does several things:\n\n1. Starts Zookeeper and Kafka\n \n2. Starts PostgreSQL and Redis\n \n3. Starts Prometheus and Grafana\n \n4. Waits for infrastructure to be ready\n \n5. Creates the Kafka topic with 3 partitions\n \n6. Builds and starts all 4 application services\n \n\nYou’ll see output showing each service starting. The whole process takes about 2-3 minutes.\n\n Starting infrastructure services...Waiting for infrastructure to be ready...Checking Kafka...Creating Kafka topic...Building and starting application services...System is starting up...\n\nStep 2: Verify All Services Are Running\n---------------------------------------\n\nCheck Docker containers:\n\nbash\n\n docker-compose ps\n\nYou should see 11 containers running:\n\n* zookeeper\n \n* kafka\n \n* postgres\n \n* redis\n \n* prometheus\n \n* grafana\n \n* log-producer\n \n* log-shipper\n \n* log-consumer\n \n* api-gateway\n \n\nAll should show “Up” status.\n\nStep 3: Check Service Health\n----------------------------\n\nEach service has a health endpoint. Test them:\n\nbash\n\n # Log Producercurl http://localhost:8081/actuator/health# Log Shipper (batching layer)curl http://localhost:8082/api/logs/health# Log Consumercurl http://localhost:8083/actuator/health# API Gatewaycurl http://localhost:8080/api/query/health\n\nEach should return`{”status”:”UP”}`or similar.\n\n* * *\n\nTesting the Batching System\n===========================\n\nIntegration Test\n----------------\n\nRun the automated integration test suite:\n\nbash\n\n ./integration-tests/test-system.sh\n\nThis script:\n\n1. Waits 30 seconds for services to stabilize\n \n2. Checks all health endpoints\n \n3. Waits 60 seconds for logs to flow through the system\n \n4. Queries the API Gateway for recent logs\n \n5. Verifies metrics are being collected\n \n\nExpected output:\n\n Running integration tests...Test 1: Health checks...Test 2: Waiting for log processing...Test 3: Querying recent logs...✓ Found 247 logs in systemTest 4: Checking metrics...✓ Metrics endpoint working✓ All integration tests passed!\n\nIf any test fails, check the service logs:\n\nbash\n\n # View shipper logsdocker-compose logs log-shipper# View consumer logsdocker-compose logs log-consumer\n\nManual Verification: Watch Logs Flow\n------------------------------------\n\nOpen three terminal windows to watch the system in action.\n\n**Terminal 1 - Producer metrics:**\n\nbash\n\n watch -n 2 ‘curl -s http://localhost:8081/actuator/metrics/logs.generated | jq’\n\nThis shows how many logs the producer has generated.\n\n**Terminal 2 - Shipper metrics:**\n\nbash\n\n watch -n 2 ‘curl -s http://localhost:8082/actuator/metrics/batches.sent | jq’\n\nThis shows batches sent to Kafka. The count increases every 5 seconds.\n\n**Terminal 3 - Buffer size:**\n\nbash\n\n watch -n 1 ‘curl -s http://localhost:8082/actuator/metrics/buffer.size | jq’\n\nWatch the buffer fill up (0 to ~50 logs) then drop to 0 when flushed. This demonstrates the batching cycle.\n\nQuery Stored Logs\n-----------------\n\nAfter a minute, logs are in PostgreSQL. Query them:\n\nbash\n\n # Get 100 most recent logscurl http://localhost:8080/api/query/recent | jq ‘.[0:5]’\n\nYou’ll see output like:\n\njson\n\n [ { “id”: “a7f3c9d2-...”, “level”: “INFO”, “service”: “payment-service”, “message”: “Payment transaction initiated”, “timestamp”: “2025-10-09T14:32:15Z”, “traceId”: “8f3a2c1d” }, ...]\n\nQuery by service:\n\nbash\n\n curl http://localhost:8080/api/query/service/auth-service | jq ‘.[0:3]’\n\nQuery by log level:\n\nbash\n\n curl http://localhost:8080/api/query/level/ERROR | jq\n\n* * *\n\nLoad Testing: Proving Batching Works\n====================================\n\nNow let’s stress test the system to see batching in action.\n\nRun the Load Test\n-----------------\n\nbash\n\n ./load-test.sh\n\nThis script generates 10,000 log requests over 60 seconds (166 requests/second). You’ll see progress:\n\n Starting load test...This will generate 10,000 log requests over 60 secondsSent 100 requests... (Success: 100, Failed: 0)Sent 200 requests... (Success: 200, Failed: 0)...Sent 10000 requests... (Success: 10000, Failed: 0)Load test completed!Duration: 60sSuccessful: 10000Failed: 0Rate: 166 req/s\n\nObserve Batching Efficiency\n---------------------------\n\nWhile the load test runs, open these monitoring dashboards:\n\n**Prometheus (**\n\nhttp://localhost:9090\n\n**)**\n\nRun these queries:\n\n1. **Throughput:**How many logs per second are being sent?\n \n\npromql\n\n rate(logs_sent_total[1m])\n\nYou should see ~166 logs/second.\n\n1. **Batch rate:**How many batches per second?\n \n\npromql\n\n rate(batches_sent_total[1m])\n\nYou should see ~0.8-1.5 batches/second. This is the magic: 166 logs/sec entering, only 1 batch/sec leaving. That’s 166x reduction in network calls.\n\n1. **Average batch size:**\n \n\npromql\n\n rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n\nYou should see ~800-1000 logs per batch. When traffic is high, batches fill up and trigger the size limit (1000 logs).\n\n1. **Buffer utilization:**\n \n\npromql\n\n buffer_size\n\nDuring the load test, you’ll see the buffer fluctuate between 0 and 1000 as it fills and flushes.\n\n[![](https://substackcdn.com/image/fetch/$s_!PS8z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83764f4c-8286-4b6c-a0a1-74d3802d51eb_623x554.png)](https://substackcdn.com/image/fetch/$s_!PS8z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83764f4c-8286-4b6c-a0a1-74d3802d51eb_623x554.png)\n\n**Grafana (**\n\nhttp://localhost:3000\n\n**)**\n\nLogin: admin / admin\n\nCreate a dashboard with these panels:\n\n1. Logs received vs logs sent (should match)\n \n2. Batches sent over time (shows batching frequency)\n \n3. Average batch size (shows batching efficiency)\n \n4. Buffer utilization (shows memory pressure)\n \n\n[![](https://substackcdn.com/image/fetch/$s_!U-Wa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcb47d1-5352-4b15-a195-1b829b211b04_866x625.png)](https://substackcdn.com/image/fetch/$s_!U-Wa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcb47d1-5352-4b15-a195-1b829b211b04_866x625.png)\n\nCompare: Without Batching\n-------------------------\n\nTo understand the improvement, let’s calculate what would happen without batching:\n\n**With Batching (actual):**\n\n* Network calls: 1 batch/second\n \n* Kafka messages: 1 message/second\n \n* Network overhead: ~500 bytes/second\n \n* Total bandwidth: ~100 KB/second\n \n\n**Without Batching (hypothetical):**\n\n* Network calls: 166 requests/second\n \n* Kafka messages: 166 messages/second\n \n* Network overhead: ~83 KB/second (500 bytes × 166)\n \n* Total bandwidth: ~100 KB/second + 83 KB overhead = 183 KB/second\n \n\nBatching reduced network calls by 166x and saved 45% bandwidth. At Netflix scale (500 billion events/day), this saves millions of dollars in infrastructure costs.\n\n* * *\n\nMonitoring Production Metrics\n=============================\n\nUnderstanding your metrics is crucial for operating this system in production.\n\nKey Metrics Explained\n---------------------\n\n**logs.received (counter)**Total logs accepted by the shipper. This should match what producers send.\n\n**logs.sent (counter)**Total logs successfully transmitted to Kafka. Should match logs.received (no loss).\n\n**logs.dropped (counter)**Logs discarded due to buffer full. This should always be 0 in healthy operation. If non-zero, you’re losing data.\n\n**batches.sent (counter)**Number of batches transmitted. Divide logs.sent by this to get average batch size.\n\n**batch.flush.duration (timer)**Time taken to serialize and send a batch to Kafka. P99 should be under 100ms.\n\n**buffer.size (gauge)**Current number of logs waiting in the buffer. Normal: 0-1000. Warning: 7000+. Critical: 9500+.\n\n**buffer.remaining.capacity (gauge)**How much space is left. If this stays near 0, increase buffer size or consumer throughput.\n\nSetting Up Alerts\n-----------------\n\nIn production, you’d configure these alerts:\n\n**Critical Alerts (page immediately):**\n\n1. `logs_dropped_total`\\> 0: Data loss occurring\n \n2. `buffer_remaining_capacity`< 500 for 5 minutes: About to drop logs\n \n3. `kafka_send_failures`\\> 5% for 2 minutes: Kafka connectivity issues\n \n\n**Warning Alerts (investigate soon):**\n\n1. `buffer_size`\\> 7000 for 10 minutes: Approaching capacity\n \n2. `batch_flush_duration`P99 > 500ms: Performance degrading\n \n3. `logs_sent / logs_received`< 0.99: Losing logs somewhere\n \n\n**Info Alerts (monitor trends):**\n\n1. Average batch size< 100: Not batching effectively\n \n2. Batch rate > 10/second: Consider tuning batch parameters\n \n\n* * *\n\nTesting Failure Scenarios\n=========================\n\nReal production systems fail. Let’s test how our batching handles failures gracefully.\n\nScenario 1: Kafka Goes Down\n---------------------------\n\nStop Kafka and watch what happens:\n\nbash\n\n docker-compose stop kafka\n\nObserve the shipper behavior:\n\nbash\n\n docker-compose logs -f log-shipper\n\nYou’ll see:\n\n WARN Failed to send batch to KafkaWARN Buffer size: 1000WARN Buffer size: 2000...WARN Buffer size: 9500WARN Buffer full, dropping log\n\nThe shipper:\n\n1. Continues accepting logs\n \n2. Buffer fills up\n \n3. Starts dropping logs to prevent OOM\n \n4. Emits metrics showing data loss\n \n\nCheck metrics:\n\nbash\n\n curl -s http://localhost:8082/actuator/metrics/logs.dropped\n\nRestart Kafka:\n\nbash\n\n docker-compose start kafka\n\nWithin 30 seconds, the buffer drains and normal operation resumes. Logs dropped during downtime are lost—this is our trade-off for low latency.\n\nScenario 2: Consumer Can’t Keep Up\n----------------------------------\n\nStop the consumer to simulate slow processing:\n\nbash\n\n docker-compose stop log-consumer\n\nThe Kafka topic accumulates messages but doesn’t affect the shipper. This demonstrates decoupling: the shipper’s job is just to get logs into Kafka. Downstream problems don’t cascade upstream.\n\nCheck Kafka lag:\n\nbash\n\n docker-compose exec kafka kafka-consumer-groups \\ --bootstrap-server localhost:9092 \\ --describe --group log-consumer-group\n\nYou’ll see increasing LAG values. This tells you the consumer is behind.\n\nRestart the consumer:\n\nbash\n\n docker-compose start log-consumer\n\nIt catches up by processing the backlog. No data lost because Kafka persisted everything.\n\nScenario 3: Graceful Shutdown\n-----------------------------\n\nStop the shipper cleanly:\n\nbash\n\n docker-compose stop log-shipper\n\nCheck the logs:\n\nbash\n\n docker-compose logs log-shipper | tail -20\n\nYou should see:\n\n INFO Shutting down, flushing remaining 47 logsINFO Flushed batch of 47 logs to KafkaINFO Application shutdown complete\n\nThe ApplicationListener triggered, flushing the partial batch before exit. Restart and verify no logs were lost by comparing producer and consumer counts.\n\n* * *\n\nUnderstanding the Performance Gains\n===================================\n\nLet’s analyze what we’ve achieved with concrete numbers.\n\nBaseline: No Batching\n---------------------\n\nImagine we send each log individually:\n\n* Producer generates: 10 logs/second\n \n* Network calls: 10 HTTP requests/second\n \n* Each request has ~500 bytes overhead (headers, handshake, etc.)\n \n* Overhead bandwidth: 5 KB/second\n \n* Log payload: 100 bytes × 10 = 1 KB/second\n \n* Total bandwidth: 6 KB/second\n \n* Kafka messages: 10/second\n \n* Context switches: ~40/second (producer, network, Kafka)\n \n\nWith Batching (Our System)\n--------------------------\n\nSame producer workload:\n\n* Producer generates: 10 logs/second\n \n* Shipper batches into ~2 batches every 10 seconds\n \n* Network calls: 0.2 HTTP requests/second\n \n* Request overhead: 500 bytes × 0.2 = 100 bytes/second\n \n* Log payload: Still 1 KB/second\n \n* Total bandwidth: 1.1 KB/second\n \n* Kafka messages: 0.2/second\n \n* Context switches: ~1/second\n \n\nThe Math\n--------\n\n* Network calls: 10 → 0.2 (50x reduction)\n \n* Overhead bandwidth: 5 KB → 0.1 KB (50x reduction)\n \n* Total bandwidth: 6 KB → 1.1 KB (5.5x reduction)\n \n* Kafka load: 10 msg/s → 0.2 msg/s (50x reduction)\n \n* CPU utilization: ~40% reduction from fewer system calls\n \n\nAt 1000 logs/second (load test):\n\n* Network calls: 1000 → 5 (200x reduction)\n \n* Kafka load: 1000 msg/s → 5 msg/s (200x reduction)\n \n\nThis is why batching is fundamental to every high-scale system. The gains multiply with load.\n\n* * *\n\nTuning Batch Parameters\n=======================\n\nThe default configuration (1000 logs or 5 seconds) works well for most cases, but you might need to tune for your specific requirements.\n\nAdjusting Batch Size\n--------------------\n\nEdit`log-shipper/src/main/resources/application.yml`:\n\nyaml\n\n batching: max-batch-size: 1000 # Change this buffer-capacity: 10000\n\n**Guidelines:**\n\n* **Increase**if you have high-volume, small logs (e.g., 5000 for 50-byte logs)\n \n* **Decrease**if you have large logs (e.g., 100 for 10KB logs)\n \n* Target: 100KB-1MB total batch payload\n \n\nRebuild and test:\n\nbash\n\n docker-compose up -d --build log-shipper\n\nMonitor average batch size:\n\npromql\n\n rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n\nAdjusting Time Interval\n-----------------------\n\nEdit`LogBatchingService.java`:\n\njava\n\n @Scheduled(fixedDelay = 5000) // milliseconds\n\n**Guidelines:**\n\n* **Decrease**for latency-sensitive systems (e.g., 1000ms = 1 second)\n \n* **Increase**for throughput-focused systems (e.g., 30000ms = 30 seconds)\n \n* Balance: Shorter interval = lower latency but less batching efficiency\n \n\nAdjusting Buffer Capacity\n-------------------------\n\nyaml\n\n batching: buffer-capacity: 10000 # Change this\n\n**Guidelines:**\n\n* Set to handle burst traffic:`peak_logs_per_second × burst_duration_seconds`\n \n* Example: 1000 logs/sec × 30 sec burst = 30,000 capacity\n \n* Memory cost: ~100 bytes/log = 3MB for 30,000 capacity\n \n* Always leave headroom: Set capacity 2x your expected max\n \n\n* * *\n\nCommon Issues and Solutions\n===========================\n\nProblem: Logs Not Appearing in PostgreSQL\n-----------------------------------------\n\n**Symptoms:**Producer shows logs generated, but API Gateway returns empty results.\n\n**Diagnosis:**\n\nbash\n\n # Check each stagecurl http://localhost:8081/actuator/metrics/logs.generatedcurl http://localhost:8082/actuator/metrics/logs.sentcurl http://localhost:8083/actuator/metrics/logs.processed\n\nIf logs.generated > logs.sent: Shipper isn’t flushing\n\nbash\n\n docker-compose logs log-shipper\n\nLook for Kafka connection errors.\n\nIf logs.sent > logs.processed: Consumer isn’t working\n\nbash\n\n docker-compose logs log-consumer\n\nCheck PostgreSQL connection.\n\n**Solution:**Restart the failing service:\n\nbash\n\n docker-compose restart log-shipper# ordocker-compose restart log-consumer\n\nProblem: High Memory Usage\n--------------------------\n\n**Symptoms:**Docker shows high memory usage, system slows down.\n\n**Diagnosis:**\n\nbash\n\n curl http://localhost:8082/actuator/metrics/buffer.size\n\nIf buffer.size stays near capacity (9000+), logs aren’t flushing.\n\n**Solutions:**\n\n1. Increase consumer throughput (scale horizontally)\n \n2. Reduce batch size (flush more frequently)\n \n3. Increase buffer capacity (if you have RAM)\n \n4. Accept data loss (current behavior: drops logs at capacity)\n \n\nProblem: Metrics Not Showing in Prometheus\n------------------------------------------\n\n**Symptoms:**Prometheus queries return no data.\n\n**Diagnosis:**\n\nCheck Prometheus targets:[http://localhost:9090/targets](http://localhost:9090/targets)\n\nAll should be “UP”. If “DOWN”, check the service health endpoint directly.\n\n**Solution:**\n\nRestart Prometheus:\n\nbash\n\n docker-compose restart prometheus\n\nWait 15 seconds for scraping to resume.\n\n* * *\n\nCleaning Up\n===========\n\nWhen you’re done experimenting:\n\nStop All Services\n-----------------\n\nbash\n\n docker-compose down\n\nThis stops containers but preserves data volumes.\n\nRemove All Data\n---------------\n\nbash\n\n docker-compose down -v\n\nThis deletes PostgreSQL data, Redis cache, Prometheus metrics, and Grafana dashboards.\n\nRemove Images\n-------------\n\nbash\n\n docker-compose down --rmi all\n\nThis frees disk space by removing all Docker images.\n\nComplete Reset\n--------------\n\nbash\n\n docker-compose down -v --rmi allcd ..rm -rf log-batching-system\n\nThis removes everything and lets you start fresh.\n\n* * *\n\nWhat You’ve Learned\n===================\n\nBy completing this lesson, you now understand:\n\n**Batching Fundamentals**\n\n* Why individual transmission is wasteful at scale\n \n* How dual triggers (size + time) balance latency and throughput\n \n* The trade-off between memory usage and batching efficiency\n \n\n**Production Patterns**\n\n* Bounded queues prevent memory leaks during backpressure\n \n* Graceful degradation (dropping logs) is better than crashing\n \n* Monitoring and alerting are essential for reliable operation\n \n\n**Real-World Applications**\n\n* This exact pattern powers Netflix, AWS, Uber, and Google systems\n \n* 200x network reduction is typical for high-volume systems\n \n* Combined with compression (tomorrow), gains reach 1000x\n \n\n**System Design Skills**\n\n* How to measure and prove system improvements\n \n* Failure modes and how to handle them gracefully\n \n* Performance tuning through metrics-driven decisions\n \n\nTomorrow, you’ll add compression to multiply these gains by another 5-10x. But the batching pattern you learned today is already production-grade and ready for real systems.","freeMarkdown","kind1EventId","kind30023EventId","1fb8e75040b76af6ab6ebeb41b870d191736709fcbaa8bea0e64e58c338f120b","kind30023",{"_14":211,"_186":214,"_188":215,"_62":216,"_201":21,"_152":230,"_203":231},"c06cf3b62d126315f27b3ed36f9e0cd072bd24399a25863a2ab653aa431622348bf6527636ffe649e26ca93be0a363ac104e3acd877759c59bd7961b6d3cdc07",30023,[217,220,221,224,225,227,228,229],[218,219],"r","https://drewdru.syndichain.com/articles/9a7c2aa2-7b4e-4b55-ac68-7ab44f96abde",[26,138],[222,223],"published_at","1762338728",[29,144],[226,133],"d",[173,139],[198,24],[200,21,24],"Day 11: Implement Batching in the Log Shipper to Optimize Network Usage\nWhat We’re Building Today\n-------------------------\n\n* **Intelligent batching log shipper**that accumulates logs before transmission\n \n* **Configurable batching strategies**with size-based and time-based triggers\n \n* **Memory-efficient buffer management**with backpressure handling\n \n* **Production-ready Spring Boot service**with Kafka integration and comprehensive monitoring\n \n\n* * *\n\nWhy This Matters: The Network Is Your Bottleneck\n------------------------------------------------\n\nEvery log transmission carries overhead: TCP handshakes, TLS negotiations, HTTP headers, kernel context switches. At scale, this overhead dominates your infrastructure costs. Netflix processes 500+ billion events daily—sending each individually would require millions of network connections per second. Instead, they batch aggressively, reducing network calls by 99% while maintaining sub-second latency.\n\nThe batching problem surfaces everywhere: Uber’s trip updates, Airbnb’s search queries, AWS CloudWatch metrics. The pattern is universal: accumulate data points, wait for a trigger (time or size), then transmit in bulk. But naive batching creates new problems: memory pressure from unbounded buffers, data loss during crashes, and head-of-line blocking where one slow batch stalls the entire pipeline.\n\nThis lesson implements production-grade batching that Netflix and AWS use in their ingestion pipelines. You’ll learn to balance throughput against latency, handle backpressure gracefully, and maintain data durability guarantees even when batching introduces delays.\n\n* * *\n\nSystem Design Deep Dive: Batching Strategies and Trade-offs\n-----------------------------------------------------------\n\n### Pattern 1: Dual-Trigger Batching (Time + Size)\n\nThe fundamental batching challenge: wait too long for full batches and latency suffers; send too early and throughput collapses. Production systems use dual triggers: flush when either batch size OR time interval is reached, whichever comes first.\n\n**Trade-off Analysis:**Size-based batching (e.g., 1000 records) maximizes throughput during high traffic but creates unbounded latency during quiet periods. Time-based batching (e.g., 5 seconds) bounds worst-case latency but wastes network efficiency during traffic spikes. Combining both gives you throughput during peaks and latency guarantees during valleys.\n\n**Real-world calibration:**AWS Kinesis uses 500KB or 1-second batches. Kafka producers default to 16KB or 10ms. Netflix’s edge proxies tune dynamically based on observed traffic patterns. Start with 1000 records or 5 seconds, then measure p99 latency and network utilization to tune.\n\n### Pattern 2: Bounded Queue with Backpressure\n\nUnbounded batching buffers are a memory leak waiting to happen. When your producer outpaces consumers, memory usage grows until OOM kills your process. Production systems use bounded queues: when full, apply backpressure by blocking producers or dropping logs.\n\n**Implementation consideration:**Java’s ArrayBlockingQueue provides thread-safe bounded buffering with configurable blocking behavior. Set capacity based on expected burst size (e.g., 10,000 records = ~1MB for 100-byte logs). Configure`offer()`with timeout for non-blocking backpressure or`put()`for blocking behavior.\n\n**Anti-pattern:**Never batch indefinitely without bounds. LinkedIn learned this when a batching bug caused servers to accumulate 40GB of logs in memory, triggering cascading OOM failures across their fleet.\n\n### Pattern 3: Graceful Degradation Under Load\n\nWhen system load exceeds capacity, you must choose: maintain latency by dropping data, or maintain durability by increasing latency. This choice depends on your data’s criticality. Application logs are often droppable; billing events never are.\n\n**Strategy:**Implement adaptive batching with load shedding. Under normal load, use standard batch parameters. When queue depth exceeds 80%, emit metrics and begin sampling (send 10% of logs). At 95%, drop non-critical logs entirely. Always preserve error logs and transactions.\n\n**Monitoring trigger:**Track`batch_queue_depth`,`logs_dropped_count`, and`average_batch_fill_ratio`. Alert when queue depth stays above 70% for 5 minutes—you’re approaching capacity limits.\n\n### Pattern 4: At-Least-Once Delivery Semantics\n\nBatching introduces a durability gap: logs in your batch buffer aren’t yet persisted to Kafka. If the shipper crashes, those logs vanish. Production systems accept this trade-off but minimize exposure through rapid batch flushes and periodic checkpointing.\n\n**Mitigation strategy:**Flush batches every 5 seconds maximum, even partially full. On shutdown signals (SIGTERM), flush immediately before exiting. For critical logs, write to a local disk buffer before batching (Write-Ahead Log pattern), then mark committed after Kafka acknowledgment.\n\n**Real-world compromise:**Most log aggregation accepts 5-10 seconds of potential data loss during crashes. Critical paths (payments, user actions) use immediate sends or persist locally first. Know your SLA and architect accordingly.\n\n### Pattern 5: Batch Size Distribution and Network Efficiency\n\nNot all batches are created equal. A batch of 1000 small logs (100 bytes each) behaves differently than 1000 large logs (10KB each). Network efficiency depends on payload size relative to overhead. Small payloads waste bandwidth on headers; large payloads risk timeout during transmission.\n\n**Optimal strategy:**Target 100KB-1MB batch payloads regardless of record count. Use separate batch size limits for small vs large logs. Compress batches before transmission (tomorrow’s lesson) to maximize payload density.\n\n**Performance math:**A 100-byte log with 500 bytes of overhead = 83% waste. Batch 1000 logs = 100KB payload + 500 bytes overhead = 99.5% efficiency. This 200x improvement in network utilization is why every high-scale system batches aggressively.\n\n* * *\n\nImplementation Walkthrough: Building the Batching Log Shipper\n-------------------------------------------------------------\n\nOur implementation uses Spring Boot’s scheduled executors for time-based flushing and ArrayBlockingQueue for bounded buffering. The LogBatchingService accumulates logs in memory and flushes to Kafka when either trigger fires.\n\n**Core architecture decision:**We separate batching logic from Kafka interaction. The BatchAccumulator handles buffering and trigger logic. The KafkaProducerService handles transmission with retry. This separation allows testing batching behavior without Kafka and enables swapping messaging layers if needed.\n\n**Key implementation details:**\n\n1. **Thread-safe accumulation:**Use ConcurrentLinkedQueue for lock-free log appends. Multiple producer threads can add logs without contention.\n \n2. **Scheduled flushing:**Spring’s @Scheduled annotation triggers time-based flushes every 5 seconds. Use fixedDelay to ensure flushes complete before scheduling the next.\n \n3. **Size-based triggers:**After each append, check buffer size. If >= maxBatchSize, submit a flush task to a separate executor to avoid blocking producers.\n \n4. **Graceful shutdown:**Implement ApplicationListenerto flush pending batches before shutdown. This prevents data loss during deployments.\n \n\nThe KafkaProducerService sends batches with acknowledgment=all for durability. We serialize batches as JSON arrays for simplicity, though Protocol Buffers would reduce payload size by 60% in production.\n\n**Load testing reveals:**At 1000 logs/second with 100-byte payloads, batching reduces Kafka requests from 1000/sec to 5/sec (200x reduction). Network bandwidth drops from 500KB/s to 120KB/s (4.2x improvement) due to reduced overhead. CPU usage decreases 40% from fewer kernel calls.\n\n* * *\n\nProduction Considerations: Failure Modes and Monitoring\n-------------------------------------------------------\n\n**Memory pressure monitoring:**Track heap usage and queue depth. Alert if queue depth exceeds 70% or heap usage grows unexpectedly. Set JVM max heap to 2x expected buffer size to prevent OOM.\n\n**Latency distribution:**Measure p50, p95, p99 time-to-delivery. Batching introduces variable latency: logs at batch start wait up to maxInterval; logs triggering size flush send immediately. Monitor to ensure p99 stays under your SLA.\n\n**Data loss scenarios:**Logs in buffer vanish if JVM crashes before flush. Mitigate with frequent flushes (5s max) and local disk buffering for critical logs. Track`logs_in_buffer`metric—persistent high values indicate flush problems.\n\n**Kafka unavailability:**When Kafka is down, batches accumulate in memory until buffer fills. Configure backpressure to drop logs rather than OOM. Emit`kafka_send_failures`metric and page on-call when failure rate exceeds 5%.\n\n**Thundering herd on restart:**If 100 shippers crash simultaneously and restart, all flush their buffers at once, overwhelming Kafka. Implement jittered startup delays (random 0-30 seconds) to stagger the load.\n\n* * *\n\nScale Connection: How FAANG Systems Batch\n-----------------------------------------\n\n**Netflix Edge Proxies:**Batch view events with 1-second windows to reduce backend load 100x. During peak (Friday 8pm), batching prevents their event ingestion from requiring millions of connections/second.\n\n**AWS Kinesis Streams:**Implements automatic batching in their SDK with 500KB or 1-second triggers. Their customers don’t think about batching—it’s transparent—but it reduces their infrastructure costs by 90%.\n\n**Uber’s Real-Time Analytics:**Batches trip events with 5-second windows before feeding their machine learning pipelines. Batching reduces their Kafka cluster size from 500 nodes to 50 nodes while maintaining sub-10-second freshness.\n\n**Google Cloud Logging:**Uses adaptive batching that adjusts interval based on traffic. During quiet periods, flushes every 10 seconds; during spikes, flushes every 100ms with 1000-log batches. This maintains 99th percentile latency under 15 seconds while maximizing throughput.\n\n* * *\n\nNext Steps: Compression Multiplies Batching Gains\n-------------------------------------------------\n\nTomorrow, we add compression to our batches. Gzip typically reduces log payloads by 5-10x because logs contain repetitive text. Combined with batching, you’ll achieve 1000x reduction in network usage: 200x from batching, 5x from compression. This combination is why modern log systems can ingest terabytes per day on modest infrastructure.\n\nWe’ll implement streaming compression to handle large batches without memory spikes, compare gzip vs snappy for latency-sensitive paths, and measure compression ratio vs CPU trade-offs. You’ll learn when compression hurts more than it helps (small payloads, low repetition) and how Netflix decides which compression algorithm to use for each data type.\n\n* * *\n\nSystem Architecture Overview\n============================\n\n[](https://substackcdn.com/image/fetch/$s_!fcuL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc6efc417-e39b-4a3e-a58c-02970fa22386_1009x621.png)\n\nThe system consists of four Spring Boot microservices backed by Kafka, PostgreSQL, and Redis:\n\n**Log Producer (Port 8081)**Simulates application servers generating logs at 10 logs/second. Sends individual log events to the shipper.\n\n**Log Shipper (Port 8082)**The batching layer. Accumulates logs in a bounded buffer (10,000 capacity) and flushes when either 1000 logs accumulate OR 5 seconds elapse. This is where the magic happens.\n\n**Apache Kafka**Message broker providing durability and decoupling. The log-events topic has 3 partitions for parallel processing.\n\n**Log Consumer (Port 8083)**Reads batched logs from Kafka and persists them to PostgreSQL for long-term storage.\n\n**API Gateway (Port 8080)**Query interface with Redis caching (60-second TTL) for fast log retrieval.\n\n**PostgreSQL**Persistent storage with indexes on timestamp, service, and log level for efficient queries.\n\n**Redis**Cache layer reducing database load for repeated queries.\n\n**Prometheus + Grafana**Monitoring stack collecting metrics from all services. Tracks batching efficiency, buffer utilization, and system health.\n\nYoutube Video:\n--------------\n\n* * *\n\nBuilding the System\n===================\n\n### Github Link :\n\n https://github.com/sysdr/sdc-java/tree/main/day11/log-batching-system\n\nPrerequisites\n-------------\n\nYou need these installed on your machine:\n\n* Docker Desktop (minimum 4GB RAM allocated)\n \n* Git\n \n* A terminal (Command Prompt, PowerShell, or Terminal)\n \n* Port availability: 8080-8083, 9090, 3000, 5432, 6379, 9092, 2181\n \n\nCheck Docker is running:\n\nbash\n\n docker --versiondocker-compose --version\n\nStep 1: Generate the Project Files\n----------------------------------\n\nDownload the`generate_system_files.sh`script from the lesson materials. This script creates the entire project structure with all code and configuration files.\n\nbash\n\n # Make the script executablechmod +x generate_system_files.sh# Run it./generate_system_files.sh\n\nYou’ll see output like:\n\n Generating Day 11: Batching Log Shipper System...✓ System generated successfully!\n\nThe script creates a`log-batching-system`directory containing:\n\n log-batching-system/├── log-producer/ # Generates synthetic logs├── log-shipper/ # Batching layer (main focus)├── log-consumer/ # Kafka consumer + PostgreSQL├── api-gateway/ # Query API + Redis cache├── docker-compose.yml # Infrastructure setup├── setup.sh # Automated deployment├── load-test.sh # Performance testing├── integration-tests/ # System validation├── monitoring/ # Prometheus + Grafana configs└── README.md\n\nStep 2: Understanding the Batching Service\n------------------------------------------\n\nLet’s look at the core batching logic before we run it. Open`log-shipper/src/main/java/com/example/logshipper/LogBatchingService.java`:\n\n**The buffer:**An ArrayBlockingQueue with 10,000 capacity prevents memory leaks:\n\njava\n\n private final BlockingQueuebuffer = new ArrayBlockingQueue<>(10000);\n\n**Adding logs with backpressure:**The`offer()`method returns false if buffer is full—we drop the log rather than crash:\n\n**Size-based trigger:**After adding, check if we hit the batch size limit:\n\njava\n\n if (currentBatchSize.get() >= maxBatchSize) { flushBatch(); // Send immediately when full}\n\n**Time-based trigger:**Spring scheduler flushes every 5 seconds regardless of size:\n\njava\n\n @Scheduled(fixedDelay = 5000)public void scheduledFlush() { if (currentBatchSize.get() > 0) { flushBatch(); }}\n\n**Graceful shutdown:**Before the application stops, flush remaining logs:\n\njava\n\n @Overridepublic void onApplicationEvent(ContextClosedEvent event) { log.info(”Shutting down, flushing remaining logs”); flushBatch();}\n\nThis design ensures logs never wait more than 5 seconds, network efficiency improves dramatically, and memory usage stays bounded.\n\n* * *\n\nDeploying and Running the System\n================================\n\nStep 1: Start the Infrastructure\n--------------------------------\n\nNavigate into the project directory:\n\nbash\n\n cd log-batching-system\n\nRun the automated setup:\n\nbash\n\n ./setup.sh\n\nThis script does several things:\n\n1. Starts Zookeeper and Kafka\n \n2. Starts PostgreSQL and Redis\n \n3. Starts Prometheus and Grafana\n \n4. Waits for infrastructure to be ready\n \n5. Creates the Kafka topic with 3 partitions\n \n6. Builds and starts all 4 application services\n \n\nYou’ll see output showing each service starting. The whole process takes about 2-3 minutes.\n\n Starting infrastructure services...Waiting for infrastructure to be ready...Checking Kafka...Creating Kafka topic...Building and starting application services...System is starting up...\n\nStep 2: Verify All Services Are Running\n---------------------------------------\n\nCheck Docker containers:\n\nbash\n\n docker-compose ps\n\nYou should see 11 containers running:\n\n* zookeeper\n \n* kafka\n \n* postgres\n \n* redis\n \n* prometheus\n \n* grafana\n \n* log-producer\n \n* log-shipper\n \n* log-consumer\n \n* api-gateway\n \n\nAll should show “Up” status.\n\nStep 3: Check Service Health\n----------------------------\n\nEach service has a health endpoint. Test them:\n\nbash\n\n # Log Producercurl http://localhost:8081/actuator/health# Log Shipper (batching layer)curl http://localhost:8082/api/logs/health# Log Consumercurl http://localhost:8083/actuator/health# API Gatewaycurl http://localhost:8080/api/query/health\n\nEach should return`{”status”:”UP”}`or similar.\n\n* * *\n\nTesting the Batching System\n===========================\n\nIntegration Test\n----------------\n\nRun the automated integration test suite:\n\nbash\n\n ./integration-tests/test-system.sh\n\nThis script:\n\n1. Waits 30 seconds for services to stabilize\n \n2. Checks all health endpoints\n \n3. Waits 60 seconds for logs to flow through the system\n \n4. Queries the API Gateway for recent logs\n \n5. Verifies metrics are being collected\n \n\nExpected output:\n\n Running integration tests...Test 1: Health checks...Test 2: Waiting for log processing...Test 3: Querying recent logs...✓ Found 247 logs in systemTest 4: Checking metrics...✓ Metrics endpoint working✓ All integration tests passed!\n\nIf any test fails, check the service logs:\n\nbash\n\n # View shipper logsdocker-compose logs log-shipper# View consumer logsdocker-compose logs log-consumer\n\nManual Verification: Watch Logs Flow\n------------------------------------\n\nOpen three terminal windows to watch the system in action.\n\n**Terminal 1 - Producer metrics:**\n\nbash\n\n watch -n 2 ‘curl -s http://localhost:8081/actuator/metrics/logs.generated | jq’\n\nThis shows how many logs the producer has generated.\n\n**Terminal 2 - Shipper metrics:**\n\nbash\n\n watch -n 2 ‘curl -s http://localhost:8082/actuator/metrics/batches.sent | jq’\n\nThis shows batches sent to Kafka. The count increases every 5 seconds.\n\n**Terminal 3 - Buffer size:**\n\nbash\n\n watch -n 1 ‘curl -s http://localhost:8082/actuator/metrics/buffer.size | jq’\n\nWatch the buffer fill up (0 to ~50 logs) then drop to 0 when flushed. This demonstrates the batching cycle.\n\nQuery Stored Logs\n-----------------\n\nAfter a minute, logs are in PostgreSQL. Query them:\n\nbash\n\n # Get 100 most recent logscurl http://localhost:8080/api/query/recent | jq ‘.[0:5]’\n\nYou’ll see output like:\n\njson\n\n [ { “id”: “a7f3c9d2-...”, “level”: “INFO”, “service”: “payment-service”, “message”: “Payment transaction initiated”, “timestamp”: “2025-10-09T14:32:15Z”, “traceId”: “8f3a2c1d” }, ...]\n\nQuery by service:\n\nbash\n\n curl http://localhost:8080/api/query/service/auth-service | jq ‘.[0:3]’\n\nQuery by log level:\n\nbash\n\n curl http://localhost:8080/api/query/level/ERROR | jq\n\n* * *\n\nLoad Testing: Proving Batching Works\n====================================\n\nNow let’s stress test the system to see batching in action.\n\nRun the Load Test\n-----------------\n\nbash\n\n ./load-test.sh\n\nThis script generates 10,000 log requests over 60 seconds (166 requests/second). You’ll see progress:\n\n Starting load test...This will generate 10,000 log requests over 60 secondsSent 100 requests... (Success: 100, Failed: 0)Sent 200 requests... (Success: 200, Failed: 0)...Sent 10000 requests... (Success: 10000, Failed: 0)Load test completed!Duration: 60sSuccessful: 10000Failed: 0Rate: 166 req/s\n\nObserve Batching Efficiency\n---------------------------\n\nWhile the load test runs, open these monitoring dashboards:\n\n**Prometheus (**\n\nhttp://localhost:9090\n\n**)**\n\nRun these queries:\n\n1. **Throughput:**How many logs per second are being sent?\n \n\npromql\n\n rate(logs_sent_total[1m])\n\nYou should see ~166 logs/second.\n\n1. **Batch rate:**How many batches per second?\n \n\npromql\n\n rate(batches_sent_total[1m])\n\nYou should see ~0.8-1.5 batches/second. This is the magic: 166 logs/sec entering, only 1 batch/sec leaving. That’s 166x reduction in network calls.\n\n1. **Average batch size:**\n \n\npromql\n\n rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n\nYou should see ~800-1000 logs per batch. When traffic is high, batches fill up and trigger the size limit (1000 logs).\n\n1. **Buffer utilization:**\n \n\npromql\n\n buffer_size\n\nDuring the load test, you’ll see the buffer fluctuate between 0 and 1000 as it fills and flushes.\n\n[![](https://substackcdn.com/image/fetch/$s_!PS8z!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83764f4c-8286-4b6c-a0a1-74d3802d51eb_623x554.png)](https://substackcdn.com/image/fetch/$s_!PS8z!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F83764f4c-8286-4b6c-a0a1-74d3802d51eb_623x554.png)\n\n**Grafana (**\n\nhttp://localhost:3000\n\n**)**\n\nLogin: admin / admin\n\nCreate a dashboard with these panels:\n\n1. Logs received vs logs sent (should match)\n \n2. Batches sent over time (shows batching frequency)\n \n3. Average batch size (shows batching efficiency)\n \n4. Buffer utilization (shows memory pressure)\n \n\n[![](https://substackcdn.com/image/fetch/$s_!U-Wa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcb47d1-5352-4b15-a195-1b829b211b04_866x625.png)](https://substackcdn.com/image/fetch/$s_!U-Wa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdbcb47d1-5352-4b15-a195-1b829b211b04_866x625.png)\n\nCompare: Without Batching\n-------------------------\n\nTo understand the improvement, let’s calculate what would happen without batching:\n\n**With Batching (actual):**\n\n* Network calls: 1 batch/second\n \n* Kafka messages: 1 message/second\n \n* Network overhead: ~500 bytes/second\n \n* Total bandwidth: ~100 KB/second\n \n\n**Without Batching (hypothetical):**\n\n* Network calls: 166 requests/second\n \n* Kafka messages: 166 messages/second\n \n* Network overhead: ~83 KB/second (500 bytes × 166)\n \n* Total bandwidth: ~100 KB/second + 83 KB overhead = 183 KB/second\n \n\nBatching reduced network calls by 166x and saved 45% bandwidth. At Netflix scale (500 billion events/day), this saves millions of dollars in infrastructure costs.\n\n* * *\n\nMonitoring Production Metrics\n=============================\n\nUnderstanding your metrics is crucial for operating this system in production.\n\nKey Metrics Explained\n---------------------\n\n**logs.received (counter)**Total logs accepted by the shipper. This should match what producers send.\n\n**logs.sent (counter)**Total logs successfully transmitted to Kafka. Should match logs.received (no loss).\n\n**logs.dropped (counter)**Logs discarded due to buffer full. This should always be 0 in healthy operation. If non-zero, you’re losing data.\n\n**batches.sent (counter)**Number of batches transmitted. Divide logs.sent by this to get average batch size.\n\n**batch.flush.duration (timer)**Time taken to serialize and send a batch to Kafka. P99 should be under 100ms.\n\n**buffer.size (gauge)**Current number of logs waiting in the buffer. Normal: 0-1000. Warning: 7000+. Critical: 9500+.\n\n**buffer.remaining.capacity (gauge)**How much space is left. If this stays near 0, increase buffer size or consumer throughput.\n\nSetting Up Alerts\n-----------------\n\nIn production, you’d configure these alerts:\n\n**Critical Alerts (page immediately):**\n\n1. `logs_dropped_total`\\> 0: Data loss occurring\n \n2. `buffer_remaining_capacity`< 500 for 5 minutes: About to drop logs\n \n3. `kafka_send_failures`\\> 5% for 2 minutes: Kafka connectivity issues\n \n\n**Warning Alerts (investigate soon):**\n\n1. `buffer_size`\\> 7000 for 10 minutes: Approaching capacity\n \n2. `batch_flush_duration`P99 > 500ms: Performance degrading\n \n3. `logs_sent / logs_received`< 0.99: Losing logs somewhere\n \n\n**Info Alerts (monitor trends):**\n\n1. Average batch size< 100: Not batching effectively\n \n2. Batch rate > 10/second: Consider tuning batch parameters\n \n\n* * *\n\nTesting Failure Scenarios\n=========================\n\nReal production systems fail. Let’s test how our batching handles failures gracefully.\n\nScenario 1: Kafka Goes Down\n---------------------------\n\nStop Kafka and watch what happens:\n\nbash\n\n docker-compose stop kafka\n\nObserve the shipper behavior:\n\nbash\n\n docker-compose logs -f log-shipper\n\nYou’ll see:\n\n WARN Failed to send batch to KafkaWARN Buffer size: 1000WARN Buffer size: 2000...WARN Buffer size: 9500WARN Buffer full, dropping log\n\nThe shipper:\n\n1. Continues accepting logs\n \n2. Buffer fills up\n \n3. Starts dropping logs to prevent OOM\n \n4. Emits metrics showing data loss\n \n\nCheck metrics:\n\nbash\n\n curl -s http://localhost:8082/actuator/metrics/logs.dropped\n\nRestart Kafka:\n\nbash\n\n docker-compose start kafka\n\nWithin 30 seconds, the buffer drains and normal operation resumes. Logs dropped during downtime are lost—this is our trade-off for low latency.\n\nScenario 2: Consumer Can’t Keep Up\n----------------------------------\n\nStop the consumer to simulate slow processing:\n\nbash\n\n docker-compose stop log-consumer\n\nThe Kafka topic accumulates messages but doesn’t affect the shipper. This demonstrates decoupling: the shipper’s job is just to get logs into Kafka. Downstream problems don’t cascade upstream.\n\nCheck Kafka lag:\n\nbash\n\n docker-compose exec kafka kafka-consumer-groups \\ --bootstrap-server localhost:9092 \\ --describe --group log-consumer-group\n\nYou’ll see increasing LAG values. This tells you the consumer is behind.\n\nRestart the consumer:\n\nbash\n\n docker-compose start log-consumer\n\nIt catches up by processing the backlog. No data lost because Kafka persisted everything.\n\nScenario 3: Graceful Shutdown\n-----------------------------\n\nStop the shipper cleanly:\n\nbash\n\n docker-compose stop log-shipper\n\nCheck the logs:\n\nbash\n\n docker-compose logs log-shipper | tail -20\n\nYou should see:\n\n INFO Shutting down, flushing remaining 47 logsINFO Flushed batch of 47 logs to KafkaINFO Application shutdown complete\n\nThe ApplicationListener triggered, flushing the partial batch before exit. Restart and verify no logs were lost by comparing producer and consumer counts.\n\n* * *\n\nUnderstanding the Performance Gains\n===================================\n\nLet’s analyze what we’ve achieved with concrete numbers.\n\nBaseline: No Batching\n---------------------\n\nImagine we send each log individually:\n\n* Producer generates: 10 logs/second\n \n* Network calls: 10 HTTP requests/second\n \n* Each request has ~500 bytes overhead (headers, handshake, etc.)\n \n* Overhead bandwidth: 5 KB/second\n \n* Log payload: 100 bytes × 10 = 1 KB/second\n \n* Total bandwidth: 6 KB/second\n \n* Kafka messages: 10/second\n \n* Context switches: ~40/second (producer, network, Kafka)\n \n\nWith Batching (Our System)\n--------------------------\n\nSame producer workload:\n\n* Producer generates: 10 logs/second\n \n* Shipper batches into ~2 batches every 10 seconds\n \n* Network calls: 0.2 HTTP requests/second\n \n* Request overhead: 500 bytes × 0.2 = 100 bytes/second\n \n* Log payload: Still 1 KB/second\n \n* Total bandwidth: 1.1 KB/second\n \n* Kafka messages: 0.2/second\n \n* Context switches: ~1/second\n \n\nThe Math\n--------\n\n* Network calls: 10 → 0.2 (50x reduction)\n \n* Overhead bandwidth: 5 KB → 0.1 KB (50x reduction)\n \n* Total bandwidth: 6 KB → 1.1 KB (5.5x reduction)\n \n* Kafka load: 10 msg/s → 0.2 msg/s (50x reduction)\n \n* CPU utilization: ~40% reduction from fewer system calls\n \n\nAt 1000 logs/second (load test):\n\n* Network calls: 1000 → 5 (200x reduction)\n \n* Kafka load: 1000 msg/s → 5 msg/s (200x reduction)\n \n\nThis is why batching is fundamental to every high-scale system. The gains multiply with load.\n\n* * *\n\nTuning Batch Parameters\n=======================\n\nThe default configuration (1000 logs or 5 seconds) works well for most cases, but you might need to tune for your specific requirements.\n\nAdjusting Batch Size\n--------------------\n\nEdit`log-shipper/src/main/resources/application.yml`:\n\nyaml\n\n batching: max-batch-size: 1000 # Change this buffer-capacity: 10000\n\n**Guidelines:**\n\n* **Increase**if you have high-volume, small logs (e.g., 5000 for 50-byte logs)\n \n* **Decrease**if you have large logs (e.g., 100 for 10KB logs)\n \n* Target: 100KB-1MB total batch payload\n \n\nRebuild and test:\n\nbash\n\n docker-compose up -d --build log-shipper\n\nMonitor average batch size:\n\npromql\n\n rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n\nAdjusting Time Interval\n-----------------------\n\nEdit`LogBatchingService.java`:\n\njava\n\n @Scheduled(fixedDelay = 5000) // milliseconds\n\n**Guidelines:**\n\n* **Decrease**for latency-sensitive systems (e.g., 1000ms = 1 second)\n \n* **Increase**for throughput-focused systems (e.g., 30000ms = 30 seconds)\n \n* Balance: Shorter interval = lower latency but less batching efficiency\n \n\nAdjusting Buffer Capacity\n-------------------------\n\nyaml\n\n batching: buffer-capacity: 10000 # Change this\n\n**Guidelines:**\n\n* Set to handle burst traffic:`peak_logs_per_second × burst_duration_seconds`\n \n* Example: 1000 logs/sec × 30 sec burst = 30,000 capacity\n \n* Memory cost: ~100 bytes/log = 3MB for 30,000 capacity\n \n* Always leave headroom: Set capacity 2x your expected max\n \n\n* * *\n\nCommon Issues and Solutions\n===========================\n\nProblem: Logs Not Appearing in PostgreSQL\n-----------------------------------------\n\n**Symptoms:**Producer shows logs generated, but API Gateway returns empty results.\n\n**Diagnosis:**\n\nbash\n\n # Check each stagecurl http://localhost:8081/actuator/metrics/logs.generatedcurl http://localhost:8082/actuator/metrics/logs.sentcurl http://localhost:8083/actuator/metrics/logs.processed\n\nIf logs.generated > logs.sent: Shipper isn’t flushing\n\nbash\n\n docker-compose logs log-shipper\n\nLook for Kafka connection errors.\n\nIf logs.sent > logs.processed: Consumer isn’t working\n\nbash\n\n docker-compose logs log-consumer\n\nCheck PostgreSQL connection.\n\n**Solution:**Restart the failing service:\n\nbash\n\n docker-compose restart log-shipper# ordocker-compose restart log-consumer\n\nProblem: High Memory Usage\n--------------------------\n\n**Symptoms:**Docker shows high memory usage, system slows down.\n\n**Diagnosis:**\n\nbash\n\n curl http://localhost:8082/actuator/metrics/buffer.size\n\nIf buffer.size stays near capacity (9000+), logs aren’t flushing.\n\n**Solutions:**\n\n1. Increase consumer throughput (scale horizontally)\n \n2. Reduce batch size (flush more frequently)\n \n3. Increase buffer capacity (if you have RAM)\n \n4. Accept data loss (current behavior: drops logs at capacity)\n \n\nProblem: Metrics Not Showing in Prometheus\n------------------------------------------\n\n**Symptoms:**Prometheus queries return no data.\n\n**Diagnosis:**\n\nCheck Prometheus targets:[http://localhost:9090/targets](http://localhost:9090/targets)\n\nAll should be “UP”. If “DOWN”, check the service health endpoint directly.\n\n**Solution:**\n\nRestart Prometheus:\n\nbash\n\n docker-compose restart prometheus\n\nWait 15 seconds for scraping to resume.\n\n* * *\n\nCleaning Up\n===========\n\nWhen you’re done experimenting:\n\nStop All Services\n-----------------\n\nbash\n\n docker-compose down\n\nThis stops containers but preserves data volumes.\n\nRemove All Data\n---------------\n\nbash\n\n docker-compose down -v\n\nThis deletes PostgreSQL data, Redis cache, Prometheus metrics, and Grafana dashboards.\n\nRemove Images\n-------------\n\nbash\n\n docker-compose down --rmi all\n\nThis frees disk space by removing all Docker images.\n\nComplete Reset\n--------------\n\nbash\n\n docker-compose down -v --rmi allcd ..rm -rf log-batching-system\n\nThis removes everything and lets you start fresh.\n\n* * *\n\nWhat You’ve Learned\n===================\n\nBy completing this lesson, you now understand:\n\n**Batching Fundamentals**\n\n* Why individual transmission is wasteful at scale\n \n* How dual triggers (size + time) balance latency and throughput\n \n* The trade-off between memory usage and batching efficiency\n \n\n**Production Patterns**\n\n* Bounded queues prevent memory leaks during backpressure\n \n* Graceful degradation (dropping logs) is better than crashing\n \n* Monitoring and alerting are essential for reliable operation\n \n\n**Real-World Applications**\n\n* This exact pattern powers Netflix, AWS, Uber, and Google systems\n \n* 200x network reduction is typical for high-volume systems\n \n* Combined with compression (tomorrow), gains reach 1000x\n \n\n**System Design Skills**\n\n* How to measure and prove system improvements\n \n* Failure modes and how to handle them gracefully\n \n* Performance tuning through metrics-driven decisions\n \n\nTomorrow, you’ll add compression to multiply these gains by another 5-10x. But the batching pattern you learned today is already production-grade and ready for real systems.\nYou can include dynamic values by using placeholders like: https://drewdru.syndichain.com/articles/9a7c2aa2-7b4e-4b55-ac68-7ab44f96abde , Drew Dru, https://sdcourse.substack.com/p/day-11-implement-batching-in-the-017 , npub1897arz, npub1897arz, npub1897arz, \n\nnpub1897arz These will automatically be replaced with the actual data when the message is sent. https://drewdru.syndichain.com/articles/9a7c2aa2-7b4e-4b55-ac68-7ab44f96abde npub1897arz\n",1762338728,"publishedToNostr","htmlContentKind30023","

What We’re Building Today

\n
Intelligent batching log shipperthat accumulates logs before transmission
\n
\n
Configurable batching strategieswith size-based and time-based triggers
\n
\n
Memory-efficient buffer managementwith backpressure handling
\n
\n
Production-ready Spring Boot servicewith Kafka integration and comprehensive monitoring
\n

Why This Matters: The Network Is Your Bottleneck

System Design Deep Dive: Batching Strategies and Trade-offs

Pattern 1: Dual-Trigger Batching (Time + Size)

**Trade-off Analysis:**Size-based batching (e.g., 1000 records) maximizes throughput during high traffic but creates unbounded latency during quiet periods. Time-based batching (e.g., 5 seconds) bounds worst-case latency but wastes network efficiency during traffic spikes. Combining both gives you throughput during peaks and latency guarantees during valleys.

**Real-world calibration:**AWS Kinesis uses 500KB or 1-second batches. Kafka producers default to 16KB or 10ms. Netflix’s edge proxies tune dynamically based on observed traffic patterns. Start with 1000 records or 5 seconds, then measure p99 latency and network utilization to tune.

Pattern 2: Bounded Queue with Backpressure

**Implementation consideration:**Java’s ArrayBlockingQueue provides thread-safe bounded buffering with configurable blocking behavior. Set capacity based on expected burst size (e.g., 10,000 records = ~1MB for 100-byte logs). Configureoffer()with timeout for non-blocking backpressure orput()for blocking behavior.

**Anti-pattern:**Never batch indefinitely without bounds. LinkedIn learned this when a batching bug caused servers to accumulate 40GB of logs in memory, triggering cascading OOM failures across their fleet.

Pattern 3: Graceful Degradation Under Load

**Strategy:**Implement adaptive batching with load shedding. Under normal load, use standard batch parameters. When queue depth exceeds 80%, emit metrics and begin sampling (send 10% of logs). At 95%, drop non-critical logs entirely. Always preserve error logs and transactions.

**Monitoring trigger:**Trackbatch_queue_depth,logs_dropped_count, andaverage_batch_fill_ratio. Alert when queue depth stays above 70% for 5 minutes—you’re approaching capacity limits.

Pattern 4: At-Least-Once Delivery Semantics

**Mitigation strategy:**Flush batches every 5 seconds maximum, even partially full. On shutdown signals (SIGTERM), flush immediately before exiting. For critical logs, write to a local disk buffer before batching (Write-Ahead Log pattern), then mark committed after Kafka acknowledgment.

**Real-world compromise:**Most log aggregation accepts 5-10 seconds of potential data loss during crashes. Critical paths (payments, user actions) use immediate sends or persist locally first. Know your SLA and architect accordingly.

Pattern 5: Batch Size Distribution and Network Efficiency

**Optimal strategy:**Target 100KB-1MB batch payloads regardless of record count. Use separate batch size limits for small vs large logs. Compress batches before transmission (tomorrow’s lesson) to maximize payload density.

**Performance math:**A 100-byte log with 500 bytes of overhead = 83% waste. Batch 1000 logs = 100KB payload + 500 bytes overhead = 99.5% efficiency. This 200x improvement in network utilization is why every high-scale system batches aggressively.

Implementation Walkthrough: Building the Batching Log Shipper

**Core architecture decision:**We separate batching logic from Kafka interaction. The BatchAccumulator handles buffering and trigger logic. The KafkaProducerService handles transmission with retry. This separation allows testing batching behavior without Kafka and enables swapping messaging layers if needed.

Key implementation details:

\n
**Thread-safe accumulation:**Use ConcurrentLinkedQueue for lock-free log appends. Multiple producer threads can add logs without contention.
\n
\n
**Scheduled flushing:**Spring’s @Scheduled annotation triggers time-based flushes every 5 seconds. Use fixedDelay to ensure flushes complete before scheduling the next.
\n
\n
**Size-based triggers:**After each append, check buffer size. If >= maxBatchSize, submit a flush task to a separate executor to avoid blocking producers.
\n
\n
**Graceful shutdown:**Implement ApplicationListenerto flush pending batches before shutdown. This prevents data loss during deployments.
\n

**Load testing reveals:**At 1000 logs/second with 100-byte payloads, batching reduces Kafka requests from 1000/sec to 5/sec (200x reduction). Network bandwidth drops from 500KB/s to 120KB/s (4.2x improvement) due to reduced overhead. CPU usage decreases 40% from fewer kernel calls.

Production Considerations: Failure Modes and Monitoring

**Memory pressure monitoring:**Track heap usage and queue depth. Alert if queue depth exceeds 70% or heap usage grows unexpectedly. Set JVM max heap to 2x expected buffer size to prevent OOM.

**Latency distribution:**Measure p50, p95, p99 time-to-delivery. Batching introduces variable latency: logs at batch start wait up to maxInterval; logs triggering size flush send immediately. Monitor to ensure p99 stays under your SLA.

**Data loss scenarios:**Logs in buffer vanish if JVM crashes before flush. Mitigate with frequent flushes (5s max) and local disk buffering for critical logs. Tracklogs_in_buffermetric—persistent high values indicate flush problems.

**Kafka unavailability:**When Kafka is down, batches accumulate in memory until buffer fills. Configure backpressure to drop logs rather than OOM. Emitkafka_send_failuresmetric and page on-call when failure rate exceeds 5%.

**Thundering herd on restart:**If 100 shippers crash simultaneously and restart, all flush their buffers at once, overwhelming Kafka. Implement jittered startup delays (random 0-30 seconds) to stagger the load.

Scale Connection: How FAANG Systems Batch

**Netflix Edge Proxies:**Batch view events with 1-second windows to reduce backend load 100x. During peak (Friday 8pm), batching prevents their event ingestion from requiring millions of connections/second.

**AWS Kinesis Streams:**Implements automatic batching in their SDK with 500KB or 1-second triggers. Their customers don’t think about batching—it’s transparent—but it reduces their infrastructure costs by 90%.

**Uber’s Real-Time Analytics:**Batches trip events with 5-second windows before feeding their machine learning pipelines. Batching reduces their Kafka cluster size from 500 nodes to 50 nodes while maintaining sub-10-second freshness.

**Google Cloud Logging:**Uses adaptive batching that adjusts interval based on traffic. During quiet periods, flushes every 10 seconds; during spikes, flushes every 100ms with 1000-log batches. This maintains 99th percentile latency under 15 seconds while maximizing throughput.

Next Steps: Compression Multiplies Batching Gains

System Architecture Overview

The system consists of four Spring Boot microservices backed by Kafka, PostgreSQL, and Redis:

**Log Producer (Port 8081)**Simulates application servers generating logs at 10 logs/second. Sends individual log events to the shipper.

**Log Shipper (Port 8082)**The batching layer. Accumulates logs in a bounded buffer (10,000 capacity) and flushes when either 1000 logs accumulate OR 5 seconds elapse. This is where the magic happens.

Apache KafkaMessage broker providing durability and decoupling. The log-events topic has 3 partitions for parallel processing.

**Log Consumer (Port 8083)**Reads batched logs from Kafka and persists them to PostgreSQL for long-term storage.

**API Gateway (Port 8080)**Query interface with Redis caching (60-second TTL) for fast log retrieval.

PostgreSQLPersistent storage with indexes on timestamp, service, and log level for efficient queries.

RedisCache layer reducing database load for repeated queries.

Prometheus + GrafanaMonitoring stack collecting metrics from all services. Tracks batching efficiency, buffer utilization, and system health.

Youtube Video:

Building the System

Github Link :

https://github.com/sysdr/sdc-java/tree/main/day11/log-batching-system\n

Prerequisites

You need these installed on your machine:

\n
Docker Desktop (minimum 4GB RAM allocated)
\n
\n
Git
\n
\n
A terminal (Command Prompt, PowerShell, or Terminal)
\n
\n
Port availability: 8080-8083, 9090, 3000, 5432, 6379, 9092, 2181
\n

Check Docker is running:

bash

docker --versiondocker-compose --version\n

Step 1: Generate the Project Files

Download thegenerate_system_files.shscript from the lesson materials. This script creates the entire project structure with all code and configuration files.

bash

# Make the script executablechmod +x generate_system_files.sh# Run it./generate_system_files.sh\n

You’ll see output like:

Generating Day 11: Batching Log Shipper System...✓ System generated successfully!\n

The script creates alog-batching-systemdirectory containing:

log-batching-system/├── log-producer/ # Generates synthetic logs├── log-shipper/ # Batching layer (main focus)├── log-consumer/ # Kafka consumer + PostgreSQL├── api-gateway/ # Query API + Redis cache├── docker-compose.yml # Infrastructure setup├── setup.sh # Automated deployment├── load-test.sh # Performance testing├── integration-tests/ # System validation├── monitoring/ # Prometheus + Grafana configs└── README.md\n

Step 2: Understanding the Batching Service

Let’s look at the core batching logic before we run it. Openlog-shipper/src/main/java/com/example/logshipper/LogBatchingService.java:

**The buffer:**An ArrayBlockingQueue with 10,000 capacity prevents memory leaks:

java

private final BlockingQueuebuffer = new ArrayBlockingQueue<>(10000);\n

**Adding logs with backpressure:**Theoffer()method returns false if buffer is full—we drop the log rather than crash:

**Size-based trigger:**After adding, check if we hit the batch size limit:

java

if (currentBatchSize.get() >= maxBatchSize) { flushBatch(); // Send immediately when full}\n

**Time-based trigger:**Spring scheduler flushes every 5 seconds regardless of size:

java

@Scheduled(fixedDelay = 5000)public void scheduledFlush() { if (currentBatchSize.get() > 0) { flushBatch(); }}\n

**Graceful shutdown:**Before the application stops, flush remaining logs:

java

@Overridepublic void onApplicationEvent(ContextClosedEvent event) { log.info(”Shutting down, flushing remaining logs”); flushBatch();}\n

This design ensures logs never wait more than 5 seconds, network efficiency improves dramatically, and memory usage stays bounded.

Deploying and Running the System

Step 1: Start the Infrastructure

Navigate into the project directory:

bash

cd log-batching-system\n

Run the automated setup:

bash

./setup.sh\n

This script does several things:

\n
Starts Zookeeper and Kafka
\n
\n
Starts PostgreSQL and Redis
\n
\n
Starts Prometheus and Grafana
\n
\n
Waits for infrastructure to be ready
\n
\n
Creates the Kafka topic with 3 partitions
\n
\n
Builds and starts all 4 application services
\n

You’ll see output showing each service starting. The whole process takes about 2-3 minutes.

Starting infrastructure services...Waiting for infrastructure to be ready...Checking Kafka...Creating Kafka topic...Building and starting application services...System is starting up...\n

Step 2: Verify All Services Are Running

Check Docker containers:

bash

docker-compose ps\n

You should see 11 containers running:

\n
zookeeper
\n
\n
kafka
\n
\n
postgres
\n
\n
redis
\n
\n
prometheus
\n
\n
grafana
\n
\n
log-producer
\n
\n
log-shipper
\n
\n
log-consumer
\n
\n
api-gateway
\n

All should show “Up” status.

Step 3: Check Service Health

Each service has a health endpoint. Test them:

bash

# Log Producercurl http://localhost:8081/actuator/health# Log Shipper (batching layer)curl http://localhost:8082/api/logs/health# Log Consumercurl http://localhost:8083/actuator/health# API Gatewaycurl http://localhost:8080/api/query/health\n

Each should return{”status”:”UP”}or similar.

Testing the Batching System

Integration Test

Run the automated integration test suite:

bash

./integration-tests/test-system.sh\n

This script:

\n
Waits 30 seconds for services to stabilize
\n
\n
Checks all health endpoints
\n
\n
Waits 60 seconds for logs to flow through the system
\n
\n
Queries the API Gateway for recent logs
\n
\n
Verifies metrics are being collected
\n

Expected output:

Running integration tests...Test 1: Health checks...Test 2: Waiting for log processing...Test 3: Querying recent logs...✓ Found 247 logs in systemTest 4: Checking metrics...✓ Metrics endpoint working✓ All integration tests passed!\n

If any test fails, check the service logs:

bash

# View shipper logsdocker-compose logs log-shipper# View consumer logsdocker-compose logs log-consumer\n

Manual Verification: Watch Logs Flow

Open three terminal windows to watch the system in action.

Terminal 1 - Producer metrics:

bash

watch -n 2 ‘curl -s http://localhost:8081/actuator/metrics/logs.generated | jq’\n

This shows how many logs the producer has generated.

Terminal 2 - Shipper metrics:

bash

watch -n 2 ‘curl -s http://localhost:8082/actuator/metrics/batches.sent | jq’\n

This shows batches sent to Kafka. The count increases every 5 seconds.

Terminal 3 - Buffer size:

bash

watch -n 1 ‘curl -s http://localhost:8082/actuator/metrics/buffer.size | jq’\n

Watch the buffer fill up (0 to ~50 logs) then drop to 0 when flushed. This demonstrates the batching cycle.

Query Stored Logs

After a minute, logs are in PostgreSQL. Query them:

bash

# Get 100 most recent logscurl http://localhost:8080/api/query/recent | jq ‘.[0:5]’\n

You’ll see output like:

json

[ { “id”: “a7f3c9d2-...”, “level”: “INFO”, “service”: “payment-service”, “message”: “Payment transaction initiated”, “timestamp”: “2025-10-09T14:32:15Z”, “traceId”: “8f3a2c1d” }, ...]\n

Query by service:

bash

curl http://localhost:8080/api/query/service/auth-service | jq ‘.[0:3]’\n

Query by log level:

bash

curl http://localhost:8080/api/query/level/ERROR | jq\n

Load Testing: Proving Batching Works

Now let’s stress test the system to see batching in action.

Run the Load Test

bash

./load-test.sh\n

This script generates 10,000 log requests over 60 seconds (166 requests/second). You’ll see progress:

Starting load test...This will generate 10,000 log requests over 60 secondsSent 100 requests... (Success: 100, Failed: 0)Sent 200 requests... (Success: 200, Failed: 0)...Sent 10000 requests... (Success: 10000, Failed: 0)Load test completed!Duration: 60sSuccessful: 10000Failed: 0Rate: 166 req/s\n

Observe Batching Efficiency

While the load test runs, open these monitoring dashboards:

Prometheus (

http://localhost:9090

)

Run these queries:

**Throughput:**How many logs per second are being sent?

promql

rate(logs_sent_total[1m])\n

You should see ~166 logs/second.

**Batch rate:**How many batches per second?

promql

rate(batches_sent_total[1m])\n

You should see ~0.8-1.5 batches/second. This is the magic: 166 logs/sec entering, only 1 batch/sec leaving. That’s 166x reduction in network calls.

Average batch size:

promql

rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n

You should see ~800-1000 logs per batch. When traffic is high, batches fill up and trigger the size limit (1000 logs).

Buffer utilization:

promql

buffer_size\n

During the load test, you’ll see the buffer fluctuate between 0 and 1000 as it fills and flushes.

$\"\"$

Grafana (

http://localhost:3000

)

Create a dashboard with these panels:

\n
Logs received vs logs sent (should match)
\n
\n
Batches sent over time (shows batching frequency)
\n
\n
Average batch size (shows batching efficiency)
\n
\n
Buffer utilization (shows memory pressure)
\n

$\"\"$

Compare: Without Batching

To understand the improvement, let’s calculate what would happen without batching:

With Batching (actual):

\n
Network calls: 1 batch/second
\n
\n
Kafka messages: 1 message/second
\n
\n
Network overhead: ~500 bytes/second
\n
\n
Total bandwidth: ~100 KB/second
\n

Without Batching (hypothetical):

\n
Network calls: 166 requests/second
\n
\n
Kafka messages: 166 messages/second
\n
\n
Network overhead: ~83 KB/second (500 bytes × 166)
\n
\n
Total bandwidth: ~100 KB/second + 83 KB overhead = 183 KB/second
\n

Batching reduced network calls by 166x and saved 45% bandwidth. At Netflix scale (500 billion events/day), this saves millions of dollars in infrastructure costs.

Monitoring Production Metrics

Understanding your metrics is crucial for operating this system in production.

Key Metrics Explained

**logs.received (counter)**Total logs accepted by the shipper. This should match what producers send.

**logs.sent (counter)**Total logs successfully transmitted to Kafka. Should match logs.received (no loss).

**logs.dropped (counter)**Logs discarded due to buffer full. This should always be 0 in healthy operation. If non-zero, you’re losing data.

**batches.sent (counter)**Number of batches transmitted. Divide logs.sent by this to get average batch size.

**batch.flush.duration (timer)**Time taken to serialize and send a batch to Kafka. P99 should be under 100ms.

**buffer.size (gauge)**Current number of logs waiting in the buffer. Normal: 0-1000. Warning: 7000+. Critical: 9500+.

**buffer.remaining.capacity (gauge)**How much space is left. If this stays near 0, increase buffer size or consumer throughput.

Setting Up Alerts

In production, you’d configure these alerts:

Critical Alerts (page immediately):

\n
logs_dropped_total> 0: Data loss occurring
\n
\n
buffer_remaining_capacity< 500 for 5 minutes: About to drop logs
\n
\n
kafka_send_failures> 5% for 2 minutes: Kafka connectivity issues
\n

Warning Alerts (investigate soon):

\n
buffer_size> 7000 for 10 minutes: Approaching capacity
\n
\n
batch_flush_durationP99 > 500ms: Performance degrading
\n
\n
logs_sent / logs_received< 0.99: Losing logs somewhere
\n

Info Alerts (monitor trends):

\n
Average batch size< 100: Not batching effectively
\n
\n
Batch rate > 10/second: Consider tuning batch parameters
\n

Testing Failure Scenarios

Real production systems fail. Let’s test how our batching handles failures gracefully.

Scenario 1: Kafka Goes Down

Stop Kafka and watch what happens:

bash

docker-compose stop kafka\n

Observe the shipper behavior:

bash

docker-compose logs -f log-shipper\n

You’ll see:

WARN Failed to send batch to KafkaWARN Buffer size: 1000WARN Buffer size: 2000...WARN Buffer size: 9500WARN Buffer full, dropping log\n

The shipper:

\n
Continues accepting logs
\n
\n
Buffer fills up
\n
\n
Starts dropping logs to prevent OOM
\n
\n
Emits metrics showing data loss
\n

Check metrics:

bash

curl -s http://localhost:8082/actuator/metrics/logs.dropped\n

Restart Kafka:

bash

docker-compose start kafka\n

Within 30 seconds, the buffer drains and normal operation resumes. Logs dropped during downtime are lost—this is our trade-off for low latency.

Scenario 2: Consumer Can’t Keep Up

Stop the consumer to simulate slow processing:

bash

docker-compose stop log-consumer\n

The Kafka topic accumulates messages but doesn’t affect the shipper. This demonstrates decoupling: the shipper’s job is just to get logs into Kafka. Downstream problems don’t cascade upstream.

Check Kafka lag:

bash

docker-compose exec kafka kafka-consumer-groups \\ --bootstrap-server localhost:9092 \\ --describe --group log-consumer-group\n

You’ll see increasing LAG values. This tells you the consumer is behind.

Restart the consumer:

bash

docker-compose start log-consumer\n

It catches up by processing the backlog. No data lost because Kafka persisted everything.

Scenario 3: Graceful Shutdown

Stop the shipper cleanly:

bash

docker-compose stop log-shipper\n

Check the logs:

bash

docker-compose logs log-shipper | tail -20\n

You should see:

INFO Shutting down, flushing remaining 47 logsINFO Flushed batch of 47 logs to KafkaINFO Application shutdown complete\n

The ApplicationListener triggered, flushing the partial batch before exit. Restart and verify no logs were lost by comparing producer and consumer counts.

Understanding the Performance Gains

Let’s analyze what we’ve achieved with concrete numbers.

Baseline: No Batching

Imagine we send each log individually:

\n
Producer generates: 10 logs/second
\n
\n
Network calls: 10 HTTP requests/second
\n
\n
Each request has ~500 bytes overhead (headers, handshake, etc.)
\n
\n
Overhead bandwidth: 5 KB/second
\n
\n
Log payload: 100 bytes × 10 = 1 KB/second
\n
\n
Total bandwidth: 6 KB/second
\n
\n
Kafka messages: 10/second
\n
\n
Context switches: ~40/second (producer, network, Kafka)
\n

With Batching (Our System)

Same producer workload:

\n
Producer generates: 10 logs/second
\n
\n
Shipper batches into ~2 batches every 10 seconds
\n
\n
Network calls: 0.2 HTTP requests/second
\n
\n
Request overhead: 500 bytes × 0.2 = 100 bytes/second
\n
\n
Log payload: Still 1 KB/second
\n
\n
Total bandwidth: 1.1 KB/second
\n
\n
Kafka messages: 0.2/second
\n
\n
Context switches: ~1/second
\n

The Math

\n
Network calls: 10 → 0.2 (50x reduction)
\n
\n
Overhead bandwidth: 5 KB → 0.1 KB (50x reduction)
\n
\n
Total bandwidth: 6 KB → 1.1 KB (5.5x reduction)
\n
\n
Kafka load: 10 msg/s → 0.2 msg/s (50x reduction)
\n
\n
CPU utilization: ~40% reduction from fewer system calls
\n

At 1000 logs/second (load test):

\n
Network calls: 1000 → 5 (200x reduction)
\n
\n
Kafka load: 1000 msg/s → 5 msg/s (200x reduction)
\n

This is why batching is fundamental to every high-scale system. The gains multiply with load.

Tuning Batch Parameters

The default configuration (1000 logs or 5 seconds) works well for most cases, but you might need to tune for your specific requirements.

Adjusting Batch Size

Editlog-shipper/src/main/resources/application.yml:

yaml

batching: max-batch-size: 1000 # Change this buffer-capacity: 10000\n

Guidelines:

\n
Increaseif you have high-volume, small logs (e.g., 5000 for 50-byte logs)
\n
\n
Decreaseif you have large logs (e.g., 100 for 10KB logs)
\n
\n
Target: 100KB-1MB total batch payload
\n

Rebuild and test:

bash

docker-compose up -d --build log-shipper\n

Monitor average batch size:

promql

rate(logs_sent_total[5m]) / rate(batches_sent_total[5m])\n

Adjusting Time Interval

EditLogBatchingService.java:

java

@Scheduled(fixedDelay = 5000) // milliseconds\n

Guidelines:

\n
Decreasefor latency-sensitive systems (e.g., 1000ms = 1 second)
\n
\n
Increasefor throughput-focused systems (e.g., 30000ms = 30 seconds)
\n
\n
Balance: Shorter interval = lower latency but less batching efficiency
\n

Adjusting Buffer Capacity

yaml

batching: buffer-capacity: 10000 # Change this\n

Guidelines:

\n
Set to handle burst traffic:peak_logs_per_second × burst_duration_seconds
\n
\n
Example: 1000 logs/sec × 30 sec burst = 30,000 capacity
\n
\n
Memory cost: ~100 bytes/log = 3MB for 30,000 capacity
\n
\n
Always leave headroom: Set capacity 2x your expected max
\n

Common Issues and Solutions

Problem: Logs Not Appearing in PostgreSQL

**Symptoms:**Producer shows logs generated, but API Gateway returns empty results.

Diagnosis:

bash

# Check each stagecurl http://localhost:8081/actuator/metrics/logs.generatedcurl http://localhost:8082/actuator/metrics/logs.sentcurl http://localhost:8083/actuator/metrics/logs.processed\n

If logs.generated > logs.sent: Shipper isn’t flushing

bash

docker-compose logs log-shipper\n

Look for Kafka connection errors.

If logs.sent > logs.processed: Consumer isn’t working

bash

docker-compose logs log-consumer\n

Check PostgreSQL connection.

**Solution:**Restart the failing service:

bash

docker-compose restart log-shipper# ordocker-compose restart log-consumer\n

Problem: High Memory Usage

**Symptoms:**Docker shows high memory usage, system slows down.

Diagnosis:

bash

curl http://localhost:8082/actuator/metrics/buffer.size\n

If buffer.size stays near capacity (9000+), logs aren’t flushing.

Solutions:

\n
Increase consumer throughput (scale horizontally)
\n
\n
Reduce batch size (flush more frequently)
\n
\n
Increase buffer capacity (if you have RAM)
\n
\n
Accept data loss (current behavior: drops logs at capacity)
\n

Problem: Metrics Not Showing in Prometheus

**Symptoms:**Prometheus queries return no data.

Diagnosis:

Check Prometheus targets:http://localhost:9090/targets

All should be “UP”. If “DOWN”, check the service health endpoint directly.

Solution:

Restart Prometheus:

bash

docker-compose restart prometheus\n

Wait 15 seconds for scraping to resume.

Cleaning Up

When you’re done experimenting:

Stop All Services

bash

docker-compose down\n

This stops containers but preserves data volumes.

Remove All Data

bash

docker-compose down -v\n

This deletes PostgreSQL data, Redis cache, Prometheus metrics, and Grafana dashboards.

Remove Images

bash

docker-compose down --rmi all\n

This frees disk space by removing all Docker images.

Complete Reset

bash

docker-compose down -v --rmi allcd ..rm -rf log-batching-system\n

This removes everything and lets you start fresh.

What You’ve Learned

By completing this lesson, you now understand:

Batching Fundamentals

\n
Why individual transmission is wasteful at scale
\n
\n
How dual triggers (size + time) balance latency and throughput
\n
\n
The trade-off between memory usage and batching efficiency
\n

Production Patterns

\n
Bounded queues prevent memory leaks during backpressure
\n
\n
Graceful degradation (dropping logs) is better than crashing
\n
\n
Monitoring and alerting are essential for reliable operation
\n

Real-World Applications

\n
This exact pattern powers Netflix, AWS, Uber, and Google systems
\n
\n
200x network reduction is typical for high-volume systems
\n
\n
Combined with compression (tomorrow), gains reach 1000x
\n

System Design Skills

\n
How to measure and prove system improvements
\n
\n
Failure modes and how to handle them gracefully
\n
\n
Performance tuning through metrics-driven decisions
\n

Tomorrow, you’ll add compression to multiply these gains by another 5-10x. But the batching pattern you learned today is already production-grade and ready for real systems.

","pubDate","2025-11-05T10:32:08.000Z","modifiedAt","2025-11-06T10:48:08.805Z","storyId","likes","reposts","comments",{"_14":15,"_16":17,"_18":19,"_20":21,"_22":-5,"_23":24,"_25":24,"_26":27,"_28":24,"_29":24,"_30":24,"_31":-5,"_32":-5,"_33":24,"_34":35,"_36":37,"_38":37,"_39":40,"_41":244,"_74":246},{"_14":40,"_43":44,"_45":44,"_46":27,"_28":47,"_48":49,"_50":-5,"_51":52,"_53":-5,"_54":55,"_56":-5,"_57":47,"_58":27,"_59":21,"_60":61,"_62":245,"_64":-5,"_65":66,"_67":55,"_68":55,"_69":-5,"_70":-5,"_71":-5,"_36":72,"_38":73},[],{"_76":15,"_77":-5,"_78":-5,"_79":80,"_81":82,"_83":247,"_87":248,"_91":92},{"_85":66,"_86":66},[89,90],"relatedArticles",[],"actionData","errors"]