What We’re Building Today

Today we’re transforming our basic log storage system into a production-grade distributed query engine that handles real-time log processing at scale. You’ll implement:

Distributed Log Query API: RESTful service with advanced filtering, aggregation, and real-time search capabilities
Event-Driven Processing Pipeline: Kafka-based system processing 10K+ logs/second with guaranteed delivery
Intelligent Caching Layer: Redis-powered query optimization reducing response times from 2s to 50ms
Production Monitoring Stack: Complete observability with Prometheus metrics, Grafana dashboards, and distributed tracing

Why This Matters: From Local Files to Planet-Scale Systems

The jump from flat file storage to distributed query processing represents one of the most critical scaling challenges in modern systems. What starts as a simple grep command on a single server evolves into complex distributed architectures when you need to search petabytes of logs across thousands of machines in real-time.
Consider Netflix’s challenge: processing 500+ billion events daily across global infrastructure. Their logs aren’t just stored—they’re actively queried by fraud detection systems, recommendation engines, and operational dashboards. A single query might need to aggregate data across 50 different microservices, each running in multiple regions. The system you’re building today demonstrates the foundational patterns that make this scale possible.
The core architectural challenge isn’t just storage or retrieval—it’s maintaining query performance and system availability while data volume grows exponentially. Today’s implementation introduces you to the distributed systems trade-offs that separate senior engineers from those still thinking in single-machine terms.

System Design Deep Dive: Five Core Patterns for Distributed Query Processing

1. Event Sourcing with Command Query Responsibility Segregation (CQRS)

Traditional systems couple write and read operations, creating bottlenecks when query patterns differ from write patterns. Our log processing system implements CQRS by separating:

Write Side: High-throughput log ingestion via Kafka with minimal validation
Read Side: Optimized query structures with pre-computed aggregations and indexes

This separation allows us to scale reads and writes independently. When Netflix needs to process millions of viewing events while simultaneously running complex analytics queries, CQRS prevents read-heavy analytics from slowing down real-time event ingestion.

Trade-off Analysis: CQRS introduces eventual consistency between write and read models. Your query results might be 100-500ms behind real-time events, but you gain the ability to handle 10x more concurrent queries. For log processing, this trade-off is almost always worth it—operational dashboards can tolerate slight delays if it means avoiding system overload during traffic spikes.

2. Distributed Caching with Cache-Aside Pattern

Raw log queries against PostgreSQL become prohibitively expensive beyond 1M records. Our Redis implementation uses the cache-aside pattern where:

Application code manages cache population and invalidation
Cache misses trigger database queries with automatic cache warming
TTL policies balance data freshness with performance

Key Insight: Cache hit ratios above 95% are achievable for log queries because of temporal locality—recent logs are queried far more frequently than historical data. However, cache invalidation becomes complex when implementing real-time updates.

Anti-Pattern Warning: Never implement write-through caching for high-velocity log data. The cache invalidation overhead will negate performance benefits and create consistency nightmares during failure scenarios.

3. Circuit Breaker Pattern for Fault Isolation

Distributed systems fail in cascade patterns—one slow component triggers timeouts across the entire system. Our Resilience4j implementation provides:

Automatic failure detection with configurable thresholds
Fast-fail responses preventing resource exhaustion
Gradual recovery with half-open state testing

When Uber’s map matching service experiences high latency, circuit breakers prevent the delay from propagating to rider assignment and pricing services. The same pattern protects our log query system from database overload scenarios.

4. Asynchronous Processing with Guaranteed Delivery

Kafka’s producer-consumer model ensures log processing continues even when downstream services are temporarily unavailable. Our implementation includes:

At-least-once delivery with producer retries and consumer acknowledgments
Dead letter queues for handling poison messages
Consumer group management for horizontal scaling

Scalability Implication: This pattern allows you to decouple ingestion rate from processing rate. During traffic spikes, Kafka acts as a buffer, preventing log loss while consumers catch up.

5. Distributed Observability with Correlation IDs

Production distributed systems require comprehensive observability. Our implementation includes:

Distributed tracing with correlation IDs propagated across service boundaries
Structured logging with consistent metadata for aggregation
Custom metrics measuring business logic, not just infrastructure

Implementation Walkthrough: Building Production-Grade Components

Core Architecture Decisions

Our system implements a three-service architecture optimized for independent scaling:

Log Producer Service: Handles high-velocity ingestion with minimal processing overhead. Critical design decision: validation occurs asynchronously to maintain ingestion throughput. Invalid logs are routed to dead letter queues rather than blocking the main pipeline.

Log Consumer Service: Processes Kafka events and maintains read-optimized data structures. Key optimization: batch processing with configurable flush intervals balances latency with throughput. During high load, larger batches improve database write efficiency.

API Gateway Service: Provides query interface with intelligent routing and caching. Implements query complexity analysis to prevent expensive operations from overwhelming the system.

Database Schema Optimization

Traditional log tables become unusable beyond 10M records without proper indexing strategy:

-- Composite index optimized for time-range queries\nCREATE INDEX idx_logs_timestamp_level ON logs(timestamp DESC, log_level);\n\n-- Partial index for error-only queries (common pattern)\nCREATE INDEX idx_logs_errors ON logs(timestamp DESC) WHERE log_level = ‘ERROR’;\n

Caching Strategy Implementation

Redis serves three distinct caching layers:

Query Result Caching: Exact query matches with 5-minute TTL
Aggregation Caching: Pre-computed hourly/daily summaries with 1-hour TTL
Hot Data Caching: Recent logs (last 15 minutes) with 30-second TTL

Cache key design prevents collisions while enabling pattern-based invalidation during data updates.

Production Considerations: Performance and Reliability

Performance Benchmarks

Under normal load (1,000 logs/second), the system maintains:

Query response times under 100ms (95th percentile)
Cache hit ratios above 94%
End-to-end processing latency under 200ms

During spike conditions (10,000 logs/second), performance gracefully degrades:

Query response times increase to 300ms (acceptable for operational dashboards)
Circuit breakers prevent cascade failures
Kafka buffering maintains zero data loss

Failure Scenarios and Recovery

Database Connection Loss: Circuit breakers activate after 3 consecutive failures, routing all queries to cached data. System continues serving recent logs while database recovers.

Kafka Cluster Outage: Producer buffering and retry logic maintain data integrity. Consumer lag is measured and alerts trigger when processing falls behind ingestion.

Redis Cluster Failure: System automatically falls back to direct database queries with degraded performance. Cache warming begins immediately upon Redis recovery.

Scale Connection: Enterprise Production Patterns

This architecture mirrors production systems at scale. Airbnb’s log processing system uses identical patterns to handle 2M+ events per minute across their booking, pricing, and fraud detection systems. The key scaling strategies you’ve implemented today—CQRS separation, distributed caching, and asynchronous processing—remain constant as you move from thousands to millions of events per second.

The primary difference at massive scale is data partitioning strategy and cross-region replication, concepts we’ll explore in Week 3 when building geographically distributed systems.

Working Code Demo:

Next Steps: Tomorrow’s Advanced Concepts

Day 7 focuses on service mesh integration and advanced routing patterns. You’ll implement intelligent load balancing, A/B testing infrastructure, and cross-service authentication—the missing pieces for production deployment at enterprise scale.

Building Your System: Step-by-Step Implementation

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day6/distributed-log-processor

Prerequisites

Before starting, make sure you have these tools installed on your computer:

Java 17 or newer - The programming language we’re using
Maven 3.6 or newer - Builds our Java projects
Docker and Docker Compose - Runs our infrastructure services
A code editor - VS Code, IntelliJ IDEA, or Eclipse

You can verify your installations by running these commands in your terminal:

java -version\nmvn -version\ndocker --version\ndocker-compose --version\n

Step 1: Generate the Project Files

We’ve created a script that automatically generates all the code and configuration files you need. This saves you from typing thousands of lines of code manually.

Run the generator script:

chmod +x generate_system_files.sh\n./generate_system_files.sh\n

This creates a folder called distributed-log-processor containing:

Three Spring Boot services (Producer, Consumer, Gateway)
Database configurations
Docker setup files
Testing scripts
Complete documentation

Navigate into the project:

cd distributed-log-processor\n

Step 2: Start the Infrastructure Services

Our application needs several supporting services to run. Think of these as the foundation of your house—you need them in place before you can build the walls.

Start all infrastructure services:

./setup.sh\n

This script starts:

Kafka - Message streaming platform
PostgreSQL - Main database for storing logs
Redis - Fast cache for query results
Prometheus - Collects performance metrics
Grafana - Creates visual dashboards

The script takes about 30-60 seconds to start everything. You’ll see messages as each service comes online.

Verify everything is running:

docker-compose ps\n

You should see all services showing “Up” status.

Step 3: Build the Application Services

Now we’ll compile our three Spring Boot services into executable JAR files.

Build all services at once:

mvn clean install\n

This command:

Downloads all necessary libraries
Compiles your Java code
Runs automated tests
Packages everything into JAR files

The build takes 2-3 minutes on first run. Subsequent builds are faster because Maven caches the libraries.

You’ll see output showing each service being built. Look for “BUILD SUCCESS” messages for all three modules.

Step 4: Start the Application Services

Now comes the exciting part—starting your distributed system! Open three separate terminal windows to run each service. This lets you see the logs from each one independently.

Terminal 1 - Start the Log Producer:

java -jar log-producer/target/log-producer-1.0.0.jar\n

Wait until you see: “Started LogProducerApplication”

Terminal 2 - Start the Log Consumer:

java -jar log-consumer/target/log-consumer-1.0.0.jar\n

Wait until you see: “Started LogConsumerApplication”

Terminal 3 - Start the API Gateway:

java -jar api-gateway/target/api-gateway-1.0.0.jar\n

Wait until you see: “Started ApiGatewayApplication”

Step 5: Verify System Health

Let’s make sure all services can communicate with each other. We’ll check the health endpoints that each service provides.

Check each service:

# Producer health check\ncurl http://localhost:8081/actuator/health\n\n# Consumer health check\ncurl http://localhost:8082/actuator/health\n\n# Gateway health check\ncurl http://localhost:8080/actuator/health\n

Each should return: {”status”:”UP”}

If any service returns an error, check its terminal window for error messages.

Step 6: Your First Log Event

Time to send your first log through the system! We’ll use curl to send an HTTP request to the producer service.

Send a test log:

curl -X POST http://localhost:8081/api/v1/logs \\\n  -H “Content-Type: application/json” \\\n  -d ‘{\n    “message”: “My first distributed log event!”,\n    “level”: “INFO”,\n    “source”: “learning-system”,\n    “metadata”: {\n      “student”: “your-name”,\n      “lesson”: “day-6”\n    }\n  }’\n

You should see: “Log event accepted for processing”

Watch the terminal windows—you’ll see the log flow through the system:

Producer receives it and sends to Kafka
Consumer reads from Kafka and saves to database
Gateway makes it available for queries

Step 7: Query Your Logs

Now let’s retrieve the log you just sent. The gateway service provides a powerful query API.

Get recent logs:

curl “http://localhost:8080/api/v1/logs?size=10”\n

You’ll see a JSON response containing your log entry with all its details.

Try more advanced queries:

# Get only ERROR level logs\ncurl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n\n# Search for specific keywords\ncurl “http://localhost:8080/api/v1/logs?keyword=distributed”\n\n# Get logs from a specific source\ncurl “http://localhost:8080/api/v1/logs?source=learning-system”\n

IMAGE PLACEHOLDER: Query Response Example Formatted JSON output showing log query results

Step 8: Send Multiple Logs (Batch Processing)

Our system can handle many logs at once. Let’s test the batch endpoint.

Send a batch of logs:

curl -X POST http://localhost:8081/api/v1/logs/batch \\\n  -H “Content-Type: application/json” \\\n  -d ‘[\n    {\n      “message”: “User logged in successfully”,\n      “level”: “INFO”,\n      “source”: “auth-service”\n    },\n    {\n      “message”: “Failed to connect to database”,\n      “level”: “ERROR”,\n      “source”: “db-service”\n    },\n    {\n      “message”: “Payment processed”,\n      “level”: “INFO”,\n      “source”: “payment-service”\n    }\n  ]’\n

Query to see all your logs:

curl “http://localhost:8080/api/v1/logs?size=20”\n

Step 9: View System Statistics

The gateway provides an analytics endpoint that shows you interesting statistics about your logs.

Get statistics:

curl “http://localhost:8080/api/v1/logs/stats”\n

This returns:

Count of logs by level (INFO, ERROR, etc.)
Top sources generating logs
Total log count in time range

Step 10: Run the Load Test

Let’s stress test your system to see how it handles high volume. The load test script sends 1,000 logs and makes 50 queries simultaneously.

Run the load test:

./load-test.sh\n

Watch the terminal output as it:

Sends 10 batches of 100 logs each
Performs 50 different queries
Reports success/failure for each operation

The test takes about 30-40 seconds to complete.

Step 11: Monitor Performance with Prometheus

Prometheus collects metrics from all your services. Let’s see what it’s tracking.

Open Prometheus in your browser:

http://localhost:9090\n

Try these queries in the Prometheus UI:

Query 1 - Log ingestion rate:

rate(logs_received_total[5m])\n

Query 2 - Cache hit ratio:

rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) * 100\n

Query 3 - Average query response time:

rate(log_query_duration_seconds_sum[5m]) / rate(log_query_duration_seconds_count[5m])\n

Click “Execute” and “Graph” to see visual representations.

Step 12: Create Dashboards in Grafana

Grafana turns your metrics into beautiful, real-time dashboards.

Open Grafana in your browser:

http://localhost:3000\n

Username: admin
Password: admin

(You’ll be prompted to change the password—you can skip this for now)

The system comes with a pre-built dashboard called “Log Processor System Dashboard” that shows:

Log ingestion rate over time
Cache hit percentage
Query performance
Error rates

Step 13: Run Integration Tests

Integration tests verify that all components work together correctly. These tests simulate real-world usage patterns.

Run the integration test suite:

./integration-tests/system-integration-test.sh\n

The tests check:

All services are responding
Logs can be sent and retrieved
Queries return correct results
Statistics are calculated properly

You’ll see “All integration tests passed!” if everything works.

Understanding What You Built

Let’s review what each component does and how they work together:

The Log Producer Service (Port 8081)

This is the entry point for logs. Applications send their logs here via HTTP requests. The producer:

Validates incoming logs
Immediately returns a response (doesn’t wait for processing)
Sends logs to Kafka for asynchronous processing
Tracks how many logs it receives

Apache Kafka (Port 9092)

Think of Kafka as a super-fast, reliable message highway. It:

Stores logs temporarily in topics (named channels)
Guarantees logs won’t be lost even if services crash
Allows multiple consumers to read the same logs
Handles millions of messages per second

The Log Consumer Service (Port 8082)

This service reads from Kafka and saves logs to the database. It:

Processes logs in batches for efficiency
Acknowledges messages only after successful storage
Handles failures gracefully with retries
Keeps track of processing progress

PostgreSQL Database (Port 5432)

The permanent storage for all logs. It features:

Optimized indexes for fast time-based queries
Support for complex filtering and aggregation
ACID guarantees for data consistency
Ability to store billions of log records

Redis Cache (Port 6379)

An extremely fast in-memory cache that:

Stores frequently accessed query results
Reduces database load by 90%+
Expires old data automatically
Returns results in milliseconds

The API Gateway Service (Port 8080)

Your query interface. It:

Checks Redis cache first (cache-aside pattern)
Falls back to database on cache miss
Implements circuit breakers for fault tolerance
Provides REST APIs for searching logs

Prometheus (Port 9090)

Collects metrics from all services:

Scrapes metrics every 15 seconds
Stores time-series data
Enables alerting on thresholds
Powers Grafana dashboards

Grafana (Port 3000)

Visualizes your metrics:

Creates real-time dashboards
Shows trends and patterns
Helps identify performance issues
Makes data accessible to everyone

Key Concepts You’ve Learned

Eventual Consistency

When you send a log to the producer, it returns immediately. But the log isn’t queryable instantly—it takes 100-500ms to flow through Kafka, get processed by the consumer, and land in the database. This delay is called eventual consistency.

Why it matters: This trade-off lets your system handle way more logs. If every log had to be saved before responding, your system would be much slower.

Circuit Breaker Pattern

If the database goes down, the circuit breaker “opens” after detecting failures. Instead of trying the database repeatedly, the gateway serves cached data only. Once the database recovers, the circuit breaker gradually “closes” again.

Why it matters: Prevents cascade failures where one slow component brings down the entire system.

Cache-Aside Pattern

The gateway doesn’t automatically populate the cache. Instead:

Check if result exists in cache
If yes, return it (cache hit)
If no, query database (cache miss)
Store result in cache for next time

Why it matters: Simple to implement and works well for read-heavy workloads like log queries.

Horizontal Scaling

Each service is stateless—it doesn’t store any data in memory that matters. This means you can run multiple copies:

3 producer instances can share incoming requests
5 consumer instances can divide Kafka partitions
4 gateway instances can handle more queries

Why it matters: As your traffic grows, just add more servers. No fundamental redesign needed.

Troubleshooting Common Issues

Problem: Services won’t start

Check 1: Are the ports already in use?

lsof -i :8080  # Check gateway port\nlsof -i :8081  # Check producer port\nlsof -i :8082  # Check consumer port\n

Solution: Stop any other programs using these ports.

Check 2: Is Docker running?

docker ps\n

Solution: Start Docker Desktop or the Docker daemon.

Problem: Can’t connect to Kafka

Check: Is Kafka healthy?

docker exec kafka kafka-topics --list --bootstrap-server localhost:9092\n

Solution: Restart Kafka:

docker-compose restart kafka\n

Problem: Queries are very slow

Check: What’s the cache hit rate?

curl http://localhost:8080/actuator/metrics/cache.gets\n

Solution: If hit rate is low, your cache TTL might be too short. Check Redis:

docker exec redis redis-cli info stats\n

Problem: Consumer isn’t processing logs

Check: Consumer lag in Kafka:

docker exec kafka kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-consumer-group\n

Solution: If lag is growing, the consumer might be overwhelmed. Check the consumer service logs for errors.

Experimenting Further

Now that your system is running, try these experiments to learn more:

Experiment 1: Break the Database

Stop PostgreSQL and see what happens:

docker-compose stop postgres\n

Send logs and try queries. Notice:

Producer still accepts logs (Kafka buffers them)
Gateway serves cached queries
Circuit breaker prevents errors

Restart the database:

docker-compose start postgres\n

Watch the consumer catch up on processing the buffered logs.

Experiment 2: Cache Performance

Send the same query twice and compare response times:

time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\ntime curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n

The second query should be much faster (cached result).

Experiment 3: Batch vs. Individual

Compare sending 100 logs individually vs. one batch:

Individual (slow):

for i in {1..100}; do\n  curl -X POST http://localhost:8081/api/v1/logs \\\n    -H “Content-Type: application/json” \\\n    -d ‘{”message”:”Log ‘$i’”,”level”:”INFO”,”source”:”test”}’\ndone\n

Batch (fast): Use the batch endpoint with 100 logs in one request.

Notice how much faster batching is!

What’s Next

You’ve built a distributed system that handles thousands of logs per second. Here’s what you can explore next:

Add More Features:

User authentication for the API
Rate limiting to prevent abuse
Data retention policies (auto-delete old logs)
Full-text search with Elasticsearch

Scale It Further:

Deploy to Kubernetes
Add multiple Kafka brokers
Set up database replication
Implement cross-region deployment

Production Hardening:

Add encryption for data in transit
Implement audit logging
Set up automated backups
Create runbooks for incidents

Tomorrow in Day 7, we’ll add a service mesh to handle advanced routing, authentication between services, and implement A/B testing capabilities.

Cleaning Up

When you’re done experimenting, shut everything down:

# Stop application services\n# Press Ctrl+C in each terminal window\n\n# Stop infrastructure\ndocker-compose down\n\n# Remove data volumes (optional - deletes all logs)\ndocker-compose down -v\n

Your code and configuration files remain intact, so you can restart anytime by running the setup script again.

","contentEncoded","freeContentEncoded","textContent","What We’re Building TodayToday we’re transforming our basic log storage system into a production-grade distributed query engine that handles real-time log processing at scale. You’ll implement:Distributed Log Query API: RESTful service with advanced filtering, aggregation, and real-time search capabilitiesEvent-Driven Processing Pipeline: Kafka-based system processing 10K+ logs/second with guaranteed deliveryIntelligent Caching Layer: Redis-powered query optimization reducing response times from 2s to 50msProduction Monitoring Stack: Complete observability with Prometheus metrics, Grafana dashboards, and distributed tracingWhy This Matters: From Local Files to Planet-Scale SystemsThe jump from flat file storage to distributed query processing represents one of the most critical scaling challenges in modern systems. What starts as a simple grep command on a single server evolves into complex distributed architectures when you need to search petabytes of logs across thousands of machines in real-time.Consider Netflix’s challenge: processing 500+ billion events daily across global infrastructure. Their logs aren’t just stored—they’re actively queried by fraud detection systems, recommendation engines, and operational dashboards. A single query might need to aggregate data across 50 different microservices, each running in multiple regions. The system you’re building today demonstrates the foundational patterns that make this scale possible.The core architectural challenge isn’t just storage or retrieval—it’s maintaining query performance and system availability while data volume grows exponentially. Today’s implementation introduces you to the distributed systems trade-offs that separate senior engineers from those still thinking in single-machine terms.System Design Deep Dive: Five Core Patterns for Distributed Query Processing1. Event Sourcing with Command Query Responsibility Segregation (CQRS)Traditional systems couple write and read operations, creating bottlenecks when query patterns differ from write patterns. Our log processing system implements CQRS by separating:Write Side: High-throughput log ingestion via Kafka with minimal validationRead Side: Optimized query structures with pre-computed aggregations and indexesThis separation allows us to scale reads and writes independently. When Netflix needs to process millions of viewing events while simultaneously running complex analytics queries, CQRS prevents read-heavy analytics from slowing down real-time event ingestion.Trade-off Analysis: CQRS introduces eventual consistency between write and read models. Your query results might be 100-500ms behind real-time events, but you gain the ability to handle 10x more concurrent queries. For log processing, this trade-off is almost always worth it—operational dashboards can tolerate slight delays if it means avoiding system overload during traffic spikes.2. Distributed Caching with Cache-Aside PatternRaw log queries against PostgreSQL become prohibitively expensive beyond 1M records. Our Redis implementation uses the cache-aside pattern where:Application code manages cache population and invalidationCache misses trigger database queries with automatic cache warmingTTL policies balance data freshness with performanceKey Insight: Cache hit ratios above 95% are achievable for log queries because of temporal locality—recent logs are queried far more frequently than historical data. However, cache invalidation becomes complex when implementing real-time updates.Anti-Pattern Warning: Never implement write-through caching for high-velocity log data. The cache invalidation overhead will negate performance benefits and create consistency nightmares during failure scenarios.3. Circuit Breaker Pattern for Fault IsolationDistributed systems fail in cascade patterns—one slow component triggers timeouts across the entire system. Our Resilience4j implementation provides:Automatic failure detection with configurable thresholdsFast-fail responses preventing resource exhaustionGradual recovery with half-open state testingWhen Uber’s map matching service experiences high latency, circuit breakers prevent the delay from propagating to rider assignment and pricing services. The same pattern protects our log query system from database overload scenarios.4. Asynchronous Processing with Guaranteed DeliveryKafka’s producer-consumer model ensures log processing continues even when downstream services are temporarily unavailable. Our implementation includes:At-least-once delivery with producer retries and consumer acknowledgmentsDead letter queues for handling poison messagesConsumer group management for horizontal scalingScalability Implication: This pattern allows you to decouple ingestion rate from processing rate. During traffic spikes, Kafka acts as a buffer, preventing log loss while consumers catch up.5. Distributed Observability with Correlation IDsProduction distributed systems require comprehensive observability. Our implementation includes:Distributed tracing with correlation IDs propagated across service boundariesStructured logging with consistent metadata for aggregationCustom metrics measuring business logic, not just infrastructureImplementation Walkthrough: Building Production-Grade ComponentsCore Architecture DecisionsOur system implements a three-service architecture optimized for independent scaling:Log Producer Service: Handles high-velocity ingestion with minimal processing overhead. Critical design decision: validation occurs asynchronously to maintain ingestion throughput. Invalid logs are routed to dead letter queues rather than blocking the main pipeline.Log Consumer Service: Processes Kafka events and maintains read-optimized data structures. Key optimization: batch processing with configurable flush intervals balances latency with throughput. During high load, larger batches improve database write efficiency.API Gateway Service: Provides query interface with intelligent routing and caching. Implements query complexity analysis to prevent expensive operations from overwhelming the system.Database Schema OptimizationTraditional log tables become unusable beyond 10M records without proper indexing strategy:-- Composite index optimized for time-range queries CREATE INDEX idx_logs_timestamp_level ON logs(timestamp DESC, log_level); -- Partial index for error-only queries (common pattern) CREATE INDEX idx_logs_errors ON logs(timestamp DESC) WHERE log_level = ‘ERROR’; Caching Strategy ImplementationRedis serves three distinct caching layers:Query Result Caching: Exact query matches with 5-minute TTLAggregation Caching: Pre-computed hourly/daily summaries with 1-hour TTLHot Data Caching: Recent logs (last 15 minutes) with 30-second TTLCache key design prevents collisions while enabling pattern-based invalidation during data updates.Production Considerations: Performance and ReliabilityPerformance BenchmarksUnder normal load (1,000 logs/second), the system maintains:Query response times under 100ms (95th percentile)Cache hit ratios above 94%End-to-end processing latency under 200msDuring spike conditions (10,000 logs/second), performance gracefully degrades:Query response times increase to 300ms (acceptable for operational dashboards)Circuit breakers prevent cascade failuresKafka buffering maintains zero data lossFailure Scenarios and RecoveryDatabase Connection Loss: Circuit breakers activate after 3 consecutive failures, routing all queries to cached data. System continues serving recent logs while database recovers.Kafka Cluster Outage: Producer buffering and retry logic maintain data integrity. Consumer lag is measured and alerts trigger when processing falls behind ingestion.Redis Cluster Failure: System automatically falls back to direct database queries with degraded performance. Cache warming begins immediately upon Redis recovery.Scale Connection: Enterprise Production PatternsThis architecture mirrors production systems at scale. Airbnb’s log processing system uses identical patterns to handle 2M+ events per minute across their booking, pricing, and fraud detection systems. The key scaling strategies you’ve implemented today—CQRS separation, distributed caching, and asynchronous processing—remain constant as you move from thousands to millions of events per second.The primary difference at massive scale is data partitioning strategy and cross-region replication, concepts we’ll explore in Week 3 when building geographically distributed systems.Working Code Demo:Next Steps: Tomorrow’s Advanced ConceptsDay 7 focuses on service mesh integration and advanced routing patterns. You’ll implement intelligent load balancing, A/B testing infrastructure, and cross-service authentication—the missing pieces for production deployment at enterprise scale.Building Your System: Step-by-Step ImplementationGithub Link:https://github.com/sysdr/sdc-java/tree/main/day6/distributed-log-processorPrerequisitesBefore starting, make sure you have these tools installed on your computer:Java 17 or newer - The programming language we’re usingMaven 3.6 or newer - Builds our Java projectsDocker and Docker Compose - Runs our infrastructure servicesA code editor - VS Code, IntelliJ IDEA, or EclipseYou can verify your installations by running these commands in your terminal:java -version mvn -version docker --version docker-compose --version Step 1: Generate the Project FilesWe’ve created a script that automatically generates all the code and configuration files you need. This saves you from typing thousands of lines of code manually.Run the generator script:chmod +x generate_system_files.sh ./generate_system_files.sh This creates a folder called distributed-log-processor containing:Three Spring Boot services (Producer, Consumer, Gateway)Database configurationsDocker setup filesTesting scriptsComplete documentationNavigate into the project:cd distributed-log-processor Step 2: Start the Infrastructure ServicesOur application needs several supporting services to run. Think of these as the foundation of your house—you need them in place before you can build the walls.Start all infrastructure services:./setup.sh This script starts:Kafka - Message streaming platformPostgreSQL - Main database for storing logsRedis - Fast cache for query resultsPrometheus - Collects performance metricsGrafana - Creates visual dashboardsThe script takes about 30-60 seconds to start everything. You’ll see messages as each service comes online.Verify everything is running:docker-compose ps You should see all services showing “Up” status.Step 3: Build the Application ServicesNow we’ll compile our three Spring Boot services into executable JAR files.Build all services at once:mvn clean install This command:Downloads all necessary librariesCompiles your Java codeRuns automated testsPackages everything into JAR filesThe build takes 2-3 minutes on first run. Subsequent builds are faster because Maven caches the libraries.You’ll see output showing each service being built. Look for “BUILD SUCCESS” messages for all three modules.Step 4: Start the Application ServicesNow comes the exciting part—starting your distributed system! Open three separate terminal windows to run each service. This lets you see the logs from each one independently.Terminal 1 - Start the Log Producer:java -jar log-producer/target/log-producer-1.0.0.jar Wait until you see: “Started LogProducerApplication”Terminal 2 - Start the Log Consumer:java -jar log-consumer/target/log-consumer-1.0.0.jar Wait until you see: “Started LogConsumerApplication”Terminal 3 - Start the API Gateway:java -jar api-gateway/target/api-gateway-1.0.0.jar Wait until you see: “Started ApiGatewayApplication”Step 5: Verify System HealthLet’s make sure all services can communicate with each other. We’ll check the health endpoints that each service provides.Check each service:# Producer health check curl http://localhost:8081/actuator/health # Consumer health check curl http://localhost:8082/actuator/health # Gateway health check curl http://localhost:8080/actuator/health Each should return: {”status”:”UP”}If any service returns an error, check its terminal window for error messages.Step 6: Your First Log EventTime to send your first log through the system! We’ll use curl to send an HTTP request to the producer service.Send a test log:curl -X POST http://localhost:8081/api/v1/logs \\ -H “Content-Type: application/json” \\ -d ‘{ “message”: “My first distributed log event!”, “level”: “INFO”, “source”: “learning-system”, “metadata”: { “student”: “your-name”, “lesson”: “day-6” } }’ You should see: “Log event accepted for processing”Watch the terminal windows—you’ll see the log flow through the system:Producer receives it and sends to KafkaConsumer reads from Kafka and saves to databaseGateway makes it available for queriesStep 7: Query Your LogsNow let’s retrieve the log you just sent. The gateway service provides a powerful query API.Get recent logs:curl “http://localhost:8080/api/v1/logs?size=10” You’ll see a JSON response containing your log entry with all its details.Try more advanced queries:# Get only ERROR level logs curl “http://localhost:8080/api/v1/logs?logLevel=ERROR” # Search for specific keywords curl “http://localhost:8080/api/v1/logs?keyword=distributed” # Get logs from a specific source curl “http://localhost:8080/api/v1/logs?source=learning-system” IMAGE PLACEHOLDER: Query Response Example Formatted JSON output showing log query resultsStep 8: Send Multiple Logs (Batch Processing)Our system can handle many logs at once. Let’s test the batch endpoint.Send a batch of logs:curl -X POST http://localhost:8081/api/v1/logs/batch \\ -H “Content-Type: application/json” \\ -d ‘[ { “message”: “User logged in successfully”, “level”: “INFO”, “source”: “auth-service” }, { “message”: “Failed to connect to database”, “level”: “ERROR”, “source”: “db-service” }, { “message”: “Payment processed”, “level”: “INFO”, “source”: “payment-service” } ]’ Query to see all your logs:curl “http://localhost:8080/api/v1/logs?size=20” Step 9: View System StatisticsThe gateway provides an analytics endpoint that shows you interesting statistics about your logs.Get statistics:curl “http://localhost:8080/api/v1/logs/stats” This returns:Count of logs by level (INFO, ERROR, etc.)Top sources generating logsTotal log count in time rangeStep 10: Run the Load TestLet’s stress test your system to see how it handles high volume. The load test script sends 1,000 logs and makes 50 queries simultaneously.Run the load test:./load-test.sh Watch the terminal output as it:Sends 10 batches of 100 logs eachPerforms 50 different queriesReports success/failure for each operationThe test takes about 30-40 seconds to complete.Step 11: Monitor Performance with PrometheusPrometheus collects metrics from all your services. Let’s see what it’s tracking.Open Prometheus in your browser:http://localhost:9090 Try these queries in the Prometheus UI:Query 1 - Log ingestion rate:rate(logs_received_total[5m]) Query 2 - Cache hit ratio:rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) * 100 Query 3 - Average query response time:rate(log_query_duration_seconds_sum[5m]) / rate(log_query_duration_seconds_count[5m]) Click “Execute” and “Graph” to see visual representations.Step 12: Create Dashboards in GrafanaGrafana turns your metrics into beautiful, real-time dashboards.Open Grafana in your browser:http://localhost:3000 Login credentials:Username: adminPassword: admin(You’ll be prompted to change the password—you can skip this for now)The system comes with a pre-built dashboard called “Log Processor System Dashboard” that shows:Log ingestion rate over timeCache hit percentageQuery performanceError ratesStep 13: Run Integration TestsIntegration tests verify that all components work together correctly. These tests simulate real-world usage patterns.Run the integration test suite:./integration-tests/system-integration-test.sh The tests check:All services are respondingLogs can be sent and retrievedQueries return correct resultsStatistics are calculated properlyYou’ll see “All integration tests passed!” if everything works.Understanding What You BuiltLet’s review what each component does and how they work together:The Log Producer Service (Port 8081)This is the entry point for logs. Applications send their logs here via HTTP requests. The producer:Validates incoming logsImmediately returns a response (doesn’t wait for processing)Sends logs to Kafka for asynchronous processingTracks how many logs it receivesApache Kafka (Port 9092)Think of Kafka as a super-fast, reliable message highway. It:Stores logs temporarily in topics (named channels)Guarantees logs won’t be lost even if services crashAllows multiple consumers to read the same logsHandles millions of messages per secondThe Log Consumer Service (Port 8082)This service reads from Kafka and saves logs to the database. It:Processes logs in batches for efficiencyAcknowledges messages only after successful storageHandles failures gracefully with retriesKeeps track of processing progressPostgreSQL Database (Port 5432)The permanent storage for all logs. It features:Optimized indexes for fast time-based queriesSupport for complex filtering and aggregationACID guarantees for data consistencyAbility to store billions of log recordsRedis Cache (Port 6379)An extremely fast in-memory cache that:Stores frequently accessed query resultsReduces database load by 90%+Expires old data automaticallyReturns results in millisecondsThe API Gateway Service (Port 8080)Your query interface. It:Checks Redis cache first (cache-aside pattern)Falls back to database on cache missImplements circuit breakers for fault toleranceProvides REST APIs for searching logsPrometheus (Port 9090)Collects metrics from all services:Scrapes metrics every 15 secondsStores time-series dataEnables alerting on thresholdsPowers Grafana dashboardsGrafana (Port 3000)Visualizes your metrics:Creates real-time dashboardsShows trends and patternsHelps identify performance issuesMakes data accessible to everyoneKey Concepts You’ve LearnedEventual ConsistencyWhen you send a log to the producer, it returns immediately. But the log isn’t queryable instantly—it takes 100-500ms to flow through Kafka, get processed by the consumer, and land in the database. This delay is called eventual consistency.Why it matters: This trade-off lets your system handle way more logs. If every log had to be saved before responding, your system would be much slower.Circuit Breaker PatternIf the database goes down, the circuit breaker “opens” after detecting failures. Instead of trying the database repeatedly, the gateway serves cached data only. Once the database recovers, the circuit breaker gradually “closes” again.Why it matters: Prevents cascade failures where one slow component brings down the entire system.Cache-Aside PatternThe gateway doesn’t automatically populate the cache. Instead:Check if result exists in cacheIf yes, return it (cache hit)If no, query database (cache miss)Store result in cache for next timeWhy it matters: Simple to implement and works well for read-heavy workloads like log queries.Horizontal ScalingEach service is stateless—it doesn’t store any data in memory that matters. This means you can run multiple copies:3 producer instances can share incoming requests5 consumer instances can divide Kafka partitions4 gateway instances can handle more queriesWhy it matters: As your traffic grows, just add more servers. No fundamental redesign needed.Troubleshooting Common IssuesProblem: Services won’t startCheck 1: Are the ports already in use?lsof -i :8080 # Check gateway port lsof -i :8081 # Check producer port lsof -i :8082 # Check consumer port Solution: Stop any other programs using these ports.Check 2: Is Docker running?docker ps Solution: Start Docker Desktop or the Docker daemon.Problem: Can’t connect to KafkaCheck: Is Kafka healthy?docker exec kafka kafka-topics --list --bootstrap-server localhost:9092 Solution: Restart Kafka:docker-compose restart kafka Problem: Queries are very slowCheck: What’s the cache hit rate?curl http://localhost:8080/actuator/metrics/cache.gets Solution: If hit rate is low, your cache TTL might be too short. Check Redis:docker exec redis redis-cli info stats Problem: Consumer isn’t processing logsCheck: Consumer lag in Kafka:docker exec kafka kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-consumer-group Solution: If lag is growing, the consumer might be overwhelmed. Check the consumer service logs for errors.Experimenting FurtherNow that your system is running, try these experiments to learn more:Experiment 1: Break the DatabaseStop PostgreSQL and see what happens:docker-compose stop postgres Send logs and try queries. Notice:Producer still accepts logs (Kafka buffers them)Gateway serves cached queriesCircuit breaker prevents errorsRestart the database:docker-compose start postgres Watch the consumer catch up on processing the buffered logs.Experiment 2: Cache PerformanceSend the same query twice and compare response times:time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR” time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR” The second query should be much faster (cached result).Experiment 3: Batch vs. IndividualCompare sending 100 logs individually vs. one batch:Individual (slow):for i in {1..100}; do curl -X POST http://localhost:8081/api/v1/logs \\ -H “Content-Type: application/json” \\ -d ‘{”message”:”Log ‘$i’”,”level”:”INFO”,”source”:”test”}’ done Batch (fast): Use the batch endpoint with 100 logs in one request.Notice how much faster batching is!What’s NextYou’ve built a distributed system that handles thousands of logs per second. Here’s what you can explore next:Add More Features:User authentication for the APIRate limiting to prevent abuseData retention policies (auto-delete old logs)Full-text search with ElasticsearchScale It Further:Deploy to KubernetesAdd multiple Kafka brokersSet up database replicationImplement cross-region deploymentProduction Hardening:Add encryption for data in transitImplement audit loggingSet up automated backupsCreate runbooks for incidentsTomorrow in Day 7, we’ll add a service mesh to handle advanced routing, authentication between services, and implement A/B testing capabilities.Cleaning UpWhen you’re done experimenting, shut everything down:# Stop application services # Press Ctrl+C in each terminal window # Stop infrastructure docker-compose down # Remove data volumes (optional - deletes all logs) docker-compose down -v Your code and configuration files remain intact, so you can restart anytime by running the setup script again.","published","strategy","content-ready","locked","deletedAt","isSyncedFromOwnBlog","original","duplicatesCount","duplicateVersion","originalArticleId","errorsCount","appearance","color","backgroundColor","shortSummary","summary","bullets","persons","places","words","sentiment","subject","meta","eventId","f4354ede6471b83ec95c6c3fe59aa8a18c6ac3b50113822385cc8097988a83af","nostrRootEvent","event",{"_14":182,"_186":187,"_188":189,"_62":190,"_202":21,"_152":203,"_204":205},"sig","1922514de2ecb2f78a4f4180d2fe8fa1955a7bedad1069fae28bde575f626bf7701bbfc23d7b36cdf4a6d1dcd103057994f0c035426c1c1686d0544436c9f03a","kind",1,[191,192,193,197,200],[179,138],[29,144],[194,195,24,196],"a","30023:d12614be726049c7ecfeb7307a63d38418bb82b91866b8764a903ff28ed99bc6:c2d99fcc-d5e0-4576-8c78-b87b71493fb5","mention",[198,199],"client","Layer3.press",[201,21,24],"p","pubkey","Day 6: Building a Distributed Log Query Engine with Real-Time Processing\nWhat We’re Building Today\n\n\nnostr:naddr1qvzqqqr4gupzp5fxzjl8yczfclk0ades0f3a8pqchwptjxrxhpmy4ypl728dnx7xqqjxxvny8yukvcmr94jr2efs956r2dek95uxxdec943rsdmzxucngwfnve3r2kwvjaf ","created_at",1761220157,"authorNPub","markdown","What We’re Building Today\n-------------------------\n\n> Today we’re transforming our basic log storage system into a production-grade distributed query engine that handles real-time log processing at scale. You’ll implement:\n\n* **Distributed Log Query API**: RESTful service with advanced filtering, aggregation, and real-time search capabilities\n \n* **Event-Driven Processing Pipeline**: Kafka-based system processing 10K+ logs/second with guaranteed delivery\n \n* **Intelligent Caching Layer**: Redis-powered query optimization reducing response times from 2s to 50ms\n \n* **Production Monitoring Stack**: Complete observability with Prometheus metrics, Grafana dashboards, and distributed tracing\n \n\n* * *\n\nWhy This Matters: From Local Files to Planet-Scale Systems\n----------------------------------------------------------\n\n> The jump from flat file storage to distributed query processing represents one of the most critical scaling challenges in modern systems. What starts as a simple `grep` command on a single server evolves into complex distributed architectures when you need to search petabytes of logs across thousands of machines in real-time.\n> \n> Consider Netflix’s challenge: processing 500+ billion events daily across global infrastructure. Their logs aren’t just stored—they’re actively queried by fraud detection systems, recommendation engines, and operational dashboards. A single query might need to aggregate data across 50 different microservices, each running in multiple regions. The system you’re building today demonstrates the foundational patterns that make this scale possible.\n> \n> The core architectural challenge isn’t just storage or retrieval—it’s maintaining query performance and system availability while data volume grows exponentially. Today’s implementation introduces you to the distributed systems trade-offs that separate senior engineers from those still thinking in single-machine terms.\n\n* * *\n\nSystem Design Deep Dive: Five Core Patterns for Distributed Query Processing\n----------------------------------------------------------------------------\n\n### 1\\. Event Sourcing with Command Query Responsibility Segregation (CQRS)\n\nTraditional systems couple write and read operations, creating bottlenecks when query patterns differ from write patterns. Our log processing system implements CQRS by separating:\n\n* **Write Side**: High-throughput log ingestion via Kafka with minimal validation\n \n* **Read Side**: Optimized query structures with pre-computed aggregations and indexes\n \n\nThis separation allows us to scale reads and writes independently. When Netflix needs to process millions of viewing events while simultaneously running complex analytics queries, CQRS prevents read-heavy analytics from slowing down real-time event ingestion.\n\n**Trade-off Analysis**: CQRS introduces eventual consistency between write and read models. Your query results might be 100-500ms behind real-time events, but you gain the ability to handle 10x more concurrent queries. For log processing, this trade-off is almost always worth it—operational dashboards can tolerate slight delays if it means avoiding system overload during traffic spikes.\n\n### 2\\. Distributed Caching with Cache-Aside Pattern\n\nRaw log queries against PostgreSQL become prohibitively expensive beyond 1M records. Our Redis implementation uses the cache-aside pattern where:\n\n* Application code manages cache population and invalidation\n \n* Cache misses trigger database queries with automatic cache warming\n \n* TTL policies balance data freshness with performance\n \n\n**Key Insight**: Cache hit ratios above 95% are achievable for log queries because of temporal locality—recent logs are queried far more frequently than historical data. However, cache invalidation becomes complex when implementing real-time updates.\n\n**Anti-Pattern Warning**: Never implement write-through caching for high-velocity log data. The cache invalidation overhead will negate performance benefits and create consistency nightmares during failure scenarios.\n\n### 3\\. Circuit Breaker Pattern for Fault Isolation\n\nDistributed systems fail in cascade patterns—one slow component triggers timeouts across the entire system. Our Resilience4j implementation provides:\n\n* **Automatic failure detection** with configurable thresholds\n \n* **Fast-fail responses** preventing resource exhaustion\n \n* **Gradual recovery** with half-open state testing\n \n\nWhen Uber’s map matching service experiences high latency, circuit breakers prevent the delay from propagating to rider assignment and pricing services. The same pattern protects our log query system from database overload scenarios.\n\n### 4\\. Asynchronous Processing with Guaranteed Delivery\n\nKafka’s producer-consumer model ensures log processing continues even when downstream services are temporarily unavailable. Our implementation includes:\n\n* **At-least-once delivery** with producer retries and consumer acknowledgments\n \n* **Dead letter queues** for handling poison messages\n \n* **Consumer group management** for horizontal scaling\n \n\n**Scalability Implication**: This pattern allows you to decouple ingestion rate from processing rate. During traffic spikes, Kafka acts as a buffer, preventing log loss while consumers catch up.\n\n### 5\\. Distributed Observability with Correlation IDs\n\nProduction distributed systems require comprehensive observability. Our implementation includes:\n\n* **Distributed tracing** with correlation IDs propagated across service boundaries\n \n* **Structured logging** with consistent metadata for aggregation\n \n* **Custom metrics** measuring business logic, not just infrastructure\n \n\n* * *\n\nImplementation Walkthrough: Building Production-Grade Components\n----------------------------------------------------------------\n\n### Core Architecture Decisions\n\nOur system implements a three-service architecture optimized for independent scaling:\n\n**Log Producer Service**: Handles high-velocity ingestion with minimal processing overhead. Critical design decision: validation occurs asynchronously to maintain ingestion throughput. Invalid logs are routed to dead letter queues rather than blocking the main pipeline.\n\n**Log Consumer Service**: Processes Kafka events and maintains read-optimized data structures. Key optimization: batch processing with configurable flush intervals balances latency with throughput. During high load, larger batches improve database write efficiency.\n\n**API Gateway Service**: Provides query interface with intelligent routing and caching. Implements query complexity analysis to prevent expensive operations from overwhelming the system.\n\n[\n\n![](https://substackcdn.com/image/fetch/$s_!ymNu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aeb09c3-dd9a-4a5a-a40d-751cc542ceaa_935x615.png)\n\n\n\n](https://substackcdn.com/image/fetch/$s_!ymNu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aeb09c3-dd9a-4a5a-a40d-751cc542ceaa_935x615.png)\n\n### Database Schema Optimization\n\nTraditional log tables become unusable beyond 10M records without proper indexing strategy:\n\n -- Composite index optimized for time-range queries\n CREATE INDEX idx_logs_timestamp_level ON logs(timestamp DESC, log_level);\n \n -- Partial index for error-only queries (common pattern)\n CREATE INDEX idx_logs_errors ON logs(timestamp DESC) WHERE log_level = ‘ERROR’;\n \n\n### Caching Strategy Implementation\n\nRedis serves three distinct caching layers:\n\n1. **Query Result Caching**: Exact query matches with 5-minute TTL\n \n2. **Aggregation Caching**: Pre-computed hourly/daily summaries with 1-hour TTL\n \n3. **Hot Data Caching**: Recent logs (last 15 minutes) with 30-second TTL\n \n\nCache key design prevents collisions while enabling pattern-based invalidation during data updates.\n\n* * *\n\nProduction Considerations: Performance and Reliability\n------------------------------------------------------\n\n### Performance Benchmarks\n\nUnder normal load (1,000 logs/second), the system maintains:\n\n* Query response times under 100ms (95th percentile)\n \n* Cache hit ratios above 94%\n \n* End-to-end processing latency under 200ms\n \n\nDuring spike conditions (10,000 logs/second), performance gracefully degrades:\n\n* Query response times increase to 300ms (acceptable for operational dashboards)\n \n* Circuit breakers prevent cascade failures\n \n* Kafka buffering maintains zero data loss\n \n\n### Failure Scenarios and Recovery\n\n**Database Connection Loss**: Circuit breakers activate after 3 consecutive failures, routing all queries to cached data. System continues serving recent logs while database recovers.\n\n**Kafka Cluster Outage**: Producer buffering and retry logic maintain data integrity. Consumer lag is measured and alerts trigger when processing falls behind ingestion.\n\n**Redis Cluster Failure**: System automatically falls back to direct database queries with degraded performance. Cache warming begins immediately upon Redis recovery.\n\n* * *\n\nScale Connection: Enterprise Production Patterns\n------------------------------------------------\n\nThis architecture mirrors production systems at scale. Airbnb’s log processing system uses identical patterns to handle 2M+ events per minute across their booking, pricing, and fraud detection systems. The key scaling strategies you’ve implemented today—CQRS separation, distributed caching, and asynchronous processing—remain constant as you move from thousands to millions of events per second.\n\nThe primary difference at massive scale is data partitioning strategy and cross-region replication, concepts we’ll explore in Week 3 when building geographically distributed systems.\n\nWorking Code Demo:\n------------------\n\n* * *\n\nNext Steps: Tomorrow’s Advanced Concepts\n----------------------------------------\n\nDay 7 focuses on service mesh integration and advanced routing patterns. You’ll implement intelligent load balancing, A/B testing infrastructure, and cross-service authentication—the missing pieces for production deployment at enterprise scale.\n\n* * *\n\n* * *\n\nBuilding Your System: Step-by-Step Implementation\n=================================================\n\n### Github Link:\n\n https://github.com/sysdr/sdc-java/tree/main/day6/distributed-log-processor\n\nPrerequisites\n-------------\n\nBefore starting, make sure you have these tools installed on your computer:\n\n* **Java 17 or newer** - The programming language we’re using\n \n* **Maven 3.6 or newer** - Builds our Java projects\n \n* **Docker and Docker Compose** - Runs our infrastructure services\n \n* **A code editor** - VS Code, IntelliJ IDEA, or Eclipse\n \n\nYou can verify your installations by running these commands in your terminal:\n\n java -version\n mvn -version\n docker --version\n docker-compose --version\n \n\n* * *\n\nStep 1: Generate the Project Files\n----------------------------------\n\nWe’ve created a script that automatically generates all the code and configuration files you need. This saves you from typing thousands of lines of code manually.\n\nRun the generator script:\n\n chmod +x generate_system_files.sh\n ./generate_system_files.sh\n \n\nThis creates a folder called `distributed-log-processor` containing:\n\n* Three Spring Boot services (Producer, Consumer, Gateway)\n \n* Database configurations\n \n* Docker setup files\n \n* Testing scripts\n \n* Complete documentation\n \n\nNavigate into the project:\n\n cd distributed-log-processor\n \n\n* * *\n\nStep 2: Start the Infrastructure Services\n-----------------------------------------\n\nOur application needs several supporting services to run. Think of these as the foundation of your house—you need them in place before you can build the walls.\n\nStart all infrastructure services:\n\n ./setup.sh\n \n\nThis script starts:\n\n* **Kafka** - Message streaming platform\n \n* **PostgreSQL** - Main database for storing logs\n \n* **Redis** - Fast cache for query results\n \n* **Prometheus** - Collects performance metrics\n \n* **Grafana** - Creates visual dashboards\n \n\nThe script takes about 30-60 seconds to start everything. You’ll see messages as each service comes online.\n\nVerify everything is running:\n\n docker-compose ps\n \n\nYou should see all services showing “Up” status.\n\n* * *\n\nStep 3: Build the Application Services\n--------------------------------------\n\nNow we’ll compile our three Spring Boot services into executable JAR files.\n\nBuild all services at once:\n\n mvn clean install\n \n\nThis command:\n\n* Downloads all necessary libraries\n \n* Compiles your Java code\n \n* Runs automated tests\n \n* Packages everything into JAR files\n \n\nThe build takes 2-3 minutes on first run. Subsequent builds are faster because Maven caches the libraries.\n\nYou’ll see output showing each service being built. Look for “BUILD SUCCESS” messages for all three modules.\n\n* * *\n\nStep 4: Start the Application Services\n--------------------------------------\n\nNow comes the exciting part—starting your distributed system! Open three separate terminal windows to run each service. This lets you see the logs from each one independently.\n\n**Terminal 1 - Start the Log Producer:**\n\n java -jar log-producer/target/log-producer-1.0.0.jar\n \n\nWait until you see: “Started LogProducerApplication”\n\n**Terminal 2 - Start the Log Consumer:**\n\n java -jar log-consumer/target/log-consumer-1.0.0.jar\n \n\nWait until you see: “Started LogConsumerApplication”\n\n**Terminal 3 - Start the API Gateway:**\n\n java -jar api-gateway/target/api-gateway-1.0.0.jar\n \n\nWait until you see: “Started ApiGatewayApplication”\n\n* * *\n\nStep 5: Verify System Health\n----------------------------\n\nLet’s make sure all services can communicate with each other. We’ll check the health endpoints that each service provides.\n\nCheck each service:\n\n # Producer health check\n curl http://localhost:8081/actuator/health\n \n # Consumer health check\n curl http://localhost:8082/actuator/health\n \n # Gateway health check\n curl http://localhost:8080/actuator/health\n \n\nEach should return: `{”status”:”UP”}`\n\nIf any service returns an error, check its terminal window for error messages.\n\n* * *\n\nStep 6: Your First Log Event\n----------------------------\n\nTime to send your first log through the system! We’ll use curl to send an HTTP request to the producer service.\n\nSend a test log:\n\n curl -X POST http://localhost:8081/api/v1/logs \\\n -H “Content-Type: application/json” \\\n -d ‘{\n “message”: “My first distributed log event!”,\n “level”: “INFO”,\n “source”: “learning-system”,\n “metadata”: {\n “student”: “your-name”,\n “lesson”: “day-6”\n }\n }’\n \n\nYou should see: “Log event accepted for processing”\n\nWatch the terminal windows—you’ll see the log flow through the system:\n\n1. Producer receives it and sends to Kafka\n \n2. Consumer reads from Kafka and saves to database\n \n3. Gateway makes it available for queries\n \n\n* * *\n\nStep 7: Query Your Logs\n-----------------------\n\nNow let’s retrieve the log you just sent. The gateway service provides a powerful query API.\n\nGet recent logs:\n\n curl “http://localhost:8080/api/v1/logs?size=10”\n \n\nYou’ll see a JSON response containing your log entry with all its details.\n\nTry more advanced queries:\n\n # Get only ERROR level logs\n curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n \n # Search for specific keywords\n curl “http://localhost:8080/api/v1/logs?keyword=distributed”\n \n # Get logs from a specific source\n curl “http://localhost:8080/api/v1/logs?source=learning-system”\n \n\n**IMAGE PLACEHOLDER: Query Response Example** _Formatted JSON output showing log query results_\n\n* * *\n\nStep 8: Send Multiple Logs (Batch Processing)\n---------------------------------------------\n\nOur system can handle many logs at once. Let’s test the batch endpoint.\n\nSend a batch of logs:\n\n curl -X POST http://localhost:8081/api/v1/logs/batch \\\n -H “Content-Type: application/json” \\\n -d ‘[\n {\n “message”: “User logged in successfully”,\n “level”: “INFO”,\n “source”: “auth-service”\n },\n {\n “message”: “Failed to connect to database”,\n “level”: “ERROR”,\n “source”: “db-service”\n },\n {\n “message”: “Payment processed”,\n “level”: “INFO”,\n “source”: “payment-service”\n }\n ]’\n \n\nQuery to see all your logs:\n\n curl “http://localhost:8080/api/v1/logs?size=20”\n \n\n* * *\n\nStep 9: View System Statistics\n------------------------------\n\nThe gateway provides an analytics endpoint that shows you interesting statistics about your logs.\n\nGet statistics:\n\n curl “http://localhost:8080/api/v1/logs/stats”\n \n\nThis returns:\n\n* Count of logs by level (INFO, ERROR, etc.)\n \n* Top sources generating logs\n \n* Total log count in time range\n \n\n* * *\n\nStep 10: Run the Load Test\n--------------------------\n\nLet’s stress test your system to see how it handles high volume. The load test script sends 1,000 logs and makes 50 queries simultaneously.\n\nRun the load test:\n\n ./load-test.sh\n \n\nWatch the terminal output as it:\n\n* Sends 10 batches of 100 logs each\n \n* Performs 50 different queries\n \n* Reports success/failure for each operation\n \n\nThe test takes about 30-40 seconds to complete.\n\n* * *\n\nStep 11: Monitor Performance with Prometheus\n--------------------------------------------\n\nPrometheus collects metrics from all your services. Let’s see what it’s tracking.\n\nOpen Prometheus in your browser:\n\n http://localhost:9090\n \n\nTry these queries in the Prometheus UI:\n\n**Query 1 - Log ingestion rate:**\n\n rate(logs_received_total[5m])\n \n\n**Query 2 - Cache hit ratio:**\n\n rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) * 100\n \n\n**Query 3 - Average query response time:**\n\n rate(log_query_duration_seconds_sum[5m]) / rate(log_query_duration_seconds_count[5m])\n \n\nClick “Execute” and “Graph” to see visual representations.\n\n* * *\n\nStep 12: Create Dashboards in Grafana\n-------------------------------------\n\nGrafana turns your metrics into beautiful, real-time dashboards.\n\nOpen Grafana in your browser:\n\n http://localhost:3000\n \n\nLogin credentials:\n\n* Username: admin\n \n* Password: admin\n \n\n(You’ll be prompted to change the password—you can skip this for now)\n\nThe system comes with a pre-built dashboard called “Log Processor System Dashboard” that shows:\n\n* Log ingestion rate over time\n \n* Cache hit percentage\n \n* Query performance\n \n* Error rates\n \n\n* * *\n\nStep 13: Run Integration Tests\n------------------------------\n\nIntegration tests verify that all components work together correctly. These tests simulate real-world usage patterns.\n\nRun the integration test suite:\n\n ./integration-tests/system-integration-test.sh\n \n\nThe tests check:\n\n* All services are responding\n \n* Logs can be sent and retrieved\n \n* Queries return correct results\n \n* Statistics are calculated properly\n \n\nYou’ll see “All integration tests passed!” if everything works.\n\n* * *\n\nUnderstanding What You Built\n----------------------------\n\nLet’s review what each component does and how they work together:\n\n### The Log Producer Service (Port 8081)\n\nThis is the entry point for logs. Applications send their logs here via HTTP requests. The producer:\n\n* Validates incoming logs\n \n* Immediately returns a response (doesn’t wait for processing)\n \n* Sends logs to Kafka for asynchronous processing\n \n* Tracks how many logs it receives\n \n\n### Apache Kafka (Port 9092)\n\nThink of Kafka as a super-fast, reliable message highway. It:\n\n* Stores logs temporarily in topics (named channels)\n \n* Guarantees logs won’t be lost even if services crash\n \n* Allows multiple consumers to read the same logs\n \n* Handles millions of messages per second\n \n\n### The Log Consumer Service (Port 8082)\n\nThis service reads from Kafka and saves logs to the database. It:\n\n* Processes logs in batches for efficiency\n \n* Acknowledges messages only after successful storage\n \n* Handles failures gracefully with retries\n \n* Keeps track of processing progress\n \n\n### PostgreSQL Database (Port 5432)\n\nThe permanent storage for all logs. It features:\n\n* Optimized indexes for fast time-based queries\n \n* Support for complex filtering and aggregation\n \n* ACID guarantees for data consistency\n \n* Ability to store billions of log records\n \n\n### Redis Cache (Port 6379)\n\nAn extremely fast in-memory cache that:\n\n* Stores frequently accessed query results\n \n* Reduces database load by 90%+\n \n* Expires old data automatically\n \n* Returns results in milliseconds\n \n\n### The API Gateway Service (Port 8080)\n\nYour query interface. It:\n\n* Checks Redis cache first (cache-aside pattern)\n \n* Falls back to database on cache miss\n \n* Implements circuit breakers for fault tolerance\n \n* Provides REST APIs for searching logs\n \n\n### Prometheus (Port 9090)\n\nCollects metrics from all services:\n\n* Scrapes metrics every 15 seconds\n \n* Stores time-series data\n \n* Enables alerting on thresholds\n \n* Powers Grafana dashboards\n \n\n### Grafana (Port 3000)\n\nVisualizes your metrics:\n\n* Creates real-time dashboards\n \n* Shows trends and patterns\n \n* Helps identify performance issues\n \n* Makes data accessible to everyone\n \n\n* * *\n\nKey Concepts You’ve Learned\n---------------------------\n\n### Eventual Consistency\n\nWhen you send a log to the producer, it returns immediately. But the log isn’t queryable instantly—it takes 100-500ms to flow through Kafka, get processed by the consumer, and land in the database. This delay is called eventual consistency.\n\n**Why it matters**: This trade-off lets your system handle way more logs. If every log had to be saved before responding, your system would be much slower.\n\n### Circuit Breaker Pattern\n\nIf the database goes down, the circuit breaker “opens” after detecting failures. Instead of trying the database repeatedly, the gateway serves cached data only. Once the database recovers, the circuit breaker gradually “closes” again.\n\n**Why it matters**: Prevents cascade failures where one slow component brings down the entire system.\n\n### Cache-Aside Pattern\n\nThe gateway doesn’t automatically populate the cache. Instead:\n\n1. Check if result exists in cache\n \n2. If yes, return it (cache hit)\n \n3. If no, query database (cache miss)\n \n4. Store result in cache for next time\n \n\n**Why it matters**: Simple to implement and works well for read-heavy workloads like log queries.\n\n### Horizontal Scaling\n\nEach service is stateless—it doesn’t store any data in memory that matters. This means you can run multiple copies:\n\n* 3 producer instances can share incoming requests\n \n* 5 consumer instances can divide Kafka partitions\n \n* 4 gateway instances can handle more queries\n \n\n**Why it matters**: As your traffic grows, just add more servers. No fundamental redesign needed.\n\n* * *\n\nTroubleshooting Common Issues\n-----------------------------\n\n### Problem: Services won’t start\n\n**Check 1**: Are the ports already in use?\n\n lsof -i :8080 # Check gateway port\n lsof -i :8081 # Check producer port\n lsof -i :8082 # Check consumer port\n \n\n**Solution**: Stop any other programs using these ports.\n\n**Check 2**: Is Docker running?\n\n docker ps\n \n\n**Solution**: Start Docker Desktop or the Docker daemon.\n\n### Problem: Can’t connect to Kafka\n\n**Check**: Is Kafka healthy?\n\n docker exec kafka kafka-topics --list --bootstrap-server localhost:9092\n \n\n**Solution**: Restart Kafka:\n\n docker-compose restart kafka\n \n\n### Problem: Queries are very slow\n\n**Check**: What’s the cache hit rate?\n\n curl http://localhost:8080/actuator/metrics/cache.gets\n \n\n**Solution**: If hit rate is low, your cache TTL might be too short. Check Redis:\n\n docker exec redis redis-cli info stats\n \n\n### Problem: Consumer isn’t processing logs\n\n**Check**: Consumer lag in Kafka:\n\n docker exec kafka kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-consumer-group\n \n\n**Solution**: If lag is growing, the consumer might be overwhelmed. Check the consumer service logs for errors.\n\n* * *\n\nExperimenting Further\n---------------------\n\nNow that your system is running, try these experiments to learn more:\n\n### Experiment 1: Break the Database\n\nStop PostgreSQL and see what happens:\n\n docker-compose stop postgres\n \n\nSend logs and try queries. Notice:\n\n* Producer still accepts logs (Kafka buffers them)\n \n* Gateway serves cached queries\n \n* Circuit breaker prevents errors\n \n\nRestart the database:\n\n docker-compose start postgres\n \n\nWatch the consumer catch up on processing the buffered logs.\n\n### Experiment 2: Cache Performance\n\nSend the same query twice and compare response times:\n\n time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n \n\nThe second query should be much faster (cached result).\n\n### Experiment 3: Batch vs. Individual\n\nCompare sending 100 logs individually vs. one batch:\n\n**Individual (slow):**\n\n for i in {1..100}; do\n curl -X POST http://localhost:8081/api/v1/logs \\\n -H “Content-Type: application/json” \\\n -d ‘{”message”:”Log ‘$i’”,”level”:”INFO”,”source”:”test”}’\n done\n \n\n**Batch (fast):** Use the batch endpoint with 100 logs in one request.\n\nNotice how much faster batching is!\n\n* * *\n\nWhat’s Next\n-----------\n\nYou’ve built a distributed system that handles thousands of logs per second. Here’s what you can explore next:\n\n**Add More Features:**\n\n* User authentication for the API\n \n* Rate limiting to prevent abuse\n \n* Data retention policies (auto-delete old logs)\n \n* Full-text search with Elasticsearch\n \n\n**Scale It Further:**\n\n* Deploy to Kubernetes\n \n* Add multiple Kafka brokers\n \n* Set up database replication\n \n* Implement cross-region deployment\n \n\n**Production Hardening:**\n\n* Add encryption for data in transit\n \n* Implement audit logging\n \n* Set up automated backups\n \n* Create runbooks for incidents\n \n\nTomorrow in Day 7, we’ll add a service mesh to handle advanced routing, authentication between services, and implement A/B testing capabilities.\n\n* * *\n\nCleaning Up\n-----------\n\nWhen you’re done experimenting, shut everything down:\n\n # Stop application services\n # Press Ctrl+C in each terminal window\n \n # Stop infrastructure\n docker-compose down\n \n # Remove data volumes (optional - deletes all logs)\n docker-compose down -v\n \n\nYour code and configuration files remain intact, so you can restart anytime by running the setup script again.","freeMarkdown","kind1EventId","kind30023EventId","7f850f0687aad4e9a0d52b9c7750ed7ea40788cad580893fb4e1a7a4cf89eaac","kind30023",{"_14":212,"_186":215,"_188":216,"_62":217,"_202":21,"_152":231,"_204":232},"3137cfffe423a995fbf95da58cb12ce61b49b15619cb2618fec68e0eaa888be979e2ddb76e3f3e027ef4c0a6221d195250ac7d7de690920041872d5495b303fa",30023,[218,221,222,225,226,228,229,230],[219,220],"r","https://drewdru.syndichain.com/articles/c2d99fcc-d5e0-4576-8c78-b87b71493fb5",[26,138],[223,224],"published_at","1760605519",[29,144],[227,133],"d",[173,139],[198,199],[201,21,24],"Day 6: Building a Distributed Log Query Engine with Real-Time Processing\nWhat We’re Building Today\n-------------------------\n\n> Today we’re transforming our basic log storage system into a production-grade distributed query engine that handles real-time log processing at scale. You’ll implement:\n\n* **Distributed Log Query API**: RESTful service with advanced filtering, aggregation, and real-time search capabilities\n \n* **Event-Driven Processing Pipeline**: Kafka-based system processing 10K+ logs/second with guaranteed delivery\n \n* **Intelligent Caching Layer**: Redis-powered query optimization reducing response times from 2s to 50ms\n \n* **Production Monitoring Stack**: Complete observability with Prometheus metrics, Grafana dashboards, and distributed tracing\n \n\n* * *\n\nWhy This Matters: From Local Files to Planet-Scale Systems\n----------------------------------------------------------\n\n> The jump from flat file storage to distributed query processing represents one of the most critical scaling challenges in modern systems. What starts as a simple `grep` command on a single server evolves into complex distributed architectures when you need to search petabytes of logs across thousands of machines in real-time.\n> \n> Consider Netflix’s challenge: processing 500+ billion events daily across global infrastructure. Their logs aren’t just stored—they’re actively queried by fraud detection systems, recommendation engines, and operational dashboards. A single query might need to aggregate data across 50 different microservices, each running in multiple regions. The system you’re building today demonstrates the foundational patterns that make this scale possible.\n> \n> The core architectural challenge isn’t just storage or retrieval—it’s maintaining query performance and system availability while data volume grows exponentially. Today’s implementation introduces you to the distributed systems trade-offs that separate senior engineers from those still thinking in single-machine terms.\n\n* * *\n\nSystem Design Deep Dive: Five Core Patterns for Distributed Query Processing\n----------------------------------------------------------------------------\n\n### 1\\. Event Sourcing with Command Query Responsibility Segregation (CQRS)\n\nTraditional systems couple write and read operations, creating bottlenecks when query patterns differ from write patterns. Our log processing system implements CQRS by separating:\n\n* **Write Side**: High-throughput log ingestion via Kafka with minimal validation\n \n* **Read Side**: Optimized query structures with pre-computed aggregations and indexes\n \n\nThis separation allows us to scale reads and writes independently. When Netflix needs to process millions of viewing events while simultaneously running complex analytics queries, CQRS prevents read-heavy analytics from slowing down real-time event ingestion.\n\n**Trade-off Analysis**: CQRS introduces eventual consistency between write and read models. Your query results might be 100-500ms behind real-time events, but you gain the ability to handle 10x more concurrent queries. For log processing, this trade-off is almost always worth it—operational dashboards can tolerate slight delays if it means avoiding system overload during traffic spikes.\n\n### 2\\. Distributed Caching with Cache-Aside Pattern\n\nRaw log queries against PostgreSQL become prohibitively expensive beyond 1M records. Our Redis implementation uses the cache-aside pattern where:\n\n* Application code manages cache population and invalidation\n \n* Cache misses trigger database queries with automatic cache warming\n \n* TTL policies balance data freshness with performance\n \n\n**Key Insight**: Cache hit ratios above 95% are achievable for log queries because of temporal locality—recent logs are queried far more frequently than historical data. However, cache invalidation becomes complex when implementing real-time updates.\n\n**Anti-Pattern Warning**: Never implement write-through caching for high-velocity log data. The cache invalidation overhead will negate performance benefits and create consistency nightmares during failure scenarios.\n\n### 3\\. Circuit Breaker Pattern for Fault Isolation\n\nDistributed systems fail in cascade patterns—one slow component triggers timeouts across the entire system. Our Resilience4j implementation provides:\n\n* **Automatic failure detection** with configurable thresholds\n \n* **Fast-fail responses** preventing resource exhaustion\n \n* **Gradual recovery** with half-open state testing\n \n\nWhen Uber’s map matching service experiences high latency, circuit breakers prevent the delay from propagating to rider assignment and pricing services. The same pattern protects our log query system from database overload scenarios.\n\n### 4\\. Asynchronous Processing with Guaranteed Delivery\n\nKafka’s producer-consumer model ensures log processing continues even when downstream services are temporarily unavailable. Our implementation includes:\n\n* **At-least-once delivery** with producer retries and consumer acknowledgments\n \n* **Dead letter queues** for handling poison messages\n \n* **Consumer group management** for horizontal scaling\n \n\n**Scalability Implication**: This pattern allows you to decouple ingestion rate from processing rate. During traffic spikes, Kafka acts as a buffer, preventing log loss while consumers catch up.\n\n### 5\\. Distributed Observability with Correlation IDs\n\nProduction distributed systems require comprehensive observability. Our implementation includes:\n\n* **Distributed tracing** with correlation IDs propagated across service boundaries\n \n* **Structured logging** with consistent metadata for aggregation\n \n* **Custom metrics** measuring business logic, not just infrastructure\n \n\n* * *\n\nImplementation Walkthrough: Building Production-Grade Components\n----------------------------------------------------------------\n\n### Core Architecture Decisions\n\nOur system implements a three-service architecture optimized for independent scaling:\n\n**Log Producer Service**: Handles high-velocity ingestion with minimal processing overhead. Critical design decision: validation occurs asynchronously to maintain ingestion throughput. Invalid logs are routed to dead letter queues rather than blocking the main pipeline.\n\n**Log Consumer Service**: Processes Kafka events and maintains read-optimized data structures. Key optimization: batch processing with configurable flush intervals balances latency with throughput. During high load, larger batches improve database write efficiency.\n\n**API Gateway Service**: Provides query interface with intelligent routing and caching. Implements query complexity analysis to prevent expensive operations from overwhelming the system.\n\n[\n\n![](https://substackcdn.com/image/fetch/$s_!ymNu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aeb09c3-dd9a-4a5a-a40d-751cc542ceaa_935x615.png)\n\n\n\n](https://substackcdn.com/image/fetch/$s_!ymNu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aeb09c3-dd9a-4a5a-a40d-751cc542ceaa_935x615.png)\n\n### Database Schema Optimization\n\nTraditional log tables become unusable beyond 10M records without proper indexing strategy:\n\n -- Composite index optimized for time-range queries\n CREATE INDEX idx_logs_timestamp_level ON logs(timestamp DESC, log_level);\n \n -- Partial index for error-only queries (common pattern)\n CREATE INDEX idx_logs_errors ON logs(timestamp DESC) WHERE log_level = ‘ERROR’;\n \n\n### Caching Strategy Implementation\n\nRedis serves three distinct caching layers:\n\n1. **Query Result Caching**: Exact query matches with 5-minute TTL\n \n2. **Aggregation Caching**: Pre-computed hourly/daily summaries with 1-hour TTL\n \n3. **Hot Data Caching**: Recent logs (last 15 minutes) with 30-second TTL\n \n\nCache key design prevents collisions while enabling pattern-based invalidation during data updates.\n\n* * *\n\nProduction Considerations: Performance and Reliability\n------------------------------------------------------\n\n### Performance Benchmarks\n\nUnder normal load (1,000 logs/second), the system maintains:\n\n* Query response times under 100ms (95th percentile)\n \n* Cache hit ratios above 94%\n \n* End-to-end processing latency under 200ms\n \n\nDuring spike conditions (10,000 logs/second), performance gracefully degrades:\n\n* Query response times increase to 300ms (acceptable for operational dashboards)\n \n* Circuit breakers prevent cascade failures\n \n* Kafka buffering maintains zero data loss\n \n\n### Failure Scenarios and Recovery\n\n**Database Connection Loss**: Circuit breakers activate after 3 consecutive failures, routing all queries to cached data. System continues serving recent logs while database recovers.\n\n**Kafka Cluster Outage**: Producer buffering and retry logic maintain data integrity. Consumer lag is measured and alerts trigger when processing falls behind ingestion.\n\n**Redis Cluster Failure**: System automatically falls back to direct database queries with degraded performance. Cache warming begins immediately upon Redis recovery.\n\n* * *\n\nScale Connection: Enterprise Production Patterns\n------------------------------------------------\n\nThis architecture mirrors production systems at scale. Airbnb’s log processing system uses identical patterns to handle 2M+ events per minute across their booking, pricing, and fraud detection systems. The key scaling strategies you’ve implemented today—CQRS separation, distributed caching, and asynchronous processing—remain constant as you move from thousands to millions of events per second.\n\nThe primary difference at massive scale is data partitioning strategy and cross-region replication, concepts we’ll explore in Week 3 when building geographically distributed systems.\n\nWorking Code Demo:\n------------------\n\n* * *\n\nNext Steps: Tomorrow’s Advanced Concepts\n----------------------------------------\n\nDay 7 focuses on service mesh integration and advanced routing patterns. You’ll implement intelligent load balancing, A/B testing infrastructure, and cross-service authentication—the missing pieces for production deployment at enterprise scale.\n\n* * *\n\n* * *\n\nBuilding Your System: Step-by-Step Implementation\n=================================================\n\n### Github Link:\n\n https://github.com/sysdr/sdc-java/tree/main/day6/distributed-log-processor\n\nPrerequisites\n-------------\n\nBefore starting, make sure you have these tools installed on your computer:\n\n* **Java 17 or newer** - The programming language we’re using\n \n* **Maven 3.6 or newer** - Builds our Java projects\n \n* **Docker and Docker Compose** - Runs our infrastructure services\n \n* **A code editor** - VS Code, IntelliJ IDEA, or Eclipse\n \n\nYou can verify your installations by running these commands in your terminal:\n\n java -version\n mvn -version\n docker --version\n docker-compose --version\n \n\n* * *\n\nStep 1: Generate the Project Files\n----------------------------------\n\nWe’ve created a script that automatically generates all the code and configuration files you need. This saves you from typing thousands of lines of code manually.\n\nRun the generator script:\n\n chmod +x generate_system_files.sh\n ./generate_system_files.sh\n \n\nThis creates a folder called `distributed-log-processor` containing:\n\n* Three Spring Boot services (Producer, Consumer, Gateway)\n \n* Database configurations\n \n* Docker setup files\n \n* Testing scripts\n \n* Complete documentation\n \n\nNavigate into the project:\n\n cd distributed-log-processor\n \n\n* * *\n\nStep 2: Start the Infrastructure Services\n-----------------------------------------\n\nOur application needs several supporting services to run. Think of these as the foundation of your house—you need them in place before you can build the walls.\n\nStart all infrastructure services:\n\n ./setup.sh\n \n\nThis script starts:\n\n* **Kafka** - Message streaming platform\n \n* **PostgreSQL** - Main database for storing logs\n \n* **Redis** - Fast cache for query results\n \n* **Prometheus** - Collects performance metrics\n \n* **Grafana** - Creates visual dashboards\n \n\nThe script takes about 30-60 seconds to start everything. You’ll see messages as each service comes online.\n\nVerify everything is running:\n\n docker-compose ps\n \n\nYou should see all services showing “Up” status.\n\n* * *\n\nStep 3: Build the Application Services\n--------------------------------------\n\nNow we’ll compile our three Spring Boot services into executable JAR files.\n\nBuild all services at once:\n\n mvn clean install\n \n\nThis command:\n\n* Downloads all necessary libraries\n \n* Compiles your Java code\n \n* Runs automated tests\n \n* Packages everything into JAR files\n \n\nThe build takes 2-3 minutes on first run. Subsequent builds are faster because Maven caches the libraries.\n\nYou’ll see output showing each service being built. Look for “BUILD SUCCESS” messages for all three modules.\n\n* * *\n\nStep 4: Start the Application Services\n--------------------------------------\n\nNow comes the exciting part—starting your distributed system! Open three separate terminal windows to run each service. This lets you see the logs from each one independently.\n\n**Terminal 1 - Start the Log Producer:**\n\n java -jar log-producer/target/log-producer-1.0.0.jar\n \n\nWait until you see: “Started LogProducerApplication”\n\n**Terminal 2 - Start the Log Consumer:**\n\n java -jar log-consumer/target/log-consumer-1.0.0.jar\n \n\nWait until you see: “Started LogConsumerApplication”\n\n**Terminal 3 - Start the API Gateway:**\n\n java -jar api-gateway/target/api-gateway-1.0.0.jar\n \n\nWait until you see: “Started ApiGatewayApplication”\n\n* * *\n\nStep 5: Verify System Health\n----------------------------\n\nLet’s make sure all services can communicate with each other. We’ll check the health endpoints that each service provides.\n\nCheck each service:\n\n # Producer health check\n curl http://localhost:8081/actuator/health\n \n # Consumer health check\n curl http://localhost:8082/actuator/health\n \n # Gateway health check\n curl http://localhost:8080/actuator/health\n \n\nEach should return: `{”status”:”UP”}`\n\nIf any service returns an error, check its terminal window for error messages.\n\n* * *\n\nStep 6: Your First Log Event\n----------------------------\n\nTime to send your first log through the system! We’ll use curl to send an HTTP request to the producer service.\n\nSend a test log:\n\n curl -X POST http://localhost:8081/api/v1/logs \\\n -H “Content-Type: application/json” \\\n -d ‘{\n “message”: “My first distributed log event!”,\n “level”: “INFO”,\n “source”: “learning-system”,\n “metadata”: {\n “student”: “your-name”,\n “lesson”: “day-6”\n }\n }’\n \n\nYou should see: “Log event accepted for processing”\n\nWatch the terminal windows—you’ll see the log flow through the system:\n\n1. Producer receives it and sends to Kafka\n \n2. Consumer reads from Kafka and saves to database\n \n3. Gateway makes it available for queries\n \n\n* * *\n\nStep 7: Query Your Logs\n-----------------------\n\nNow let’s retrieve the log you just sent. The gateway service provides a powerful query API.\n\nGet recent logs:\n\n curl “http://localhost:8080/api/v1/logs?size=10”\n \n\nYou’ll see a JSON response containing your log entry with all its details.\n\nTry more advanced queries:\n\n # Get only ERROR level logs\n curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n \n # Search for specific keywords\n curl “http://localhost:8080/api/v1/logs?keyword=distributed”\n \n # Get logs from a specific source\n curl “http://localhost:8080/api/v1/logs?source=learning-system”\n \n\n**IMAGE PLACEHOLDER: Query Response Example** _Formatted JSON output showing log query results_\n\n* * *\n\nStep 8: Send Multiple Logs (Batch Processing)\n---------------------------------------------\n\nOur system can handle many logs at once. Let’s test the batch endpoint.\n\nSend a batch of logs:\n\n curl -X POST http://localhost:8081/api/v1/logs/batch \\\n -H “Content-Type: application/json” \\\n -d ‘[\n {\n “message”: “User logged in successfully”,\n “level”: “INFO”,\n “source”: “auth-service”\n },\n {\n “message”: “Failed to connect to database”,\n “level”: “ERROR”,\n “source”: “db-service”\n },\n {\n “message”: “Payment processed”,\n “level”: “INFO”,\n “source”: “payment-service”\n }\n ]’\n \n\nQuery to see all your logs:\n\n curl “http://localhost:8080/api/v1/logs?size=20”\n \n\n* * *\n\nStep 9: View System Statistics\n------------------------------\n\nThe gateway provides an analytics endpoint that shows you interesting statistics about your logs.\n\nGet statistics:\n\n curl “http://localhost:8080/api/v1/logs/stats”\n \n\nThis returns:\n\n* Count of logs by level (INFO, ERROR, etc.)\n \n* Top sources generating logs\n \n* Total log count in time range\n \n\n* * *\n\nStep 10: Run the Load Test\n--------------------------\n\nLet’s stress test your system to see how it handles high volume. The load test script sends 1,000 logs and makes 50 queries simultaneously.\n\nRun the load test:\n\n ./load-test.sh\n \n\nWatch the terminal output as it:\n\n* Sends 10 batches of 100 logs each\n \n* Performs 50 different queries\n \n* Reports success/failure for each operation\n \n\nThe test takes about 30-40 seconds to complete.\n\n* * *\n\nStep 11: Monitor Performance with Prometheus\n--------------------------------------------\n\nPrometheus collects metrics from all your services. Let’s see what it’s tracking.\n\nOpen Prometheus in your browser:\n\n http://localhost:9090\n \n\nTry these queries in the Prometheus UI:\n\n**Query 1 - Log ingestion rate:**\n\n rate(logs_received_total[5m])\n \n\n**Query 2 - Cache hit ratio:**\n\n rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) * 100\n \n\n**Query 3 - Average query response time:**\n\n rate(log_query_duration_seconds_sum[5m]) / rate(log_query_duration_seconds_count[5m])\n \n\nClick “Execute” and “Graph” to see visual representations.\n\n* * *\n\nStep 12: Create Dashboards in Grafana\n-------------------------------------\n\nGrafana turns your metrics into beautiful, real-time dashboards.\n\nOpen Grafana in your browser:\n\n http://localhost:3000\n \n\nLogin credentials:\n\n* Username: admin\n \n* Password: admin\n \n\n(You’ll be prompted to change the password—you can skip this for now)\n\nThe system comes with a pre-built dashboard called “Log Processor System Dashboard” that shows:\n\n* Log ingestion rate over time\n \n* Cache hit percentage\n \n* Query performance\n \n* Error rates\n \n\n* * *\n\nStep 13: Run Integration Tests\n------------------------------\n\nIntegration tests verify that all components work together correctly. These tests simulate real-world usage patterns.\n\nRun the integration test suite:\n\n ./integration-tests/system-integration-test.sh\n \n\nThe tests check:\n\n* All services are responding\n \n* Logs can be sent and retrieved\n \n* Queries return correct results\n \n* Statistics are calculated properly\n \n\nYou’ll see “All integration tests passed!” if everything works.\n\n* * *\n\nUnderstanding What You Built\n----------------------------\n\nLet’s review what each component does and how they work together:\n\n### The Log Producer Service (Port 8081)\n\nThis is the entry point for logs. Applications send their logs here via HTTP requests. The producer:\n\n* Validates incoming logs\n \n* Immediately returns a response (doesn’t wait for processing)\n \n* Sends logs to Kafka for asynchronous processing\n \n* Tracks how many logs it receives\n \n\n### Apache Kafka (Port 9092)\n\nThink of Kafka as a super-fast, reliable message highway. It:\n\n* Stores logs temporarily in topics (named channels)\n \n* Guarantees logs won’t be lost even if services crash\n \n* Allows multiple consumers to read the same logs\n \n* Handles millions of messages per second\n \n\n### The Log Consumer Service (Port 8082)\n\nThis service reads from Kafka and saves logs to the database. It:\n\n* Processes logs in batches for efficiency\n \n* Acknowledges messages only after successful storage\n \n* Handles failures gracefully with retries\n \n* Keeps track of processing progress\n \n\n### PostgreSQL Database (Port 5432)\n\nThe permanent storage for all logs. It features:\n\n* Optimized indexes for fast time-based queries\n \n* Support for complex filtering and aggregation\n \n* ACID guarantees for data consistency\n \n* Ability to store billions of log records\n \n\n### Redis Cache (Port 6379)\n\nAn extremely fast in-memory cache that:\n\n* Stores frequently accessed query results\n \n* Reduces database load by 90%+\n \n* Expires old data automatically\n \n* Returns results in milliseconds\n \n\n### The API Gateway Service (Port 8080)\n\nYour query interface. It:\n\n* Checks Redis cache first (cache-aside pattern)\n \n* Falls back to database on cache miss\n \n* Implements circuit breakers for fault tolerance\n \n* Provides REST APIs for searching logs\n \n\n### Prometheus (Port 9090)\n\nCollects metrics from all services:\n\n* Scrapes metrics every 15 seconds\n \n* Stores time-series data\n \n* Enables alerting on thresholds\n \n* Powers Grafana dashboards\n \n\n### Grafana (Port 3000)\n\nVisualizes your metrics:\n\n* Creates real-time dashboards\n \n* Shows trends and patterns\n \n* Helps identify performance issues\n \n* Makes data accessible to everyone\n \n\n* * *\n\nKey Concepts You’ve Learned\n---------------------------\n\n### Eventual Consistency\n\nWhen you send a log to the producer, it returns immediately. But the log isn’t queryable instantly—it takes 100-500ms to flow through Kafka, get processed by the consumer, and land in the database. This delay is called eventual consistency.\n\n**Why it matters**: This trade-off lets your system handle way more logs. If every log had to be saved before responding, your system would be much slower.\n\n### Circuit Breaker Pattern\n\nIf the database goes down, the circuit breaker “opens” after detecting failures. Instead of trying the database repeatedly, the gateway serves cached data only. Once the database recovers, the circuit breaker gradually “closes” again.\n\n**Why it matters**: Prevents cascade failures where one slow component brings down the entire system.\n\n### Cache-Aside Pattern\n\nThe gateway doesn’t automatically populate the cache. Instead:\n\n1. Check if result exists in cache\n \n2. If yes, return it (cache hit)\n \n3. If no, query database (cache miss)\n \n4. Store result in cache for next time\n \n\n**Why it matters**: Simple to implement and works well for read-heavy workloads like log queries.\n\n### Horizontal Scaling\n\nEach service is stateless—it doesn’t store any data in memory that matters. This means you can run multiple copies:\n\n* 3 producer instances can share incoming requests\n \n* 5 consumer instances can divide Kafka partitions\n \n* 4 gateway instances can handle more queries\n \n\n**Why it matters**: As your traffic grows, just add more servers. No fundamental redesign needed.\n\n* * *\n\nTroubleshooting Common Issues\n-----------------------------\n\n### Problem: Services won’t start\n\n**Check 1**: Are the ports already in use?\n\n lsof -i :8080 # Check gateway port\n lsof -i :8081 # Check producer port\n lsof -i :8082 # Check consumer port\n \n\n**Solution**: Stop any other programs using these ports.\n\n**Check 2**: Is Docker running?\n\n docker ps\n \n\n**Solution**: Start Docker Desktop or the Docker daemon.\n\n### Problem: Can’t connect to Kafka\n\n**Check**: Is Kafka healthy?\n\n docker exec kafka kafka-topics --list --bootstrap-server localhost:9092\n \n\n**Solution**: Restart Kafka:\n\n docker-compose restart kafka\n \n\n### Problem: Queries are very slow\n\n**Check**: What’s the cache hit rate?\n\n curl http://localhost:8080/actuator/metrics/cache.gets\n \n\n**Solution**: If hit rate is low, your cache TTL might be too short. Check Redis:\n\n docker exec redis redis-cli info stats\n \n\n### Problem: Consumer isn’t processing logs\n\n**Check**: Consumer lag in Kafka:\n\n docker exec kafka kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-consumer-group\n \n\n**Solution**: If lag is growing, the consumer might be overwhelmed. Check the consumer service logs for errors.\n\n* * *\n\nExperimenting Further\n---------------------\n\nNow that your system is running, try these experiments to learn more:\n\n### Experiment 1: Break the Database\n\nStop PostgreSQL and see what happens:\n\n docker-compose stop postgres\n \n\nSend logs and try queries. Notice:\n\n* Producer still accepts logs (Kafka buffers them)\n \n* Gateway serves cached queries\n \n* Circuit breaker prevents errors\n \n\nRestart the database:\n\n docker-compose start postgres\n \n\nWatch the consumer catch up on processing the buffered logs.\n\n### Experiment 2: Cache Performance\n\nSend the same query twice and compare response times:\n\n time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n \n\nThe second query should be much faster (cached result).\n\n### Experiment 3: Batch vs. Individual\n\nCompare sending 100 logs individually vs. one batch:\n\n**Individual (slow):**\n\n for i in {1..100}; do\n curl -X POST http://localhost:8081/api/v1/logs \\\n -H “Content-Type: application/json” \\\n -d ‘{”message”:”Log ‘$i’”,”level”:”INFO”,”source”:”test”}’\n done\n \n\n**Batch (fast):** Use the batch endpoint with 100 logs in one request.\n\nNotice how much faster batching is!\n\n* * *\n\nWhat’s Next\n-----------\n\nYou’ve built a distributed system that handles thousands of logs per second. Here’s what you can explore next:\n\n**Add More Features:**\n\n* User authentication for the API\n \n* Rate limiting to prevent abuse\n \n* Data retention policies (auto-delete old logs)\n \n* Full-text search with Elasticsearch\n \n\n**Scale It Further:**\n\n* Deploy to Kubernetes\n \n* Add multiple Kafka brokers\n \n* Set up database replication\n \n* Implement cross-region deployment\n \n\n**Production Hardening:**\n\n* Add encryption for data in transit\n \n* Implement audit logging\n \n* Set up automated backups\n \n* Create runbooks for incidents\n \n\nTomorrow in Day 7, we’ll add a service mesh to handle advanced routing, authentication between services, and implement A/B testing capabilities.\n\n* * *\n\nCleaning Up\n-----------\n\nWhen you’re done experimenting, shut everything down:\n\n # Stop application services\n # Press Ctrl+C in each terminal window\n \n # Stop infrastructure\n docker-compose down\n \n # Remove data volumes (optional - deletes all logs)\n docker-compose down -v\n \n\nYour code and configuration files remain intact, so you can restart anytime by running the setup script again.\nYou can include dynamic values by using placeholders like: https://drewdru.syndichain.com/articles/c2d99fcc-d5e0-4576-8c78-b87b71493fb5 , Drew Dru, https://sdcourse.substack.com/p/day-6-building-a-distributed-log , drewdru, drewdru, drewdru, drewdru These will automatically be replaced with the actual data when the message is sent.",1760606111,"publishedToNostr","htmlContentKind30023","

What We’re Building Today

\n
Today we’re transforming our basic log storage system into a production-grade distributed query engine that handles real-time log processing at scale. You’ll implement:
\n

\n
Distributed Log Query API: RESTful service with advanced filtering, aggregation, and real-time search capabilities
\n
\n
Event-Driven Processing Pipeline: Kafka-based system processing 10K+ logs/second with guaranteed delivery
\n
\n
Intelligent Caching Layer: Redis-powered query optimization reducing response times from 2s to 50ms
\n
\n
Production Monitoring Stack: Complete observability with Prometheus metrics, Grafana dashboards, and distributed tracing
\n

Why This Matters: From Local Files to Planet-Scale Systems

\n
The jump from flat file storage to distributed query processing represents one of the most critical scaling challenges in modern systems. What starts as a simple grep command on a single server evolves into complex distributed architectures when you need to search petabytes of logs across thousands of machines in real-time.
\n
Consider Netflix’s challenge: processing 500+ billion events daily across global infrastructure. Their logs aren’t just stored—they’re actively queried by fraud detection systems, recommendation engines, and operational dashboards. A single query might need to aggregate data across 50 different microservices, each running in multiple regions. The system you’re building today demonstrates the foundational patterns that make this scale possible.
\n
The core architectural challenge isn’t just storage or retrieval—it’s maintaining query performance and system availability while data volume grows exponentially. Today’s implementation introduces you to the distributed systems trade-offs that separate senior engineers from those still thinking in single-machine terms.
\n

System Design Deep Dive: Five Core Patterns for Distributed Query Processing

1. Event Sourcing with Command Query Responsibility Segregation (CQRS)

Traditional systems couple write and read operations, creating bottlenecks when query patterns differ from write patterns. Our log processing system implements CQRS by separating:

\n
Write Side: High-throughput log ingestion via Kafka with minimal validation
\n
\n
Read Side: Optimized query structures with pre-computed aggregations and indexes
\n

2. Distributed Caching with Cache-Aside Pattern

Raw log queries against PostgreSQL become prohibitively expensive beyond 1M records. Our Redis implementation uses the cache-aside pattern where:

\n
Application code manages cache population and invalidation
\n
\n
Cache misses trigger database queries with automatic cache warming
\n
\n
TTL policies balance data freshness with performance
\n

3. Circuit Breaker Pattern for Fault Isolation

Distributed systems fail in cascade patterns—one slow component triggers timeouts across the entire system. Our Resilience4j implementation provides:

\n
Automatic failure detection with configurable thresholds
\n
\n
Fast-fail responses preventing resource exhaustion
\n
\n
Gradual recovery with half-open state testing
\n

4. Asynchronous Processing with Guaranteed Delivery

Kafka’s producer-consumer model ensures log processing continues even when downstream services are temporarily unavailable. Our implementation includes:

\n
At-least-once delivery with producer retries and consumer acknowledgments
\n
\n
Dead letter queues for handling poison messages
\n
\n
Consumer group management for horizontal scaling
\n

Scalability Implication: This pattern allows you to decouple ingestion rate from processing rate. During traffic spikes, Kafka acts as a buffer, preventing log loss while consumers catch up.

5. Distributed Observability with Correlation IDs

Production distributed systems require comprehensive observability. Our implementation includes:

\n
Distributed tracing with correlation IDs propagated across service boundaries
\n
\n
Structured logging with consistent metadata for aggregation
\n
\n
Custom metrics measuring business logic, not just infrastructure
\n

Implementation Walkthrough: Building Production-Grade Components

Core Architecture Decisions

Our system implements a three-service architecture optimized for independent scaling:

API Gateway Service: Provides query interface with intelligent routing and caching. Implements query complexity analysis to prevent expensive operations from overwhelming the system.

[

$\"\"$

](https://substackcdn.com/image/fetch/$s_!ymNu!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3aeb09c3-dd9a-4a5a-a40d-751cc542ceaa_935x615.png)

Database Schema Optimization

Traditional log tables become unusable beyond 10M records without proper indexing strategy:

-- Composite index optimized for time-range queries\nCREATE INDEX idx_logs_timestamp_level ON logs(timestamp DESC, log_level);\n\n-- Partial index for error-only queries (common pattern)\nCREATE INDEX idx_logs_errors ON logs(timestamp DESC) WHERE log_level = ‘ERROR’;\n

Caching Strategy Implementation

Redis serves three distinct caching layers:

\n
Query Result Caching: Exact query matches with 5-minute TTL
\n
\n
Aggregation Caching: Pre-computed hourly/daily summaries with 1-hour TTL
\n
\n
Hot Data Caching: Recent logs (last 15 minutes) with 30-second TTL
\n

Cache key design prevents collisions while enabling pattern-based invalidation during data updates.

Production Considerations: Performance and Reliability

Performance Benchmarks

Under normal load (1,000 logs/second), the system maintains:

\n
Query response times under 100ms (95th percentile)
\n
\n
Cache hit ratios above 94%
\n
\n
End-to-end processing latency under 200ms
\n

During spike conditions (10,000 logs/second), performance gracefully degrades:

\n
Query response times increase to 300ms (acceptable for operational dashboards)
\n
\n
Circuit breakers prevent cascade failures
\n
\n
Kafka buffering maintains zero data loss
\n

Failure Scenarios and Recovery

Database Connection Loss: Circuit breakers activate after 3 consecutive failures, routing all queries to cached data. System continues serving recent logs while database recovers.

Kafka Cluster Outage: Producer buffering and retry logic maintain data integrity. Consumer lag is measured and alerts trigger when processing falls behind ingestion.

Redis Cluster Failure: System automatically falls back to direct database queries with degraded performance. Cache warming begins immediately upon Redis recovery.

Scale Connection: Enterprise Production Patterns

The primary difference at massive scale is data partitioning strategy and cross-region replication, concepts we’ll explore in Week 3 when building geographically distributed systems.

Working Code Demo:

Next Steps: Tomorrow’s Advanced Concepts

Building Your System: Step-by-Step Implementation

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day6/distributed-log-processor\n

Prerequisites

Before starting, make sure you have these tools installed on your computer:

\n
Java 17 or newer - The programming language we’re using
\n
\n
Maven 3.6 or newer - Builds our Java projects
\n
\n
Docker and Docker Compose - Runs our infrastructure services
\n
\n
A code editor - VS Code, IntelliJ IDEA, or Eclipse
\n

You can verify your installations by running these commands in your terminal:

java -version\nmvn -version\ndocker --version\ndocker-compose --version\n

Step 1: Generate the Project Files

We’ve created a script that automatically generates all the code and configuration files you need. This saves you from typing thousands of lines of code manually.

Run the generator script:

chmod +x generate_system_files.sh\n./generate_system_files.sh\n

This creates a folder called distributed-log-processor containing:

\n
Three Spring Boot services (Producer, Consumer, Gateway)
\n
\n
Database configurations
\n
\n
Docker setup files
\n
\n
Testing scripts
\n
\n
Complete documentation
\n

Navigate into the project:

cd distributed-log-processor\n

Step 2: Start the Infrastructure Services

Our application needs several supporting services to run. Think of these as the foundation of your house—you need them in place before you can build the walls.

Start all infrastructure services:

./setup.sh\n

This script starts:

\n
Kafka - Message streaming platform
\n
\n
PostgreSQL - Main database for storing logs
\n
\n
Redis - Fast cache for query results
\n
\n
Prometheus - Collects performance metrics
\n
\n
Grafana - Creates visual dashboards
\n

The script takes about 30-60 seconds to start everything. You’ll see messages as each service comes online.

Verify everything is running:

docker-compose ps\n

You should see all services showing “Up” status.

Step 3: Build the Application Services

Now we’ll compile our three Spring Boot services into executable JAR files.

Build all services at once:

mvn clean install\n

This command:

\n
Downloads all necessary libraries
\n
\n
Compiles your Java code
\n
\n
Runs automated tests
\n
\n
Packages everything into JAR files
\n

The build takes 2-3 minutes on first run. Subsequent builds are faster because Maven caches the libraries.

You’ll see output showing each service being built. Look for “BUILD SUCCESS” messages for all three modules.

Step 4: Start the Application Services

Now comes the exciting part—starting your distributed system! Open three separate terminal windows to run each service. This lets you see the logs from each one independently.

Terminal 1 - Start the Log Producer:

java -jar log-producer/target/log-producer-1.0.0.jar\n

Wait until you see: “Started LogProducerApplication”

Terminal 2 - Start the Log Consumer:

java -jar log-consumer/target/log-consumer-1.0.0.jar\n

Wait until you see: “Started LogConsumerApplication”

Terminal 3 - Start the API Gateway:

java -jar api-gateway/target/api-gateway-1.0.0.jar\n

Wait until you see: “Started ApiGatewayApplication”

Step 5: Verify System Health

Let’s make sure all services can communicate with each other. We’ll check the health endpoints that each service provides.

Check each service:

# Producer health check\ncurl http://localhost:8081/actuator/health\n\n# Consumer health check\ncurl http://localhost:8082/actuator/health\n\n# Gateway health check\ncurl http://localhost:8080/actuator/health\n

Each should return: {”status”:”UP”}

If any service returns an error, check its terminal window for error messages.

Step 6: Your First Log Event

Time to send your first log through the system! We’ll use curl to send an HTTP request to the producer service.

Send a test log:

curl -X POST http://localhost:8081/api/v1/logs \\\n  -H “Content-Type: application/json” \\\n  -d ‘{\n    “message”: “My first distributed log event!”,\n    “level”: “INFO”,\n    “source”: “learning-system”,\n    “metadata”: {\n      “student”: “your-name”,\n      “lesson”: “day-6”\n    }\n  }’\n

You should see: “Log event accepted for processing”

Watch the terminal windows—you’ll see the log flow through the system:

\n
Producer receives it and sends to Kafka
\n
\n
Consumer reads from Kafka and saves to database
\n
\n
Gateway makes it available for queries
\n

Step 7: Query Your Logs

Now let’s retrieve the log you just sent. The gateway service provides a powerful query API.

Get recent logs:

curl “http://localhost:8080/api/v1/logs?size=10”\n

You’ll see a JSON response containing your log entry with all its details.

Try more advanced queries:

# Get only ERROR level logs\ncurl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n\n# Search for specific keywords\ncurl “http://localhost:8080/api/v1/logs?keyword=distributed”\n\n# Get logs from a specific source\ncurl “http://localhost:8080/api/v1/logs?source=learning-system”\n

IMAGE PLACEHOLDER: Query Response Example Formatted JSON output showing log query results

Step 8: Send Multiple Logs (Batch Processing)

Our system can handle many logs at once. Let’s test the batch endpoint.

Send a batch of logs:

curl -X POST http://localhost:8081/api/v1/logs/batch \\\n  -H “Content-Type: application/json” \\\n  -d ‘[\n    {\n      “message”: “User logged in successfully”,\n      “level”: “INFO”,\n      “source”: “auth-service”\n    },\n    {\n      “message”: “Failed to connect to database”,\n      “level”: “ERROR”,\n      “source”: “db-service”\n    },\n    {\n      “message”: “Payment processed”,\n      “level”: “INFO”,\n      “source”: “payment-service”\n    }\n  ]’\n

Query to see all your logs:

curl “http://localhost:8080/api/v1/logs?size=20”\n

Step 9: View System Statistics

The gateway provides an analytics endpoint that shows you interesting statistics about your logs.

Get statistics:

curl “http://localhost:8080/api/v1/logs/stats”\n

This returns:

\n
Count of logs by level (INFO, ERROR, etc.)
\n
\n
Top sources generating logs
\n
\n
Total log count in time range
\n

Step 10: Run the Load Test

Let’s stress test your system to see how it handles high volume. The load test script sends 1,000 logs and makes 50 queries simultaneously.

Run the load test:

./load-test.sh\n

Watch the terminal output as it:

\n
Sends 10 batches of 100 logs each
\n
\n
Performs 50 different queries
\n
\n
Reports success/failure for each operation
\n

The test takes about 30-40 seconds to complete.

Step 11: Monitor Performance with Prometheus

Prometheus collects metrics from all your services. Let’s see what it’s tracking.

Open Prometheus in your browser:

http://localhost:9090\n

Try these queries in the Prometheus UI:

Query 1 - Log ingestion rate:

rate(logs_received_total[5m])\n

Query 2 - Cache hit ratio:

rate(cache_hits_total[5m]) / (rate(cache_hits_total[5m]) + rate(cache_misses_total[5m])) * 100\n

Query 3 - Average query response time:

rate(log_query_duration_seconds_sum[5m]) / rate(log_query_duration_seconds_count[5m])\n

Click “Execute” and “Graph” to see visual representations.

Step 12: Create Dashboards in Grafana

Grafana turns your metrics into beautiful, real-time dashboards.

Open Grafana in your browser:

http://localhost:3000\n

\n
Username: admin
\n
\n
Password: admin
\n

(You’ll be prompted to change the password—you can skip this for now)

The system comes with a pre-built dashboard called “Log Processor System Dashboard” that shows:

\n
Log ingestion rate over time
\n
\n
Cache hit percentage
\n
\n
Query performance
\n
\n
Error rates
\n

Step 13: Run Integration Tests

Integration tests verify that all components work together correctly. These tests simulate real-world usage patterns.

Run the integration test suite:

./integration-tests/system-integration-test.sh\n

The tests check:

\n
All services are responding
\n
\n
Logs can be sent and retrieved
\n
\n
Queries return correct results
\n
\n
Statistics are calculated properly
\n

You’ll see “All integration tests passed!” if everything works.

Understanding What You Built

Let’s review what each component does and how they work together:

The Log Producer Service (Port 8081)

This is the entry point for logs. Applications send their logs here via HTTP requests. The producer:

\n
Validates incoming logs
\n
\n
Immediately returns a response (doesn’t wait for processing)
\n
\n
Sends logs to Kafka for asynchronous processing
\n
\n
Tracks how many logs it receives
\n

Apache Kafka (Port 9092)

Think of Kafka as a super-fast, reliable message highway. It:

\n
Stores logs temporarily in topics (named channels)
\n
\n
Guarantees logs won’t be lost even if services crash
\n
\n
Allows multiple consumers to read the same logs
\n
\n
Handles millions of messages per second
\n

The Log Consumer Service (Port 8082)

This service reads from Kafka and saves logs to the database. It:

\n
Processes logs in batches for efficiency
\n
\n
Acknowledges messages only after successful storage
\n
\n
Handles failures gracefully with retries
\n
\n
Keeps track of processing progress
\n

PostgreSQL Database (Port 5432)

The permanent storage for all logs. It features:

\n
Optimized indexes for fast time-based queries
\n
\n
Support for complex filtering and aggregation
\n
\n
ACID guarantees for data consistency
\n
\n
Ability to store billions of log records
\n

Redis Cache (Port 6379)

An extremely fast in-memory cache that:

\n
Stores frequently accessed query results
\n
\n
Reduces database load by 90%+
\n
\n
Expires old data automatically
\n
\n
Returns results in milliseconds
\n

The API Gateway Service (Port 8080)

Your query interface. It:

\n
Checks Redis cache first (cache-aside pattern)
\n
\n
Falls back to database on cache miss
\n
\n
Implements circuit breakers for fault tolerance
\n
\n
Provides REST APIs for searching logs
\n

Prometheus (Port 9090)

Collects metrics from all services:

\n
Scrapes metrics every 15 seconds
\n
\n
Stores time-series data
\n
\n
Enables alerting on thresholds
\n
\n
Powers Grafana dashboards
\n

Grafana (Port 3000)

Visualizes your metrics:

\n
Creates real-time dashboards
\n
\n
Shows trends and patterns
\n
\n
Helps identify performance issues
\n
\n
Makes data accessible to everyone
\n

Key Concepts You’ve Learned

Eventual Consistency

Why it matters: This trade-off lets your system handle way more logs. If every log had to be saved before responding, your system would be much slower.

Circuit Breaker Pattern

Why it matters: Prevents cascade failures where one slow component brings down the entire system.

Cache-Aside Pattern

The gateway doesn’t automatically populate the cache. Instead:

\n
Check if result exists in cache
\n
\n
If yes, return it (cache hit)
\n
\n
If no, query database (cache miss)
\n
\n
Store result in cache for next time
\n

Why it matters: Simple to implement and works well for read-heavy workloads like log queries.

Horizontal Scaling

Each service is stateless—it doesn’t store any data in memory that matters. This means you can run multiple copies:

\n
3 producer instances can share incoming requests
\n
\n
5 consumer instances can divide Kafka partitions
\n
\n
4 gateway instances can handle more queries
\n

Why it matters: As your traffic grows, just add more servers. No fundamental redesign needed.

Troubleshooting Common Issues

Problem: Services won’t start

Check 1: Are the ports already in use?

lsof -i :8080  # Check gateway port\nlsof -i :8081  # Check producer port\nlsof -i :8082  # Check consumer port\n

Solution: Stop any other programs using these ports.

Check 2: Is Docker running?

docker ps\n

Solution: Start Docker Desktop or the Docker daemon.

Problem: Can’t connect to Kafka

Check: Is Kafka healthy?

docker exec kafka kafka-topics --list --bootstrap-server localhost:9092\n

Solution: Restart Kafka:

docker-compose restart kafka\n

Problem: Queries are very slow

Check: What’s the cache hit rate?

curl http://localhost:8080/actuator/metrics/cache.gets\n

Solution: If hit rate is low, your cache TTL might be too short. Check Redis:

docker exec redis redis-cli info stats\n

Problem: Consumer isn’t processing logs

Check: Consumer lag in Kafka:

docker exec kafka kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group log-consumer-group\n

Solution: If lag is growing, the consumer might be overwhelmed. Check the consumer service logs for errors.

Experimenting Further

Now that your system is running, try these experiments to learn more:

Experiment 1: Break the Database

Stop PostgreSQL and see what happens:

docker-compose stop postgres\n

Send logs and try queries. Notice:

\n
Producer still accepts logs (Kafka buffers them)
\n
\n
Gateway serves cached queries
\n
\n
Circuit breaker prevents errors
\n

Restart the database:

docker-compose start postgres\n

Watch the consumer catch up on processing the buffered logs.

Experiment 2: Cache Performance

Send the same query twice and compare response times:

time curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\ntime curl “http://localhost:8080/api/v1/logs?logLevel=ERROR”\n

The second query should be much faster (cached result).

Experiment 3: Batch vs. Individual

Compare sending 100 logs individually vs. one batch:

Individual (slow):

for i in {1..100}; do\n  curl -X POST http://localhost:8081/api/v1/logs \\\n    -H “Content-Type: application/json” \\\n    -d ‘{”message”:”Log ‘$i’”,”level”:”INFO”,”source”:”test”}’\ndone\n

Batch (fast): Use the batch endpoint with 100 logs in one request.

Notice how much faster batching is!

What’s Next

You’ve built a distributed system that handles thousands of logs per second. Here’s what you can explore next:

Add More Features:

\n
User authentication for the API
\n
\n
Rate limiting to prevent abuse
\n
\n
Data retention policies (auto-delete old logs)
\n
\n
Full-text search with Elasticsearch
\n

Scale It Further:

\n
Deploy to Kubernetes
\n
\n
Add multiple Kafka brokers
\n
\n
Set up database replication
\n
\n
Implement cross-region deployment
\n

Production Hardening:

\n
Add encryption for data in transit
\n
\n
Implement audit logging
\n
\n
Set up automated backups
\n
\n
Create runbooks for incidents
\n

Tomorrow in Day 7, we’ll add a service mesh to handle advanced routing, authentication between services, and implement A/B testing capabilities.

Cleaning Up

When you’re done experimenting, shut everything down:

# Stop application services\n# Press Ctrl+C in each terminal window\n\n# Stop infrastructure\ndocker-compose down\n\n# Remove data volumes (optional - deletes all logs)\ndocker-compose down -v\n

Your code and configuration files remain intact, so you can restart anytime by running the setup script again.

","pubDate","2025-10-16T09:05:19.000Z","modifiedAt","2025-10-16T09:15:11.840Z","storyId","likes","reposts","comments",{"_14":15,"_16":17,"_18":19,"_20":21,"_22":-5,"_23":24,"_25":24,"_26":27,"_28":24,"_29":24,"_30":24,"_31":-5,"_32":-5,"_33":24,"_34":35,"_36":37,"_38":37,"_39":40,"_41":245,"_74":247},{"_14":40,"_43":44,"_45":44,"_46":27,"_28":47,"_48":49,"_50":-5,"_51":52,"_53":-5,"_54":55,"_56":-5,"_57":47,"_58":27,"_59":21,"_60":61,"_62":246,"_64":-5,"_65":66,"_67":55,"_68":55,"_69":-5,"_70":-5,"_71":-5,"_36":72,"_38":73},[],{"_76":15,"_77":-5,"_78":-5,"_79":80,"_81":82,"_83":248,"_87":249,"_91":92},{"_85":66,"_86":66},[89,90],"relatedArticles",[],"actionData","errors"]