What We’re Building Today

Today marks a critical milestone in our distributed systems journey. We’re integrating all the components we’ve built over the past week into a complete, production-ready log processing pipeline that runs on a single machine but follows distributed system principles. By the end of this lesson, you’ll have:

• A multi-service Spring Boot system with log producers, consumers, and a query API gateway • Event-driven architecture using Apache Kafka for reliable message delivery and decoupling • Persistent storage layer with PostgreSQL for long-term log retention and Redis for high-performance caching • Production observability with Prometheus metrics, distributed tracing, and Grafana dashboards

Why This Foundation Matters

What we’re building today isn’t just a local system—it’s a scaled-down version of the same architectural patterns that power Netflix’s real-time analytics, Uber’s event streaming platform, and Amazon’s CloudWatch logs. Every design decision we make reflects the same trade-offs these companies face at massive scale.

The critical insight here is understanding that distributed systems aren’t about having multiple machines—they’re about designing with distribution principles from day one. When Netflix processes billions of log events daily, they use the exact same event sourcing patterns, message queue semantics, and observability strategies we’ll implement today. The only difference is scale.

By building these patterns locally first, you’ll understand the fundamental challenges: message ordering guarantees, failure recovery, backpressure handling, and data consistency models. These challenges don’t change whether you’re processing 1,000 events per second or 1 million—only your infrastructure capacity changes.

System Design Deep Dive: Core Distributed Patterns

Event Sourcing and Message Queuing Architecture

Our system implements event sourcing through Kafka, where every log event becomes an immutable fact in our system’s history. This isn’t just message passing—it’s a fundamental shift from request-response to event-driven thinking.

The key trade-off here is consistency versus availability. By using Kafka’s partitioning strategy, we guarantee message ordering within partitions but accept eventual consistency across the system. This is the same decision Uber made for their real-time data platform: they prioritized availability and partition tolerance over immediate consistency, accepting that different services might have slightly different views of the world at any given moment.

Anti-pattern warning: Don’t fall into the trap of using Kafka like a traditional message queue with immediate acknowledgments. The power comes from treating it as a distributed log where consumers can replay events and multiple subscribers can process the same stream independently.

Cache-Aside Pattern with Redis

We implement cache-aside caching where our API gateway checks Redis first, then falls back to PostgreSQL for cache misses. This pattern handles the classic trade-off between read latency and data freshness.

The architectural insight here is understanding cache invalidation strategies. We use time-based expiration (TTL) for simplicity, but production systems like Amazon’s DynamoDB Accelerator (DAX) implement sophisticated invalidation based on write-through patterns and event triggers.

Scalability bottleneck: Redis becomes a single point of failure in this design. At Netflix scale, this evolves into Redis clusters with consistent hashing, but for our local system, we accept this limitation while building the foundational patterns correctly.

Circuit Breaker and Bulkhead Patterns

Using Resilience4j, we implement circuit breakers around external service calls and bulkhead patterns to isolate different types of requests. This prevents cascade failures where one slow database query doesn’t bring down your entire log ingestion pipeline.

The trade-off is between system resilience and operational complexity. Circuit breakers add monitoring overhead and require careful tuning, but they’re essential for systems that need to handle partial failures gracefully. This is exactly how Spotify prevents their recommendation service outages from affecting music playback.

Observable Systems Architecture

We instrument everything with Micrometer metrics and distributed tracing. This isn’t just monitoring—it’s designing for debuggability at scale. Every service exposes health checks, custom metrics, and trace context propagation.

Production consideration: Observability has performance costs. Each metric collection and trace span adds latency. We carefully choose what to measure, focusing on RED metrics (Rate, Errors, Duration) rather than monitoring everything.

Database Connection Pool Management

Our system uses HikariCP connection pooling with carefully tuned parameters. This addresses the fundamental challenge of database connections as a finite resource that must be shared across concurrent requests.

The key insight is that connection pools are your first line of defense against database overload. The configuration we use (maximum 20 connections, 30-second timeout) reflects production patterns where you’d rather fail fast with connection timeouts than overwhelm your database with too many concurrent queries.

Implementation Walkthrough

Service Architecture Decisions

We structure our system as three distinct Spring Boot services:

Log Producer Service: Exposes REST endpoints for log ingestion and publishes to Kafka. The architectural decision here is to separate ingestion from processing, allowing us to handle traffic spikes by buffering events in Kafka rather than overwhelming downstream services.

Log Consumer Service: Subscribes to Kafka topics and persists logs to PostgreSQL. We implement idempotent processing using log event IDs, ensuring that message redelivery doesn’t create duplicate records. This reflects the same exactly-once semantics that Apache Flink uses for stream processing.

API Gateway Service: Provides query endpoints with Redis caching. The gateway pattern centralizes cross-cutting concerns like authentication, rate limiting, and response caching. This is the same pattern Amazon uses for API Gateway, though our implementation is obviously simpler.

[Image placeholder: Service interaction diagram showing request/response flow between services]

Message Processing Strategy

We configure Kafka consumers with manual acknowledgment, giving us control over when messages are considered processed. This prevents message loss during service restarts and allows for sophisticated error handling strategies.

The consumer group configuration ensures that multiple consumer instances can process messages in parallel while maintaining ordering guarantees within partitions. This is crucial for scaling—you can add more consumer instances without changing your application code.

Database Design Philosophy

Our PostgreSQL schema includes proper indexing for time-range queries, which are the most common pattern for log systems. We also implement connection pooling and prepared statements for optimal performance.

Performance insight: We use JSONB for storing log metadata, taking advantage of PostgreSQL’s native JSON indexing. This gives us the flexibility of document storage with the performance of relational queries—exactly the approach Discord uses for storing billions of messages.

Production Considerations

Performance Optimization

The system includes JVM tuning parameters optimized for throughput over latency. We configure G1GC for consistent performance under varying load conditions. Kafka producers use batching and compression to maximize throughput while minimizing network overhead.

Monitoring strategy: We track JVM metrics, Kafka lag, database connection pool utilization, and Redis hit rates. These four metrics tell you everything you need to know about system health in production.

Failure Scenarios and Recovery

We implement graceful degradation where Redis failures don’t break the API—queries just become slower as they hit PostgreSQL directly. Kafka failures are handled by Spring Retry with exponential backoff, preventing overwhelming the broker during recovery.

The critical insight is that systems fail in partial ways. Netflix’s chaos engineering approach teaches us to build systems that continue operating even when individual components fail. Our circuit breaker configuration reflects this philosophy.

Deployment and Scaling Strategies

The Docker Compose configuration includes resource limits and restart policies that reflect production deployment practices. We use health checks to ensure services are genuinely ready before receiving traffic.

Capacity planning: The system is configured to handle approximately 1,000 events per second on standard development hardware. The bottleneck is typically PostgreSQL writes, which is exactly what you’d expect in a log processing system.

Scale Connection: Enterprise Patterns

This exact architecture pattern appears in every major tech company, scaled up with additional infrastructure:

Netflix: Their event streaming platform processes billions of events using the same Kafka partitioning strategy and consumer group patterns we implement
Uber: Their real-time data infrastructure uses identical circuit breaker and bulkhead patterns for handling partial failures across thousands of microservices
Amazon: CloudWatch Logs uses the same cache-aside pattern with ElastiCache (Redis) for high-performance log queries

The architectural principles don’t change at scale—only the infrastructure capacity and operational complexity increase.

Next Steps

Tomorrow we’ll break our single-machine constraint and implement TCP-based log collection, introducing network partition handling and service discovery patterns. This builds naturally on today’s foundation by extending our event-driven architecture across machine boundaries.

The key insight you should take from today: distributed systems design isn’t about having multiple machines—it’s about designing with distribution principles that make your system resilient, observable, and scalable from day one.

Youtube Video:

Implementation Guide

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day7/distributed-log-processor

Setting Up Your Development Environment

Before we start building, let’s make sure you have everything you need installed on your computer.

Required Software

Java Development Kit 17+

java -version\n# Should show version 17 or higher\n

Maven 3.8+

mvn -version\n# Should show version 3.8 or higher\n

Docker and Docker Compose

docker --version\ndocker-compose --version\n# Both should be installed and running\n

If you’re missing any of these, install them before continuing. Docker is especially important since it will run our databases and message queues.

Building the Complete System

Step 1: Generate the Project Structure

Run the system generator script we created:

chmod +x generate_system_files.sh\n./generate_system_files.sh\n

This creates a complete project with three Spring Boot services and all the configuration files we need. You should see a new folder called distributed-log-processor with everything inside.

[Image placeholder: Terminal screenshot showing successful project generation]

Step 2: Start the Infrastructure Services

Navigate to your project folder and start the supporting services:

cd distributed-log-processor\n./setup.sh\n

This command starts:

PostgreSQL database (port 5432)
Apache Kafka with Zookeeper (port 9092)
Redis cache (port 6379)
Prometheus monitoring (port 9090)
Grafana dashboards (port 3000)

Wait for the setup script to complete. It will check that all services are healthy before finishing.

Step 3: Build the Java Applications

Now let’s compile our three Spring Boot services:

# Build all services at once\nmvn clean package -DskipTests\n\n# This might take a few minutes the first time\n# Maven will download all the dependencies we need\n

You should see “BUILD SUCCESS” for each of the three services: log-producer, log-consumer, and api-gateway.

Step 4: Start the Services

Open three separate terminal windows and start each service:

Terminal 1 - Log Producer:

cd log-producer\nmvn spring-boot:run\n

Terminal 2 - Log Consumer:

cd log-consumer  \nmvn spring-boot:run\n

Terminal 3 - API Gateway:

cd api-gateway\nmvn spring-boot:run\n

Each service will take about 30-60 seconds to start. You’ll know they’re ready when you see “Started [ServiceName]Application” in the logs.

[Image placeholder: Three terminal windows showing services starting up]

Testing Your System

Health Check Test

Let’s verify everything is working by checking the health of each service:

# Check Log Producer (port 8081)\ncurl http://localhost:8081/producer/api/logs/health\n\n# Check Log Consumer (port 8082) \ncurl http://localhost:8082/actuator/health\n\n# Check API Gateway (port 8080)\ncurl http://localhost:8080/gateway/api/query/health\n

Each command should return a JSON response with “status”: “healthy”.

Send Your First Log Event

Let’s send a log message to our system:

curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n  -H “Content-Type: application/json” \\\n  -d ‘{\n    “level”: “INFO”,\n    “message”: “Hello from our distributed log system!”,\n    “source”: “test-application”,\n    “metadata”: {\n      “userId”: “student123”,\n      “action”: “first_test”\n    }\n  }’\n

You should get back a response like:

{\n  “status”: “accepted”,\n  “logId”: “some-unique-id”,\n  “message”: “Log event successfully ingested”\n}\n

Query Your Log Back

Wait about 5 seconds for the message to be processed, then query it back:

curl “http://localhost:8080/gateway/api/query/logs?source=test-application”\n

You should see your log message in the response, along with timestamps and the metadata you sent.

Run the Full Integration Test

We’ve provided a complete test suite that verifies all the functionality:

./integration-tests/run-tests.sh\n

This script tests:

Health checks for all services
Single log ingestion
Batch log ingestion
Log querying and search
Statistics generation

All tests should pass with green checkmarks.

Performance Testing

Load Test

Let’s see how much load our system can handle:

./load-test.sh\n

This sends 600 log messages over 60 seconds (10 per second) and measures response times. Watch the terminal output to see how your system performs under load.

Monitor Performance

While the load test is running, open Grafana in your web browser:

http://localhost:3000\n

Requests per second
Response times
Error rates
Database and cache performance

[Image placeholder: Grafana dashboard showing live metrics during load test]

Understanding the Data Flow

Let’s trace what happens when you send a log message:

HTTP Request → Your curl command sends JSON to the Log Producer
Kafka Publishing → Producer converts JSON to Kafka message and sends to log-events topic
Message Consumption → Log Consumer receives message from Kafka
Database Write → Consumer saves log to PostgreSQL with proper indexing
Query Path → API Gateway checks Redis cache first, then PostgreSQL if needed
Monitoring → All steps generate metrics that Prometheus collects

Viewing the Raw Data

You can see your logs directly in the database:

# Connect to PostgreSQL  \ndocker exec -it postgres psql -U loguser -d logprocessor\n\n# Query your logs\nSELECT id, level, message, source, timestamp FROM log_events ORDER BY timestamp DESC LIMIT 10;\n\n# Exit the database\n\\q\n

Checking Kafka Topics

You can also see the messages in Kafka:

# List Kafka topics\ndocker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list\n\n# See recent messages (press Ctrl+C to stop)\ndocker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic log-events --from-beginning\n

Exploring Advanced Features

Circuit Breaker Testing

Let’s trigger the circuit breaker by stopping the database:

# Stop PostgreSQL\ndocker stop postgres\n\n# Try to send logs - should get circuit breaker response\ncurl -X POST http://localhost:8081/producer/api/logs/ingest \\\n  -H “Content-Type: application/json” \\\n  -d ‘{”level”: “ERROR”, “message”: “Test during outage”, “source”: “test”}’\n\n# Restart PostgreSQL  \ndocker start postgres\n

The system should return a “service unavailable” response when the database is down, demonstrating our resilience patterns.

Cache Performance

Let’s test the cache-aside pattern:

# First query (cache miss) - will be slower\ntime curl “http://localhost:8080/gateway/api/query/stats”\n\n# Second query (cache hit) - much faster  \ntime curl “http://localhost:8080/gateway/api/query/stats”\n

The second request should be noticeably faster because it’s served from Redis.

Batch Processing

Test sending multiple logs at once:

curl -X POST http://localhost:8081/producer/api/logs/batch \\\n  -H “Content-Type: application/json” \\\n  -d ‘[\n    {”level”: “INFO”, “message”: “Batch message 1”, “source”: “batch-test”},\n    {”level”: “WARN”, “message”: “Batch message 2”, “source”: “batch-test”}, \n    {”level”: “ERROR”, “message”: “Batch message 3”, “source”: “batch-test”}\n  ]’\n

This is much more efficient than sending logs one at a time.

Monitoring and Debugging

Key Metrics to Watch

In Grafana, focus on these important metrics:

Throughput Metrics:

log_ingestion_total - How many logs per second we’re processing
kafka_messages_sent_total - Messages successfully sent to Kafka
kafka_messages_consumed_total - Messages processed by consumers

Performance Metrics:

log_ingestion_duration - How long it takes to process each log
http_server_requests - API response times
log_processing_duration - Consumer processing time

Health Metrics:

Circuit breaker states (open/closed/half-open)
Database connection pool usage
Redis hit/miss rates
JVM memory and garbage collection

Common Issues and Solutions

“Connection refused” errors: Make sure Docker containers are running with docker-compose ps

Slow queries: Check the database connection pool isn’t exhausted in the metrics

High memory usage: Monitor JVM heap size and consider increasing it with -Xmx4g

Kafka lag: If consumers fall behind, increase the number of consumer instances

Real-World Applications

The system you’ve built demonstrates the same patterns used by:

Netflix: Their event streaming processes billions of user interactions daily using identical Kafka partitioning and consumer group patterns.

Uber: Their dispatch system uses the same circuit breaker patterns to handle partial failures across thousands of services.

Discord: Their message storage uses PostgreSQL with JSONB for metadata, just like our log system.

Amazon CloudWatch: Uses cache-aside patterns with Redis for high-performance log queries.

The only differences at scale are:

More machines and larger clusters
Additional monitoring and alerting
More sophisticated deployment automation
Higher performance requirements

But the fundamental architectural patterns remain the same.

[Image placeholder: Comparison diagram showing local system vs enterprise scale]

What You’ve Accomplished

Congratulations! You’ve built a production-ready distributed log processing system that includes:

Event-driven architecture with proper message ordering
Resilience patterns that handle failures gracefully
High-performance caching with Redis
Comprehensive monitoring with Prometheus and Grafana
Horizontal scaling capabilities built-in from day one

This foundation prepares you for tomorrow’s lesson where we’ll extend this system across network boundaries, introducing service discovery and network partition handling.

The most important lesson: distributed systems aren’t about having multiple machines—they’re about designing with distribution principles that make systems resilient, observable, and scalable from the start.

Cleanup

When you’re done experimenting, stop all services:

# Stop Spring Boot services (Ctrl+C in each terminal)\n\n# Stop Docker infrastructure\ndocker-compose down\n\n# Remove volumes if you want to start fresh next time\ndocker-compose down -v\n

This frees up system resources and stops all the background processes.

","contentEncoded","freeContentEncoded","textContent","What We’re Building TodayToday marks a critical milestone in our distributed systems journey. We’re integrating all the components we’ve built over the past week into a complete, production-ready log processing pipeline that runs on a single machine but follows distributed system principles. By the end of this lesson, you’ll have:• A multi-service Spring Boot system with log producers, consumers, and a query API gateway • Event-driven architecture using Apache Kafka for reliable message delivery and decoupling • Persistent storage layer with PostgreSQL for long-term log retention and Redis for high-performance caching • Production observability with Prometheus metrics, distributed tracing, and Grafana dashboardsWhy This Foundation MattersWhat we’re building today isn’t just a local system—it’s a scaled-down version of the same architectural patterns that power Netflix’s real-time analytics, Uber’s event streaming platform, and Amazon’s CloudWatch logs. Every design decision we make reflects the same trade-offs these companies face at massive scale.The critical insight here is understanding that distributed systems aren’t about having multiple machines—they’re about designing with distribution principles from day one. When Netflix processes billions of log events daily, they use the exact same event sourcing patterns, message queue semantics, and observability strategies we’ll implement today. The only difference is scale.By building these patterns locally first, you’ll understand the fundamental challenges: message ordering guarantees, failure recovery, backpressure handling, and data consistency models. These challenges don’t change whether you’re processing 1,000 events per second or 1 million—only your infrastructure capacity changes.System Design Deep Dive: Core Distributed PatternsEvent Sourcing and Message Queuing ArchitectureOur system implements event sourcing through Kafka, where every log event becomes an immutable fact in our system’s history. This isn’t just message passing—it’s a fundamental shift from request-response to event-driven thinking.The key trade-off here is consistency versus availability. By using Kafka’s partitioning strategy, we guarantee message ordering within partitions but accept eventual consistency across the system. This is the same decision Uber made for their real-time data platform: they prioritized availability and partition tolerance over immediate consistency, accepting that different services might have slightly different views of the world at any given moment.Anti-pattern warning: Don’t fall into the trap of using Kafka like a traditional message queue with immediate acknowledgments. The power comes from treating it as a distributed log where consumers can replay events and multiple subscribers can process the same stream independently.Cache-Aside Pattern with RedisWe implement cache-aside caching where our API gateway checks Redis first, then falls back to PostgreSQL for cache misses. This pattern handles the classic trade-off between read latency and data freshness.The architectural insight here is understanding cache invalidation strategies. We use time-based expiration (TTL) for simplicity, but production systems like Amazon’s DynamoDB Accelerator (DAX) implement sophisticated invalidation based on write-through patterns and event triggers.Scalability bottleneck: Redis becomes a single point of failure in this design. At Netflix scale, this evolves into Redis clusters with consistent hashing, but for our local system, we accept this limitation while building the foundational patterns correctly.Circuit Breaker and Bulkhead PatternsUsing Resilience4j, we implement circuit breakers around external service calls and bulkhead patterns to isolate different types of requests. This prevents cascade failures where one slow database query doesn’t bring down your entire log ingestion pipeline.The trade-off is between system resilience and operational complexity. Circuit breakers add monitoring overhead and require careful tuning, but they’re essential for systems that need to handle partial failures gracefully. This is exactly how Spotify prevents their recommendation service outages from affecting music playback.Observable Systems ArchitectureWe instrument everything with Micrometer metrics and distributed tracing. This isn’t just monitoring—it’s designing for debuggability at scale. Every service exposes health checks, custom metrics, and trace context propagation.Production consideration: Observability has performance costs. Each metric collection and trace span adds latency. We carefully choose what to measure, focusing on RED metrics (Rate, Errors, Duration) rather than monitoring everything.Database Connection Pool ManagementOur system uses HikariCP connection pooling with carefully tuned parameters. This addresses the fundamental challenge of database connections as a finite resource that must be shared across concurrent requests.The key insight is that connection pools are your first line of defense against database overload. The configuration we use (maximum 20 connections, 30-second timeout) reflects production patterns where you’d rather fail fast with connection timeouts than overwhelm your database with too many concurrent queries.Implementation WalkthroughService Architecture DecisionsWe structure our system as three distinct Spring Boot services:Log Producer Service: Exposes REST endpoints for log ingestion and publishes to Kafka. The architectural decision here is to separate ingestion from processing, allowing us to handle traffic spikes by buffering events in Kafka rather than overwhelming downstream services.Log Consumer Service: Subscribes to Kafka topics and persists logs to PostgreSQL. We implement idempotent processing using log event IDs, ensuring that message redelivery doesn’t create duplicate records. This reflects the same exactly-once semantics that Apache Flink uses for stream processing.API Gateway Service: Provides query endpoints with Redis caching. The gateway pattern centralizes cross-cutting concerns like authentication, rate limiting, and response caching. This is the same pattern Amazon uses for API Gateway, though our implementation is obviously simpler.[Image placeholder: Service interaction diagram showing request/response flow between services]Message Processing StrategyWe configure Kafka consumers with manual acknowledgment, giving us control over when messages are considered processed. This prevents message loss during service restarts and allows for sophisticated error handling strategies.The consumer group configuration ensures that multiple consumer instances can process messages in parallel while maintaining ordering guarantees within partitions. This is crucial for scaling—you can add more consumer instances without changing your application code.Database Design PhilosophyOur PostgreSQL schema includes proper indexing for time-range queries, which are the most common pattern for log systems. We also implement connection pooling and prepared statements for optimal performance.Performance insight: We use JSONB for storing log metadata, taking advantage of PostgreSQL’s native JSON indexing. This gives us the flexibility of document storage with the performance of relational queries—exactly the approach Discord uses for storing billions of messages.Production ConsiderationsPerformance OptimizationThe system includes JVM tuning parameters optimized for throughput over latency. We configure G1GC for consistent performance under varying load conditions. Kafka producers use batching and compression to maximize throughput while minimizing network overhead.Monitoring strategy: We track JVM metrics, Kafka lag, database connection pool utilization, and Redis hit rates. These four metrics tell you everything you need to know about system health in production.Failure Scenarios and RecoveryWe implement graceful degradation where Redis failures don’t break the API—queries just become slower as they hit PostgreSQL directly. Kafka failures are handled by Spring Retry with exponential backoff, preventing overwhelming the broker during recovery.The critical insight is that systems fail in partial ways. Netflix’s chaos engineering approach teaches us to build systems that continue operating even when individual components fail. Our circuit breaker configuration reflects this philosophy.Deployment and Scaling StrategiesThe Docker Compose configuration includes resource limits and restart policies that reflect production deployment practices. We use health checks to ensure services are genuinely ready before receiving traffic.Capacity planning: The system is configured to handle approximately 1,000 events per second on standard development hardware. The bottleneck is typically PostgreSQL writes, which is exactly what you’d expect in a log processing system.Scale Connection: Enterprise PatternsThis exact architecture pattern appears in every major tech company, scaled up with additional infrastructure:Netflix: Their event streaming platform processes billions of events using the same Kafka partitioning strategy and consumer group patterns we implementUber: Their real-time data infrastructure uses identical circuit breaker and bulkhead patterns for handling partial failures across thousands of microservicesAmazon: CloudWatch Logs uses the same cache-aside pattern with ElastiCache (Redis) for high-performance log queriesThe architectural principles don’t change at scale—only the infrastructure capacity and operational complexity increase.Next StepsTomorrow we’ll break our single-machine constraint and implement TCP-based log collection, introducing network partition handling and service discovery patterns. This builds naturally on today’s foundation by extending our event-driven architecture across machine boundaries.The key insight you should take from today: distributed systems design isn’t about having multiple machines—it’s about designing with distribution principles that make your system resilient, observable, and scalable from day one.Youtube Video:Implementation GuideGithub Link:https://github.com/sysdr/sdc-java/tree/main/day7/distributed-log-processorSetting Up Your Development EnvironmentBefore we start building, let’s make sure you have everything you need installed on your computer.Required SoftwareJava Development Kit 17+java -version # Should show version 17 or higher Maven 3.8+mvn -version # Should show version 3.8 or higher Docker and Docker Composedocker --version docker-compose --version # Both should be installed and running If you’re missing any of these, install them before continuing. Docker is especially important since it will run our databases and message queues.Building the Complete SystemStep 1: Generate the Project StructureRun the system generator script we created:chmod +x generate_system_files.sh ./generate_system_files.sh This creates a complete project with three Spring Boot services and all the configuration files we need. You should see a new folder called distributed-log-processor with everything inside.[Image placeholder: Terminal screenshot showing successful project generation]Step 2: Start the Infrastructure ServicesNavigate to your project folder and start the supporting services:cd distributed-log-processor ./setup.sh This command starts:PostgreSQL database (port 5432)Apache Kafka with Zookeeper (port 9092)Redis cache (port 6379)Prometheus monitoring (port 9090)Grafana dashboards (port 3000)Wait for the setup script to complete. It will check that all services are healthy before finishing.Step 3: Build the Java ApplicationsNow let’s compile our three Spring Boot services:# Build all services at once mvn clean package -DskipTests # This might take a few minutes the first time # Maven will download all the dependencies we need You should see “BUILD SUCCESS” for each of the three services: log-producer, log-consumer, and api-gateway.Step 4: Start the ServicesOpen three separate terminal windows and start each service:Terminal 1 - Log Producer:cd log-producer mvn spring-boot:run Terminal 2 - Log Consumer:cd log-consumer mvn spring-boot:run Terminal 3 - API Gateway:cd api-gateway mvn spring-boot:run Each service will take about 30-60 seconds to start. You’ll know they’re ready when you see “Started [ServiceName]Application” in the logs.[Image placeholder: Three terminal windows showing services starting up]Testing Your SystemHealth Check TestLet’s verify everything is working by checking the health of each service:# Check Log Producer (port 8081) curl http://localhost:8081/producer/api/logs/health # Check Log Consumer (port 8082) curl http://localhost:8082/actuator/health # Check API Gateway (port 8080) curl http://localhost:8080/gateway/api/query/health Each command should return a JSON response with “status”: “healthy”.Send Your First Log EventLet’s send a log message to our system:curl -X POST http://localhost:8081/producer/api/logs/ingest \\ -H “Content-Type: application/json” \\ -d ‘{ “level”: “INFO”, “message”: “Hello from our distributed log system!”, “source”: “test-application”, “metadata”: { “userId”: “student123”, “action”: “first_test” } }’ You should get back a response like:{ “status”: “accepted”, “logId”: “some-unique-id”, “message”: “Log event successfully ingested” } Query Your Log BackWait about 5 seconds for the message to be processed, then query it back:curl “http://localhost:8080/gateway/api/query/logs?source=test-application” You should see your log message in the response, along with timestamps and the metadata you sent.Run the Full Integration TestWe’ve provided a complete test suite that verifies all the functionality:./integration-tests/run-tests.sh This script tests:Health checks for all servicesSingle log ingestionBatch log ingestionLog querying and searchStatistics generationAll tests should pass with green checkmarks.Performance TestingLoad TestLet’s see how much load our system can handle:./load-test.sh This sends 600 log messages over 60 seconds (10 per second) and measures response times. Watch the terminal output to see how your system performs under load.Monitor PerformanceWhile the load test is running, open Grafana in your web browser:http://localhost:3000 Login with username admin and password admin. You’ll see dashboards showing:Requests per secondResponse timesError ratesDatabase and cache performance[Image placeholder: Grafana dashboard showing live metrics during load test]Understanding the Data FlowLet’s trace what happens when you send a log message:HTTP Request → Your curl command sends JSON to the Log ProducerKafka Publishing → Producer converts JSON to Kafka message and sends to log-events topicMessage Consumption → Log Consumer receives message from KafkaDatabase Write → Consumer saves log to PostgreSQL with proper indexingQuery Path → API Gateway checks Redis cache first, then PostgreSQL if neededMonitoring → All steps generate metrics that Prometheus collectsViewing the Raw DataYou can see your logs directly in the database:# Connect to PostgreSQL docker exec -it postgres psql -U loguser -d logprocessor # Query your logs SELECT id, level, message, source, timestamp FROM log_events ORDER BY timestamp DESC LIMIT 10; # Exit the database \\q Checking Kafka TopicsYou can also see the messages in Kafka:# List Kafka topics docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list # See recent messages (press Ctrl+C to stop) docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic log-events --from-beginning Exploring Advanced FeaturesCircuit Breaker TestingLet’s trigger the circuit breaker by stopping the database:# Stop PostgreSQL docker stop postgres # Try to send logs - should get circuit breaker response curl -X POST http://localhost:8081/producer/api/logs/ingest \\ -H “Content-Type: application/json” \\ -d ‘{”level”: “ERROR”, “message”: “Test during outage”, “source”: “test”}’ # Restart PostgreSQL docker start postgres The system should return a “service unavailable” response when the database is down, demonstrating our resilience patterns.Cache PerformanceLet’s test the cache-aside pattern:# First query (cache miss) - will be slower time curl “http://localhost:8080/gateway/api/query/stats” # Second query (cache hit) - much faster time curl “http://localhost:8080/gateway/api/query/stats” The second request should be noticeably faster because it’s served from Redis.Batch ProcessingTest sending multiple logs at once:curl -X POST http://localhost:8081/producer/api/logs/batch \\ -H “Content-Type: application/json” \\ -d ‘[ {”level”: “INFO”, “message”: “Batch message 1”, “source”: “batch-test”}, {”level”: “WARN”, “message”: “Batch message 2”, “source”: “batch-test”}, {”level”: “ERROR”, “message”: “Batch message 3”, “source”: “batch-test”} ]’ This is much more efficient than sending logs one at a time.Monitoring and DebuggingKey Metrics to WatchIn Grafana, focus on these important metrics:Throughput Metrics:log_ingestion_total - How many logs per second we’re processingkafka_messages_sent_total - Messages successfully sent to Kafkakafka_messages_consumed_total - Messages processed by consumersPerformance Metrics:log_ingestion_duration - How long it takes to process each loghttp_server_requests - API response timeslog_processing_duration - Consumer processing timeHealth Metrics:Circuit breaker states (open/closed/half-open)Database connection pool usageRedis hit/miss ratesJVM memory and garbage collectionCommon Issues and Solutions“Connection refused” errors: Make sure Docker containers are running with docker-compose psSlow queries: Check the database connection pool isn’t exhausted in the metricsHigh memory usage: Monitor JVM heap size and consider increasing it with -Xmx4gKafka lag: If consumers fall behind, increase the number of consumer instancesReal-World ApplicationsThe system you’ve built demonstrates the same patterns used by:Netflix: Their event streaming processes billions of user interactions daily using identical Kafka partitioning and consumer group patterns.Uber: Their dispatch system uses the same circuit breaker patterns to handle partial failures across thousands of services.Discord: Their message storage uses PostgreSQL with JSONB for metadata, just like our log system.Amazon CloudWatch: Uses cache-aside patterns with Redis for high-performance log queries.The only differences at scale are:More machines and larger clustersAdditional monitoring and alertingMore sophisticated deployment automationHigher performance requirementsBut the fundamental architectural patterns remain the same.[Image placeholder: Comparison diagram showing local system vs enterprise scale]What You’ve AccomplishedCongratulations! You’ve built a production-ready distributed log processing system that includes:Event-driven architecture with proper message orderingResilience patterns that handle failures gracefullyHigh-performance caching with RedisComprehensive monitoring with Prometheus and GrafanaHorizontal scaling capabilities built-in from day oneThis foundation prepares you for tomorrow’s lesson where we’ll extend this system across network boundaries, introducing service discovery and network partition handling.The most important lesson: distributed systems aren’t about having multiple machines—they’re about designing with distribution principles that make systems resilient, observable, and scalable from the start.CleanupWhen you’re done experimenting, stop all services:# Stop Spring Boot services (Ctrl+C in each terminal) # Stop Docker infrastructure docker-compose down # Remove volumes if you want to start fresh next time docker-compose down -v This frees up system resources and stops all the background processes.","published","strategy","content-ready","locked","deletedAt","isSyncedFromOwnBlog","original","duplicatesCount",0,"duplicateVersion","originalArticleId","errorsCount","appearance","color","backgroundColor","shortSummary","summary","bullets","persons","places","words","sentiment","subject","meta","eventId","e8d4cd0aafa2fdcbba6b64be9db535cdcbe306dc6de3b8821dc7e4fec3d6aed3","nostrRootEvent","event",{"_25":170,"_198":223,"_200":224,"_103":225,"_203":31,"_139":226,"_205":227},"authorNPub","markdown","What We’re Building Today\n-------------------------\n\n> Today marks a critical milestone in our distributed systems journey. We’re integrating all the components we’ve built over the past week into a complete, production-ready log processing pipeline that runs on a single machine but follows distributed system principles. By the end of this lesson, you’ll have:\n\n> • **A multi-service Spring Boot system** with log producers, consumers, and a query API gateway • **Event-driven architecture** using Apache Kafka for reliable message delivery and decoupling • **Persistent storage layer** with PostgreSQL for long-term log retention and Redis for high-performance caching • **Production observability** with Prometheus metrics, distributed tracing, and Grafana dashboards\n\n[\n\n](https://substackcdn.com/image/fetch/$s_!1-cL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16e359f1-1a55-4c49-b3d1-2d40dfaf9f39_965x610.png)\n\nWhy This Foundation Matters\n---------------------------\n\nWhat we’re building today isn’t just a local system—it’s a scaled-down version of the same architectural patterns that power Netflix’s real-time analytics, Uber’s event streaming platform, and Amazon’s CloudWatch logs. Every design decision we make reflects the same trade-offs these companies face at massive scale.\n\nThe critical insight here is understanding that distributed systems aren’t about having multiple machines—they’re about designing with distribution principles from day one. When Netflix processes billions of log events daily, they use the exact same event sourcing patterns, message queue semantics, and observability strategies we’ll implement today. The only difference is scale.\n\nBy building these patterns locally first, you’ll understand the fundamental challenges: message ordering guarantees, failure recovery, backpressure handling, and data consistency models. These challenges don’t change whether you’re processing 1,000 events per second or 1 million—only your infrastructure capacity changes.\n\nSystem Design Deep Dive: Core Distributed Patterns\n--------------------------------------------------\n\n### Event Sourcing and Message Queuing Architecture\n\nOur system implements event sourcing through Kafka, where every log event becomes an immutable fact in our system’s history. This isn’t just message passing—it’s a fundamental shift from request-response to event-driven thinking.\n\nThe key trade-off here is **consistency versus availability**. By using Kafka’s partitioning strategy, we guarantee message ordering within partitions but accept eventual consistency across the system. This is the same decision Uber made for their real-time data platform: they prioritized availability and partition tolerance over immediate consistency, accepting that different services might have slightly different views of the world at any given moment.\n\n**Anti-pattern warning**: Don’t fall into the trap of using Kafka like a traditional message queue with immediate acknowledgments. The power comes from treating it as a distributed log where consumers can replay events and multiple subscribers can process the same stream independently.\n\n### Cache-Aside Pattern with Redis\n\nWe implement cache-aside caching where our API gateway checks Redis first, then falls back to PostgreSQL for cache misses. This pattern handles the classic trade-off between read latency and data freshness.\n\nThe architectural insight here is understanding cache invalidation strategies. We use time-based expiration (TTL) for simplicity, but production systems like Amazon’s DynamoDB Accelerator (DAX) implement sophisticated invalidation based on write-through patterns and event triggers.\n\n**Scalability bottleneck**: Redis becomes a single point of failure in this design. At Netflix scale, this evolves into Redis clusters with consistent hashing, but for our local system, we accept this limitation while building the foundational patterns correctly.\n\n### Circuit Breaker and Bulkhead Patterns\n\nUsing Resilience4j, we implement circuit breakers around external service calls and bulkhead patterns to isolate different types of requests. This prevents cascade failures where one slow database query doesn’t bring down your entire log ingestion pipeline.\n\nThe trade-off is between system resilience and operational complexity. Circuit breakers add monitoring overhead and require careful tuning, but they’re essential for systems that need to handle partial failures gracefully. This is exactly how Spotify prevents their recommendation service outages from affecting music playback.\n\n### Observable Systems Architecture\n\nWe instrument everything with Micrometer metrics and distributed tracing. This isn’t just monitoring—it’s designing for debuggability at scale. Every service exposes health checks, custom metrics, and trace context propagation.\n\n**Production consideration**: Observability has performance costs. Each metric collection and trace span adds latency. We carefully choose what to measure, focusing on RED metrics (Rate, Errors, Duration) rather than monitoring everything.\n\n### Database Connection Pool Management\n\nOur system uses HikariCP connection pooling with carefully tuned parameters. This addresses the fundamental challenge of database connections as a finite resource that must be shared across concurrent requests.\n\nThe key insight is that connection pools are your first line of defense against database overload. The configuration we use (maximum 20 connections, 30-second timeout) reflects production patterns where you’d rather fail fast with connection timeouts than overwhelm your database with too many concurrent queries.\n\nImplementation Walkthrough\n--------------------------\n\n### Service Architecture Decisions\n\nWe structure our system as three distinct Spring Boot services:\n\n**Log Producer Service**: Exposes REST endpoints for log ingestion and publishes to Kafka. The architectural decision here is to separate ingestion from processing, allowing us to handle traffic spikes by buffering events in Kafka rather than overwhelming downstream services.\n\n**Log Consumer Service**: Subscribes to Kafka topics and persists logs to PostgreSQL. We implement idempotent processing using log event IDs, ensuring that message redelivery doesn’t create duplicate records. This reflects the same exactly-once semantics that Apache Flink uses for stream processing.\n\n**API Gateway Service**: Provides query endpoints with Redis caching. The gateway pattern centralizes cross-cutting concerns like authentication, rate limiting, and response caching. This is the same pattern Amazon uses for API Gateway, though our implementation is obviously simpler.\n\n_\\[Image placeholder: Service interaction diagram showing request/response flow between services\\]_\n\n### Message Processing Strategy\n\nWe configure Kafka consumers with manual acknowledgment, giving us control over when messages are considered processed. This prevents message loss during service restarts and allows for sophisticated error handling strategies.\n\nThe consumer group configuration ensures that multiple consumer instances can process messages in parallel while maintaining ordering guarantees within partitions. This is crucial for scaling—you can add more consumer instances without changing your application code.\n\n### Database Design Philosophy\n\nOur PostgreSQL schema includes proper indexing for time-range queries, which are the most common pattern for log systems. We also implement connection pooling and prepared statements for optimal performance.\n\n**Performance insight**: We use JSONB for storing log metadata, taking advantage of PostgreSQL’s native JSON indexing. This gives us the flexibility of document storage with the performance of relational queries—exactly the approach Discord uses for storing billions of messages.\n\nProduction Considerations\n-------------------------\n\n### Performance Optimization\n\nThe system includes JVM tuning parameters optimized for throughput over latency. We configure G1GC for consistent performance under varying load conditions. Kafka producers use batching and compression to maximize throughput while minimizing network overhead.\n\n**Monitoring strategy**: We track JVM metrics, Kafka lag, database connection pool utilization, and Redis hit rates. These four metrics tell you everything you need to know about system health in production.\n\n### Failure Scenarios and Recovery\n\nWe implement graceful degradation where Redis failures don’t break the API—queries just become slower as they hit PostgreSQL directly. Kafka failures are handled by Spring Retry with exponential backoff, preventing overwhelming the broker during recovery.\n\nThe critical insight is that systems fail in partial ways. Netflix’s chaos engineering approach teaches us to build systems that continue operating even when individual components fail. Our circuit breaker configuration reflects this philosophy.\n\n### Deployment and Scaling Strategies\n\nThe Docker Compose configuration includes resource limits and restart policies that reflect production deployment practices. We use health checks to ensure services are genuinely ready before receiving traffic.\n\n**Capacity planning**: The system is configured to handle approximately 1,000 events per second on standard development hardware. The bottleneck is typically PostgreSQL writes, which is exactly what you’d expect in a log processing system.\n\nScale Connection: Enterprise Patterns\n-------------------------------------\n\nThis exact architecture pattern appears in every major tech company, scaled up with additional infrastructure:\n\n* **Netflix**: Their event streaming platform processes billions of events using the same Kafka partitioning strategy and consumer group patterns we implement\n \n* **Uber**: Their real-time data infrastructure uses identical circuit breaker and bulkhead patterns for handling partial failures across thousands of microservices\n \n* **Amazon**: CloudWatch Logs uses the same cache-aside pattern with ElastiCache (Redis) for high-performance log queries\n \n\nThe architectural principles don’t change at scale—only the infrastructure capacity and operational complexity increase.\n\nNext Steps\n----------\n\nTomorrow we’ll break our single-machine constraint and implement TCP-based log collection, introducing network partition handling and service discovery patterns. This builds naturally on today’s foundation by extending our event-driven architecture across machine boundaries.\n\nThe key insight you should take from today: distributed systems design isn’t about having multiple machines—it’s about designing with distribution principles that make your system resilient, observable, and scalable from day one.\n\nYoutube Video:\n--------------\n\n* * *\n\nImplementation Guide\n====================\n\n### Github Link:\n\n https://github.com/sysdr/sdc-java/tree/main/day7/distributed-log-processor\n\nSetting Up Your Development Environment\n---------------------------------------\n\nBefore we start building, let’s make sure you have everything you need installed on your computer.\n\n### Required Software\n\n**Java Development Kit 17+**\n\n java -version\n # Should show version 17 or higher\n \n\n**Maven 3.8+**\n\n mvn -version\n # Should show version 3.8 or higher\n \n\n**Docker and Docker Compose**\n\n docker --version\n docker-compose --version\n # Both should be installed and running\n \n\nIf you’re missing any of these, install them before continuing. Docker is especially important since it will run our databases and message queues.\n\nBuilding the Complete System\n----------------------------\n\n### Step 1: Generate the Project Structure\n\nRun the system generator script we created:\n\n chmod +x generate_system_files.sh\n ./generate_system_files.sh\n \n\nThis creates a complete project with three Spring Boot services and all the configuration files we need. You should see a new folder called `distributed-log-processor` with everything inside.\n\n_\\[Image placeholder: Terminal screenshot showing successful project generation\\]_\n\n### Step 2: Start the Infrastructure Services\n\nNavigate to your project folder and start the supporting services:\n\n cd distributed-log-processor\n ./setup.sh\n \n\nThis command starts:\n\n* **PostgreSQL database** (port 5432)\n \n* **Apache Kafka** with Zookeeper (port 9092)\n \n* **Redis cache** (port 6379)\n \n* **Prometheus monitoring** (port 9090)\n \n* **Grafana dashboards** (port 3000)\n \n\nWait for the setup script to complete. It will check that all services are healthy before finishing.\n\n### Step 3: Build the Java Applications\n\nNow let’s compile our three Spring Boot services:\n\n # Build all services at once\n mvn clean package -DskipTests\n \n # This might take a few minutes the first time\n # Maven will download all the dependencies we need\n \n\nYou should see “BUILD SUCCESS” for each of the three services: log-producer, log-consumer, and api-gateway.\n\n### Step 4: Start the Services\n\nOpen three separate terminal windows and start each service:\n\n**Terminal 1 - Log Producer:**\n\n cd log-producer\n mvn spring-boot:run\n \n\n**Terminal 2 - Log Consumer:**\n\n cd log-consumer \n mvn spring-boot:run\n \n\n**Terminal 3 - API Gateway:**\n\n cd api-gateway\n mvn spring-boot:run\n \n\nEach service will take about 30-60 seconds to start. You’ll know they’re ready when you see “Started \\[ServiceName\\]Application” in the logs.\n\n_\\[Image placeholder: Three terminal windows showing services starting up\\]_\n\nTesting Your System\n-------------------\n\n### Health Check Test\n\nLet’s verify everything is working by checking the health of each service:\n\n # Check Log Producer (port 8081)\n curl http://localhost:8081/producer/api/logs/health\n \n # Check Log Consumer (port 8082) \n curl http://localhost:8082/actuator/health\n \n # Check API Gateway (port 8080)\n curl http://localhost:8080/gateway/api/query/health\n \n\nEach command should return a JSON response with “status”: “healthy”.\n\n### Send Your First Log Event\n\nLet’s send a log message to our system:\n\n curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n -H “Content-Type: application/json” \\\n -d ‘{\n “level”: “INFO”,\n “message”: “Hello from our distributed log system!”,\n “source”: “test-application”,\n “metadata”: {\n “userId”: “student123”,\n “action”: “first_test”\n }\n }’\n \n\nYou should get back a response like:\n\n {\n “status”: “accepted”,\n “logId”: “some-unique-id”,\n “message”: “Log event successfully ingested”\n }\n \n\n### Query Your Log Back\n\nWait about 5 seconds for the message to be processed, then query it back:\n\n curl “http://localhost:8080/gateway/api/query/logs?source=test-application”\n \n\nYou should see your log message in the response, along with timestamps and the metadata you sent.\n\n### Run the Full Integration Test\n\nWe’ve provided a complete test suite that verifies all the functionality:\n\n ./integration-tests/run-tests.sh\n \n\nThis script tests:\n\n* Health checks for all services\n \n* Single log ingestion\n \n* Batch log ingestion\n \n* Log querying and search\n \n* Statistics generation\n \n\nAll tests should pass with green checkmarks.\n\nPerformance Testing\n-------------------\n\n### Load Test\n\nLet’s see how much load our system can handle:\n\n ./load-test.sh\n \n\nThis sends 600 log messages over 60 seconds (10 per second) and measures response times. Watch the terminal output to see how your system performs under load.\n\n### Monitor Performance\n\nWhile the load test is running, open Grafana in your web browser:\n\n http://localhost:3000\n \n\nLogin with username `admin` and password `admin`. You’ll see dashboards showing:\n\n* Requests per second\n \n* Response times\n \n* Error rates\n \n* Database and cache performance\n \n\n_\\[Image placeholder: Grafana dashboard showing live metrics during load test\\]_\n\nUnderstanding the Data Flow\n---------------------------\n\nLet’s trace what happens when you send a log message:\n\n1. **HTTP Request** → Your curl command sends JSON to the Log Producer\n \n2. **Kafka Publishing** → Producer converts JSON to Kafka message and sends to `log-events` topic\n \n3. **Message Consumption** → Log Consumer receives message from Kafka\n \n4. **Database Write** → Consumer saves log to PostgreSQL with proper indexing\n \n5. **Query Path** → API Gateway checks Redis cache first, then PostgreSQL if needed\n \n6. **Monitoring** → All steps generate metrics that Prometheus collects\n \n\n### Viewing the Raw Data\n\nYou can see your logs directly in the database:\n\n # Connect to PostgreSQL \n docker exec -it postgres psql -U loguser -d logprocessor\n \n # Query your logs\n SELECT id, level, message, source, timestamp FROM log_events ORDER BY timestamp DESC LIMIT 10;\n \n # Exit the database\n \\q\n \n\n### Checking Kafka Topics\n\nYou can also see the messages in Kafka:\n\n # List Kafka topics\n docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list\n \n # See recent messages (press Ctrl+C to stop)\n docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic log-events --from-beginning\n \n\nExploring Advanced Features\n---------------------------\n\n### Circuit Breaker Testing\n\nLet’s trigger the circuit breaker by stopping the database:\n\n # Stop PostgreSQL\n docker stop postgres\n \n # Try to send logs - should get circuit breaker response\n curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n -H “Content-Type: application/json” \\\n -d ‘{”level”: “ERROR”, “message”: “Test during outage”, “source”: “test”}’\n \n # Restart PostgreSQL \n docker start postgres\n \n\nThe system should return a “service unavailable” response when the database is down, demonstrating our resilience patterns.\n\n### Cache Performance\n\nLet’s test the cache-aside pattern:\n\n # First query (cache miss) - will be slower\n time curl “http://localhost:8080/gateway/api/query/stats”\n \n # Second query (cache hit) - much faster \n time curl “http://localhost:8080/gateway/api/query/stats”\n \n\nThe second request should be noticeably faster because it’s served from Redis.\n\n### Batch Processing\n\nTest sending multiple logs at once:\n\n curl -X POST http://localhost:8081/producer/api/logs/batch \\\n -H “Content-Type: application/json” \\\n -d ‘[\n {”level”: “INFO”, “message”: “Batch message 1”, “source”: “batch-test”},\n {”level”: “WARN”, “message”: “Batch message 2”, “source”: “batch-test”}, \n {”level”: “ERROR”, “message”: “Batch message 3”, “source”: “batch-test”}\n ]’\n \n\nThis is much more efficient than sending logs one at a time.\n\nMonitoring and Debugging\n------------------------\n\n### Key Metrics to Watch\n\nIn Grafana, focus on these important metrics:\n\n**Throughput Metrics:**\n\n* `log_ingestion_total` - How many logs per second we’re processing\n \n* `kafka_messages_sent_total` - Messages successfully sent to Kafka\n \n* `kafka_messages_consumed_total` - Messages processed by consumers\n \n\n**Performance Metrics:**\n\n* `log_ingestion_duration` - How long it takes to process each log\n \n* `http_server_requests` - API response times\n \n* `log_processing_duration` - Consumer processing time\n \n\n**Health Metrics:**\n\n* Circuit breaker states (open/closed/half-open)\n \n* Database connection pool usage\n \n* Redis hit/miss rates\n \n* JVM memory and garbage collection\n \n\n### Common Issues and Solutions\n\n**“Connection refused” errors:** Make sure Docker containers are running with `docker-compose ps`\n\n**Slow queries:** Check the database connection pool isn’t exhausted in the metrics\n\n**High memory usage:** Monitor JVM heap size and consider increasing it with `-Xmx4g`\n\n**Kafka lag:** If consumers fall behind, increase the number of consumer instances\n\nReal-World Applications\n-----------------------\n\nThe system you’ve built demonstrates the same patterns used by:\n\n**Netflix:** Their event streaming processes billions of user interactions daily using identical Kafka partitioning and consumer group patterns.\n\n**Uber:** Their dispatch system uses the same circuit breaker patterns to handle partial failures across thousands of services.\n\n**Discord:** Their message storage uses PostgreSQL with JSONB for metadata, just like our log system.\n\n**Amazon CloudWatch:** Uses cache-aside patterns with Redis for high-performance log queries.\n\nThe only differences at scale are:\n\n* More machines and larger clusters\n \n* Additional monitoring and alerting\n \n* More sophisticated deployment automation\n \n* Higher performance requirements\n \n\nBut the fundamental architectural patterns remain the same.\n\n_\\[Image placeholder: Comparison diagram showing local system vs enterprise scale\\]_\n\nWhat You’ve Accomplished\n------------------------\n\nCongratulations! You’ve built a production-ready distributed log processing system that includes:\n\n* **Event-driven architecture** with proper message ordering\n \n* **Resilience patterns** that handle failures gracefully\n \n* **High-performance caching** with Redis\n \n* **Comprehensive monitoring** with Prometheus and Grafana\n \n* **Horizontal scaling** capabilities built-in from day one\n \n\nThis foundation prepares you for tomorrow’s lesson where we’ll extend this system across network boundaries, introducing service discovery and network partition handling.\n\nThe most important lesson: distributed systems aren’t about having multiple machines—they’re about designing with distribution principles that make systems resilient, observable, and scalable from the start.\n\nCleanup\n-------\n\nWhen you’re done experimenting, stop all services:\n\n # Stop Spring Boot services (Ctrl+C in each terminal)\n \n # Stop Docker infrastructure\n docker-compose down\n \n # Remove volumes if you want to start fresh next time\n docker-compose down -v\n \n\nThis frees up system resources and stops all the background processes.","freeMarkdown","kind1EventId","kind30023EventId","2f761f18e019edea68c04d26d10c9a7f8ba3df6fc4f1f222b36c52c51cf560bc","kind30023",{"_25":180,"_198":199,"_200":201,"_103":202,"_203":31,"_139":204,"_205":206},"publishedToNostr","htmlContentKind30023","

What We’re Building Today

\n
Today marks a critical milestone in our distributed systems journey. We’re integrating all the components we’ve built over the past week into a complete, production-ready log processing pipeline that runs on a single machine but follows distributed system principles. By the end of this lesson, you’ll have:
\n

\n
• A multi-service Spring Boot system with log producers, consumers, and a query API gateway • Event-driven architecture using Apache Kafka for reliable message delivery and decoupling • Persistent storage layer with PostgreSQL for long-term log retention and Redis for high-performance caching • Production observability with Prometheus metrics, distributed tracing, and Grafana dashboards
\n

[

](https://substackcdn.com/image/fetch/$s_!1-cL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16e359f1-1a55-4c49-b3d1-2d40dfaf9f39_965x610.png)

Why This Foundation Matters

System Design Deep Dive: Core Distributed Patterns

Event Sourcing and Message Queuing Architecture

Cache-Aside Pattern with Redis

Circuit Breaker and Bulkhead Patterns

Observable Systems Architecture

Database Connection Pool Management

Implementation Walkthrough

Service Architecture Decisions

We structure our system as three distinct Spring Boot services:

[Image placeholder: Service interaction diagram showing request/response flow between services]

Message Processing Strategy

Database Design Philosophy

Production Considerations

Performance Optimization

Failure Scenarios and Recovery

Deployment and Scaling Strategies

Scale Connection: Enterprise Patterns

This exact architecture pattern appears in every major tech company, scaled up with additional infrastructure:

\n
Netflix: Their event streaming platform processes billions of events using the same Kafka partitioning strategy and consumer group patterns we implement
\n
\n
Uber: Their real-time data infrastructure uses identical circuit breaker and bulkhead patterns for handling partial failures across thousands of microservices
\n
\n
Amazon: CloudWatch Logs uses the same cache-aside pattern with ElastiCache (Redis) for high-performance log queries
\n

The architectural principles don’t change at scale—only the infrastructure capacity and operational complexity increase.

Next Steps

Youtube Video:

Implementation Guide

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day7/distributed-log-processor\n

Setting Up Your Development Environment

Before we start building, let’s make sure you have everything you need installed on your computer.

Required Software

Java Development Kit 17+

java -version\n# Should show version 17 or higher\n

Maven 3.8+

mvn -version\n# Should show version 3.8 or higher\n

Docker and Docker Compose

docker --version\ndocker-compose --version\n# Both should be installed and running\n

If you’re missing any of these, install them before continuing. Docker is especially important since it will run our databases and message queues.

Building the Complete System

Step 1: Generate the Project Structure

Run the system generator script we created:

chmod +x generate_system_files.sh\n./generate_system_files.sh\n

This creates a complete project with three Spring Boot services and all the configuration files we need. You should see a new folder called distributed-log-processor with everything inside.

[Image placeholder: Terminal screenshot showing successful project generation]

Step 2: Start the Infrastructure Services

Navigate to your project folder and start the supporting services:

cd distributed-log-processor\n./setup.sh\n

This command starts:

\n
PostgreSQL database (port 5432)
\n
\n
Apache Kafka with Zookeeper (port 9092)
\n
\n
Redis cache (port 6379)
\n
\n
Prometheus monitoring (port 9090)
\n
\n
Grafana dashboards (port 3000)
\n

Wait for the setup script to complete. It will check that all services are healthy before finishing.

Step 3: Build the Java Applications

Now let’s compile our three Spring Boot services:

# Build all services at once\nmvn clean package -DskipTests\n\n# This might take a few minutes the first time\n# Maven will download all the dependencies we need\n

You should see “BUILD SUCCESS” for each of the three services: log-producer, log-consumer, and api-gateway.

Step 4: Start the Services

Open three separate terminal windows and start each service:

Terminal 1 - Log Producer:

cd log-producer\nmvn spring-boot:run\n

Terminal 2 - Log Consumer:

cd log-consumer  \nmvn spring-boot:run\n

Terminal 3 - API Gateway:

cd api-gateway\nmvn spring-boot:run\n

Each service will take about 30-60 seconds to start. You’ll know they’re ready when you see “Started [ServiceName]Application” in the logs.

[Image placeholder: Three terminal windows showing services starting up]

Testing Your System

Health Check Test

Let’s verify everything is working by checking the health of each service:

# Check Log Producer (port 8081)\ncurl http://localhost:8081/producer/api/logs/health\n\n# Check Log Consumer (port 8082) \ncurl http://localhost:8082/actuator/health\n\n# Check API Gateway (port 8080)\ncurl http://localhost:8080/gateway/api/query/health\n

Each command should return a JSON response with “status”: “healthy”.

Send Your First Log Event

Let’s send a log message to our system:

curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n  -H “Content-Type: application/json” \\\n  -d ‘{\n    “level”: “INFO”,\n    “message”: “Hello from our distributed log system!”,\n    “source”: “test-application”,\n    “metadata”: {\n      “userId”: “student123”,\n      “action”: “first_test”\n    }\n  }’\n

You should get back a response like:

{\n  “status”: “accepted”,\n  “logId”: “some-unique-id”,\n  “message”: “Log event successfully ingested”\n}\n

Query Your Log Back

Wait about 5 seconds for the message to be processed, then query it back:

curl “http://localhost:8080/gateway/api/query/logs?source=test-application”\n

You should see your log message in the response, along with timestamps and the metadata you sent.

Run the Full Integration Test

We’ve provided a complete test suite that verifies all the functionality:

./integration-tests/run-tests.sh\n

This script tests:

\n
Health checks for all services
\n
\n
Single log ingestion
\n
\n
Batch log ingestion
\n
\n
Log querying and search
\n
\n
Statistics generation
\n

All tests should pass with green checkmarks.

Performance Testing

Load Test

Let’s see how much load our system can handle:

./load-test.sh\n

This sends 600 log messages over 60 seconds (10 per second) and measures response times. Watch the terminal output to see how your system performs under load.

Monitor Performance

While the load test is running, open Grafana in your web browser:

http://localhost:3000\n

\n
Requests per second
\n
\n
Response times
\n
\n
Error rates
\n
\n
Database and cache performance
\n

[Image placeholder: Grafana dashboard showing live metrics during load test]

Understanding the Data Flow

Let’s trace what happens when you send a log message:

\n
HTTP Request → Your curl command sends JSON to the Log Producer
\n
\n
Kafka Publishing → Producer converts JSON to Kafka message and sends to log-events topic
\n
\n
Message Consumption → Log Consumer receives message from Kafka
\n
\n
Database Write → Consumer saves log to PostgreSQL with proper indexing
\n
\n
Query Path → API Gateway checks Redis cache first, then PostgreSQL if needed
\n
\n
Monitoring → All steps generate metrics that Prometheus collects
\n

Viewing the Raw Data

You can see your logs directly in the database:

# Connect to PostgreSQL  \ndocker exec -it postgres psql -U loguser -d logprocessor\n\n# Query your logs\nSELECT id, level, message, source, timestamp FROM log_events ORDER BY timestamp DESC LIMIT 10;\n\n# Exit the database\n\\q\n

Checking Kafka Topics

You can also see the messages in Kafka:

# List Kafka topics\ndocker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list\n\n# See recent messages (press Ctrl+C to stop)\ndocker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic log-events --from-beginning\n

Exploring Advanced Features

Circuit Breaker Testing

Let’s trigger the circuit breaker by stopping the database:

# Stop PostgreSQL\ndocker stop postgres\n\n# Try to send logs - should get circuit breaker response\ncurl -X POST http://localhost:8081/producer/api/logs/ingest \\\n  -H “Content-Type: application/json” \\\n  -d ‘{”level”: “ERROR”, “message”: “Test during outage”, “source”: “test”}’\n\n# Restart PostgreSQL  \ndocker start postgres\n

The system should return a “service unavailable” response when the database is down, demonstrating our resilience patterns.

Cache Performance

Let’s test the cache-aside pattern:

# First query (cache miss) - will be slower\ntime curl “http://localhost:8080/gateway/api/query/stats”\n\n# Second query (cache hit) - much faster  \ntime curl “http://localhost:8080/gateway/api/query/stats”\n

The second request should be noticeably faster because it’s served from Redis.

Batch Processing

Test sending multiple logs at once:

curl -X POST http://localhost:8081/producer/api/logs/batch \\\n  -H “Content-Type: application/json” \\\n  -d ‘[\n    {”level”: “INFO”, “message”: “Batch message 1”, “source”: “batch-test”},\n    {”level”: “WARN”, “message”: “Batch message 2”, “source”: “batch-test”}, \n    {”level”: “ERROR”, “message”: “Batch message 3”, “source”: “batch-test”}\n  ]’\n

This is much more efficient than sending logs one at a time.

Monitoring and Debugging

Key Metrics to Watch

In Grafana, focus on these important metrics:

Throughput Metrics:

\n
log_ingestion_total - How many logs per second we’re processing
\n
\n
kafka_messages_sent_total - Messages successfully sent to Kafka
\n
\n
kafka_messages_consumed_total - Messages processed by consumers
\n

Performance Metrics:

\n
log_ingestion_duration - How long it takes to process each log
\n
\n
http_server_requests - API response times
\n
\n
log_processing_duration - Consumer processing time
\n

Health Metrics:

\n
Circuit breaker states (open/closed/half-open)
\n
\n
Database connection pool usage
\n
\n
Redis hit/miss rates
\n
\n
JVM memory and garbage collection
\n

Common Issues and Solutions

“Connection refused” errors: Make sure Docker containers are running with docker-compose ps

Slow queries: Check the database connection pool isn’t exhausted in the metrics

High memory usage: Monitor JVM heap size and consider increasing it with -Xmx4g

Kafka lag: If consumers fall behind, increase the number of consumer instances

Real-World Applications

The system you’ve built demonstrates the same patterns used by:

Netflix: Their event streaming processes billions of user interactions daily using identical Kafka partitioning and consumer group patterns.

Uber: Their dispatch system uses the same circuit breaker patterns to handle partial failures across thousands of services.

Discord: Their message storage uses PostgreSQL with JSONB for metadata, just like our log system.

Amazon CloudWatch: Uses cache-aside patterns with Redis for high-performance log queries.

The only differences at scale are:

\n
More machines and larger clusters
\n
\n
Additional monitoring and alerting
\n
\n
More sophisticated deployment automation
\n
\n
Higher performance requirements
\n

But the fundamental architectural patterns remain the same.

[Image placeholder: Comparison diagram showing local system vs enterprise scale]

What You’ve Accomplished

Congratulations! You’ve built a production-ready distributed log processing system that includes:

\n
Event-driven architecture with proper message ordering
\n
\n
Resilience patterns that handle failures gracefully
\n
\n
High-performance caching with Redis
\n
\n
Comprehensive monitoring with Prometheus and Grafana
\n
\n
Horizontal scaling capabilities built-in from day one
\n

This foundation prepares you for tomorrow’s lesson where we’ll extend this system across network boundaries, introducing service discovery and network partition handling.

Cleanup

When you’re done experimenting, stop all services:

# Stop Spring Boot services (Ctrl+C in each terminal)\n\n# Stop Docker infrastructure\ndocker-compose down\n\n# Remove volumes if you want to start fresh next time\ndocker-compose down -v\n

This frees up system resources and stops all the background processes.

","pubDate",["D",1760954705000],"modifiedAt",["D",1760955014197],["D",1760955014197],"storyId","likes","zaps","comments","reposts","tweets",[],"sig","5b8178453786a2d89a237256fc041cb37023be91e0ae64071b6e8dd4959f812634dc1724d7cdfd76b76ef39365abc8b64322c25a236ffa96e3206e43bf0cdd9c","kind",30023,[207,208,209,210,211,212,213,214],"pubkey","Day 7: Building Your First End-to-End Log Processing Pipeline\nWhat We’re Building Today\n-------------------------\n\n> Today marks a critical milestone in our distributed systems journey. We’re integrating all the components we’ve built over the past week into a complete, production-ready log processing pipeline that runs on a single machine but follows distributed system principles. By the end of this lesson, you’ll have:\n\n> • **A multi-service Spring Boot system** with log producers, consumers, and a query API gateway • **Event-driven architecture** using Apache Kafka for reliable message delivery and decoupling • **Persistent storage layer** with PostgreSQL for long-term log retention and Redis for high-performance caching • **Production observability** with Prometheus metrics, distributed tracing, and Grafana dashboards\n\n[\n\n](https://substackcdn.com/image/fetch/$s_!1-cL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16e359f1-1a55-4c49-b3d1-2d40dfaf9f39_965x610.png)\n\nWhy This Foundation Matters\n---------------------------\n\nWhat we’re building today isn’t just a local system—it’s a scaled-down version of the same architectural patterns that power Netflix’s real-time analytics, Uber’s event streaming platform, and Amazon’s CloudWatch logs. Every design decision we make reflects the same trade-offs these companies face at massive scale.\n\nThe critical insight here is understanding that distributed systems aren’t about having multiple machines—they’re about designing with distribution principles from day one. When Netflix processes billions of log events daily, they use the exact same event sourcing patterns, message queue semantics, and observability strategies we’ll implement today. The only difference is scale.\n\nBy building these patterns locally first, you’ll understand the fundamental challenges: message ordering guarantees, failure recovery, backpressure handling, and data consistency models. These challenges don’t change whether you’re processing 1,000 events per second or 1 million—only your infrastructure capacity changes.\n\nSystem Design Deep Dive: Core Distributed Patterns\n--------------------------------------------------\n\n### Event Sourcing and Message Queuing Architecture\n\nOur system implements event sourcing through Kafka, where every log event becomes an immutable fact in our system’s history. This isn’t just message passing—it’s a fundamental shift from request-response to event-driven thinking.\n\nThe key trade-off here is **consistency versus availability**. By using Kafka’s partitioning strategy, we guarantee message ordering within partitions but accept eventual consistency across the system. This is the same decision Uber made for their real-time data platform: they prioritized availability and partition tolerance over immediate consistency, accepting that different services might have slightly different views of the world at any given moment.\n\n**Anti-pattern warning**: Don’t fall into the trap of using Kafka like a traditional message queue with immediate acknowledgments. The power comes from treating it as a distributed log where consumers can replay events and multiple subscribers can process the same stream independently.\n\n### Cache-Aside Pattern with Redis\n\nWe implement cache-aside caching where our API gateway checks Redis first, then falls back to PostgreSQL for cache misses. This pattern handles the classic trade-off between read latency and data freshness.\n\nThe architectural insight here is understanding cache invalidation strategies. We use time-based expiration (TTL) for simplicity, but production systems like Amazon’s DynamoDB Accelerator (DAX) implement sophisticated invalidation based on write-through patterns and event triggers.\n\n**Scalability bottleneck**: Redis becomes a single point of failure in this design. At Netflix scale, this evolves into Redis clusters with consistent hashing, but for our local system, we accept this limitation while building the foundational patterns correctly.\n\n### Circuit Breaker and Bulkhead Patterns\n\nUsing Resilience4j, we implement circuit breakers around external service calls and bulkhead patterns to isolate different types of requests. This prevents cascade failures where one slow database query doesn’t bring down your entire log ingestion pipeline.\n\nThe trade-off is between system resilience and operational complexity. Circuit breakers add monitoring overhead and require careful tuning, but they’re essential for systems that need to handle partial failures gracefully. This is exactly how Spotify prevents their recommendation service outages from affecting music playback.\n\n### Observable Systems Architecture\n\nWe instrument everything with Micrometer metrics and distributed tracing. This isn’t just monitoring—it’s designing for debuggability at scale. Every service exposes health checks, custom metrics, and trace context propagation.\n\n**Production consideration**: Observability has performance costs. Each metric collection and trace span adds latency. We carefully choose what to measure, focusing on RED metrics (Rate, Errors, Duration) rather than monitoring everything.\n\n### Database Connection Pool Management\n\nOur system uses HikariCP connection pooling with carefully tuned parameters. This addresses the fundamental challenge of database connections as a finite resource that must be shared across concurrent requests.\n\nThe key insight is that connection pools are your first line of defense against database overload. The configuration we use (maximum 20 connections, 30-second timeout) reflects production patterns where you’d rather fail fast with connection timeouts than overwhelm your database with too many concurrent queries.\n\nImplementation Walkthrough\n--------------------------\n\n### Service Architecture Decisions\n\nWe structure our system as three distinct Spring Boot services:\n\n**Log Producer Service**: Exposes REST endpoints for log ingestion and publishes to Kafka. The architectural decision here is to separate ingestion from processing, allowing us to handle traffic spikes by buffering events in Kafka rather than overwhelming downstream services.\n\n**Log Consumer Service**: Subscribes to Kafka topics and persists logs to PostgreSQL. We implement idempotent processing using log event IDs, ensuring that message redelivery doesn’t create duplicate records. This reflects the same exactly-once semantics that Apache Flink uses for stream processing.\n\n**API Gateway Service**: Provides query endpoints with Redis caching. The gateway pattern centralizes cross-cutting concerns like authentication, rate limiting, and response caching. This is the same pattern Amazon uses for API Gateway, though our implementation is obviously simpler.\n\n_\\[Image placeholder: Service interaction diagram showing request/response flow between services\\]_\n\n### Message Processing Strategy\n\nWe configure Kafka consumers with manual acknowledgment, giving us control over when messages are considered processed. This prevents message loss during service restarts and allows for sophisticated error handling strategies.\n\nThe consumer group configuration ensures that multiple consumer instances can process messages in parallel while maintaining ordering guarantees within partitions. This is crucial for scaling—you can add more consumer instances without changing your application code.\n\n### Database Design Philosophy\n\nOur PostgreSQL schema includes proper indexing for time-range queries, which are the most common pattern for log systems. We also implement connection pooling and prepared statements for optimal performance.\n\n**Performance insight**: We use JSONB for storing log metadata, taking advantage of PostgreSQL’s native JSON indexing. This gives us the flexibility of document storage with the performance of relational queries—exactly the approach Discord uses for storing billions of messages.\n\nProduction Considerations\n-------------------------\n\n### Performance Optimization\n\nThe system includes JVM tuning parameters optimized for throughput over latency. We configure G1GC for consistent performance under varying load conditions. Kafka producers use batching and compression to maximize throughput while minimizing network overhead.\n\n**Monitoring strategy**: We track JVM metrics, Kafka lag, database connection pool utilization, and Redis hit rates. These four metrics tell you everything you need to know about system health in production.\n\n### Failure Scenarios and Recovery\n\nWe implement graceful degradation where Redis failures don’t break the API—queries just become slower as they hit PostgreSQL directly. Kafka failures are handled by Spring Retry with exponential backoff, preventing overwhelming the broker during recovery.\n\nThe critical insight is that systems fail in partial ways. Netflix’s chaos engineering approach teaches us to build systems that continue operating even when individual components fail. Our circuit breaker configuration reflects this philosophy.\n\n### Deployment and Scaling Strategies\n\nThe Docker Compose configuration includes resource limits and restart policies that reflect production deployment practices. We use health checks to ensure services are genuinely ready before receiving traffic.\n\n**Capacity planning**: The system is configured to handle approximately 1,000 events per second on standard development hardware. The bottleneck is typically PostgreSQL writes, which is exactly what you’d expect in a log processing system.\n\nScale Connection: Enterprise Patterns\n-------------------------------------\n\nThis exact architecture pattern appears in every major tech company, scaled up with additional infrastructure:\n\n* **Netflix**: Their event streaming platform processes billions of events using the same Kafka partitioning strategy and consumer group patterns we implement\n \n* **Uber**: Their real-time data infrastructure uses identical circuit breaker and bulkhead patterns for handling partial failures across thousands of microservices\n \n* **Amazon**: CloudWatch Logs uses the same cache-aside pattern with ElastiCache (Redis) for high-performance log queries\n \n\nThe architectural principles don’t change at scale—only the infrastructure capacity and operational complexity increase.\n\nNext Steps\n----------\n\nTomorrow we’ll break our single-machine constraint and implement TCP-based log collection, introducing network partition handling and service discovery patterns. This builds naturally on today’s foundation by extending our event-driven architecture across machine boundaries.\n\nThe key insight you should take from today: distributed systems design isn’t about having multiple machines—it’s about designing with distribution principles that make your system resilient, observable, and scalable from day one.\n\nYoutube Video:\n--------------\n\n* * *\n\nImplementation Guide\n====================\n\n### Github Link:\n\n https://github.com/sysdr/sdc-java/tree/main/day7/distributed-log-processor\n\nSetting Up Your Development Environment\n---------------------------------------\n\nBefore we start building, let’s make sure you have everything you need installed on your computer.\n\n### Required Software\n\n**Java Development Kit 17+**\n\n java -version\n # Should show version 17 or higher\n \n\n**Maven 3.8+**\n\n mvn -version\n # Should show version 3.8 or higher\n \n\n**Docker and Docker Compose**\n\n docker --version\n docker-compose --version\n # Both should be installed and running\n \n\nIf you’re missing any of these, install them before continuing. Docker is especially important since it will run our databases and message queues.\n\nBuilding the Complete System\n----------------------------\n\n### Step 1: Generate the Project Structure\n\nRun the system generator script we created:\n\n chmod +x generate_system_files.sh\n ./generate_system_files.sh\n \n\nThis creates a complete project with three Spring Boot services and all the configuration files we need. You should see a new folder called `distributed-log-processor` with everything inside.\n\n_\\[Image placeholder: Terminal screenshot showing successful project generation\\]_\n\n### Step 2: Start the Infrastructure Services\n\nNavigate to your project folder and start the supporting services:\n\n cd distributed-log-processor\n ./setup.sh\n \n\nThis command starts:\n\n* **PostgreSQL database** (port 5432)\n \n* **Apache Kafka** with Zookeeper (port 9092)\n \n* **Redis cache** (port 6379)\n \n* **Prometheus monitoring** (port 9090)\n \n* **Grafana dashboards** (port 3000)\n \n\nWait for the setup script to complete. It will check that all services are healthy before finishing.\n\n### Step 3: Build the Java Applications\n\nNow let’s compile our three Spring Boot services:\n\n # Build all services at once\n mvn clean package -DskipTests\n \n # This might take a few minutes the first time\n # Maven will download all the dependencies we need\n \n\nYou should see “BUILD SUCCESS” for each of the three services: log-producer, log-consumer, and api-gateway.\n\n### Step 4: Start the Services\n\nOpen three separate terminal windows and start each service:\n\n**Terminal 1 - Log Producer:**\n\n cd log-producer\n mvn spring-boot:run\n \n\n**Terminal 2 - Log Consumer:**\n\n cd log-consumer \n mvn spring-boot:run\n \n\n**Terminal 3 - API Gateway:**\n\n cd api-gateway\n mvn spring-boot:run\n \n\nEach service will take about 30-60 seconds to start. You’ll know they’re ready when you see “Started \\[ServiceName\\]Application” in the logs.\n\n_\\[Image placeholder: Three terminal windows showing services starting up\\]_\n\nTesting Your System\n-------------------\n\n### Health Check Test\n\nLet’s verify everything is working by checking the health of each service:\n\n # Check Log Producer (port 8081)\n curl http://localhost:8081/producer/api/logs/health\n \n # Check Log Consumer (port 8082) \n curl http://localhost:8082/actuator/health\n \n # Check API Gateway (port 8080)\n curl http://localhost:8080/gateway/api/query/health\n \n\nEach command should return a JSON response with “status”: “healthy”.\n\n### Send Your First Log Event\n\nLet’s send a log message to our system:\n\n curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n -H “Content-Type: application/json” \\\n -d ‘{\n “level”: “INFO”,\n “message”: “Hello from our distributed log system!”,\n “source”: “test-application”,\n “metadata”: {\n “userId”: “student123”,\n “action”: “first_test”\n }\n }’\n \n\nYou should get back a response like:\n\n {\n “status”: “accepted”,\n “logId”: “some-unique-id”,\n “message”: “Log event successfully ingested”\n }\n \n\n### Query Your Log Back\n\nWait about 5 seconds for the message to be processed, then query it back:\n\n curl “http://localhost:8080/gateway/api/query/logs?source=test-application”\n \n\nYou should see your log message in the response, along with timestamps and the metadata you sent.\n\n### Run the Full Integration Test\n\nWe’ve provided a complete test suite that verifies all the functionality:\n\n ./integration-tests/run-tests.sh\n \n\nThis script tests:\n\n* Health checks for all services\n \n* Single log ingestion\n \n* Batch log ingestion\n \n* Log querying and search\n \n* Statistics generation\n \n\nAll tests should pass with green checkmarks.\n\nPerformance Testing\n-------------------\n\n### Load Test\n\nLet’s see how much load our system can handle:\n\n ./load-test.sh\n \n\nThis sends 600 log messages over 60 seconds (10 per second) and measures response times. Watch the terminal output to see how your system performs under load.\n\n### Monitor Performance\n\nWhile the load test is running, open Grafana in your web browser:\n\n http://localhost:3000\n \n\nLogin with username `admin` and password `admin`. You’ll see dashboards showing:\n\n* Requests per second\n \n* Response times\n \n* Error rates\n \n* Database and cache performance\n \n\n_\\[Image placeholder: Grafana dashboard showing live metrics during load test\\]_\n\nUnderstanding the Data Flow\n---------------------------\n\nLet’s trace what happens when you send a log message:\n\n1. **HTTP Request** → Your curl command sends JSON to the Log Producer\n \n2. **Kafka Publishing** → Producer converts JSON to Kafka message and sends to `log-events` topic\n \n3. **Message Consumption** → Log Consumer receives message from Kafka\n \n4. **Database Write** → Consumer saves log to PostgreSQL with proper indexing\n \n5. **Query Path** → API Gateway checks Redis cache first, then PostgreSQL if needed\n \n6. **Monitoring** → All steps generate metrics that Prometheus collects\n \n\n### Viewing the Raw Data\n\nYou can see your logs directly in the database:\n\n # Connect to PostgreSQL \n docker exec -it postgres psql -U loguser -d logprocessor\n \n # Query your logs\n SELECT id, level, message, source, timestamp FROM log_events ORDER BY timestamp DESC LIMIT 10;\n \n # Exit the database\n \\q\n \n\n### Checking Kafka Topics\n\nYou can also see the messages in Kafka:\n\n # List Kafka topics\n docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list\n \n # See recent messages (press Ctrl+C to stop)\n docker exec -it kafka kafka-console-consumer --bootstrap-server localhost:9092 --topic log-events --from-beginning\n \n\nExploring Advanced Features\n---------------------------\n\n### Circuit Breaker Testing\n\nLet’s trigger the circuit breaker by stopping the database:\n\n # Stop PostgreSQL\n docker stop postgres\n \n # Try to send logs - should get circuit breaker response\n curl -X POST http://localhost:8081/producer/api/logs/ingest \\\n -H “Content-Type: application/json” \\\n -d ‘{”level”: “ERROR”, “message”: “Test during outage”, “source”: “test”}’\n \n # Restart PostgreSQL \n docker start postgres\n \n\nThe system should return a “service unavailable” response when the database is down, demonstrating our resilience patterns.\n\n### Cache Performance\n\nLet’s test the cache-aside pattern:\n\n # First query (cache miss) - will be slower\n time curl “http://localhost:8080/gateway/api/query/stats”\n \n # Second query (cache hit) - much faster \n time curl “http://localhost:8080/gateway/api/query/stats”\n \n\nThe second request should be noticeably faster because it’s served from Redis.\n\n### Batch Processing\n\nTest sending multiple logs at once:\n\n curl -X POST http://localhost:8081/producer/api/logs/batch \\\n -H “Content-Type: application/json” \\\n -d ‘[\n {”level”: “INFO”, “message”: “Batch message 1”, “source”: “batch-test”},\n {”level”: “WARN”, “message”: “Batch message 2”, “source”: “batch-test”}, \n {”level”: “ERROR”, “message”: “Batch message 3”, “source”: “batch-test”}\n ]’\n \n\nThis is much more efficient than sending logs one at a time.\n\nMonitoring and Debugging\n------------------------\n\n### Key Metrics to Watch\n\nIn Grafana, focus on these important metrics:\n\n**Throughput Metrics:**\n\n* `log_ingestion_total` - How many logs per second we’re processing\n \n* `kafka_messages_sent_total` - Messages successfully sent to Kafka\n \n* `kafka_messages_consumed_total` - Messages processed by consumers\n \n\n**Performance Metrics:**\n\n* `log_ingestion_duration` - How long it takes to process each log\n \n* `http_server_requests` - API response times\n \n* `log_processing_duration` - Consumer processing time\n \n\n**Health Metrics:**\n\n* Circuit breaker states (open/closed/half-open)\n \n* Database connection pool usage\n \n* Redis hit/miss rates\n \n* JVM memory and garbage collection\n \n\n### Common Issues and Solutions\n\n**“Connection refused” errors:** Make sure Docker containers are running with `docker-compose ps`\n\n**Slow queries:** Check the database connection pool isn’t exhausted in the metrics\n\n**High memory usage:** Monitor JVM heap size and consider increasing it with `-Xmx4g`\n\n**Kafka lag:** If consumers fall behind, increase the number of consumer instances\n\nReal-World Applications\n-----------------------\n\nThe system you’ve built demonstrates the same patterns used by:\n\n**Netflix:** Their event streaming processes billions of user interactions daily using identical Kafka partitioning and consumer group patterns.\n\n**Uber:** Their dispatch system uses the same circuit breaker patterns to handle partial failures across thousands of services.\n\n**Discord:** Their message storage uses PostgreSQL with JSONB for metadata, just like our log system.\n\n**Amazon CloudWatch:** Uses cache-aside patterns with Redis for high-performance log queries.\n\nThe only differences at scale are:\n\n* More machines and larger clusters\n \n* Additional monitoring and alerting\n \n* More sophisticated deployment automation\n \n* Higher performance requirements\n \n\nBut the fundamental architectural patterns remain the same.\n\n_\\[Image placeholder: Comparison diagram showing local system vs enterprise scale\\]_\n\nWhat You’ve Accomplished\n------------------------\n\nCongratulations! You’ve built a production-ready distributed log processing system that includes:\n\n* **Event-driven architecture** with proper message ordering\n \n* **Resilience patterns** that handle failures gracefully\n \n* **High-performance caching** with Redis\n \n* **Comprehensive monitoring** with Prometheus and Grafana\n \n* **Horizontal scaling** capabilities built-in from day one\n \n\nThis foundation prepares you for tomorrow’s lesson where we’ll extend this system across network boundaries, introducing service discovery and network partition handling.\n\nThe most important lesson: distributed systems aren’t about having multiple machines—they’re about designing with distribution principles that make systems resilient, observable, and scalable from the start.\n\nCleanup\n-------\n\nWhen you’re done experimenting, stop all services:\n\n # Stop Spring Boot services (Ctrl+C in each terminal)\n \n # Stop Docker infrastructure\n docker-compose down\n \n # Remove volumes if you want to start fresh next time\n docker-compose down -v\n \n\nThis frees up system resources and stops all the background processes.\nYou can include dynamic values by using placeholders like: https://drewdru.syndichain.com/articles/e9f4a4e4-2267-4df4-9b7e-64b609bf9629 , Drew Dru, https://sdcourse.substack.com/p/day-7-building-your-first-end-to , drewdru, drewdru, drewdru, drewdru These will automatically be replaced with the actual data when the message is sent.","created_at",1760955014,[221,222],[36,125],[219,220],[39,131],[218,119],[161,126],[216,217],[215,31,41],"p","client","Layer3.press","d","published_at","1760954705","r","https://drewdru.syndichain.com/articles/e9f4a4e4-2267-4df4-9b7e-64b609bf9629","b6925710cad826144eea85f55ac2b1dd9602e1a7528ba4ccd78502fe603e121926be15fabd3ca9460b2bbe9aec26eabccebe9c987926a3bfb8e9da7994b2a705",1,[228,229,230,231,232],"Day 7: Building Your First End-to-End Log Processing Pipeline\nWhat We’re Building Today\n\n\nnostr:naddr1qvzqqqr4gupzp5fxzjl8yczfclk0ades0f3a8pqchwptjxrxhpmy4ypl728dnx7xqqjx2wtxx3sngef595eryd3h956xge3595ukydm995mrgc3kxqukye3excerjtl5zdq ",1761219875,[167,125],[39,131],[233,234,41,235],[216,217],[215,31,41],"a","30023:d12614be726049c7ecfeb7307a63d38418bb82b91866b8764a903ff28ed99bc6:e9f4a4e4-2267-4df4-9b7e-64b609bf9629","mention","userSubscription","theme","light","currentUser","currentUserVersion2","currentPath","/articles/e9f4a4e4-2267-4df4-9b7e-64b609bf9629","customDomain","defaultLocale",[],"domainPrimaryAccentColor","relays",[289,290,291,292,289],"plans",[267,268],[77,78],"isDomainOwner","userInterfaceLanguage","interfacePreferences","session",{"_100":-5,"_263":12,"_264":265,"_72":245},"locale","sessionNpub","sessionPublicKey","hostname","drewdru.syndichain.com","syndicated","isAdmin","rights",{"_266":12},"owner",{"_25":283,"_270":284,"_38":285,"_273":286,"_275":276,"_277":278,"_279":153,"_280":23,"_47":287,"_49":288},{"_25":269,"_270":271,"_38":272,"_273":274,"_275":276,"_277":278,"_279":153,"_280":23,"_47":281,"_49":282},"3982e6ff-f607-4102-a24c-36c9fd4baf5e","name","Test Plan on Syndichain","TEST Plan description","price","6.00","currency","USD","billingInterval","monthly","trialDays","isActive",["D",1756209805178],["D",1756209805178],"1fd9543f-3ba9-4c83-914a-a4376d4dd8d6","Free plan","Free notifications about new articles","0.00",["D",1758187216959],["D",1758187216959],"wss://articles.layer3.news","wss://nostr-01.yakihonne.com","wss://relay.primal.net/","wss://relay.primal.net"]