Tag: big-data
All the articles with the tag "big-data".
Replacing Database LIKE Queries with Elasticsearch: Approaches and Implementation Details
For a large order dataset, Elasticsearch replaces database LIKE searches on contact information, followed by database lookups for orders. The article covers index design, historical and incremental synchronization, UTC handling, pagination, database fallback, and synchronization gaps.
A Practical Guide to Elasticsearch in Application Development, with a Real Optimization Case
A checklist for using Elasticsearch in application development, covering analyzer validation, field mappings, index planning, lifecycle management, and capacity estimates. A real logging use case illustrates performance problems caused by cleanup and writes, along with improvements using data streams, circuit breakers, and batching.
Building Elastic Stack from the Official Documentation: A Three-Node Elasticsearch Cluster, Kibana, Filebeat, Metricbeat, and Migration Without Downtime
Uses Docker Compose to build Elastic Stack with a three-node Elasticsearch cluster, Kibana, Filebeat, and Metricbeat, explaining TLS and monitoring configuration. Drawing on a production migration, it discusses moving from one node to a cluster through domain switching, data migration, and connection verification.
Kafka (Part 6): Oracle-to-PostgreSQL CDC with Kafka Connect and Debezium, and Cache Consistency
Introduces CDC and synchronizes Oracle data to PostgreSQL through a Kafka Connect and Debezium example. It discusses database synchronization, Elasticsearch imports, and cache consistency, comparing the conditions under which different approaches are suitable.
Kafka (Part 5): Consumer Callbacks, Scheduled Retries, and Rebalancing
This largely completes a messaging system built with Kafka. The key code has been explained across the articles; see the source repository for the complete design and implementation. The next Kafka article is planned to cover synchronizing Oracle data to PostgreSQL with Kafka Connect.
Kafka (Part 3): Sending JSON with a Shared Serializer and Improving Producer Throughput
Implements a Kafka producer that sends JSON messages, using a shared serializer for different objects and configuring asynchronous sending, failure recording, and retries. Based on actual email-message sizes, it discusses how compression, batch size, request size, and waiting time affect throughput.
Kafka (Part 1): A Single-Node KRaft Setup with Docker Compose, Kafka UI, and Prometheus JMX Exporter
Builds a single-node Kafka environment in KRaft mode with Docker Compose and integrates Kafka UI and Prometheus JMX Exporter. It explains listeners, roles, memory, and monitoring configuration, then covers startup verification, JMX port conflicts, and resource-related troubleshooting.
Using Solr (Part 2): Full and Incremental Imports from MySQL
Explains how to configure Solr DataImportHandler for full and incremental imports from MySQL and periodically refresh data with operating-system scheduled jobs. It also covers tinyint fields, multivalued fields, and Transformers, and outlines DIH components and changes in later versions.