Even in one of these domains, larger organizations don’t use a single data warehouse or data lake. The data warehouse and the data lake are not dead but more relevant than ever before in a data-driven world. However, architectural concepts like domain-driven design, microservices, and data mesh show that decentralized ownership is the right choice for modern enterprise architecture : The sky (and your budget) is the limit with current big data and cloud technologies. The traditional data warehouse respectively data lake approach is to ingest all data from all sources into a central storage system for centralized data ownership. Flexibility through decentralization and best-of-breed With these basics about processing events, let’s understand why storing all events in a single, central data lake is not the solution to all problems. Non-critical analytics is usually not real-time: Finding insights in historical data is usually done in a batch process using paradigms like complex SQL queries, map-reduce, or complex algorithms (e.g., reporting model training with machine learning algorithms forecasting).Critical analytics is usually real-time: Critical analytics very often requires real-time processing (e.g., detecting the fraud before it happens predicting a machine failure before it breaks upselling to a customer before he leaves the store).Business transactions are often real-time: A transaction like a payment usually requires real-time processing (e.g., before the customer leaves the store before you ship the item before you leave the ride-hailing car).If you don’t need real-time decisions, batch processing (i.e., after minutes, hours, days) or on-demand (i.e., request-reply) is sufficient. Real-time usually means end-to-end processing within milliseconds or seconds. Example: Reporting and business intelligence to forecast demand. Non-critical analytics : Downtime and data loss are not good but do not kill the whole business.Example: Continuous monitoring of IoT sensor data and a (predictive) machine failure alert. Alerting on the aggregation of events is more critical. Data loss of a single sensor event might be okay. Critical analytics : Ideally, zero downtime.Example: Payments need to be processed exactly once. Business transaction : Ideally, zero downtime and zero data loss.Potential impacts can be increased revenue, reduced risk, reduced cost, or improved customer experience. The criticality of an event defines the outcome. A business process in the real world requires the correlation of various events. Think about the following flow of events across applications, domains, and organizations:Īn event is business information or technical information. Today, nobody questions that data-driven business processes change the world and enable innovation across industries.ĭata-driven business processes require both real-time data processing and batch processing. The last decade offered many articles, blogs, and presentations about data becoming the new oil. Subscribe to my newsletter to get an email after each publication (no spam or ads). I will link the blogs here as soon as they are available (in the next few weeks). Stay tuned for a dedicated blog post for each topic as part of this blog series.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |