{"id":2435,"date":"2020-10-03T13:56:13","date_gmt":"2020-10-03T13:56:13","guid":{"rendered":"https:\/\/merlinsourcing.com\/?p=2435"},"modified":"2026-03-19T12:23:51","modified_gmt":"2026-03-19T12:23:51","slug":"setting-up-a-high-performance-scalable-real-time-data-processing-engine","status":"publish","type":"post","link":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/","title":{"rendered":"Setting up a high-performance scalable real-time data processing engine"},"content":{"rendered":"<div data-elementor-type=\"wp-post\" data-elementor-id=\"2435\" class=\"elementor elementor-2435\" data-elementor-settings=\"[]\">\n\t\t\t\t\t\t\t<div class=\"elementor-section-wrap\">\n\t\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-b5c9706 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"b5c9706\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-widget-wrap elementor-element-populated elementor-element elementor-element-7ee7c708\" data-id=\"7ee7c708\" data-element_type=\"column\">\n\t\t\t\t\t\t\t\t\t\t<div class=\"elementor-element elementor-element-6fe4a722 elementor-widget elementor-widget-text-editor\" data-id=\"6fe4a722\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t<!-- wp:yoast-seo\/breadcrumbs \/--><!-- wp:yoast-seo\/table-of-contents -->\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_61 counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" >Inhaltsverzeichnis<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #000000;color:#000000\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewbox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #000000;color:#000000\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewbox=\"0 0 24 24\" version=\"1.2\" baseprofile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#What_is_the_context\" title=\"What is the context?\">What is the context?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#What_is_the_Problem_Statement\" title=\"What is the Problem Statement?\">What is the Problem Statement?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#What_are_the_options\" title=\"What are the options?\">What are the options?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#How_did_we_compare_and_what_did_we_choose\" title=\"How did we compare and what did we choose?\">How did we compare and what did we choose?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#How_did_we_Integrate_them_with_our_stack\" title=\"How did we Integrate them with our stack?\">How did we Integrate them with our stack?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#What_were_the_challenges_and_how_did_we_solve_them\" title=\"What were the challenges and how did we solve them?\">What were the challenges and how did we solve them?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#What_we_achieved\" title=\"What we achieved?\">What we achieved?<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What_is_the_context\"><\/span>What is the context?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:paragraph -->\n<p>We wanted to develop an auction module for our flagship product MeRLIN. An e-Auction by nature is time-bound, where participants from across the globe login and submit their bids. During this period, the system is expected to provide competitive information on the participant\u2019s bids against other participants\u2019. Some auctions are very competitive and run for a very brief period. Hence it is pivotal that such competitive information is provided real-time.<\/p>\n<!-- \/wp:paragraph --><!-- wp:heading -->\n<h2 id=\"h-what-is-the-problem-statement\"><span class=\"ez-toc-section\" id=\"What_is_the_Problem_Statement\"><\/span><strong>What is the Problem Statement?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:paragraph -->\n<p>MeRLIN is architected on Microsoft stack with SQL Server being the backend. While typical transactions are stored in SQL Server, attempting to store such large and live transactions in SQL Server will place huge load and potentially impact the performance of other transactions during this period.<\/p>\n<!-- \/wp:paragraph --><!-- wp:paragraph -->\n<p>Moreover, the requirement to perform real-time calculations on the bid competition and bring them live to participants and administrators can not be done in SQL Server and typical HTTP request-response pair.<\/p>\n<!-- \/wp:paragraph --><!-- wp:heading -->\n<h2 id=\"h-what-are-the-options\"><span class=\"ez-toc-section\" id=\"What_are_the_options\"><\/span><strong>What are the options?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:paragraph -->\n<p>While SQL Server is ruled out for computation and data storage, any other technology that involves disk I\/O will not be optimal, however efficient it is. On cloud platforms such as Azure, where disk I\/O is costed, such large I\/O operations will push the infrastructure cost too. Hence, for computation purposes, an in-memory system is an optimal solution. We looked at both Memcached and Redis.<\/p>\n<!-- \/wp:paragraph --><!-- wp:paragraph -->\n<p>For notifications of computational results, we needed a push mechanism. When we looked at browser push notifications, we realised that they had security flaws and handling them to make changes to web application data will be very challenging. We looked at both HTTP notifications and the ubiquitous SignalR.<\/p>\n<!-- \/wp:paragraph --><!-- wp:image {\"align\":\"center\",\"id\":1299} -->\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" class=\"wp-image-1299\" src=\"https:\/\/blog.rheincs.com\/wp-content\/uploads\/Picture3.png\" alt=\"\" \/><\/figure>\n<\/div>\n<!-- \/wp:image --><!-- wp:heading -->\n<h2 id=\"h-how-did-we-compare-and-what-did-we-choose\"><span class=\"ez-toc-section\" id=\"How_did_we_compare_and_what_did_we_choose\"><\/span><strong>How did we compare and what did we choose?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:paragraph -->\n<p>While we have several push notification techniques such as Pushers, Socket.io etc., our natural choice for .NET based application. Considering the relative advantages and ease of use, we decided to proceed with Redis and SignalR.<\/p>\n<!-- \/wp:paragraph --><!-- wp:heading -->\n<h2 id=\"h-how-did-we-integrate-them-with-our-stack\"><span class=\"ez-toc-section\" id=\"How_did_we_Integrate_them_with_our_stack\"><\/span><strong>How did we Integrate them with our stack?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:image {\"align\":\"center\",\"id\":1294} -->\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" class=\"wp-image-1294\" src=\"https:\/\/blog.rheincs.com\/wp-content\/uploads\/sample.png\" alt=\"\" \/><\/figure>\n<\/div>\n<!-- \/wp:image --><!-- wp:paragraph -->\n<p>The next task was to integrate them with our existing stack. Also, Redis and SignaR combination is used only during the period of the auction. Once the auction is complete, the application should retrieve data from SQL Server.<\/p>\n<!-- \/wp:paragraph --><!-- wp:heading -->\n<h2 id=\"h-what-were-the-challenges-and-how-did-we-solve-them\"><span class=\"ez-toc-section\" id=\"What_were_the_challenges_and_how_did_we_solve_them\"><\/span><strong>What were the challenges and how did we solve them?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:list -->\n<ul>\n<li><em>What happens if the in-memory Redis server goes down during the auction period?<\/em><\/li>\n<\/ul>\n<!-- \/wp:list --><!-- wp:paragraph -->\n<p>We decided to dump the Redis data into SQL Server periodically so that at least we can resume the auction from the point of the storage.<\/p>\n<!-- \/wp:paragraph --><!-- wp:list -->\n<ul>\n<li><em>What happens if the web socket used by SignalR gets closed?<\/em><\/li>\n<\/ul>\n<!-- \/wp:list --><!-- wp:paragraph -->\n<p>We implemented a self reconnection mechanism so that a broken connection does not permanently break a participant\u2019s notification.<\/p>\n<!-- \/wp:paragraph --><!-- wp:list -->\n<ul>\n<li><em>How do we show live charts for the price changes to Auction owner?<\/em><\/li>\n<\/ul>\n<!-- \/wp:list --><!-- wp:paragraph -->\n<p>We have been heavily using High Charts for all our visualisations and hence High Charts live was our natural choice.<\/p>\n<!-- \/wp:paragraph --><!-- wp:heading -->\n<h2 id=\"h-what-we-achieved\"><span class=\"ez-toc-section\" id=\"What_we_achieved\"><\/span><strong>What we achieved?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<!-- \/wp:heading --><!-- wp:image {\"align\":\"center\",\"id\":1306} -->\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" class=\"wp-image-1306\" src=\"https:\/\/blog.rheincs.com\/wp-content\/uploads\/sdasd.png\" alt=\"\" \/><\/figure>\n<\/div>\n<!-- \/wp:image --><!-- wp:paragraph -->\n<p>We ended up creating an in-memory calculation along with SignalR based notification during the period of the auction and then a regular SQL Server-based storage after the period of the auction. Here is a detailed sequence diagram of the final implementation.<\/p>\n<!-- \/wp:paragraph --><!-- wp:image {\"align\":\"center\",\"id\":1307} -->\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter\"><img decoding=\"async\" class=\"wp-image-1307\" src=\"https:\/\/blog.rheincs.com\/wp-content\/uploads\/image001.gif\" alt=\"\" \/><\/figure>\n<\/div>\n<!-- \/wp:image --><!-- wp:paragraph -->\n<p>Here is how Auction can be monitored live by the Administrators.<\/p>\n<!-- \/wp:paragraph --><!-- wp:paragraph -->\n<p><strong>Kudos to the team:<\/strong><\/p>\n<!-- \/wp:paragraph --><!-- wp:paragraph -->\n<p>The development team of\u00a0<strong>Froze Khan<\/strong>,\u00a0<strong>Karthick Pugalengthi<\/strong>,\u00a0<strong>Raju R<\/strong>\u00a0and<strong>\u00a0Gowtham Venkatesh<\/strong>\u00a0of our MeRLIN team under the able guidance of\u00a0<strong>Raja Chandramohan<\/strong>\u00a0navigated through various technical challenges and POCs to crack the code of real-time auctions.<\/p>\n<!-- \/wp:paragraph -->\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>","protected":false},"excerpt":{"rendered":"<p>Set up a powerful and adaptable system to efficiently process data in real time while maintaining high performance and scalability.<\/p>","protected":false},"author":10,"featured_media":2436,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19],"tags":[34,37],"class_list":["post-2435","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-procurement-analytics","tag-analytics-engine","tag-eauction","entry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v24.4 (Yoast SEO v21.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing<\/title>\n<meta name=\"description\" content=\"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\" \/>\n<meta property=\"og:locale\" content=\"de_DE\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing\" \/>\n<meta property=\"og:description\" content=\"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\" \/>\n<meta property=\"og:site_name\" content=\"MeRLIN\u202f\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/MeRLINstrategicsourcing\" \/>\n<meta property=\"article:published_time\" content=\"2020-10-03T13:56:13+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-19T12:23:51+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/merlinsourcing.com\/wp-content\/uploads\/2020\/10\/banner-opt2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"650\" \/>\n\t<meta property=\"og:image:height\" content=\"230\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Viswanathan Srinivasan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@MeRLINsourcing\" \/>\n<meta name=\"twitter:site\" content=\"@MeRLINsourcing\" \/>\n<meta name=\"twitter:label1\" content=\"Verfasst von\" \/>\n\t<meta name=\"twitter:data1\" content=\"Viswanathan Srinivasan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Gesch\u00e4tzte Lesezeit\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\"},\"author\":{\"name\":\"Viswanathan Srinivasan\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/5275c0cd7933a181ce4516cfe4dde897\"},\"headline\":\"Setting up a high-performance scalable real-time data processing engine\",\"datePublished\":\"2020-10-03T13:56:13+00:00\",\"dateModified\":\"2026-03-19T12:23:51+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\"},\"wordCount\":586,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/#organization\"},\"keywords\":[\"analytics engine\",\"eAuction\"],\"articleSection\":[\"Procurement Analytics\"],\"inLanguage\":\"de-DE\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#respond\"]}],\"accessibilityFeature\":[\"tableOfContents\"]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\",\"url\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\",\"name\":\"Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing\",\"isPartOf\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/#website\"},\"datePublished\":\"2020-10-03T13:56:13+00:00\",\"dateModified\":\"2026-03-19T12:23:51+00:00\",\"description\":\"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.\",\"breadcrumb\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#breadcrumb\"},\"inLanguage\":\"de-DE\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/merlinsourcing.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Setting up a high-performance scalable real-time data processing engine\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#website\",\"url\":\"https:\/\/merlinsourcing.com\/de\/\",\"name\":\"MeRLIN\u202f\",\"description\":\"Strategic Sourcing Simplified\",\"publisher\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/merlinsourcing.com\/de\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"de-DE\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#organization\",\"name\":\"MeRLIN\u202f\",\"url\":\"https:\/\/merlinsourcing.com\/de\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de-DE\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#\/schema\/logo\/image\/\",\"url\":\"\",\"contentUrl\":\"\",\"caption\":\"MeRLIN\u202f\"},\"image\":{\"@id\":\"https:\/\/merlinsourcing.com\/de\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/MeRLINstrategicsourcing\",\"https:\/\/twitter.com\/MeRLINsourcing\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/5275c0cd7933a181ce4516cfe4dde897\",\"name\":\"Viswanathan Srinivasan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"de-DE\",\"@id\":\"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/705f03b3db24425e082496323d4ed214?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/705f03b3db24425e082496323d4ed214?s=96&d=mm&r=g\",\"caption\":\"Viswanathan Srinivasan\"},\"url\":\"https:\/\/merlinsourcing.com\/de\/blog\/author\/viswanathan-srinivasan\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing","description":"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/","og_locale":"de_DE","og_type":"article","og_title":"Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing","og_description":"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.","og_url":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/","og_site_name":"MeRLIN\u202f","article_publisher":"https:\/\/www.facebook.com\/MeRLINstrategicsourcing","article_published_time":"2020-10-03T13:56:13+00:00","article_modified_time":"2026-03-19T12:23:51+00:00","og_image":[{"width":650,"height":230,"url":"https:\/\/merlinsourcing.com\/wp-content\/uploads\/2020\/10\/banner-opt2.jpg","type":"image\/jpeg"}],"author":"Viswanathan Srinivasan","twitter_card":"summary_large_image","twitter_creator":"@MeRLINsourcing","twitter_site":"@MeRLINsourcing","twitter_misc":{"Verfasst von":"Viswanathan Srinivasan","Gesch\u00e4tzte Lesezeit":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#article","isPartOf":{"@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/"},"author":{"name":"Viswanathan Srinivasan","@id":"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/5275c0cd7933a181ce4516cfe4dde897"},"headline":"Setting up a high-performance scalable real-time data processing engine","datePublished":"2020-10-03T13:56:13+00:00","dateModified":"2026-03-19T12:23:51+00:00","mainEntityOfPage":{"@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/"},"wordCount":586,"commentCount":0,"publisher":{"@id":"https:\/\/merlinsourcing.com\/de\/#organization"},"keywords":["analytics engine","eAuction"],"articleSection":["Procurement Analytics"],"inLanguage":"de-DE","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#respond"]}],"accessibilityFeature":["tableOfContents"]},{"@type":"WebPage","@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/","url":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/","name":"Building a High-Performance Real-Time Data Processing Engine | Merlin Sourcing","isPartOf":{"@id":"https:\/\/merlinsourcing.com\/de\/#website"},"datePublished":"2020-10-03T13:56:13+00:00","dateModified":"2026-03-19T12:23:51+00:00","description":"Learn how to build a scalable, high-performance real-time data processing engine to handle high-volume data streams, enable faster decision-making, and support advanced sourcing and analytics use cases.","breadcrumb":{"@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#breadcrumb"},"inLanguage":"de-DE","potentialAction":[{"@type":"ReadAction","target":["https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/merlinsourcing.com\/de\/blog\/setting-up-a-high-performance-scalable-real-time-data-processing-engine\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/merlinsourcing.com\/"},{"@type":"ListItem","position":2,"name":"Setting up a high-performance scalable real-time data processing engine"}]},{"@type":"WebSite","@id":"https:\/\/merlinsourcing.com\/de\/#website","url":"https:\/\/merlinsourcing.com\/de\/","name":"MeRLIN\u202f","description":"Strategic Sourcing Simplified","publisher":{"@id":"https:\/\/merlinsourcing.com\/de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/merlinsourcing.com\/de\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"de-DE"},{"@type":"Organization","@id":"https:\/\/merlinsourcing.com\/de\/#organization","name":"MeRLIN\u202f","url":"https:\/\/merlinsourcing.com\/de\/","logo":{"@type":"ImageObject","inLanguage":"de-DE","@id":"https:\/\/merlinsourcing.com\/de\/#\/schema\/logo\/image\/","url":"","contentUrl":"","caption":"MeRLIN\u202f"},"image":{"@id":"https:\/\/merlinsourcing.com\/de\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/MeRLINstrategicsourcing","https:\/\/twitter.com\/MeRLINsourcing"]},{"@type":"Person","@id":"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/5275c0cd7933a181ce4516cfe4dde897","name":"Viswanathan Srinivasan","image":{"@type":"ImageObject","inLanguage":"de-DE","@id":"https:\/\/merlinsourcing.com\/de\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/705f03b3db24425e082496323d4ed214?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/705f03b3db24425e082496323d4ed214?s=96&d=mm&r=g","caption":"Viswanathan Srinivasan"},"url":"https:\/\/merlinsourcing.com\/de\/blog\/author\/viswanathan-srinivasan\/"}]}},"_links":{"self":[{"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/posts\/2435"}],"collection":[{"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/comments?post=2435"}],"version-history":[{"count":9,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/posts\/2435\/revisions"}],"predecessor-version":[{"id":43085,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/posts\/2435\/revisions\/43085"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/media\/2436"}],"wp:attachment":[{"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/media?parent=2435"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/categories?post=2435"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merlinsourcing.com\/de\/wp-json\/wp\/v2\/tags?post=2435"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}