What is the context?
We wanted to develop an auction module for our flagship product MeRLIN. An e-Auction by nature is time-bound, where participants from across the globe login and submit their bids. During this period, the system is expected to provide competitive information on the participant’s bids against other participants’. Some auctions are very competitive and run for a very brief period. Hence it is pivotal that such competitive information is provided real-time.
What is the Problem Statement?
MeRLIN is architected on Microsoft stack with SQL Server being the backend. While typical transactions are stored in SQL Server, attempting to store such large and live transactions in SQL Server will place huge load and potentially impact the performance of other transactions during this period.
Moreover, the requirement to perform real-time calculations on the bid competition and bring them live to participants and administrators can not be done in SQL Server and typical HTTP request-response pair.
What are the options?
While SQL Server is ruled out for computation and data storage, any other technology that involves disk I/O will not be optimal, however efficient it is. On cloud platforms such as Azure, where disk I/O is costed, such large I/O operations will push the infrastructure cost too. Hence, for computation purposes, an in-memory system is an optimal solution. We looked at both Memcached and Redis.
For notifications of computational results, we needed a push mechanism. When we looked at browser push notifications, we realised that they had security flaws and handling them to make changes to web application data will be very challenging. We looked at both HTTP notifications and the ubiquitous SignalR.
How did we compare and what did we choose?
While we have several push notification techniques such as Pushers, Socket.io etc., our natural choice for .NET based application. Considering the relative advantages and ease of use, we decided to proceed with Redis and SignalR.
How did we Integrate them with our stack?
The next task was to integrate them with our existing stack. Also, Redis and SignaR combination is used only during the period of the auction. Once the auction is complete, the application should retrieve data from SQL Server.
What were the challenges and how did we solve them?
- What happens if the in-memory Redis server goes down during the auction period?
We decided to dump the Redis data into SQL Server periodically so that at least we can resume the auction from the point of the storage.
- What happens if the web socket used by SignalR gets closed?
We implemented a self reconnection mechanism so that a broken connection does not permanently break a participant’s notification.
- How do we show live charts for the price changes to Auction owner?
We have been heavily using High Charts for all our visualisations and hence High Charts live was our natural choice.
What we achieved?
We ended up creating an in-memory calculation along with SignalR based notification during the period of the auction and then a regular SQL Server-based storage after the period of the auction. Here is a detailed sequence diagram of the final implementation.
Here is how Auction can be monitored live by the Administrators.
Kudos to the team:
The development team of Froze Khan, Karthick Pugalengthi, Raju R and Gowtham Venkatesh of our MeRLIN team under the able guidance of Raja Chandramohan navigated through various technical challenges and POCs to crack the code of real-time auctions.
