Consuming messages directly from a queue which belongs to a third party system

by RishikeshDhokare   Last Updated July 07, 2018 05:05 AM

We are working on a project which interacts with multiple third party systems and they need each other's data to function. Now, for some data which needs to be in sync in multiple systems, we are planning to introduce event based architecture where if the data is created/updated in one system, it will publish an event and the other's will subscribe to those events.

The problem(?) I have is, these are third party systems. One is deployed to Microsoft Azure, other is deployed to AWS and some other is deployed to their own data center etc. In short their is no certainly what kind of infrastructure the third party systems might be using.

Currently, they communicate with each other over HTTP.

The question is - while introducing event driven architecture, should my system directly consume from a queue which is in third party infrastructure? OR let them write the consumer as well which will continue to communicate with my system over HTTP?

Answers 2

I'm not clear on the two options. It appears you're saying you could read directly from a queue, or let a third party write something that reads from the queue and pushes the message to you via an api endpoint.

If this is the choice, I would much prefer the queue. Queues are designed to be tolerant of consumer downtime so if your consumer is down for maintenance the messages just wait until you're back up. If your vendor writes a queue reader and pushes to your endpoint, they may not be as tolerant and you could lose messages.

In fact, if the queue they have internally is RabbitMQ or something similar, these all support HTTP push notifications so the end result is an HTTP call to your system anyway.

As for the different service providers, there's not much difference reading from a queue in AWS, Azure, or on premises. The syntax to pull a message off a queue is going to be slightly different for each implementation, but the basic ideas are the same.

Brad Irby
Brad Irby
June 06, 2018 13:48 PM

(I could not add this as comment hence adding as an answer)

@Brad Irby,

Your interpretation of two options is correct. As long as the queue implementations have push notifications, I don't think I have any problems.

With the approach of reading directly from queue, I see few problems -

  1. Based on different queue implementations e.g rabbitmq, aws sqs, kafka, azure's queue service, every service will need to write different consumer implementations using their libraries. IMO, It is not scalable and maintainable.
  2. If the third party puts the message in queue, I think their responsibility should also include to push the message out of their network via any mechanism (I prefer HTTP since that is the widely used protocol for communication between web components). As far as loosing the message is concerned, there is a solution - when you push a message outside your network, don't delete the message from the queue till you receive HTTP 200/202.

There is no right or wrong approaches to this. Do my concerns make sense?

June 07, 2018 03:55 AM

Related Questions

facebook/linkedin like newsfeed

Updated January 16, 2018 00:05 AM

Information sharing in event-based systems

Updated December 16, 2017 19:05 PM

Java event driven - choreography with microservices

Updated August 25, 2017 08:05 AM