Duplicate Message Filtering

Multiple Event Instances Filtering

Technical Complexity

Simple

Purpose

Allow the listener to not have to manage possible multiple instances of the same event.

Situation

A machine is being detected as down. A periodic check is done, and each time the check is done, an event is sent. But the event listener is only interested in one instance of the event for the down state (at most once).
An ATM allows for money withdrawal. But to avoid rapid fire events, only one withdrawal event is allowed on a single account every 10 seconds.

Design

The notion of filtering multiple copies of the same event on a stream can be rephrased as, not having the same event twice in a row. While not being ground breaking, this change in wording allow us to do the following, if we have ON event and OFF event, and the inbound stream looks like:

[ON,ON,ON,OFF,OFF,OFF,OFF,ON,OFF,OFF,...] and what we want to send out is

[ON,OFF,ON,OFF,....]

We basically show a two state event. So, if we have a sliding window of 2 just to keep the last two events:

From [ON,ON] being the last two events, we know we only want [ON],
Same with [OFF,OFF] we only want [OFF].
But if we have [ON,OFF] or [OFF,ON] as the last two, we will want to see then both.

So in short, we want the last two event, that are distinct from one another.

To be able to forward this to a stream, we have to use a stream converter. For this case we want the new event that would be inserted in the stream. So, out of the distinct, we would get the last entry in relation, and put it into the outbound stream.

IEP Constructs Used

Process Flow

Inbound stream
Stream to Relation Tuple Window Converter
Distinct Relation
Inserted Event Relation to Stream Converter
Outbound Stream

Tuple Window Converter

Sample Files