Managing Continuos Data Feed with Subscriber/Publisher Pattern
Software Engineering Department
Pattern Name and Classification
Subscriber Purpose: Behavioral, Scope:Object
Publisher Purpose: Behavioral, Scope:Object
Allow a continuos data generator to accept subscription request, manage subscription list and maintain the data feed to any number of consumer in a distributed environment with very little exposure to substrate system primitives and very high genericity/reusability.
Also Known As
Subscriber is a specialization of the Observer Pattern of the MVC framework to maintain continuos data feed.
Our application scenario is one of providing prices of financial instruments to one or more customers. There are several unique aspects to this application. In our environment the prices are generated in a main frame environment which provides marginal support for message passing. The Feed Server system and its components reside in a UNIX environment as shown in Figure 1. The source server and the customers are not synchronized, in that, the source may initiate the feed while there are no active customers. Or conversely one or more customers may request for the feed before the source. Further, in the financial domain, customers and servers are separated by geographical distances and prone to link failures. Consequently, recovery and replay protocols are very important and the recovery protocols are never the same and are particular to the business needs of the customer. And, for obvious reasons failure on one customer link should not affect other customers. Furthermore, the pricing model and the class of instruments of interest to a customer is particular to that customer (customer specific). The data feed must be customizable according to particular customer requirements. Thus, the application scenario demands a highly reliable, fault tolerant, and flexible feed delivery service customizable to individual customer specification.
The Publisher/Subscriber Pattern can be used to:
In this section we present in detail the Subscriber-Publisher pattern used in this application. Subscriber-Publisher is a simplified but distributed rendition of the Smalltalk fame Model-View-Controller (MVC) framework . The MVC is a rather complex design construct and is usually made up of several simple and cohesive design patterns and one of them is the Observer pattern. The Observer pattern is an essential sub-component to implement the more general MVC design construct and it is defined to be " .. used to decouple objects so that changes to one can affect any number of clients without requiring the changed object to know the details of the others" .
The relevance of the Observer pattern in our application scenario is that the stream of messages generated by the Source represents the "model" and the customized stream of messages generated by the subscribers represent the "views". The role of Publisher is to notify the list of subscribers which subscribe to the feed.
Notifying the subscribers of a new message arrival is equivalent to the updating the views in the original MVC model. The relationship is presented in Booch Diagram in Figure 2 and the Booch interaction diagram is presented in Figure 3.
The Publisher-Subscriber implemented here is a domain specific customization of the Observer pattern. These are composite objects which are made up of IPC wrappers and various foundation classes, such as the Acceptor, Connector, Courier and MultiPlexor wrappers discussed in . Acceptor is the communication engine for the Publisher objects to accept incoming subscription requests. The Connector is the communication engine for the Subscribers to initiate a subscribe request with the Publisher. Courier is a transport object. The MultiPlexor is a wrapper to maintain multiple subscription streams.
- Accepts Subscribe/Unsubscribe Requests
- Enforces distribution criteria
- Publishes message units
- Subscribes to data sources
- Retrieves Messages as they arrive
- Maintains a Courier Object for communication
- Maintains a list of SubscriberInfo
- Publishes the message to all the subscribers maintained by the ListManager
1. Maintaining Subscriber List
IPC Wrappers, such as, Courier, Connector and Acceptor are used to maintain references, make a connection and accept an incoming subscription requests, respectively. These are network distributed references. Robustness and richness of the communication capabilities are delegated to these wrappers.
2. Message Independence
The framework presented here is oblivious to the kind of message being exchanged. Messages are treated as a stream of bytes of definite length. The framework is not responsible for data representation issues. The framework cohesively addresses the problem of publishing messages as they arrive.
3. Recovering from Communication Errors
Distributor encapsulates all tasks relating to detection and recovery from underlying communication failures. Distributor is used to multicast the message and provide fault tolerance.
4. Maintaining Priority Subscribers
The ListManager maintains various lists of equivalent subscription priorities and enforces the distribution policies.
5. Who initiates message exchange?
In our framework messages are generated by an external source and is delivered to the Publisher which then distributes to the set of current subscribers. The subscribers customize and deliver to the end client. Thus, the whole process is driven by the source.
6. Resolving Subscriber Location
The Connector and Acceptor Wrappers provide these resolution services. Publishers and Subscribers are not concerned about these aspects.
The Publisher/Subscription framework can be used to maintain multiple streams. With modifications the service may be extended to multimedia services as well.
This system was designed and implemented for Asset Corporation, Holmdel, New Jersey on contract to a major financial firm. This system would not have been possible without the support of the major financial firm, senior Vice President, the Project Manager and the team involved. The author acknowledges the anonymous referees for their many useful, thoughtful and constructive comments. And, the Software Engineering Department, Center for Telematics and the Monmouth University for supporting the preparation of this manuscript and other activities of the DNA Laboratory. DNA is an unfunded laboratory comprised wholly of volunteers.
1. Gamma, E., R. Helm, R. Johnson, and J. Vlissides, Design Patterns: Elements of Reusable Object Oriented Software, Addison Wesley, Reading, MA, 1994.
2. Glenn E. Krasner and Stephen T. Pope. A Cookbook for Using the Model View-Controller User Interface Paradigm in Smalltalk-80, Journal Of Object Oriented Programming, August-September 1988, 26-49.
3. Booch, D. Object Oriented Analysis and Design with Applications. Second Edition. Benjamin/Cummings Publishing Company, Inc., Redwood City, CA 94065, 1994.
4. Douglas C. Schmidt, ASX: An Object Oriented Framework for Developing Distributed Applications, USENIX C++ Conference, 1994.
5. Raman Kannan, Application Partitioning with Patterns, TR-MU-SE-DNA-95-002.