Edward R. Murrow once said, “People say conversation is a lost art; how often I have wished it were.” How true it is in the vendor and media-spun world of IT and even more so in the case of SOA. Many people think they are doing or talking about Service-Oriented Architecture (SOA), but most of the time they’re really doing Web Services Integration (WSI), developing Just a Bunch of Web Services (JBOWS), or even just point-to-point XML (what many would like to call REST, but even in most cases is not even truly Representational State Transfer [REST], which is also an architectural form). Even after 8 years of evangelism, research, advisory, and deep consulting, we’re still finding a large percentage of so-called enterprise architects pursuing SOA projects as if they were point-to-point integration projects.

We’ve discussed many times from an economic, technical, architectural, and philosophical perspective the problem with not understanding that SOA is an architectural concept, so I won’t bore or nag you further in understanding that SOA is not just another name for integration. However, we need to crack this nut and crack it definitively if this industry is to make progress from its current inefficient, ineffective, unpredictable, overly-expensive, overly-complex, and rudimentary form to the predictable, reliable, cost-effective, disciplined, and easily measured part of the business organization that we know it should be.

While we’ve made our pleas to the business, architectural, and process-driven parts of the organization and to vendors in previous ZapFlashes, we want to focus this ZapFlash on where the problem specifically lies: within the IT organization, and specifically the part that concerns themselves with so-called integration issues. Folks (and I’m speaking directly to you integration-centric techies), you are getting in the way of making SOA a success probably due to your own misunderstandings and misconceptions of how SOA should work. And to correct your misunderstanding, I need to speak your language: the language of integration technology.

The Problem of Service Consumer to Service Provider Communication
In typical integration environments, you’re primarily concerned with the movement of data or the amalgamation of business logic to achieve a specific task. In that vein, most integration developers think about inputs (“gazintas”), outputs (“gazattas”), transfer, and transformation. That is, they think about how they’re going to get data or application functionality out of one system, how they’ll get it into another system, the mechanism they’ll use to get that information or functionality from one place to another, and then what they need to do with it while it’s en-route to transform it from one format, protocol, context, process, or policy into another. So, while architects are chemists — that is, scientists who are trying to understand a system and how it can all be put together, integration developers are chemical engineers — focusing on just the mechanisms of transformation.

From this integration-centric mentality, most integration developers see the concept of the Service as just another means of gazinta or gazatta. That means these integration-heads need to find something else to do transfer and transformation. Vendors to the rescue! Most vendors responded to this desire by simply dusting off their existing integration middleware. End-users responded easily to this insignificant change and voila, now they have transfer and transform that can deal with their Service-based gazintas/gazattas. But does this make SOA? Oh no, no, no. But we’ve already beaten you on the head with that a million times. So, let’s take a different tack. The reason why this approach is wrong is because Services Consumers (what you might think of as a gazinta) should never communicate directly to Service Providers (what you might erroneously think is a gazatta).

Why? First, the main concept of SOA is that we want to deal with frequent and unpredictable change by constructing an architectural model, discipline, and abstraction that loosely-couples the providers of capability from the consumers of capability in an environment of continuous heterogeneity. This means that we have to somehow develop capabilities without the knowledge of how they might be used. As we delved into the Seven Aspects of Loose Coupling in a previous ZapFlash, this means that we need to have significant variability at a variety of aspects including the implementation, infrastructure, contract, process, policy, data schema, and semantic aspects. Having this variability allows companies to have stability in their business and IT even though the IT and business continue to change. Agility, predictability, reliability, visibility, and cost-effectiveness all become that much more realistic when we can achieve that sort of abstraction.

But to achieve that abstraction means that we can’t afford things to break when things change. Lots of things change in an IT or business environment: location and availability of services, versioning of Service implementations and contracts, changes to business processes and policies, new or changing requirements for reporting, visibility, and control, and of course the need for new capabilities to solve emerging business problems or capitalize on new business opportunities. However, if you think that a Service is just an API, then all hell breaks loose when you try to connect a Service Consumer directly to a Service Provider. What if the provider moves? What if the Service contract changes? What if the provider is no longer available? What if you now need a new communication mechanism or change to a data schema?

The first knee-jerk reaction to this complexity by most integration “architects” is to put some piece of technology between the Service consumers and providers to ameliorate this problem. The impulse is correct: you can’t have direct communication if you want to have variability. However, the execution is all wrong. If you are relying on black-box, and often proprietary, technology to do the heavy lifting of managing Service communication differences you’ll simply shift all the complexity and work from the end-points (the Services) to an increasingly more complex, expensive, and brittle middle point. Welcome to EAI 2.0, the secret and unpublicized real name for most so-called Enterprise Service Bus (ESB) products — or at least the way that many are using them.

(Re)Introducing the Service Proxy
So, if reworked middleware isn’t the solution, what is? Well, of course the answer is architecture. But that’s too pat of an answer, so to speak again in the parlance of integration developers, let’s explore how a system would be designed if you can’t count on any particular vendor’s solution to be in the middle to make it all work.

Let’s say you want to build a Service Consumer to consume or compose some functionality, but you don’t know where that functionality is or even how to communicate with it. There are at least three ways of handling this particular challenge. First, you can talk to a proxy (or broker or intermediary) in a language that you do understand and that’s in a location that you already know and send a request to this proxy that will act on the behalf of the Consumer. This is the “envelope within an envelope” concept. Basically, you enclose a message to deliver to someone whom you know, but don’t know where they are, within an envelope that is addressed to a party that you do know. That party then opens the envelope and addresses the internal envelope to the end recipient based on a look-up of some sort. This approach simplifies the problem of the Service consumer having to know about changes since they just need to know about the proxy and the end-recipient. From a technology perspective, the use of WS-Addressing simplifies this approach. In essence, a Service consumer just needs to know the WS-Address of a Provider and hand that to a Service proxy to resolve and deliver.

We’ve discussed this concept of the Service Intermediary before. The problem with this approach is that the Service proxy still needs to find a way to deliver the message, and if the proxy contains all the rules internally in the black-box, we have the same problem we have with EAI 2.0-style ESB solutions. This is where the registry steps in. Service proxies should not store their own rules, but rather get all their routing, location, and policy-based binding rules from a central system of record. In SOA, this is the role of the registry. Proxies can cache the results of registry look-ups for performance enhancement, but the metadata themselves should come from the registry.

So, registry-configured Service proxies solve many of the problems of Service communication, but it still leaves open the problem of the connection between the Service consumer and the Service proxy. What if the proxies move or are unavailable? And how do Service consumers find out about the Service proxy in the first place? The answer to this challenge is the use of registry-based “late binding”. In this scenario, the Service consumer pings the registry to find out where the proxy is, perhaps based on where the Service consumer is currently located and with what protocols it can communicate. One would think that we can make the Service consumer directly bind to a Service provider after finding its metadata in the registry, but that would be a mistake. The problem is that if the Service provider is using some alternate protocol or is in some location that the Service consumer can’t reach, we’ll have a communications problem. So, even in the case where we’re discovering metadata in a registry, we still don’t want direct communication between a Service consumer and provider.

Therefore, the optimal solution here is to use registry-based “late-binding” to help the Service consumers find proxies and also to help them resolve Service providers to WS-Addresses. Then, the Service consumer communicates with a proxy, providing the WS-Address and required metadata for communication. This allows Service consumers to be quite simple, leveraging a network of distributed, and most likely heterogeneous, Service proxies doing the communication on behalf of Service consumers. In this scenario, companies can implement Service proxies using and mixing a wide range of technologies and approaches including hardware-based network intermediaries, software proxies, and even agents that reside on Service consumer end-points.

Finally, in the above scenario, where’s the ESB? Nowhere, unless you believe that the ESB is a pattern and not a product (as IBM once espoused). Is the ESB needed? Not necessarily. Will it make you feel better to buy one anyways? Maybe. Of course, what are these proxies? Surely ESB vendors will want to have you believe that their products can be used as proxies in this configuration, and that’s absolutely true. They certainly can. The point is not that ESB’s can’t make good proxies. They certainly can, and I want to specifically point that out. But you can also get away with not using one or using your existing infrastructure and it will all still work. The sad truth is that most integration architects are not using ESBs as Service proxies, but rather as EAI 2.0 hub-and-spoke or message-bus brokers that facilitate Web Services Integration, not SOA.

The ZapThink Take
If you think about it, we’ve already implemented complex, heterogeneous networks in this fashion before. Computers leverage DHCP, gateways, and routers to do networking in a way that’s far superior to direct, point-to-point IP communications. And this is basically what we’re saying about Service communication. Understanding this is a realization that this is not a technology issue, but an architectural one.

To get SOA, you have to get it in your head that the Service concept is an abstraction, not an interface. Once you understand the abstraction, then it will all become that much clearer. So, you need to do some soul-searching with regards to your SOA initiative. Are you doing Web Services Integration (WSI) with EAI 2.0, or have you really changed your architecture to become Service-oriented in an environment of heterogeneity and continuous change? If you are doing WSI on EAI 2.0, you will find yourself continuously solving the same problems. With SOA, you will find yourself able to continuously solve new problems.

Discussion
48 comments for “Why Service Consumers and Service Providers Should Never Directly Communicate”
Service-Mediation Scenarios « Actionable's Blog Avatar

[...] Schmelzer (2008) Why Service Consumers and Service Providers Should Never Directly Communicate , zapthink [...]

Economics Tuition Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Economics Tuition | July 24, 2016
Calile Malouf Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Calile Malouf | August 1, 2016
Buy Steroids Online Avatar

... [Trackback]

[...] There you will find 26833 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Buy Steroids Online | August 13, 2016
Printing Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Printing | August 21, 2016
satta matka Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by satta matka | August 22, 2016
click here link click click this link good site here is the site http://www.clemsondeckbuilders.com/ www.clemsondeckbuilders.com/ clemsondeckbuilders.com/ http://www.clemsondeckbuilders.com/ www.clemsondeckbuilders.com/ clemsondeckbuilders.com/ http://www Avatar

... [Trackback]

[...] Read More Infos here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Margaret Cunniffe is an Australian Fraudster based in Melbourne Victoria who is a lying drunkard who creates the facade of a rich person by drinking champagne and living the highlife. Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Agen Bandarq Avatar

... [Trackback]

[...] Read More Infos here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Agen Bandarq | August 31, 2016
economics tuition Avatar

... [Trackback]

[...] Informations on that Topic: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by economics tuition | September 3, 2016
yandex ankara Avatar

... [Trackback]

[...] There you will find 4796 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by yandex ankara | September 4, 2016
The definition of Civil Liability at MyLawDictionary.org Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

yandex ankara Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by yandex ankara | September 10, 2016
آموزش طراحی سایت Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by آموزش طراحی سایت | September 18, 2016
hgh gh Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by hgh gh | September 23, 2016
google my business Avatar

... [Trackback]

[...] Read More Infos here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by google my business | September 30, 2016
motogolf Avatar

... [Trackback]

[...] There you will find 72070 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by motogolf | October 2, 2016
Imprimiss changing the rules Avatar

... [Trackback]

[...] There you will find 83990 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Imprimiss changing the rules | October 14, 2016
Best Newspaper in India Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Best Newspaper in India | October 21, 2016
ankara escort Avatar

... [Trackback]

[...] There you will find 35414 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by ankara escort | October 30, 2016
Best Newspaper in India Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Best Newspaper in India | November 2, 2016
Best Newspaper in India Avatar

... [Trackback]

[...] Find More Informations here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Best Newspaper in India | November 2, 2016
Best Newspaper in India Avatar

... [Trackback]

[...] There you will find 80933 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Best Newspaper in India | November 2, 2016
buy best steroids Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by buy best steroids | November 4, 2016
kryptodyne Avatar

... [Trackback]

[...] There you will find 72941 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by kryptodyne | November 4, 2016
rhodium Avatar

... [Trackback]

[...] There you will find 64583 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by rhodium | November 4, 2016
rhodium Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by rhodium | November 5, 2016
joseph de saram Avatar

... [Trackback]

[...] There you will find 25285 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by joseph de saram | November 5, 2016
joseph shihara rukshan de saram Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by joseph shihara rukshan de saram | November 6, 2016
joseph shihara rukshan de saram Avatar

... [Trackback]

[...] Informations on that Topic: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by joseph shihara rukshan de saram | November 6, 2016
ethanol brander Avatar

... [Trackback]

[...] Read More Infos here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by ethanol brander | November 7, 2016
gay parenting Avatar

... [Trackback]

[...] There you will find 83407 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by gay parenting | November 15, 2016
gp test cyp 250 Avatar

... [Trackback]

[...] There you will find 76040 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by gp test cyp 250 | November 15, 2016
kimim ben Avatar

... [Trackback]

[...] There you will find 93227 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by kimim ben | November 19, 2016
big ass xxx Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by big ass xxx | November 21, 2016
click here to hire an injury attorney Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by click here to hire an injury attorney | November 28, 2016
celtic jewelry pendants Avatar

... [Trackback]

[...] Informations on that Topic: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by celtic jewelry pendants | December 10, 2016
http://www.godwinsremovals.co.uk/international-removals/northern-ireland Avatar

... [Trackback]

[...] There you will find 59866 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Celebrity Management & Casting Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Celebrity Management & Casting | December 12, 2016
papasalpormayor.cl Avatar

... [Trackback]

[...] There you will find 6067 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by papasalpormayor.cl | December 12, 2016
download free Avatar

... [Trackback]

[...] Informations on that Topic: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by download free | December 18, 2016
APKBucket Avatar

... [Trackback]

[...] Read More Infos here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by APKBucket | December 18, 2016
100 layers of glue Avatar

... [Trackback]

[...] There you will find 51179 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by 100 layers of glue | December 19, 2016
Skrota bilen Göteborg Avatar

... [Trackback]

[...] There you will find 74822 more Infos: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Skrota bilen Göteborg | December 23, 2016
Skrota bilen Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by Skrota bilen | December 23, 2016
trump Avatar

... [Trackback]

[...] Read More here: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by trump | December 28, 2016
chimney Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by chimney | December 30, 2016
VA Home Loans Avatar

... [Trackback]

[...] Read More: zapthink.com/2008/02/28/why-service-consumers-and-service-providers-should-never-directly-communicate/ [...]

Posted by VA Home Loans | January 8, 2017