You can argue about its roots, but modern middleware got going in earnest with the advent of (proprietary) IBM MQ in the early 1990s. Banking was the sweet spot for this asynchronous messenger. Who would have thought message queuing would come to encompass open source software and applications ranging from cloud computing to big data and beyond? Well, it did.
SearchSOA.com was able to speak with one of middleware’s great thought leaders when we hooked up with James Strachan, senior consulting software engineer at RedHat, at the recent CamelOne 2012 event in Boston. Strachan, former CTO of FuseSource and a founder of Apache ActiveMQ, contrasted the somewhat simpler past days of middleware with the present -- which, without question, is increasingly complex.
More and more people are involved with message queuing and middleware development as opposed to years ago. And yet, it's also not just strictly transactional or analytical systems.
It used to be a little bit niche. Then with the "big data" thing, we're almost seeing a combination of data, database, big data and messaging. Throw caching in the mix and everything's starting to kind of cross over. Most people are starting to use messaging and big data. Big data is a system of record or the offline batch processing or random access to historic things and the messaging is the real-time low-latency do things as quickly as possible. People are starting to combine the two together. I guess the concepts behind databases and big data and messaging are very much combining.
Yet there are a lot of people with a lot of programming experience who are just getting into message queuing.
Now I think it must be hard for new programmers to really comprehend all of it. I think it's only old folks like us that really have at least a vague understanding of the whole space. Because it's huge now. Just understanding Hadoop is really tricky. There's a lot in Hadoop…it's a massive, complex piece of software. Just understanding one piece can take a long time.
But in terms of the concepts behind this stuff, it's all kind of similar. You have your registries, you have your way of coordinating. Incidentally, a parallel: FuseFabric uses ZooKeeper, Hadoop uses ZooKeeper—they're all using the same coordinator. ZooKeeper's an Apache project. It's basically a distributed coordination engine. When you're making distributed systems, it's very hard to know who's the master and who's the slave…how do a hundred machines decide to make a decision? You tend to use a leadership election system for that -- because if everyone was making a decision, you'd have chaos.
Follow us on Twitter at @SearchSOA.