Microservices have been positioned as an almost all-around solution to the problems organizations face working...
with monolithic architectures. But while rapid deployment and efficiency are on the mind of potential adopters, management complexity may be looming in the mist.
In this interview with SearchMicroservices, Gwen Shapira, systems architect at Confluent, an open source streaming platform based on Apache Kafka, explains not only what problems microservices may address, but also which ones they may bring in through the back door. She also explained why she thinks there is a lack of available tooling for microservices management and what tools people should be considering.
What problems do microservices solve, and what problems do they introduce?
Gwen Shapira: I think the biggest problems that microservices are supposed to solve are both time to delivery and how safe you are making a change. If you're a hotel and you want to add a new application that handles some kind of new promotion, how fast can you get that to market? And what is the risk in running it? What other things are going to be impacted because you are making this one change? Microservices are an attempt to decouple everything so that things can move at their own pace with very little risk.
But there are two major problems that I see. One is that, now, your business logic is distributed across multiple different services. The code base is suddenly all torn apart. And if you're trying to predict everything that will happen when a customer updates his mailing address, for example, it's really hard to tell what else is going on, because everything is in those independent pieces and you don't always know about it. It also means that if the wrong thing happened, it's really hard to figure out how the wrong thing happened.
Also, you can make a small change that is actually going to have super wide-ranging impact on the organization, and you're not going to know about it because it's all torn apart and distributed. For example, I saw a presentation by Uber where they explained how they made a tiny change to how they stored the dates in one service, and they had five different applications all break down because they were dependent on the dates being slightly different. They couldn't have known anything about that. I see this happen over and over in different organizations. So, that's definitely an issue with just lacking the big picture; [it] is a really difficult problem.
Even engineers at a company experienced in microservices management, like Uber, wouldn't be able to know about that?
Shapira: Even if they thought about it, how would you go about actually solving it? You'd have to have a big meeting of every group and say, 'I'm going to do this. Is it going to have impact?' And will individual engineers who wrote a particular service maybe two years ago remember that this thing depends on that other thing?
When it was in monolithic, it was easy because everything had a lot of very good tests. And I feel that, in microservices, end-to-end testing actually incorporates entire large chains, which are not something you normally see. And [the availability of] a lot of tools to just validate that your change is not breaking anyone else and visualizing who depends on what is something that is still missing. I've seen few companies building tools for that, but I feel like the entire area is still super young. There's no one good solution that every company -- Uber, Netflix, LinkedIn, Airbnb -- uses to solve things in their own special way.
What kinds of microservices management tools are available that can actually help?
Shapira: We keep seeing a lot of companies build what's called the schema registry. [There are many] events that go between the systems, and it registers what those events look like. And everyone can agree that, yes, this is what a customer is, this is what an address is. And then, if someone tries to make a change, it has to register a new version. If it's compatible and changeable, it goes through -- no problem. If it's incompatible, the person who is trying to register immediately knows that this is not compatible, and they will be prevented from writing incompatible events that are going to later break down everyone else's system. Even if it's not the entire solution, it's definitely part of the solution.
Does that risk slowing down the rate at which you can push out changes to applications and services?
Shapira: To a very small extent. Because it's something that the engineer can see immediately, they can test it on his machine and immediately see feedback. So, yes, if they broke something, they'll have to think about what they did. But it's not like calling an entire company and getting approval. I used to be a database administrator, and I was part of this. Making a change to a database process could take two to four months because you had to get approval from so many different stakeholders -- and that was a time when I actually knew who all the stakeholders were.
The four-month change process is exactly what organizations are trying to step away from, and empowering engineers to do this kind of validation on machines, just like they do other unit tests, is huge.
So it sounds like implementing a larger authorization process at the beginning can potentially save you a lot of time in the long run.
Shapira: Exactly. And if you do event-driven microservices or event sourcing, basically, the whole point is that events are immutable. They are not going to change, which means that if you wrote a bad event in the system and you store it for a very long time and those event sourcing systems need to go back and process it maybe months after, you just created large technical debt for the entire organization. They need to know how to handle this bad event you accidentally wrote.
Five tips for adopting your own microservices architecture
Learn how to manage your APIs properly for microservices
Discover why continuous integration is a must for microservices