Thoughts on “if you can’t build a well-structured monolith, what makes you think microservices is the answer?”

A couple of weeks ago at the DevNexus conference Simon Brown gave a keynote (slides) titled the “Modular Monolith”.  It was an interesting presentation and he made a number of good points. His final slide asked this excellent question:

If you can’t build a well-structured monolith, what makes you think microservices is the answer?

He had made a similar point on Twitter:

As had other tweets:

 

 

I don’t disagree with the sentiment of Simon’s question. But I think it misses a critical point.

Most teams know how to build a modular system. They want to build a modular system. Most teams usually do. The challenge is maintaining the modularity. A large and/or complex application will often degenerate into a big ball of mud.

The challenge is maintaining modularity

There are several factors that make it difficult to maintain modularity:

  • Current programming languages lack genuinely useful modularity mechanisms. Java visibility rules are insufficient. Perhaps Java 9 modules will finally solve this problem.
  • Monoliths often become too complex for a developer understand. This increases the chance that changes will violate implicit modular boundaries causing the design to degrade.
  • Time pressure. There are no depths to which a developer will not sink (especially under pressure from management) in order to meet a deadline. Who cares about modularity when you need to ship software?!

Microservices can help with modularity

Microservices are certainly not a silver bullet. But one benefit they offer is that your application is functionality decomposed into smaller (not necessarily tiny) services. Each service is much easier to understand. In theory, it is easier to preserve its architecture and modularity.

More importantly, the service API is an impermeable barrier that encapsulates the service implementation. It takes real effort and determination to breach that barrier.  A developer is less likely to make a quick hack to solve a problem the night before a deadline. As a result, it is likely that it will be easier to maintain the modularity of the system.

Posted in microservices, Uncategorized | Tagged , , , | 1 Comment

Microxchg 2016: A pattern language for microservices

A couple of weeks ago, I spoke at the MicroXchg conference in Berlin. It is a small, two track conference that is just about microservices. Lots of great topics. In particular, it was nice to see that the discussion has moved beyond the basics of what is a microservice? Quite a few speakers shared their practical experiences with using to microservices. Notable speakers included James Lewis, Suzanne Kaiser,  and Praveena Fernandes.

I gave a talk on a pattern language for microservices. Here are the slides.

Here is the video.

I also took part in a panel.

Berlin has some good restaurants. I had some great Turkish food at Hassir Mitte.

Learn more

To learn more about microservices:

 

Posted in microservices, Uncategorized | Tagged , , , , | 1 Comment

Microservice chassis pattern

Build your microservices using a microservice chassis, which is a framework that handles cross-cutting concerns

When you build a microservice you must put in place the mechanisms to handle cross-cutting concerns such as logging, externalized configuration, health checks, metrics, service discovery, and circuit breakers. You cannot afford to spend a significant amount of time doing this for each service. Therefore, you should build your microservices using a microservice chassis, which is a framework that handles these cross-cutting concerns.

For a full description of this pattern see Microservice chassis pattern.

Posted in microservices, Uncategorized | Tagged , | Leave a comment

Microservices – more than just infrastructure

Eventuate.IO

Matt Miller of Sequoia recently published a map of the microservices ecosystem.

The ecosystem map is very focussed on infrastructure. It lists many of the usual suspects including Docker, Kafka, Cloud Foundry, Azure, and Chef. It includes some developer frameworks such as Hystrix but unfortunately, the focus on infrastructure means there are some surprising omissions.

Microservices need a chassis

James Watters from Pivotal, for instance, points out that Spring Boot and Spring Cloud are nowhere to be seen:

That is a shame because if you are building microservices you need a microservice chassis, such as Spring Cloud + Spring Boot. A microservice chassis is a framework that enables you…

View original post 262 more words

Posted in Uncategorized | Leave a comment

API Gateway pattern

Have a service that acts as the sole entry point into a microservices-based application

Consider a mobile shopping application.  The product information that it needs to show to the user includes the description, price, reviews, recommendations and shipping information. The ownership of that data is spread among a number of microservices. In theory, the application could access each microservice separately. But in practice, it makes more sense for the mobile application to retrieve the needed information by making a single request to a service known as an API Gateway, which aggregates data from multiple services.

For a full description of this pattern see the API Gateway Pattern.

Posted in microservices, Uncategorized | Tagged , | Leave a comment

Microservices architecture pattern

Functionally decompose an enterprise server-side application into collection of services that can be developed, deployed and scaled independently

A successful application typically grows in complexity as developers implement more and more features. At some point the application becomes so complex that modularity breaks down and it degenerates into an unmaintainable ball of mud. Agile development and deployment becomes impossible. The development team can no longer keep up with the needs of the business.

The Microservices Architecture pattern functionally decomposes the application into a collection of services that are easier to understand. Each service can be developed, deployed and scaled independently. The velocity of the development team increases dramatically.

For a full description of this pattern see the Microservices Architecture Pattern.

Posted in microservices, Uncategorized | Tagged , , | Leave a comment

Does each microservice really need its own database?

The short answer is yes. However, before you start hyperventilating about the cost of all those extra Oracle licenses, lets first explore why it is essential to do this and then discuss what is meant by the term ‘database’.

The main benefit of the microservice architecture is that it dramatically improves agility and velocity. That’s because when you  correctly decompose a system into microservices, you can develop and deploy each microservice independently and in parallel with the other services. In order to be able to independently develop microservices , they must be loosely coupled. Each microservice’s persistent data must be private to that service and only accessible via it’s API . If two or more microservices were to share persistent data then you need to carefully coordinate changes to the data’s schema, which would slow down development.

There are a few different ways to keep a service’s persistent data private. You do not need to provision a database server for each service. For example,  if you are using a relational database then the options are:

  • Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
  • Schema-per-service – each service has a database schema that’s private to that service
  • Database-server-per-service – each service has it’s own database server.

Private-tables-per-service and schema-per-service have the lowest overhead.  Using a schema per service is appealing since it makes ownership clearer. For some applications, it might make sense for database intensive services to have their own database server.

It is a good idea to create barriers that enforce this modularity. You could, for example, assign a different database user id to each service and use a database access control mechanism such as grants. Without some kind of barrier to enforce encapsulation, developers will always be tempted to bypass a service’s API and access it’s data directly.

It might also make sense to have a polyglot persistence architecture. For each service you choose the type of database that is best suited to that service’s requirements. For example, a service that does text searches could use ElasticSearch. A service that manipulates a social graph could use Neo4j. It might not make sense to use a relational database for every service.

There are some downsides to keeping a service’s persistent data private. Most notably, it can be challenging to implement business transactions that update data owned by multiple services. Rather than using distributed transaction, you typically must use an eventually consistent, event-driven approach to maintain database consistency.

Another problem, is that it is difficult to implement some queries because you can’t do database joins across the data owned by multiple services. Sometimes, you can join the data within a service. In other situations, you will need to use Command Query Responsibility Segregation (CQRS) and maintain denormalizes views.

Another challenge is that  services sometimes need to share data. For example, let’s imagine that several services need access to user profile data. One option is to encapsulate the user profile data with a service, that’s then called by other services. Another option is to use an event-driven mechanism to replicate data to each service that needs it.

In summary,  it is important that each service’s persistent data is private. There are, however,  a few different ways to accomplish this such as a schema-per-service. Some applications benefit from a polyglot persistence architecture that uses a mixture of database types.  A downside of not sharing databases is that maintaining data consistency and implementing queries is more challenging.

Posted in architecture, microservices | Tagged , , | 16 Comments

Monstrous monoliths – how bad can it get?

I recently posted a survey on twitter asking folks to describe their monolithic application:

So far I’ve had around 12 responses. As of 5/13/15, here are the “winning” answers to each question:

How many lines of code? 2.2M
How many jar files? 418
How long to startup? 12 minutes
How large is the WAR? 373 MB

The ages of the applications ranged from 2 to 13 years.

Do you have some winning metrics about your monstrous monolith you would like to share? If so, fill in this form.

Posted in microservices | Tagged | 1 Comment

Need help with microservices?

These days I’m focussed on microservices.

Microservices patterns

My book Microservices patterns has been published.


Consulting and training

I provide consulting services and training classes that help you get started with microservices.

Contact me if you want to learn more or need help with developing microservices-based applications.

My startup

I am the founder of Eventuate, Inc, a startup that is building a microservices application platform.

Learn more about microservices

Read my book or take a look at learn.microservices.io for patterns, articles, presentations, and example code.

Posted in microservices | Tagged | Leave a comment

Need to install MongoDB, RabbitMQ, or MySQL? Use Docker to simplify dev and test

Almost every interesting application uses at least one infrastructure service such as a database or a message broker. For example, if you tried to build and/or run the Spring Boot-based user registration service you would have discovered that it needs both MongoDB and RabbitMQ.

One option, of course, is to install both of those services on your machine. Unfortunately, installing a service is not always easy. Also, different projects might need different incompatible versions. Moreover, I’m not a fan of cluttering my machine with random services. Fortunately, there is a great way to solve this problem: Docker. You install Docker and use it to run the services that you need as containers.

Docker only directly runs on Linux so if you are using Mac OSX or Windows the first step is to install Boot2Docker (Mac OSX, Window). Boot2Docker installs the Docker command line locally but runs the Docker daemon in a Virtual Box VM.  As a result, the user experience is very similar to running Docker directly on Linux.

The following command runs a VM containing the Docker daemon:

$ boot2docker up

You then need to set some environment variables in order for the Docker command line to know the host and port of the Docker daemon.

$ boot2docker shellinit
Writing /Users/c/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/c/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/c/.boot2docker/certs/boot2docker-vm/key.pem
    export DOCKER_HOST=tcp://192.168.59.103:2376
    export DOCKER_CERT_PATH=/Users/c/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1

You can set those variables by simply running this command:

$ $(boot2docker shellinit)

Once you have done that you are ready to start using Docker from you Mac or Windows machine. For example, docker ps shows the running Docker containers.

$ docker ps
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                                                                                        NAMES
6fcf0779a5e1        dockerfile/mongodb:latest         "mongod"               8 days ago          Up 8 days           28017/tcp, 0.0.0.0:27017->27017/tcp                                                          mongodb                 

So what about running MongoDB and RabbitMQ? One of the great features of the Docker ecosystem is https://hub.docker.com, which is a website where the community shares Docker images. In particular, you can find images for infrastructure services such as RabbitMQ and MongoDB. You can, for example, run MongoDB with the following command:

docker run -d -p 27017:27017 --name mongodb dockerfile/mongodb

This command will, if necessary, download the dockerfile/mongodb image from Docker Hub and launch the container running MongoDB listening on port 27017. Similarly, you can run RabbitMQ using this command:

docker run -d -p 5672:5672 -p 15672:15672 --name rabbitmq dockerfile/rabbitmq

These containers are, of course, running in the Virtual Box VM so you need to connect to them using the appropriate IP address. Fortunately, boot2docker makes that easy to configure:

export DOCKER_HOST_IP=$(boot2docker ip)
export SPRING_DATA_MONGODB_URI=mongodb://${DOCKER_HOST_IP}/userregistration
export SPRING_RABBITMQ_HOST=${DOCKER_HOST_IP}

The “boot2docker ip” command outputs the IP address that you use to connect to the containers.

Now that you have setup the MongoDB and RabbitMQ, you can now run the User Registration Service:

docker run -d -p 8080:8080 -e SPRING_DATA_MONGODB_URI -e SPRING_RABBITMQ_HOST --name sb_rest_svc
sb_rest_svc
Posted in docker, microservices, mongodb, rabbitmq | Tagged , , | Leave a comment