Need help with microservices?

These days I’m focussed on microservices.

I provide consulting services and training classes that help you get started with microservices.

I am the founder of Eventuate, Inc, a startup that is building a microservices application platform.

Contact me if you want to learn more or need help with developing microservice-based applications.

Here is some of my latest articles and presentations:

Posted in microservices | Tagged | Leave a comment

API Gateway pattern

Have a service that acts as the sole entry point into a microservices-based application

Consider a mobile shopping application.  The product information that it needs to show to the user includes the description, price, reviews, recommendations and shipping information. The ownership of that data is spread among a number of microservices. In theory, the application could access each microservice separately. But in practice, it makes more sense for the mobile application to retrieve the needed information by making a single request to a service known as an API Gateway, which aggregates data from multiple services.

For a full description of this pattern see the API Gateway Pattern.

Posted in microservices, Uncategorized | Tagged , | Leave a comment

Microservices architecture pattern

Functionally decompose an enterprise server-side application into collection of services that can be developed, deployed and scaled independently

A successful application typically grows in complexity as developers implement more and more features. At some point the application becomes so complex that modularity breaks down and it degenerates into an unmaintainable ball of mud. Agile development and deployment becomes impossible. The development team can no longer keep up with the needs of the business.

The Microservices Architecture pattern functionally decomposes the application into a collection of services that are easier to understand. Each service can be developed, deployed and scaled independently. The velocity of the development team increases dramatically.

For a full description of this pattern see the Microservices Architecture Pattern.

Posted in microservices, Uncategorized | Tagged , , | Leave a comment

Does each microservice really need its own database?

The short answer is yes. However, before you start hyperventilating about the cost of all those extra Oracle licenses, lets first explore why it is essential to do this and then discuss what is meant by the term ‘database’.

The main benefit of the microservice architecture is that it dramatically improves agility and velocity. That’s because when you  correctly decompose a system into microservices, you can develop and deploy each microservice independently and in parallel with the other services. In order to be able to independently develop microservices , they must be loosely coupled. Each microservice’s persistent data must be private to that service and only accessible via it’s API . If two or more microservices were to share persistent data then you need to carefully coordinate changes to the data’s schema, which would slow down development.

There are a few different ways to keep a service’s persistent data private. You do not need to provision a database server for each service. For example,  if you are using a relational database then the options are:

  • Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
  • Schema-per-service – each service has a database schema that’s private to that service
  • Database-server-per-service – each service has it’s own database server.

Private-tables-per-service and schema-per-service have the lowest overhead.  Using a schema per service is appealing since it makes ownership clearer. For some applications, it might make sense for database intensive services to have their own database server.

It is a good idea to create barriers that enforce this modularity. You could, for example, assign a different database user id to each service and use a database access control mechanism such as grants. Without some kind of barrier to enforce encapsulation, developers will always be tempted to bypass a service’s API and access it’s data directly.

It might also make sense to have a polyglot persistence architecture. For each service you choose the type of database that is best suited to that service’s requirements. For example, a service that does text searches could use ElasticSearch. A service that manipulates a social graph could use Neo4j. It might not make sense to use a relational database for every service.

There are some downsides to keeping a service’s persistent data private. Most notably, it can be challenging to implement business transactions that update data owned by multiple services. Rather than using distributed transaction, you typically must use an eventually consistent, event-driven approach to maintain database consistency.

Another problem, is that it is difficult to implement some queries because you can’t do database joins across the data owned by multiple services. Sometimes, you can join the data within a service. In other situations, you will need to use Command Query Responsibility Segregation (CQRS) and maintain denormalizes views.

Another challenge is that  services sometimes need to share data. For example, let’s imagine that several services need access to user profile data. One option is to encapsulate the user profile data with a service, that’s then called by other services. Another option is to use an event-driven mechanism to replicate data to each service that needs it.

In summary,  it is important that each service’s persistent data is private. There are, however,  a few different ways to accomplish this such as a schema-per-service. Some applications benefit from a polyglot persistence architecture that uses a mixture of database types.  A downside of not sharing databases is that maintaining data consistency and implementing queries is more challenging.

Posted in architecture, microservices | Tagged , , | 4 Comments

Monstrous monoliths – how bad can it get?

I recently posted a survey on twitter asking folks to describe their monolithic application:

So far I’ve had around 12 responses. As of 5/13/15, here are the “winning” answers to each question:

How many lines of code? 2.2M
How many jar files? 418
How long to startup? 12 minutes
How large is the WAR? 373 MB

The ages of the applications ranged from 2 to 13 years.

Do you have some winning metrics about your monstrous monolith you would like to share? If so, fill in this form.

Posted in microservices | Tagged | 1 Comment

Need to install MongoDB, RabbitMQ, or MySQL? Use Docker to simplify dev and test

Almost every interesting application uses at least one infrastructure service such as a database or a message broker. For example, if you tried to build and/or run the Spring Boot-based user registration service you would have discovered that it needs both MongoDB and RabbitMQ.

One option, of course, is to install both of those services on your machine. Unfortunately, installing a service is not always easy. Also, different projects might need different incompatible versions. Moreover, I’m not a fan of cluttering my machine with random services. Fortunately, there is a great way to solve this problem: Docker. You install Docker and use it to run the services that you need as containers.

Docker only directly runs on Linux so if you are using Mac OSX or Windows the first step is to install Boot2Docker (Mac OSX, Window). Boot2Docker installs the Docker command line locally but runs the Docker daemon in a Virtual Box VM.  As a result, the user experience is very similar to running Docker directly on Linux.

The following command runs a VM containing the Docker daemon:

$ boot2docker up

You then need to set some environment variables in order for the Docker command line to know the host and port of the Docker daemon.

$ boot2docker shellinit
Writing /Users/c/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/c/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/c/.boot2docker/certs/boot2docker-vm/key.pem
    export DOCKER_HOST=tcp://192.168.59.103:2376
    export DOCKER_CERT_PATH=/Users/c/.boot2docker/certs/boot2docker-vm
    export DOCKER_TLS_VERIFY=1

You can set those variables by simply running this command:

$ $(boot2docker shellinit)

Once you have done that you are ready to start using Docker from you Mac or Windows machine. For example, docker ps shows the running Docker containers.

$ docker ps
CONTAINER ID        IMAGE                             COMMAND                CREATED             STATUS              PORTS                                                                                        NAMES
6fcf0779a5e1        dockerfile/mongodb:latest         "mongod"               8 days ago          Up 8 days           28017/tcp, 0.0.0.0:27017->27017/tcp                                                          mongodb                 

So what about running MongoDB and RabbitMQ? One of the great features of the Docker ecosystem is https://hub.docker.com, which is a website where the community shares Docker images. In particular, you can find images for infrastructure services such as RabbitMQ and MongoDB. You can, for example, run MongoDB with the following command:

docker run -d -p 27017:27017 --name mongodb dockerfile/mongodb

This command will, if necessary, download the dockerfile/mongodb image from Docker Hub and launch the container running MongoDB listening on port 27017. Similarly, you can run RabbitMQ using this command:

docker run -d -p 5672:5672 -p 15672:15672 --name rabbitmq dockerfile/rabbitmq

These containers are, of course, running in the Virtual Box VM so you need to connect to them using the appropriate IP address. Fortunately, boot2docker makes that easy to configure:

export DOCKER_HOST_IP=$(boot2docker ip)
export SPRING_DATA_MONGODB_URI=mongodb://${DOCKER_HOST_IP}/userregistration
export SPRING_RABBITMQ_HOST=${DOCKER_HOST_IP}

The “boot2docker ip” command outputs the IP address that you use to connect to the containers.

Now that you have setup the MongoDB and RabbitMQ, you can now run the User Registration Service:

docker run -d -p 8080:8080 -e SPRING_DATA_MONGODB_URI -e SPRING_RABBITMQ_HOST --name sb_rest_svc
sb_rest_svc
Posted in docker, microservices, mongodb, rabbitmq | Tagged , , | Leave a comment

The world is an amazing place – the photos on my business cards

I like to order business cards from moo.com since they let you put photos on the back.

Here are the photos from the January 2015 order.

P8210484

P8200426

P8170098

P8132326

IMG_8327

IMG_7967

IMG_6338

IMG_6201

IMG_2864 - Version 2

IMG_2649

IMG_2172

IMG_1615

IMG_1576

Posted in photos, travel | Tagged , | Leave a comment