So you have too many (micro-) services..
It is quite common to have docker compose for your local development. Sometimes even for production. But as your project grows you have more and more services and containers. It takes a lot of resources to run them at once.
To prevent docker compose from starting all services, you can use service profiles for this.
Official documentation contains some examples, but I will provide another real-world example below.
Yet another web app example
Let’s say our application consists of the frontend application, backend API, database, two background workers and one cronjob:
services: db: image: postgres frontend: image: your-app/frontend backend: image: your-app/backend command: api backend-worker-1: image: your-app/backend command: worker-1 backend-worker-2: image: your-app/backend command: worker-2 cronjob: image: your-app/backend command: cronjob
So if you run
docker compose up -d now, it will start all 6 services at once. Quite possible, you only wanted to start backend and frontend.
You can introduce profiles for the services that don’t need to start:
backend-worker-1: image: your-app/backend command: worker-1 ++ profiles: ["backend", "worker"] backend-worker-2: image: your-app/backend command: worker-2 ++ profiles: ["backend", "worker"] cronjob: image: your-app/backend command: cronjob ++ profiles: ["backend", "cron"]
With those changes when you run
docker compose up -d, those extra services will no longer start. Only
If you want to run those services, you have three options:
docker compose up -d backend-worker-2this will auto-enable profile and start specific service
docker compose --profile worker up -dwill start both workers, as both of them have this profile. Keep in mind, profile argument is positional, you cannot use it after
COMPOSE_PROFILES=backend docker compose up -dwill start all backend workers and cronjob using env var.
I’ve noticed that second approach is quite tedious. You need to remember which profile you used. This will be a problem once you try to stop all containers with
docker compose down and you will notice that those extra workers didn’t stop.
Note: if you use networks, you might additionally see this error:
failed to remove network network_name: Error response from daemon: error while removing network: network network_name id f3f6e7fc5 has active endpoints.
This is because
downcommand didn’t know about those extra profiles. To solve this you need to call it with exactly same profile names:
docker compose --profile worker down.
And with environmental variable you can do:
export COMPOSE_PROFILES=backend docker compose up -d # will start all # ... docker compose down # this will stop all according to COMPOSE_PROFILES
Of course, this only works in the same terminal session, or if you have this variable set globally.
We started using profiles in my current project - Taskcluster because of the big amount of services.
Currently Taskcluster’s docker-compose.yml defines more than 50 services, most of which are periodic cronjobs or background workers that are not needed during local development.