MF-655 Proper usage of docker volumes #657

chombium · 2019-03-18T06:44:25Z

What does this do?

adds names for the persistent volumes that we already use them
adds filters for the cleanghost to filter out only the containers which we have created
adds a switch which when is present the persistent volumes will not be deleted. The default behavior or volume deletion is not changed.

Which issue(s) does this PR fix/relate to?

Resolves #655

List any changes that modify/break current functionality

none

Have you included tests for your changes?

no

Did you document any new/modified functionality?

added description about the changes in the dev guide

Notes

See the issue for more details

chombium · 2019-03-18T06:49:01Z

I'm still working on this. I've opened the pull request to get feedback.

The current state is:

$ docker volume ls
DRIVER              VOLUME NAME
local               cassandra-writer_mainflux-cassandra-volume
local               docker_mainflux-redis-volume
local               docker_mainflux-things-db-volume
local               docker_mainflux-things-redis-volume
local               docker_mainflux-users-db-volume
local               influxdb-writer_mainflux-grafana-volume
local               influxdb-writer_mainflux-influxdb-volume
local               mongodb-writer_mainflux-mongodb-configdb-volume
local               mongodb-writer_mainflux-mongodb-db-volume

the make cleanghost make task works only with "mainflux artifacts" and the delete switch works.
I still need to do more testing and eventually adjust the commands in cleandocker and cleanghost make tasks

codecov-io · 2019-03-18T06:54:12Z

Codecov Report

Merging #657 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #657   +/-   ##
=======================================
  Coverage   85.58%   85.58%           
=======================================
  Files          64       64           
  Lines        4011     4011           
=======================================
  Hits         3433     3433           
  Misses        395      395           
  Partials      183      183

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update baebe8a...15be7d7. Read the comment docs.

chombium · 2019-04-16T13:49:37Z

The fixes which this PR brings are:

no more "ghost volumes"
better filtering for the docker containers, images and volumes, so that we are sure that we are only working with the artifacts that we created
optional data preservation (the persistent volumes are not deleted) when executing make cleandocker. The default behavior stays the same; no parameter, everything is deleted

The summary of the changes:

added persistent volumes in the docker-compose.yml files for all of the components (we were already using persistent volumes anyways)
removed the make cleanghost target as there are no more ghost volumes
added switch to the make cleandocker so that the persistent volumes are not deleted. The command is make spv=true cleandocker.
added documentation of the switch in the dev guide
added filtering for all of the docker commands, so that we work only with our containers, images and volumes

nmarcetic · 2019-04-17T17:43:00Z

@chombium Can you pls checkout conflicts and update branch. Thanks!

chombium · 2019-04-18T07:14:46Z

@nmarcetic done ;)

anovakovic01

Looks good, just fix these two typos. Great job @chombium!

anovakovic01 · 2019-04-18T08:19:04Z

docs/dev-guide.md

+__Note:__ Please store your customizations to some folder outside the Mainflux's source folder and maybe add them to some other git repository. You can always apply your customizations by pointing to the right file using `docker-compose -f ...`.
+
+
+### Cleaning up your dockerized Mainlfux setup


anovakovic01 · 2019-04-18T08:19:18Z

docs/dev-guide.md

+
+
+### Cleaning up your dockerized Mainlfux setup
+If you want to clean your whole dockerized Mainlfux installation you can use the `make cleandocker` command. Please note that __by default the `make cleandocker` command will stop and delete all of the containers, images and persistent volumes__. If you want to keep gathered data in the system (the persistent volumes) please use the following command `make spv=true cleandocker` (spv = skip persistent volumes). This form of the command will stop and delete the containers and images, but won't delete the persistent volumes.


anovakovic01 · 2019-04-18T08:35:01Z

docs/dev-guide.md

@@ -65,7 +65,7 @@ make docker_http

 > N.B. Mainflux creates `FROM scratch` docker containers which are compact and small in size.

-> N.B. The `things-db` and `users-db` containers are built from a vanilla PostgreSQL docker image downloaded from docker hub which does not persist the data when these containers are rebuilt. Thus, __rebuilding of all docker containers with `make dockers` or rebuilding the `things-db` and `users-db` containers separately with `make docker_things-db` and `make docker_users-db` respectively, will cause data loss. All your users, things, channels and connections between them will be lost!__ As we use this setup only for development, we don't guarantee any permanent data persistence. If you need to retain the data between the container rebuilds you can attach volume to the `things-db` and `users-db` containers. Check the official docs on how to use volumes [here](https://docs.docker.com/storage/volumes/) and [here](https://docs.docker.com/compose/compose-file/#volumes). For examples on how to add persistent volumes check the [Overriding the default docker-compose configuration](#overriding-the-default-docker-compose-configuration) section.
+> N.B. The `things-db` and `users-db` containers are built from a vanilla PostgreSQL docker image downloaded from docker hub which does not persist the data when these containers are rebuilt. Thus, __rebuilding of all docker containers with `make dockers` or rebuilding the `things-db` and `users-db` containers separately with `make docker_things-db` and `make docker_users-db` respectively, will cause data loss. All your users, things, channels and connections between them will be lost!__ As we use this setup only for development, we don't guarantee any permanent data persistence. If you want to update your Mainflux dockerized installation and want to keep your data, use `make spv=true cleandocker` to clean the containers and images and keep the data and then `make run` to update the images and the containers. Check the [Cleaning up your dockerized Mainlfux setup](#cleaning-up-your-dockerized-mainlfux-setup) section for details. Please note that this kind of updating might not work if there are database changes.


chombium · 2019-04-18T09:50:49Z

@anovakovic01 really good catch. LOLed to myself. The same type in two consecutive sentences :)

anovakovic01 · 2019-04-18T10:16:15Z

docs/dev-guide.md

@@ -65,7 +65,7 @@ make docker_http

 > N.B. Mainflux creates `FROM scratch` docker containers which are compact and small in size.

-> N.B. The `things-db` and `users-db` containers are built from a vanilla PostgreSQL docker image downloaded from docker hub which does not persist the data when these containers are rebuilt. Thus, __rebuilding of all docker containers with `make dockers` or rebuilding the `things-db` and `users-db` containers separately with `make docker_things-db` and `make docker_users-db` respectively, will cause data loss. All your users, things, channels and connections between them will be lost!__ As we use this setup only for development, we don't guarantee any permanent data persistence. If you need to retain the data between the container rebuilds you can attach volume to the `things-db` and `users-db` containers. Check the official docs on how to use volumes [here](https://docs.docker.com/storage/volumes/) and [here](https://docs.docker.com/compose/compose-file/#volumes). For examples on how to add persistent volumes check the [Overriding the default docker-compose configuration](#overriding-the-default-docker-compose-configuration) section.
+> N.B. The `things-db` and `users-db` containers are built from a vanilla PostgreSQL docker image downloaded from docker hub which does not persist the data when these containers are rebuilt. Thus, __rebuilding of all docker containers with `make dockers` or rebuilding the `things-db` and `users-db` containers separately with `make docker_things-db` and `make docker_users-db` respectively, will cause data loss. All your users, things, channels and connections between them will be lost!__ As we use this setup only for development, we don't guarantee any permanent data persistence. If you want to update your Mainflux dockerized installation and want to keep your data, use `make spv=true cleandocker` to clean the containers and images and keep the data and then `make run` to update the images and the containers. Check the [Cleaning up your dockerized Mainflux setup](#cleaning-up-your-dockerized-mainlfux-setup) section for details. Please note that this kind of updating might not work if there are database changes.


Just update the link #cleaning-up-your-dockerized-mainflux-setup.

thanks once again. I've missed that one :( it is fixed now

anovakovic01

@chombium Don't worry, it happens. At least you were consistent. 😄 LGTM!

anovakovic01

@chombium please rebase

Signed-off-by: Jovan Kostovski <[email protected]>

chombium · 2019-04-18T15:01:52Z

@anovakovic01 done

anovakovic01

LGTM!

drasko · 2019-04-18T15:32:36Z

docs/dev-guide.md

+
+
+### Cleaning up your dockerized Mainflux setup
+If you want to clean your whole dockerized Mainflux installation you can use the `make cleandocker` command. Please note that __by default the `make cleandocker` command will stop and delete all of the containers, images and persistent volumes__. If you want to keep gathered data in the system (the persistent volumes) please use the following command `make spv=true cleandocker` (spv = skip persistent volumes). This form of the command will stop and delete the containers and images, but won't delete the persistent volumes.


I think it is better to skip persistent volume by default, but add explicitly extra parameter if you want to delete it also - like pv=true.

I think this can help avoinding errors and making idiot-proof solution. I see more a situations where I cleaned dockers, but I fortgot vlumes - so I run the command again with correct param, no big deal. However, the other way around, if I cleaned everything because I forgot a param - it is too late :).

I let you decide on this, but I would personaly do it like this.

@drasko I completely agree with you from data persistence view point. My idea was to to leave the current make cleandocker behavior as it is and offer an option to skip the deletetion of the persistent volumes.

On the second thought, taking in account all the complaints about data loss, I would by default skip the volume deletion.

I hope the possible running of the same command twice by the @mainflux/contributors won't be a problem.

I will prepare a new pr

* added filter and a switch for volume deletion Signed-off-by: Jovan Kostovski <[email protected]> * added persistent volume configuration Signed-off-by: Jovan Kostovski <[email protected]>

chombium added the work-in-progress label Mar 18, 2019

chombium changed the title ~~Mf 655~~ MF-655 Proper usage of docker volumes Mar 18, 2019

chombium force-pushed the MF-655 branch from 55e5fa8 to 02e8ab1 Compare April 16, 2019 13:26

chombium removed the work-in-progress label Apr 16, 2019

chombium force-pushed the MF-655 branch from 02e8ab1 to 17bbdc2 Compare April 17, 2019 20:20

anovakovic01 suggested changes Apr 18, 2019

View reviewed changes

chombium force-pushed the MF-655 branch from 17bbdc2 to 79418e7 Compare April 18, 2019 09:21

anovakovic01 reviewed Apr 18, 2019

View reviewed changes

chombium force-pushed the MF-655 branch from 79418e7 to 0a7814e Compare April 18, 2019 10:35

anovakovic01 previously approved these changes Apr 18, 2019

View reviewed changes

manuio previously approved these changes Apr 18, 2019

View reviewed changes

anovakovic01 reviewed Apr 18, 2019

View reviewed changes

chombium added 2 commits April 18, 2019 17:00

added filter and a switch for volume deletion

a4a9bee

Signed-off-by: Jovan Kostovski <[email protected]>

added persistent volume configuration

15be7d7

Signed-off-by: Jovan Kostovski <[email protected]>

chombium dismissed stale reviews from manuio and anovakovic01 via 15be7d7 April 18, 2019 15:01

chombium force-pushed the MF-655 branch from 0a7814e to 15be7d7 Compare April 18, 2019 15:01

anovakovic01 approved these changes Apr 18, 2019

View reviewed changes

manuio approved these changes Apr 18, 2019

View reviewed changes

anovakovic01 merged commit a30666d into absmach:master Apr 18, 2019

drasko reviewed Apr 18, 2019

View reviewed changes

chombium deleted the MF-655 branch April 20, 2019 11:37

chombium mentioned this pull request Apr 20, 2019

NOISSUE - follow up of PR #657 #723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MF-655 Proper usage of docker volumes #657

MF-655 Proper usage of docker volumes #657

chombium commented Mar 18, 2019 •

edited

Loading

chombium commented Mar 18, 2019

codecov-io commented Mar 18, 2019 •

edited

Loading

chombium commented Apr 16, 2019

nmarcetic commented Apr 17, 2019

chombium commented Apr 18, 2019

anovakovic01 left a comment

anovakovic01 Apr 18, 2019

anovakovic01 Apr 18, 2019

anovakovic01 Apr 18, 2019

chombium commented Apr 18, 2019

anovakovic01 Apr 18, 2019

chombium Apr 18, 2019

anovakovic01 left a comment

anovakovic01 left a comment

chombium commented Apr 18, 2019

anovakovic01 left a comment

drasko Apr 18, 2019 •

edited

Loading

chombium Apr 18, 2019

		__Note:__ Please store your customizations to some folder outside the Mainflux's source folder and maybe add them to some other git repository. You can always apply your customizations by pointing to the right file using `docker-compose -f ...`.


		### Cleaning up your dockerized Mainlfux setup



		### Cleaning up your dockerized Mainlfux setup
		If you want to clean your whole dockerized Mainlfux installation you can use the `make cleandocker` command. Please note that __by default the `make cleandocker` command will stop and delete all of the containers, images and persistent volumes__. If you want to keep gathered data in the system (the persistent volumes) please use the following command `make spv=true cleandocker` (spv = skip persistent volumes). This form of the command will stop and delete the containers and images, but won't delete the persistent volumes.



		### Cleaning up your dockerized Mainflux setup
		If you want to clean your whole dockerized Mainflux installation you can use the `make cleandocker` command. Please note that __by default the `make cleandocker` command will stop and delete all of the containers, images and persistent volumes__. If you want to keep gathered data in the system (the persistent volumes) please use the following command `make spv=true cleandocker` (spv = skip persistent volumes). This form of the command will stop and delete the containers and images, but won't delete the persistent volumes.

MF-655 Proper usage of docker volumes #657

MF-655 Proper usage of docker volumes #657

Conversation

chombium commented Mar 18, 2019 • edited Loading

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

Notes

chombium commented Mar 18, 2019

codecov-io commented Mar 18, 2019 • edited Loading

Codecov Report

chombium commented Apr 16, 2019

nmarcetic commented Apr 17, 2019

chombium commented Apr 18, 2019

anovakovic01 left a comment

Choose a reason for hiding this comment

anovakovic01 Apr 18, 2019

Choose a reason for hiding this comment

anovakovic01 Apr 18, 2019

Choose a reason for hiding this comment

anovakovic01 Apr 18, 2019

Choose a reason for hiding this comment

chombium commented Apr 18, 2019

anovakovic01 Apr 18, 2019

Choose a reason for hiding this comment

chombium Apr 18, 2019

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

chombium commented Apr 18, 2019

anovakovic01 left a comment

Choose a reason for hiding this comment

drasko Apr 18, 2019 • edited Loading

Choose a reason for hiding this comment

chombium Apr 18, 2019

Choose a reason for hiding this comment

chombium commented Mar 18, 2019 •

edited

Loading

codecov-io commented Mar 18, 2019 •

edited

Loading

drasko Apr 18, 2019 •

edited

Loading