Jan-Lukas Else

Thoughts of an IT expert

How I sync my Nextcloud with an S3 bucket using Drone CI and rclone

Published on 2021-03-13 in 👨‍💻 Dev

Short link: https://b.jlel.se/s/3fc

⚠️ This entry is already over one year old. It may no longer be up to date. Opinions may have changed.

A few days ago, an OVH data center went up in flames and was completely destroyed. This reminded me that it is important to store backups in another place on earth if possible, so better not in the same data center as the server that might burn down.

Lately, I’ve been using the Nextcloud instance I rent from Hetzner as my backup storage. However, that is in the same data center as some of my virtual servers, so not very secure. Instead, I now have backups set up with Scaleway S3 buckets. Using restic and rclone, my servers are backed up to storage buckets at Scaleway in Paris.

But in addition to that, I also set up a regular (daily) sync of my Nextcloud to a storage bucket. rclone makes it easy to copy files back and forth between different cloud storage providers and across different protocols.

On my Git server, I created a private repository that contains both the rclone configuration and a configuration for my self-hosted Drone CI instance. In Drone CI, a container is started with rclone, which then syncs the files from Nextcloud with the files in the Scaleway bucket.

kind: pipeline
type: docker
name: default

steps:
- name: sync
  image: rclone/rclone:1.54
  commands:
  - "rclone --config rclone.conf sync nextcloud: scaleway:mynextcloudbucket"

In the Drone settings it is then possible to set up a cronjob.

And I have to say that works pretty well. The VPS has a fast network connection to the Nextcloud, but also to Scaleway and within minutes many gigabytes can be copied. If I tried to do that from home, it would take hours.

Tags: Backups, Nextcloud, rclone

Interactions & Comments

Jan-Lukas Else I moved my files to OneDrive Probably some readers will now think about this post: “What’s wrong with you? Where have your ideals and principles gone? Microsoft is evil!” But why do I actually have to justify myself? I have my reasons and I am free in what I can and cannot do.I just cancelled my Nextcloud at Hetzner and moved my files to Microsoft OneDrive. I bought a discounted Microsoft 365 Single subscription for that.Why am I switching to OneDrive?The flood in western Germany got me thinking. What if something like t…
- Anonymous For me, there is no ideological element to where I choose to store my backups. Thanks to PGP, you can get the same degree of privacy anywhere. In this respect, OneDrive is a perfectly good choice. These days, though, I'm using BorgBackup, which unfortunately rules out the cheapest options (b2, OneDrive, and so on), but offers some unique features in return.
- Frank Meeuwsen Bookmark: I moved my files to OneDrive I moved my files to OneDrive door Jan-Lukas ElseJan-Lukas Else (jlelse.blog) Probably some readers will now think about this post: “What’s wrong with you? Where have your ideals and principles gone? Microsoft is evil!” But why do I actually have to justify myself? I have my reasons and I am free in what I can and cannot do. I just cancelled my Nextcloud at Hetzner and … Ik vind dat Jan-Lukas duidelijk argumenten geeft om voor een Big Tech provider als Microsoft te kiezen voor de backups van …
- Kristófer Reykjalín @en @jle your reasons are almost precisely why I choose to use iCloud instead of self-hosting some other cloud software.For me it’s especially about the tight integration with my Mac and iPhone.For files I care about keeping “safe” I use #Cryptomator to encrypt the files before they’re uploaded, but that’s just the paranoia in me speaking when it comes down to it :)
- Vít Skalický :manjaro: @enIf I didn't host my own nextcloud, I would pay for Mega.
- Jan-Lukas Else Mein August ‘21 im Rückblick Eigentlich hatte ich vor, meinen Monatsrückblick erst morgen zu schreiben, aber das wäre doch blöd diese Routine zu unterbrechen. Also hier jetzt ein kurzer Überblick, was im August bei mir los war:Anfang des Monats habe ich mich ein wenig mit dem Ort, an dem ich online meine Dateien ablege, auseinandergesetzt. Von Hetzners Storage Share bin ich nun zu Microsofts OneDrive gewechselt. Eine kontroverse Entscheidung, aber ich denke sie war richtig. Ich vertraue darauf, dass meine Dateien nun bes…
- Jan-Lukas Else My August ‘21 in Review Actually, I had planned to write my monthly review only tomorrow, but it would be stupid to interrupt this routine. So here is a short overview of what was going on in August:At the beginning of the month, I spent some time looking at where I store my files online. I’ve now switched from Hetzner’s Storage Share to Microsoft’s OneDrive. A controversial decision, but I think it was the right one. I trust that my files are now better protected against data loss. Plus, for almost the same price, …
- Jan-Lukas Else How I archive my favorite TV show to get rid of Amazon Prime I’m a fan of the German TV series “Notruf Hafenkante”, which is shown on the public broadcaster ZDF and is also available on their website. New episodes are released weekly, and older episodes are also made available on a regular basis. After a few months, however, they are no longer available for free. At the moment I have to buy an extra subscription for a Prime Video Channel to be able to watch all the old episodes. But that makes me dependent on Amazon, and I don’t like that. So is there …