Backups Aren't Simple

readthenotes1

"Obviously, none of this is worth anything if you don’t actually test restores." One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.

PunchyHamster

So is apparently making readable site. Software like https://restic.net/ does a good job. Few choices of backend Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it can mount FUSE directory with all backups on sensible OSes. https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)

bloomingeek

"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)

pizzafeelsright

I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

innocent_name

>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user. and a privesc also.

gerdesj

One of our customers decided to go for off site backups with us - they had been a customer for 15 odd years for other services. Backups are not our core business, its one of the services and I got around to setting them up sooner rather than later. Two weeks later they had a fire in one of their huge printing machines which caused quite a lot of direct damage due to heat and rendered a lot of equipment broken due to smoke/fumes. It was also, as it turns out, a week before the first COVID 19 lockdown in the UK. Their backups landed on a XFS file system with reflinks enabled. I cloned their repo and fired up NFS with their VMs running on our gear. It took me another hour to spin up another OpenVPN server (CA etc) for them to use and sort out a few other details (coffee doesn't brew itself). They ran via VPN out of our data centre for most of the pandemic. That was an utter triumph but I also have some rather less triumphant stories about backups and lack thereof. Let's skip over those 8)

senectus1

Amazon are finding this out atm :-P https://archive.is/Ay7RJ

publlus_enigma

There are four times in my life I have suffered regrettable data loss incidents. The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything. The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files. The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously. And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.

ebrahimh

I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest. All my hosts run the same CoreOS setup ( https://github.com/ebrahim37/infra-template ), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup. I plan to implement it like this: vps1: - restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours homelab: - backrest container, to back up volumes/, do prune/check, replicate repo to offsite - rest-server container, will store backups from vps1, homelab, offsite offsite: - restic container, backs up volumes/ to homelab every 24 hours - rest-server container, store copy of backups from homelab Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc. The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost

AdieuToLogic

A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying: We are not in the backup business. We are in the restoration business. 0 - https://en.wikipedia.org/wiki/Backup_Exec

snizovtsev

I'm polishing my backup system right now, and found that `tar | zstd | gpg` plus a small vibe-coded wrapper is pretty much all I need. GNU tar has its own incremental index via `--listed-incremental=FILE`. Unlike Borg or Restic, which have their own more complicated repository formats, this leaves me with just one additional file (the `.snar`) alongside a dumb, portable full-disk tar archive. The nice part is that, unlike repositories that require both read/write access patterns, tar can compute deltas using only the small `.snar` file, while the main `full-disk.tar` can remain buried in write-only Glacier storage. This makes it a much better fit for Glacier's write-once model and 180-day minimum retention. My current plan is to upload a full-disk tar to an external HDD + Glacier on a weekly cadence, and the `.snar`-based deltas daily. That gives me a pretty simple cloud backup solution for a few bucks a month (after burning through $100+ of free credits).

chasil

For my Oracle Linux systems, I rely on the vendor ReaR RPM package for the baseline backup. This is similar to the "Ignite" utility that I used in my HP-UX days. https://relax-and-recover.org/ This is two-fold useful, first as it enables boot from backup media for total recovery, second in that it creates a backup.tar.gz that holds everything that was not explitictly ignored in the /etc/rear/local.conf file. I'm running Oracle databases on these systems (and the local.conf is configured to ignore the datafile directories). I have standby databases (not Dataguard in that I don't have online redolog replication) that allow me to recover them to the primary in a disaster, with some loss of committed transactions. I finally have rsync configured for some scratch temporary files. I have dallied with btrfs snapshot replication for home directories (in a loopback mount). My vendor support (via a CSI number) has been on and off, so I don't use btrfs in areas that we need it most. My replacements do not like the complexity (and I am retiring).

darvo31

Yep, backup is a verb, restore is a noun. Testing restores is where the real work begins.

smallerfish

On my home PC I use restic, and (shockingly) I run it manually (via a script). I have a terminal banner that nags me when backup hasn't run in 24 hours, or displays spanish verb tables when it has (so I will notice if the banner doesn't show). I've been burned in the past losing a backup cron, and dead man's snitch seemed overkill for my needs here. And I've also forgotten what backup system was in place on a prior install, and it took me way longer than I liked figuring out how it was setup when I needed to do a restore. Obviously for production systems automation & monitoring is the way. I kind of like this setup on my own machine though.

mrheosuper

As an average-techie person, the only piece of data that i truly practice 3-2-1 is photos and videos. 1 on my phone, 1 on my small server running Immich, and 1 is on Google Photo(sadly, the quality is loss, but i can live with that). Most of other data i only have 2 backup, usually 1 at my local machine(code) and 1 online(github). Yeah it's not ideal, but another question to ask yourself, is it truly worth it. I trust the engineers at MS doing much better job at backing up their data than me.

bakinator

On the topic of backups, 7zip recently changed its handling of the FILE_ATTRIBUTE_ARCHIVE bit on unzipped files which caused an issue with some old software I was using. This is from the old DOS days with the BACKUP/RESTORE commands - the bit was set if any backup was made of the file. It's comically naive because it only assumes a single backup and this bit was forgotten to the sands of time until it was disturbed by some meddling kids!

m463

I've given out jwz's link to several people: https://www.jwz.org/doc/backups.html (maybe copy/paste this into browser instead of following link, since referrer from hn apparently does something)

justsomehnguy

> So our first principle is to have a backup, i.e. a copy of your files someplace else. So far so good. So who are we? A household with a localhost administrator?[0] Then you are surely doesn't need encrypted, chunk-level corporate level bla-bla-bla. You need a Syncthing copy to some other device not at home - and an additional backup procedure to maintain the history and protect against PEBKAC errors - which can run on your local Syncthing copy or /and the other one. And no, nobody needs your 555GB of RAWs of sunsets/flowers/precious_family_moments you shot - not even you.[1] [0] well considering the tone of TFA and most of the comments here [1] that's Instagram/Google Photos/iCloud for nowadays, totally with "Remember this day N years ago?"

crossroadsguy

If you don't control your data size, it will end up controlling you and your backup choices, which will eventually lead to many avoidable, disastrous outcomes. Somehow, this gets missed in data storage and backup planning.

jongjong

When it comes to software, it's almost never simple. What people think of as simple is rarely so. For example: Requirement: "Just copy a file from folder A to folder B." Minimal translation: "Implement the capability to copy any file, in any format and of any size from SharePoint, located at a configurable path, with appropriate authentication and access control checks, then stream it in chunks to a different, configurable path inside an S3 bucket, also with appropriate authentication and access controls in place. Ensure that any disruption in either service which may occur while the file is streaming can be recovered from at the point it failed instead of having to restart from the beginning. Ensure that the retry mechanism is built-in and that the retry window is configurable; if the file cannot be copied within the specific time window, then an error should be sent via email to a configured email address. Ensure that the entire transfer is encrypted in transit... If the file happens to be a folder, then you must copy across all of its contents recursively up to a certain configured MAX_DEPTH to avoid DoS and ensuring that the system does not get caught in an infinite loop due to symlinks pointing to a parent folder... In this case, send an email to the configured address... Etc... Etc..." And the thing is; if you tell AI "Just copy a file from folder A to folder B." - It will not meet your 'basic' needs because even if it does a great job at filling the gaps in your requirements, it will still take shortcuts. In order for an AI to avoid taking shortcuts, it would have to make you fill out a questionnaire and make you sign up for and configure services; it would not be a pleasant user experience. The user experience cannot be pleasant, because the AI cannot read your mind and it cannot know your intent.