Personal Documentation

Methods and reasoning on keeping personal data safe, secure, and easily accessible

It’s scary how important factual and accurate documents are to a country’s national history. Here in America, where I live, our historical documents (e.g. The Constitution, Bill of Rights, Letters from an American Farmer) lay not only the groundwork for our nation but hold the conversations, actions, and thought processes of generations before my own. Yesterday, Richardson1 mentioned that Trump supporters submited false election certificates to the National Archives and Records Administration (NARA), actually affixing the state seals of Michigan and Arizona to the papers (I won’t mention how ridiculous this attempt at subverting our democratic process was).

There’s a common quote that history is written by the victors2. How subversive and detrimental it would be for anyone—especially the losers3—to rewrite their own current fiction into historical fact. The consequences would be dire; we would make decisions based on lies, alter programs that actually benefit our society, and perhaps cause a reckoning that we could not reckon with.

But that is not what I want to write about today. Rather, reading yesterday’s post added another puzzle piece to my thinking around one’s own documentation process, how one keeps an accurate record of their life, thoughts, and schedules, all the while keeping that documentation secure and private without the highly likely scenario of those documents being leaked or traded for advertising dollars. The one thing that is guaranteed to always be one’s own is our thoughts and our minds. But as we move more and more into a digital space, especially during the pandemic years, we have to accept that our thoughts are ever more present outside of our minds.

As I’ve stated, I’m a writer. Most of the time, I don’t know exactly what I’m thinking until I write the thoughts down, discuss them with people I trust, and then rewrite again. As you might expect, I have a lot of words written. I tend to make decisions based on the patterns of these words. Before moving from Boston to New York City, six months worth of diary writing showed how unhappy I was. It wasn’t until I reread those words that I saw it was a persistent ache, which spurred me to take action.

The above may work for something as subjective as feelings, where thoughts are written on lined paper and stored in a bookcase. But when it comes to creating a personal knowledge-base, or managing my finances with Ledger4, storing pictures on my phone to share with my girlfriend, or simply tracking a to-do list accessible across devices, keeping information in a paper-bound notebook doesn’t exactly make things searchable or allow me to run calculations, does it? Add to this that I am a fairly private person, regardless of what I put on this site. I prefer to control where and how I store information. Lastly, I run Linux on my laptop and desktop and own an Android phone.

Requirements

In the past, I’ve often used Git5 to manage syncing files and backing up. It’s usually free, since I use the Free tier of GitLab, yet I’m still using a third-party to store and host my Git repositories. Here’s what I need:

The Setup

Believe it or not, I’ve been able to fulfill all of my requirements with a few open-source programs and a Linode server.

First, I consider my desktop, a used Dell Optiplex 7050 off a three year lease that runs Pop!_OS6, as my one source of truth. I plan to purchase a NAS (network attached storage) at some point in the future, which will take my desktop’s place as the source of truth.

From here, I need to sync data to my laptop and mobile phone, without using an intermediary. What happens if the service goes down (we’ve seen a number of AWS outages over the past few months) or a service starts to charge more money than I’m willing to pay?

Syncing

I think it was Hacker News where I first heard about Syncthing, a continuous file synchronization program. I run this software on all three devices. This little program performs a two-way sync of my ~/Documents folder between my desktop and laptop. It syncs the ~/vimwiki7 folder between my desktop, laptop, and phone. I use Markor’s Markdown editor to manage my VimWiki on the phone, since the files are just Markdown files anyway (got to love plain text, right?). I don’t get the same functionality as I do when working in Vim, but I’m often just writing down thoughts or checking the todo list in my yearly log file, and often don’t need the functionality.

The synchronization is a job that runs in the background and updates without me having to initiate it. So far, it just works. I’ve even had my VimWiki index.md file open on my desktop and phone at the same time and it syncs without an issue (I may need to close and reopen the file to see the changes but there isn’t an error, as far as I’m aware).

Photos

Syncthing also does a one-way sync between my phone and ~/Pictures folder on my desktop. I own an unlocked Samsung Ultra S21 phone, which has amazing cameras on it. I live in a very beautiful county, take care of horses and goats in the morning, and all these things are begging for me to take pictures of them. I also love taking pictures of our small family; I want to celebrate and remember these days. Losing these photos would make me a sad girl.

I take photos using the built-in Camera app on the phone and then, when I get home, Syncthing syncs them to the ~/Pictures folder. I also have Photoprism running on a Linode server on a subdomain of Wild Mind, which uploads photos every time I take picture using PhotoSync8. Photoprism is an AI-powered app that will recognize faces and reads the location date from the Exif data on each picture, placing photos on a map. I like having this data but I don’t want anyone other than me having this data.

Backup

Even without a specific backup solution, the above setup has backups of a sort inherently built in. If my data is synced between different devices, I have the data on at least two devices at any one time and, with my photos, they are backed up to an off-site server. However, I want to ensure that my data is explicitly backed up. I want a local backup, that gives me easy and quick access to lost data, and an off-site backup that is encrypted. There aren’t many backup up services for Linux specifically but I have heard nothing but praise for Tarsnap. In my experience, Tarsnap has been wonderful.

“Tarsnap is a secure, efficient online backup service” (Tarsnap) that is encrypted and very inexpensive. I followed their easy documentation and each night, at 20:00, a cronjob creates a backup of my ~/Documents, ~/Pictures, and ~/vimwiki directories, which are pushed up to their S3 buckets.

Lastly

This setup is more work than using Google for everything or syncing with Dropbox. There is a little more overhead in keeping my services up-to-date. What I like, though, is that I have multiple points of redundancy, thereby making data loss less of an issue. I am not reliant on a service that is mostly a black box. And, if it all comes crashing down (e.g. the world implodes and the internet goes out), at least I still have things locally and I don’t have to worry about losing data because I can no longer access it.

Bibliography


  1. Heather Cox Richardson, an American historian and Boston College history professor, writes prolifically at Letters from an American. She has a wonderful way of distilling the day’s news and how today’s events relate to America’s historical narrative. ↩︎

  2. There’s some interesting history to the statement itself. Hermann Göring, at the Nuremberg trials, stated “The victor will always be the judge, and the vanquished the accused.” ↩︎

  3. Losers is a loaded term, and I am specifically referring to Trump and his faction of the GOP in this instance. ↩︎

  4. Ledger is a double-entry, command-line program that allows me to track, forecast, and report on my finances. It has a little learning curve but keeps financial records in plain text and has powerful reporting features. ↩︎

  5. Git is a distributed versional control system, for those who don’t know. ↩︎

  6. Pop!_OS is a Linux distrobution based off of Ubuntu. I recently moved away from Arch after one too many mornings of having to research why a rolling upgrade broke my monitor setup or audio playback. When I get up in the morning to write, I don’t want my computer to be working against me! ↩︎

  7. I have used many different programs to store a personal wiki but I keep coming back to VimWiki. Since I program and write in Vim, it is so much easier to type ,ww to pull up the wiki rather than having to switch programs. ↩︎

  8. I purchased PhotoSync Bundle, which includes Autotransfer. It gets installed on your phone and has numerous services to transfer/backup photos and videos. I specifically use WebDAV, which seemed to be the easiest with Photoprism. ↩︎