January

January was a relatively light programming month for me. I was focused on finishing up end of year blog posts and other tasks. Since Lastfmeoystats is used to generate the stats I need for my end of year music post, I worked on it a little to make some fixes.  The biggest fix was to change the chart titles not to be hardcoded. I didn’t realize it until I was reviewing my blog post, but I had hard-coded the year when I first wrote the code a couple years ago. I also changed the limits on some of the data I was collecting so that I could do more expansive trending for my overall stats.

I started working my way back towards spending more time programming as the summer started (in between getting re-addicted to CDProjektRed’s Gwent).

I started off by working on my btrfs snapshot program, Snap in Time. I finally added in the ability for the remote culling to take place. (My backup directories had started getting a LITTLE too big) I also added in official text log files so that I wouldn’t have to rely on my cronjob log file hack.

Back in 2019, when I did my programming retrospective I made a few predictions. How did those go?

• Work on my Extra Life Donation Tracker? Yup! See below!
• Write more C++ thanks to Arduino? Not so much.
• C# thanks to Unity? Yes, but not in the way I thought. I only did minor work on my game, but I did start a new GameDev.tv class.
• Learning Ruby? Well, I wouldn’t necessarily say I learned Ruby. I did finish the book Ruby Wizardry and I took copious notes. But until I do some practice - maybe via some code katas, I don’t think I’ll have solidified it in my mind.
• 3D Game Dev? Nope, not really.
• Rust and Go? Not even close. Although I did make sure to get some books on the languages.

So, what happened? On the programming front, I wanted to continue my journey to truly master Python after having used it at a surface level for the past 15ish years. I dedicated myself to doing the Python Morsels challenges (more on that below) and working through various Python development exercises. Outside of programming, the time I had off from COVID was used to play with my kids and they wanted to play lots of video games. So we took advantage of having way more time than usual to do that. So a lot of my goals slipped. We’ll get to 2021 predictions at the end, so let’s take a look at 2020!

If you’ve been following my blog for a long time, you know that back in 2014 I was working on a Python program to create hourly btrfs snapshots and cull them according to a certain algorithm. (See all the related posts here: 1, 2, 3, 4, 5, 6, 7, 8, 9) The furthest I ever got was weekly culling. Frankly, life and school contributed a good excuse not to keep going because I had created a huge headache for myself by attempting to figure out the date and cover all the possible corner cases with unit tests. This is what my code looked like in 2014.

It’s been a while since btrfs was first introduced to me via a Fedora version that had it as the default filesystem. At the time, it was especially brittle when it came to power outages. I ended up losing a system to one such use case. But a few years ago, I started using btrfs on my home directory. And even developed a program to manage snapshots. My two favorite features of btrfs are that Copy on Write (COW) allows me to make snapshots that only take up space when the file that was snapshot changes and the ability to dynamically set up and grow RAID levels. I was able to use this recently to get my photo hard drive on RAID1 without having to have an extra hard drive (because most RAID solutions destroy what’s on the drive).

This was the status at the end of the scrub:

[root@supermario ~]# /usr/sbin/btrfs scrub start -Bd /media/Photos/
scrub device /dev/sdd1 (id 1) done
 scrub started at Tue Mar 21 17:18:13 2017 and finished after 05:49:29
 total bytes scrubbed: 2.31TiB with 0 errors
scrub device /dev/sda1 (id 2) done
 scrub started at Tue Mar 21 17:18:13 2017 and finished after 05:20:56
 total bytes scrubbed: 2.31TiB with 0 errors

I’m a bit perplexed at this information. Since this is a RAID1, I would expect it to be comparing info between disks - is this not so? If not, why? Because I would have expected both disks to end at the same time. Also, interesting to note that the 1TB/hr stopped being the case at some point.

Here’s the output of the status command:

[root@supermario ~]# btrfs scrub status /media/Photos/
scrub status for 27cc1330-c4e3-404f-98f6-f23becec76b5
 scrub started at Tue Mar 21 17:18:13 2017, running for 01:05:38
 total bytes scrubbed: 1.00TiB with 0 errors

So on Fedora 25 with an AMD-8323 (8 core, no hyperthreading) and 24GB of RAM with this hard drive and its 3TB brother in RAID1 , it takes about an hour per Terabyte to do a scrub. (Which seems about equal to what a coworker told me his system takes to do a zfs scrub - 40ish hours for about 40ish TB)

A little under 3 years ago, I started exploring btrfs for its ability to help me limit data loss. Since then I’ve implemented a snapshot script to take advantage of the Copy-on-Write features of btrfs. But I hadn’t yet had the funds and the PC case space to do RAID1. I finally was able to implement it for my photography hard drive. This means that, together with regular scrubs, I should have a near miniscule chance of bit rot ruining any photos it hasn’t already corrupted.

I’ve been looking at NAS implementations for a long time. I looked at FreeNAS for a while then OpenMediaVault. But what I really wanted was to be able to take advantage of btrfs and its great RAID abilities - especially its ability to dynamically expand. So I was happy when I discovered Rockstor on Reddit. Here are some videos in which I explore the interface and how to work with Rockstor using a VM before setting it up on bare metal.

Looks like I was right about the non-commit and possibly also about the df -h.

Last night at the time I wrote the post:

# btrfs fi show /home
Label: 'Home1' uuid: 89cfd56a-06c7-4805-9526-7be4d24a2872
 Total devices 1 FS bytes used 1.91TiB
 devid 1 size 2.73TiB used 1.99TiB path /dev/sdb1

$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 600K 3.9G 1% /dev/shm
tmpfs 3.9G 976K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda3 146G 52G 87G 38% /
tmpfs 3.9G 296K 3.9G 1% /tmp
/dev/sda1 240M 126M 114M 53% /boot
/dev/sdb1 2.8T 2.0T 769G 73% /home
babyluigi.mushroomkingdom:/media/xbmc 1.8T 1.6T 123G 93% /media/nfs/xbmc-mount
babyluigi.mushroomkingdom:/fileshares 15G 6.5G 7.5G 47% /media/nfs/babyluigi
tmpfs 795M 28K 795M 1% /run/user/500

And today: