Exploring btrfs for backups Part 6: Backup Drives and changing RAID levels VM

Hard drives are relatively cheap, especially nowadays. But I still want to stay within my budget as I setup my backups and system redundancies. So, ideally, for my backup RAID I’d take advantage of btrs’ ability to change RAID types on the fly and start off with one drive. Then I’d add another and go to RAID1. Then another and RAID5. Finally, the fourth drive and RAID6. At that point I’d have to be under some sort of Job-like God/Devil curse if all my drives failed at once, negating the point of the RAID. The best thinking right now is that you want to have backups, but want to try not to have to use them because of both offline time and the fact that a restore is never as clean as you hope it’ll be.

Let’s get started! I’ve added a third drive to my VM. Time to format and partition the drive. I do this with gParted. Interestingly, after the last post – gParted is a bit confused about what’s going on – again, btrfs isn’t exactly transparent with what it’s doing, especially when you have RAID setup. It shows the data as being stored on the second hard drive and nothing on the first one. The fact that sdb1 and sdb2 are unknown file systems and empty seems to go along with what I said yesterday about the fact that I don’t think it was properly set up to be able to boot from that system should the main hard drive die. So if you followed along with part 5 – make sure you take care of that if you’re doing RAID on your boot hard drive.

OK, now that the partition is created, I create a backup directory under /media/backup. Then I create a btrfs subvolume there.

$ sudo mount -t btrfs /dev/sdc1 /media/backup/
$ sudo btrfs subvolume create /media/backup/backups
Create subvolume '/media/backup/backups'
$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.83GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 1 FS bytes used 208.00KiB
 devid 1 size 8.00GiB used 855.00MiB path /dev/sdc1

Looks like I’m in a good place. Just need to add this to fstab. Alright, everything should now be set to grow this into a RAID1. Time to shut off the VM to add another hard drive.

Alright, so far everything is working correctly. I created a couple folders and a file to make sure the data survives in tact. So, in yesterday’s post I said I wouldn’t be using the sfdisk thing, but I think it’s probably a great shortcut to make sure everything is right in case I do something weird in gParted.

$ sudo sfdisk -d /dev/sdc | sudo sfdisk /dev/sdd
sfdisk: Checking that no-one is using this disk right now ...
sfdisk: OK
Disk /dev/sdd: 1044 cylinders, 255 heads, 63 sectors/track
sfdisk: /dev/sdd: unrecognized partition table type
Old situation:
sfdisk: No partitions found
New situation:
Units: sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/sdd1 1 16777215 16777215 ee GPT
/dev/sdd2 0 - 0 0 Empty
/dev/sdd3 0 - 0 0 Empty
/dev/sdd4 0 - 0 0 Empty
sfdisk: Warning: partition 1 does not end at a cylinder boundary
sfdisk: Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table
Re-reading the partition table ...
sfdisk: If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
Looks good. Time to add it to btrfs. Strangely, it doesn't appear to work. (Complains there isn't an sdd1)  Ok, let's try gParted, then.
$ sudo btrfs device add -f /dev/sdd1 /media/backup/

I had to use -f because I’d put a btrfs partition on there – I probably should have selected unallocated or something. Let’s make sure this makes sense:

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.84GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 2 FS bytes used 208.00KiB
 devid 1 size 8.00GiB used 855.00MiB path /dev/sdc1
 devid 2 size 8.00GiB used 0.00 path /dev/sdd1

Looks good to me. Time to RAID1 it.

$ sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /media/backup/
Done, had to relocate 5 out of 5 chunks

It took 30 seconds because there wasn’t really anything to move.

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.84GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 2 FS bytes used 464.00KiB
 devid 1 size 8.00GiB used 1.28GiB path /dev/sdc1
 devid 2 size 8.00GiB used 1.28GiB path /dev/sdd1

And, checking the RAID levels:

$ sudo btrfs fi df /media/backup/
Data, RAID1: total=1.00GiB, used=320.00KiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=256.00MiB, used=128.00KiB
unknown, single: total=16.00MiB, used=0.00

Excellent! But this isn’t anything special over yesterday’s post. Now let’s add a third backup harddrive. This time I go straight for gParted and use a partition type of unformatted.

Now a quick check that my files are there:

$ tree /media/backup/
/media/backup/
├── test1
└── test2
 └── Iamintest2

Yup! Let’s keep going.

$ sudo btrfs device add /dev/sde1 /media/backup/

No errors this time. As always, a double-check.

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.85GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 3 FS bytes used 464.00KiB
 devid 1 size 8.00GiB used 1.28GiB path /dev/sdc1
 devid 2 size 8.00GiB used 1.28GiB path /dev/sdd1
 devid 3 size 8.00GiB used 0.00 path /dev/sde1

Perfect. Time for RAID5!

$ sudo btrfs balance start -dconvert=raid5 -mconvert=raid5 /media/backup/
Done, had to relocate 3 out of 3 chunks

Again, a quick finish because not many files there; 10-20 seconds. So, the checks:

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.85GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 3 FS bytes used 720.00KiB
 devid 1 size 8.00GiB used 1.16GiB path /dev/sdc1
 devid 2 size 8.00GiB used 1.16GiB path /dev/sdd1
 devid 3 size 8.00GiB used 1.16GiB path /dev/sde1

And RAID check:

$ sudo btrfs fi df /media/backup/
Data, RAID5: total=2.00GiB, used=576.00KiB
System, RAID5: total=64.00MiB, used=16.00KiB
Metadata, RAID5: total=256.00MiB, used=128.00KiB
unknown, single: total=16.00MiB, used=0.00

Alright. No issues expected with that. So let’s see if RAID6 is just as easy. Created another file so see if RAID5 overhead caused any issues. None that I could see, but I’m not exactly running this on a critical database or something.

$ tree /media/backup/
/media/backup/
├── test1
│   └── hahaha
└── test2
 └── Iamintest2

OK. let’s get to it! gParted again. Add the device:

$ sudo btrfs device add /dev/sdf1 /media/backup/

And the usual checks:

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.85GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 4 FS bytes used 720.00KiB
 devid 1 size 8.00GiB used 1.16GiB path /dev/sdc1
 devid 2 size 8.00GiB used 1.16GiB path /dev/sdd1
 devid 3 size 8.00GiB used 1.16GiB path /dev/sde1
 devid 4 size 8.00GiB used 0.00 path /dev/sdf1

Looks fine. Also, just realized they don’t do the CS thing of counting from 0. OK, moment of truth:

sudo btrfs balance start -dconvert=raid6 -mconvert=raid6 /media/backup/
Done, had to relocate 3 out of 3 chunks

Alright! Balance check:

$ sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.85GiB
 devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3
Label: 'Backup' uuid: 7042f4b7-9815-44f4-aef9-81103fc5855b
 Total devices 4 FS bytes used 720.00KiB
 devid 1 size 8.00GiB used 1.16GiB path /dev/sdc1
 devid 2 size 8.00GiB used 1.16GiB path /dev/sdd1
 devid 3 size 8.00GiB used 1.16GiB path /dev/sde1
 devid 4 size 8.00GiB used 1.16GiB path /dev/sdf1

And RAID check:

$ sudo btrfs fi df /media/backup/
Data, RAID6: total=2.00GiB, used=576.00KiB
System, RAID6: total=64.00MiB, used=16.00KiB
Metadata, RAID6: total=256.00MiB, used=128.00KiB
unknown, single: total=16.00MiB, used=0.00

Perfect. So, it’s just that easy. This has been a great demonstration for me because it means I can buy my harddrives little by little instead of all at once. Sure, all at once is nicer in that you don’t have to take hours doing your balancing, but sometimes that’s just not an option and it’s nice to know that btrfs can handle it on a live system. No offline time necessary.

With my current grad school load, the next post will most likely be me converting my home subvolume on SuperMario to RAID1. Then, probably this winter, setting up my backup drive on SuperMario and setting up my Snap-in-Time scripts to send snapshots from the main system to the backup drive.  See ya then!

Exploring btrfs for backups Part 5: RAID1 on the Main Disks in the VM

So, back when I started this project, I laid out that one of the reasons I wanted to use btrfs on my home directory (don’t think it’s ready for / just yet) is that with RAID1, btrfs is self-healing. Obviously, magic can’t be done, but a checksum is stored as part of the data’s metadata and if the file doesn’t match the checksum on one disk, but does on the other, the file can be fixed. This can help protect against bitrot, which is the biggest thing that’s going to keep our children’s digital photos from lasting as long as the ones printed on archival paper. So, like I did the first time, I’ll first be trying it out in a Fedora VM that mostly matches my version, kernel, and btrfs-progs version. So, I went and added another virtual hard drive of the same size to my VM.

btrfs-RAID1-two hard drives in VirtualBox Next comes a part that I won’t be doing on my real machine because I don’t have root on the non-VM system I want to RAID1:

#>sudo sfdisk -d /dev/sda | sudo sfdisk /dev/sdb
[sudo] password for ermesa: [sudo] password for ermesa: 
sfdisk: Checking that no-one is using this disk right now ...
sfdisk: OK
Disk /dev/sdb: 1044 cylinders, 255 heads, 63 sectors/track
sfdisk: /dev/sdb: unrecognized partition table type
Old situation:
sfdisk: No partitions found
Sorry, try again.
New situation:
Units: sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/sdb1 * 2048 1026047 1024000 83 Linux
/dev/sdb2 1026048 2703359 1677312 82 Linux swap / Solaris
/dev/sdb3 2703360 16777215 14073856 83 Linux
/dev/sdb4 0 - 0 0 Empty
sfdisk: Warning: partition 1 does not end at a cylinder boundary
sfdisk: Warning: partition 2 does not start at a cylinder boundary
sfdisk: Warning: partition 2 does not end at a cylinder boundary
sfdisk: Warning: partition 3 does not start at a cylinder boundary
sfdisk: Warning: partition 3 does not end at a cylinder boundary
Successfully wrote the new partition table
Re-reading the partition table ...
sfdisk: If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

OK, so now I have to install grub. Again, I wouldn’t do this on SuperMario, but since the VM has btrfs on the whole system, I’m going to do it here.

#>sudo grub2-install /dev/sdb
Installation finished. No error reported.

Excellent. Now the btrfs-specific parts.

#> sudo btrfs device add /dev/sdb1 /

Before the (hopefully) last step, let’s see what this gives us in the current btrfs filesystem:

 #>sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.82GiB
 devid 1 size 6.71GiB used 4.07GiB path /dev/sda3
 devid 2 size 500.00MiB used 0.00 path /dev/sdb1

Oh, this allowed me to catch something. It should have been sdb3 before not 1. Let me see if this fixes things.

#>sudo btrfs device delete /dev/sdb1 /
#> sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 1 FS bytes used 2.82GiB
 devid 1 size 6.71GiB used 4.07GiB path /dev/sda3

OK, good. That appears to have put us back where we started. Let me try the correct parameters this time.

#>sudo btrfs device add /dev/sdb3 /
#> sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.82GiB
 devid 1 size 6.71GiB used 4.07GiB path /dev/sda3
 devid 2 size 6.71GiB used 0.00 path /dev/sdb3

Much better. See that both devices are the same size? Good. A df shows me that /boot is on sda1. So I’m not 100% convinced we’ll end up with a system that can boot no matter which hard drive fails. I’m not going to worry about that since in SuperMario I’ll just be doing a home drive, but you may want to check documenation if you’re doing this for your boot hard drive as well. Time for the final command to turn it into a RAID 1:

#>sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /

That hammers the system for a while. I got the following error:

ERROR: error during balancing '/' - Read-only file system

I wonder what happened. And we can see that it truly was not balanced.

#>sudo btrfs fi show
Label: 'fedora' uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
 Total devices 2 FS bytes used 2.88GiB
 devid 1 size 6.71GiB used 5.39GiB path /dev/sda3
 devid 2 size 6.71GiB used 3.34GiB path /dev/sdb3

Hmm. Checking dmesg shows that it’s a systemd issue. I’ll reboot the VM in case the filesystem ended up in a weird state. It has definitely been acting a bit strange. The fact that it doesn’t want to reboot isn’t encouraging. Since it’s just a VM, I decide to go for a hard reset. When I tried to run it again, it said operation now in progress. I guess it saw that it wasn’t able to complete it last time? I’m not sure. If so, that’d be awesome. And maybe that’s why the reboot wouldn’t happen? But it gave me errors, so that’s a bit unintuitive if that’s what was going on.

Here’s what dmesg showed:

[ 224.975078] BTRFS info (device sda3): found 12404 extents
[ 243.538313] BTRFS info (device sda3): found 12404 extents
[ 244.061442] BTRFS info (device sda3): relocating block group 389611520 flags 1
[ 354.881373] BTRFS info (device sda3): found 14154 extents
[ 387.088152] BTRFS info (device sda3): found 14154 extents
[ 387.450010] BTRFS info (device sda3): relocating block group 29360128 flags 36
[ 404.492103] hrtimer: interrupt took 3106176 ns
[ 417.499860] BTRFS info (device sda3): found 8428 extents
[ 417.788591] BTRFS info (device sda3): relocating block group 20971520 flags 34
[ 418.079598] BTRFS info (device sda3): found 1 extents
[ 418.832913] BTRFS info (device sda3): relocating block group 12582912 flags 1
[ 421.570949] BTRFS info (device sda3): found 271 extents
[ 425.489926] BTRFS info (device sda3): found 271 extents
[ 426.188314] BTRFS info (device sda3): relocating block group 4194304 flags 4
[ 426.720475] BTRFS info (device sda3): relocating block group 0 flags 2

So it does look like it’s working on that. When it was done, I had:

#>sudo btrfs fi show
Label: ‘fedora’ uuid: e5d5f485-4ca8-4846-b8ad-c00ca8eacdd9
Total devices 2 FS bytes used 2.83GiB
devid 1 size 6.71GiB used 3.62GiB path /dev/sda3
devid 2 size 6.71GiB used 3.62GiB path /dev/sdb3

It’s encouraging that the same space is taken up on each drive. But how to confirm it’s RAID1? The answer is found in btrfs’ version of df. btrfs needs its own version because, as of right now, a lot of what makes it an awesome COW filesystem means that the usual GNU programs don’t truly know how much free space you have. So, let’s try it:

#>sudo btrfs fi df /
Data, RAID1: total=3.34GiB, used=2.70GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=256.00MiB, used=133.31MiB
unknown, single: total=48.00MiB, used=0.00

I’m slightly nervous about the “unknown” entry – but a quick Google shows that it’s no big deal.

3.15 has this commit, it's the cause of the unknown.  We'll roll the progs patch 
into the next progs release, but it's nothing at all to worry about.

-chris

Author: David Sterba <dsterba <at> suse.cz>
Date:   Fri Feb 7 14:34:12 2014 +0100

    btrfs: export global block reserve size as space_info

    Introduce a block group type bit for a global reserve and fill the space
    info for SPACE_INFO ioctl. This should replace the newly added ioctl
    (01e219e8069516cdb98594d417b8bb8d906ed30d) to get just the 'size' part
    of the global reserve, while the actual usage can be now visible in the
    'btrfs fi df' output during ENOSPC stress.

    The unpatched userspace tools will show the blockgroup as 'unknown'.

    CC: Jeff Mahoney <jeffm <at> suse.com>
    CC: Josef Bacik <jbacik <at> fb.com>
    Signed-off-by: David Sterba <dsterba <at> suse.cz>
    Signed-off-by: Chris Mason <clm <at> fb.com>

So, there you go, relatively simple to setup a RAID1 on a btrfs system. Took just under an hour – but it was only 3 GB to balance. Larger  drive takes longer (which is why RAID6 is better as you can have another drive fail while you are balancing in your replacement drive) The best thing is that it all runs on a live system so you don’t need to suffer being unable to use the computer while the balance runs. Again, if you’re doing this on your boot drive, use Google to confirm that the /boot and all that is setup correctly or you won’t quite have the redudancy protection you think you do. Next time’s going to get a bit interesting as I simulate what I want to do with my backup btrfs hard drives. After that it’ll either be more Snap-In-Time code or my live migration to RAID1 on my home btrfs subvolume.

Exploring btrfs for backups Part 4: Weekly Culls and Unit Testing

Back in August I finally had some time to do some things I’d been wanting to do with my Snap-in-Time btrfs program for a while now. First of all, I finally added the weekly code. So now my snapshots are cleaned up every three days and then every other week. Next on the docket is quarterly cleanups followed up yearly cleanups. Second, the big thing I’d wanted to do for a while now: come up with unit tests! Much more robust than my debug code and testing scripts, it helped me find corner cases. If you look at my git logs you can see that it helped me little-by-little figure out just what I needed to do as well as when my “fixes” broke other things. Yay! My first personal project with regression testing!

A small note: to accommodate the unit testing, I had to change the file name – so the one you want to use is the one without dashes. I am not 100% sure how to get rid of the old file without losing commit history, but I think it’s not too big of a deal for now.

If I can get my way, the next update will be when I setup the self-healing RAID 1 followed by setting up the backup harddive and btrfs send/receive.

Exploring btrfs for backups Part 3: The Script in Practice

Night of the second day:

# btrfs sub list /home
ID 275 gen 3201 top level 5 path home
ID 1021 gen 3193 top level 275 path .snapshots
ID 1023 gen 1653 top level 275 path .snapshots/2014-03-13-2146
ID 1024 gen 1697 top level 275 path .snapshots/2014-03-13-2210
ID 1025 gen 1775 top level 275 path .snapshots/2014-03-13-2300
ID 1027 gen 1876 top level 275 path .snapshots/2014-03-14-0000
ID 1028 gen 1961 top level 275 path .snapshots/2014-03-14-0100
ID 1029 gen 2032 top level 275 path .snapshots/2014-03-14-0200
ID 1030 gen 2105 top level 275 path .snapshots/2014-03-14-0300
ID 1031 gen 2211 top level 275 path .snapshots/2014-03-14-0400
ID 1032 gen 2284 top level 275 path .snapshots/2014-03-14-0500
ID 1033 gen 2357 top level 275 path .snapshots/2014-03-14-0600
ID 1035 gen 2430 top level 275 path .snapshots/2014-03-14-0700
ID 1036 gen 2506 top level 275 path .snapshots/2014-03-14-0800
ID 1037 gen 2587 top level 275 path .snapshots/2014-03-14-0900
ID 1038 gen 2667 top level 275 path .snapshots/2014-03-14-1700
ID 1039 gen 2774 top level 275 path .snapshots/2014-03-14-1800
ID 1040 gen 2879 top level 275 path .snapshots/2014-03-14-1900
ID 1041 gen 2982 top level 275 path .snapshots/2014-03-14-2000
ID 1042 gen 3088 top level 275 path .snapshots/2014-03-14-2100
ID 1043 gen 3193 top level 275 path .snapshots/2014-03-14-2200

Morning of the third day:

# btrfs sub list /home
ID 275 gen 4602 top level 5 path home
ID 1021 gen 4558 top level 275 path .snapshots
ID 1025 gen 1775 top level 275 path .snapshots/2014-03-13-2300
ID 1027 gen 1876 top level 275 path .snapshots/2014-03-14-0000
ID 1028 gen 1961 top level 275 path .snapshots/2014-03-14-0100
ID 1029 gen 2032 top level 275 path .snapshots/2014-03-14-0200
ID 1030 gen 2105 top level 275 path .snapshots/2014-03-14-0300
ID 1031 gen 2211 top level 275 path .snapshots/2014-03-14-0400
ID 1032 gen 2284 top level 275 path .snapshots/2014-03-14-0500
ID 1033 gen 2357 top level 275 path .snapshots/2014-03-14-0600
ID 1035 gen 2430 top level 275 path .snapshots/2014-03-14-0700
ID 1036 gen 2506 top level 275 path .snapshots/2014-03-14-0800
ID 1037 gen 2587 top level 275 path .snapshots/2014-03-14-0900
ID 1038 gen 2667 top level 275 path .snapshots/2014-03-14-1700
ID 1039 gen 2774 top level 275 path .snapshots/2014-03-14-1800
ID 1040 gen 2879 top level 275 path .snapshots/2014-03-14-1900
ID 1041 gen 2982 top level 275 path .snapshots/2014-03-14-2000
ID 1042 gen 3088 top level 275 path .snapshots/2014-03-14-2100
ID 1043 gen 3193 top level 275 path .snapshots/2014-03-14-2200
ID 1044 gen 3305 top level 275 path .snapshots/2014-03-14-2300
ID 1045 gen 3418 top level 275 path .snapshots/2014-03-15-0000
ID 1046 gen 3529 top level 275 path .snapshots/2014-03-15-0100
ID 1047 gen 3640 top level 275 path .snapshots/2014-03-15-0200
ID 1048 gen 3754 top level 275 path .snapshots/2014-03-15-0300
ID 1049 gen 3872 top level 275 path .snapshots/2014-03-15-0400
ID 1050 gen 3986 top level 275 path .snapshots/2014-03-15-0500
ID 1052 gen 4102 top level 275 path .snapshots/2014-03-15-0600
ID 1053 gen 4216 top level 275 path .snapshots/2014-03-15-0700
ID 1054 gen 4331 top level 275 path .snapshots/2014-03-15-0800
ID 1055 gen 4445 top level 275 path .snapshots/2014-03-15-0900
ID 1056 gen 4558 top level 275 path .snapshots/2014-03-15-1000

As you can see, it has removed the first two snapshots. Since all three snapshots for the first day were in the last quarter of the day, that is the correct behaviour. Tomorrow we will have a much better demonstration that it is 100% working as it should. To see the cron job go to part 2 of this series. To get the Python script, go to Github.

Creating Nice-Looking Buttons in QML on KDE

Back in October I created a GUI for my Python amortization table program.  One of the things I lamented was that the buttons in QML look like crap.  I want my buttons to look like buttons, not blue patches no a white screen.  I don’t really know what made me look, but yesterday (during the No Snow Snow Day) I was looking through the file system and it turned out I had Plasma Components installed on my system.  So I added

import org.kde.plasma.components 0.1 as PlasmaComponents

to the top and then changed Button{…} in my code to

PlasmaComponents.Button{
id: calculate
x: 80
y: 135
width: 200
text: "Calculate"
onClicked: calculateit(principle_input.text,interest_input1.text,paymets_input1.text)
}

And voila!  I went from

Amortization gui in QML
Amortization gui in QML

to

 

Amortization  GUI  with Plasma-style button
Amortization GUI with Plasma-style button

Much better, right?  Now, here’s the caveat – the ideal solution would be Desktop Components, not Plasma Components.  Because if well implemented Desktop components would be cross-platform.  So if you wrote your backend code to cross-platform you GUI would work on all the operating systems that QT supports.   I’m hoping that since the KDE guys are very much involved with QT stuff that the APIs will be similar enough to allow me to switch over to the Desktop Components code without much work.  I haven’t pushed my new code up to Github yet, but it’ll be there soon.  Now I just need to change the text input boxes to the usual code which, when reviewing the Plasma Components source code, I learned will allow select-all to work (right now you need to backspace to delete the default text).  One more caveat – I think you need at least KDE 4.9 to have these libraries (but I could be wrong)

My Second Ever Useful GUI Program

A while ago I wrote about my first ever useful GUI program.  And in one of the series of posts that followed I explained that the reason I hadn’t made a useful GUI program before now is because all the typical stuff has already been made over and over.  We don’t need any more tetris clones, word processors, or music programs.  Recently I had reason to create my second program.  A while ago I created a an amortization calculator in python which you can find here.

My wife was asking me to run some numbers and I wanted to make it nice and easy for her to do it without needed to specify commandline options.  So I created this GUI for her.  It only required a slight modification to my Python code which I think I’ll be able to merge back into the commandline code and just have one codebase.  Once I get things cleaned up I’ll have the code up on that page I linked to before.

Amortization gui in QML
Amortization gui in QML

The awesome thing about QML is that it only took me about an hour to code up a GUI to my python program and I think that’s awesome!  If I could have one wishlist item for QML it would be an easy way to make QT widgets that fit in with KDE.  And maybe it exists and I just haven’t come across the right documentation or tutorials, but I’d prefer if my “calculate” button looks like a regular button.

Developing My First Plasmoid: The QML Code

Back in February I posted the code to the data engine I developed for my plastmoid.  At the time I’d wanted to clean up my plasmoid before posting it on here, however, I’ve become stuck on a key feature so I was hoping that maybe by posting the code I could get some help.  (As well as provide an example to others)

First of all, plasmoids need a certain structure to the folders. Here’s what mine looks like:

.
├── contents
│   └── ui
│   ├── Button.qml
│   ├── Button.qml~
│   ├── content
│   │   ├── ItemDelegate.qml
│   │   └── ItemDelegate.qml~
│   ├── flickrhelpers
│   │   ├── addtogroup.py
│   │   ├── addtogroup.py~
│   │   ├── removefromgroup.py
│   │   └── removefromgroup.py~
│   ├── flickrviews.py~
│   ├── flickrviews.qml
│   ├── flickrviews.qml~
│   └── test.qml~
├── metadata.desktop
└── metadata.desktop~

As you’ll see, I need to have a lot more in the ui folder, but I haven’t taken that optimization step yet. First off is the metadata.desktop file. A lot of this is self-explanatory, but one important part is the X-Plasma-RequiredExtensions which allows me to launch the flickrhelpers files.


[Desktop Entry]
Name=flickr views
Comment=A widget for flickr views
Icon=chronometer

X-Plasma-API=declarativeappletscript
X-Plasma-MainScript=ui/flickrviews.qml
X-Plasma-DefaultSize=750,700
X-Plasma-RequiredExtensions=LaunchApp

X-KDE-PluginInfo-Author=Eric Mesa
X-KDE-PluginInfo-Email=
X-KDE-PluginInfo-Website=http:/server.ericsbinaryworld.com/
X-KDE-PluginInfo-Category=Internet
X-KDE-PluginInfo-Name=org.kde.flickrviews
X-KDE-PluginInfo-Version=0.2

X-KDE-PluginInfo-Depends=
X-KDE-PluginInfo-License=GPL
X-KDE-PluginInfo-EnabledByDefault=true
X-KDE-ServiceTypes=Plasma/Applet
Type=Service

Moving on, we go to Button.qml which allows us to definte a button once and then use it throughout the main file without cluttering things up too much.

import QtQuick 1.0

Rectangle{

id: button
width: 50; height: 43

Text{
id: buttonLabel
anchors.centerIn: parent
text: bLabel
}

property color buttonColor: "lightblue"
property color onHoverColor: "gold"
property color borderColor: "white"
property string bLabel: "button label"

signal buttonClick()
onButtonClick:{
console.log(buttonLabel.text + " clicked")
}

MouseArea{
id: buttonMouseArea
anchors.fill: parent
onClicked: buttonClick()
hoverEnabled: true
onEntered: parent.border.color = onHoverColor
onExited: parent.border.color = borderColor
}

//detmermines color of button by using the conditional operator
color: buttonMouseArea.pressed ? Qt.darker(buttonColor, 1.5) : buttonColor

Next up is the ItemDelegate code. This code takes the XML from the data engine and parses it out to make the list on the left-hand side of all the photos.

import QtQuick 1.0

Item {
id: delegate
height: column.height + 40
width: delegate.ListView.view.width

Column {
id: column
x: 20; y: 20
width: parent.width - 40

Text {
id: titleText
text: title; width: parent.width; wrapMode: Text.WordWrap
font { bold: true; family: "Helvetica"; pointSize: 14 }
}

Text {
id: viewsText
text: "Views: " + views; width: parent.width; wrapMode: Text.WordWrap
font { bold: false; family: "Helvetica"; pointSize: 12 }
}
}

MouseArea {
anchors.fill: delegate
onClicked: {
delegate.ListView.view.currentIndex = index
window.photoURL = url
window.jpegURL = jpeg
window.photoID = photo
}
}

}

So, with the helper files out of the way, let’s take a look at the main code. First come the imports:

import Qt 4.7
import "content"
import QtWebKit 1.0
import org.kde.plasma.core 0.1 as PlasmaCore

Then comes the code that starts my plasmoid. All of the plasmoid is within a Rectangle. Next up is how I access my engine. This pulls all the data from the engine into my plasmoid.

Rectangle {
id: window
width: window.width
height: window.height

PlasmaCore.DataSource {
id: viewsSource
engine: "flickrviewsengine2"
interval: 86400000
Component.onCompleted: connectedSources = sources
onSourceAdded: connectSource(source)
}

Then I setup the variables I use throughout the plasmoid.

property string currentGroup: "" //this is used to hold the current views group to display
property string photoURL: "" //this is used to hold the flickr webpage of the current image
property string jpegURL: "" //this is to hold the photo that appaers in the webview
property string photoID: "" //this is to hold the photo ID that appaers in the webview
property string currentphotogroup: "" //this is to hold the photo's current group that appaers in the webview
property string nextphotogroup: "" //this is to hold the group it should go into
property bool loading: feedModel.status == XmlListModel.Loading

Then comes the XML liste model. This is what pulls all the data out of the XML. Isn’t it so nice and easy to read? It’s easier than any other XML parser I’ve used in other programming languages.

XmlListModel{
id:feedModel
xml: window.currentGroup
query: "/photos/photo"
XmlRole { name: "title"; query: "title/string()"}
XmlRole { name: "views"; query: "views/string()"}
XmlRole { name: "url"; query: "URL/string()"}
XmlRole { name: "jpeg"; query: "JPEG/string()"}
XmlRole { name: "photo"; query: "id/string()"}
}

Now we move back into visual stuff rather than program logic. We use the buttons I defined before and use them to populate some of the variables I declared above. There are a lot of buttons (one for each group) so I will just include the first two in this post:

Row {
width: window.width
height: 43
spacing: 2
//spacing: parent.width/14
Button{
id: v25
bLabel: "25"
onButtonClick:{ currentGroup = viewsSource.data["25"]["Group 25"]; window.currentphotogroup = "25"; window.nextphotogroup="50"}
}
Button{
id: v50
bLabel: "50"
onButtonClick:{ currentGroup = viewsSource.data["50"]["Group 50"]; window.currentphotogroup = "50"; window.nextphotogroup="75"}
}

Now we get to the column along the left-hand side that uses the ItemDelegate I shared above:

Row {
x: 0
y: 43
width: window.width
height: window.height
Rectangle {
width: window.width/3+10; height: window.height
color: "#efefef"

//This uses the ItemDelegate in the "contents" folder to create the list of images

ListView {
id: list
width: window.width/3; height: window.height
model: feedModel
delegate: ItemDelegate {}

highlight: Rectangle { color: "lightgray" }
highlightMoveSpeed: 9999999
}

}

This is where I define the box that holds the preview image.

Rectangle {
width: window.width/3
height: window.height
Row{
spacing: 0
WebView {
id: webView
width: 250
html: ""
//url: photoURL
}

Finally, the action buttons – the part that makes my plasmoid more useful than the commandline program I’d written. This is the part where I really need help from readers as I will explain after this section.

Column{

Button{
id: launchwebpage
width: 200
bLabel: "View on Flickr"
onButtonClick: plasmoid.openUrl(photoURL) //plasmoid.runCommand("firefox", ["http://www.google.com"]) //plasmoid.openUrl([photoURL]) //requires: X-Plasma-RequiredExtensions=LaunchApp
}
Button{
id: launchdiscuss
width: 200
bLabel: "Discuss"
onButtonClick: plasmoid.openUrl("http://www.flickr.com/groups/views"+window.currentphotogroup+"/discuss/")
}
Button{
id: addtogroup
width: 200
bLabel: "Add to Next Group"
onButtonClick: plasmoid.runCommand("/home/ermesa/bin/qml/plasmoids/flickrviews2/contents/ui/flickrhelpers/addtogroup.py", [photoID,nextphotogroup])
}
Button{
id: removefromgroup
width: 200
bLabel: "Remove from Current Group"
onButtonClick: plasmoid.runCommand("/home/ermesa/bin/qml/plasmoids/flickrviews2/contents/ui/flickrhelpers/removefromgroup.py", [photoID,currentphotogroup])
}
}

}
}
}
}

As you can see, in “Add to Next Group” and “Remove from Current Group” I am launching a python program that either adds or removes the photo from the group. However, sometimes this command fails. The biggest reason is that only 5 photos can be added to a group at a time. And I may have added some photos earlier in the day or via a cron job that I have that adds in the photos to the groups if they’ve never been in any groups. When it fails it raises an exception. I’d like to know how I can communicate that back to my plasmoid. This part is the only part where I haven’t been able to get help from the mailing lists or IRC. Without it, I have to launch the plasmoid from the cli and constantly check to see if there was a failure or success. If any readers can help me out with this, I would be so, so happy. I’d finally be done with functionality and be able to work on cleaning up the code and beautifying the plasmoid.

Developing my first plasmoid: The Data Engine (in python)

I figured it’d be neat to show you how my plasmoid works so you could use it when developing your own plasmoids.  Here’s the main.py of my data engine.  The indentation is off, in case you try to copy and past this in.

Here are the imports:

from PyQt4.QtCore import *
from PyKDE4.kdecore import *
from PyKDE4 import plasmascript
#for flickr
import views

Those are pretty standard.  The last one is the part of my engine that interacts with flickr.  Right now I have some work to do to get that presentable, but all you need to know is that it outputs XML to this main part of the data engine.

class PyFlickrEngine(plasmascript.DataEngine):
def __init__(self,parent,args=None):
plasmascript.DataEngine.__init__(self,parent)

def init(self):
self.setMinimumPollingInterval(333)
#for flickr
views.initialize()

The above is mostly standard.  Below are the sources.  These are the sources you’ll present to any plasmoids that want to grab from your data engine.

def sources(self):
sources = [“25”, “50”, “75”, “100”, “200”, “300”, “400”, “500”, “600”, “700”, “800”, “900”, “1000”, “1250”, “1500”, “1750”, “2000”,”3000″, “4000”, “5000”, “10000”]
return sources

The following is also pretty standard:

def sourceRequestEvent(self, name):
print “source request event” #debugging
return self.updateSourceEvent(name)

Here’s the main part of the engine.  This is how the data actually ends up in the data engine.  So I’m updating all of my sources here:

def updateSourceEvent(self,group):
print “updateSourceEvent”
#grouplist = []
if group == “25”:
print “i’m @ 25” #debug
grouplist = views.analyzeviews(views.views25, views.views50)
#self.setData(grouplist, “Group List”, QVariant.List) #original line
self.setData(“25”, “Group 25”, grouplist)
elif group == “50”:
print “i’m @ 50” #debug
grouplist = views.analyzeviews(views.views50, views.views75)
self.setData(“50″,”Group 50”, grouplist)
elif group == “75”:
print “i’m @ 75” #debug
grouplist = views.analyzeviews(views.views75,views.views100)
self.setData(“75″,”Group 75”, grouplist)
elif group == “100”:
print “i’m @ 100” #debug
grouplist = views.analyzeviews(views.views100,views.views200)
self.setData(“100″,”Group 100”, grouplist)
elif group == “200”:
print “i’m @ 200” #debug
grouplist = views.analyzeviews(views.views200, views.views300)
self.setData(“200″,”Group 200”, grouplist)
elif group == “300”:
print “i’m @ 300” #debug
grouplist = views.analyzeviews(views.views300, views.views400)
self.setData(“300″,”Group 300”, grouplist)
elif group == “400”:
print “i’m @ 400” #debug
grouplist = views.analyzeviews(views.views400, views.views500)
self.setData(“400″,”Group 400”, grouplist)
elif group == “500”:
print “i’m @ 500” #debug
grouplist = views.analyzeviews(views.views500, views.views600)
self.setData(“500″,”Group 500”, grouplist)
elif group == “600”:
print “i’m @ 600” #debug
grouplist = views.analyzeviews(views.views600, views.views700)
self.setData(“600″,”Group 600”, grouplist)
elif group == “700”:
print “i’m @ 700” #debug
grouplist = views.analyzeviews(views.views700, views.views800)
self.setData(“700″,”Group 700”, grouplist)
elif group == “800”:
print “i’m @ 800” #debug
grouplist = views.analyzeviews(views.views800, views.views900)
self.setData(“800″,”Group 800”, grouplist)
elif group == “900”:
print “i’m @ 900” #debug
grouplist = views.analyzeviews(views.views900, views.views1000)
self.setData(“900″,”Group 900”, grouplist)
elif group == “1000”:
print “i’m @ 1000” #debug
grouplist = views.analyzeviews(views.views1000, views.views1250)
self.setData(“1000″,”Group 1000”, grouplist)
elif group == “1250”:
print “i’m @ 1250” #debug
grouplist = views.analyzeviews(views.views1250, views.views1500)
self.setData(“1250″,”Group 1250”, grouplist)
elif group == “1500”:
print “i’m @ 1500” #debug
grouplist = views.analyzeviews(views.views1500, views.views1750)
self.setData(“1500″,”Group 1500”, grouplist)
elif group == “1750”:
print “i’m @ 1750” #debug
grouplist = views.analyzeviews(views.views1750, views.views2000)
self.setData(“1750″,”Group 1750”, grouplist)
elif group == “2000”:
print “i’m @ 2000” #debug
grouplist = views.analyzeviews(views.views2000, views.views3000)
self.setData(“2000″,”Group 2000”, grouplist)
elif group == “3000”:
print “i’m @ 3000” #debug
grouplist = views.analyzeviews(views.views3000, views.views4000)
self.setData(“3000″,”Group 3000”, grouplist)
elif group == “4000”:
print “i’m @ 4000” #debug
grouplist = views.analyzeviews(views.views4000, views.views5000)
self.setData(“4000″,”Group 4000”, grouplist)
elif group == “5000”:
print “i’m @ 5000” #debug
grouplist = views.analyzeviews(views.views5000, views.views10000)
self.setData(“5000″,”Group 5000”, grouplist)
elif group == “10000”:
print “i’m @ 10000” #debug
grouplist = views.analyzeviews(views.views10000, views.views10000)
self.setData(“10000″,”Group 10000”, grouplist)
return True

And, finally, initializing the engine – a standard part:

def CreateDataEngine(parent):
return PyFlickrEngine(parent)

 

As you can see, writing a data engine in python is pretty simple.  Most of the work is done in the other python file, but the data engine part is so simple, eh?  Between this and the official tutorial you should be in pretty good shape.

Developing my first plamoid Part 2

OK, so it’s been a year since I last blogged about working on this program.  I tried working on the data engine in August and then got stuck right around the time I started my first semester of grad school so I had to drop it, even though it got stuck in my head and I was thinking about it for weeks until the light bulb went off.  I wrote my idea and just had the idea to work on it recently.  I have to say that, overall, QML is pretty awesome for making a quick GUI.  I’ve always struggled with GUI code, but with QML I was able to put together a quick GUI in about 20 minutes.  Now, don’t judge QML too harshly because my plasmoid looks ugly.  It looks ugly because I just put together the minimum GUI to implement grabbing data from my Data Engine.  Once I get everything working right, I’ll fix it up.  After all, GUIs are really so easy in QML that it can be the icing on the cake in the end.

flickr views plasmoid
flickr views plasmoid - my first ever original plasmoid!

Man, when I got my plasmoid to this point I was soooo happy!  Unfortunately, I wasted countless hours trying to figure out why my plasmoid couldn’t talk to my data engine and it turned out it was because I was using QT Creator (an AWESOME IDE for QML development) and for some reason using that instead of the plasmoidviewer commandline argument to launch my plasmoid means that it can’t talk to engines.  It was so frustrating when I found out that was the reason for the issues.  I had literally spent hours trying to figure that out.  I’d say, use QT Creator to make your GUI and then maybe create some fake data you can read out of a file to make sure the GUI’s working.  Then move to plasmoidviwer for your final development.

One other annoyance was that in the XmlListView I had to change had to change source: to xml: – another hour or so wasted from not quite understanding the documentation well enough.  Basically it can only read xml from a data engine if the type is xml.  If you use source, it only works if it’s a web page or a file on your computer.  That’s kinda non-intuitive and wasted at least another hour of pointless debugging.

But I’m really happy because now I’m at the point where I’ve essentially ironed out enough of how QML and data engines work in order to finalize the look/feel of the plasmoid.  Then, once that’s done I’ll need to get help from the IRC or the mailing list to iron out the final bugs.  And after that I’ll post my code because there’s a real dearth of example code out there because QML and data engines are so new.  And it’s not always clear from the code documentation exactly how things could fit together.  Speaking of help, sreich on the plasma IRC room and Sebastian Kugler on the mailing list.  They were very patient with me as I worked otu what was going wrong with my plasmoid.  sreich even used his homework procrastination time to help me out.

Hopefully I’ll be able to have another post really soon where things are working at 80-90% of where I want them.  It all depends on how much time I can find to work on the code.

Automatically Posting your Top 3 Artists from Last.fm onto Twitter (with Python!)

I wrote this code a while back because a website that does the same thing seemed to miss my posts every other week.  So I figured I’d write my own in python to do the same thing to me.  Then I just put it into a cron job to automatically run it every Sunday.  I’m going to be posting the code on my GPL code page.  Here it is for you to see and for Google to index.  Just fill in the appropriate variables with the secret keys you get from each site’s API.

#!/usr/bin/python

__author__ = “Eric Mesa”

__version__ = “v0.1”

__license__ = “GNU GPL v3.0”

__copyright__ = “(c) 2010 Eric Mesa”

__email__ = “ericsbinaryworld at gmail dot com”

import pylast

import twitter

#variables

#last.fm

API_KEY = “”

API_SECRET = “”

#twitter

CONSUMER_KEY=””

CONSUMER_SECRET=””

ACCESS_TOKEN_KEY=””

ACCESS_TOKEN_SECRET=””

#last.fm code

network = pylast.get_lastfm_network(api_key = API_KEY, api_secret = API_SECRET)

user = network.get_user(“username”)

topartists = user.get_top_artists(period=’7day’)

post = “My top 3 weekly #lastfm artists: %s (%s), %s (%s), %s (%s)” % (topartists[0].item, topartists[0].weight, topartists[1].item, topartists[1].weight, topartists[2].item, topartists[2].weight )

#twitter part

api = twitter.Api(consumer_key=CONSUMER_KEY,

consumer_secret=CONSUMER_SECRET, access_token_key=ACCESS_TOKEN_KEY, access_token_secret=ACCESS_TOKEN_SECRET)

status = api.PostUpdate(post)

Programming to the Rescue: Amortization Automation

What I love about programming is the instant feedback.  In most programming languages, after you set up a framework for the barest bones of a program you can then run it at every step of the way to confirm that you are moving towards your goal.  What I love second-most about programming is the fact that I am using my computer to solve a problem and free myself from tedium.

Case in point, yesterday Danielle and I wanted to figure out what kind of a difference it would make to our mortgage if we paid off extra principal.  We have, built in to MoneyDance, a program that will do that, but no way to save the results.  So every time we wanted to examine it, we would have to recreate it.  I said, rather flippantly, “I should probably write a program to do this.”  And then that seed grew and grew until I knew I had to do it.  It took about an hour to get everything perfect, but I was so happy to have it completed.  But then I realized another problem, it looked great on my terminal screen, but what if I wanted to share it with someone?  So I figured I should have it output to a csv file.  I did that (about another half hour or so) and it imported into OpenOffice.org and Excel just fine.  woohoo!  Then I thought back to when I was trying to figure this out or see if someone else had written a program.  There were a lot of results on Google that led to porn or other sites that had nothing at all to do with amortization at all!  So I figured, I could generalize the program so that anyone could use it for any purpose.  Well, anyone with Python installed.  So, just download it from the link in this post and run python amortization.py and it will tell you what the commandline options are and how to see how paying extra principal affects the results.

Right-click and “save as” -> amortization.py

365 Graph

Today I worked on a python program to create a graph of the views of all my photos in my 365 Project set. Here’s the result:  (click for full size)

365 Project Views Graph

I was curious how they stacked up and I wanted to see if I could detect any patterns.  Except for a few outliers, they’re mostly below 50 views per photo.  I also expected to see more views after I started adding my photos to more 365 photo groups, but this is not the case.  There do appear to be clusters of views.  In other words, one highly viewed photo seemed to lift the ones by it.  That one huge outlier is my photo making fun of causes of the swine flu. The other big one is my Heroin Chic photo, no surprises there.  It was fun to create this and I learned a new python module.  Also, even though it probably took me a bit more time to program than to do it manually, it will now automatically generate whenever I want, and that’s worth the time it took.

FarmerJoe – An easier render solution.

I’ve been very happy until now with drqueue. Developed by Jorge Daza, it’s a very nice render farm management software with a nice GUI. It worked relatively well for me for “Jose’s Dinner” and “Schrodinger’s Cat“. However, there were two big kinks in using drqueue. First of all, Windows support was so sketchy it might as well be non-existent. So my most powerful computer was left out of the render-pool. Second, with the latest iteration, it no longer works on BSD computers. I tried for a few nights to make it work, but to no avail. So I turned to FarmerJoe.

FarmerJoe technically does not have FreeBSD support, but I emailed the developer and he sent me an unofficial FreeBSD script. I’m hoping I can help him include it in the official release for the next release. FarmerJoe is almost infinitely easier to setup than drqueue. That’s not to say there aren’t some slipups. It still takes about a day or two to get used to what it’s asking you to do and how it works, but it’s pretty easy once you get the hang of it. Of course, one of the best parts of FarmerJoe is that it can be launched from within Blender with a nice Python GUI. Sure, it’s not so horrible to launch a render from drqueue, but it’s so much more intuitive to be able to do it from within Blender. I really like this solution so much I think it may be the first project at which I truly contribute to get the features I’d like built in.

Here’s the method I use to setup and run FarmerJoe.

1) Setup a directory that is shared to all of your computers
2) Put unzip FarmerJoe to that directory
3) Edit the conf file to point to this directory and edit the path to Blender on each of the operating systems

# Master Server Configuration
port = 2006
master = 192.168.0.104

jobs = jobs
logs = logs

linux_root = /mnt/render/FJ
linux_blender = blender
linux_composite = /usr/bin/composite

### Added by Sven Mertens ###
freebsd_root = /mnt/render/FJ
freebsd_blender = blender
freebsd_composite = /usr/bin/composite
### /end of modification by Sven Mertens ###

windows_root = z:\FJ
windows_blender = C:\Program Files\Blender Foundation\Blender\blender.exe
windows_composite = composite

osx_root = /Volumes/farmerjoe
osx_blender = /Volumes/farmerjoe/bin/osx/blender/blender.app/Contents/MacOS/blender
osx_composite = /usr/local/bin/composite

# Application server Configuration
appserver_port = 2007

4) Go to the master computer and run the master/webserver like so ./Farmerjoe.pl –master && ./Farmerjoe.pl –appserver

5) Check on the website to see that all the slaves have connected

6) Go into Blender on any computer and run the python script to submit the render to the master

7) chmod -R 777 * the jobs directory – and you probably want to do this before you connect any slaves

8) Go to each slave and run the slave as so  ./Farmerjoe.linux for linux ./Farmerjoe.pl for BSD (if you have the modified Farmerjoe.pl that works with BSD) and Farmerjoe.exe for Windows
9) Go do something while your animation is rendered blazingly fast (depending on how many machines are involved)
10) Come back and collect all of the rendered frames from the frames directory
11) Do it again? Or Quit the slave and master programs and turn your computers off and stop heating your office/basement

So it’s incredibly easy to use and the author is very aproachable. He usually answers email within 24 hours and I imagine he gets a lot of it. So I’d say this is a great program and a good example of when it’s not a bad thing that in the FOSS world two people develop programs that do the same thing. One can end up being a little more complicated, but perhaps better suited to studios (studios have indeed used drqueue) and another can end up being a lot easier to use with a bit of compromise. For example, it’s not easy (or possible?) to re-render only particular frames that have messed up in FarmerJoe, but this is much easier to do with drqueue. (Although not without the occassional bug)

In the end, you must do what has been counciled time and again – try them both out and see which works best for you.