My Programming Projects and Progress in 2020

EricMesa

- 22 minutes read - 4487 words

Back in 2019, when I did my programming retrospective I made a few predictions. How did those go?

  • Work on my Extra Life Donation Tracker? Yup! See below!
  • Write more C++ thanks to Arduino? Not so much.
  • C# thanks to Unity? Yes, but not in the way I thought. I only did minor work on my game, but I did start a new GameDev.tv class.
  • Learning Ruby? Well, I wouldn’t necessarily say I learned Ruby. I did finish the book Ruby Wizardry and I took copious notes. But until I do some practice - maybe via some code katas, I don’t think I’ll have solidified it in my mind.
  • 3D Game Dev? Nope, not really.
  • Rust and Go? Not even close. Although I did make sure to get some books on the languages.

So, what happened? On the programming front, I wanted to continue my journey to truly master Python after having used it at a surface level for the past 15ish years. I dedicated myself to doing the Python Morsels challenges (more on that below) and working through various Python development exercises. Outside of programming, the time I had off from COVID was used to play with my kids and they wanted to play lots of video games. So we took advantage of having way more time than usual to do that. So a lot of my goals slipped. We’ll get to 2021 predictions at the end, so let’s take a look at 2020!

Using git commits as a metric, it was a busier programming year than last year. I went from 769 commits for the year to 1231 commits.

As you can see before, on Github my contributions were:

  • 84% commits
  • 9% issues
  • 7% pull requests

Let’s take a look at the various projects I worked on in 2020:

lastfmeoystats

I can’t believe it took me this long to come up with this script to gather the data for my end of year music post.I’ve been using the last.fm API to post my top artists each week for a few years now. And each year it would take me a couple hours to gather and format all the data for the end of year last.fm posts. But this year I wanted to make some graphs and I’d learned how to use the matplotlib library earlier in the year, so I knew I’d generate the graphs programmatically. But as soon as I thought about it, I realized I could also generate all the statistical text for the post automatically and formatted the way I needed it. It’s a pretty simple script, but it does exactly what Python excels at - Automating the Boring Stuff.

Extra Life Donation Tracker

My current programming magnum opus continued to be refined. I made 23 releases (boy, that surprised me - I hadn’t realized how many releases I made in one year!) I went from version 3.2 to v5.2.3. One thing that’s pretty awesome from the point of view of someone putting stuff out there, is that a few of those releases were due to bug fixes reported from users. So I continue to have users! Other releases contained various improvements like:

  • changing the file structure to allow for PyPi releases
  • the ability to check automatically if there’s an update available
  • a help menu that takes users to the documentation
  • the ability for the user to change the font for the tracker window
  • the ability to validate user and team IDs
  • color-coded output to help the user know if something has gone wrong

I’ve got some changes planned for this year to continue to make the code more Pythonic (based on things I’ve learned in 2020) and hopefully easier to test via Unit Tests. I got Brian Okken’s book on Pytest so I hope to use that to improve my unit tests in my various projects. I also want to do some user-facing improvements like providing images from the Extra Life site.

PyInvaders

I made one tiny change to this code that I copied from Linux Format Magazine nearly a decade ago (and then improved upon slightly) so that it would run on Python 3.

Amortization

This program came a long way last year. What could I do to improve upon it in 2020? Well, during some of my Python education this year, I learned why the numbers were always slightly off on the program. It turns out that when you’re doing complex math, you don’t really want to use the built-in numbers because the rounding errors just build up like crazy. Instead, you want to use the Decimal package. This ensures a certain level of precision is maintained. Once I did that, all the errors went away. Then I decided to use what I’d learned about argparse last year to remove a lot of code that was a pain to maintain and have Python’s argparse generate the user help output for me - improving maintainability!

Raspi_garage_alert

I fully developed this code in 2020. Originally I intended to fork this guy’s code and add in the improvements I needed .But when I looked at how he had things setup, I would have wanted to make some radical changes so I decided to just start from scratch. Other than another project I’ll mention below, it’s the only project in which I’ve had pull requests come in from others; very exciting for me! The code still needs some work before it’s very useful to others, but it works perfectly for my purpose - checking whether the garage door was left open. I’ve set up a Home Assistant integration to send me an alert if it’s open after sunset.

Podmanapis

I started this work after I found out that there was a rest API for Podman. I figured I could make really good use of the API to automate some of my container work. A couple months later it was announced that the podman group had released their own Python API. So this project is mothballed, but I did learn a lot about creating an API translation layer.

Enviropi

This code grew out of my desire to use Pimoroni’s Enviropi pHat (or bonnet) on a Raspberry Pi Zero W in my master bath to measure humidity, temperature, and light levels. I’d been using an Arduino MKR WIFI 1010 and environment shield for this work, but it was slightly flaky and it’s a lot harder to debug a flaky Arduino since you can’t just ssh into it or check its logs. My code uses Pimoroni’s code as the base and then I made a few adaptations to make it work for me. Namely, it sends the data via MQTT to my server and it keeps the display off unless I purposely turn it on.

TurtleSnowFlake

This year, as I’ll talk about below (or you’ve seen on the blog in previous posts), I did the Raspberry Pi Foundation’s Scratch and Python projects with the kids. It was a fun way to introduce them to the idea of programming - so the kids could understand that they had the power to control the computer to make their own fun games or do fun things with the computer. This was a project to use Python’s implementation of the turtle (see this wikipedia page about turtle graphics) to draw snowflakes. As was typical when doing the Raspberry Pi Foundation projects, we started off by copying their code for the base. Then we branched off into the direction Sam wanted to go.

Prophecy Practicum

This was a project born during the COVID break. A friend had the need for a website to help him out with a weekly task that was robbing him of tons of hours when it could easily be automated. I wanted to learn flask and not just create a dummy site. I find that, with some of these technologies - like Flask or Django - the online tutorials tend to be trivial to understand and very hard to apply to a real web app when you want to make one. So, after completing a flask video tutorial, I started to work on this code. I was able to get a rough website working as well as an SQLite database. But when it came time to write the admin backend, the fact that Flask requires the user to build everything started to get tiresome. I think I may redo this in Django now that I have a better idea of the requirements. I don’t think my flask knowledge was completely wasted. I just think it’s a better framework when you don’t need an admin backend.

Matrix-nio

As I have mentioned in a previous post, I started learning Python in order to contribute to an open source project. So it was great to contribute to this Matrix protocol API this year. Partway through the contribution, the original author started making some changes that broke my changes, so I stopped. I’ll need to see if I can finish my pull request or whether it should be abandoned.

Snap-In-Time

My original code for this project to manage btrfs snapshots was a huge mess I created through my lack of knowledge of core Python concepts. This year I decided to clean up the code, make it work with Python 3, and use newer concepts I’d learned to make it more Pythonic and less for a spaghetti code mess. It’s now in a more maintainable format and includes the ability to push snapshots to my server. The first version of this project was my crowning achievement before my Extra Life Donation program and so it’s nice to have it once again be something I can be proud of. There is still some work to be done, so I hope to find some time for that in 2020.

Photo_stats

I’m a big stats and graphs junkie. (See my 2020 music post as an example) Back when I used to use Adobe Lightroom, I had a user-created plugin to generate stats and graphs on various EXIF data from my photos. I missed that, so I decided to make a little Python program to do that. It was through this project that I learned how to use the matplotlib module. I can already sense a pretty full slate for 2021, but maybe I’ll explore turning this into a plugin for Digikam.

Mathgenerator

I’d created this script in order to provide math problems for the wife. With my oldest starting multiplication, I added that to the script.

Pool Signup

As I mentioned, I wrote a script to sign me up for swimming so that I don’t have to risk dropping my phone in the pool.

Eric’s Comet Competition

I worked on this before the pandemic, so I completely forgot that I’d done any work at all on this game. I needed to look up some math in order to get the angles working for going from one end of the screen to the other end. Hoping that this year I at least add the ability to shoot at the asteroids and destroy them.

With the Kids

Scratch

2020 was the year of Scratch at our home. The Raspberry Pi Foundation provided a series of lessons each week and I chose one to do with each kid. They really enjoyed them and looked forward to them each week. Scarlett even tweaked some of them after we were done, so we’ll see if she or any of the others ends up developing a curiosity about programming. For me it came from being the only way I could use a computer back when I was a kid. So I’m not 100% sure what it’s going to take for the kids to develop the desire to program.

Our projects included:

MS Make Code

With the success I was having with the kids and Scratch, I decided to get some Adafruit Circuit Playground Expresses. Those can be programmed in MS Makecode, which uses blocks just like Scratch. So first I created an electronic spinner that Scarlett could use as a sibling chooser when we needed to randomly pick one of the kids for something. Turns out it wasn’t random enough, so I had to make some mods.

I then made a snake charmer in a box with Stella. She had a lot more fun pushing the button and watching the snake pop out than I thought she would.

Circuit Python

I got to relive my childhood and do some Circuit Python coding with Sam when we made a Simon Clone for the Circuit Playground Express. It was my first time working with Circuit Python and I found it wasn’t too terribly different from regular CPython. It might be fun to redo this project with a Raspberry Pi Zero to keep track of high scores.

Solo Electronics Projects

Adafruit

I didn’t end up doing anything with the Adafruit Clue that arrived with my first Adabox, although I would like to eventually use it for some of the example projects. I did have a lot of fun with the next Adabox, though. It was a Matrix Portal and large Matrix that I used to make a Halloween spooky pair of eyes to show through the window. Then I took the example code for the Halloween countdown timer and adapted it to make a Christmas countdowns clock that the kids really enjoyed. I had Scarlett create the images that showed between the time intervals. They’ve asked me to use it as a birthday countdown clock in the future. I’d like to see what else I can come up with using that hardware.

The final Adabox for the year included a Magtag. I have used that, along with sample code from Adafruit, to create a weather station. I’d like to adapt it in the future to perhaps have some other features. However, for the moment I have some debugging to do because it doesn’t seem to want to go to sleep as it should.

Arduino

I didn’t do as much with Arduino as I thought I would. I was expecting to at least work towards finishing my BBQ thermostat project. But I ran into some trouble when trying to control the fan with a BJT so the project was temporarily put on hold.

When I switched to using a Raspberry Pi Zero W for the bathroom project, I took the Arduino I had in there and adapted it to measure temperature and humidity levels from one of the bedrooms. I was able to find that the temperature differences among the rooms were not as large as we thought they were - at least at night.

Raspberry Pi

Early on in 2020 I brought my Raspberry Pi 1 B out of retirement to monitor my garage door. It could let me know if I’d left it open, especially after sunset. At some point the SD card became corrupted (a problem with those older Raspberry Pis) so I switched it out for a Raspberry Pi Zero W. The Raspberry Pi 1 B now monitors the temperature and humidity in the office - the hottest room in my house (by as much as 10 degrees fahrenheit in the summer). As I mentioned a few times before, I also put a Raspberry Pi Zero W in the bathroom to take over for the Arduino. It’s helping to monitor humidity levels and take action based on the measurements.

Improving Programming Skills

As I mentioned in the introduction, 2020 wasn’t the year of becoming a programming polyglot I thought it might be. (I spent nearly a whole weekend at my in-laws pre-pandemic looking at resources for Go and TinyGo - the equivalent of Circuit Python or Micro Python) But it was a great year for improving my Python. Through a variety of books (like Serious Python, Clean Code, and Impractical Python) and exercises (like Python Morsels) I seriously improved my grasp on Pythonic code and finally understood things that had been eluding me, like list comprehension. I did also do some work in other languages.

Python

Python Morsels

One of the many programming Humble Bundles came with training packages from various providers. One of the clinchers for me was a training video I haven’t yet done which involved list comprehensions. I decided to jump into Python Morsels first. So far it has been an incredibly positive experience. Each week Trey Hunner sends you a Python homework problem. The base problem is usually easy or just a mild stretch. The biggest value is the work he puts into the bonus sections. These take the solution you just made and stretch them to fit new, harder requirements. Depending on the way your brain works, you might need to overhaul your solution or it may work with merely a few tweaks. Another thing that makes this a top notch site include the built-in Python evaluator that allows you to check your code against the unit tests he writes to see if your code matches. I usually download the unit test rather than constantly using the in-built checker, but it’s awesome that it’s there. Even better than all this are the answers that Trey provides. I’m always excited to read them, even if I was able to solve the base problem and every bonus. He usually reveals an incredibly more Pythonic answer than the one I came up with. As the year went on and I started being closer to his answer, it was a great marker of my progress - I was starting to think more Pythonically. For a good example of the problems and the progress I made, check out this post.

Impractical Python

I got this awesome book, by Lee Vaughn, as part of a Humble Bundle. It has definitely been worth reading and I recommend it for anyone who’s gone beyond a beginner level in Python. This year I did chapters 3 and 4. They continued to introduce me to Python modules I didn’t know about and more Pythonic ways to solve problems. It’s slowly getting ever so slightly out of date as Python 3 continues to evolve (for example there isn’t any use of F-strings), but still worth getting and working through the problems.

Ruby

This year I finished reading through the kid’s Ruby book, Ruby Wizardry, by Eric Weinstein. I thought it was very well written and did a good job introducing the basics of Ruby. I took some notes on how various features work. It seems, in a lot of ways, to be pretty close to Python. That said, I’d need some practice and/or a deeper book before I’d say that I really know Ruby. Still, it was neat to see how the language worked after all the hype I’d heard about it.

C# and Unity

RTS Online Multiplayer class

Sure, after completing my GameDev.tv 2D Unity class, I still haven’t finished either of the two games I wanted to create, but I really wanted to know how to do online multiplayer for Unity. One of the games I’d like to develop - a Ludo/Parcheesi clone would be so that Ican play online with family members. I spent a couple days in 2019 trying to see if there was an easy way to do it. There wasn’t anything I could find that made it too easy. Then I saw this class this year and took it, since I’d already had a great experience with the GameDev.tv professors. After a first section, in which we focused on moving a ball around, we started working on an RTS (think Command and Conquer, Age of Empires, or Warcraft). It’s been pretty fascinating to see how this works. It is a lot more complex and will probably take me another review of the material to truly understand how to do it for my games, but I’m very glad to have taken the class to better understand the issues involved. I am about halfway through the RTS section and will complete the class in 2021.

Advent of Code

This was my first year participating in the Advent of Code challenge. Essentially, each of the first 25 days in December contains a programming challenge followed by a bonus challenge that takes the first problem and adds some wrinkles to make it harder. Each day is harder than the last. Additionally, there’s a cute little story that goes along with the entire challenge and each day’s problem advances the story. The first few are pretty trivial to solve, so some folks took to esoteric languages or methods to solve those first problems. I heard about it via the Python discord server. The Advent of Code servers give a score to the first 100 participants. (First place takes 100 pts, second 99pts, etc) I would never have scored because a) the puzzles come out at midnight EST and b) were often solved within a few minutes. I think my fastest solution was an hour or so. Fortunately, the Python discord server was more generous with points and I was able to make the leaderboard.

For the first dozen or so days I made really good progress at solving the puzzles, even if it took me hours. Eventually, I came across a few I just couldn’t solve - neither the primary puzzle nor the bonus. (see my solutions in this Github repo) What made it worth it to push through was that a good challenge makes finding a solution very rewarding. Also, there was a great community on reddit (and the Python discord) sharing various solutions, visualizations, and lots of help. I definitely plan to participate next year. Eventually I ended up placing 136/574. I thought that was a pretty decent score for someone for whom CS principles do not come naturally. That is to say, I love programming and have been doing it since I was 8, but I would often see others’ solutions and realize I’d come up with the most convoluted solution possible in comparison. My biggest tip for saving your sanity is to write unit tests. Since some example input and output is given, the unit tests will make sure your code works before attempting the real input. (even though I still found myself sometimes coming up against corner cases - just like in real life programs I’ve written) I will definitey be reading this tips and tricks page now and once again just before the 2021 Advent of Code. (Took a peek before this blog post was published and I learned something fascinating about how AoC is set up! - Each puzzle comes with input data that is custom-generated for each participant. To solve the puzzle, the participant must write code that uses the input to produce a matching answer. While all the participants have to come up with the same general solution, everyone’s answer is different.)

Time will tell whether I stick with Python (so that the only challenge is the problem itself) or if I use it (particularly the early days) as a way to reinforce other languages I learn in 2021.

Goals and Hopes for 2021

Ok, after all that, what are my goals and hopes for 2021? Well, I currently have an issue in Github that I need to fix for the Extra Life Donation Tracker. Doing that is probably going to rekindle my desire to work on that project. It’s reached the mature stage, so I don’t have too many new features to add. Mostly I want to make some changes to make it more Pythonic and easier to maintain. I’m hoping to do that via the planned 5.3 and 6.0 releases this year. Maybe I’ll get to 7.0 and use flask to make a web GUI - especially combined with learning React or some other Javascript framework. I’d also like to help my friend out and finish the Prophecy Practicum project - probably by converting it to Django. I’ve also been considering redoing my flickr views project, but with PyQT, modern Pythonic programming, and using secrets properly so that I could post it to Github. I’d also like to make progress on Eric’s Comet Cleaners.

In 2020 I went somewhat nuts with programming Humble Bundles. I’ve got at least one book for nearly every major programming language. I’m currently eyeing a frontend web development bundle because I’m still stuck in the old HTML world (even pre-CSS). I haven’t done any Javascript since the days of Geocities and Tripod websites. Realistically, I’d like to have some time for Rust and Go - at least to get the basics down. Stretching (a HUGE stretch) I’d like to look at Kotlin (an improved version of Java that Google recommends for Android app development). I learned Java back in undergrad - Cornell’s equivalents of CS100 and 101 were Java (with a splash of Matlab). I do also have some Java books to see how the language has evolved. And, while we’re just shooting for the moon - I got some books to play around with Machine Learning. Between ML work in Python and computers getting stronger, even a Raspberry Pi can do some machine learning/ image recognition work. It’s something I’d like to play with in the future.Realistically I’m going to get through Brian Okken’s book on Pytest and then MAYBE get to Rust or Go by the end of 2021.

In the electronics realm, I’d like to continue with my BBQ Thermostat project. I’ve moved to using an ESP32-S2 and Circuit Python. We’ll have to see, once Circuit Python support for the board improves, how far that goes. I’d also like to continue developing projects that are fun to do with the kids.

I also have plans to continue working with the kids in Scratch and seeing if perhaps Scarlett is ready to try anything else - Javascript, Ruby, or Python. The other kids - I’ll have to wait until they’re old enough to type. I know with Scarlett, the annoyance of programming input when she didn’t know how to read or type was one thing that put her off early on. Also, we’ll have to see how the twins’ personalities match with programming or if I can even keep their interest. Maybe there are just too many distractions nowadays compared to when I was a kid and for TV there was only saturday morning cartoons and the video games were so hard you never played more than a couple hours.

We’ll have to see next year how things stack up compared to what I wanted for this year.

Stay tuned on this blog for more End of Year posts on the software I’m using for programming, video games, and cooking.