Marc Littlemore

Rescuing My Bricked M1 MacBook Pro

2026-01-25T00:00:00Z

Back in 2021, I joined a logistics startup called Beacon. I didn't stay long, but I walked away with a company MacBook Pro M1. It should have been sent back, but a major family health emergency happened at the same time and I completely forgot about it in the back of my wardrobe.

Fast forward to September 2025. My son had just gone off to university and, within a week, he'd managed to damage his laptop screen with an ill-timed AirPod case incident. Since he was struggling to use it, I went digging in the cupboards for a spare.

I rediscovered the MacBook and thought I'd see if it was still usable after I'd previously wiped it. It booted up fine, but it was completely locked down with JAMF, and I didn't have any credentials to log in. JAMF is a Mobile Device Management (MDM) system that companies use to manage their company devices.

What followed was a massive rabbit hole of trying to unlock a machine that everyone told me was a "brick." As I struggled to find any concrete information online, I thought I'd share the process here for anyone else facing the same headache.

This won't help you unlock a stolen MacBook. This post is only for legitimately owned devices that have been released from MDM profiles. It won't help you bypass Activation Lock or features designed to prevent theft.

The MDM brick wall

Beacon used JAMF to manage their devices. When I tried to boot the laptop, I was greeted with a JAMF login screen asking for a Google Account I no longer had.

I contacted an old friend who is still there. As he's now the head of engineering, they were happy for me to keep the laptop and asked their IT provider to remove it from their MDM system and Apple Business Manager.

I spoke to the provider on the phone and they confirmed it was done. However, when I tried to use the laptop, it still booted to that same JAMF screen. After another call, they told me I'd have to reformat the machine to properly remove the profile.

The problem? When I tried to boot into recovery mode to reinstall macOS, it asked for an admin recovery key. I'd never been given one, and the IT provider had no record of it. It looked like I was stuck.

Understanding Mac security

This is where I went down a rabbit hole. Since this was an Apple Silicon M1 chip, the security works differently than the old Intel-based Macs.

Essentially, there are two different layers of the security onion:

FileVault is disk encryption. It makes your data unreadable if the laptop is stolen, but it doesn't stop someone from wiping the whole machine.
Recovery Lock is what I was dealing with. It's an MDM hurdle that locks the recovery partition itself. It's there to stop someone from simply hitting "factory reset" on a stolen laptop and reusing it.

I was eventually told by the IT provider to call Apple Support. They suggested that Apple could remove the recovery lock if I provided proof of purchase (which I had). When I finally escalated the issue to someone technical at Apple, he was a bit patronising but somewhat helpful. He confirmed they couldn't actually remove the recovery key as it's a core security feature.

He suggested the MDM hadn't actually been removed since I could still see the login screen. It turned out that this wasn't actually correct, but it confirmed one thing: I was on my own to figure out how to wipe the hardware as nobody knew how to!

Even after a device is removed from Apple Business Manager, M1 Macs seem to keep cached records that still see the machine as a corporate device. Without the recovery key, you're a bit buggered as you can't even enter recovery mode to wipe the drive.

The solution: DFU restore

After a lot of research, I discovered that recovery locks on M1 Macs can be bypassed using DFU (Device Firmware Update) mode to completely restore the machine. Again, this can only be done if the laptop has been unenrolled from MDM.

DFU mode allows you to reinstall the firmware and OS, bypassing all locks. The downside is that it erases everything on the Mac, but since I couldn't access it anyway, that wasn't an issue.

To give this a go, I needed:

Another Mac running macOS Sonoma or later.
Apple Configurator 2 (a free app from the App Store).
A USB-C to USB-C data cable (not a Thunderbolt cable).
A hell of a lot of patience!

Note: I found that one of my USB cables didn't work and I'm not 100% sure why. You might have to try a few different ones to get it to click.

Booting into DFU mode

Getting the MacBook Pro into DFU mode was by far the hardest part. The timing is incredibly precise and it took me multiple attempts over a few hours to get it right.

Make sure the locked Mac is completely powered off (hold the power button for 15 seconds).
Connect it to your working Mac with the USB-C cable.
Open Apple Configurator 2 on your working Mac.

Now for the "Voodoo" timing part:

Press and release the power button on the locked Mac.
Immediately press and hold: Power button + Left Control + Left Option + Right Shift.
Hold all four for exactly 10 seconds. (I found counting "one-thousand, two-thousand" out loud actually helped).
After 10 seconds, release the three keys but keep holding the power button.
Wait until you see "DFU" appear in Apple Configurator 2 (usually another 5-10 seconds).

The screen should stay completely black. If you see an Apple logo, it's started booting normally and you need to try again. I had to try this at least 20 times before I got the timing right. Don't give up!

The IPSW secret

Once I got into DFU mode, the restore process kept failing with a generic error while halfway through the update:

AMRestoreErrorDomain error 10

After some help from Claude, I found the key: don't let Apple Configurator download the firmware. Instead, download the macOS firmware file (called an IPSW file) manually.

Go to ipsw.me on the working Mac.
Select your specific MacBook model.
Download the latest signed IPSW file (it's usually around 14GB).

In Apple Configurator 2, you just drag that downloaded IPSW file from your folder and drop it onto the large "DFU" icon. This single change made everything work on the first try.

Victory was mine

The restore took about 30 minutes. When the MacBook finally restarted and showed the "Hello!" welcome screen, I think I actually shouted "FUCK YEAH!" 😂

The recovery lock was gone. The MDM was gone. It was completely clean and ready to set up as a new laptop. I went through the setup, and it's now working perfectly for my son.

What I learned

If you're facing a similar situation, here are the small wins to keep in mind:

The Cable Matters: Not all USB-C cables are created equal. If DFU isn't triggering, swap the cable.
Manual Downloads: Always get the IPSW file yourself. It's much more reliable than letting the app handle it.
Counting Out Loud: It sounds stupid, but it's the only way to hit that 10-second window accurately for DFU mode.

It's a frustrating process, but it's definitely possible to recover a bricked device if you have the patience to work through it! 😄

Recovering My Apple Podcast Feed

2026-01-20T00:00:00Z

Discovering podcasts

Way back in the mid 2000s, I was still a mildly successful DJ under my alter ego of DJ Cruze. I was regularly travelling to Germany and playing at various clubs and I wanted to share my mixes with a wider audience, and yes, try to get a few more DJ gigs!

As I was a video games programmer at the time, I wasn't an expert on the web so I had naïvely set up my DJ Cruze website as a Blogger site. This worked for a while but I moved over to self-hosted Wordpress to get a bit more control over it once I started hitting the limitations of Blogger. This move gave me the ability to learn about, and set up, an RSS feed for my blog posts. As it was a DJ site, I also wanted to start sharing my DJ mixes as mp3 files.

I had just discovered podcasts for the first time and started listened to a few nerdy tech ones, mainly Leo Laporte's "This Week In Tech (TWIT)" and Adam Curry's "The Daily Source Code". Once I started to dig into the technology that powered podcasts, I discovered that it was just a variation of RSS under the hood. As I knew that Wordpress supported RSS, I found a couple of extra Wordpress plugins, and soon I had a podcast RSS feed ready to go.

I started recording a few mixes, added some voice overs to avoid getting into trouble for sharing unreleased tracks in a DJ mix, compressed them as terrible quality MP3 files, and started The DJ Cruze Podcast. The files were kindly hosted on a friend's server for a while as I had no idea what I was doing nor any concept of bandwidth.

Around the same time, Apple leaned heavily into podcasting and allowed you to submit your podcast to their iTunes directory. I submitted my podcast into the ether and thought nothing much of it. However, after a couple of months I soon gained some fans and quickly discovered that the iTunes crawler was reading my RSS feed over and over again. This quickly used up all of my hosting bandwidth. I quickly needed to cache my feed to stop Dreamhost from charging me more money, so I started looking for a better solution. And then I discovered Feedburner.

👋 Hello Feedburner

Feedburner was a web service that let you manage RSS feeds and, more importantly for me, it could cache them and handle all the bandwidth. I was a cheapskate back then and I didn't want to pay extra for better hosting so this seemed like the perfect solution. I quickly set up a redirect so my Wordpress feed URL would redirect to Feedburner instead. Bandwidth saved! 🎉

I left it like this for years and didn't really think much of it. I stopped making the podcast in 2011 but the feed remained live on Feedburner for anyone who wanted to subscribe to it. In 2012, Google acquired Feedburner and then left it to rot, as they often do with services they buy. I didn't really think much of it for the next decade.

Moving to my own feed

I eventually decided to move my blog away from Wordpress in the late 2010s and wanted to build a static site built with my favourite static site generator, Eleventy. I also wanted to take back control of my podcast feed. Back in 2021, I'd actually written about how to create an Eleventy podcast feed, so I already had the technical understanding of how to do it. My new feed would live at https://www.djcruze.co.uk/podcasts/feed.xml and I wanted to migrate all my old episodes over to this new feed.

As I was hosting the new site on Netlify, I set up redirects from the old Wordpress feed URL to the new Eleventy feed URL. This meant that Feedburner would now be pulling from my new Eleventy feed instead of the old Wordpress one. And then Apple Podcasts would pull from the new RSS feed too.

But I had one problem I wanted to solve: how do you tell Apple Podcasts and all your subscribers to switch to the new feed URL without all of the redirection?

Apple Podcasts had a fancy new web portal for podcasters as podcasting had finally become a big deal. Although Apple still had my podcast listed in their directory, I had no way of managing it as I'd never set up an account with them. I needed to prove ownership of the feed somehow.

So what do you do?

Redirecting subscribers

The answer is a special iTunes RSS tag called <itunes:new-feed-url>. When Apple encounters this tag in the feed markup, they automatically update their subscription to point to the new URL.

Here's what I added to my RSS podcast feed:

<itunes:new-feed-url>https://www.djcruze.co.uk/podcasts/feed.xml</itunes:new-feed-url>

This tag tells Apple that you feed has moved to this new feed URL. You'll need to make sure you leave the old HTTP 301 redirect in place from the old feed URL to the new one as well.

Apple recommends keeping this redirect in place for at least four weeks to give subscribers time to migrate. I've kept mine in the feed forever just to be safe. After this, the old feed can be retired and all subscribers should be receiving updates from the new feed.

The importance of GUIDs

Every episode in your podcast feed has a GUID (Globally Unique Identifier). This can be anything you want it to be but you can use URLs as these ideally should never change. If you've not read it, then you should look at this cool URIs never change article. Podcast apps use these GUIDs to track which episodes a subscriber has already downloaded or listened to.

Here's an example GUID from one of my episodes which has the original Wordpress post ID URL:

<guid isPermaLink="false">http://www.djcruze.co.uk/cms/?p=820</guid>

If you change the GUIDs when migrating your feed, you're in for a world of pain. Podcast apps will think all your episodes are brand new, and subscribers will re-download everything. They'll also lose their play progress and which episodes they've already heard.

When I migrated from Feedburner to my own feed, I made absolutely certain that every episode kept its original Wordpress GUID. I went through my Feedburner feed XML, noted down each episode's GUID, and made sure my new Eleventy-generated feed used the exact same identifiers.

This was tedious but I didn't want my 50+ podcast episodes to be redownloaded by any subscribers I might have left!

Claiming ownership in Apple Podcasts

The final piece of the puzzle was claiming ownership of my podcast in Apple Podcasts. Because I'd submitted the feed nearly 20 years ago through the old iTunes system, I'd never had access to Apple Podcasts Connect, their modern dashboard for podcast management.

Thanks to Rob Knight's excellent article on verifying an Apple Podcast feed, I learned about the claiming process. Here's how it works:

Step 1: Request to claim your podcast

In Apple Podcasts Connect, you search for your podcast and request to claim it. Apple needs to verify you actually own the feed, so they generate a unique verification code, a UUID-style identifier.

Step 2: Add the verification tag to your feed

You add this verification code to your podcast feed XML as an <itunes:applepodcastsverify> tag:

<itunes:applepodcastsverify>8c011b37-3cc1-4bb9-8c38-example</itunes:applepodcastsverify>

This tag goes in your RSS feed's channel section, alongside your other iTunes tags like <itunes:author> and <itunes:category>.

Step 3: Complete verification

Once the tag is in your feed and live on your server, you go back to Apple Podcasts Connect and click to complete the verification. Apple's servers fetch your feed, find the verification tag, confirm it matches their generated code, and you are now verified as the owner of your podcast in their system.

After verification succeeded, I had full access to my podcast's analytics, could update any metadata, and most importantly, I finally had complete control over my feed URL again. Success! 🎉

Own your content

So after two decades, I finally have complete control of my podcast feed again. It lives on my domain, hosted on Netlify, with my RSS implementation, that I built and understand myself. If I want to change hosting providers, I can do it without anyone noticing. If I want to move to a different static site generator, the URL stays the same.

With all of the current turmoil in the tech world, and big tech doing plenty of bad things, I feel much better knowing that my content is mine again. There's quite a backlash to using third-party services for hosting your content these days. Services come and go, get acquired and sold, or just shut down, so it's much better to own your own stuff.

Just remember to preserve those podcast GUIDs, test thoroughly, and take back control!

Writing Again

2025-01-11T00:00:00Z

I was looking at my site the other day and realised that I've managed to write quite a few blog posts in the past 10 years. I was really proud of writing 30+ articles in 2021 alone when I challenged myself to write more. However, looking at my the site markdown files, it dawned on me that I've only managed to add 2 more to this site since 2022 and I was a bit disappointed in myself.

But as everyone who has a partner or family knows, there's always a lot to deal with in real life.

Life challenges

Over the past few years I've been helping two teenage children to grow and thrive into young adults. This isn't always easy as they navigate growing up and the challenges of school, college, work, and their own relationships. My wife and I have had to become partial carers for my mum who sadly suffered from a stroke back in 2022. It's hard for her now that she's lost a lot of independence so we help by doing the shopping, washing, cooking, and taking her to the many medical appointments. And then in May 2024, I was, yet again, laid off from my tech job at Netlify after working there for 2 years. I had to spend a few months trying to find another role in a difficult market and that puts a lot of pressure on life. So I'm sure you can understand that that blogging wasn't top of mind for me.

I used to post a lot on social media but my behaviour has changed over the past few years. Twitter has become a bit of a cesspool after Elmo took over so I've not felt as comfortable sharing my thoughts there. Facebook now mostly serves posts from groups you're not part of rather than showing updates from your friends and it is also now allowing even more misinformation and hate speech due to the change in their policies. I found myself using Bluesky a little but more recently as it has been making strides to become a nice place to share your thoughts with the world. But as the world of media changes, I do feel like I need to keep my own little place on the internet and so I want to start posting a bit more on here again.

Writing for myself again

I always felt like I had to have big ideas to write about and that my posts should be really insightful and more long-form in length. But I've also read lots of great blog posts that are just a few paragraphs long and are often just someone sharing their thoughts and ideas. I have tried to start writing again a few times but I find myself paralysed by the thought that people would judge what I've written or I'm not knowledgeable enough to write on certain subjects. Thanks imposter syndrome!

I've come to realise that I should just write for myself and if people read it, then that's a bonus. ❤️

I really should read my own blog and create a writing habit!

Inspiration for writing more

I've also been inspired by a few blog posts that I've read recently.

My old Netlify colleague Den Delimarsky wrote a great post about being a property owner and not a renter on the internet. He's writing a great post eschewing social media platforms and ensuring that we all help to build our own little homes on the internet where platform owners can't pull the rug from under us.

The ever-brilliant Simon Willison has written lots about blogging on his site. I don't know how he manages to publish multiple blog posts per day but he has some great advice on his post called what to blog about. He suggests writing more "today I learned" (TIL) posts about the challenges you encounter and how you fixed them. He also says you should write more about any projects you're working on so it's time to actually work on some more projects in the first place!

This post from Danny Guo called "Why I blog" makes some really great points on why you should blog. I love that writing helps to clarify your thinking as you have to write something down and make it coherent.

There's also the classic post by Jeff Atwood on how to achieve ultimate blog success from way back in 2007 which is still really relevant today.

Finally, I'm always inspired by my friend Sam Rose who has got some brilliant long-form articles on computer science topics on his wonderful blog. You should really check this out if you haven't already as he writes incredible articles on complex computer science topics and includes his own wonderful visualisations. His posts are really engaging and he writes so anyone can understand them.

What's next?

So 2025 will be a year where I post more regularly and don't worry about what others think. Everything will continue to be syndicated via my RSS feed so if you're still an old-school RSS reader like me, you can get all of my posts there.

I'm also planning on making more music under my DJ Cruze moniker. I might even publish some videos or live-streams of me making the music too so keep your eyes peeled for that.

Continue to be excellent to each other and see you online soon!

Automate Your Static Site Rebuilds with Netlify Scheduled Functions

2024-03-17T00:00:00Z

I laugh every year when I see this meme about updating your website footer:

https://twitter.com/iamdevloper/status/1212320688050229249

If you're a developer who has come from the world of Wordpress where using PHP to dynamically update the year in the footer then I'm sure you're wondering what the fuss is about. As PHP is a server-side language, you're constantly rebuilding a page when someone requests a URL.

But if you have a static site then your code only updates when you build it manually and redeploy the latest version. If you have dynamic content on the page, such as a year in your footer, then you have to remember to rebuild the site when the data changes or maybe use some client-side JavaScript.

This site is generated with 11ty which is staticly generated so I need to remember to rebuild my site each year if I want to do something like update my footer. But clicking on a deploy button in the Netlify dashboard is a bit of a faff and I'm likely to forget to do it. So why not automate it? 🤖

Let's start automating

My original plan for rebuilding my sites was to trigger this rebuild via a cron service, like cron-job.org or Easy Cron or automation platform like Zapier.

I tried using the automation service make.com to schedule the yearly rebuild of my site. However, I soon hit the limits of their free tier which only allows 2 "scenarios" (a set of automation steps) and I wanted to update multiple websites. You could bundle multiple rebuild requests into a single scenario and update each site in a sequence, but then you lack single responsibility and it's harder to debug if anything goes wrong with a single update.

So as my site was already hosted there, I decided to use Netlify.

Disclaimer: I am a Senior Software Engineering Manager at Netlify but I've used their platform for years before I worked there.

Using Netlify to rebuild your site

I host all of my sites on Netlify and have done for a long time. Netlify provides you with the ability to rebuild your site via a build hook. This is special kind of webhook, a function that responds to a HTTP POST request, which allows you to kick-off a new build and deployment of your site.

While there are plenty of other options for running scheduled tasks, I thought I'd just move over to using Netlify's scheduled functions to automate this. This has the advantage of having complete control over the code and when the rebuild happens. I can also add in any additional notifications, like sending me a message on Slack or Telegram, and it allows me to commit it to my site's repository as part of the infrastructure, known as infrastructure as code (IaC). This allows me to version my scheduled tasks along with the code and stops me from having to remember which service I used to rebuild the site.

This approach is easy to implement, even for beginners with no prior experience with Netlify functions. So, buckle up and get ready to make website maintenance easy with the power of automation!

Create a Netlify build hook

First, we need to create a build hook in Netlify. This is a URL that you can use to trigger a new build and deployment of your Netlify site.

Go to the Netlify dashboard and select the site you want to add your build hook to. Next, go to the "Build & deploy" section and scroll down to the "Build hooks" section. Click the "Add build hook" button and give it a name. I called mine build-production-site as, unsuprisingly, it rebuilds my production website.

Remember that this build hook URL should be considered a secret and must be kept safe. If you accidentally expose it, you allow anyone to trigger a new build and deployment of your site. This might not seem like a big deal but you don't want people to be able to rack up build minutes on your behalf.

It will give you a URL that you can use to trigger a new build and deployment of your site and you can test it out using a CURL command to send a HTTP POST request to it.

curl -X POST -d {} https://api.netlify.com/build_hooks/<your-build-hook>

Take a look at your deployments in the Netlify application and you should see a new deployment being created.

Add the build hook to your environment variables

To keep this URL safe, you can add it to your environment variables in your Netlify site settings. This will allow you to access it in your Netlify functions without exposing it in your code.

Go to your Netlify site settings and select the "Build & deploy" section. Scroll down to the "Environment" section, click the "Add a variable" button, and choose "Add a single variable". Add a new environment variable called NETLIFY_REBUILD_HOOK and paste in the URL of your build hook.

Note: You can limit the scope of this to "Functions" as this will only be used in our scheduled functions.

Ok great! So now we've got a build hook so let's create some code in a Netlify scheduled function to trigger it.

Adding a Netlify rebuild scheduled function

If you've never used scheduled functions before, think of them as robot workers who can automate tasks for you while you sleep and do this on a consistent schedule! 🤖

Create a new file in your netlify/functions directory called rebuild-yearly.mts. This will be the scheduled code that calls our rebuild hook.

Node.js supports two distinct module formats with different capabilities and APIs: ECMAScript modules (or ES modules), an official standard format for JavaScript packages, and CommonJS, a legacy format specific to Node.js.

I'm using the .mts file extension to indicate that the file is an ECMAScript module.

To rebuild the site, we're going to make a fetch request to the build hook URL to trigger a new build and deployment. Netlify's rebuild hooks allow us to add a query parameter to the URL called trigger_title which gives us a clearer title for the build in the Netlify dashboard. This will help us to identify why we're rebuilding the site.

Add the following code:

Yearly rebuild scheduled function ./netlify/functions/rebuild-yearly.mts

import type { Config } from '@netlify/functions'

// Stop TypeScript from complaining about
// the missing process.env.NETLIFY_REBUILD_HOOK
declare var process : {
    env: {
        NETLIFY_REBUILD_HOOK: string
    }
}

// An asynchronous function to call
// the Netlify build hook to rebuild your site
const rebuildSite = async (triggerTitle: string) => {
    // Construct the URL for the Netlify rebuild hook
    const url = new URL(process.env.NETLIFY_REBUILD_HOOK);

    // Add the title to the query string
    url.searchParams.append('trigger_title', triggerTitle);

    // Make a POST request to the Netlify webhook
    return await fetch(url.toString(), {
        method: 'POST',
    });
};

// Always update your footer every year! :)
export default async (request: Request) => {
    await rebuildSite('Yearly rebuild to update footer');
};

// Netlify scheduled function cron syntax
// Run every year on the 1st of January at 00:00
export const config: Config = {
    schedule: '0 0 1 1 *'
};

You have to export a config object with a schedule property to tell Netlify when to run the function. The schedule property uses the cron syntax to specify when the function should run. In my example, the function will run every year on the 1st of January at 00:00. You can use a cron generator like crontab.guru to help you generate the cron syntax but Netlify also supports cron extensions like @yearly which might be a bit easier to read.

Testing the scheduled function locally

You'll want to test the scheduled function locally using the Netlify CLI to check that it works as expected.

Install the Netlify CLI if you haven't already:

npm install -g netlify-cli

Then run the following command to start the development server for your static site:

netlify dev

This will start a local server and you can access your scheduled function at http://localhost:8888/.netlify/functions/rebuild-yearly in your browser.

You can't test scheduled functions, so you'll have to invoke the function manually. In another terminal window, you can trigger the function by running the following CURL command:

netlify functions:invoke rebuild-yearly

Note: The name of the function matches the name of the file without the file extension.

You can check the Netlify dashboard to see if a new deployment has been created. If it has, then your scheduled function is working as expected. 🎉

Now that we've got a scheduled function to rebuild our site, we can update the footer with the latest year. You can do this by using a JavaScript function to get the current year and update the footer text.

For my 11ty site I use the following liquid template code to update the footer:

<footer>
    <p>Copyright &copy; 2007-{{"now" | date: "%Y"}}</p>
</footer>

Liquid templates allow the keyword now or today to get today's date and then we format it to only take the year. This will update the footer with the latest year when the site is rebuilt.

Deploying the scheduled function

Once you're happy with your scheduled function, you can deploy it to Netlify. If you have a git repository linked to a Netlify site, you can simply push your changes to the repository and Netlify will automatically deploy your scheduled function.

You can also use the Netlify CLI. Run the following command to deploy your function to production (or leave off the --prod flag to create a deploy preview):

netlify deploy --prod

This will deploy your function to Netlify and you can check the Netlify dashboard to see your scheduled function in the "Functions" section.

Your function will now run every year and you'll see it in your deploys in the Netlify dashboard:

And that's it! All done! 🎉

Example use cases

I've used this to update the footer on my site but you can use this approach to automate any other tasks that you need to run on a schedule.

Weekly newsletter updates: I use the EmailOctopus email service to send out a newsletter and each week I automatically update my site with the latest newsletter from their API.
Scheduled blog posts: If you're missing Wordpress scheduled posts, you can do the same thing with a static site by setting a publish date in your post metadata. Use Netlify scheduled functions to automatically publish them on your site on a specific date. Remy Sharp has a great post on scheduled and draft blog posts with Eleventy which will show you how to do this.
Activity feed: I use the GoodReads API to update my site with the latest books I've read and you can regularly update this with a scheduled function. You could do the same with Strava for your latest runs or rides, or Last.fm for your latest music listens.
Bookmarks: Use a bookmark manager like Pocket or Raindrop.io to save your favourite links and use their APIs to update your site with the latest bookmarks you've saved.

That's all folks!

Automating regular website updates for your static site using Netlify scheduled functions is a great way to set it and forget it. You can stop worrying about manual updates and focus on creating new content for your site.

Go and automate your static site updates with Netlify scheduled functions today! 🤖

Easily Create Gravatar Images With Eleventy

2022-01-30T00:00:00Z

If you're like me, your early blogs were probably hosted using Wordpress. Most of my personal sites started off with a self-hosted version of Wordpress on a server that I had to maintain and update. I'm getting too old for the constant updates that owning a server requires so I have moved to using static site generators and hosting them using Netlify.

I'm a big fan of Eleventy as my static site generator as it's super simple to use. One of the most recent sites had lots of comments and I wanted to keep them in the new Eleventy build. All of the comments have a user image on them and they use a Gravatar image.

What is a Gravatar?

A Gravatar is a Globally Recognised Avatar. It was an idea started by the ex-cofounder of GitHub, Tom Preston-Werner, and his Gravatar site and API was eventually bought by Automattic, the company who build Wordpress. A Gravatar is a user profile image or "avatar" which is attached to a specified email address. It can be uploaded and set on the Gravatar website, and used on any websites that support them. It makes it easy to update your picture in a single place and for it to appear on multiple websites.

So now we know what a Gravatar is, how do we generate one for Eleventy?

Adding an Eleventy shortcode

We're going to add an Eleventy shortcode to convert an email address into an image URL. This will allow us to easily use it within either an HTML image tag, or from a Markdown file.

First, we add a shortcode to the Eleventy configuration file .eleventy.js to expose it in our template files. Let's call it gravatar.

Filename .eleventy.js

const gravatarShortcode = (email) => {
    // TODO: Return the Gravatar URL
};


modules.export = (eleventyConfig) => {
    // Expose the shortcode for the template
    eleventyConfig.addShortcode('gravatar', gravatarShortcode);

    // Other configuration code...
};

Adding this shortcode into the Eleventy configuration file allows us to add it into one of our Markdown files to render an image. Here's an example of how we might use it in a blog post's markdown to render the Gravatar for an email address.

Filename posts/example-markdown.md

![A Gravatar Image]({% gravatar "email@fakedomain.com" %})

Creating a Gravatar URL

A Gravatar image URL is created by calculating an MD5 hash of the email you want to use. We can use the Node.js crypto module to easily generate this.

We should ensure that the email address we pass to the hashing function is cleaned up. We can remove any leading or trailing spaces and make it lowercase. That will ensure it creates an identical hash if someone accidently calls it with a different case or some extra whitespace.

Finally, we need to return a Gravatar URL with the hash appended to it.

Filename .eleventy.js

const crypto = require('crypto');

const gravatarShortcode = (email) => {
    // Clean up the email address
    // - Remove any leading or trailing spaces
    // - Make it lowercase
    const cleanEmail = email.trim().toLowerCase();

    // Create an MD5 hash from the cleaned email address
    const emailHash = crypto
        .createHash('md5')
        .update(cleanEmail)
        .digest('hex');

    // Return a URL image with the hash appended
    return `https://www.gravatar.com/avatar/${emailHash}`;
};

In our template we'll now have an image URL for the email if a Gravatar exists for it.

Filename posts/example-markdown.md

![A Gravatar Image]({% gravatar "email@fakedomain.com" %})

The above gravatar shortcode becomes this image URL:

Filename posts/example-markdown.md

![A Gravatar Image](https://www.gravatar.com/avatar/4cfeb4eed871ce152cfef83b542ad62f)

If you wanted to use the shortcode in some HTML as an image tag you can do that too:

Filename example.html

    <img
        src="{% gravatar "email@fakedomain.com" %}" 
        title="Jane Doe"
        alt="Avatar of user"
        width="150"
        height="150"
    />

Here is a Gravatar of me embedded into this Markdown file:

Adding a size

The Gravatar URL allows you to append some additional query parameters to change the size of the image that the URL returns. We can update our shortcode to allow us to pass in a size so that the Gravatar URL renders it at the resolution we need.

Filename .eleventy.js

const crypto = require('crypto');

// Add a size parameter which defaults to 80
const gravatarShortcode = (email, size = 80) => {
    // Clean up the email address
    // - Remove any leading or trailing spaces
    // - Make it lowercase
    const cleanEmail = email.trim().toLowerCase();

    // Create an MD5 hash from the cleaned email address
    const emailHash = crypto
        .createHash('md5')
        .update(cleanEmail)
        .digest('hex');

    // Return a URL image with the hash appended
    // Append a size query parameter
    return `https://www.gravatar.com/avatar/${emailHash}?s=${size}`;
};

Let's try it out and render a larger Gravatar of me!

Adding a default image

Finally, the Gravatar URL allows us to use a default image which will be rendered if no Gravatar is found for the email hash. We can add that as another query parameter in the generated URL.

Filename .eleventy.js

const crypto = require('crypto');

// Add a defaultImage parameter which defaults to 'mp' (mystery person)
const gravatarShortcode = (email, size = 80, defaultImage = 'mp') => {
    // Clean up the email address
    // - Remove any leading or trailing spaces
    // - Make it lowercase
    const cleanEmail = email.trim().toLowerCase();

    // Create an MD5 hash from the cleaned email address
    const emailHash = crypto
        .createHash('md5')
        .update(cleanEmail)
        .digest('hex');

    // Return a URL image with the hash appended
    // Append a default image query parameter
    return `https://www.gravatar.com/avatar/${emailHash}?s=${size}&d=${defaultImage}`;
};

Here's a default image for an unknown email address which uses the "mystery person" image.

Filename posts/example-markdown.md

![Gravatar]({% gravatar "unknown@email.com" 150 %})

You could also render a robot as the default image instead.

Filename posts/example-markdown.md

![Gravatar]({% gravatar "unknown@email.com" 150 "robohash" %})

Conclusion

See how easy it is to add a Gravatar with Eleventy? Eleventy's awesome shortcode system gives us the ability to easily extend it and expose Gravatars in our templates. Try it out if you're moving your Wordpress site over to Eleventy.

I hope you found it helpful but let me know if you have any questions or spot any errors in the code.

Improving Your Workflow with JavaScript Testing: Best Practices

2021-07-24T00:00:00Z

One of the questions people ask when starting their journey into testing their JavaScript code is where to start. What functions should the team test? Do we need to test every function, or should we look at testing the whole API end-to-end? Do we have mock out all of our backend services like databases or cloud storage?

It can be overwhelming to know where to start.

This paralysis often stops people from adding tests at all.

Starting to test your code

Testing is often an area that's overlooked when starting to writing code, especially if your team is unfamiliar with the process. You start with with a proof of concept demo project and before you know it, it has become the production code and deployed without any tests.

There's a common misconception that tests are there to find bugs in your code. This is a great side effect, and you can write additional tests to confirm bugs are fixed when you find them, but its not the main reason. The primary goal is to make sure that your code is working as expected and to give you and the whole team confidence in your codebase.

When you write tests, they should be testing one thing: given an input into a function you are checking that you always receive the same output. Tests are there to give you confidence in your code so that you can make changes and ensure that everything still produces the same results as it did before.

Another common worry is that testing will take too much time and effort.

Teams often think it's not worth the time to add tests to your code. It's true that testing takes some upfront thinking about your code, but this often makes your design better. Not being able to test your code well is a "code smell" - something which doesn't feel right and suggests that there are other problems.

While testing your code takes some time and effort, the benefits pay for themselves later when we deploy the code to your production environment. Being able to run all of your tests on your continuous integration server, and confirm the tests all pass, gives you confidence that your end users will see fewer or no issues.

The types of tests you will want to add to your code may differ depending on how much code has already been written.

Adding tests to an existing codebase

If you've already got an existing project with lots of code, a good approach is to start writing end-to-end tests. These test the functionality that the user will see and don't attempt to mock out any of your dependencies.

For end-to-end tests, you can use a fresh development environment that you create to run the tests. This can match your production environment as closely as it can but it's created and intialised when you start running your end-to-end tests. Alternatively, you can initialise your data before your tests run and remove this data at the end of the test cycle. This is often known as the setup and tear-down process in a lot of test frameworks.

Your database could be intialised via fixture files, data which is loaded from a file, and contain the initial state of your application that you want to test. Your tests can assert that the data changes correctly as you run your test cases. It's good practice to ensure that your tests can run independently. You want to avoid sharing state between tests. You don't want a test to fail because a previous test has run incorrectly.

End-to-end tests are great because they mimic what a real user would do when they use your application. However, they can take longer to run and can take development time to get the data and environment set up correctly.

If you want to start writing some end-to-end tests for your code then take a look at Cypress, Test Cafe, or Codecept. These are all fantastic testing frameworks which will help you to get started.

A good compromise, and a complement to end-to-end tests, are integration tests.

Integration tests can be used to test your code but for these tests you can mock or stub your dependencies with fake data. This means that things like upstream APIs or databases can have mocked data that you control. This allows you to avoid having to depend on parts of your system that you might not easily control. You are able to provide your application with both the happy path, when the code works as expected, and the unhappy paths, when there are errors or upstream APIs don't work as expected.

A good rule of thumb is to mock or stub dependecies which aren't under your control. For example, if you have a database or API that you don't control, you can mock it out to provide fake data.

Integration tests can be harder to get started with as you have to understand how to provide fake data for your systems. You must match the data structure that is returned from your database call or API endpoints and ensure that it is consistent with the real services. How this is done will depend on how your code is architected. A common JavaScript or Node.js pattern is to use mocking libraries such as Sinon, Proxyquite or Jest mock functions to provide test doubles for modules. You can also mock HTTP requests with Nock and spin up your API locally using Supertest.

How many tests to write

Don't get hung up on your code's test coverage. This is a metric to determine what percentage of your code modules, functions, and logic branches are covered by tests. The goal is to write tests that are easy to understand and maintain, not to test everything. If you're not sure how many tests you need to write, start small and add more as you go.

Start by testing the primary flows of your code and then add more as you continue to add new features. It's more import to get started with small test cases than to have an exhaustive test suite.

Decide on which parts of the flows could fail and how important that would be to the team or your users. If it's important part of your code, then you should write tests for that. For example, if you're developing an e-commerce application, consider testing the code to add and delete items from the shopping cart and ensure that your payment flows work. These would be the key business areas that should rarely fail as you'd lose money if they did.

Conclusion

It is easier than you think to get started with testing your code. If you're new to the process, it can be a little overwhelming. Don't worry, it's not that hard and you can get started really quickly.

Let me know if you have any questions or comments and I'd be happy to help.

Want to learn more about testing your JavaScript code? Check out my other JavaScript testing articles for more tips and techniques!

Streamline Your Workflow: Automate GitLab Releases with Semantic-Release

2021-07-10T00:00:00Z

Each week one of my new team makes a release and deployment of our API service from our GitLab repository. It is a very manual process that involves working out what has changed in the previous commits, determining what sort of version it should be, tagging it in GitLab, and running a GitLab pipeline to deploy it to the correct server. It's a tedious process that can be prone to human error.

On a previous project, I'd used a tool called semantic-release in combination with conventional commit messages to automate the process. This gave a quick and easy release process without any human interaction. I love automation so it felt like the perfect tool for the job.

What is a release?

A release will depend on your stack of choice and platform. For a desktop application, you could be taking the latest code, compiling the code, and creating an executable file. For web applications, this is often an update of a release number and re-deploying the application to your servers.

Our team creates a Node.js API that is consumed by multiple desktop and mobile clients. A release for us is a git tagged release which is then deployed to our development, QA, or production environments. We also bump our Node package.json version number, although as we're not versioning a library, this isn't strictly necessary.

Version numbers in the Node.js ecosystem use the standard semantic versioning (also known as SemVer). This gives us a release number in the format of major.minor.patch. A major version is determined by a breaking change, for example, the API makes incompatible changes. A minor version is generated if new functionality is added but the functionality is backwards compatible with previous releases. Finally, a patch version is changed if any bug fixes are made.

So how do we determine when to change the version number? This is up to the development team to think about the changes that are made. However, we can infer it automatically if the team uses consistent git commit messages.

Conventional Commits

The conventional commits specification is a lightweight convention that can be adopted for commit messages. It provides you with an easy set of rules to follow to create consistent commit messages. This allows automated tools to parse the git messages to determine a semantic version number.

A commit message is structured in the following way:

type(optional scope): description of commit

[optional body]

[optional footer]

For example:

feat(user): add validation schema on user create endpoint

Now we have a consistent format for our commit messages, how do we ensure that the team sticks to it?

For this, we can look at a tool to lint the commit messages before anyone is allowed to commit their code. A tool called commitlint can be used to validate commit messages. We can also use the husky package to add automated git hooks to the project. With this, a git hook can be added which will lint the commit message to our standardised format before the code is committed to our project.

Assuming you have a Node.js project set up, commitlint can be installed like this:

# Install commitlint CLI
npm install --save-dev @commitlint/cli

Although commitlint allows you to validate against a variety of commit message formats, including your configuration, we chose to use the config-conventional configuration which adheres to the conventional commit message format. You can install it as follows:

# Install conventional config for commitlint
npm install --save-dev @commitlint/config-conventional

Now you can set up the commitlint configuration file called commitlint.config.js so it uses the config-conventional message format:

echo "module.exports = { extends: ['@commitlint/config-conventional'] };" > commitlint.config.js

Now we've set up commitlint, we can ensure that it gets called when we attempt to commit code into git. We can install husky and add a commit-msg hook:

# Install Husky
npm install husky --save-dev

# Activate Husky git hooks
npx husky install

# Add the commit-msg hook
# It runs commitlint with the given commit message
npx husky add .husky/commit-msg 'npx --no-install commitlint --edit $1'

Now all commit messages must adhere to the config-conventional commit format and git will fail to commit any code that has messages that don't validate.

The team should now all be using the same commit message format so we can use that with semantic-release to determine how we want to release.

Semantic Release

The semantic-release package automates your release workflow. It analyses the git commit messages, determines the next version number, generates release notes and a changelog file, and publishes the package for you.

By default, semantic-release has several plugins which are part of the main package. These allow you to automatically determine the release, update the version in package.json, deploy to the npm registry for public packages, and tag a release in GitHub. As my team has its code stored in a private GitLab repository, we need some additional configuration.

First, let's install semantic-release:

npm install --save-dev semantic-release

We want to create a CHANGELOG.md file that can be shared outside of the team, commit that file and the updated package.json to our git repository, and tag a release in GitLab. For this we'll need to add the following plugins:

npm install --save-dev @semantic-release/changelog @semantic-release/git @semantic-release/gitlab

To override the normal semantic-release configuration, we can add a releases key to our package.json and configure the additional plugins:

{
  ...,
  "release": {
    "plugins": [
      "@semantic-release/commit-analyzer",
      "@semantic-release/release-notes-generator",
      "@semantic-release/changelog",
      "@semantic-release/npm",
      "@semantic-release/git",
      "@semantic-release/gitlab"
    ]
  },
  ...
}

Determine when a release should be created

Once semantic-release is set up correctly, it will use its commit-analyzer plugin to parse your commit messages and determine whether a new release should be created. By default, it creates a release when the following commit types are present in the commit message:

fix or perf - A fix or performance commit message will trigger a new patch release
feat - A feature commit message will trigger a new minor release
If BREAKING CHANGE is written in the commit message, it will trigger a major release.

These defaults can be changed by adding additional rules to commit-analyzer using the plugin configuration in your package.json. We found some issues with it not creating releases when we expected it to, so see the gotchas section for rules that we added.

GitLab pipelines

Now that the script and plugins are ready to determine when a new release should be tagged, it's time to set up a GitLab pipeline to do so.

How you do this will vary depending on your project's need. For our project, we first want to run our tests and linting jobs to ensure they all pass. After that, we can determine if we should create a release by running semantic-release. We only want to check this on our main branch so that we don't attempt to create a release for all branches pushed to GitLab which are used for merge requests. Finally, we have a deployment job that will run only if a new release has been tagged. This is our continuous deployment step.

As we want to use GitLab's API to tag a release, we need to create an environment variable called GITLAB_TOKEN which is exposed to the pipeline. This will allow us to have write access to the API.

To do this you need to create a new access token in your project in GitLab by going to Settings -> Access Tokens. The access token will need read and write access to the API and also the ability to read and write to the repository. It should look something like this:

For security, your access token will only be shown once so copy the token ready to add it as an environment variable into your GitLab project.

Once the environment variable has been added to the pipeline, we can build the GitLab pipeline to use the access token with semantic-release.

Here's an example GitLab YAML pipeline:

# semantic-release requires optional chaining which
# appears in Node 14.5+
image: node:14.15

# Assuming we're:
# - running tests
# - creating a new release if applicable
# - deploying the release if applicable
stage:
  - test
  - release
  - deploy

# An example job to run the unit tests
run-unit-tests:
  stage: test
  script:
    - npm run test:unit
  tags:
    - <your-runner>

# An example job to run integration tests
run-integration-tests:
  stage: test
  script:
    - npm run test:integration
  tags:
    - <your-runner>

# A job to cut a new release
# This will run before our deployment
create-release:

  # This is part of the release stage
  stage: release

  # Only attempt to run this job when we merge to the main branch
  only:
    refs:
      - main
  
  # Expose the GITLAB_TOKEN environment variable to this job
  # as a GitLab variable to be used by semantic-release
  variables:
    GITLAB_TOKEN: $GITLAB_TOKEN
  
  # Execute semantic-release to create a new release
  script:
    - npx semantic-release
  
  # Add the runner that will execute this job
  tags:
    - <your-runner>

# An example deployment script
development-deploy:
  stage: deploy

  # Only run this job when a release has been tagged
  rules:
    - if: $COMMIT_TAG
      when: always
  script:
    - <your-deployment-script>
  tags:
    - <your-runner>

Using the above example GitLab pipeline, a new release will be created if your commit messages should update the version number when you merge your code to your main branch from a merge request.

Advantages of automatic releases

Using semantic-release to automatically generate your releases gives many advantages.

Moving towards consistent commit messages using commitlint allows the team to move towards smaller, more atomic pull or merge requests. This makes code reviews much easier as your team will gravitate towards less code for each branch.

The automatic generation of release notes and a changelog file allows you to automatically inform stakeholders by sharing a link to the latest release. It avoids the team having to perform git comparisons between commits to determine what has changed.

If you're publishing a library, the npm plugin allows you to automatically publish to the npm registry. No more manual steps to publish your latest and greatest work.

Note: If your project is marked as private in your package.json file, there will be no attempt to publish the application if you use the npm plugin. It will just be used to update your package.json version number.

It avoids the team having to think about releasing manually. This is often a job that happens daily or weekly and it now happens automatically. Ultimately, you'll start to see a faster release cadence with these smaller, atomic changes.

Gotchas

While using semantic-release has made life easier, there are some potential issues you might have to deal with.

Firstly, semantic-release claims to work with Node v10+. However, when I tried it in the GitLab pipeline, one of the plugins depends on a new JavaScript operator called the optional chaining operator. This only appears in Node 14.5 and beyond so make sure that any Docker containers, or the GitLab Node image you use, use Node 14.5 or later.

If you use the @semantic-release/git plugin, it will commit the changes in your CHANGELOG.md or package.json files to your repository. This git commit will trigger a new pipeline on your main branch, and hence this can start another pipeline to re-check the release.

Tagging a new version in git also causes an additional pipeline to trigger. Again, you may have to play with your pipeline jobs to avoid unnecessarily triggering another pipeline.

If you use the revert type commit message, it won't trigger a new release unless you use a git revert command or the message has This reverts commit <your-commit-sha>. If reverting code, you'd expect the version to be changed as functionality will have changed. Using the refactor type also doesn't trigger a new release. In theory, a refactor shouldn't change the functionality so it's understandable that it might not trigger the release. In practice, a refactor often needs to be deployed to sanity check that nothing has broken.

To fix both of these issues, you can add additional config to the commit-analyzer plugin to trigger release off the revert or refactor commit types.

In the example below, a revert type will trigger a new patch release and a refactor gives a new minor release.

{
  ...,
  "release": {
    "plugins": [
      [
        "@semantic-release/commit-analyzer",
        {
          "releaseRules": [
            {
              "type": "revert",
              "release": "patch"
            },
            {
              "type": "refactor",
              "release": "minor"
            }
          ]
        }
      ],
      ...
    ]
  },
  ...
}

Conclusion

Using sematic-release is a great way to take the stress out of your release process and help your team to move towards a continuous deployment process. Although this post mainly concentrates on a Node.js automatic release pipeline, the tools suggested could be used to create automated releases with other languages too.

Let me know if you have any questions or spot any errors.

Lessons learned from 30 days of writing

2021-04-22T00:00:00Z

In the middle of January 2021, I decided to start a habit of writing.

I was inspired by the #ship30for30 community I'd seen on Twitter. The hashtags were appearing daily and I thought a writing habit would be good to start and could be a useful tool to clarify my thinking.

Here are some of the lessons I learned after my 30 days of writing experiment was over.

It took more than 30 days

My original plan was take the #ship30for30 hashtag literally and write 30 articles in 30 days. The Ship 30 community revolves around making much smaller articles than I ended up writing. Their idea is to write an "atomic essay" of approximately 250 to 500 words to summarise a topic.

I initially attempted to keep the posts small but I struggled to articulate what I wanted to say in so few words. Anyone that knows me will tell you that I can talk a lot and my writing style matches this. I tried to get better at editing but even then, most articles ended up being between 500 and 1500 words.

In the end, my 30-day writing challenge took 55 days. I'm still incredibly proud of what I achieved, but it didn't happen as quickly as I initially thought it would.

I enjoy writing

I've always struggled with thinking that I was more mathematically-minded and "wasn't a great writer". Part of this experiment was me trying to dispel the myth that I'd built up in my own head.

I knew that I wasn't bad at technical writing. In my day job, I wrote a lot and I could be clear and succinct when I needed to be. Writing a new article every couple of days really helped me to exercise my writing muscle and I found out that I actually enjoyed writing regularly.

People enjoyed reading the articles

Having a website where you share personal views is a double-edged sword. On the one hand, you worry that nobody is going to read what you write. On the other hand, you worry that everyone will see what you write, especially if you make it more personal.

Every time I wrote a new article, I shared it on Twitter and on Linked In. While the aim was never to attempt to go viral, I found that people enjoyed reading the articles. Twitter wasn't the biggest driver of traffic but as I ended up writing more articles around team cultures, Linked In became a great place for a discussion. I found that my writing resonated with people and they commented on each article I shared.

I talked about sharing your network and learning in public in one of my articles and I found that sharing my work expanded my own network. I talked to a few recruiters who had "heard of me" from my articles. It also allowed me to have great conversations with new people. I especially enjoyed a Zoom chat I had with Duncan Skelton about leadership and coaching. Writing has really opened up my network to more opportunities.

Finding my cadence

I mentioned that I didn't manage to write 30 articles in 30 days but I did manage to find a cadence that worked for me. My wife is currently a teacher and also tutors some students for a couple of hours on Monday to Wednesday from around 6.30 pm. I found that attempting to write at this time gave me a chance to focus for 2 hours on writing. Sometimes I'd be helping my children with homework, but most of the time, this routine worked well.

Trying to write on Thursday and Friday evenings was much harder as my wife wasn't tutoring and I was involved more with the family. I'm an early riser and often wake up early at the weekends. I found that trying to write for an hour on Saturday or Sunday morning worked well too. I'd often finished a new article before the rest of the family were out of bed.

Ideas didn't always flow

Some evenings I really struggled with what to write about. I wrote an article about creating an idea habit and even then I sometimes couldn't focus on a single idea. I hadn't prepared a list of ideas in advance and so this is why it took much longer than 30 days.

Ideas often came from great conversations I had with my team in their 1:1 meetings, or when I caught up with colleagues. I wrote down these ideas as soon as I could and with them, I was able to more easily outline the article. In future I need to ensure I keep a list of article ideas so that I can just pick from them and start writing.

Taking notes makes writing easy

While sometimes I lacked ideas, the articles I found the most easy to write were those where I'd previously written a bunch of notes in Obsidian. It wasn't just a single idea. It was a collection of ideas around a single topic which I'd often outlined into key points. Being able to pick one of these notes made it easy to write an article and I could often finish a 1000 word article within a couple of hours.

Technical articles take a long time

I wasn't sure of the intial direction that my 30-day writing challenge was going to take. As an Engineering Manager, I tend to write less code in my day job. However, that means I'm often spending my evenings reading about new languages or frameworks and wanting to try them out.

I decided to write a few technical articles as part of my writing challenge but I found they were much harder. When I write a technical post, I care that the reader is getting a learning experience. That means I spent a lot of time ensuring that I'd captured every step in writing the code for the article. This slowed down the writing process and meant that a technical article often took 4 or 5 hours to write.

Whilst I really enjoy writing technical articles, the time taken meant I wrote fewer of those. Writing one per week would be much easier and I'll look to do that in the future.

Would I do it again?

I very much enjoyed the challenge and I think its made me a better writer. My website readership has increased and I'm starting to rank for various keywords now too. I don't think I'd attempt to write every day again as this was difficult at times. However, writing a new article weekly is what I want to do from now on.

You should try a 30-day writing challenge if you haven't.

I promise that it'll change the way you think.

You've got this: become a confident developer

2021-03-09T00:00:00Z

I've never liked the title Junior Developer. I much prefer Less-Experienced Developer. It doesn't imply age. It just says "you don't have the knowledge yet".

Being a Less-Experienced Developer is where we all start.

We might be less-experienced in software development as a whole. Or we might be less-experienced in a specific programming languange. Or a specific domain. Or testing. Or in front-end skills. Or databases. Or infrastructure. Or tooling. And so on...

It can be overwhelming to think of all of the software development concepts that you don't know. I've been a professional software developer since 1994 and I have a list as long as my arm of all of the things I don't yet know. And I like that. I'm always learning so it gives me something else to look at when I feel I've gained enough experience in the current technology I'm investing my time in.

But it can feel too much at times. You might think that being a software developer is too formidable and you don't know where to start. As I say to the less-experienced developers on my team:

“Be confident. You've got this!”

You are eventually going to be the biggest asset to your team and company. It'll just take a bit of time.

So how do you help to keep up that confidence?

Ask all the questions

We have a phrase in my team: "there's no such thing as a silly question". I love it when the team ask questions. It's a chance for more-experienced team members to step up and share what they know. Or for another less-experienced person to share what they've just learnt in that area. Asking questions is the key to learning. You're not expected to know everything so ask away!

Hopefully you're working in a team with a positive culture where senior engineers or managers don't disparage others for not knowing things. If you're a team lead, it should be your job to enable a culture of openness. If a less-experienced engineer is under confident, and perhaps isn't understanding something, why not try to open the discussion for them by asking someone to explain the concept. Be their advocate. Not everyone is comfortable in asking for help.

Understand the team's applications

Every team will build their applications in a different way. There will be technology decisions which have been made many years ago. There will be specific tools that the team use. Their will be team understanding around code design patterns. The folder structures and testing methods will have been chosen for a variety of reasons. It's time to dig in and start to understand why they've been built in this way.

Try and get a high level overview of the applications that the team owns, what they do, and how they work. You won't be expected to know everything, especially in a large codebase, but having a basic understand will really help to build your confidence. Once you've learnt a little bit about the systems, why not share it with others?

Communicate with your team and find their superpowers

You should get to know your teammates. Each of them will be different and have expertise in various technical topics. There will often be someone on the team who ends up being the most knowledgeable about specific applications or technologies. Another might be the expert in your tooling and continuous integration servers. If you're feeling stuck, then work out who has the superpower in that area and chat to them.

In my experience, the best developers are born out of great communicators. Building a great working relationship with the right person, and asking for their help at the right time, can really help you to grow as a developer. Pay attention to what they're helping you with and how they're explaining things. Reiterate the information back to them to clarify your understanding. They are there to help you. A good senior engineer should be a good mentor to those with less experience.

Document all the things

Documentation is the thing that most developers are the worst at. It should be part of the product that we're developing but we often leave it as an afterthought. When you're learning from others, always make your own documentation. Some of the less-experienced developers on my team keep a developer diary of what they've learnt each day. They then use that to confirm their understanding at the end of the week. Suggest to the team that you document what you're learning for everyone. Update the project's README file with what you've learnt and raise a pull request. Doing so will really help everyone but especially you.

Your notes can be a list of the concepts you've learnt during the day. It could be those pesky git commands that you need to learn. It could be links to online resources that you've found helpful. I've always liked to keep my developer resources in a GitHub repository which I can add to. By using version control, you can keep a history of what you've added and why. It also helps you to learn how a tool like git and GitHub works. If you don't want to use version control, then you can keep your notes in your favourite note-taking application. I love Obsidian but it could be Evernote or another application.

Set solid learning objective

One of my team used to be a primary school teacher and is one of the fantastic developers on my team. She's taught me that having a good learning objective is the key to learning a topic. Understand what you want to learn and why. This could be learning how to create a unit test for your code. Or how to store some state in a database.

Decide on what you want to learn and focus on that for a period of time. Don't jump from one language to another or from one new library to the next. This is very easy to do and won't help to build your confidence in your ability. You don't have to become an expert in everything overnight so having a single, focused learning objective will really help.

Help others once you know

Once you've learnt a technological topic, teach it to someone else. Being one step ahead of someone else, and teaching them what you know, will really help to solidify your learning. I always like to build a culture of lifting the team up and this should apply to everyone and not just the team lead.

Help to lift your colleagues up by running your own learning sessions for the team.

Be excellent to yourself and each other

Finally, please remember that you deserve to be a software engineer. You might be early in your career, and feel like there's too much to learn. Just remember that you'll definitely get there but it takes time. You probably don't remember how much you have already learnt. Look back at your notes and rediscover what you learnt last month, 3 months ago, or even a year ago.

Celebrate your wins, however small they are! You'll have come much further than you think. 🎉

Don't forget to be excellent to yourself. You're doing great!

And in the words of Bill and Ted, "be excellent to each other!"

Never get too comfortable

2021-03-07T00:00:00Z

Being comfortable is a good thing, especially when you start a new job.

You've got to know your colleagues, you've found your way around the office or if you're working remote, the company intranet. You can quickly find documentation on Confluence and know how to find Jira tickets, and you've finally found your stride with your work. Every day you make a significant impact in your role.

After a while you can easily drift towards the familiar daily tasks. The exciting can quickly become mundane. Maybe a restructure has left you drifting away from what you loved about your role. Perhaps there are few opportunities to grow within your team or department. Often you end up feeling like you're working hard each day but without making any impact by the end of the week. You feel like you've stagnated.

Sometimes it's easy to do nothing as you've become too comfortable.

Feeling stuck

If you've been doing the same role for the past few years, it's easy to feel like you're stuck. You might have reached your career goals, but you've become comfortable with the status quo and don't feel that you can reach outside of your comfort zone.

Feeling stuck like this can often lead to a sense of dissatisfaction. Everything that you do can seem too easy and you're no longer learning anything new. You can do one of two things if this happens. You can sit back and stay comfortable, or you can challenge yourself by expanding your role or asking for more.

Try to kick-start a change in what you do. Reach out to people outside of your immediate network and see if you can help other teams. Embrace a change in your day-to-day work to find a new challenge.

Find opportunties for growth

Doing the same thing every week means that you're missing out on your personal and professional growth. Doing the same work repeatedly means you'll potentially miss other opportunities to learn something new.

If you're feeling unchallenged in your work, it could be time to look at making changes to your routine and finding other areas in which you can grow. If you're a software developer who mainly writes code each day, now could be the time to look at mentoring less experienced developers. At my current job, I looked to grow my career and decided to gain confidence in public speaking. I knew this was one of the keys to unlocking a wider network of people I could help and who could potentially help me. This definitely helped to challenge myself and pushed me further in my career. It felt uncomfortable at the time, but that was a good thing.

Work out your goals

If you're not feeling challenged each day, you need to determine what would challenge you. Is it a need to take more responsibility? Do you want to learn a different skill or aspect to your current role? Only you can define what will make you feel uncomfortably challenged.

Reach out to your peers or manager and ask for more responsibility. If you're a developer then maybe you could look at learning a new framework or programming language. If you're a team lead or engineering manager then look for opportunities to be a force multiplier for your team or external teams.

Time to move on

Ultimately, if you're feeling too comfortable in your current role, and you can't find the challenges within your company, it may be time to look elsewhere.

It's a difficult decision to look for a new role but sometimes a fresh start can help to recharge your batteries. A new role is always a new challenge and it can often help to redefine your role and responsibilities. Making impactful change in a new role is always a great way to learn more and to grow your career.

Challenge yourself

So stop and think about where you are now. Are you too comfortable in what you're doing? Is it time to make a change?

Don't repeat your years.

Don't play your job on easy mode.

Find chances to change what you do and help yourself to grow.

Avoid becoming too comfortable.

Now might be the time to make the change that you need.

Be a force multiplier

2021-03-02T00:00:00Z

In software development, there's always been the idea of the 10x or rockstar developer. A programmer that's so talented, that they can write software in double quick time, with their eyes closed, and also in their sleep. I've worked with developers like this. With a few exceptions, I'd never want to work with them again.

In my experience, the 10x developer is a maverick who likes to solve issues alone and not share how. Who doesn't document their complex algorithms. Who breaks code and often doesn't take responsibility for fixing up other related interfaces. Who likes the challenge of initially solving the hard problems, but doesn't want to write production ready code. Who wants to take the glory and isn't concerned with the rest of the team.

I don't want to work with 10x developers.

I much prefer force multipliers.

Force multiplier

Much like the Jedi Knights, you can use the force for good and improve life for you and your team. You can become a force multiplier in multiple areas:

Technical: Use your software engineering skills beyond your own role. Look at mentoring and coaching your team and share your knowledge with them. Or maybe look outside your own team and become an advisor for other teams and projects.
Cultural: Focus on lifting your team up and set out to improve the team culture. Make your team the best place to be such that others want to join you too.
Process: Use your skills to focus on improving the team or department's processes. Can you improve the continuous integration or deployment servers? Or improve inter-team communication?

All of these are force multipliers. Working in these areas will help to set an example of the change you want to see. You can work alone or also work with others to achieve the changes. Using force multipliers will help to lift the whole team up.

Technical force multipliers

If you're like me, you might be a manager of a software engineering team and have a technical background. You can use your this knowledge with any less experienced members of your team with technical mentoring. I run regular "team learning" sessions with my team where we look at programming concepts and patterns, and software development ideas like test-driven development (TDD).

Carving out the time each week to help your team means you'll have less time for any hands-on coding yourself, but you're helping to increase the knowledge of others. Spend some time pair programming with your team and be available as a technical advisor when people need it. Get involved in code reviews and share different solutions if you think they might help.

Being a technical force multiplier doesn't just have to be within your team. If you have the time, you can offer technical help to other teams. Share your experience of what has worked for your team and what hasn't. Share your network and introduce your engineers to engineers on other teams. Helping others that are missing domain knowledge that your team has will help both of your teams to thrive.

By being a technical force multiplier, you are encouraging others to become technical mentors too.

Cultural force multipliers

Culture can be difficult to define. It's a combination of the values, beliefs, attitudes, and behaviours shared by the team as a whole. It's how people work well together and how they treat each other.

How can you improve the team culture? Think about what a culture of diversity, positivity, and inclusivity would do to team morale. Encourage constant feedback from yourself as the team manager, and peer feedback from others. Be honest, open, and transparent in all that you do.

In my own team I've tried to build a culture where every voice is heard. I've encouraged people to share their concerns and for everyone to be honest about what they know and don't. Asking questions is encouraged. We all know that there's no such thing as a stupid questions. It's just something somebody hasn't yet learnt.

Don't forget to encourage taking time out from the day-to-day of technical work and build a culture of sharing parts of your lives that you're comfortable with others knowing. Informal chats and "water cooler" breaks are important, especially as more of us are now remote. Digital tea breaks where people can chat about life can help the team to bond. Don't wait for company mandated activities. Build them yourself with your team.

By being a cultural force multipler, you make your team, and hopefully the company, a fantastic place to work.

Process force multipliers

What slows your team down? Are code reviews taking too long? Does deploying to production take an eternity? Is your documentation left undocumented or hard to find? Look at how you can work with the team to enable making these things easier.

These problems are often best looked at from the bottom up rather than being mandated from the management team but you can help to kickstart the initiative. Look to make your day-to-day processes slightly easier each day. Once you get the ball rolling, you can chip away at the bigger problems.

By being a process force multiplier you can make the team more efficient.

Use the force

As your journey develops into more senior roles, your job might become less technical and more managerial. Use your force multipliers and expand your impact both within your team and in the wider department or company. It'll help you to grow and help all of your team on the way.

May the force multiplier be with you...always.

Become a producer not a consumer

2021-02-28T00:00:00Z

When I was young, I used to love watching TV when I got in from school. I was a child of the 70s and 80s here in the UK so I only had about 1.5 to 2 hours of children's TV until the soap operas and news started. I'd eat my dinner and then jump on to my VIC-20 or C64. They had no internet connection. We had no games without spending 30 minutes waiting for a game to load from a cassette tape. Instead, I'd write my own programs. I was a content producer.

My kids love YouTube and could spend hours on it watching videos of their favourite gamers. I love YouTube too. It's easy to watch one video and then get sucked down a rabbit hole of related videos. Before you know it you've spent 4 hours watching videos on food gadgets for your kitchen, how to light Zoom videos, or how to take better notes. Or is that just my feed?

I guesstimate that 90%+ of the people who consume content stay in that phase.

And that's fine.

But by starting to produce content you'll start to find what you enjoy. You'll discover what you're good at and want you're not. It can set a direction for your hobbies or maybe your work life. Repeat this regularly and you'll become a content producer and a creator.

Getting started with content production

Set the bar low to begin with. I'm almost at the end of a 30-day writing habit. I've sometimes struggled with attempting to write too much in each blog post, but if you aim for 200 to 500 words then it shouldn't take a long. It could be creating a quick drawing. Or a 5-minute piece of audio recorded on your phone. Just get started.

Document what you've made and share it with others. Post a link, a picture or a short video on Twitter, Instagram, TikTok or YouTube. It's a small amount of extra effort to document your work, but sharing with others will get them excited about what you do. Having a network, and then sharing your network with others, will help you to grow.

Schedule some time to create something. I've found that setting a specific time each evening has really helped me to write more. I've not always achieved it every night but I've definitely written much more in the past 30 days than I have in the previous 5 years. Taking control of your schedule will change you into a producer of interesting content and not just a passive consumer.

Turn off notifications and block your applications. I've found this hard at times. While writing this I had WhatsApp enabled on my laptop. A friend messaged me about a Twitch DJ stream I wanted to hear. It delayed me from starting to write this post for over an hour as we exchanged messages about it. I should have turned off all none essential applications and blocked any websites except this text editor I'm using to write this post. Create a quiet space away from the noise of the internet and start to create.

Build a streak and form a habit. Jerry Seinfeld wrote every day and created a habit of writing. He wasn't looking for results immediately. He was looking for consistency which helped to hone his craft and over time, it would bring the results. That's what you need to do to become a creator. Just get started.

And finally, look to repurpose your content in multiple ways. A video can be cut up for Instagram or TikTok. A blog post can be linked to on Twitter or cut up and shared as a Twitter tweet thread. Use quotes from your blog post to answer questions on Quora. There are so many ways to share your content and increase your reach.

So do you want to just sit there and watch another YouTube video? Or do you want to make something?

Get creating!

Asking for feedback as a leader

2021-02-24T00:00:00Z

I had an interesting catch up with a wonderful colleague, Blanca Garcia Gil, today. We talked about the role of a software engineering team lead and also about giving and receiving feedback. After giving me a compliment which I didn't expect, she noticed that I was taken aback by it. The thing that came to light from our conversation was that managers don't receive as much feedback as their team do and they probably should.

As a leader, your job is to guide your team and part of that is to offer constructive feedback on a regular basis. Feedback is a process that needs regular attention. If it needs to be given then it should be around that time that any situation has happened for both good and not-so-good feedback. It's not a process that should wait for a yearly review otherwise it comes as a shock. Frequent, informal feedback should feel normal and shouldn't be difficult to deliver, nor unexpected for your team.

I realised that I need to improve at asking for feedback so it made me wonder what I should do. Asking for feedback demonstrates a growth mindset and a need for continuous improvement. I know that I should be taking the initiative to get better at asking for feedback as well as giving it.

Who should I be asking for feedback from?

My manager

First and foremost, I need to be checking in with my own manager in our weekly 1:1s and asking for feedback on my own work. Sometimes 1:1s can head into a discussion of the day-to-day work of the team rather than my own development. I need to ensure that we talk about my own development and how I can improve on a regular basis.

My team

Asking for feedback from my team in their 1:1s is something I try to do but sometimes without realising it. If you've built a relationship of trust with your team, this should be a natural conversation. As our team transitioned into a new way of working in early 2020, I often asked what I could do better to support my team. If you've built a healthy team cullture then you can get some good feedback in this way and not just from the most senior or confident people in your team.

My peers

I'm currently a Software Engineering Team Lead (but this will change soon!) and I need to engage with other team leads to gain more peer feedback. We'll all be going through similar challenges so to compare how we address them is helpful to build a community around the technical leadership team. It'll help me to understand how I can improve and any changes I need to make.

Asking for feedback

Asking for feedback is key to improving as a person and leader but how should you do it? You should prepare both yourself and your colleague or manager to ask for it to ensure you're both comfortable.

Find an appropriate time

While it's easy to approach your manager and ask them for feedback, it's probably better to set aside some time when you can both discuss things in more detail. This gives both of you some time to prepare for the conversation. It can be a good idea to forewarn them of your expectations.

Prepare questions to ask

Asking "can you give me some feedback" without context is too general and a hard question to answer. Instead, preparing some questions in advance, and potentially sharing them with your manager, could be helpful.

Perhaps you want to ask about specific strengths and weaknesses or other areas you'd like to concentrate on. Some examples could be:

How can I get better at supporting the mission of our team?
What advice would you give me to help me improve my communication?
How can I prepare for our upcoming project X?
Which parts of my presentation could have been better?

Asking for specific advice makes it easier for you to understand the areas you need to concentrate on in order to grow.

Take notes

Anyone that knows me, knows that I take a lot of notes. I'm currently a big fan of Obsidian, which allows me to link concepts in my notes together really easily, but a pen and notebook will do just as well. Ensure you capture any ideas from your manager or peers as to how you can improve.

Learn from the feedback

There's little point in asking for feedback if you don't then use it. Take some time to reflect on what's been said, and the notes you've taken, especially if there is room for improvement. Consider how you could apply any suggestions to how you work and make a development plan for yourself. Perhaps creating a list of goals would be useful here.

Say thank you

Finally, don't forget to thank people for taking the time to give you feedback. It's not easy to give feedback and we should all be doing it more regularly than we probably do. Make sure you thank people for their time and thoughts and use it to help your personal development and growth.

How do you measure success?

2021-02-18T00:00:00Z

The dictionary defines success as:

“The accomplishment of an aim or purpose”

Everyone wants to feel successful. But what does it mean to you? What is the aim or purpose of your life?

Peter Drucker, one of the great theorists on management techniques, has been attributed with the saying:

“You can't manage what you can't measure”

But how do you measure success?

Create your own definition of success

I've got a friend who often compares themselves to others. How their own life is not as good as mine or one of our mutual friends. They tell me how successful I've been and how they've not. I live my own life optimistically. I've always felt that this was the wrong way to consider success by looking at what others do. Success should be what's truly important to you and to nobody else.

Everyone's definition of success is different.

Your definition of success will be different to mine. Many people look successful from the outside but only they can define what it means. They may not feel as successful as you think they are.

Success changes over time

When I was in my late teens, I started DJing as my DJ Cruze alter ego. I came to Manchester for university in 1990 and slowly started DJing at the student nights. I met my good friend Mark and sent him DJ mixtapes over to Germany when he lived out there. Before I knew it, I was DJing every other weekend in Germany and in Manchester and I became a pretty successful DJ. It felt great to spin new house music tracks to adoring clubbers every weekend. This continued for almost 18 years but towards the end, it felt less and less like success. Sometimes it felt like going through the motions and playing the music because I felt I had to rather than I wanted to.

It's never easy to admit, but sometimes your idea of what success looks like changes. We should embrace this but we often don't.

I got married in 2006 and we had our children in 2007 and 2009. These key moments in my life changed my measure of success. It was no longer about partying every weekend. Having a happy marriage and a healthy family meant much more to me than anything else.

How I measure success now

I was incredibly ill in 2014 and almost died. I was training for a half marathon at the time and one of the consultants commented that having exercised as much as I did, and therefore having a healthy heart and lungs, was one of the key factors in keeping me alive. Being healthy is one of the key success criteria for me in the past few years and will continue to be so in 2021. I don't think I'm necessarily as healthy as I could be, but I'm definitely walking and cycling much more than I ever have. I want to ensure that I keep active and eat well and I encourage my family to do the same. Being out in the fresh air and enjoying life after near-death feels like a big success for me.

I'm lucky that I found a career that I enjoy. I got my first home computer, a Commodore VIC-20, when I was about 11 or 12 and I've continued to love computers and software development since those early years. Getting to work with people who also love writing software and to be able to help them to develop their own careers is another key success measurement for me. It's helped me to earn a good salary for what I do, and it's enough to keep my family and I well cared for.

In my earlier career, I worked in the video games industry. While I loved games, and still do, the industry suffered from a tendency to push the developers to work beyond the normal working week. The so-called "crunch time" was one of the main reasons why I decided to move out of games and into the world of web-based applications. One of my other success indicators is to ensure I have a good work-life balance. There are other ways I could look to earn more money in the software industry but I'd never want it to come at the expense of my family. A healthy work-life balance will always been one of my key success measures nowadays.

While 2020 and 2021 have been years like no others, I see my own success as something I love to share with others. Sharing my money, time, experience, or help with other people is something I cherish. Spending time with friends and family, or helping others I perhaps don't know as well, is something I love. A successful life is a happy life and being able to share it with others really makes me happy too.

Don't compare yourself to other people's definition of success. Only you can define your own values and what it means to be successful. Define your own succesful life and strive to make it happen.

Share your network

2021-02-16T00:00:00Z

I went to the Lead Developer Conference back in 2019. Pat Kau gave a fantastic talk called "Flavours of technical leadership". In the talk, he shared different ways that engineering leaders develop and make a difference to their teams. One of his thoughts really resonated with me:

“Share your network”

Different types of networks

Anyone who has been in an idustry for a while has a network of people that they know and trust. This builds up over time and expands the impact and opportunities you have.

There are differing networks you can be a part of. The Harvard Business Review's "How leaders create and use networks", Herminia Ibarra and Mark Lee Hunter suggest that there are three main types of networks:

1. Operation network

An operation network consists of people that help to fulfil your current responsiblities and day-to-day tasks. They may be people in your immediate team and direct managers or stakeholders.

2. Personal network

You can find people outside of your team, department, or company who can help you or you can help them. This is your personal network. They can be people with similar interests or thinking and you can leverage their help for your own success. Always remember that you should be helping them too.

3. Strategic network

A strategic network are people within or external to your organisation which will help you to find new directions for your team. They may be external stakeholders that you can work with or find support from. They'll help to open up interesting opportunities for you and your team.

These three networks will consist of different people who will be valuable to both you and others that you introduce them too. To help to build a high-performing team, you can make introductions to others in these networks and share your network.

I wrote a little bit about sharing your network in my article on lifting your team up. Most strong leaders use networking to develop relationships in their team and beyond. Talking to people is a good way to forge personal and professional relationships and enable you and your team to do your best work.

By introducing your team to others, you're creating new opportunities. These can help your team to grow and learn from people with different experiences than your own. You'll find the potential for mentoring and coaching outside of your own knowledge and this will help your own team to thrive. Networking with others is important for your own career so by sharing the network you have it helps to grow the careers of others. You will understand your team's needs and can help them to connect with relevant people.

An introduction to a senior leader or engineer is an endorsement or sponsorship from someone trusted. Hopefully you've gained a reputation within the company as being someone reliable. By encouraging your team to talk to people outside of the department, you're subconciously telling others how much you trust and believe in your team member. This goes much further than you think in helping to grow their careers.

What differentiates a leader from a manager, is the ability to understand who to talk to enlist the help that is needed. By sharing this with your team, you're unlocking their ability to become independent and grow their potential both with and without you.

Take a look at your own network today and see who you can share it with. Empower your team to be amazing.

Watch the talk

https://www.youtube.com/watch?v=YbGwoU2Ow4A

References

How leaders create and use networks

Remote working is the future

2021-02-13T00:00:00Z

People have talked about remote working in the technology industry for many years. Some companies, such as Basecamp, GitHub, and GitLab have embraced it for years whilst others have pushed back, expecting most teams to co-locate in office spaces. Some have had a mix of remote-friendly teams who may have some split of the two.

And then 2020 and Coronavirus COVID-19 changed everything.

Many countries introduced lockdowns to stop the movement of people and the spread of the disease. Companies had to quickly switch their working practices to allow people to work from away from their offices. For those that had already embraced remote-first working practices, the transition was easier. For those that had not started to look at remote working, I'm sure it was much more difficult.

After almost a year of remote working, I hope that most technology companies will move towards remote-first thinking. I believe it's the future of work.

It won't be easy but it'll be worth it

I became very ill in 2014 and almost died. It took 5 months for me to return to work and thankfully the BBC were great in allowing me to work remotely to a team which was mostly based in London.

However, being the only remote working to a co-located team wasn't easy.

Being an individual on the end of a Skype or Zoom call was tough. A room full of people all talking to each other with a single laptop in front of one person often meant that I couldn't communicate effectively. Trying to be heard over a meeting room of people who can easily interject and take visual cues from each other was tough. It took some conversations with my team, and a long time, to help to drive a change towards "remote-first". This is when you build your team's culture around the idea that everyone should be considered remote, even if co-located in the same office. I helped to drive this change in culture in our team and for the past 3 or 4 years, we've definitely become remote-first.

Constantly iterate

Remote working doesn't mean attempting to replicate what you do in the office but from home. It shouldn't mean a day of Zoom calls which mirror office meetings.

Companies and teams need to get better at changing what remote working looks like. Companies like GitLab have been running 100% remotely and have built their practices around their teams and how they collaborate. Communication and the discoverability and visibility of that information can become more difficult in the remote world. Defining what information is required and where it should live will make your team's life easier.

As technology teams, we have to become better at writing instead of communicating face-to-face. Starting to think about remote ways of working like asynchronous meetings will help us embrace remote-friendly ways of working. Being clear and concise in our written communication will allow us to work together but apart.

We'll have to get better at building software remotely too.

I've always liked pair programming as a process which helps to write better software. While it's a great way to share code, enable a mentoring relationship, and spot issues earlier, it can be more challenging in the remote environment. The expectations of having the same schedule and for pairing over Zoom for 5 hours idea can be tiring. We potentially need to look at new ways of working collaborating more effectively. If we look to the open source communities, who have been building great software in public but by distributed software engineers, we can think about new ways to build applications.

Thinking about the way we work will be necessary to avoid bringing old office ways of working to the remote world. Being more communicative, organised, and intentional in what we do will be great for everyone.

Companies will adapt

There will have to be big changes to the way that companies build their teams.

Having remote-first teams will mean we can start to hire people from anywhere. Having flexibility around how we work will allow us to look more deeply at diversity and inclusion in our teams. Being able to work asynchronously can help us to encourage people who wouldn't normally feel that technology is for them to join our teams. Talented people no longer have to be within commuting distance of an office space. We'll have to ensure that we can support new members of the team with better remote on-boarding, mentoring, and coaching. People like me, in engineering management, need to make this better.

If we no longer need teams to be in offices, companies might consider cutting their commercial office spaces. These may become more like tech hubs where teams can come together and collaborate from time-to-time instead of all of the time. Some companies may decide to get rid of the office spaces all together. I hope that companies will look to spend some of the money they save by not needing office spaces on enabling great remote setups for their employees. For example, we all need good chairs if we're not in an office space. Or allowing people to work from shared spaces should they wish too and perhaps don't have the ability to work from home as easily.

Companies need to adapt to the new normal of many people not wanting to return to the 9 am to 5 pm, 5-days per week in an office. They want to save their commuting time for time with their families, to exercise, or to just get up later!

Remote working is the future for those companies that can embrace it. While there may be a mix of teams coming together in real life, and also working remotely, I can't see a future where companies will bring teams back into the office for 5-days a week. I think we're being naive if we believe that their teams want to do that even after the pandemic. Most of the larger technology companies are now publicly stating that they will be building remote-friendly teams. And this is a great step forward from the past few years.

The future is remote-first working. The pendulum won't swing back now.

References and further reading

Writing great pull requests

2021-02-10T00:00:00Z

Teams want to write great code. They want to communicate and collaborate on their software. Sharing code via a pull request is a great way to present your code for review by your peers.

A pull request is a way to create a new feature branch which you're asking to be merged into your main branch. It's also a way to document and share your thinking around the changes and allow others to review the code and suggest changes.

Here are some ideas as to what you can do to help to write a great pull request.

Why does it matter?

Keeping your codebase simple and easy to read is hard. Ensuring code consistency isn't easy. Reviewing code can be difficult and defects can easily occur without the whole team helping to validate any potential changes.

Each change can be neatly captured in a pull request. Keeping these pull requests descriptive, easy to read, and consistent allows your team to use their mental energy on reviewing the code. It avoids having to jump through hoops to understand why the change is being made.

Make small pull requests

It's difficult to review large pull requests. If you have too many changes then reviewers tend to skim read the code and often merge it without a thorough review. A study by SmartBear showed that developers should review no more than 200 to 400 lines of code at a time. The brain can only process so much information and beyond 400 lines of code, it was shown that the ability to spot issues was greatly reduced.

Look to make your smaller pull requests and then you can raise them more often. Try to have a single context for each new request. Sometimes you think "I'll just fix this other thing while I'm here". It's a great idea to refactor your code while in the codebase, but maybe save it for a separate pull request to avoid losing the context around the current changes. Having atomic contextual changes makes it easy to review.

A useful title

We often go back through the code and look at closed pull requests. Writing descriptive titles allows you to see what happened for each set of merged changes.

Add a summary of what changes have been applied for this pull request. This should be a short description of the changes at a high level. If you use a feature or defect tracking application such as Jira, you can use a pattern of adding a ticket number to the title as my team currently does. This helps to have a quick reference to a potentially more descriptive document with more context around the changes.

You can also add a prefix to your title to define the type of pull request. For example, adding a [WIP] (work in progress) would suggest that the pull request is unfinished. Or adding [RFC] (request for comments) could suggest that it's not yet a mergeable pull request and that you'd like people to comment on the suggested architectural pattern.

GitHub also allows you to create a draft pull request which can be reviewed and commented upon but not merged into the main branch. This is a useful way to review an idea before embarking on the full implementation of the code.

Tell them what changed

Next up is the description for the pull request.

Firstly, we should tell the reviewers what has changed. Your job here is to make it easier for the reviewer to see what this pull request achieves.

Write a high level description which explains the changes being made and what the expected results are. You can also add a list of the key changes that have been made so the reviewer can see them at a glance. Don't just make this the list of commits to the request. Those can be easily seen as the commits to this pull request. Instead, list the key changes which matter.

Tell them why its changed

In addition to telling the reviewer what has changed, it's good to give a context as to why this change is happening. If it's related to other pull requests or architectural changes, add links to these documents so the reviewer can read them. Often the developer reading the code won't have been involved in the changes and may not have the full picture of why this change is necessary. Given them a solid context as to the reason for the pull request will definitely help with their understanding.

Adding screenshots or videos to your pull request makes it much easier to review. For visual changes, the screenshots could be a before and after shot to show what changes the pull request provides. For more complicated changes, a short video is a fantastic way to share information about user flows and expectations.

Tags and labels

If your pull request references previous changes, make sure to link them in your description. You can tag other team members if you think that they can provide additional context around the changes or you want them involved in the discussion. In GitHub you can also tag teams if it's relevant to a whole team and not just individuals.

Use a pull requests labels to give the reviewer some additional information around the changes. This could be a bug label for a quick fix or a do not merge label if the code is not yet ready for prime time. You can add your own team's custom labels if relevant too.

Keep consistent

The best thing for a team to do is to keep a consistent pull request format. Context switching between pull requests with different information in them can be challenging and slow down the review process.

GitHub provides the ability to add a standard pull request Markdown template which gets added to the description when a pull request gets raised. This is a useful way to capture the information above into a consistent format. For GitHub, this involves creating a file named pull_request_template.md in the .github directory of your project.

Wrapping up

These are just some of the ways you can help to write great pull requests. They are not mandatory rules but they've definitely helped my team to write better pull requests over the years. As with all processes, they should adapt and change over time. Your teams needs might be different from mine. Take the best ideas and add your own.

Let me know if you have any other fantastic ideas for writing great pull requests by sending me a message or tweeting at me.

Useful resources

Don't repeat your years

2021-02-09T00:00:00Z

I recently attended an "away day" for the leadership team of our department. In early 2021, an "away day" obviously means another set of Zoom meetings. It was a great day of digging into the challenges of the team and there were a couple of guest speakers with amazingly inspiring talks.

One of the speakers talked about her experience in moving from being a managing director at a newspaper to becoming a business manager at the BBC. She had to work her way up again after having senior role. And then she said one thing that really resonated with me:

“Don't repeat your years.”

She'd stayed in a role for over a year because she didn't believe she was ready to take the next step up even though she had the experience. She taken the easier route because she didn't yet believe in herself and the skills she had. She did the same thing she'd done the previous year for the next 12 months.

She'd repeated her year.

Complacency kills time

Time is scarce. Life is short. I know this as I almost lost it. If you don't use your time wisely, it slips away like the sands in an hourglass. You never get that time back.

As a software engineer, and now as a team lead and manager of software engineers, I enjoy a challenge. I enjoy helping to creating a new application. I love digging through a codebase to understand how it works. I can't wait for my 1:1s with my team and to help them on to achieve their career aspirations. I love making an impact in the code or by helping my team.

But sometimes you go through the motions and find yourself no longer learning new things.

If the work lacks challenges, or even worse, you do the same things that you've done in the past month or two, it's easy to become complacent. Work shifts towards boredom and you lack the motivation to rise to new challenges. You can easily do the same thing without feeling a sense of purpose.

It's easy to repeat your years.

Don't waste your career

I read a fantastic article called "How to waste your career, one comfortable year at a time" by Apoorva Govind. It's a great post so please take the time out to read it. Apoorva lays out a framework for deciding when it's time for a change. It's an incredibly useful way to determine if you're wasting your years. She suggests looking at the following every quarter and giving yourself a score in the following areas.

Accomplishment - Have you achieved anything significant in the past three months?
Impact - Could you write a new line in your resume about the things you've worked on in the past three months? If you were hiring for a role, would you value this experience?
Growth or future alignment - Have you acquired valuable insights or skills? Do these align with how you want to grow into your career?
Challenge - Do you spend your days thinking of the challenges at work? Do you try and solve these problems in the shower? (I love it when you get to do this!)
Community - Are you happy to go to work every day? Even if this is on Zoom? Do you believe in the mission, vision, and leadership of the team or company?

If you mark each of these five areas together as a percentage, and you're scoring less than 50%, you have to question if you're still feeling useful in your role. If you're not then it's time to change and avoid becoming stuck in your role.

Time for a change

If you feel you're repeating your years, it doesn't necessarily mean a new job.

It can be moving teams.

It could be working on a project you're less familiar with.

If you have the option to do this then ask to change and see if this changes your framework scores.

If not, it might be time to think about looking for a new role elsewhere.

Don't live the life of others

You have to shape your own life. It's easy to plod along and feel a lack of achievement. It's especially easy to do this in a larger organisation. But it's up to you. You have to shape your career yourself, and maybe with the help of a supportive team lead like myself.

I read an article from a palliative nurse who recorded the most common regrets of people on their deathbeds. This one stood out to me.

“I wish I'd had the courage to live a life true to myself, not the life others expected of me.”

Don't spend your own life waiting for life to happen to you.

Be proactive in doing what you want to do.

Don't repeat your years.

Create an Eleventy podcast feed

2021-02-08T00:00:00Z

At the end of last year I converted my DJ Cruze website from Wordpress to Eleventy. Much like this website, it's easy to deploy using Netlify. Moving to a statically generated site made it much easier to update and avoided me having to maintain a server, PHP, Wordpress versions, and so on.

For many years I created the DJ Cruze podcast show so I wanted to move over the podcast RSS feed from Wordpress to Eleventy. Creating a custom RSS feed based on my podcast episodes took a bit of work so I thought I'd share how I did it.

Create the podcast data file

Eleventy supports a wide range of data sources.

For my podcast I define a JSON file which contains the metadata needed for the podcast. All of this data will be used to populate the RSS feed. You may have this information in other data files in your Eleventy site but I wanted to keep most of the data for the podcast together.

It's stored in the data directory like this: /data/podcast.json. This will expose a global data object called podcast which matches the filename of the JSON file and can be used in your template files.

{
  "title": "DJ Cruze House Music Podcast",
  "description": "DJ Cruze is in the house! Spinning funky and chunky house music since 1988. Manchester is in the house!",
  "category": "Music",
  "author": "DJ Cruze",
  "channelImage": "/images/podcasts/dj-cruze-podcast-logo-1400x1400.jpg",
  "owner": {
    "name": "Marc Littlemore",
    "email": "info@djcruze.co.uk"
  },
  "feedPath": "/podcasts/feed.xml",
  "episode": {
    "defaultDescription": "The latest episode of the DJ Cruze podcast features new and old funky and chunky house music."
  }
}

Most of the metadata properties should be obvious.

Both iTunes and Google Podcasts support the standard RSS schema but with additional tags. Google Podcasts is essentially the same as an iTunes feeds and uses the iTunes specific tags in its feed too. Make sure that your category property matches the expected iTunes categories and be careful that you also match the correct case. If you don't, it may well be rejected. In my case I'm using the "Music" category. Note that you can use sub-categories too but I'm not in my example.

Set up a podcast post

In Wordpress, each of my podcasts was a new post with a specific tag. I exported these as Markdown files and added some custom frontmatter to mark it up with per-podcast metadata.

---
media:
    # The episode number
    episode: 57

    # The image file for this episode
    # You can ignore this if you don't want per-episode artwork
    image: '/images/podcasts/dj-cruze-podcast-episode-57-june-2011.jpg'

    # Your MP3 file 
    content: '/podcasts/dj-cruze-podcast-episode-57-june-2011.mp3'

    # The duration of the episode in seconds
    duration: '4230'

    # The filesize of the MP3 file in bytes
    fileSize: '67973718'

    # A per-episode description if you want it
    description: 'Another DJ set of funky house music from DJ Cruze!'

# Additional layout data here...
---

I use Eleventy's directory specific data files to mark up all podcasts with the tag podcast. The enables us to create an Eleventy collection of podcasts to iterate through.

{
  "tags": ["podcast"]
}

Custom filters

Our podcast feed needs some custom filters adding to the Eleventy configuration.

RSS feeds expect some dates using the RFC822 date format. There's an npm package we can install to do the conversion for us. Install it as a development dependency like this:

npm install --save-dev rfc822-date

We also need to escape some of our text to ensure any special characters are encoded correctly. I use the Lodash escape method to do this. Again, lets install this a development dependency as follows:

npm install --save-dev lodash.escape

Lastly, we need to know what the date of the last episode was so we can add a special LastBuildDate tag. This tells any clients when the podcast was last modified. By adding this, any new podcasts will update to the latest date and allow new episodes to be downloaded.

Here are the additional filters we add to our .eleventy.js configuration file. I've only added the filters we've created for the podcast feed and intentionally left out any other Eleventy configuration.

const escape = require('lodash.escape');
const rfc822Date = require('rfc822-date');

module.exports = (eleventyConfig) => {
    // RSS
    eleventyConfig.addLiquidFilter('rfc822Date', (dateValue) => {
        return rfc822Date(dateValue);
    });

    // Escape characters for XML feed
    eleventyConfig.addLiquidFilter('xmlEscape', (value) => {
        return escape(value);
    });

    // Newest date in the collection
    eleventyConfig.addFilter('collectionLastUpdatedDate', (collection) => {
        if (!collection || !collection.length) {
            throw new Error(
                'Collection is empty in collectionLastUpdatedDate filter.'
            );
        }

        return rfc822Date(
            new Date(
                Math.max(...collection.map((item) => {
                    return item.date;
                }))
            )
        );
    });

    // Rest of Eleventy config goes here...
};

Template file

Now we can create a post with a custom template in it to define our podcast URL. In this template we can iterate over our podcast collection and render each podcast episode correctly into the expected RSS XML format.

I created a podcast Liquid template file with a /podcast/feed.xml permalink. I'm excluding this file from other collections and from my sitemap. You might want to do this too.

The podcast feed is split up into the intial channel metadata which is mostly created from our podcast.json metadata. Ensure that any text is escaped correctly by passing it through the escape filter.

As I'm attempting to redirect my old Wordpress theme from an old URL to a new URL, I've also added in the itunes:new-feed-url tag as follows. You won't need this unless you are migration your podcast feed too.

<itunes:new-feed-url>{{site.url}}{{podcast.feedPath}}</itunes:new-feed-url>

The second part of the podcast feed iterates through our collections.podcast feed in reverse order. In that way, we always get the latest episode first in the RSS feed. The metadata used for each episode comes from the Markdown frontmatter that we defined earlier.

The enclosure tag defines the media file needed to play the podcast. As I host my podcasts on a different URL to the DJ Cruze site itself, I have an additional URL in my site.json data file which defines the external server to load them from. This explains the site.mediaFilesUrl URL which is prepended before the content path.

Here is the full RSS template.

---
permalink: "/podcasts/feed.xml"
eleventyExcludeFromCollections: true
sitemap:
  exclude: yes
---
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>{{podcast.title | xmlEscape}}</title>
    <link>{{site.url}}</link>
    <language>en-us</language>
    <copyright>&#169; 2005 - {{"now" | date: "%Y"}} {{podcast.author}}</copyright>
    <lastBuildDate>{{collections.podcast | collectionLastUpdatedDate}}</lastBuildDate>
    <generator>Eleventy</generator>
    <description>{{podcast.description | xmlEscape}}</description>
    <itunes:author>{{podcast.author}} </itunes:author>
    <itunes:explicit>false</itunes:explicit>
    <itunes:type>episodic</itunes:type>
    <itunes:image href="{{podcast.channelImage | prepend: site.url}}" />
    <itunes:owner>
      <itunes:name>{{podcast.owner.name}}</itunes:name>
      <itunes:email>{{podcast.owner.email}}</itunes:email>
    </itunes:owner>
    <itunes:category text="{{podcast.category}}" />
    <itunes:new-feed-url>{{site.url}}{{podcast.feedPath}}</itunes:new-feed-url>
    
    {%- for podcastEpisode in collections.podcast reversed -%}
      <item>
        <pubDate>{{ podcastEpisode.date | rfc822Date }}</pubDate>
        <link>{{ podcastEpisode.url | prepend: site.url }}</link>

        {% if podcastEpisode.data.media.guid %}
          <guid>{{ podcastEpisode.data.media.guid }}</guid>
        {% else %}
          <guid>{{ podcastEpisode.url | prepend: site.url }}</guid>
        {% endif %}

        {% if podcastEpisode.data.media.title %}
          <title>{{podcastEpisode.data.media.title | xmlEscape}}</title>
        {% else %}
          <title>{{podcastEpisode.data.title}}</title>
        {% endif %}

        {% if podcastEpisode.data.media.description %}
          <description>{{ podcastEpisode.data.media.description | xmlEscape }}</description>
        {% else %}
          <description>{{ podcast.episode.defaultDescription | xmlEscape }}</description>
        {% endif %}
        
        <enclosure
          url="{{podcastEpisode.data.media.content | prepend: site.mediaFilesUrl}}"
          length="{{podcastEpisode.data.media.fileSize}}"
          type="audio/mpeg"
        />

        {% comment %} iTunes specific tags {% endcomment %}

        {% if podcastEpisode.data.media.episodeType %}
            <itunes:episodeType>{{podcastEpisode.data.media.episodeType}}</itunes:episodeType>
        {% else %}
            <itunes:episodeType>full</itunes:episodeType>
        {% endif %}

        {% if podcastEpisode.data.media.image %}
            <itunes:image href="{{podcastEpisode.data.media.image | prepend: site.url}}" />
        {% endif %}

        {% if podcastEpisode.data.media.episode %}
            <itunes:episode>{{podcastEpisode.data.media.episode}}</itunes:episode>
        {% endif %}

        {% if podcastEpisode.data.media.season %}
            <itunes:season>{{podcastEpisode.data.media.season}}</itunes:season>
        {% endif %}

        {% if podcastEpisode.data.media.duration %}
            <itunes:duration>{{podcastEpisode.data.media.duration}}</itunes:duration>
        {% endif %}        
      </item>
    {%- endfor -%}
  </channel>
</rss>

If you head to my DJ Cruze podcast feed, you can see all of the episodes rendered correctly.

Next steps

There is some metadata which could be automatically generated from the media files. I add to the duration and file size properties to my Markdown frontmatter for each podcast episode. I do this by hand but it could easily be generated by reading the file when Eleventy builds the feed.

Here is the DJ Cruze website on GitHub. Feel free to take a look through the code and adapt it for your own use.

I hope you find this useful. Let me know if you have any questions or comments.

Useful resources

Always be learning

2021-02-07T00:00:00Z

We often think of learning as something we do at school, college, or university. You learn new skills as you start your career but often, people stop thinking they need to continue as they get older.

It often feels like learning is something that we encourage our children to do but we forgot how fun it can be as adults. Learning something new encourages those dopamine hits and makes you feel motivated. If you challenge yourself and solve small problems by learning something new, you'll soon find that childlike wonder and love of learning all over again.

Learning is natural - embrace it.

Most knowledge workers like me know that our industries change quickly. I've learnt so much in the past 25+ years as a professional sofware developer. Computer languages change. Software development environments are constantly evolving. Every week in the world of JavaScript there's a new framework or library. Software development changes at an incredible pace but that doesn't mean you have to learn everything that the software industry throws at you. Think about just-in-time learning and learn as you need them.

Companies look for people who can learn new skills quickly and embrace change. If you can constantly adapt to new challenges, you'll never be out of a job.

What stops people from learning?

It's hard work. You have to put some effort in to learn something new. But the challenge of the unknown is worth it for the rewards you'll find.

Taking notes and learning a new skill helps to clarify your thoughts. If you expose yourself to new ways of thinking, you'll know if your current thinking is effective and worth continuing iwht. You can find different, and often simpler, ways of achieving the same goal with your new found knowledge.

There are so many places to learn new skills now.

Lots of people are sharing their knowledge online so you can find online courses for most things. Take a look on Udemy, Skillshare, or just use Google, to learn anything from writing, machine learning, or photography, to beer brewing and watercolour painting. You name it and there will be a course for it.

YouTube is filled with people teaching and this is all free. There are fantastic tutorials from talented people so dive in and take a look.

If you prefer reading then Google for tutorials on the topic you want to learn or browse through Amazon to find a book on the subject.

Make learning a priority. If you're not learning then you can quickly become stagnant. Learn how to learn. And don't forget to share your notes and learning in public.

Always be learning.

Creating an idea habit

2021-02-06T00:00:00Z

If you're playing along at home, you'll know that I embarked on a 30-day writing habit challenge to myself.

I want to attempt to kickstart a habit, and feel like I was getting better at putting pen to paper, or fingers to keyboard. I don't think I'm a bad writer, but I've always struggled to get started with writing.

Once I start writing, I love it. But some days I hit a roadblock.

What should I write about?

A habit of ideas

I found a great Twitter thread from Mark Shpuntov which summarised a workshop where Ali Abdaal and David Perell discussed "building an idea factory". I thought this was a fantastic description of what your brain could become.

Very few people have 100% original ideas. You don't have to constantly come up with unique ideas to create good content. When you combine an idea with your voice and your knowledge and experience of somthing, it becomes original.

Ali Abdaal is a doctor and top YouTube creator in the productivity space. He suggests a some simple ways to generate new ideas.

Find content creators that you watch regularly and consider them to be great in their area.
Be inspired from their content and watch or read what they create.
Create your own ideas and content based upon their work but give it a unique angle.
Make sure you reference the original idea and creator.

You can ensure that you don't feel like you're just stealing their ideas by using more than just a single source to create your own. Read multiple articles or watch videos from different creators to gain the initial inspiration but apply your own thinking and experience, to enable you to make it different.

To be more creative you can diversify the content you consume. As Steve Jobs once said:

“Creativity is just connecting things”

Take inspiration from different places, different fields of study, and different places. The more things you can connect, the better your ideas will become. Try to consume quality content and look for inspiration and ideas in unique books, documentaries, YouTube videos, podcasts, and blogs.

One thing that I'm trying to get better at is capturing the things I learn using Obsidian. Good ideas are everywhere but our brains can struggle to notice them. As with most things, I believe that this can be improved with practice. Having a reliable process for capturing ideas is a great way to let your brain think of the ideas and then dump them into a place that you can curate them. Pay attention to any ideas you may have and attempt to write each one down.

Telling others your ideas can help you to clarify your thinking and maybe allow you to see it in a new way. This could be writing it down in a blog post like this, tweeting it out on Twitter, or having a chat with others. Having a discussion allows people to ask more questions which will lead to new ideas and thinking.

The last thing I'm planning on doing before my next blog post is to brainstorm ideas. I have a lot of diverse interests and sometimes it's hard to slim down all of my thoughts into simple ideas. A 30-minute session where I concentrate on coming up with as many thoughts as possible will help me to consider options and ideas I'd not previously considered.

It's time to break out the mind mapping software before tomorrow's post.

Building an audience

2021-02-03T00:00:00Z

Building an online audience is a great way to build your career.

The more people who see your writing or videos, the more your network can expand. This can help you to connect with more people and create more opportunities for your career.

This video by the writer and entrepreneur Nate Eliason shows us how he builds an audience for his top-selling online course - Effortless Output in Roam.

Here are my notes from watching it and learning from an online expert.

The video

https://www.youtube.com/watch?v=LSrcELpdgX8

Lessons to learn

Start an email list

Most people start posting their messages on social media. A new tweet, an Instagram photo, a post on Facebook, or a YouTube video only gets you so far. Twitter, Facebook, or Google own their platforms and they control how many of your followers see your content. They decide what the people see. And they decide if you're allowed on the platform.

By building an email list, you control the distribution of your content and message. It's not uncommon to see 30 to 50% open rates of emails if the audience is engaged and wants to hear from you. With an email service provider, you control the emails in your list. You have permission from the users to email them. You can export the email addresses to a spreadsheet and take them to another email provider with ease.

Nates likes ConvertKit and I've had a good experience with Mailchimp.

But making an email list isn't enough...

How to get people to join your list

There are a lot of emails lists and people aren't going to sign up to yours without a compelling reason to do so. Just asking people to sign up to your newsletter isn't enough. You have to create a "super freebie".

A "super freebie" is something so good that they don't even think twice about signing up to your email list. For example, Nate and his friend Justin Mares created a 7 day series of free lesons for their technical marketing course. Every lesson was at least 1500 to 2000 words long with screenshots, videos, and free code examples. They made sure that each daily lesson had a really high value. The subscribers were so blown away by the free content, that they were asking for the paid course before finishing all of the lessons!

Some other examples of a high value "super freebie" are:

A valuable digital download
A video course
Free templates for a design tool
Free code snippets

It has to be so good that people don't hesitate to share their email address to get it.

The easiest way to start collecting email addresses, and starting to build your audience, is to set up a landing page. On the page you should explain to them what they're getting when they sign up. On all of your future videos, articles, or social media posts, you can send users to this landing page. It will have a clear call to action and it'll be understood what they'll receive when they join your mailing list.

But how do you get people to show up on the landing page in the first place?

Involve others

When you're first starting out, you probably don't have much of an audience. While you wait to build up your own followers, you can leverage other people's credibility by involving them in what you're doing.

Interview a dozen experts in the field of what you're trying to build a product around. Share those interviews on your blog, in videos, or on a podcast. Point the audience at the "super freebie" that they should look at if they found the interview useful or interesting. If you do a good job with the interviews, and all of your guests feel appreciated, then they'll be happy to share the article with their audience. This will help to build your credibility and drive traffic to your landing page.

This isn't just about you though.

It should be a win-win deal where you're helping them. If you give them more value than they're giving you, then they'll be happy to help you boost what you're doing. People know when you're being selfish and just out for yourself.

People can tell if you're just wanting to promote your own work. If you're reaching out to people to help them, don't ask them to write anything. Writing is an intensive and time consuming process. Asking for a guest post takes a lot more energy than asking for an interview. It's much easier to show up and talk about their life for 30 minutes than it is to write 1000 words. If you do want an article, ask them for some old content. This could be something that they'd previously shared with their own audience.

Once you've created your content, offer it out to them and their audience to share as part of their products too. Give them full permission to use it as their own. This makes it really easy to build a mutually advantageous relationship which helps you both.

The more you help others, the easier it is to promote your "super freebie".

Promote the super freebie

Once you have an email list and an audience, this will be the main way you promote your work. However, this is the chicken-and-egg situation. You need an audience to build an audience. This is where online communities can help.

Look for Reddit subreddits or Facebook groups that focus around the subject you want to build your product around. You should be able to find some of these communities online which care about your product. Actively participate in the conversations in these groups and bring value to them. Over time you'll be able to introduce your "super freebie" or product that you're working on. Don't you try and sell to them immediately or it becomes annoying. It's all about building relationships with like-minded people so you can kickstart your own audience.

Using SEO for passively growing the list

For long-term audience growth you should be considering SEO for your video or blog content as you create it. For Nate, he put up the technical marketer lessons on the blog and over time they started ranking for keywords like "automate your twitter activity" and "automatically reply to tweets". People would find the content, read it, and sign up to the email list to get all of the other lessons in the course. This drove between 10 and 30 people per day onto the email list without them intending for this to happen.

With Nate's Roam Research blog post, he managed to get the article on the first page of Google for the keywords "Roam research". Once someone has found the article and read it, there's a call to action at the bottom to sign up for the email list to learn more about how to use Roam. This is another great way to build up the audience over time.

Encourage referrals

Morning Brew is a huge email newsletter that has recently been acquired by Business Insider. They grew successfully by encouraging people to invite their friends to join up in exchange for Morning Brew products. This baked virality into their newsletter so that every sign up brought in more people until they had millions of subscribers. But you don't have to use any fancy referral software to do this.

When someone signs up to your newsletter, send them an email asking them to share it on Twitter. In your email you can include some simple share links for Twitter or Facebook or wherever you want them to share it. You can then use your email tool, like ConvertKit to track anyone who clicks on one of the share links and send the orginator another email with a link to an additional freebie. This gives them another great reason to share our content.

For example, Nate and Justin had an additional PDF for their technical marketer course which shared some common automation tasks with Zapier that people would get if they shared the course with their friends. They found that 20-30% of the people would share the course on Twitter to get the new valuable freebie. This brought in a few thousand subscribes.

Put your content on a blog

Finally, you should always share the content and link to your freebies on your blog. Starting a blog is a great way to start building that audience.

Hopefully you enjoyed these notes on Nate's video and found them useful.

Ship your code with confidence

2021-02-02T00:00:00Z

All developers want to ship code on a daily basis. But how can we ensure that the code we're pushing out to our users won't break?

There are plenty of good practices we can use to ensure we can confidently release new versions of our applications. Here are a few that I think are really valuable.

Version control

I'm always surprised when I have to mention this, but I've still met some teams and individuals who still don't use version control for their source code. Even if you're a solo developer, make sure you use a source control system. Having revisions of your code is a must for everyone's sanity. Having previous versions of your code makes it easier to spot any differences should you have errors when you deploy to production. You can also use this revision history to roll back to a previous version should you need to.

Write tests and run locally

When you write your first computer program, you test it manually as it often doesn't do a lot. Once things get more complicated, manually verifying all of your changes can take a long time. And how do you know that you haven't broken anything else with your new code?

This is why you should start writing automated tests for your code. Investing the time into learning about unit, integration, and end-to-end tests will help you to feel confident that your new code works and that it hasn't broken any other code.

You can run your tests on your local computer but it's also great to add in hooks to run tests automatically when you attempt to commit or push code. I use git for my version control and I add git hooks so that the tests have to run and complete successfully before I can push any code to a remote repository like GitHub.

If you're writing JavaScript code, and are new to testing, then download my FREE course on JavaScript testing.

Pair programming and code reviews

I place a lot of value in having more than one person looking at a code change. Pair programming is the practice of two people working together on code. This is conventionally done by two people sitting next to each other but in these times of remote working, it's just as easy to do this remotely with something like Visual Studio Live Share. Working together on a problem can help to spot errors early and helps to create consistent patterns and strategies for the code.

Code reviews

Once the code is complete, I use a branch-based workflow in GitHub. With this method I create pull requests to allow feature branch code reviews. This discussion around the code again gives the team another opportunity to spot any errors.

Before the code can be merged to a main branch, we can also run our tests on our continuous integration server. This allows us to check that all of our tests pass in a completely new environment and not just on a developer's local machine. Try and avoid "it works on my machine" syndrome in your teams.

Small iterations and often

A few years ago, it was commonplace to spend a month or two adding features and fixing bugs for an application and then have a big bang release of thousands of lines of code. This isn't the way to build code confidence!

Breaking a problem into small iterations that you can release quickly and often is the way to combat this. It helps to speed up delivery and gives the team confidence in your code deployments. If there is a problem once you've released to production, it's such a small change that you can more easily roll it back or understand how to fix it.

Feature flags

One of the best techniques we implemented to build confidence in our code has been feature flags. These are variables which we can use to determine whether a logical path should be followed or not. They are often booleans but can just as easily have multiple values. These flags allows us to release code quickly and often but not yet enable a feature until we decide that the time is right.

There are many ways to implement feature flags but in our current system, we have a user interface which allows us to change the value for the feature flag and this gets propagated to any running instances of our applications. This allows a feature flag's value to be altered and it's an almost instant change for the end users. They are also great if any issues are seen in the production environment because the new feature can almost immediately be turned off too.

Note that it does incur some technical debt when you add feature flags. The code can end up littered with feature flag logic and branching paths in both the code and tests. Make sure you remove the feature flag code as soon as you're happy that the new code works as expected.

Development environments

In order to ensure you can perform some testing prior to the production environment, it's a good idea to consider multiple development environments. This can vary based upon your own needs. We use multiple environments for testing our integration with other products prior to the code going live.

Depending on the scale of your application, you might want to consider performance testing on one of your environments. This tends to be a staging environment where you attempt to make it as close to the production environment as possible to mimic real-world conditions.

Live testing

Finally, once your code has been deployed to the production environment, you can test it here. These can be smoke tests or regression tests to ensure that no features have been broken. You can also use any monitoring you have to ensure that alarms trigger if performance degrades or tests fail on the live environment.

I hope this has given you some ideas for giving you more confidence in your development processes. Let me know on Twitter or via the contact form if you have any others.

Create animated gifs using Camtasia

2021-01-31T00:00:00Z

I love gifs!

I used to create them in the 90s when I was learning how to build my early websites. It was probably mostly the animated "under construction" gif. They went out of fashion in the late 90s due to better file formats and a patent on the gif format. When Slack was launched in 2013, and other messaging apps became commonplace, they made a reappearance as an animated response to messages. I think I probably send at least 10 gifs a day but I'd only made a few of my own. As I use Camtasia for my own videos on YouTube, let's look at how we can make our own using that.

Oh and of course, I pronounce it "gif" with a hard G like "gift". That's the correct way right? 😉

Getting started

To start off you need to record some video footage. This can be a video of yourself, your family, or some other fun situation.

You can record this easily using your mobile phone and then upload it to your computer. If you have a webcam, you can also record it straight into Camtasia instead.

A basic Camtasia gif project

The main thing to remember with a gif is that it needs to be small. You're not looking for ultra-HD footage here. If you're planning on uploading the gif to a site like Giphy or Tenor then, although they support 720p images, its best to stick to a 480p resolution of 852 x 480 pixels.

Set up the project in Camtasia by clicking on Edit > Project Settings and set the resolution to a width of 852 and height of 480 pixels. Leave the frame rate as 30 fps.

Drag the footage that you've created already into the Camtasia Media Bin. If you're recording footage in Camtasia, ensure your webcam is enabled and record your footage directly into the application instead.

Next, drag the video you've added or created onto the timeline. You'll probably have recorded more footage than you need so use Camtasia's trimming tools to resize it to only the footage you need. Ideally this will be under 6 seconds long. For my gif, this was around 2 seconds long.

A gif needs to be short and sweet.

Add text

If you want to add some text to your gif, you can do this by using Camtasia's annotations tools in the left menu bar. Choose the annotation type that you'd like and drag it to the timeline. Make sure that you place it above the video on the timeline so that you can see it overlaid on top of the video footage.

If you want to use the classic meme font, it's Impact.

Move the text to the correct place for your video footage as shown in the screenshot below.

For my footage, I added a behaviour to the text so that it grows into the screen. Drag the behaviour of your choice on top of the text to apply it.

Depending on the video footage colour, you might need to add an additional background shape behind the text. If you do, make sure it sits underneath the text but above the video in the Camtasia timeline.

You can add any additional transitions to the text that you want such as fade in or out.

Exporting

Now you've created your masterpiece, it's time to export it. If you're planning on uploading the gif to Giphy or Tenor, you can export the timeline as an MP4.

Choose Share > Local File from the menu and export it to the folder of your choice as an MP4.

Alternatively, if you want the gif file, choose the same Share > Local File option from the menu but change the output format to animated gif and hit export.

If you're exporting it as a gif file, it often has quite a large file size because we can't go lower than 30 frame per second for a Camtasia project. You can optimise the gif file using a tool like EZGif. Experiment with the different optimisation formats. I used the lossy gif encoder at 35% which had a good reduction in file size without making the gif look too bad.

Upload to Giphy or Tenor

Now you can upload it to the gif platform of your choice. I use Giphy a lot so I've created an account and channel there. Upload your MP4 or gif file to your channel. You can then add tags so that it's searchable via their search engine. This will make it appear in apps like Slack too. Make sure you add relevant tags to the gif. In Giphy you can add a maximum of 20 tags so make sure they count.

Animated gif best practices

Gifs are not high quality videos so think about the following for optimising them for the gif platforms.

Try and make your gif short. Under 6 seconds is best.
Make your source video resolution should be 720p max, but we recommend you keep it at 480p.
Most gifs appear on a phone or in messaging apps like Slack or WhatsApp. You don't need huge files at high resolution.
Try and keep the number of frames in the animation low. Try to keep it at 200 frames or less. At 30 frames per second, that's just over 6.5 seconds in length.

Enjoy and keep on creating great gifs!

Rubber duck debugging

2021-01-30T00:00:00Z

I wrote video games for about 20 years and for a long time I made games for the Sony PlayStation consoles.

Sony were famous for using rubber ducks in their console announcement demos. Here's some old footage of the PlayStation 3 announcement from the wonderful Phil Harrison:

https://www.youtube.com/watch?v=7UaDvIPpQnQ

At Sony's Studio Liverpool, we had hundreds of rubber ducks in the offices. They became part of our debugging process.

Yes, I used to talk to them when I had coding problems. 🦆

What is rubber duck debugging?

Writing computer software is a very precise skill. You have to be explicit in your code and clear in the logic to get the results you want. Humans are often not so good at this. Requirements can seem ambigiously or perhaps we don't have enough details before we start coding a solution. It's easy to become blind to errors that we've created. Our code can be incorrect but it can be difficult to understand why.

The idea of rubber duck debugging is that when you encounter a problem in your code, you talk through your code to a rubber duck. It doesn't have to be a duck. It can be another inanimate object, your children, another member of your team, your partner, or especially in these times of working from home, it could be your cat, dog, or hamster. 🐹

So what do you need to do?

1. Read through your code out loud

Explain your code at a high level to your duck friend. Tell it what the code is meant to do if it was working. This helps you to set the understanding for the problem at hand.

2. Explain each line and functions

Next you should try and explain each line on its own. Describe the flow of your function and what each line does. Say how it changes any variable values or state within the function. Don't skip any of the details. Ducks like details.

3. Discuss any state or global variables

If your code isn't purely functional, you may rely on state or global variables from outside of this code. Describe what this is, where it is set, and what its values should be.

4. Find your problem

At this point you should have your "DOH!" moment. Talking about the code out loud will help you to realise the assumptions you've made which might not be true. Maybe you'll spot the typo where you've misnamed a variable. Or perhaps you'll see that your logic is incorrect.

Your duck will have helped you to find your problem! Aren't ducks great? 🐤

Why does rubber duck debugging work?

You'll be amazed that this type of debugging actually works. I've often explained a problem to one of my team and as I explain each line, I end up discovering the issue. While writing new code, you can easily enter a state of flow, also known as "being in the zone". In this state, your brain is thinking quickly and your hands are busy typing. You think you understand what you need to do but you might miss the details.

Once you start explaining a problem to your rubber duck, your brain slows down to read and understand what's going on. You think a lot faster than you talk. Talking out loud helps you to focus on what's gone wrong.

Try out rubber duck debugging when you next have a problem. You'll find that you quickly gain clarity on what your code does rather than what you think it does. And don't forget to bring some bread to feed your duck. 🦆

Lift your team up

2021-01-27T00:00:00Z

I've always loved the following saying:

“A rising tide lifts all boats”

Although John F Kennedy may have borrowed it from the chamber of commerce of the New England Council, the idea behind it really resonates with me. The true meaning behind it was to suggest that an improved economy will benefit all participants. However, I like to apply it to how you build a positive team culture. By helping one person, you quickly help to lift the whole team up.

As a manager of a software engineering team, I help to set the tone for team. It's my responsibility to ensure the team are happy, that they're constantly learning, and that they can grow their careers. By building an environment where everyone helps to bring their team mates up, we end up with a positive and inclusive culture.

So how do you help to lift your whole team up?

Be encouraging

Believe in your team. You hired them for a reason. Don't let them join your team and then leave them to get on with the work in isolation. Give regular feedback. Help them to thrive. When they're doing a great job then tell them. Share their successes with others. We have a kudos Slack channel where I love posting my team's small and big wins. Encouragement and positivity is infectious.

Be understanding

We're all human. Life is never easy and we all face challenges in our own lives, especially in 2020 and 2021. Have regular 1:1 meetings with your team and be an active listener. Accept what they're willing to share with you and share your own life, as much as you both feel comfortable with. It'll help you both to understand each other and build a positive relationship. Feeling understood will help them to feel accepted as part of the team.

Let everyone feel heard

To make everyone feel included they must feel heard. Ensure you enable every to voice their opinion in your meetings. Don't only allow those with the most confidence or loudest voices to speak. Lift everyone's opinions so that they feel part of the team, even if you can't always act upon everyone's views. As Simon Sinek quite nicely puts it, "be the last to speak".

Multiply the makers

As a team lead, your job is to multiply the makers on your team. As a technical leader, your job isn't to always know the right answer. It's to find the answer by allowing your team to be the experts that you know they are. See the potential in your team and help to amplify their voices. You'll have much more impact as a leader and it'll bring your whole team along on the journey with you.

Build a learning environment

I love learning. It's something I actively try and encourage my children to understand. Every day should be a day of discovery in finding something new to learn about. Create learning opportunities for your team. I run a weekly team learning session where I mentor some of my less experienced team members to increase their knowledge and understanding of both programming and our team's domain expertise. I share my own knowledge and help to bring them up too. It's not a zero-sum game. By sharing what I know, I'm helping the team and in turn, they'll teach someone else. I've learnt so much more by teaching my team and it's helped to lift me up too.

You can lift the whole team up by sharing your network, especially if you've been at a company for a while. You'll have gained a reputation, and hopefully a good one. Use this to sponsor members of your team. Introduce them to other people they might not normally meet and create those opportunities for career growth. Lend your privilege to others who may not have it yet. A short introduction to another team might not mean much to you, but it'll do wonders for the confidence of your engineer who is being raised up to represent your team.

Inspire the team

Inspiration is a funny term and I often struggle with it and the idea of being inspirational. You should strive to inspire your team to do their best work by aspiring to do some or all of the things I've talked about above. You can share the history of why your team has created the application architecture that you have. It can inspire them that they can help to build similar too. You can also set the vision of the future and be an inspiration for what could be. By listening to your team, you can help them to build the team's vision for the future too.

Stay humble

You might not realise how much you help the team by lifting them up. Your accomplishments will mean a lot to the team but remember to be happy to do so without a fuss. Sometimes you might not even realise you're doing a great job until somebody tells you so.

Build that positive environment and keep on helping to lift the team's boats.

The rewards will also come to you too.

It's not too late to start

2021-01-26T00:00:00Z

I once found a wonderful comic by Zach Weinersmith of Saturday Morning Breakfast Cereal (SMBC). I've told a lot of people about it so please go and read it now.

It states that most people think that they only live once and spend their life thinking they can only have one career. But if it takes around 7 years to master a skill, you've actually got 11 opportunities to do something from age 11 to 88.

It's not too late to start.

I spent my early career as a programmer writing video games. That's the part of the technology industry I knew best for nearly 20 years. I never thought I'd need to think about any other technology. And then I got made redundant in 2010.

After a couple of years of contracting in the games industry, I started developing web-based technologies like API services. It was something I knew very little about, but I kept learning. I showed up every day and learnt a little bit more than I had the day before. This new knowledge helped me to get a job at the BBC.

I was using programming languages I'd never touched before, on platforms I knew nothing about. I watched graduate programmers talk about things I knew nothing about and had never previously needed to. But I carried on learning and while my job was still in technology, I started a new career in web services and cloud technologies.

I had reinvented myself. I'd started again.

I see a lot of people online asking if it's too late to become a software developer. They're often in another career, or think they're too old. Just remember:

It's not too late to start.

The BBC ran a fantastic programme of work called Step Into Tech in 2018 and 2019. It encouraged women to sign up for a 12-week training course to have the opportunity to learn some fundamental skills to kick-start their career in the technology sector. All of the women were of differing ages and most had no background in software engineering. It was great to see them move from their varied careers such as working for the NHS, working in music production, and teaching, to reinvent themselves as software developers. They're all doing wonderfully well now and have yet again proved that:

It's not too late to start.

So what do you need to do to start?

Turn up. Start doing. And be consistent.

Am I a good writer? Probably not yet. But 12 days ago I decided to turn up and start writing every day. By showing up every day, I've written about 14,000 words. Not all of them are great I'm sure. But I started.

I appreciate that this advice is easy to give but hard to follow. Consistency is hard inbetween your work and family life. If you can set some time aside to work on your goals, then:

It's not too late to start.

And finally, I love this video from Sean Wes. It's called "Show up every day for 2 years". Follow his advice and you can't lose.

https://www.youtube.com/watch?v=pUhnIWUTTts

Learning to learn about software development

2021-01-25T00:00:00Z

One of my team asked me a fantastic question today.

“How do I stay on top of learning about software development technologies and best practices?”

There's an information overload in the software development world. Every day there's a new framework, a new library, a new technology stack, or a new programming language. It feels like you have to learn everything, especially if you're not an experienced developer yet.

But you don't!

So how do you manage to learn new skills when there's a massive influx of articles and videos competing for your attention?

My day-to-day role is as a software engineering team lead. This means I'm not as hands-on with development work as I used to be. I have to pick and choose what I have time to learn and, with the exception of our team's 10% time (similar to Google's 20% time), most of my learning is done after work in the evenings or at weekends.

Knowing what to learn

I'm somewhat addicted to Twitter, although I try my best not to be. This is my first port of call when it comes to ideas for new technologies to investigate. You can follow some interesting developers and technologists on Twitter but it's sometimes difficult to keep up with your Twitter timeline if you follow too many people. Alternatively, you can create and follow lists of people, or follow lists that others have made. This is a great way to expose yourself to new people and their ideas. Don't forget to keep your ideas diverse by following more than just people who look and think like you or who use the same technologies.

Another great source of new technologies are technology aggregators such as Hacker News (don't read the comments!), Dev.to, Reddit, and Hashnode.

I also watch a lot of YouTube videos. YouTube is a fantastic search engine but by also following the right people, its algorithm will also suggest other technology videos you might be interested in. I enjoy watching Fireship, Adam Wathan, Firebase, Google Chrome Developers and many more.

You'll find plenty to read on Twitter, YouTube, and these sites and it'll help you to discover what you're interested in learning.

Scratch your own itch

I always like to investigate new ideas and technologies by "scratching my own itch". This means that I have an idea for something which solves a current problem I have and it allows me to learn whilst I'm implementing a solution.

For example, my wife was watching a series of online webinars that were only available for the next 5 days. She had no time to watch them so wondered if I could download the videos so she could watch them later. The videos were hosted by Wistia so I investigated how I could find the source video files by digging through the websites source code and discovering how the video was embedded. As I needed to download multiple videos, I wondered if I could automate the process through a command line tool. This took me off on a voyage of discovery as I scraped a webpage, used regular expressions to parse script tags to pull out JSON data, and worked out which hidden binary file URL to download. This led me to Node.js streams, which I'd not used before with any force, to stream the video file and write it to disk. All of this from a simple question from my wife!

As you can see, working a project that you're interested in can open up a world of learning opportunities.

Open source GitHub repositories

Once you've discovered what you want to learn it's a good idea to dig into other people's code. When I'm thinking about functionality I need for a project, I have to decide whether it's worth writing the code myself or exploring an open source library which might do what I need.

If it's an open source library I'm using, I quickly jump to their GitHub, GitLab, or BitBucket repository. A well-written README file can teach you a lot about the architecture and design of their library. Once I've got an idea of how it works, I'll attempt to use the library in my project to understand their public API. If I struggle to get something working, now is a good time to jump into the code. The great thing about open source projects is that their code is available for you to read. Take a look at the code modules and work out the code path for the action that you're attempting. You'll fix your own bug and learn how their code is designed.

As I've mentioned previously, I take a lot of notes using Obsidian. I use this to build up a list of technologies, languages or frameworks I'd like to learn as I discover them. Every few weeks I revisit my notes to see what I want to learn next. As I start using something new, I take notes as I'm learning and note down any problems I've encountered, other libraries that they use, or how they're structured. This helps me to make my own technical decisions for my code and allows me to expand my area of knowledge in new directions.

Once you've done that, try and write up your notes so that you can learn in public. You might not know all of the details yet but you're one step ahead of someone who's not yet discovered it. Why not write up some notes on Twitter or if you have a personal website, write a blog post about it.

Just-in-time learning

If you're anything like me, you've probably now got hundreds of ideas of what you want to build. But you can't learn everything. You have to be ruthless with your time, especially if you have other commitments like a partner or family. I try and employ just-in-time learning. I don't go off and attempt to become an expert in a technology that I don't currently need. I learn it when I need it for my job or my side projects.

You can't learn everything so gain your knowledge at the right time. Before you know it, you've learnt an awful lot more than you thought.

Add YouTube playlists to your Eleventy site

2021-01-24T00:00:00Z

Over the past couple of years I've been moving my personal websites from using WordPress to simple static HTML sites. WordPress is fantastic but I spent too much time being concerned about security risks, updating to new versions, and worrying about managing PHP versions on a server. These are things I didn't want to have to concern myself with so I wanted to use a static site generator to build plain HTML, CSS, and a little bit of client-side JavaScript where necessary. I discovered Eleventy back in early 2020 and this has quickly become my tool of choice.

My chatbots page has a list of YouTube videos on it. Initially this was a static list of YouTube video IDs that I iterated over to generate the markup. I realised that using Eleventy's great data flow, I could automate this using the YouTube API. This means that I only need to regenerate the site when I add new videos to the playlist and I don't have to cut and paste a YouTube video ID into a markdown file each time. This makes things much simpler.

Let's look at how to do this using Eleventy's data files.

Obtaining YouTube API credentials

I'm assuming that you have a Google account. Without this you can't access the developer console to register an application or generate a new API key for the YouTube API.

Sign into your account and head to the Google Developers Console. Create a new project for your website, or use an existing project if you have one.

You must obtain authorisation credentials in order to generate an API key. Open the credentials page in the Google console and click "Create credentials" to create a new API key.

After creating the key you must enable the YouTube Data API (v3) for this API key. Go to the developer console here and enable it. Ensure that you save your API key somewhere safe so we can use it to access the YouTube API later.

The playlist we will attempt to read is public and so we can use an API key to retrieve this data. If you require a user's private data, such as a private YouTube playlist, then you need to use an OAuth 2.0 token with your request.

Add your API key locally

API keys should always be kept safe. If you expose it publicly then others can use it to make requests against your Google account. To avoid this, we're going to store the API key as an environment variable. We can then use this in our YouTube playlist fetcher without exposing the key to anyone.

In Node.js, a great way to expose environment variables to your application is using the dotenv npm package. This allows us to create a .env file which contains all of our secret keys.

Install dotenv as a development dependency for your Eleventy project:

npm install -D dotenv

You should attempt to require dotenv as early as you can in your JavaScript project. For Eleventy, this means you can add to the top of your .eleventy.js coniguration file.

require('dotenv').config();

// The rest of your Eleventy configuration goes below
module.exports = (eleventyConfig) => {
    // Your Eleventy plugins and so on
};

Create a .env file in the root of your project and add in your YouTube API key.

# Add your secret API key
YOUTUBE_API_KEY=my-secret-youtube-api-key

When your Eleventy build is executed, you'll have access to the API key using the Node process environment variables like this:

const myApiKey = process.env.YOUTUBE_API_KEY;

We can now use this exposed API key inside our data files to read a YouTube playlist.

Make sure you add the .env file to your .gitignore file so that you don't accidentally add secret keys to your git repository.

Eleventy data files

One of the super powers of Eleventy is that it allows you to retrieve data in multiple ways. This can be data in your markdown frontmatter, data in a layout directory, template data, data directories, and more.

I often use static global data as JSON files in my _data directory but you can also use JavaScript files to generate it programatically. Using this method, we export a single function which gets called by Eleventy as it builds are site. One of the great features of using this method is that we can use asynchronous functions to retrieve data from external sources. Let's look at how we can use this to load a public playlist using the YouTube API.

Install the Google APIs Node.js package as a development dependency. We'll use this to call the YouTube API without having to write our own HTTP requests and authentication.

npm install -D googleapis

Create a file in your _data directory called youTubeVideos.js. The name of the file will match the name of the global object available to your Eleventy build.

We'll require the Google APIs package and create a YouTube client to use to retrieve the data we want from the YouTube API. It can be used for any public API call and not just for playlists.

// Require the Google API package
const {google} = require('googleapis');

// Create a YouTube service
const youTubeClient = google.youtube({
    // Use the v3 API
    version: 'v3',

    // Don't hardcode your API key
    // Read it from an environment variable
    auth: process.env.YOUTUBE_API_KEY
});

Now that we have the YouTube client, let's create a function which retrieves the data we want from our playlist. We can then expose it to Eleventy using module.exports. We will use the YouTube client we created earlier to calll the YouTube playlistItems endpoint and list all of the videos for a specified playlist. The API expects you to tell it what fields you want to retrieve. For this example we'll retrieve the id and the snippets (title, description, position, resourceId).

// Note that we're exposing an asynchronous function
// Eleventy will wait for this to finish before exposing the data
// to the rest of our code
module.exports = async () => {
    try {
        // Call the YouTube API using the client we created earlier
        const response = await youTubeClient.playlistItems.list({
            // Pass your own public playlist ID here
            playlistId: 'your-public-playlist-id',

            // Tell it what data you want to retrieve
            part: 'id,snippet',

            // Maximum results you require from 0 to 50
            maxResults: 50
        });

        // Return an array of video items
        return response.data.items;
    }
    catch(error) {
        // Log any errors so we can see them
        console.log(error.toString());

        // Allow the build to work but without video data
        return [];
    }
};

If the API call is successful then we'll be given the playlist data in response.data. This will contain an array of items which we'll return to Eleventy. Notice that I've added error checking to the code. If there's an error then we can log it out so we'll see it on our command line. I also return an empty array so that our Eleventy build won't just stop due to an error referencing null data.

In the example, I'm only attempting to read the first 50 videos in the playlist. This is the maximum you can retrieve in 1 call. The response will return you a nextPageToken if more results are available. In that case, you can call the API again and pass a page: nextPageToken key-value pair in the options passed to the API. Keep repeating this process to retrieve all of the results until you no longer receive a nextPageToken which means you've received all results.

Note that the YouTube API has a quota of 10,000 units per day and each operation has a different cost associated with it, depending on what operation you're doing it. For most uses, this is more than enough but read more here in case you think you might have a high volume use case.

Using the data in your template

Now that we've retrieved the data, how do we use it?

As we named the data file youTubeVideos.js, we now have a global variable called youTubeVideos which matches the name of the data file. This variable is exposed to our templates so we can iterate over the data in it to render each YouTube video link from the playlist.

Here's an example using a Liquid template.

{% for video in youTubeVideos %}
<article>
    <a href="https://www.youtube.com/watch?v={{video.snippet.resourceId.videoId}}" target="_blank" rel="noreferrer">
        <img src="{{video.snippet.thumbnails.high.url}}" alt="{{video.snippet.title}}" />
        <dl>
            <dt>Title</dt>
            <dd>{{video.snippet.title}}</dd>
        </dl>
    </a>
</article>
{% endfor %}

Take a look at the chatbots page to see it in action.

Deployment to your environment

Now that we've generated the YouTube playlist as a data file and used it in our local build, we mustn't forget to add the YOUTUBE_API_KEY environment variable to our production environment. I use Netlify for my production builds and you add this easily using their site settings. For other deployment platforms, such as Vercel or GitHub pages you can add enviroment variables in their settings too. Just ensure you've added the key prior to the build process or you won't see your videos.

Hopefully that's given you a good idea of how to use the YouTube API with Eleventy. To speed up your development build, and avoid calling the API repeatedly, you could look to add a cache to save the playlist data locally. This isn't a necessity but it will speed up the build.

Build simple software

2021-01-21T00:00:00Z

Nobody sets to build complex software.

All developers set out with best laid plans to keep applications simple. We separate the concerns of the code with nice APIs and interfaces. We abstract our business logic away from our domain services. It's as simple as we can make it but over time the code gets more complex. But why?

There are times when developers over-engineer a problem. We add Gang of four design patterns to the code because we care about separations of concern but often these layers of factories on top of facades on top of proxies make our code much more difficult to read. Code complexity is not always driven by the software engineers working on the code. There are plenty of reasons why it's not only a technical issue.

Sofware isn't useful until it provides business value and is in front of users. There are often time constraints and deadlines so new features are added to the application in a short time frame. If external teams integrate with your services, there are often requirements from yet more stakeholders. Software is written with the best knowledge you have at the time of writing it. Software architecture changes over time as new requirements appear. The requirements of the application are based on the needs of a product team, the user experience (UX) team, and the developers working on it. Over time the engineering team may change and steer the architecture in different directions. As a software developer you'll always be learning and discover new patterns which make more sense. Ultimately, all software will rot and incur technical debt. There's no getting around this so we have to put plans in place to make it easier.

So what can we as software developers do?

How do we reduce software complexity?

Ross, the principal engineer on my team at the BBC, loves this quote from well known software engineer Kent Beck and I think this is applicable in software design.

https://twitter.com/KentBeck/status/250733358307500032

In this instance, Kent was referring to refactoring your code but it also applies to the design of your software. You simplify the design of your code such that you can lay simpler foundations for future requirements.

Simplicity in code is not necessarily easy to achieve but there are some ways we can help to point us in the right direction.

Make it human readable

It's easy to write more code than you need to and overcomplicate your software. Try to eliminate unnecessary code where possible but not to the point of losing readibility. As developers we often spend more time reading code than writing it, especially if it's not a new project. Optimise for making this easy.

"There are only two hard things in Computer Science: cache invalidation and naming things." - Phil Karlton

People underestimate how important it is to clearly name your variables and functions. If you can read the code like you're reading it set of instructions, it means you've done a good job. If there's a potential to cause confusion in what the code does, consider simplifying the names. If you agree on naming conventions with your team, this makes life simpler.

Easily testable

If you follow the concept of test-driven development (TDD) then you will be writing your tests before your code. This naturally leads you to only writing the code you need to pass the tests. Even if you're not following strict TDD and writing your tests afterwards, if you find that the code is difficult to test, then you've perhaps you need to simplify the code to make this easier.

Separation of concerns

You should look at removing complexity by separating your code modules into their areas of concern. For example, if you're writing an API service, move your business logic away from your HTTP transport layer. You don't want to mix database calls and HTTP JSON responses together. Think about how you can isolate your code into only the areas that it should be concerned about. It simplifies the application and will also make testing easier.

Don't repeat yourself - but within reason

The "don't repeat yourself" (DRY) software principal is the idea that looks to reduce the repetition of code. If you start to repeat the same block of code multiple times, then look to abstract it out into its own function or class. However, be wary or premature optimisation. You might find that the two pieces of code actually have different business reasons for existing. You don't want to refactor into a single function which ends up with branching business logic inside it. My rule of thumb for DRY is to see if the code is used a 3rd time. Once that's the case, I'll usually refactor it to a shared function.

Reduce your dependencies

It's really easy to search the JavaScript npm registry, or the equivalent registry for other programming languages, to add some open-source code to your project. Be mindful of additional dependencies you're adding to your software by doing so. While some of the packages might be useful, it adds a layer of complexity in code that you and your team haven't written. Try not to import packages just because you can and only use them when you need to.

Avoid developing for edge cases

During tight deadlines, it's often easy to fix issues by adding special cases into your functions. Code which branches based upon state passed into the function, or even worse, based on global state. It's a slippery slope if you start adding these edge cases to your code. It's much harder to determine what a function does once there are more branches of logic to follow. Attempt to keep your code functional and avoid the need for global state.

As an example, about 20 years ago, I was involved in writing a AAA videogame. Right before mastering the finished DVD, we discovered issues with the final boss battle that I had developed the AI for. The main character had a move which was too powerful and would kill the final foe with one hit. This hadn't been thought of from the start so I was asked to add a line of code which removed the special move for this level only. So I had to add an additional if statement to check the global level state and remove the special move for this level only. This was in the main characters input code. 🤦🏻‍♂️

Don't do this!

Conclusion

There's no silver bullet for writing simple code. You have to think about your architecture before hitting the keyboard. Make time to build a good foundation for your application architecture. Allow the team to think about good coding practices and don't try and cut corners. Make sure you build in contingency for refactoring tech debt as the requirements change.

And remember to Keep It Simple Stupid!

Useful resources

A great naming cheatsheet on how to get better at naming variables and functions in your code

Stop overthinking everything

2021-01-20T00:00:00Z

Some days I can't shut my brain off.

I often worry about something that happened at work or home. Sometimes I remember a situation that happened in the mid-90s when I was DJing in a nightclub. Yes, this regularly happens and I'm always surprised I can remember that far back! Today I've been worrying whether the last 7 articles I've written for my writing habit challenge are good enough or not. I can't stop thinking about how I could have approached a situation differently and changed the outcome.

It can be mentally exhausting.

I'm sure I'm no different to many people in being an overthinker but I wonder how you can challenge your brain to organise the good thoughts and solve problems, rather than being bogged down with things you can't affect.

Overthinking can take many forms. It can be thinking too much about the past situations that you could have changed. It can be worrying about future events and how they will play out. For me it's often deliberating when attempting to make a decision on something, and then questioning the decision once I've made it. Just ask my wife Clare how long it takes me to buy anything! It can also build a running commentary in your head, putting words into others mouths in a fictional situation. Inevitably, you think you know how the conversation will end and in your own mind that's usually badly.

Overthinking isn't problem solving to find a solution. It's dwelling on the problem itself and not getting beyond that.

How can we deal with overthinking?

I'm trying to get better at scheduling my downtime to stop my brain from spinning. I used to spend too many hours in front of a laptop after 10 pm at night. Wherever possible, I put the computer and phone away to try and allow my thinking to slow down. I've also tried to read more in bed as it's a much nicer way to end the day and helps me to avoid thinking too much about what's happened.

Some research suggests, as strange as it sounds, that you schedule your worries into a set time each day. I've not tried it myself but sounds like an interesting idea.

A collegue at work ran a workshop which referenced Steven Covey's classic book "The 7 Habits of Highly Effective People". There's an idea of the circle of influence and control. I found it's a good way to rationalise what you can and can't control and where you can influence situations. There's little point in thinking about the your company's policies, unless you're on the board of directors, as it's difficult to change these situations.

Noticing that you're overthinking is a good thing. Once you realise that you're stuck on the same thoughts, you can move your mind to something else. Try and steer your thinking away from the bad and think about what great things you could do at the weekend instead.

Move from dwelling on a problem to suggesting potential actions to solve it instead. If it's something you can control, see if you can come up with one or more solutions which you can explore instead. If it's something that's not within your circle of influence, try and move it aside and focus on areas you can control.

Remember that overthinking is often your brain over-exaggerating towards the negative. Try and step back and remember a similar situation that's happened before. The chances are that it didn't turn out as bad as you thought. I know this is hard as I struggle with this a lot but it does help.

So yes, I'm trying to get better and stop overthinking.

To realise that things in the past can't change.

That things in the future haven't happened yet.

I'm trying to take more time out to reflect and meditate, especially with the current pandemic that we're all living through.

Stay safe and keep thinking of the positives instead of dwelling on the negatives.

Create a serverless chatbot for Telegram using Vercel - the ultimate guide

2021-01-19T00:00:00Z

I love helping people to automate their lives.

Back in 2018/19, I spent a lot of time building automated chatbots using the Facebook Messenger platform. It was a great way for businesses to automate interactions with their customers. More recently, Facebook have made it much harder to build bots on their platform by limiting their API, especially if you're a European user.

In the past year, I've moved over to using Telegram to build bots and often use them to integrate with other online services and APIs. They have their own fully-featured bot API which makes it much easier to interact with.

Telegram messages

Telegram supports two ways of interacting with the messages that users send to its bots. The first is long polling. This is similar to standard polling techniques where the client requests data from the server at regular intervals. With long polling, there's an expectation that the server may not reply instantly and then the client holds open the request and waits. This works if you are happy with deploying bot server code and infrastructure that never shuts down.

The alternative way to receive messages from Telegram is using webhooks. This method is event-driven and Telegram will send you a HTTP POST request only when a new message is sent from a user. This fits perfectly with the idea of event-driven architecture such as serverless functions.

Let's look at using Telegram's API and Vercel serverless functions to create a basic bot and deploy it to their cloud hosting.

Creating a new Telegram bot

The first task is to create a new bot that our user can interact with. To do this we must talk to the Botfather. This is Telegram's own bot which creates and updates new bots. Either start a new chat with Botfather or click on this URL to open the bot in Telegram: https://t.me/botfather

To create a new bot we must send it the /newbot command. It will ask you for a name and username of the bot. The name is displayed in the contact details. The username is a shortname which is used for mentions, @mybot for example, or in t.me links for sharing. The username has to end in bot. As bot usernames are unique, you can see from the screenshot that it took me a couple of attempts to get the bot name MarcDevBot that hasn't already been used by another developer.

Botfather will send you an authorisation token which is a long string. You use this to send requests to the Telegram Bot API.

Make sure you keep your Telegram authorisation token safe and don't share it with anyone. If you do, they can send messages to users on behalf of your bot by using it.

Vercel account

To deploy our chatbot code to Vercel, we must create a Vercel account. Head to the signup page and create a new account using your GitHub, GitLab, or BitBucket account. If you've already got one, then that's great!

Creating our bot serverless function

We're going to use Node.js and JavaScript and a couple of npm packages to create a serverless function. I'm going to assume that you have Node.js set up on your local development machine and understand the basics of adding packages to a project using a command terminal.

Let's get the project set up. Create a directory called test-bot and change directory into it:

mkdir test-bot
cd test-bot

Next we'll create a basic Node.js project using npm. Here we're skipping the part where we fill in the details of the project into the package.json file. I'll leave that for you to do later.

npm init -y

As we want to deploy our serverless function using Vercel, we must install their command line tools. We can add the vercel application as a global dependency, so that it's available everywhere and not just in our project, like this:

npm install -g vercel

Next we can login to vercel which allows it to know which account we're deploying the code to.

vercel login

Vercel will send you an email with a link to verify your account. Once you click on that link it will save the correct tokens to your machine to allow you to deploy your application.

Vercel serverless functions

Vercel can be used to deploy any frontend application to its cloud hosting but it also allows you to run serverless functions, using AWS Lambda and Google Cloud Functions under the hood. Making a new serverless function is easy. Vercel wants a directory called /api in the root of your project. Inside that directory you can add a JavaScript, Go, Ruby, or Python exported function and it will appear as an API route. Let's add a new function to handle our Telegram messages.

Create the /api directory in the root of your project.

mkdir api

In your editor of choice create a file called webhook.js in the /api directory.

module.exports = (request, response) => {
    response.json({
      body: request.body,
      query: request.query,
      cookies: request.cookies,
    });
};

This will expose an API function on the following path /api/webhook. For basic usage, the name of the file represents the last part of our path (i.e. webhook.js becomes /webhook). It always sits under the /api path unless you change it through configuration. We can test that the code works correctly in our browser by running a development version of Vercel using the following command:

vercel dev

As this is the first time you've run the development build, Vercel will ask you a few questions as shown below.

At the end of the process it will expose an HTTP endpoint on port 3000 by default. Hit this URL in your browser and it will make a GET request to your newly exposed serverless function.

http://localhost:3000/api/webhook?hello=world

You should see a similar response to the following:

{
  "query": {
    "hello": "world"
  },
  "cookies": {
    "_ga": "GA1.1.10178417.1582143681",
    "__smToken": "2PgmPwvNz7Ss6Os5nQexTPAl",
    "_gid": "GA1.1.327749069.1610568137",
    "io": "WRSUGikIUBDcMAQ_AAAZ"
  }
}

Yay! Our serverless function is working locally and is responding with a JSON response. 🎉

Creating the Telegram bot message handler

Telegram expects to call a webhook by sending us a POST request when a message is entered by the user. Let's build the message handler to receive this. We can install an npm package called node-telegram-bot-api which will allow us to easily receive and send messages to a Telegram bot.

npm install node-telegram-bot-api

Let's update our webhook API to receive the messages that Telegram sends us. A Telegram message contains a JSON body with a message property in it. There are other messages that Telegram can send us but this function will only handle text messages for now.

In order to send messages using the Bot API, we need to use the authorisation token that the Botfather sent us earlier. We don't want to hardcode this as a variable in our code as this would be a security risk. Instead we'll set it as an environment variable that we can use.

Copy the following code over the webhook.js code you wrote earlier.

// https://github.com/yagop/node-telegram-bot-api/issues/319#issuecomment-324963294
// Fixes an error with Promise cancellation
process.env.NTBA_FIX_319 = 'test';

// Require our Telegram helper package
const TelegramBot = require('node-telegram-bot-api');

// Export as an asynchronous function
// We'll wait until we've responded to the user
module.exports = async (request, response) => {
    try {
        // Create our new bot handler with the token
        // that the Botfather gave us
        // Use an environment variable so we don't expose it in our code
        const bot = new TelegramBot(process.env.TELEGRAM_TOKEN);

        // Retrieve the POST request body that gets sent from Telegram
        const { body } = request;

        // Ensure that this is a message being sent
        if (body.message) {
            // Retrieve the ID for this chat
            // and the text that the user sent
            const { chat: { id }, text } = body.message;

            // Create a message to send back
            // We can use Markdown inside this
            const message = `✅ Thanks for your message: *"${text}"*\nHave a great day! 👋🏻`;

            // Send our new message back in Markdown and
            // wait for the request to finish
            await bot.sendMessage(id, message, {parse_mode: 'Markdown'});
        }
    }
    catch(error) {
        // If there was an error sending our message then we 
        // can log it into the Vercel console
        console.error('Error sending message');
        console.log(error.toString());
    }
    
    // Acknowledge the message with Telegram
    // by sending a 200 HTTP status code
    // The message here doesn't matter.
    response.send('OK');
};

Testing our function locally

Telegram needs to send us a message to a public URL which has SSL/TLS encryption. As vercel dev only exposes an HTTP and not an HTTPS endpoint, we need to expose an HTTPS endpoint to the public internet so that the Telegram server can send us messages. We can do this using a great library called ngrok. This allows us to expose a secure connection to our local server.

Let's install ngrok as a development dependency for our project:

npm install --save-dev ngrok

To expose a web sever on port 3000 to the internet we can now run this command:

ngrok http 3000

ngrok will now be up and running and will give us two URLs: one for HTTP and another for HTTPS. You need the HTTPS URL to set as the webhook URL for Telegram to communicate with. Note that ngrok will generate a unique URL each time you run it. If you restart it then you'll also have to update the Telegram webhook URL.

Setting up the Telegram webhook

Using the ngrok HTTPS URL you were given, we can make a POST HTTP request to the Telegram API using curl to tell it to send messages to our local serverless function. Use the following command to do this. Make sure your replace <YOUR-BOT-TOKEN> (including the <> brackets) and the https://your-ngrok-subdomain.ngrok.io URL to the ones you've been given:

curl -X POST https://api.telegram.org/bot<YOUR-BOT-TOKEN>/setWebhook -H "Content-type: application/json" -d '{"url": "https://your-ngrok-subdomain.ngrok.io/api/webhook"}'

If this succeeds then the API will send you the following JSON response:

{"ok":true,"result":true,"description":"Webhook was set"}

Testing with the Telegram API

Let's run the vercel dev local mode whilst ngrok is running. This will start the vercel development environment and allow the function to receive requests on port 3000. In order to communicate with Telegram's API, we have to set our authorisation token as an environment variable. Replace <YOUR-BOT-TOKEN> with the one the Botfather gave you and run the following command:

TELEGRAM_TOKEN=<YOUR-BOT-TOKEN> vercel dev

Now let's chat to our bot in Telegram. Start a conversation with the bot you created earlier. Send it any message and you should get a response back.

The bot is up and running and responding to messages! 🎉

Deploying to Vercel

Now that your bot is working as expected locally, let's deploy it to Vercel. Deploying with Vercel is amazingly simple and that's why I love it. Type the following in the root of your project:

vercel

Vercel will build your project and deploy it to its servers for hosting as a serverless function. It will deploy it to the project name that you created earlier when you ran vercel login. For me this was test-bot.

Before we can switch over to using our production Telegram bot, we need to set up an environment variable for the Botfather authorisation token. There are a few ways to do this but let's set a text variable using the Vercel console for simplicity. Head to the settings area of your project. The URL will be something like this:

https://vercel.com/<your-vercel-username>/<your-bot-name>/settings/environment-variables

Add a new plaintext variable called TELEGRAM_TOKEN which is what we expect to find in our code. Set the value to the authorisation token that the Botfather gave you earlier and hit save.

Once this environment variable has been added, we'll need to redeploy the application to ensure the serverless function reads it when it starts up. Vercel makes it easy again so run the following command:

vercel

And finally...

Now that we've got our serverless Telegram bot deployed to Vercel, we need to update Telegram to tell it to route the messages to our Vercel function. Set the Telegram webhook using a curl command. Again, don't forget to replace <YOUR-BOT-TOKEN> with the Telegram authorisation token and the project-name.username.vercel.app domain name with your own URL which Vercel gave you when you deployed the application.

curl -X POST https://api.telegram.org/bot<YOUR-BOT-TOKEN>/setWebhook -H "Content-type: application/json" -d '{"url": "https://project-name.username.vercel.app/api/webhook"}'

And that's it. You've now got a serverless Telegram bot deployed to Vercel which can respond to user messages. Try typing another message in Telegram to your bot and check that it responds correctly.

What's next?

Now that you know how to deploy a Telegram bot using Vercel, it's time to update the code. Telegram has lots of fantastic features. You can send it commands and create a suite of automatic functions. You can use your bot to call out to external APIs to retrieve data for you. Have a think and see what amazing bots you can create!

Give me a shout or reach out on Twitter if you have any questions or need any help!

Useful resources

I've uploaded the code to a GitHub repository here: https://github.com/MarcL/telegram-test-bot Feel free to fork and reuse it how you wish.

Read the Telegram API for some in-depth knowledge of what bots can do.

Take a look at the Vercel documentation to see what other things you can deploy to their hosting.

Learn in public

2021-01-18T00:00:00Z

I worked in the video games industry for almost 20 years. I made quite a few games and worked on many more unreleased demos. Every company I worked at was incredibly secretive about their work. Very few teams shared what they were working on externally. Some didn't even share much between their internal teams either. We weren't really using open source software libraries. The encouragement was to create our own code and share with nobody.

I moved to the BBC in 2013 and it changed the way I thought about my work. We started using GitHub to share code. We created pull requests and reviewed and critiqued each other's code. We almost never had code reviews in my games development days so this was new to me! I started seeing the value in sharing things with others before it was 100% complete.

The whole team was learning Node.js at the time and most of us hadn't touched it before. Sharing my code with the team encouraged them to comment on it and we all started learning the best way to architecture our applications.

I realised that we were learning in public.

What is learning in public then?

I first read about the idea after following Shawn SWYX Wang on Twitter. It was the first time I'd heard the phrase "learn in public" and I was intrigued.

Most of us learn something new by consuming content. We read a book or an online article. We watch a YouTube video or a Netflix documentary. And the thinking ends there. We might use that knowledge in private but we don't often share it with others in a public forum.

Learning in public is the idea of creating a habit of re-teaching what you've learnt so that others can benefit from it. It allows you to clarify your understanding of the subject using the protégé effect. This is where teaching others also helps you, the teacher, to absorb that information more easily.

You can teach what you've learnt it in lots of different ways:

a blog post or tutorial on your website
a thread of tweets on Twitter
a YouTube video or Twitch live stream
creating a public example of code on GitHub
speaking at a local meetup
starting a newsletter

You might have learnt everyone on the subject yet but you try your best to be right with the details you know right now. However, also try not to worry if you get things wrong. As xkcd tells us, somebody will tell you when you're wrong on the internet:

When you haven't got all of the details correct, people may well tell you that your work is terrible. Accepting public criticism is tricky at first. Try not to care too much - I know that's hard initially. Ask them what's wrong and update your blog post or create a new video with the new details. As Shawn Wang says: "make the thing you wish you had found when you were learning". Many more people than you think will find it useful.

Why should I learn in public?

I don't know about you but I can't remember what I did last week.

Learning in public is a great way to help future you remember what you learnt last year. It helps the person who was in your shoes six months ago who has picked up the technology for the first time and wants to get started. Once you start writing more about a subject, you'll start to become an expert on it. People will come to you for advice and ask you questions online. I did this with my chatbots knowledge. I'd never written a Facebook Messenger chatbot before the end of 2017 and within three or four months I had become an "expert" in the field by sharing my videos of what I'd learnt.

In reality, a lot of online experts are online a few steps ahead of you. The reason you think they know so much is because they're sharing it with you. Most people don't do this. Share what you've just learnt publicly and you can quickly become an expert too. Eventually people will ask you for advice and may well offer to pay you for your time. Again, this has happened with me and I consulted on a few chatbot projects. Learning in public could lead to a new side project. It will definitely open the door to more opportunities. Who is a new employer going to look more favourably at? The person with a simple resumé or the person with a resumé which links to an expert blog post that other people reference?

Learning in public also gives you the opportunity to amplify your message. I can sit down with one person and teach them something I've learnt. But if I write it down, or make a YouTube video, I may reach 5, 50, or 500 people more easily. It allows you to scale your teaching and also your own learning.

What to do next?

Learn something and write it down. Share those notes on your Twitter account. If you've got a website then write an article on there. If you don't have a blog, learn how to set one up. And then write about the experience!

By writing out this article, it's made me attempt to clarify my thoughts on learning in public. Do the same with your learning and I look forward to reading more about it online.

Useful resources

https://www.youtube.com/watch?v=znNxtSbuBjI

Shawn SWYX Wang Learn in public

Kent C Dodds - How am I so productive - A great blog post about being productive and learning in public.

Build in public

Gain confidence at public speaking

2021-01-17T00:00:00Z

I was shy as a teenager.

Really shy.

The sort of shy where I'd blush bright red if somebody I didn't know talked to me.

The sort of shy where I'd blush bright red if I knew them too!

Since my teenage years, the thought of having to get up in front of people and give a speech or presentation absolutely terrified me. I had to explain my university dissertation to my tutors and peers at 21 and I mumbled and sweated my way through that. I then managed to avoid public speaking for a long time. I became less shy over time but I still avoided talking to large groups. And then two things changed which gave me more confidence to start giving speaking to an audience.

Firstly, I got married in 2006. I spent 12 months prior to the big day preparing for my groom's speech. It filled me with fear but I practiced it so often that I knew most of the points by heart. As the groom, you have the obvious advantage that everyone is on your side and they want you to give a great performance. I gave the heartfelt speech and it was incredibly well received. That was the first inflection point which made me think I might enjoy public speaking.

Althought I'd enjoyed talking at my wedding, I carried on with my role in the games industry for many years without much thought of public speaking again. In 2013 I decided to change my technology career and moved to work at the BBC. This was the second inflection point. This is when I decided that enough was enough.

I wanted to become a confident public speaker.

How to get better at public speaking

So what's the secret? How did I go from becoming a nervous wreck to a confident public speaker?

It was a simple difference. I said "Yes!" A lot.

I decided that my job move was a chance to reinvent myself. Every time the engineering manager asked for somebody to share their work with the team. I said "Yes!" When there was a time to present my team's work to the wider department. I said "I'll do it!" Learning sessions about technologies I knew a little bit about? "Yes! I can talk about that."

I'm sure everyone became bored of having me sharing my thoughts over and over again but it did wonders for my confidence. I was really nervous before each one but over time, I realised that I was getting better at speaking to an audience.

So what should you do to gain more confidence?

Change your mindset

Start thinking that you can be good at public speaking. Most people will want your talk to succeed and to hear you speak. The floor is yours so allow yourself to believe that you can do it. The crowd will be on your side even if your brain is telling you they're not.

Plan an outline and edit

Most of us don't have speech writers so we have to write the talks ourselves. Start by deciding on your main theme and then create an outline of the points you'd like to make. Consider using the rule of three when grouping your points. It creates a good rhythm to your presentation.

I don't like to write out every word I'm going to say. I use my outline to remember key points and I'm now confident enough to talk about each section. If your confidence won't let you do this yet, write out more of your speech but try to avoid just reading it to the audience. Take a look at it a few days later and edit it. Try short and snappy phrases. Long and winding speeches can quickly become boring.

Zach Holman has a great site called Speaking.io which has a helpful section on outlining your talk.

Practice makes perfect

I recently hosted a 300+ person online conference with some BBC colleagues. I gave the welcome speech and introduced 3 of our speakers, including a senior director at the BBC. I had so many compliments that I was a natural at presenting and public speaking. But do you know why that is?

I make it look natural because I practice. A lot.

Prior to the big day, I probably practiced my speech to nobody at least 10 times. Don't rehearse the speech in your head. Speak the words out loud like you're presenting to a room full of people. Do this at least 5 to 10 times. You'll quickly realise what works and what doesn't. Remove or rewrite any words or sentences that trip you up.

Record yourself speaking

If you have the time, it's useful to record yourself speaking. Record your talk using a microphone, maybe the ones on your headphones, but your computer microphone will work just as well. Recording the speech will give you an idea of how you sound when you speak. You'll spot the pauses and the "ums" and "ahs". Try and spot what works and what doesn't and edit your speech accordingly.

Find a small audience first

Once you've practiced then it's time to share with an audience. This could be your partner or best friend first. Ask for honest feedback from them. Even if they're not familiar with the content they can comment on the delivery. Try and share it with a few close colleagues. I've shared my talks with my team of 9 people before giving it at our internal conferences. If anything isn't clear then they can tell you.

Use fewer words per slide

If your speech is accompanied by a set of PowerPoint or Keynote slides, use these as prompts to help you to remember what to say. Don't fill them with all of your speech and read off them. Key phrases are much better than a list of bullet points. Use your outline that you wrote earlier and with all that practicing you did, it'll spark a reminder of this part of your story.

Act naturally and be confident

This will be hard for the first few talks but if you come across as your confident self that I know you are, the audience will warm to you immediately. You are the expert in front of a crowd of people who want to hear what you have to say. Remember that. If you can inject a bit of humour into the talk then that's helpful to put both you and the crowd at ease. But remember it's not a stand-up comedy gig!

Try and make some eye contact with your audience. Choose a few people in the group and imagine you're just talking to them. Spending the talk looking down at your notes, or even worse, holding them in front of your face, is not what the audience want. Don't use looking at your notes as a defence mechanism.

A final reminder - It'll be ok

Take a deep breath before you start. Remember to speak slowly and don't rush it. And remember that things might go wrong. You might forget a section of your talk. You might introduce something before you wanted to. The technology might go wrong. These are all part of public speaking. Shrug it off and carry on. It'll be better next time. I know you'll be brilliant anway!

I hope this article helps you to gain a bit more confidence. Go out and speak publicly to others and please say "yes" more when the opportunity arises.

Useful resources

Speaking.io - A fantastic site from Zach Holman which helps you to learn more about public speaking.
The Shy Person’s Guide to Public Speaking - A great article on overcoming shyness and public speaking.

A beginner's guide to using the Obsidian Notes application

2021-01-16T00:00:00Z

I write a lot of notes both in my job and in my personal life. I've tried a lot of different note taking applications over the years and more recently I've started using Obsidian Notes as my application of choice.

Let's look at how to use the Obsidian Notes application for writing clearer notes.

What is the Obsidian Notes taking app?

The Obsidian app is a beautiful, FREE note taking application which allows you to create and store your notes locally in the Markdown file format. It allows you to link notes together and create a knowledge graph of information.

Obsidian notes are stored locally as files on your computer rather than in the cloud. Obsidian has a text editor which allows you to add markdown formatting to your notes. It also has a preview mode which allows you to see how your notes will look when they are rendered as HTML.

As a developer, Markdown is a format I write in regularly and this made the Obsidian app a perfect tool for me.

While you can use the Obsidian app to edit your notes, markdown is a common format for software developers so you can also edit your notes using any text editor that supports the markdown format.

If you're a software developer, you can use a tool like Visual Studio Code to edit your notes if you like.

Obsidian notes are stored locally

An important thing to know about the Obsidian app is that it stores your markdown notes locally on your computer rather than in the cloud. Obsidian notes are simply a folder of plain text files.

As the files are stored locally, you never have to worry about a service being shut down or changing their pricing model. Markdown is an almost standard format so you can move your notes to a new note-taking application if you needed to.

The downside of the files being stored locally is that you're now responsible for backing up your data. However, this is easily done using Google Drive, Dropbox, OneDrive, or some other online storage method. If you're a developer, you can even use a git repository to store your notes.

Obsidian links notes together in a knowledge graph

Obsidian is a tool which claims to give you "a second brain, forever". This means it can store all of your notes and help you to link each note's idea together.

Linking your Obsidian notes together is known as a knowledge graph. It's a series of interlinked notes which allows you to find common themes between your notes. This knowledge graph is also found in the tool Roam Research.

Linked notes is one of the great ideas behind Obsidian and Roam Research. By tagging concepts in your notes, and linking them to other notes using internal links, you slowly build up a graph which helps to you to discover similar ideas and thinking.

Here's an example of my current set of notes in Obsidian:

What is Personal Knowledge Management?

This year I discovered the idea of personal knowledge management (PKM). This is the idea of collecting and classifying your thoughts so that they can be easily retrieved but also help you to make sense out of them. It hadn't occurred to me that I should be bringing all of my notes together so that I could build new ideas from them. I mostly used the search functionality in any notes applications to find things I'd recorded when I needed them. Reading about PKM brought me to a tool called Roam Research which shared the idea of networked thought. It gives the user a way to link notes together in a more coherent way to create a graph of connected thinking. I used this for a while but I was looking for something in the open source space. And then I finally discovered Obsidian.

Using Obsidian for the last 4 or 5 months has really helped me to organise my notes and has started me on a journey to creating new ideas from concepts I find rather than letting the notes wither away and die.

How do I get started with Obsidian?

At the time of writing this, Obsidian is a note-taking application that is available for Mac, Windows, and Linux.

You can easily download Obsidian from their website and then install it on your local computer.

Once Obsidian is installed, you'll be asked to create or open a vault. An Obsidian vault is simply a directory within which your notes will be stored as Markdown files. Obsidians notes are just a folder of plain text markdown files.

Choose "create" if you don't have any files that you've already created. Alternatively you can open a folder which contains existing Markdown files and they will be imported into Obsidian as a new vault.

Once you're set up you can click on "create note" to create a new Markdown file for a new note.

How do I create a markdown file in Obsidian?

Each new note that's created is a Markdown file. Markdown syntax is relatively easy to get started with. It's a plaintext file format with specific syntax which maps to an HTML element.

Here's are some examples of Markdown syntax:

An example Markdown file example-markdown.md

# A main header

Some text as a paragraph. But this text is **bold** and this is in *italics*.

## A secondary header

Another paragraph with some `inline code` in it.

A [link](https://another-website.com/) to an external URL.

What are Obsidian backlinks?

While Markdown supports hyperlinking to another internal or external page using its link format, Obsidian adds the idea of the backlink.

Normal links in markdown files are mono-directional which means that a link points to the new page only.

Obsidian backlinks allow bi-directional linking. This means that if page A links to page B, Obsidian also knows that page B is linking back to page A. This allows ideas to start linking together as you write new notes which creates a knowledge graph of linked notes.

Obsidian backlinks are created using the Wikilink format using double brackets. Here is an example of how to create a note which links to another page:

Obsidian backlinks example obsidian-backlink.md

# My new note

I want this note to link to [[another note]] using Obisidian backlinks.

By default, Obsidian will link to a new note but it won't create it until you follow the link. Pressing CMD + click (Mac) or CTRL + click (Windows) on the link will take you to a new page and then the note is created in your vault.

What are Obsidian mentions?

Obsidian backlinks are really useful for seeing which ideas link together in your notes. For each note, it has the concept of linked and unlinked mentions.

Obsidian linked mentions

Obsidian linked mentions are backlinks in a note that contain an internal link to the active note. This are created when you explicitly link one page to another using the [[backlink]] format.

Obsidian unlinked mentions

Obsidian unlinked mentions are backlinks in a note that contain references to the page from another note but you haven't explicitly linked to it. Using these unlinked mentions, you can quickly see how ideas from multiple notes are related and could be linked together.

The idea of linked and unlinked mentions are really useful to see how your notes are related to each other. You can use these backlinks to expand your thoughts and allow you to create new ideas from them.

Here's an example of some linked and unlinked mentions from when I was building a Google Firebase project and documenting my learning:

What are Obsidian tags?

In addition to linking notes together, Obsidian allows you to use tags. Tags are another kind of link.

Adding tags to a note can make them easier to find in searches if you have a large vault of notes. They're helpful for adding context to your notes and allowing you to group them together. In your Markdown files you add a new tag using a Twitter-style hashtag.

For example, in my own Obsidian vault, I use #toread or #towatch to gather blog posts or YouTube videos I should watch and take notes from.

Obsidian tags example obsidian-tags.md

# An example note

Here are a list of tags for this note:

#toread #towatch #a-long-tag-name

Should I use Obsidian tags or links?

You can and should use both tags and links in your notes.

Links are connections from one Obsidian note to another note. And as I mentioned earlier, they're bi-directional so the notes are both linked together with a backlink.

Tags are connections from a note to an idea. A tag is a link but it has no note linked to it.

So you don't really have to choose between links and tags. Why not use both?

I find that tags are useful for grouping notes together and links are useful for connecting ideas together. They are complimentary and you can easily use both together.

Differences between Obsidian tags and links

While you can use both tags and links together, there are a few important things to remember.

When you change the name of a file within Obsidian, all links to that folder will automatically change to be pointing to the right place. This won't happen with tags.
Clicking on a tag will open a search for all of the files that contain that tag.
Clicking on a link will open the linked note if it exists or create a new one if it doesn't.

What is the Obsidian graph view?

Obsidian allows you to create an unlimited vault of notes but its power comes in linking these notes together.

The Obsidian graph view allows you to visualise connections between your notes and encourages you to explore how your thinking is linked together.

To see the graph view click on "open graph view" in the Obsidian sidebar. This extended view allows you to filter by specific keywords or tags that you use in your notes.

Here's an example of notes related to my use of Obsidian:

Conclusion

I've only scratched the surface with what Obsidian can do for your personal knowledge management system.

Obsidian has a fantastic plugin system which allows more functionality to be added to it. These plugins are built by the Obsidian team and also from other developers.

It also has a wonderful community around it and each of the Obsidian forum members are incredibly helpful. They all want others to learn how to get the best out of Obsidian. You can find their forums here.

Obsidian is constantly being improved by the developers with new releases coming out every couple of weeks. Before you know it you'll be heading down the rabbit hole of building a second brain and Zettlekasten like I have.

Take a look at Obsidian if you're considering levelling up your note taking system and let me know what you think.

Frequently asked questions (FAQ) about Obsidian notes

What is Obsidian notes?

Obsidian is a powerful note-taking application that allows you to create a folder of plaintext files for your notes. It's a great tool for personal knowledge management and allows you to create one or more vaults of notes that are linked together.

How to backlink in Obsidian?

To create a backlink in Obsidian, you use the Wikilink format using double brackets. For example, to link to another note called "another note" you would use [[another note]]. This links your current note to "another note" and also provides a backlink from "another note" to your current note.

Is Obsidian Notes better than Notion?

Notion is a powerful web application which allows you to write notes, create tasks, and manage projects of content. While it has similarities with Obsidian in its ability to create notes, it's a very different tool for managing projects and content with a focus on collaboration.

Obsidian is a private and secure way to store your own notes and ideas on your local computer.

Are Obsidian notes safe?

By default Obsidian stores everything locally in a folder of plaintext files. This means that your notes are only accessible on your computer and not on the internet. This makes it a very secure and private way to store your notes.

You can sign up to Obsidian's paid service Obsidian Sync if you want to synchronise your notes to the cloud and share them with multiple computers. This is private and encrypted so it's still a safe way to store your notes.

Is Obsidian hard to learn?

The hardest part of learning Obsidian is understanding the Markdown format. If you're unfamiliar with it, spend a couple of hours creating notes and reading a Markdown formatting guide. Once you've got the basics down, you'll be able to create notes quickly and easily in Obsidian.

Obsidian's backlinks are really easy to learn and once you understand how they work, you'll be able to create connections between your notes in seconds.

The most complicated part of Obsidian is deciding how you want to organise your notes. This is up to the individual to decide and experiment with. There's no right or wrong way to do it. You can use tags, backlinks, folders, or a combination of both to organise your notes.

Is Obsidian good for note-taking?

I love it for note-taking! 😄

You can keep it simple and write a note in Markdown format. You can link those notes together and create a vault of information. You can link those notes together and start to see how your thinking is linked.

Does Obsidian have a mobile app?

Yes! Obsidian has a mobile app and it works in the same way as the desktop application.

You can down load the Obsidian mobile app here.

Who is Obsidian notes best for?

Obsidian is a powerful note-taking application for many different sorts of people. Not everyone will use it in the same way, but it's useful in different ways.

Obsidian for software engineers

I'm a software engineering manager but I still have to stay technical. I use Obsidian to take notes on the latest technologies and frameworks I'm exploring. I also use it to take notes with my team when I conduct 1:1s, for meeting notes, and for any notes that I take as I read books or watch videos.

Obsidian for university students

Obsidian is the perfect tool for taking notes for university students. You can create a vault for all of your subjects or you could split it into separate vaults for the different areas of your course.

The ability to link your notes together will really help you to understand how your studies are linked together. This will be especially useful when you come to revise for your exams.

Obsidian for academics and researchers

Obsidian is the perfect tool if you work in academia and are researching and writing a new paper. You can create a vault for your research and create new notes for all of your ideas.

Linking your Obsidian notes together will really help to ensure you have citations in your papers and you can easily find the sources you used.

Obsidian for writers

Obsidian is the perfect tool for writers. You can create new notes for articles that you find on the internet or for any videos that you watch.

Take notes of the ideas that you find and start to link them together using Obsidian backlinks. By capturing snippets of linked notes, you can easily research and write your next article.

Useful resources to learn more about Obsidian notes

Francesco D'Alessio from Tool Finder has a great getting started guide for Obsidian:

https://www.youtube.com/watch?v=bLoukY64gYk

Nick Milo has a really helpful set of videos on using Obsidian. He also has a course called Linking Your Thinking which helps you to build your own personal knowledge management system.

https://www.youtube.com/watch?v=QgbLb6QCK88

Justin DiRose from Effective Remote Working has a great set of videos on learning Obsidian too so start here:

https://www.youtube.com/watch?v=LDPrRQFgJxI

Refactoring code: the broken windows theory

2021-01-15T00:00:00Z

Every time we write software we incur technical debt.

Every decision to add new code to a project has the potential to introduce new bugs. Over time a team will introduce disorder to their systems as new features are built and technical architecture changes. So how can you encourage a healthy codebase and avoid it falling into a state of disrepair?

Let's think about the broken windows theory.

The Broken Windows Theory

The broken windows theory is a criminological theory that states that visible signs of crime, anti-social behaviour, and civil disorder create an urban environment that encourages further crime and disorder, including serious crimes.

This theory, from 1982, suggests that a building with broken windows, a potential indicator of criminal behavior, causes the local residents to not care about their environment and other windows soon become broken. After a while, more windows are broken and ultimately the building becomes so damaged that it becomes abandonded.

We can apply this theory to software development and the teams who build and own applications. Small hacks added to the code can quickly add up over time. Leaving badly named variables or functions can encourage the next developer to do the same. If the previous developer didn't care about the code, why should you? Over the time a software project can quickly incur more and more technical debt. Over time the project quickly becomes a tangled mess and you'll start to see your team wince when asked to work on that application.

Avoiding rotting software

Want to avoid software going bad? Let's look at fixing some broken windows we find.

Martin Fowler calls it opportunistic refactoring. This is when a developer sees code that isn't as clear as it should be and takes the chance to fix it by tidying up variables or refactoring a function. If everyone does this regularly, the team can continue to keep the codebase healthy. This little act of fixing a small piece of code encourages the whole team to care about their code. It avoids a "cut and paste" mentality and builds a consistency in the code.

By doing this we quickly build a culture of engineering excellence. 🎉

One practice that I encourage is to avoid large pull requests. These become difficult to review and often get merged with few comments due to their size. If you're writing new code and spot the opportunity to fix one of these broken windows, sometimes it's more useful to note down where the code needs fixing rather than adding it to your current feature work. Instead raise a second pull request to make these changes so that the improvements are isolated from new features. Don't leave it for too long though or that code will start to rot again.

Be pragmatic about fixing all of the issues you find. While there is a huge benefit to refactoring and fixing your broken windows, no software will ever be perfect. You can only work with the best knowledge you have right now. Consistent small changes to make the code better may make a bigger impact in the long run and will help you to balance building new features with fixing the technical debt.

Robert Baden-Powell, the father of scouting, says it nicely:

The Scouting Rule

“Try and leave the world a little better than you found it”

- Robert Baden-Powell

I think this is a wonderful rule to live your life by as well as using it to help you to decide when to refactor your software.

Become a better software engineering manager

2021-01-14T00:00:00Z

I didn't set out to become a manager of software engineers. One day I was the principal software engineer on the team. The next I'd shifted to a management track. I initially wasn't sure if I'd made the right move but I now really love building great software teams.

I've dipped in and out of management over the years but I've now been a software engineering team lead for the past 4 years. I manage a team of 8 software engineers of different levels of experience and every day presents a new challenge. Here's are some learnings which have helped me on my journey. I hope they can help you too.

It's a different career

A software engineering management role is often seen as the next step up from being a software engineer. It's not. It's a different discipline. If you love coding, you probably won't be doing as much of it if any. There are many reasons to become a manager but make sure you know what you're getting into. It shouldn't just be the next step after becoming a senior engineer. People management is something different than managing a codebase. Take a look at some reasons to not be a manager written by the awesome Charity Majors before you consider a management role.

You'll have imposter syndrome

You've spent 20+ years writing code. You can reverse a binary tree on a whiteboard while deploying a Kubernetes cluster. If you can do that, being a manager must be easy right?

STOP!

Your day of coding will be replaced 1:1s with your team, stakeholder meetings, backlog grooming in Jira, looking through résumés, writing technical documents, and so on. For a while you won't know what you're doing but perserve. The imposter syndrome may always be there but it will stop nagging you eventually. Ensure you find a network of peers that can help you to learn. Find other engineering leaders who have done this for a bit longer than you have. Share your journeys and you'll find that other people struggle at times too. A network of peers is really helpful. Meet them regularly and it will help you all to grow.

Build a culture of positivity and honesty

Your job is now to manage and lead a group of people to success. It's your job to set the tone for the team and help to define the team's culture. Be the one who builds a culture where all voices matter. A junior engineer can have great ideas like your more experienced developers but you must allow them to feel comfortable enough to share them. Ensure that your meetings are inclusive and make all of the team feel comfortable enough to share their opinions. Why not use a collaborative document instead if you feel that some people are less comfortable at public speaking with more senior engineers around. Help to set the expectations for code reviews and ensure that it feels like a learning experience rather than a telling off. Appreciate your team in public. If someone has written some fantastic code or released a great new feature, share and celebrate it both inside and outside of the team. Most people are quick to complain about mistakes but aren't so fast at thanking people for great work. Be the champion of your team!

Trust is the key

Most people won't let you down. Trust them to do their best work without micromanaging them and telling them exactly what to do. Yes, it's your resonsibility to ensure the work gets done but do this by setting the direction and not piloting the ship. There may be times when the team doesn't enjoy the work so your role is to explain the difficult choices which may have been made. If you've built an culture of honesty and openness, the team will understand. Allowing autonomy in the team will lead to greater motivation and a belief in their own decisions. If you trust your team then your team will trust you.

Become a great listener and communicator

Your new role as a manager is mostly about clear communication. You'll be talking with project stakeholders about the requirements and deadlines for the project. You'll be sharing details with your team and setting the technical strategy for what you need to do. You'll be hearing about the needs and struggles of your team in their 1:1s. You'll have to provide mentoring and feedback for your team. What you say and how you say it will matter to the team, your manager, and other people you deal with. Try and learn how to write and speak clearly. This isn't easy but it's a skill you should attempt to get better at.

Know that some conversations will be difficult

Your job as a manager means you'll have to deal with the good and the bad. Your team won't be happy all of the time. You'll need to be a great listener and able to deal with tricky conversations. This has been even more apparent over 2020 and a global pandemic! If you've built a culture of openness, these conversations should happen regularly and you should embrace the fact that your team feel comfortable enough to be honest with you about how they feel. If you don't know the answer, it's ok to say you don't know. Ensure you've got support networks to help you find answers. If it's a conflict within the team, talk to everyone involved and help to resolve it as quickly as you can.

Your job isn't to know everything

When you first start to lead a team, you expect to be able to answer all of the questions from all of the people. You quickly realise that this isn't achievable. It's time to be comfortable with saying "I don't know" to people. It's better to become an enabler for the team. You might not know the answer but you can find out from your senior engineers, or other team members. Open up your network to the team and connect them with the people who can answer their questions. You become more valuable to the team when you realise you don't have to know everything but you know where to find the answers.

Advocate for your team

Your team won't necessarily have the voice that you have as a manager. It's your job to shout loudly about how amazing they are. You should be sharing the great work they're doing with the rest of the department. You can be a supporter of team members and help to guide them in their career. It's up to you to help to mentor and develop the team. Open up your network to them and help all of your engineers to find new opportunities to grow. If you do this you'll bring the whole team up. I've always loved the saying "A rising tide lifts all boats". Bring everyone along on the journey with you and help the whole team to succeed.

Keep on learning

Being a better software engineering manager is an on-going process. Ask for help from your manager and peers. Talk to your team and ask them how you can improve what you do. Ask for a step-level 1:1 with your manager's manager and learn from them. New challenges will come at you each day but embrace them and work out strategies to deal with them. Never stop learning and share your experiences with other managers. Let's help each other!

This isn't an exhaustive list and I'm sure I'll think of more ways of becoming a better manager as I learn them myself. You can read some other great suggestions on my good friend Gavin's website. He's another software engineering manager that I trust.

If you want to read about my journey from a being a developer to a software engineering team lead then I shared my thoughts in an interview on the Dev To Manager website.

Creating a writing habit

2021-01-13T00:00:00Z

At the start of 2020, I wrote a review of 2019. One of my main goals for last year was to find my writing "voice". The plan was to write more in 2020 but I wrote a grand total of 1 blog post. I'm not sure that a single blog post can really be considered a big win for my writing! 2020 really wasn't a year that was conducive to getting things done for me. We all struggled in different ways throughtout the pandemic and I couldn't manage to kickstart a writing habit.

I've always been an avid note taker in meetings but I'd never looped back to use my bullet points to build coherent writing. At some point during 2020 I discovered Roam Research as a tool for networked thought. I moved my note taking into Roam from Dynalist but then I discovered an application called Obsidian. While very similar to Roam, it was the community that drew me in. A lot of friendly people who were encouraging each other to write more and link ideas using Obsidian as their tool of choice. I continued to use this to create private notes for myself, but have not been great and bringing them together to write more in public.

Now is the time to change this.

I've been reading a lot about consitency in everything you do. It's a fundamental for building any habit. My ability to write software and quickly pick up new technologies is born out of my love of learning and applying new languages, frameworks, or libraries when I find them. I tend to spend many evenings reading through documentation and writing code to apply what I've learnt. But with public writing I tend to stumble. I can write README files for my code repositories. It's much easier to be factual. But I get nervous about sharing my thoughts with the outside world. So is this a habit we can learn to get better at? I hope to find out.

I admired Nathan Barry of ConvertKit for his challenge to write for a whole year. This feels like a long time frame to try and commit to. Instead, I'd like to create a small habit by sharing my thoughts publicly for 30 days. I'm sure James Clear writes about similar ideas in his Atomic Habits book (which is on my Kindle but I've not yet read it). I noticed the hashtag #ship30for30 in some tweets from people I follow. It's a community of writers who are sharing their blog posts for 30 days to build their own habits. While I've not joined the community yet, I'm going to attempt to write something new every day without thinking too much about it. If I can do this for 30 days then I'll be incredibly happy and hopefully will have built a habit and found my writing style. It's really easy to get bogged down with attempting to share your best work so by being habitual I hope to stop worrying.

Here's my plan:

Capture daily thoughts in Obsidian. Most days I attempt to write down ideas that come to mind during the day. These are sometimes linked to things I've previously thought about and using Obsidian to link my notes, I can hopefully find new ideas to write about.
Find a quiet space away from where I normally work. I sit in the same room in our house every day and it's become my office given that I've worked at home 100% of the time since March 2020. By finding a different environment within the house, I hope that it'll inspire me to not think about work and instead free my mind to write from my notes and capture coherent thoughts into a blog post.
Set a time. A habit needs regularity. I'm planning on writing a blog post after my working day is done. As the whole family are currently locked down and working or schooling from home, I can normally start and finish my day early. This affords me an hour to write from around 4.30 pm.
Set a word goal. Honestly, I'd be happy to write a couple of paragraphs but it seems sensible to aim for a word count each day. I like the #ship30for30 idea of "atomic esssays". These are up to 250 words and encapsulate a theme. I think this is manageable each day. Although I'd quite like to write 1000+ word essays, I think the consitency of writing matters more than the length of the work.
Don't overthink things. Says Marc as he writes a list of rules he's going to try and stick to! Seriously though, I want to write and not worry too much about what I commit to the internet. To write, quickly edit, and share is the main goal.

This is the first of 30 days of writing. Let's see if I can keep it up.

Visualise your code in style with pretty screenshots

2020-11-14T00:00:00Z

If you're a developer, and you hang out on developer Twitter, then you'll have seen people sharing their hot code tips. They're often accompanied by code examples and the "hot tip" fire 🔥 or lightbulb 💡 emojis.

Here's a great example from Simon Høiberg:

JavaScript Tip 💡

Make an argument 'required' in JavaScript using this small trick 👇 pic.twitter.com/yZ2w5Kd58b
— Simon Høiberg (@SimonHoiberg) November 13, 2020

These are not ordinary code screenshots straight from a code editor. They contain the code, syntax highlighting for the language, and are often embedded in a browser window with a colourful background.

🔥 It's a beautiful code screenshot!

If you read the comments on these code tweets, you'll usually see a list of people asking how to create such pretty code screenshots. So how do we do it?

Read on to find out how to create beautiful code screenshots to share with your Twitter or Instagram followers.

Websites

Carbon

Carbon is one of the easiest tools to use. There are no applications or extensions to install. You can quickly create great looking code screenshots on their website. Their code is also open source on GitHub if you want to use it yourself.

Type your code into their online editor and you can see a prettified version of it inside a Mac-styled window. It allows you to choose a variety of code themes and can syntax highlight your code in various programming languages. You can also alter the background colour, padding, and many other window or editor options. Play around with it to see what you can create.

If you use GitHub gists then you can create an image from one of those. Append the identifier of your gist to the Carbon URL and you'll see a pretty version of your code. For example here's my gist and it we use the ID "0561c83e8445f3ffc801a5a13415abef" in the Carbon URL like this "https://carbon.now.sh/0561c83e8445f3ffc801a5a13415abef" then we see the screenshot above.

Ray.so

Ray.so is another easy site to create beautiful screenshots. It's created by the team behind Raycast and it's a lot more opinionated than Carbon. It only allows a minimal set of colour options but does have both light and dark modes. It supports lots of languages with its syntax highlighting support, allows you to change the title of the window, and also the padding. It offers a more curated experience than Carbon but it's similarly easy to use.

Chalk.ist

Chalk.ist is a new site which I found on Twitter in April 2022. It's created by a developer called Kasper Mikiewicz and adds some additional features to the normal code screenshot idea. It allows you to add your Twitter handle if you enable it. This will be great for social sharing.

Another useful feature is a code difference view, like you'd see in your code editor. This is useful for sharing development tutorials or for teaching. While it currently only offers 3 colour themes, it does give you line numbers and a reflection to make it look different to the other sites.

VS Code extensions

If you're a developer who uses VS Code there are a couple of extensions which will make creating pretty code screenshots incredibly easy. VS Code extensions add new features, themes, and more to the editor. If you install one of these extensions, you can avoid the need to use an external website and screenshot your code instantly.

Polacode

I've used this extension for a while and it's an easy way to create shareable screenshots of your code without leaving the editor. Install it via the extensions sidebar as you would with other extensions. You can then use the command palette to choose Polacode. This will bring up a new tab where you can cut and past code into the Polacode window.

Cutting and pasting code from your open project will allow Polacode to match your current theme. Hit the button underneath the code snippet and you'll get a lovely code screenshot to share. You can update the background colour and shadow using VS Code's settings for the extension. The only issue I have with this extension is that it struggles with longer lines of code. These often wrap onto the next line which isn't as aesthetically pleasing. For this you'll have to edit the code yourself, perhaps using Prettier or similar.

CodeSnap

Similar to Polacode, CodeSnap is another VS Code extension. Again, you can easily install it via the VS Code extensions sidebar. CodeSnap is a bit more fully featured than Polacode and allows more adjustments in the settings menu. It also has quick access by right clicking in the current window to choose it from the pop-up menu.

One of the advantages of CodeSnap over Polacode is that it automatically captures any text in your clipboard and creates a pretty code screenshot based on that. It's also attempts to

Non-code screenshots

In some cases you might not need to take screenshots of code. Instead you can make pretty screenshots of websites, photos, or any other image you might have. There are a few ways to do this.

Screely

Screely is a great website which works in a similar way to Carbon. It allows you to use any image file and displays it in a browser-style window. Again, you have the ability to style the image with a variety of options for the background colour, padding, and window type and style. Have fun and make your non-code images beautiful too.

Using your operating system

While there are websites and code editor extensions you can use, maybe you just want to use your operating system. You can just take screenshots of your code using Windows or Mac OSX. You'll have your styling and code theme exactly the same as you see it on the screen. Let's look at how we can do that.

Windows users

Windows 10 makes it simple to save a copy of whatever is on your screen by using the "Print Screen" key on your keyboard. This if normally labelled as PrtScn on a Windows keyboard and is often located on the top row near the function keys. On laptops, you often have to hold down the function key first before pressing PrtScn.

If you press PrtScn on its own then it will copy the entire screen to the clipboard. You can then paste the image into Microsoft Paint or similar picture editing software.
If you use press the Alt key and PrtScn then this copies the active window into the clipboard. Again, you'll need to paste this into your picture editing software to save it.
If you press the Windows key and PrtScn then it'll save the entire screen as an image file. This will be saved in the "Pictures" folder in a sub-folder called "Screenshots".
If you press the Windows key with Shift and S, you can screenshot a portion of the screen. The screen will be covered with an overlay and mouse cursor turns into a plus (+) symbol, which indicates that capture mode is on. At the top of the screen you'll see a menu bar with the options: rectangular snip, freeform snip, or fullscreen snip. If you choose rectangular or freeform, you can select a region on the screen. This copies that area to the clipboard.

Mac users

Mac users also have easy ways to create good looking screenshots using the OSX operating system.

Use Shift + Command + 4 to turn the cursor into a crosshair. This then allows you to drag and select part of the screen to capture as an image. If you release the mouse, this gets saved to your desktop (unless you've changed the folder you save images to.)
If you press and hold the space bar before releasing the mouse button, you can move the selected area around the screen before taking the screenshot. This is useful for changing your selection and making it pixel perfect. Release the mouse button to save the image to your selected screenshot folder.
If you press Shift + Command + 4 (⇧ + ⌘ + 4) and then press the space bar, this turns the cursor into a camera icon. If you move over any open window and press the left mouse button, this will save an image of the whole window, including the window bar, to your chosen screenshot folder. This is a great way to make screenshots that look great.

Accessibility

Don't forget that posting screenshots of code aren't accessible. Make sure that you add a link to a GitHub gist or to your code repository so that people with any visual impairments can read your code in other ways.

Got any other ways to style your images that I've not mentioned? Send me a message and let me know.

👉🏻 Like this post? You should follow me on Twitter

Different types of software tests

2020-04-28T00:00:00Z

Understanding types of software testing

New developers often don't know where to start with testing their code. They know that it's a good idea but there's an overwhelming amount of information out there. Common questions and comments I hear are:

What's a unit test?
How does it differ from an integration test?
Should I be writing end-to-end tests?
I don't know where to start!

I'll help you to understand the differences in testing types and make you a little less scared to get started on your software testing journey.

What is a test?

This seems like a simple question right but a lot of people get scared when you mention software tests. There's an easy pattern which you can follow which will simplify how you think about writing tests. It's known as Arrange, Act and Assert (AAA) and it's a common way of writing tests.

Firstly, you need to set up our pre-conditions for the function under test. This is your Arrange. Here you can initialise any objects, or data you need for your test. You can also define the output value you expect from your code after execution.

Next, you call your code with the specified inputs for your test. This is your Act. Call your function or method with the given input as though you were calling the code in production.

Lastly, you will verify your expectations based upon either the output value, or the code paths executed. This is your Assert. You know what you expect your code to do so you can now check that it happened. If it didn't, then the test has failed and your code, or sometimes your test, is incorrect.

Unit tests

Most of the code that you write will need to interact with other code. It'll call another function or class method in some way. We want to split up our code into the smallest unit that it makes sense to test. Most of my projects are written in JavaScript using Node.js so for me this is typically a function. We then test this small piece of code. Testing a small amount of code is known as a unit test. If your test still has to allow your code to use an external service, for example an API request or database call, then it's not a unit test.

A unit test should be one of the most basic tests you have. As you want to test the code in isolation, you might have to stub or mock some external dependencies to define specific scenarios. You'll find that writing unit tests often helps to define the structure and design of your code. If it's difficult to test, it's often a code smell that means you could refactor your code to make it easier to test.

One major factor which defines a unit test is its speed. Due to the isolation of the code from its other dependencies, you expect these tests to be incredibly fast. You will often end up writing a lot more unit tests than you will the other test types. A typical unit test suite may have thousands of tests and you'd expect a single test to take much less than a second to run.

Integration tests

While your unit tests aim to be simple and test a single module or function, integration tests, as their name suggest, tests how code modules fit together in your application (i.e. how they integrate). In your unit tests, the aim is to isolate the external dependencies but for your integration tests, you may allow your code to communicate with data sources like your database, or also external APIs. This isn't a requirement and you can still mock these external dependencies if you want to.

Integration tests allow you to test the full flow of your application and test how code modules interface with each other. The classic gif below shows what could happen if you have unit tests for your project and no integration tests. Both unit tests appear to pass correctly, but when the code for both is combined, it's actually incorrect.

Integration tests are generally slower than your unit tests so you'll often have fewer of them to test complete code paths. You may need configuration of external applications, like databases or HTTP requests to external APIs, if they're not mocked out. These types of tests are great for testing higher level logic and data flow through your application.

An example of an integration test is to test the full flow of an API on a server. This may involve starting a server which runs your application code, responding to HTTP requests, authorising a client or user, validating query parameters or body data, connecting to a database or other APIs, converting data to JSON, and eventually returning the data.

One thing to consider is that you can still mock external dependencies if you want to. It's often better to have a thin wrapper around an external service which can return fake data in your integration tests. This still allows you to test your complete application flow as an integration test, but not talk to a production API or database. A good example would be an application which uses the Twitter API to send tweets. You wouldn't want it to send a tweet every time you ran your test suite!

Functional / End-to-end tests

Function tests are also know as end-to-end (E2E) tests and they test the complete functionality of your code. They runn the complete application, usually in a browser, without mocking any external dependencies at all. These types of tests are the slowest to run so you'll generally have a lot less of these than integration or unit tests.

An example of a functional test is to verify that a user can sign into a website and that they can view the expected data once authorised. This test would use automated browser interaction to enter the username and password and click on the sign-in button. The test would verify that the title of the next page is as expected and that the user's data is displayed. This is often done by querying the DOM to determine if elements are present. There are a lot of pre-conditions for this test, such as having a valid username and password combination, and knowning the DOM elements to verify. This means that functional tests are often more brittle and prone to breaking than unit or integration tests. Be wary of relying on user interface data for test expectations as developers or designers often change it!

Test doubles

When writing tests, there are rarely functions or classes that operate in isolation. Most code interacts with other modules in some way. If you want to isolate your function for testing you can use a test double. This is the generic term for when you replace a production object for testing purposes with a fake. Think of it like a film where a stunt double replaces the actual actor for the dangerous scenes.

There are various kinds of test doubles that you can use in your tests and there will be different uses for each of them.

Dummy - A dummy object is used in place of your production object and will match the interface of the thing it replaces. However, you don't care what it does. Think of it as a simple placeholder which you'll need as one of your function parameters.
Stub - A stub is an object or function which responds with a pre-programmed response. It normally only responds to what has been programmed for the test.
Spy - A spy is a function which wraps an object and can record information about how the object is used. This is useful for avoiding altering the behaviour of the function, but still determining how it was interacted with.
Mock - Mocks are fake objects which have fake methods. They also have pre-programmed responses like stubs. However, the main difference to spies and stubs is that they also have pre-programmed expectations. If the mock is not used as expected by your code, the test will fail.
Fake - A fake object is similar to a stub and it will return some expected values. However, it also implements some of the functionality of the object it's replacing. An example would be an in-memory data store which replaces the production database.

Watch the video

Hopefully this been a brief but helpful walkthrough of the different types of software tests you can write. I've recorded a video which summarises what you've learnt. I talk about the different types of tests you might write as a developer and give some information on test doubles and some of the tools that exist in the JavaScript universe which will help you to write tests. You can watch it here:

https://www.youtube.com/watch?v=pMmXJcDLUFA

If this post was valuable to you, you can follow my journey on Twitter.

Want to learn more about testing your JavaScript code? Check out my other JavaScript testing articles for more tips and techniques!

Add time and date to your Messenger chatbot with the Chatfuel JSON API

2020-02-14T00:00:00Z

Watch the video

https://www.youtube.com/watch?v=UDrGpc4Dp8w

One of the main benefits of creating a Facebook Messenger chatbot is that you don't have to be present to answer messages for 24 hours per day. Your chatbot should be on hand to do this while you're fast asleep. What would be even better is if it could understand where your user lives and give them relevant content for their time of day. It could let them know whether your shop is open for business. Or it could send your users the breakfast menu rather than the dinner menu because it knows it's 9am where they are. This functionality doesn't exist in Chatfuel by default so let me teach you how to build an API which adds time and date functions to your chatbot.

Timezones & Coordinated Universal Time

When your users talk to you on Facebook Messenger, they may live anywhere around the world. This means we need to know what the time is where they live. Each user can potential live in a different time zone. This means we may need to add or subtract a number of hours to the time where your server is to allow for this time difference. Although you might think that it's always a whole number of hours, there are time zones which have half hours too. But what time is it where your API server is hosted? Well, this depends on where the data centre is that is storing and executing your code. What happens if your server is in New York, which is 5 hours behind Greenwich Mean Time (GMT), but your user is Germany, which is 1 hour ahead of GMT. And what happens if daylight saving time has happened in one of those countries? Calculating the time is hard and things can get even more complicated. That's why we have a time standard which everyone uses. It's called Coordinated Universal Time or UTC. The Wikipedia page will explain why it's not an acronym of CUT! Using UTC allows us to define all times as relative to this standard. If we store all of our times in UTC, life becomes a bit easier.

Calculating a user's time

I've set up an example Node.js Express server using Glitch which exposes an API for all of my Chatfuel demos. The API for this article is available on the following endpoint:

https://chatfuel-demo-bot.glitch.me/timedate

The first thing we need to do is to calculate the user's local time using UTC. JavaScript (and Node.js) has a global object called Date which expose a set of methods we can use to determine and manipulate the date and time. First, we need to determine the time right now. We can use the Date.now() function which will return us the UTC epoch time. This is the number of milliseconds since the start of UNIX time (at 00:00 on January 1st, 1970). We can then use this to calculate how many milliseconds difference between the user's timezone and the time now. Using JavaScript's Date object, we can then inspect this and pull out the date and time to use in our chatbot. Let's look at some code to do this:

// Calculate the number of milliseconds in an hour
// - 1000 milliseconds in a second,
// - 60 seconds in a minute
// - 60 minutes in an hour
const ONE_HOUR_IN_MILLISECONDS = 60 * 60 * 1000;

// Pass in an offset in hours which we use to calculate the user's time
const getUserDateInTimezone = offsetInHours => {
    // Calculate the UNIX timestamp in UTC
    const utcDate = Date.now();
  
    // Get the user's offset in milliseconds
    // The offsetInHours could be a negative number if
    // the user's timezone is behind UTC
    const offsetInMilliseconds = ONE_HOUR_IN_MILLISECONDS * offsetInHours;
  
    // Create a new Date object which adds the time offset to the time now
    return new Date(utcDate + offsetInMilliseconds);
};

Once we've written a function to calculate a new Date object, we can use the built in functions to inspect the data and return it to our user. As we're using Express to expose an API, we'll use its routing methods to expose a HTTP GET endpoint which allows our Chatfuel chatbot to make a request using the JSON API. We will allow it to pass through a timezone as a query parameter which Facebook Messenger can provide us. We can then set some user attributes which are returned to Chatfuel and can be used in our chatbot flows. Let's look at how we can do this:

const express = require('express');
const router = express.Router();

// ... add the time calculation code from above here

router.get('/', (request, response) => {
    // Allow a query parameter called "timezone" to be passed with the GET request
    // Default it to zero if we don't send one
    const {timezone = 0} = request.query;

    // Calculate the user's current Date in UTC using our code from above
    const userDateInTimezone = getUserDateInTimezone(timezone);
  
    // Inspect the Date object and pull out the data we need
    // Build an object which we can return to Chatfuel
    const dateObject = {
        // Get the day
        day: userDateInTimezone.getDate(),
        
        // Note that month is zero-indexed (from 0 for January to 11 for December)
        // For it to make sense for an actual month number, add one to it!
        month: userDateInTimezone.getMonth() + 1,
        
        // Get the full 4-digit year e.g. 2020
        year: userDateInTimezone.getFullYear(),

        // Get the number of hours
        hours: userDateInTimezone.getHours(),

        // Get the number of minutes
        minutes: userDateInTimezone.getMinutes(),

        // Get the number of seconds
        seconds: userDateInTimezone.getSeconds(),

        // Get a standard ISO-8601 formatted string
        isoTime: userDateInTimezone.toISOString()
    };

    // Format the data so that Chatfuel can set user attributes
    const userAttributes = {
        set_attributes: dateObject
    };

    // Return a JSON response that Chatfuel can understand
    response.json(userAttributes);
});

// If we're putting our routes into separate modules for clarity
// then we need to export the routes for use in the Express application
module.exports = router;

The last thing you need to do is to use this code in your Chatfuel chatbot. You can use the redirect blocks to check the value of "hours" returned after calling your API. If they are greater than 18, or 6pm, you could consider it to be evening and change the user responses accordingly. In a similar way, you can check if the value is greater than 6, or 6am, and then consider it to be breakfast time. Once you've got the user's time and date, you can give timely responses to your user. Watch my video for more examples.

All of the code is available on my Chatfuel demo Glitch project. Feel free to clone it and use it for your own APIs. If you spot any errors, or have any questions, then please send me a message. I love to hear from people and I'm always happy to answer your questions.

2019: Personal Retrospective

2020-01-06T00:00:00Z

I started my 2020 by thumbing through Twitter as I often do, and often try to avoid doing. Between the technology and political tweets were personal 2019 reviews and retrospectives. In software development, we use the idea of the agile sprint retrospective). A sprint is a period in which you attempt to produce new features for your software application. A sprint retrospective is a team meeting when you discuss how the last sprint went. What went well, what could have gone better, and any actions to make sprints better. These personal reviews took this idea and applied it to their lives. Here is my personal 2019 retrospective.

What went well?

One of my big goals every year is to spend quality time with Clare and our children. This year, we had a fantastic holiday to Disney World and Universal Studios in Florida. We planned to go in 2018, but issues with my feet changed our plans. While I had some toe issues before and after the holiday, we managed to walk for five miles on most days. A fantastic holiday! As the family loves theatre, we saw a surprising amount of musicals and plays throughout the year. The children also performed in four shows and I'm proud of how far they've come with their acting and singing.

Another major goal was to ensure I saw more of my friends. Once you hit your 40s, you end up with more family commitments so seeing old friends becomes harder. This year I had two major get-togethers with a great group of mates. The first one involved far too much gin! As we all head towards the wrong side of our 40s, I also managed to celebrate a couple of 50th birthdays. Music has always been a major part of my life and I saw more live music in 2019. Thanks to some tickets for my birthday, I saw The Midnight Hour in Liverpool. This ended with us meeting Ali Shaheed Muhammad from A Tribe Called Quest. A big-time legend from my hip-hop youth. It was also great to see the Loop Daddy himself, Marc Rebillet at Manchester’s Albert Hall in November. If you don't know who he is then Google him. Seeing a man improvising songs in a pair of boxer shorts, a silk dressing gown, and patent leather shoes is something to behold. Part comedy. Part sex symbol. Yet he's a brilliant musician and singer. One of the best live shows I’ve seen! The final gig of the year was Kruder & Dorfmeister. What I thought was going to be a chilled trip-hop experience turned into a middle-aged rave. Banging tunes and a great light show made it a fantastic night.

For the past three years, I've been a Software Engineering Team Lead at the BBC. This means that I build and manage high performing teams and don't write much code during the day any more. Most of my technology learning happens in the evenings after the kids have gone to bed (which gets much later as they get older!). I wondered how much software I wrote in 2019, so I had a look at my GitHub commits for the whole year. I surprised myself with the number of new technologies I picked up, most of which I hadn't touched before. These were:

BotKit
Google Cloud Platform
Firebase location-based API which imports data from Airtable
Gatsby
GraphQL
Shopify applications and Polaris
Spark AR
Markdown to Mailchimp email generation (as I dislike the Mailchimp WYSIWYG editor!)
Open Graph image generation using serverless functions
Zeit Now 2.0 (moving a Now v1.0 API to use their new serverless functions)
Firebase serverless function to scrape Amazon book rankings

"Scratching your own itch" is always a good way to gain more knowledge of areas you're unfamiliar with. Since 2017 I've become an expert in building Messenger chatbot experiences and I've often used chatbot platforms like Chatfuel to build them. As I was consulting for a Messenger-based travel company, they were looking for cheaper options. This led me to build a solution using BotKit hosted using Google Cloud Platform. This was a fantastic learning experience for GCP App Engine, Cloud Firestore. It also allowed me to create a location-based API using Firebase functions and Airtable. Unfortunately, the chatbot isn't yet live due to a lack of money from the company involved. See also "what didn't go well"!

A big win for me this year was deciding to refactor my wife Clare's author website using Gatsby. The previous incarnation used Wordpress and needed some love. Gatsby also allowed me to learn some GraphQL in the process. This led me to build some Shopify integrations for my chatbot API and understand their integration. My chatbot API needed a bit of love this year as I'd left it to stagnate using the old Zeit Now 1.0 framework. This lets you deploy an Express server and spin up compute instances as needed. Zeit used to host the whole API this way. With their new Zeit Now 2.0, they pivoted to a serverless approach. The advantage of this allows the API to scale on a per endpoint basis. Yet, this requires a separate function for each API endpoint. While this was a big rewrite on my part, it made my application API much simpler. One observation I have of my 2019 project is that one learning moment leads to another. A problem or unknown solution in the first project often brought a discovery of a new technology which could help. I just wish I'd done more learning in public as Shawn "SWYX" Wang suggests.

My work life was pretty good for most of the year. The BBC launched one of its big diversity initiatives - the "Step Into Tech" scheme and I became an advocate and advisor for this. It was a 14-week training programme to teach the basics of software engineering to women who were looking for a career change. I helped to mentor four of the 2018 cohort and encouraged them to shadow our team to see what we did. With the help of a senior engineer, all four women made a change to our codebase. They deployed their changes to our live applications, something they should be proud of. One of the women joined us in September, and it's been rewarding to mentor and guide her new journey. I also ended up spending a lot of 2019 hiring and building up my engineering team. By the end of the year, I'd interviewed and hired four fantastic software engineers. Big thanks to my team for their help and support in this. A highlight of my working year was attending the amazing Lead Developer conference in London in June 2019. As a software engineering team lead, it's much harder to see the impact you make on a team as you no longer write code. This brought together some excellent speakers from the engineering teams across the world. It was fantastic to hear presentations from likeminded team leads, and to hear the challenges of building high functioning teams.

One of my biggest wins of the year was to use a habit tracker, a Google Sheet, to see where I was spending my time. I wanted to practice quantified self and so recorded statistics about my life for the whole of 2019. I tracked the parts of my life I thought I should improve, such as when I went to bed, when I exercised (or didn't!) and so on. One of these was reading. I've often found it tricky to stick to a regular reading schedule but by recording when I read, I tricked myself into reading more. I finished ten books in 2019 so that's a personal win for me.

What could have gone better?

After my near-death experience in 2014, I knew I should attempt to get more exercise. I've encountered a few health issues following my hospitalisation so I spent 2019 attempting to get fit. This began well but as has happened in the past, I wrestled with more issues with my toe. The amputation of my right great toe creates more pressure on the next toe. This means a greater potential for ulcers (holes) in it. It's a constant balance between more cardio exercise whilst avoiding further complications. My habit tracker showed that I spent 26% of my year exercising. I should have spent more time getting my bike in order and cycling but I didn't prioritise this. This would avoid any major impact on my feet versus running, which my podiatrist advises me to avoid.

Since 2017 I've become an expert at building Messenger chatbots. I acted as a consultant for a London-based travel company based in London at the end of 2018 and the beginning of 2019. This was a side-project which I hoped would lead to more consultancy work. I leveraged the BotKit framework to build a cost-effective bot without the cost of an external platform such as Chatfuel. While this was a great learning experience, I spent a lot of time working on the project to finish the required features. As the company is a startup, it had little spare cash and failed to pay my final invoices. Given the effort I put into the project, this caused frustrations on my part. I had a similar experience in 2018 where I built a Dialogflow and Twilio chatbot for an American entrepreneur and no payment materialised. It's made me realise that I need to become better at negotiating full payment and contracts up-front.

As well as developing a better reading habit, I wanted to build a habit of regular writing. I missed the mark with this with only 11% of my days spent writing. My one success here was writing and releasing a 30-page guide to JSON for chatbots (which you can get here). I failed to find my writing voice so this is a big goal for me in 2020.

Since late 2017 I've had plans to build an online course. I love helping people and want to share my knowledge with us to generate a side income from it. My JSON guide was my initial "lead magnet" to encourage people onto my mailing list but I failed to capitalise on this. I spent too much time thinking about what I should do rather than creating. I should have been releasing early and often and learning what people want. Paralysis by analysis happens far too often so I'd like to avoid this in 2020.

Plans for 2020

My main goal for 2020 is being addressed by this blog post as I plan to find my writing "voice". Since school, I've often told myself "I'm not very good at English" and "I'm much better at maths". I often write in my day job, be it technical papers or tickets, or emails to senior stakeholders. But I rarely write for a wider audience. The plan for 2020 is to blog more often and do a lot more learning in public. I'm will endeavour to write about the technologies that I'm learning, how I'm building my online course, and more.

As 2019 wasn't a great year for fitness, I need to get much fitter this year. As I enjoy cycling, I'm going to get outside more on my mountain bike, both alone and with the family. My wife and I are also trying to get the family into a regular habit of getting outside to walk more. We live near some beautiful countryside and it seems a shame not to see more of it. This will combine cardio exercise with fresh air which always helps to clear your head.

Since stomach surgery in 2014, I've had a more difficult relationship with food. I love cooking and have always enjoyed eating great food but I often struggle to eat well. I have to consume much smaller portion sizes and I'm often sick after eating. I now find meat more difficult to digest so I plan to lean towards a more vegetarian diet. This has obvious health benefits for me and also has the potential to help with climate change. I'd love to head towards a mostly plant-based diet but I like cheese too much. I've started the year as more of a flexitarian, with a much-reduced meat diet. I'll see where this year takes me but I may well be completely vegetarian by the end of the year.

This year I want to get some traction on my side projects and aim to make a reasonable side income through it. I'll be publishing more videos on YouTube and this will be part of a marketing push to drive more traffic to my website. I am committing to publishing a short course in the first quarter of 2020. I'll be doing some research in the next couple of weeks and aim to release an outline before the end of January 2020.

As well as spending more time with the family, I want to get back to some of my old hobbies. I spend most of my evenings reading about technology or writing new software. This hasn't allowed me enough time for non-tech related hobbies. I increased my rate of reading in 2019 and the goal for 2020 is to read twenty books. I also want to resurrect my old love of making music. I've not made any new tracks since the early 2010s so it's time to dust off the audio interface and MIDI keyboard and get back to making electronic music. 2020 could be the year I dust off the decks and get DJ Cruze rocking the house again. Watch this space!

I've set myself some audacious goals for 2020 but there's little point in not trying to better myself. I look forward to reviewing them all in 2021 to see how far I've travelled. Will I have achieved most of them? Let's see next year.

This post is inspired Knut Melvær, Jason Lengstorf, Justin Jackson, Steph Smith, and Madison Kanna, while listening to the wonderful French band Phoenix and the album Wolfgang Amadeus Phoenix. A classic from 2009.

How to validate an email adress using the Chatfuel JSON API

2019-11-24T00:00:00Z

Facebook Messenger chatbots are a great addition to your marketing toolbox but they have one big problem.

Facebook controls the platform and can change its rules when it wants to.

Their recent changes to their broadcast and subscription messaging rules means it's more difficult to contact users after 24 hours if they're not engaging with your bot. While stopping people from abusing Messenger messages is the right thing, how to you keep your user's engaged?

I always recommend that you should consider Facebook Messenger as one of your marketing channels but you always need a multi-channel marketing approach. That's why it's useful to ask your chatbot users for their email address and add them to an email list in your Email Service Provider (ESP) like Mailchimp or ConvertKit. Messenger allows you to ask for an email address using one of its quick replies but if the email isn't available, or the user chooses to type it in, how do we make sure we've got valid data?

Validating an email using Chatfuel

Let's take a look at how we can validate a given user email using Chatfuel as our chatbot platform. Chatfuel allows you to ask for a user's email using it's "Save User Email" element. Adding this to your chatbot flow allows you to send a quick reply to your and ask for an email address which is then stored in a Chatfuel user attribute. If you pass this to your server side API using the JSON API plugin, you can't validate it on the server to determine if it's valid.

Watch The Video

Take a look at how I build this email validation using Glitch and a Node.js Express web server together with Chatfuel's JSON API.

https://www.youtube.com/watch?v=r4h4DD1DNE8

How to validate an email address?

Using a regular expression

Validating email addresses correctly can be tricky. One way is to use something called a regular expression. This is a sequence of characters that define a search pattern that you attempt to match against. You'll find hundreds of different email regular expressions on the internet which give you various queries to validate against. However, they often come with caveats that they make a trade-off between speed and accuracy. You can use one of these to match the given email address with a specific pattern but you may incorrectly accept or reject some email addresses. The official regular expression which matches all email addresses is ridiculously big. I can't imagine attempting to debug this if it doesn't work as expected in your programming language. While a regular expression will confirm if your email matches a specific pattern, we can't confirm whether we can send an email to it.

Send the user an email

A nicer way is to determine if you can send an email to the given email address. This works really well for verifying an email address when signing up for an online service, but it's not that easy when you need to validate an email in a chatbot flow. Flows in Messenger or WhatsApp can't wait for your user to check their email and click on a link.

Checking the MX record

However, we can do something better by checking that the email has a valid domain to send email to. If we find a domain name in the given email address, we can check the DNS (Domain Name Server) records for the domain. These records can tell us the IP address on which the domain is hosted. They can also be used to see if email can be sent to the domain. For this we can check if the domain has an MX (Mail eXchange) record. If it does, we can send email to it.

Writing an API service to check for a valid email domain

Checking for an MX record does take some time so first we can verify that the email address has the format of an email address. For this you can check for the presence of an @ symbol. If it exists then the user has at least attempted to give us something which is similar to an email address! If you retrieve the rest of the string after the @ symbol, this represents the domain name. In my video below, I use Node.js and the standard dns library to determine if an MX record exists for that domain. If it does, then you should be able to send email to that address and we can consider it valid.

Here's some example code to retrieve an email address and validate it.

const dns = require('dns');

// The dns library will return:
// ENOTFOUND - if no records exist at all for the domain
// ENODATA - if no MX record was found for the domain,
// or if the user data was invalid
const hasMxRecordError = error => (
    error && (error.code === 'ENOTFOUND' || error.code === 'ENODATA')
);

// A promise-based method to determine if the
// MX record exists for the given email
const checkValidMxRecord = email => (
    new Promise((resolve, reject) => {
        // Determine if email is in the correct format
        const isEmail = email.includes('@');
        if (!isEmail) {
            return reject(
                new Error('Invalid email address does not contain @ symbol')
            );
        }

        // Split the email address and destructure to find the domain
        // TODO: You could also validate the username if you wanted to
        const [username, domain] = email.split('@');

        // Read the DNS records and see if an MX record exists for the domain
        return dns.resolveMx(domain, (error, addresses) => {
            // Check if any errors occurred
            if (hasMxRecordError(error)) {
                return reject(new Error('Email has invalid MX record'));
            }
    
            // You may not need the records but we can return them in case
            return resolve(addresses);
        });
    }));

To make my APIs more robust, I always return user attributes back to Chatfuel to inform it whether an API has succeeded or not. This allows us to avoid hard coding any messages in the API and gives back control to Chatfuel, or whichever chatbot platform you use, and use that to display the correct user messaging. It's a good way to inform the user that they've potentially typed in their email incorrectly. You can also use this validation check to gate your content to only allow access to people who supply their email address.

Of course, this method has a flaw in that we can't confirm that the username is valid without sending an email to it. We'll find this out when we send an email to them from our mailing list but at least we got halfway there.

I hope you found this useful. All of the code is available on my Chatfuel demo Glitch project. Feel free to clone it and use it for your own APIs. If you spot any errors, or have any questions, then please [send me a message]/(/contact/). I love to hear from people and I'm always happy to answer your questions.

Lessons Learned From Running A Code Club

2017-04-01T00:00:00Z

Here in the UK we have an organisation called Code Club which aims to help all children learn to code. It's a network of volunteers and educators who run free coding clubs for 9 to 11 year olds. I have two children, aged 7 and 9, and I want to encourage them to learn more about computing and technology and not just watch gaming videos on YouTube. When my eldest son moved from his infant school to his junior school a couple of years ago, the teachers told me how ill prepared they were for the move from Information Communication Technology (ICT) to computing. This was due to ICT being more about how to use a computer rather than how to write computer code. Having over 20 years of professional programming experience, I thought that this would be a good time to get involved to help my son's school. Unfortunately, due to almost dying, it took a bit longer to help out thank expected but I finally started running a Code Club this year. It's harder work than I imagined but it's a fantastically rewarding experience. Here are a some lessons I have learned along the way which could help others.

Lessons Learned In Running My Code Club

My original plan for my Code Club was to teach one group of children for around 10 weeks. I decided to only run the club for the Y5 and Y6 pupils, ages 9 to 11, and my plan was to start with the basics and continue to build on the projects each week. When the school announced the Code Club it was over-subscribed with around 30 children enrollong. Instead, I ran the club for two 5 week periods with 15 students in each class.

Teaching is hard

My wife is an English teacher of over 17 years. She laughed at me when I returned home after the first lesson and exclaimed "that was hard work". Any teachers reading this will also be laughing at my naïvety. I blindly assumed that all children would listen and implement the coding lessons I'd prepared. They did. Sometimes.

Some children will listen. Others, not so much. There's noise, there are questions, somebody needs to go to the toilet, everyone needs to go to the toilet, somebody is showing the whole class their project and so on. It's great fun but it can get chaotic. There's a great article about embracing the chaos on the Code Club website that everyone should read. I quickly realised that allowing them to do their own thing was part of them building their confidence in computer programming.

Tip: Embrace the chaos. It may be hard initally if you're not an educator and aren't used to children in a classroom environment. It gets easier and much more fun pretty quickly though.

Class size

I had a class size of 15 children in each group. As a non-teacher, this felt like a good class size to be able to control. For the first two weeks of Code Club I was alone in teaching until another parent helper had received his DBS check. It is harder than I imagined to be able to teach a class and answer all of their questions with one teacher and 15 children. Once an additional pair of hands was on-board, it made teaching much easier and big thanks to Dan for helping every week.

Tip: Make teaching easier by having a ratio of at least 1 adult helper to 6 or 7 children.

A variety of programming experience

I chose to use Scratch for all of my projects. It's a good choice for drag-and-drop programming and introduces the children to the basics of computing in a simple environment. It avoids complicated syntax errors when using a language such as Python or JavaScript. Most of the children had used Scratch at school but there was a range of abilities. Some questioned the basics while others raced ahead adding new features to their projects. My Code Club is in the school library so I always introduce the lessons with a set of slides on the TV. I introduce the lesson and talk through the basics but then have a set of enhancements that the children could work towards if they have the time. This helped to set the goal for the slower students while challenging the quicker ones. I also shared my version of the game as a challenge to make their's better than mine.

Tip: Consider extra features that the children could add to the project. If you're not an experienced programmer why not write a list of ideas that the children could add to your resources, even if you're not sure how to do them.

Preparation of lessons is key

It's perhaps an obvious point to make, especially if you're a teacher, but you should always run through the Code Club resources a couple of nights before. This helped me to understand the Code Club approach to the problem which often differed to my own. As I mentioned above, I always create a "stretch goal" for the children to try and achieve. This could involve improving the graphics, adding start and end game screens, adding sound and any other fun ideas I could think of. This ended up as a great challenge for me too. Scratch is restrictive when you're used to writing complex applications or games but working out a Scratch-based way of thinking was great fun.

Tip: Run through the lesson the night before and you'll encounter any issues that the children will have before they do.

Using sound

I ran the "Rock Band" project on my first club. The sound of 15 computers blaring out sound effects can be tricky to manage, especially when the children find the music! Ask them to keep the noise down. If you're lucky they will. You may allow the use of headphones too.

Tip: Projects using sound can make for a noisy classroom!

Adding more graphics

Scratch provide the children with great graphics and sound effects but they want to make each project their own. For my initial Rock Band lesson, I also showed them a Pikachu version as I knew seeing a Pokemon would get them engaged. I'd put the graphics on a USB stick but formatted it on my Mac. My fatal mistake was not formatting it for a FAT32 file system so it didn't work and there was some disappointment. The other issue to consider is whether the computers have internet access and whether the school allow access. You'll find that they want to download graphics to personalise their projects so you'll need to decide if this is possible. Thankfully, most schools have good safety policies and internet filtering. If the children can use their own graphics, their projects become their own.

Show the children the costume editor in Scratch and they can start creating their own sprites or variations of the existing ones. They love doing this but be aware that they will spend the whole lesson drawing sprites if they can.

Tip: Discuss with your school what their internet policy is for the children. Allow them to get creative with their own graphics.

Rubber duck debugging

In software engineering we have a method of finding problems in our code known as rubber duck debugging. The idea is that in telling the problem to someone else, such as a rubber duck, you'll actually understand the problem and will explain the solution yourself. If you've never tried it before, please do. It works! The children sometimes couldn't work out why their code didn't work as expected. Instead of pointing out what was wrong, I'd ask them to explain what they've done. I'd ask questions about each line of Scratch code. Most of the children would instantly see the problem and could fix it themselves. Computer programming is all about failure. You have to get it wrong to fully understand how to correct it.

Tip: If children have problems, ask them questions about their code. Try not to immediately offer solutions. They'll often work it out themselves.

Schools can't afford the best technology

I use a MacBook Pro for work. I am a developer who needs high end technology to do my job. Schools are not the same. They have constant budgetary constraints. They need to decide where to spend their hard fought for money. It won't be on the best computers. My son's school had reasonably usable Windows laptops with the latest version of Scratch on them. We generally had no issue with them. However, I helped out with teaching some programming at my daughter's infant school and it is a different story. They had half a room of desktop PCs and around 10 or 12 notebook-style laptops. The desktop PCs were fine but the notebooks had tiny screens and missing keys. The Scratch interface didn't fit on the full screen meaning that children had to scroll up and down to add new script blocks. It was much harder for the children couldn't to do this.

Tip: Investigate the equipment the schools have before you start. You'll have to work with what you've got but it helps to understand the problems you may encounter before you start.

Show and tell

One of the best ideas I used was to have a "show and tell" at the end of the lesson. I didn't start this until the second week but it finishes off the lesson extremely well. The children like to show their own unique take on the projects. I was often amazed at the direction the children took and it made for interesting learning for me too. Show off their projects on the big screen or projector if you have one. Make sure all of the children watch as it encourages their creativity for next time. Remember though, if you have 15 in the class, it can take quite a while to share each project. I initially gave 10 minutes to do this but ended up needing 15 to 20 minutes in the future weeks.

Tip: Allow at least 15 minutes before the end of the lesson for the children to show their project to the rest of the class. They're proud of what they've done and it gives them a great sense of achievement in front of their classmates.

Make it fun!

Ultimately, you and the children both need to have fun. Some of the children told me "it was awesome - I can't wait for next week". It's been a pleasure to teach them all and I hope tht I've influenced some of them to eventually follow a path into software development. If you don't run a Code Club then why don't you try? The Code Club website allows you to search for clubs in the local area where you could help out. Or you could start one yourself. You don't have to be a computer expert to start one. The children will help you if you get stuck!

The projects for my club

If you're interested in following a similar structure to my club, here are the projects that we did. I've also listed some of the enhancements that I added and have given links to my versions so you can use and alter them.

Week 1 - Rockband

This project gives a great basic introduction to Scratch. It shows how to play sounds, change costumes based upon state and about user input. I expanded upon it with my Beatbox Pikachu version.

Week 2 - Ghost Catcher

This is a "whack-a-mole" style gme where you click on ghosts as they appear and disappear to score points. A great introduction to variables which record the score and display a timer. Enhancements could include varying the frequency, size and speed of ghosts and adding a high score. My version shows a complete game with a game over screen. You'll find my Ghostbusters game here.

Week 3 - Chatbot

Ths is probably my favourite project. You add code so that Scratch asks a question and the children use the response within their code. It gives a great introduction to computer logic, for example "if...then". I started simple but then asked them to create their own "artificial intelligence". My enhanced version is an AI which responds to simple commands such as "hide", "moonwalk", "basketball", "dab", "bottle flip" and my character responds accordingly. Bottle flipping was one of their favourites and everyone tried to recreate it! 😄 Here's my version on the Scratch website.

Week 4 - Clone Wars

This project was particularly enjoyed by the Y5 children in my second group. It's a space shoot-em-up with a Galaxians feel to it. In this you learn about cloning sprites for spawning bullets and enemy AI. This is an interesting project for explaining how I would experiment with variables to create a fun game. Show them how to change the speed of enemies, the time between bullets and what difference it makes to the game. My version adds a shield, lives and a game over screen so it's almost a real game. My version is on the Scratch website here.

Week 5 - Flappy Parrot

This final project was the one that the Y6 children were most excited about. Flappy Bird is an iconic mobile game for most of them and it was great to see their ideas - Flappy Taco anyone? It introduces more complex movement to the children in the form of gravity. It also shows the game developer trick of pretenting the bird moves forward by actually moving the pipes backwards. This was quite a mind-blowing trick for some of the children but great to see them understanding it. This is also a good project to show a full set of game states with. Try adding a start and end screen. Mine used the original Flappy Bird graphics and awarded a medal if you beat the high score. You can find my version of Flappy Bird here.

Simple Error Handling For Your Production Express Server

2017-02-15T00:00:00Z

Error handling is a big part of writing code and it's something that every developer should think about. When starting to write code, it's easy to overlook what happens when your code fails. As developers we like to assume we write perfect code. We never do though do we?! If you're writing a customer facing application then you should always try and gracefully catch any errors and return human-friendly error messages to give a better user experience. This is especially true if you have an application which deals with 3rd party APIs or other external dependencies, like a database server. In these cases, you can't guarantee that the services will be available and you should be ready to handle this.

In my current job I write a lot of server-side JavaScript and we have a few Express applications running with Node.js. While checking out Twitter this evening I saw an interesting question from top JavaScript afficianado Wes Bos. He asked an interesting question about error handling with Express servers:

Talk to me about your Express error handling strategy. How (not what package) do you handle app errors as well as model validation fails?
— Wes Bos (@wesbos) February 15, 2017

Our applications are used by millions of people each day so we have quite robust error handling for our Express servers. Hence this was a question I am qualified to answer. I'll explain our approach to error handling using a simple example with some basic Express middleware functions and error handlers. The code uses ES6 syntax and Promises so if you're not familiar with these, then check out Wes Bos' excellent ES6 course and read up on Promises. I've not shown the code for the service calls as it's not important to know what that code does. What is important is that we ensure that our Express application correctly handles the errors which may occur after we execute the function.

Setting Up A Simple Express Application

Firstly, we can set up a basic user-facing Express application. It'll expose a few routes for us:

/login - this will display the user login page on a GET request and allow us to POST user credentials to it
/user - this will display the user's data (whatever that may be)

The Express server and routing code could look something like this.

function startServer(port = 3000) {
    const app = express();

    // Display the user login page
    app.get('/login',
        showLoginMiddleware
    );

    // Allow the user to submit their credentials
    app.post('/login',
        validateLoginMiddleware,
        showLoginMiddleware
    );

    // Assume that the user is validated before we hit this
    app.get('/user', (request, response) => {
        return response
            .status(200)
            .render('pages/success');
    });

    // Mount 404 handler as penultimate middleware
    app.use(notFoundHandler);

    // Final middleware is our catch-all error handler
    app.use(defaultErrorHandler);

    return app.listen(port, () => {
        logger.info(`Server started on port ${port}`);
    });
}

You can see that I've added two error handlers and these are simple Express middleware functions. The first is notFoundHandler. This will handle any requests which are application isn't set up to handle, for example if somebody hits a URL which we haven't defined. The second is defaultErrorHandler. This is the most important middleware as it allows us to catch any application errors that we aren't sure what to do with. I'll explain how these are implemented shortly.

Handling Known Errors

The first question Wes asked was what error handling pattern should be used by an Express server in order to handle known developer errors, such as model validation failure or documented API failures. In the server setup above, we declared a GET and POST route for our /login URL so let's define some middleware functions for them.

The first is for the GET request and it simply renders our login page. We can assume that this is a Jade or Pug template which renders a form allowing the user to enter their username and password.

function showLoginMiddleware(request, response) {
    response.render('pages/login');
}

The next middleware is used by our POST route and retrieves the username and password properties from our request body. We then pass these to our authentication service which validates that we have a user we know about. Notice that it's promise based so we can nicely catch our errors but you can achieve the same with a callback if you're more familiar with them. If our authenticateUser service succeeds, then we simple redirect the application to our /user endpoint.

If the user's credentials are not valid then we know that our service throws a InvalidCredentialsError typed Error. This is because we either wrote the service or we know this from 3rd party API documentation. This typed error contains a message property which we can show to the user so it's now a simple case of remembering the error for use in our re-rendered page. In my example, I'm using the response.locals object to assign an error property which gets used in the subsequent page re-render. I call next() in order to call the next middleware and this just uses showLoginMiddleware again which renders the same login page but shows the error message if it exists.

function validateLoginMiddleware(request, response, next) {
    const {username, password} = request.body;

    // Authenticate the user credentials with our service
    return authenticateUser(username, password)
        .then(() => {
            // User credentials were correct
            response.redirect('/user');
        })
        .catch(InvalidCredentialsError, (loginError) => {
            // User credentials were incorrect
            // Set an error message
            response.locals.error = loginError.message;

            // Re-render the same page where we display our error message
            next();
        })
        .catch((unknownError) => {
            // Woah! Something bad happened but we don't know what
            next(unknownError);
        });
}

Note: I'm using Bluebird's typed promise catch method in the above example.

Using a Promise returned from our authentication service also allows us to add a final catch function to our middleware. But why do we need to add this final catch block? Well our authentication service needs to determine if our user credentials are correct so it could do this by checking with a 3rd party API, such as an OAuth service, or it could be using a database query. What happens if either of these resources is unavailable? They may have heavy load. They might just fail due to unexpected server downtime. Either of these could cause a request timeout or an unknown response. Our catch block can now handle this more gracefully by passing the error on to the generic Express error handling middleware.

Handling Unexpected Application Errors

Express allows us to define middleware functions which have access to the request, response and next middleware in the applications request-response cycle. It also gives us the ability to define a default middleware for our error handler. This makes it simple for us to handle two different general error conditions: an error handler for when we don't match any of our routes (i.e. a 404 handler) and a default error handler for all other Express errors.

To use a 404 error middleware, we just use the same (request, response, next) middleware function signature but we need to ensure it is the last normal middleware in the application. This means that it'll match any route that we don't already handle with the previous middleware. In the example below, I create a notFound middleware which displays a "page not found" template. I also add a logger warning. It may not be an error as such but if you're logging events in your application, and you should be in some form, then it's useful to see what resource the user is attempting to access.

function notFoundHandler(request, response) {
    logger.warn(
        'Unhandled resource',
        {
            statusCode: 404,
            error: 'Unknown resource',
            resource: request.originalUrl
        }
    );

    return response
        .status(404)
        .render('pages/notFound');
}

Finally, we have the special Express middleware which allows us to handle errors. In my user authentication middleware above, called validateLoginMiddleware, you'll notice that I catch any unknown errors and then call the next middleware but also pass an error to it i.e. next(uknownError). By calling the next middleware with an error parameter, it allows Express to bypass all of the other middleware functions for the current route and instead calls our special error middleware. In the example below, we log the error and also return a 500 error and specific error page. In this we can warn the user that we're having some problems without letting our application crash. This greatly improves our users experience even if there's nothing we can do about the error itself.

IMPORTANT: The Express error middleware has a specific signature with the parameters (error, request, response, next). If you're using a linter such as ESLint, it will complain that you have an unused parameter in next if you don't call it. If you remove it then the middleware will no longer be called via next(error) as it no longer matches. Ensure that you add an ignore rule for your linter but don't remove the unused parameter. I learnt this the hard way by fixing the lint warning and wondered why the error middleware wasn't called! You must also register this error handling middleware as the last middleware in the application.

// eslint-disable-next-line no-unused-vars
function defaultErrorHandler(error, request, response, next) {
    logger.error('Uncaught error', {statusCode: 500, error});

    return response
        .status(500)
        .render('pages/error');
}

Conclusion

Hopefully this has given you a useful insight how to properly use Express middleware for your error handling. In short, make sure you do the following:

add a standard middleware for any routes which you don't handle specifically - assign this to the Express application as the penultimate route and it will act as your 404 error handler
add an Express error middleware with the additional error parameter as your default error - assign this to the Express application as the last middleware after all other middleware functions have been registered and it will act as your default error handler
attempt to catch any other model or service errors by using Promise.catch functions or try/catch blocks to gracefully handle known errors.

Crafting High Quality Unit Tests: Tips and Best Practices

2016-11-13T00:00:00Z

One of the often asked questions about unit testing is "how do I ensure I'm writing good quality unit tests?" It's easy to start of down the path of writing unit tests for your code but you need to make sure you're writing dependable tests which are easy to maintain. In this article, I talk about the defining characteristics of solid unit testing and how to ensure that the quality is high.

What is a unit?

The first questions to ask is "what is a unit?" when talking about writing your unit tests. There are differing definitions of what a unit actually is, but it's often defined as "the smallest component that it makes sense to test". For object-oriented programming, this could be a whole class or an interface but, as with a functional programming approach, it could also be a single method or function. To test the unit you simply have to define your expectations of what it does, given any of the specified inputs. That's it! If you're the developer in charge of writing the code then the test defines what you know will happen. If you didn't write the code under test, you can write your tests based upon your expectations of what you think should happen. If either of these things change then you want your tests to fail. Your unit tests then define a solid contract with your code.

What are good qualities of unit tests?

Your unit tests should be the first line of defence for your project. As your tests are so important to your code quality, you should always attempt to write high quality unit tests. But what does that actually mean? Let's find out by looking at some of the features of good quality tests.

Human readable and understandable test names

As silly as this may sound, you should spend some time ensuring that your test names make sense. This means that they should show the developer reading them exactly what the code under test does. The tests should be written for you and your team and not for a computer. Make sure that they're easy to read and make sense as a sentence. An example could be "should add a new user to the database" or "should call the authentication API with the expected user credentials".

An advantage of writing good test names is that you should be able to scan through each test module and see what the feature requirements for the code. This suite of tests should then be treated as living documentation for your project. If your code's functionality changes then update your test names accordingly.

Isolation of dependencies

One of the most important feature of a unit test is that it's isolated from any other external influences. This ensures that you're only testing the code that you've written and aren't dependent on the state of other modules. In order to do this you should write test doubles to replace your production dependencies for testing purposes. These generally take the form of a stub or mock object which returns the data you specify in order to fulfil your test expectation.

Some examples of things that you should make sure that you isolate are:

Calls to services - these could be external 3rd party API calls, for example the Twitter API, or simply function calls to your own services.
Serialisation calls - avoid actually serialising your data to real data stores by stubbing these function calls or mocking them with fake objects.
State - ensure that any state of the system under test has pre-conditions which are initialised by you in your test setup. This could involve stubbing your configuration, initialising structures or objects with expected data or again mocking or stubbing external code dependencies.

Provide just one reason to fail

A unit test should only have one reason to fail so you should always avoid putting multiple expectations into one test. You want to know exactly why a test fails and you'll just make it harder if your test contains more than one assertion. By keeping it simple, and using only one test expectation, it allows you to quickly see why a test has failed. This will mean that you write more tests but it makes the code much easier to reason about. You'll also find that it guides you to write tests to cover more of your code.

Make them speedy

Your unit tests should execute as quickly as possibly. Each test should have a small scope and should have all of its dependencies stubbed. By avoiding executing production code, such as that which serialises data to a database or uses 3rd party API calls, you can ensure that the whole of your unit test suite runs in a matter of seconds. This gives the team the confidence to run the tests often and will result in much better code quality. You can see from the image above that in one of our projects we run 2193 tests in 18 seconds, approximately 122 tests per second. This total includes integration tests as well as unit tests but is still fast enough for us to include those in every test run.

Independent of environment

Your unit tests should always be independent of environment. This means that the tests suite passes all of the tests on your machine, your colleagues' machine and any continuous build server. You definitely want to avoid the "it works on my machine" problem.

Ensure that the unit tests execute the same on all environments by correctly stubbing or mocking any dependencies and setting any state or external data. Also make sure that there is no dependency on other tests. Each test should run in isolation and should not need the result of a previous test in order to execute correctly. In the same vein, make sure that a test doesn't have any side effects, for example adding data to a database. You need identical data inputs and state each time the test suite runs. If the tests don't have consistency, and tests begin to fail, your team will lose faith in unit testing and they will quickly be ignored.

Tests unit completely

While some tests are better than none, the best quality unit tests ensure that all of the code paths are covered. This is often done in combination with test-driven development practices which mean you never write any code without a corresponding test. Even if you write your tests after you've written your code, you should try and test every route through your code.

Another good practice is to attempt to cover all edge cases for your code if possible. You'll find that you sometimes forget some obscure inputs, or maybe receive input data that you didn't expect, but try and determine the full requirements of your code and write your tests accordingly. Even if you don't, it's still incredibly satisfying to find a new bug, add an edge case test for it that fails as you expect, and then to fix it while still ensuring that all of your previous tests still pass.

Ultimately, make unit testing easy

You should always aim to make your unit tests easy. They should be easy to run, either from a simple terminal command or from your task runner or dependency manager. They should be easy to maintain such that refactoring the internals of your code should result in little work to update your tests as your test expectations will still be met.

Simple questions to ask for each unit test

The best way to write high quality unit tests is to write the test with these questions in mind:

What are you testing?
What should the code you're testing do?
What is the actual output of the code under test?
What is the expected output of the code under test?

If you answer all of those questions, you'll find you quickly have a higher quality unit test suite with high code coverage and you and your team can have much better confidence when deploying to your production environment.

If you spot any errors or have and questions then please contact me.

How To Unit Test Express Routes

2016-11-06T00:00:00Z

I was swiping through my Twitter timeline the other night when I saw a really interesting tweet from Kent C Dodds, a great JavaScript developer who works at PayPal, and someone you should really follow on Twitter if you don't already. He asked the following question.

[POLL]: Is it reasonable to *unit* test express routes? Or is it better to do an *integration* test of your server (with supertest for ex.)?
— Kent C. Dodds (@kentcdodds) November 3, 2016

I thought the poll posed an interesting question as to whether you should test Express routes or not. As I'm quite an evangelist of test driven development at the BBC, and I like a challenge and love writing code, I thought I'd have a think about it. I like to aim for as much test coverage as possible but I'm pragmatic enough to understand that 100% code coverage doesn't necessarily mean you're always testing the right things.

Yes, you should unit test Express routes

My answer to the poll was "Yes, you should unit test Express routes", which I appreciate goes against the majority who said unit tests weren't reasonable. At the minimum, I think that you should definitely test Express routes with some solid integration testing. In our team, we use supertest for our HTTP expectations and also nock to mock responses from API calls and it works really well for testing full flows on Express routes. I'll be honest and say that we don't currently have full unit testing for the Express routes on the three main Node.js projects that our team work on, but having thought about it, I think there are some reasons why we probably should consider it.

Reasons you should unit test Express routes

Better code coverage

Firstly, it should give you better code coverage. Having 100% code coverage should never be the end goal of your testing efforts, but writing the best tests to assert your expectations of what your code should be doing is. I feel that aiming towards full coverage of your code means that you have complete confidence in it at the unit level, and you can reason that your modules should perform the functions you expect them to.

Explicit expectations of middleware used

Next, I think it helps that you're being explicit in the middleware you think you are calling for each route. It's always a good idea to split your middleware out into separate modules as this makes unit testing of each middleware much clearer by reading the test asserts and it makes writing the code much simpler. By subsequently adding tests for each route, you can also then assert that you're calling the right middleware. A good example of this would be to verify that you're adding your authentication layer to the expected routes. You want to know that you need to authenticate your user before they can see your administration, or per-user, routes and this is simple to do with an expectation that the authentication middleware is called on those specific routes.

Security best practices

Lastly, I've recently taken up the role as security champion of our team and have been working with Dinis Cruz, a developer and application security expert, to ensure that our applications are more secure and adhere to OWASP best practices. One of Dinis' suggestions is to write tests which assert the security expectations of your application. In this case, writing tests which assert that the specified routes apply your security middleware feel like an ideal use case for writing unit tests for Express routes in addition to your integration testing. In doing so it gives you much better confidence that you're writing secure code and that those specified routes are protected.

How to write Express route unit tests

So it's easy to say that we should unit test Express routes, but how do you do it in practice? As Kent said in his later tweets, there don't seem to be any good examples of how to do it so I've written a basic Express application and some corresponding tests in this project on GitHub which show you a good way to set these tests up. We'll be using mocha, chai, sinon, sinon-chai (for some syntactic sugar) and proxyquire.

Install the project as follows:

git clone git@github.com:MarcL/unit-test-express-routes.git
cd unit-test-express-routes
npm install

Now you should see a basic Express application in the src/launch.js module and the corresponding tests in the test/launch.test.js module. The application starts an Express server on port 7080 and exposes three example routes: /, /login and /dashboard. These are meant to mimic a homepage, a login page and a dashboard page which is only exposed after authentication.

import express from 'express';

import homepage from './middleware/homepage';
import login from './middleware/login';
import authenticate from './middleware/authenticate';
import dashboard from './middleware/dashboard';

function setupRoutes(app) {
    app.get('/', homepage);
    app.get('/login', login);
    app.get('/dashboard',
        authenticate,
        dashboard
    );
}

function start(port = 7080) {
    const app = express();

    setupRoutes(app);

    const server = app.listen(port, () => {
        console.log(`Server running on: ${port}`);
    });

    return server;
}

export {
    start
};

You can see from the code above, there are four example middlewares which have been split into their own modules. Splitting up your middleware like this is always a good idea as it allows you to separate the unit tests for the middeware from the route code. This makes the code much simpler and easy to read and allows you to write clean and understandable tests.

Splitting up your middleware like this is always a good idea as it allows you to separate the unit tests for the middeware from the route code.

We need to set up our tests so that we can stub the Express application that is created and then we can return a fake server that we can use in our test expectations. As each test needs this fake server to be initialised, I chose to do this using Mocha's beforeEach pre-condition block. As we don't need to return anything when we perform an app.get to set up our routes which respond to the GET request, we can use sinon spy. However, we want to return a fake HTTP server for the app.listen call, so we make this a sinon stub. We can then set up a fake Express server which is return via a sinon stub.

beforeEach(() => {
    // Initialise our spy and stub
    spyExpressGet = sinon.spy();
    stubExpressListen = sinon.stub();

    // Create fake express application with our spy and stub methods
    fakeExpress = {
        get: spyExpressGet,
        listen: stubExpressListen
    };

    // Return our fake express application when express() is called
    stubExpress = sinon.stub().returns(fakeExpress);

    ...
});

Next, we can set up a fake HTTP server which is what we'd expect app.listen to return. We do this so we can compare it in our test which checks that the app is listening as we expect.

// We never use the fake HTTP server but we want to compare it
const fakeHttpServer = {};

// app.listen returns a fake HttpServer
stubExpressListen.returns(fakeHttpServer);

It's easy to spy on each of the middlewares used for our routes by creating sinon spies. We don't need to test any of these middlewares here, so we don't need to use a stub. You should test each middleware separately from this route code.

spyHomepage = sinon.spy();
spyLogin = sinon.spy();
spyAuthenticate = sinon.spy();
spyDashboard = sinon.spy();

We now use proxyquire to override the module dependencies. This will allow us to inject our fake Express server and to spy on all of the middleware. All of these spies and stubs will be used in our test expectations. The proxyquire initialisation looks complex, but we're simply stubbing the call to app.express so that it returns our stubExpress and making sure that the calls to import or require the middleware modules in the real code will now be using our spies.

// Use proxyquire to stub required modules and return
// our spies so we can check assertions
server = proxyquire('../../src/launch', {
    express: stubExpress,
    './middleware/homepage' : {default: spyHomepage},
    './middleware/login': {default: spyLogin},
    './middleware/authenticate': {default: spyAuthenticate},
    './middleware/dashboard': {default: spyDashboard}
});

The hard part has now been done as we've set up our server before each test. We can now inspect the routes and make expectations on them so let's take a look at our tests for that.

Express route tests

Assertion that our server is listening

First, we can write tests to check that our server is returning the expected HTTP server and runs on either the default port or a port we pass when initialising. We have stubbed app.listen to return a fake HTTP server so we can check that our call to server.start() returns it correctly. We can also confirm that it is called with either the default port of 7080, or another port that we pass through to the function call. The expectations are simple and readable so our fellow developers know exactly what the code should do. Note, for simplicity I'm using the general sinon.match.func matcher to match any function for the second parameter to app.listen.

    it('should return expected http server', () => {
        const returnedServer = server.start();
        expect(returnedServer).to.eql(fakeHttpServer);
    });

    it('should listen on default port 7080', () => {
        server.start();
        stubExpressListen.should.have.been.calledWithExactly(7080, sinon.match.func)
    });

    it('should listen on expected port if passed', () => {
        const expectedPort = 8888;
        server.start(expectedPort);
        stubExpressListen.should.have.been.calledWithExactly(expectedPort, sinon.match.func)
    });

We can then check each route using our spyExpressGet which spies on app.get for our routes. This is a simple case of confirming that each call sets the expected route string and the corresponding middleware(s).

it('should setup default route', () => {
    server.start();
    spyExpressGet.should.have.been.calledWithExactly('/', spyHomepage);
});

it('should setup login route', () => {
    server.start();
    spyExpressGet.should.have.been.calledWithExactly('/login', spyLogin);
});

it('should setup dashboard route', () => {
    server.start();
    spyExpressGet.should.have.been.calledWithExactly(
        '/dashboard',
        spyAuthenticate,
        spyDashboard
    );
});

We use sinon's calledWithExactly, albeit using the syntactic sugar of sinon-chai, to assert that we're setting the exact middleware for each route. This now gives us complete confidence that each route is only using the middleware that it should.

Notice that I've only stubbed the GET requests in my example project. If you wanted to validate the other routes which use other HTTP verbs then you should create a spy or stub for it and pass it into the fake Express application fakeExpress. You can then assert your expectations that app.post, app.put or app.delete are called with the expected route string and middleware.

tl;dr

Hopefully this was an easy-to-read explanation of how to create unit tests for Express routes. In case there is too much text to read above, my opinion is that, yes, I think you can easily unit test your Express routes and you probably should. You should do the following if you're planning to do so:

Use proxyquire to override dependencies so that you can stub/spy your express application and middleware functions
Use sinon to stub your middleware, which you should have already extracted to their own modules
Use these stubs to assert your expectations that the Express application routes use the correct middleware
These tests are an additional layer of confidence and are especially useful for security concerns
You should always have additional integration tests which test the full flow of the routes

If you've got any questions, need any clarification or have any comments on this then please contact me or tweet at me.

Halloween Sound Effects Using Spotify And Raspberry Pi

2016-11-04T00:00:00Z

For Halloween back in 2015, I had an unused Raspberry Pi and decided that I should make a Halloween project in order to learn electronics again, and to scare the local children as they came trick or treating. I connected the Raspberry Pi GPIO pins to a breadboard with a big red dome push button. I hid the Pi and electronics inside a shoe box painted black, and made it look like a spooky spider as you can see in the picture below.

Our Raspberry Pi powered Halloween spider. Press the big red button if you dare! LEDs to be added tonight. #halloween #spider #raspberrypi #spooky

A photo posted by Marc Littlemore (@marclittlemore) on Oct 30, 2015 at 10:33am PDT

Unfortunately, this year I've had a major operation so wasn't feeling up to recreating or bettering my Raspberry Pi project from last year. Instead, I decided to simplify it in order to play random spooky sound effects via the Raspberry Pi and combine it with some spooky ambient music from Spotify. This is a simple project to set up so read on if you want to scare your local trick or treaters!

What You'll Need

A Raspberry Pi (v2/3 or Zero) and SD card with Raspbian installed
A PC or Mac with a Spotify account
An audio mixer (or an input into your Mac or PC for software mixing)
A set of speakers
Some audio cables (probably 3.5mm but it will depend on your setup)

The first thing you'll need to do is to install Node.js on the Rasbperry Pi, assuming that you haven't already. You'll also need git installed for for cloning the project from GitHub, but this should come with the standard Rasbian install.

Next you'll need your PC or Mac up and running and you'll want Spotify on it. You can use a Spotify free account but bear in mind that you'll have adverts popping up which might not scare the neighbours as much as the spooky music will. Alternatively, you can use your own playlist of haunting music or find another one on SoundCloud or YouTube. Again, just watch out for adverts playing or you'll ruin the effect.

You'll need a set of speakers attached to your computer in some way. I run the output of my PC to an amplifier and speakers so I can boost the volume, but you can just use a small pair of computer speakers straight from the computer. Just make sure they're loud enough for people to be able to hear them. I kept my speakers indoors and put them by open windows but if you can run your speakers outside, and they're waterproof, then that'll work too.

Finally, you'll need something to mix the two audio sources together. I've spent nearly 30 years DJing so I had my Stanton DJ mixer to hand and plugged both the PC and the Raspberry Pi into it using RCA cables. This allowed me to alter the volumes of the two input sources and the final volume of the output to my amplifier. If you don't have an audio mixer, you can easily achieve the same thing by using a software mixer on your PC or Mac. Simply attach the output of the Raspberry Pi to the microphone input of your computer using a 3.5mm RCA cable and start the audio mixer in your settings or preferences panel. Note that the output of the Raspberry Pi seems to be really quiet in comparison to the output from the PC. You may be able to change this but I didn't investigate it too much. It just means that you'll need to make the Pi audio a lot higher than that from the PC using your hardware or software mixer.

Install Node.js on your Raspberry Pi

As I love working with Node.js and JavaScript, I decided that this was an easy way to get a project up and running. I installed Node when I first got the Raspberry Pi back in April 2015 so I have version 0.10 by following this guide. However, at the time of writing, I think the easiest way to install v4.2.1 is by following the instructions on the Node Arm GitHub project which does the following:

wget http://node-arm.herokuapp.com/node_latest_armhf.deb
sudo dpkg -i node_latest_armhf.deb

You may need to update the gcc compilers too, dependent on what version of Raspbian you're using, but follow the Node Arm guide if you need to do that.

As Linux doesn't have the best out-of-the-box audio support, you'll also have to install the Advanced Linux Sound Architecture (ALSA) library to allow easier playback of MP3 files. One of the NPM packages used is node-speaker and it relies on this library to play audio files. It should be a simple installation using apt-get on the Raspberry Pi as follows:

sudo apt-get install libasound2-dev

Install the code

Next you'll need to install my Halloween Pi project in order to play the random sounds. As mentioned above, the original project used a button and some LEDs last year so the master branch of the Halloweenpi GitHub project contains that code that uses the GPIOs to send signals from the buttons and to the LEDs. However, I've added a branch which just plays random sound effects in the random-play-sounds branch. Just clone the project from this specific branch and install all of the dependencies:

git clone -b random-play-sounds git@github.com:MarcL/halloweenpi.git
cd halloweenpi
npm install

After installation you should have the project and all of the samples necessary. Run the project and you should start hearing sound effects at intervals of between 15 and 40 seconds. I found this to be about right so that you actually get to hear them without them being too repetitive.

npm start

Now you need to also play the spooky ambient music from a Spotify playlist. I created this Spotify Playlist of spooky Halloween noises which you can use. It's nearly 7 hours long so it should be enough for playing throughout the night but feel free to make your own too. You'll have to use your audio mixer to determine the correct balance between the ambient music from Spotify and the one-shot sound effects triggered by the Raspberry Pi. As I mentioned earlier, the Raspberry Pi output seems to be incredibly quiet so I probably had my inputs at around 20% of the PC output mixed with 100% of the Pi output.

All you need to do now is leave this running and position your speakers in a place, and at a volume, where they can be heard by your visitors. A lot of children and their parents commented about how great this sounded so I was pleased with the eventual outcome.

Here's a video of it running in my study, the speakers are hidden behind the curtains with the windows open so it could be heard from the street.

https://www.youtube.com/watch?v=77df-OO72U0

Updating the sound effects

If you want to update the sound effects that are being played then it's really simple. First, you need to add the MP3 files to the /assets directory. Secondly, you need to update the config file which points at them. I chose to be explicit in what sound effects I wanted playing rather than just playing every file it finds in the assets directory. Just update the /config/config.js array of paths to the sound effects and add your new effects, or remove any that you don't want.

module.exports = {
    soundFiles: [
        './assets/cuckoo-clock.mp3',
        './assets/foghorn-doorbell.mp3',
        './assets/ghost-scream.mp3',
        './assets/evil-laugh.mp3',
        './assets/female-scream.mp3',
        './assets/funeral-bells.mp3',
        './assets/howling-wolf.mp3',
        './assets/cat-scream.mp3',
        './assets/woman-shrill-scream.mp3',
        './assets/thriller-laugh.mp3',
        './assets/glados-hello-friend.mp3',
        './assets/glados-hello-where-are-you.mp3',
        './assets/glados-i-see-you.mp3',
        './assets/glados-laugh.mp3',
        './assets/glados-come-over-here.mp3',
        './assets/dalek-exterminate.mp3'
       	]
};

I hope this guide helps you to easily set up your own Halloween Raspberry Pi spooky sound player. If you have any questions then post them below and I'll answer them, or raise and bugs as GitHub issues and I can get them fixed.

How To Find Bugs Using Git Bisect With This Easy Guide

2016-09-29T00:00:00Z

A source control system is an essential tool if you're writing software, even for your own side projects. Although I've used many version control software over the past 20+ years of development, I've only been using git for the past 3 or 4 years. However it's quickly become my version control of choice due to its ability to quickly create branches, and its distributed nature. It's easy to master the basics of git but, as with most pieces of software, there's a lot of power available if you choose to go beyond them.

One git command that you might not have seen or used before is git bisect, which is something I've only recently started using too. It's a great tool for quickly tracking down bugs in your project so let's take a look at how we can use it.

A wild bug appears

I'm currently working on some high-traffic Node.js projects for work and, as I normally do at the start of a new task, I ran git pull to ensure I had the latest code from our master branch, and then ran npm install to update any dependencies. I ran the tests to check everything was ok but an odd thing happened: our test suite hung when it finished although all of the tests still passed. WHAT? Time to investigate.

I had no idea why this bug had crept into the project, nor when it first appeared in the master branch. My initial thought was that it was the last pull request that had been merged, so I checked out the previous merge instead but again, the tests were still hanging. This felt like a perfect opportunity to try out git bisect for the first time in order to quickly work out which commit had caused the error.

What does the "git bisect" command do?

Bisect performs a binary search to quickly find out the revision that caused a bug in your project's history. You do this by telling it which was the "bad" commit, the one which contains the bug, and which was a "good" commit, one in which you think the bug wasn't there. You can then use git bisect to pick a commit which is halfway between the two endpoints and you determine whether that one is bad or good. This can be repeated by choosing the revision between these two points until you determine the specific commit which caused the bug. It's then up to you to look at the code to see what has changed and is probably the cause.

The "git bisect" process

The first thing we need to do is tell git that we're about to start using bisect by using the git bisect start command. Before you do this, you should make sure you've committed or stashed any local changes to avoid losing any work as it will checkout each revision as we attempt to find the version containing the bug.

git bisect start

The next step is to determine when the bug first occurred and when we think that the repository was ok, and tell git bisect which commit versions these were. This allows it to determine the range of revisions to search through.

# Use this if the bug occurs in the current HEAD revision
git bisect bad HEAD

# But if the bug occurs in a specific revision then use its SHA hash
git bisect bad 62c5fa0

For the good revision, you can choose an arbitrary place in the past but the more commits that have happened between the two endpoints, the more potential revisions git bisect will have to check. This will depend on the speed of development in your project but it's probably wise to not choose a revision in time that was not too far away.

# Choose your good revision
git bisect good da5e24d

Now that you've chosen your two endpoints, git bisect will choose a revision that's in the middle of these two and tell you how many more revisions it will need to check from this point.

> git bisect good da5e24d
Bisecting: 54 revisions left to test after this (roughly 6 steps)
[dac781864e62c49b279c122c42d447ed26ad16e2] Merge pull request #100 from features/my-feature

It's now a process of elimination to determine which specific commit caused the bug. You do this by running your test suite or using your project to see if the bug is still present in the current revision. At this point git bisect wants to know if this commit is good or bad so you answer git bisect good or git bisect bad.

> git bisect bad
Bisecting: 27 revisions left to test after this (roughly 5 steps)
[4f455b6e42487657e0492305f025dd82021735d5] some commit message from this revision

In my case, the gulp task was still hanging so I answered that the commit was bad. The revisions are then split into two again and you need to continue to answer whether the revisions are good or bad until you reach the last revision and find the guilty commit. At this point git bisect will show you which commit caused the problem and you can check the code in this revision to determine what has changed.

02ce44a7491e9b0151169647115f9a073513e0ce is the first bad commit
commit 02ce44a7491e9b0151169647115f9a073513e0ce
Author: some-author <some-author@gmail.com>
Date:   Mon Sep 12 13:59:20 2016 +0100

    Added some code that might break! :D

Once you've determined the cause, you'll want to finish your current git bisect session. For this you just use the reset command and it will reset to your HEAD to where you were before you started.

git bisect reset

Make sure you reset otherwise you will find your local working directory in a strange state, having checked out out a previous revision of the project.

Pro-tip: Bisect automation

Manually finding the bugs in your code is helped by using git bisect but you can make things even faster by automating the process. You simply have to pass a script to be executed by git bisect run. Doing this will run the script against each bisected commit in order to determine which one causes the script to fail.

# Set up our start and endpoints to check against
git bisect start
git bisect bad 62c5fa0
git bisect good da5e24d

# Run our gulp task which executes our
# test suite for each bisected commit
git bisect run gulp test

You'll have to ensure the script that you're using fails with a non-zero return value for this to work. For the code that fails in my above discussion, our test suite hung so it wasn't as easy to use automation here.

Conclusion: Use "git bisect"

As you can see, using git bisect is an easy way to quickly determine where defects have crept into a git project. You should start using it as its an effective tool to understand. For more information have a read of this great git SCM article on debugging with git. Also, take some time to look at the git bisect documentation too.

Marc Littlemore

Rescuing My Bricked M1 MacBook Pro

The MDM brick wall

Understanding Mac security

The solution: DFU restore

Booting into DFU mode

The IPSW secret

Victory was mine

What I learned

Recovering My Apple Podcast Feed

Discovering podcasts

👋 Hello Feedburner

Moving to my own feed

Redirecting subscribers

The importance of GUIDs

Claiming ownership in Apple Podcasts

Step 1: Request to claim your podcast

Step 2: Add the verification tag to your feed

Step 3: Complete verification

Own your content

Writing Again

Life challenges

Social media sucks

Writing for myself again

Inspiration for writing more

What's next?

Automate Your Static Site Rebuilds with Netlify Scheduled Functions

Let's start automating

Using Netlify to rebuild your site

Create a Netlify build hook

Add the build hook to your environment variables

Adding a Netlify rebuild scheduled function

Testing the scheduled function locally

Updating your site footer with the latest date

Deploying the scheduled function

Example use cases

That's all folks!

Easily Create Gravatar Images With Eleventy

What is a Gravatar?

Adding an Eleventy shortcode

Creating a Gravatar URL

Adding a size

Adding a default image

Conclusion

Improving Your Workflow with JavaScript Testing: Best Practices

Starting to test your code

Adding tests to an existing codebase

How many tests to write

Conclusion

Streamline Your Workflow: Automate GitLab Releases with Semantic-Release

What is a release?

Conventional Commits

Semantic Release

Determine when a release should be created

GitLab pipelines

Advantages of automatic releases

Gotchas

Conclusion

Lessons learned from 30 days of writing

It took more than 30 days

I enjoy writing

People enjoyed reading the articles

Finding my cadence

Ideas didn't always flow

Taking notes makes writing easy

Technical articles take a long time

Would I do it again?

You've got this: become a confident developer

Ask all the questions

Understand the team's applications

Communicate with your team and find their superpowers

Document all the things

Set solid learning objective

Help others once you know

Be excellent to yourself and each other

Never get too comfortable

Feeling stuck

Find opportunties for growth

Work out your goals

Time to move on