Apr 25, 2024

Your bug reports suck

The problem

Bug reports need videos and pictures. Alright, that’s enough for today. See you next week!

I’m kidding obviously. Anyway, I’ve written quite a bit about bugs in software before, both from a usability perspective

Joel Moser

Through Your User's Eyes

Yep, we're talking about usability again

and well, telling you that bugs are good.

Joel Moser

You want bugs in your app

How to find out what features you don't need

I started my career in software testing, specifically automation, but you end up doing a fair bit of manual testing too. Regardless of the type of testing, you always run into a bug at some point or another. The bug will exist somewhere on a spectrum of severity and impact. There is usually a framework to determine where it resides in priority. If you cannot access the app, that’s obviously a P0 (drop everything and fix it now). If it is affecting only a few customers and is just a minor feature like a button being double clicked is creating duplicate records, it might be a P2 (fix in the next release) or P3 (maybe fix in an upcoming release or two, or possibly not at all). If it is affecting a lot of customers but has a fairly simple workaround, that might be a P1 (fix before starting anything new, but don’t drop what you are working on). I’ve put some hints to what the priority numbers mean, but every company has their own system of categorization and when bugs should be fixed.

Sometimes a product manager (or even CEO) will override that and something that would usually be considered a P3 becomes a P0. It can be annoying, but as long as this doesn’t happen frequently, it’s okay. It could be really important for a prospect demo that ultimately closes a big deal. Yeah, the text size was a little small, but the little things can be just as important as the big things (and how you care for small problems may honestly be an indicator of how you will handle big problems). You never know what is important to a prospective customer.

Making your bug reports better

So what actually makes for a good bug report? Well, if your testing team is sophisticated enough to have a test case database, then it is as easy as “this test case broke”. That won’t catch everything, and honestly I think exploratory testing and chaos testing often reveal more real world bugs. Just “running through the test cases” is a good sanity check, but that only covers the known issues. What about the unknown ones? How do you go about discovering those? Sometimes they will reveal themselves as a part of another test case (if you are paying enough attention), but that’s not always the case. Some weird edge cases can hide in plain sight. With that being said, I’ve been in the startup world for awhile now. Test case databases are non-existent, and honestly for good reason. Things are changing so rapidly that having hard set test cases would be a waste of time. Sure, having some sanity checks is good, but maintaining a list of every possible test case or bug is ludicrous. Just covering “is the app up and can I navigate around?” is half the battle.

Every engineer and team will have their own preferences, but from my perspective, here is what I find most helpful: videos. They share and convey so much information. Just a simple screenshot doesn’t tell you the steps a user took to get to that point. Word descriptions are just someone’s interpretation of the problem. Videos prevent incorrect assumptions about the issue. You get to see exactly what a user did to cause or encounter the bug. Sometimes you cannot immediately reproduce the bug either, so you have to go off someone’s description. It’s rough, but it happens all the time. Timing issues and race conditions are the bane of the software developer’s existence (well and context switching but that’s a separate blog post). Videos cut through that (to an extent). You can watch the bug get reproduced time and again and analyze different parts of the application. Now, that doesn’t always work, but it’s much better than words and screenshots.

One problem that often comes up with screenshots and videos is the user doesn’t have the browser console or network tab open. That’s the first place developers usually look for more information if the bug isn’t immediately apparent (or even if it is, just as a sanity check that we aren’t overlooking something else). Having those tools open make it that much easier (and faster) for a developer to identify a problem and fix it. And identifying the problem is the usually hardest part.

I can fix any problem I can find.

— Some developer I read/heard from years ago and completely forget where it came from, but it’s burned into my brain

That doesn’t mean the solution will be easy, but identifying what the actual problem is most often the hardest part. I really wish I remembered where I got that quote from. At the time, I remember thinking that was a bit arrogant, but over the years, I realized the developer was right. The operative word in that sentence is “find” not “fix”. I’ve never encountered a problem I couldn’t fix. That doesn’t mean I didn’t get help in coming up with a good fix, but it can be fixed. I absolutely have discovered problems that I’ve spent 10x the amount of time figuring out what the problem was than coming up with a solution for it.

So is there a tool for this?

All this being said, I recently stumbled across jam.dev. I’m probably late to the party, but this seems like an awesome tool that truly aims to make filing good bug reports like a walk in the park. I have not personally used it, so you can take this with a grain of salt. But looking at the features and documentation, it seems to offer the full gambit of what a bug reporter needs. It allows you to take screenshots and videos, and also stores the browser console and network tab information. This is insanely valuable. It even has integrations into many popular software and project management tools like Jira, GitHub, Sentry, etc. I think I’m going to be pitching this to the other engineering leaders on my team going forward. I truly think it will improve developers’ lives, and recently that’s become somewhat of a passion of mine. More on that soon.