Skip to content
Trevor Behnke logoTrevor Behnke

Image Comparison Bot — Automated QA Visual Regression Testing

Proprietary, Headless Browser QA Automation for Dashboard State Validation

Lead Designer, Architect & Engineer•2022
JavaScriptPythonPuppeteerPixelmatchGoogle Cloud Platform
Image Comparison Bot — Automated QA Visual Regression Testing cover
Image comparison bot—overview of results
Image Comparison Bot — Automated QA Visual Regression Testing screenshot 1
Example Dashboard Error Diff
Image Comparison Bot — Automated QA Visual Regression Testing screenshot 2
Raw Error States Output
Background
Crow Insight needed a way to ensure quality across complex data dashboards with tens of thousands of possible state combinations. Manual QA was infeasible, so a proprietary automation tool was required to reliably test every state. The Image Comparison Bot was designed to capture screenshots from two different interaction paths (URL ingestion vs. click-based navigation) and compare them pixel by pixel, flagging any differences above a set threshold. Results were compiled into detailed reports that identified errors and the exact state sequences that caused them.
Contribution
  • Refactored and redesigned the project from scratch, replacing a half-built Python implementation with a more suitable JavaScript architecture
  • Solely responsible for designing, architecting, and engineering the solution, with feedback from colleagues along the way
  • Implemented automated state exploration (randomized and exhaustive) to capture and compare tens of thousands of dashboard views
  • Built reporting functionality to flag pixel differences, identify error thresholds, and log reproducible sequences for manual verification
  • Deployed the system to a GCP VM, enabling automated test runs around the clock as needed for comprehensive QA coverage
  • Delivered a proprietary tool that significantly reduced QA overhead, improved precision, and flagged issues before production release
Learned
  • How to design a custom QA automation system tailored to unique client needs
  • The importance of choosing the right language and tools for the job — JavaScript and Puppeteer proved more effective than the original Python approach
  • Deepened understanding of pixel-diff algorithms and how to balance sensitivity thresholds to reduce false positives
  • Gained experience building reproducible, report-driven workflows that make complex errors easier to confirm and resolve
  • Learned how automation can multiply QA impact — replacing an impossible manual task with a scalable, reliable, and precise solution