A Lightning Tour of the Google Testing Blog

Google Testing Blog has a huge wealth of information for software engineers looking for resources on testing. Testing culture at Google is one that I really look up to and I am grateful to the engineers who shared these resources. With that said, keep in mind that some of the content can be outdated and/or opinionated. Testing is a widespread topic and there can be different, sometimes conflicting, answers to any given problem.

Below you will find a a categorized list of key concepts I found in the blog with links to the particular post in which it was found. The idea here is brevity. If an item catches your eye, take a look at the linked blog post(s) for more info.

I plan to keep this updated as more posts are added.

Test Reliability

• Smaller tests run faster, are less flakey, and isolate failures. As a general rule of thumb, favor a pyramid-shaped composition of 70% small, 20% medium, and 10% large tests.¹
• Test binary size and memory usage, including third-party testing tools, have a strong correlation on whether a test is flaky.²
• Beware of using UI testing to verify underlying functionality. In these cases, it is cheaper and more reliable to have smaller tests that break closer to the source of the problem.^{3 4}
• Hermetic servers add speed and reliability into end-to-end tests. A environment is considered Hermetic if it can run an entire system under test on a single machine with fake network connections and database implementations.⁵
• Filter out flakey tests by rerunning failing tests. If a test fails three times in a row, consider it a real failure.⁶
• Build testability into the product. For example, a real-time system can rely on a fake clock instead of a hardware clock. Processes can spawn other processes attached to a debugger with debugging flags.⁷

Code Quality

• Releasing often gives teams an incentive to automate testing and reduce coupling with the rest of the system.⁸
• When a team provides fakes and writes tests for them, they become clients of their own software. Experiencing the perspective of the client gives the team an incentive to make their API easier to use.⁸
• While writing the header first encourages consideration for the interface, writing tests first encourages consideration for how the interface will be used.⁹
• Good code quality is taught, not enforced. Create a culture that teaches code quality through code review, pair programming, and mentoring.⁷
• Don't obsess over coverage numbers. Use code coverage reports to identify areas that are not covered and human judgement over whether to cover it. For example, frequently changed code should be covered.¹⁰
• Consider investing in mutation testing, a automated tool that "mutates" code and expects tests to fail. Tuned properly, it can help find oversights in tests that are worth fixing. ¹¹

Productivity

• Automation is costly. Automate only the tests that you find yourself running often to reliably catch regressions on features with business value.³
• Speed up the feedback loop between test engineers and development engineers. Share the same space, tools, daily stand-ups, and design discussions.³
• Effective automation depends on test design. Good test design is built from a solid foundation of manual tests.¹²
• If a test plan isn't worth bothering to update, it isn't worth creating in the first place. A quick brainstorming session will suffice.¹³
• Use formatting tools, like clang-format, to improve readability.¹⁴

Infrastructure

• Remove the detective work of tracking down bad changes by investing in a pre-submit system that runs automated tests against the commit before it reaches the depot.^{15 6}
• Don't fall behind on updating third party dependencies. Update them quickly by setting up CI system with dependencies pinned at head.¹⁶
• Avoid making more than one branch by putting risky new changes behind feature flags.¹⁴
• Constantly look for opportunities to make the build system faster. Reduce the amount of code being compiled, replace tools with faster counterparts, and use distributed build systems.¹⁷
• Release early and release often. Services and websites can deploy rapidly. A good target for client projects is Chromes six week cycle.¹⁷

Metrics & Logging

• Pre- vs post-production defect ratio and a breakdown of defects by component or functional area help identify holes in test.¹⁸
• Premature performance optimization makes bad code. Develop in a clean, maintainable and extensible manner first, and then let benchmarks drive performance optimizations.¹⁹
• Remove unwanted noise by logging with conditional verbosity. Log all levels to a logging queue. If a transaction completes successfully, discard the unimportant levels.²⁰
• Use two sets of logging levels, one for production builds and one for development builds.²⁰
• Trace the time spent on every significant processing step. Measuring is the only way to detect performance issues or make claims about performance.²⁰
• Write automated performance tests for performance sensitive parts of your product.¹⁷