{
  "feed": {
    "items": [
      {
        "title": "Code Health: Providing Context with Commit Messages and Bug Reports",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/lTnHFI_mRTg/code-health-providing-context-with.html",
        "pubDate": "2017-09-11T21:01:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<span id=\"docs-internal-guid-96cdc92b-72a3-298c-9f87-daaddf19a3ec\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">This is another post in our </span><a href=\"https://testing.googleblog.com/2017/04/code-health-googles-internal-code.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Code Health</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> series. A version of this post originally appeared in Google bathrooms worldwide as a Google </span><a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Testing on the Toilet</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> episode. You can download a </span><a href=\"https://docs.google.com/document/d/1Ub4xrBA6cyJjJ2bgFL7_VynlvJf423p7UBKIWBRMsnU/edit?usp=sharing\" rel=\"nofollow\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">printer-friendly version</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> to display in your office.</span></span></span><br /><span style=\"background-color: white; color: #222222; font-family: &quot;arial&quot;; font-size: 9.5pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"><br /></span><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\">By Chris Lewis</span></span><br /><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><br /></span></span><span id=\"docs-internal-guid-7555d281-72b2-27f4-085f-2b65456fdd8c\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"vertical-align: baseline; white-space: pre-wrap;\">You are caught in a trap. Mechanisms whirl around you, but they make no sense. You desperately search the room and find the builder's original plans! The description of the work order that implemented the trap reads, </span><span style=\"font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">\"Miscellaneous fixes.\" </span><span style=\"vertical-align: baseline; white-space: pre-wrap;\">Oh dear.</span></span></span><br /><span style=\"vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><br /></span></span><span id=\"docs-internal-guid-7555d281-72b5-7098-346a-238d0acb67ad\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"vertical-align: baseline; white-space: pre-wrap;\">Reading other engineers' code can sometimes feel like an archaeology expedition, full of weird and wonderful statements that are hard to decipher. </span><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Code is always written with a purpose, but sometimes that purpose is not clear in the code itself.</span></span><span style=\"color: #980000; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"vertical-align: baseline; white-space: pre-wrap;\">You can address this knowledge gap by documenting the context that explains why a change was needed. Code comments provide context, but comments alone sometimes can’t provide enough.</span></span></span><br /><span style=\"vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><br /></span></span><span id=\"docs-internal-guid-7555d281-72b5-f9ae-74c8-cbd8e1c6176b\"><span style=\"vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\">There are two key ways to indicate context:</span></span></span><br /><span id=\"docs-internal-guid-7555d281-72b6-2664-d850-c25d5e29af28\"><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\">Commit Messages</span></span></span><br /><br /><ul><li><span style=\"white-space: pre-wrap;\"><b><span id=\"docs-internal-guid-7555d281-72b6-8d7f-0b46-1fa8c7bec68e\" style=\"font-weight: normal;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"font-weight: 700; vertical-align: baseline;\"><span style=\"color: purple;\">A commit message is one of the easiest, most discoverable</span></span><span style=\"color: #980000; font-weight: 700; vertical-align: baseline;\"> </span><span style=\"vertical-align: baseline;\">means of providing context. When you encounter lines of code that may be unclear, checking the commit message which introduced the code is a great way to gain more insight into what the code is meant to do.</span></span></span></b></span></li><li><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"white-space: pre-wrap;\"><b><span style=\"font-weight: normal;\"><span style=\"vertical-align: baseline;\"><span id=\"docs-internal-guid-7555d281-72b7-48b7-d0de-75b127ea1f1d\"><span style=\"color: purple;\"><span style=\"font-weight: 700; vertical-align: baseline;\">Write the first line of the commit message so it stands alone</span><span style=\"vertical-align: baseline;\">,</span></span><span style=\"color: #980000; vertical-align: baseline;\"> </span><span style=\"vertical-align: baseline;\">as tools like GitHub will display this line in commit listing pages. Stand-alone first lines allow you to </span><span style=\"font-weight: 700; vertical-align: baseline;\"><span style=\"color: purple;\">skim through code history much faster</span></span><span style=\"vertical-align: baseline;\">, quickly building up your understanding of how a source file evolved over time. </span></span></span></span></b></span><span id=\"docs-internal-guid-7555d281-72b7-d872-f4f6-c11333f0669d\"><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Example:</span><span style=\"color: #980000;\"> </span></span></span>&nbsp;</span> <div style=\"overflow-x: auto;\"><table class=\"my-bordered-table\" style=\"width: 613px;\"><tbody><tr><td style=\"background-color: #cfe2f3; vertical-align: top; width: 607px;\"><pre style=\"background-color: #cfe2f3; border: 0px; color: black; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-top: 0px;\"><span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">Add Frobber to the list of available widgets.<br /><br />This allows consumers to easily discover the new Frobber widget and<br />add it to their application.</span></pre></td></tr></tbody></table></div>  </li></ul><div><span id=\"docs-internal-guid-7555d281-72b9-5f39-e1a5-055ba3614359\"><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\">Bug Reports</span></span></span></div><div><ul><li><span style=\"white-space: pre-wrap;\"><b><span id=\"docs-internal-guid-7555d281-72b9-bf82-7122-7cdf7b359c2a\" style=\"font-weight: normal;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"font-weight: 700; vertical-align: baseline;\"><span style=\"color: purple;\">You can use a bug report to track the entire story of a bug/feature/refactoring,</span></span><span style=\"vertical-align: baseline;\"> adding context such as the original problem description, the design discussions between the team, and the commits that are used to solve the problem. This lets you easily see all related commits in one place, and allows others to easily keep track of the status of a particular problem.</span></span></span></b></span></li><li><span style=\"white-space: pre-wrap;\"><b><span style=\"font-weight: normal;\"><span style=\"vertical-align: baseline;\"><span id=\"docs-internal-guid-7555d281-72ba-4b5b-1a53-71238d76987b\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"font-weight: 700; vertical-align: baseline;\"><span style=\"color: purple;\">Most commits should reference a bug report</span></span><span style=\"vertical-align: baseline;\"><span style=\"color: purple;\">.</span> Standalone commits (e.g. one-time cleanups or other small unplanned changes) don't need their own bug report, though, since they often contain all their context within the description and the source changes.</span></span></span></span></span></b></span></li></ul><div><span id=\"docs-internal-guid-7555d281-72ba-7a34-c41b-ae90ca3524c5\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"color: #980000; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">I</span><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">nformative commit messages and bug reports go hand-in-hand,</span></span><span style=\"vertical-align: baseline; white-space: pre-wrap;\"> providing context from different perspectives.</span><span style=\"color: #980000; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"vertical-align: baseline; white-space: pre-wrap;\">Keep in mind that such</span><span style=\"color: #980000; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">context can be useful even to yourself</span></span><span style=\"vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">,</span> providing an easy reminder about the work you did last week, last quarter, or even last year. Future you will thank past you!</span></span></span></div></div><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=lTnHFI_mRTg:V6DyzhI_ZR4:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=lTnHFI_mRTg:V6DyzhI_ZR4:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=lTnHFI_mRTg:V6DyzhI_ZR4:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/lTnHFI_mRTg\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "This is another post in our Code Health series. A version of this post originally appeared in Google bathrooms worldwide as a Google Testing on the Toilet episode. You can download a printer-friendly version to display in your office.By Chris LewisYou are caught in a trap. Mechanisms whirl around you, but they make no sense. You desperately search the room and find the builder's original plans! The description of the work order that implemented the trap reads, \"Miscellaneous fixes.\" Oh dear.Reading other engineers' code can sometimes feel like an archaeology expedition, full of weird and wonderful statements that are hard to decipher. Code is always written with a purpose, but sometimes that purpose is not clear in the code itself. You can address this knowledge gap by documenting the context that explains why a change was needed. Code comments provide context, but comments alone sometimes can’t provide enough.There are two key ways to indicate context:Commit MessagesA commit message is one of the easiest, most discoverable means of providing context. When you encounter lines of code that may be unclear, checking the commit message which introduced the code is a great way to gain more insight into what the code is meant to do.Write the first line of the commit message so it stands alone, as tools like GitHub will display this line in commit listing pages. Stand-alone first lines allow you to skim through code history much faster, quickly building up your understanding of how a source file evolved over time. Example: &nbsp; Add Frobber to the list of available widgets.This allows consumers to easily discover the new Frobber widget andadd it to their application.  Bug ReportsYou can use a bug report to track the entire story of a bug/feature/refactoring, adding context such as the original problem description, the design discussions between the team, and the commits that are used to solve the problem. This lets you easily see all related commits in one place, and allows others to easily keep track of the status of a particular problem.Most commits should reference a bug report. Standalone commits (e.g. one-time cleanups or other small unplanned changes) don't need their own bug report, though, since they often contain all their context within the description and the source changes.Informative commit messages and bug reports go hand-in-hand, providing context from different perspectives. Keep in mind that such context can be useful even to yourself, providing an easy reminder about the work you did last week, last quarter, or even last year. Future you will thank past you!",
        "id": "tag:blogger.com,1999:blog-15045980.post-8046216467923860328",
        "isoDate": "2017-09-11T21:01:00.000Z"
      },
      {
        "title": "Code Health: Eliminate YAGNI Smells",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/zQwgFkJ8VWQ/code-health-eliminate-yagni-smells.html",
        "pubDate": "2017-08-14T18:45:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<span id=\"docs-internal-guid-2e5aeb06-e1f7-06a6-663b-93c217e243b7\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">This is another post in our </span><a href=\"https://testing.googleblog.com/2017/04/code-health-googles-internal-code.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Code Health</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> series. A version of this post originally appeared in Google bathrooms worldwide as a Google </span><a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Testing on the Toilet</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> episode. You can download a </span><a href=\"https://docs.google.com/document/d/1ZDlgl15g5PHBZssnW_EHktx1gvfud0inVwQ4fmXQrL8/edit?usp=sharing\" rel=\"nofollow\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">printer-friendly version</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> to display in your office.</span></span></span><br /><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"><br /></span></span><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\">By </span></span><span style=\"color: #222222; font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"white-space: pre-wrap;\">Marc Eaddy</span></span><br /><span style=\"color: #222222; font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"white-space: pre-wrap;\"><br /></span></span><span id=\"docs-internal-guid-6eaa5349-e1fc-8117-bf44-2d81283878fb\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">The majority of software development costs are due to maintenance.</span><span style=\"color: purple;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">One way to reduce maintenance costs is to </span><span style=\"background-color: white; font-style: italic; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">implement something only when you actually need it</span><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">, a.k.a. the </span><span style=\"background-color: white; font-style: italic; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">“You Aren't Gonna Need It”</span><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"> (YAGNI) design principle.</span></span><span style=\"background-color: white; color: #980000; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">How do you spot unnecessary code? Follow your nose!</span></span></span><br /><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"><br /></span></span><span id=\"docs-internal-guid-6eaa5349-e1fd-1c9f-0067-350ede75aa2d\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">A </span><span style=\"background-color: white; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">code smell</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> is a code pattern that usually indicates a design flaw. For example, creating a base class or interface with only one subclass may indicate a speculation</span><span style=\"background-color: white; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">that more subclasses will be needed in the future. Instead, </span><span style=\"color: purple;\"><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">practice </span><span style=\"background-color: white; font-style: italic; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">incremental development and design</span></span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">: don't add the second subclass until it is actually needed.</span></span></span><br /><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"><br /></span></span><span id=\"docs-internal-guid-6eaa5349-e1fd-bee8-4106-0eb06fd3b57e\"><span style=\"color: purple; font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">The following C++ code has many YAGNI smells</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">:</span></span></span><br /><div style=\"overflow-x: auto;\"><table class=\"my-bordered-table\" style=\"width: 613px;\"><tbody><tr><td style=\"background-color: #cfe2f3; vertical-align: top; width: 607px;\"><pre style=\"background-color: #cfe2f3; border: 0px; color: black; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-top: 0px;\"><span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">class Mammal { ...<br />  virtual Status Sleep(bool hibernate) = 0;<br />};<br />class Human : public Mammal { ...<br />  virtual Status Sleep(bool hibernate) {<br />    age += hibernate ? kSevenMonths : kSevenHours;<br />    return OK;<br />  }<br />};</span></pre></td></tr></tbody></table></div><br /><span id=\"docs-internal-guid-6eaa5349-e206-b27d-31ae-50bf28cffdf8\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">Maintainers are burdened with understanding, documenting, and testing both classes when only one is really needed. Code must handle the case when </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">hibernate</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> is true, even though all callers pass </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">false</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">, as well as the case when</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">Sleep</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">returns an error, even though that never happens. This results in unnecessary code that never executes.</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"color: purple;\"><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\">Eliminating those smells simplifies the code</span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">:</span></span></span></span><br /><div><br /><div style=\"overflow-x: auto;\"><table class=\"my-bordered-table\" style=\"width: 613px;\"><tbody><tr><td style=\"background-color: #d9ead3; vertical-align: top; width: 607px;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin: 0px; padding-bottom: 0px; padding-left: 0px; padding-top: 0px;\"><span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">class Human { ...<br />  void Sleep() { age += kSevenHours; }<br />};</span></pre></td></tr></tbody></table></div></div><div><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\"><br /></span></span></span><span id=\"docs-internal-guid-6eaa5349-e20a-75df-ea9f-425d9248d2b8\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Here are some other YAGNI smells</span></span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">:</span></span></span></div><div><ul><li><span style=\"white-space: pre-wrap;\"><span style=\"font-family: Times, Times New Roman, serif;\">Code that has never been executed other than by tests (a.k.a. code that is dead on arrival)</span></span></li><li><span style=\"white-space: pre-wrap;\"><span style=\"font-family: Times, Times New Roman, serif;\">Classes designed to be subclassed (have virtual methods and/or protected members) that are not actually subclassed</span></span></li><li><span style=\"white-space: pre-wrap;\"><span style=\"font-family: Times, Times New Roman, serif;\">Public or protected methods or fields that could be private</span></span></li><li><span style=\"font-family: Times, Times New Roman, serif; white-space: pre-wrap;\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\">Parameters, variables, or flags that always have the same valu</span>e</span></li></ul><div><span id=\"docs-internal-guid-6eaa5349-e20c-160d-42e8-867621960b75\" style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">Thankfully, </span><span style=\"background-color: white; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">YAGNI smells, and code smells in general, are often easy to spot by looking for simple patterns and are easy to eliminate using simple refactorings</span></span><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">.</span></span></div></div><div><span id=\"docs-internal-guid-6eaa5349-e20c-8481-d93e-91a798eaf368\"></span><br /><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 10pt;\"><span id=\"docs-internal-guid-6eaa5349-e20c-8481-d93e-91a798eaf368\"><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; vertical-align: baseline; white-space: pre-wrap;\">Are you thinking of adding code that won't be used today? Trust me, </span><span style=\"background-color: white; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">you aren't gonna need it!</span></span></span></div><span id=\"docs-internal-guid-6eaa5349-e20c-8481-d93e-91a798eaf368\"></span><br /><div><span id=\"docs-internal-guid-6eaa5349-e20c-8481-d93e-91a798eaf368\"><span style=\"background-color: white; font-size: 13pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"><br /></span></span></div><span id=\"docs-internal-guid-6eaa5349-e20c-8481-d93e-91a798eaf368\"></span></div><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=zQwgFkJ8VWQ:aZMaLSvghkM:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=zQwgFkJ8VWQ:aZMaLSvghkM:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=zQwgFkJ8VWQ:aZMaLSvghkM:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/zQwgFkJ8VWQ\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "This is another post in our Code Health series. A version of this post originally appeared in Google bathrooms worldwide as a Google Testing on the Toilet episode. You can download a printer-friendly version to display in your office.By Marc EaddyThe majority of software development costs are due to maintenance. One way to reduce maintenance costs is to implement something only when you actually need it, a.k.a. the “You Aren't Gonna Need It” (YAGNI) design principle. How do you spot unnecessary code? Follow your nose!A code smell is a code pattern that usually indicates a design flaw. For example, creating a base class or interface with only one subclass may indicate a speculation that more subclasses will be needed in the future. Instead, practice incremental development and design: don't add the second subclass until it is actually needed.The following C++ code has many YAGNI smells:class Mammal { ...  virtual Status Sleep(bool hibernate) = 0;};class Human : public Mammal { ...  virtual Status Sleep(bool hibernate) {    age += hibernate ? kSevenMonths : kSevenHours;    return OK;  }};Maintainers are burdened with understanding, documenting, and testing both classes when only one is really needed. Code must handle the case when hibernate is true, even though all callers pass false, as well as the case when Sleep returns an error, even though that never happens. This results in unnecessary code that never executes. Eliminating those smells simplifies the code:class Human { ...  void Sleep() { age += kSevenHours; }};Here are some other YAGNI smells:Code that has never been executed other than by tests (a.k.a. code that is dead on arrival)Classes designed to be subclassed (have virtual methods and/or protected members) that are not actually subclassedPublic or protected methods or fields that could be privateParameters, variables, or flags that always have the same valueThankfully, YAGNI smells, and code smells in general, are often easy to spot by looking for simple patterns and are easy to eliminate using simple refactorings.Are you thinking of adding code that won't be used today? Trust me, you aren't gonna need it!",
        "id": "tag:blogger.com,1999:blog-15045980.post-1547713588878071210",
        "isoDate": "2017-08-14T18:45:00.000Z"
      },
      {
        "title": "Code Health: To Comment or Not to Comment?",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/grfVWQaCJak/code-health-to-comment-or-not-to-comment.html",
        "pubDate": "2017-07-17T20:54:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<span id=\"docs-internal-guid-f12e4fdc-518c-da84-eea8-b6a4be0fe25f\"><i><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\">This is another post in our </span><a href=\"https://testing.googleblog.com/2017/04/code-health-googles-internal-code.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; vertical-align: baseline; white-space: pre-wrap;\">Code Health</span></a><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\"> series. A version of this post originally appeared in Google bathrooms worldwide as a Google </span><a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; vertical-align: baseline; white-space: pre-wrap;\">Testing on the Toilet</span></a><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\"> episode. You can download a </span><a href=\"https://docs.google.com/document/d/1ZgxLPQfKAfdwU-07sYplTWWnlzk6MaAxPMB1HpuOYdM/edit?usp=sharing\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; vertical-align: baseline; white-space: pre-wrap;\">printer-friendly version</span></a><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\"> to display in your office. </span></span></i></span><br /><span style=\"background-color: white; color: #222222; font-family: &quot;arial&quot;; font-size: 9.5pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"><br /></span><span style=\"font-family: &quot;times&quot; , &quot;times new roman&quot; , serif;\"><span style=\"background-color: white; color: #222222; vertical-align: baseline; white-space: pre-wrap;\">By </span><span style=\"color: #222222;\"><span style=\"white-space: pre-wrap;\">Dori Reuveni and Kevin Bourrillion</span></span></span><br /><span style=\"color: #222222; font-family: &quot;arial&quot;;\"><span style=\"font-size: 12.6667px; white-space: pre-wrap;\"><i><br /></i></span></span><span id=\"docs-internal-guid-80f4df1e-5195-7562-ccca-33b91dee58da\"><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\">While reading code, </span><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">often there is nothing more helpful than a well-placed comment.</span></span><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"> However, </span><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">comments are not always good.</span></span><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"> Sometimes the need for a comment can be a sign that the code should be refactored.</span></span><br /><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"><br /></span><span id=\"docs-internal-guid-80f4df1e-5199-e564-da1d-a76a06c36dc4\"><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Use a comment when it is infeasible to make your code self-explanatory.</span></span><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"> If you think you need a comment to explain what a piece of code does, first try one of the following:</span></span><br /><ul style=\"padding-right:0;\"><div dir=\"ltr\" style=\"margin-left: 36pt;\"></div><li style=\"overflow-x: auto;\"><span style=\"white-space: pre-wrap;\"><span id=\"docs-internal-guid-80f4df1e-519a-69a9-6808-90426cacd365\"><span style=\"vertical-align: baseline;\"><span style=\"font-size: 13pt;\">Introduce an explaining variable.</span><table class=\"my-bordered-table\">  <tbody><tr>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Subtract discount from price.</span><br />finalPrice = (numItems * itemPrice)<br />    - min(5, numItems) * itemPrice * 0.1;</pre></td>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 50%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">price = numItems * itemPrice;<br />discount =<br />    min(5, numItems) * itemPrice * 0.1;<br />finalPrice = price - discount;</pre></td>  </tr></tbody></table></span></span></span></li><li style=\"overflow-x: auto;\"><span style=\"white-space: pre-wrap;\"><span style=\"vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-519f-4335-f8b0-d5a33dcb9da9\"><span style=\"font-size: 13pt; vertical-align: baseline;\">Extract a method. <table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Filter offensive words.</span><br />for (String word : words) { ... }</pre></td>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 50%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">filterOffensiveWords(words);</pre></td>  </tr></tbody></table></span></span></span></span></li><li style=\"overflow-x: auto;\"><span style=\"white-space: pre-wrap;\"><span style=\"vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a0-170b-6ea9-9f5c5276098e\"><span style=\"vertical-align: baseline;\"><span style=\"font-size: 13pt;\">Use a more descriptive identifier name. </span><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">int width = ...; <span style=\"font-weight: bold;\">// Width in pixels.<br /></span></pre></td>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 50%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">int widthInPixels = ...;</pre></td>  </tr></tbody></table><span style=\"font-size: 13pt;\"></span></span></span></span></span></li><li style=\"overflow-x: auto;\"><span style=\"white-space: pre-wrap;\"><span style=\"vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a0-2ea1-3c01-7139b77b3d83\"><span style=\"vertical-align: baseline;\"><span style=\"font-size: 13pt;\">Add a check in case your code has assumptions.</span><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Safe since height is always &gt; 0.</span><br />return width / height;</pre></td>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 50%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">checkArgument(height &gt; 0);<br />return width / height;<br /></pre></td>  </tr></tbody></table><span style=\"font-size: 13pt;\"></span></span></span></span></span></li></ul><span id=\"docs-internal-guid-80f4df1e-51a0-8452-eab1-2cd24d48de26\"><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">There are cases where a comment can be helpful:</span></span></span><br /><div><ul style=\"padding-right: 0;\"><li style=\"overflow-x: auto;\"><span style=\"color: #980000;\"><span style=\"font-size: 17.3333px; white-space: pre-wrap;\"><b><span id=\"docs-internal-guid-80f4df1e-51a0-b549-18ae-18357c1b342e\" style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\">Reveal your intent: explain </span><span style=\"color: black; font-size: 13pt; font-style: italic; vertical-align: baseline;\">why</span><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"> the code does something (as opposed to </span><span style=\"color: black; font-size: 13pt; font-style: italic; vertical-align: baseline;\">what</span><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"> it does). <table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 100%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Compute once because it’s expensive.</span></pre></td>  </tr></tbody></table></span></span></b></span></span></li><li style=\"overflow-x: auto;\"><span style=\"color: #980000;\"><span style=\"font-size: 17.3333px; white-space: pre-wrap;\"><b><span style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a0-df5a-d7f5-d2b802cd8c8a\"><span style=\"font-size: 13pt; vertical-align: baseline;\">Protect a well-meaning future editor from mistakenly “fixing” your code.<b style=\"color: #980000; font-size: 17.3333px;\"><span id=\"docs-internal-guid-80f4df1e-51a0-b549-18ae-18357c1b342e\" style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 100%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Create a new Foo instance because Foo is not thread-safe.</span></pre></td>  </tr></tbody></table></span></span></b></span></span></span></span></b></span></span></li><li style=\"overflow-x: auto;\"><span style=\"color: #980000;\"><span style=\"font-size: 17.3333px; white-space: pre-wrap;\"><b><span style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><span style=\"font-size: 13pt; vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a1-2bf0-a976-6eff6b40c4c9\"><span style=\"font-size: 13pt; vertical-align: baseline;\">Clarification: a question that came up during code review or that readers of the code might have.<b style=\"color: #980000; font-size: 17.3333px;\"><span style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a0-df5a-d7f5-d2b802cd8c8a\"><span style=\"font-size: 13pt; vertical-align: baseline;\"><b style=\"color: #980000; font-size: 17.3333px;\"><span id=\"docs-internal-guid-80f4df1e-51a0-b549-18ae-18357c1b342e\" style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 100%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Note that order matters because...</span></pre></td>  </tr></tbody></table></span></span></b></span></span></span></span></b></span></span></span></span></span></b></span></span></li><li style=\"overflow-x: auto;\"><span style=\"vertical-align: baseline;\"><span style=\"vertical-align: baseline;\"><span style=\"font-size: 17.3333px; vertical-align: baseline; white-space: pre-wrap;\">Explain your rationale for what looks like a bad software engineering practice.<b style=\"color: #980000; font-size: 17.3333px;\"><span style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><span style=\"font-size: 13pt; vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a1-2bf0-a976-6eff6b40c4c9\"><span style=\"font-size: 13pt; vertical-align: baseline;\"><b style=\"color: #980000; font-size: 17.3333px;\"><span style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><span id=\"docs-internal-guid-80f4df1e-51a0-df5a-d7f5-d2b802cd8c8a\"><span style=\"font-size: 13pt; vertical-align: baseline;\"><b style=\"color: #980000; font-size: 17.3333px;\"><span id=\"docs-internal-guid-80f4df1e-51a0-b549-18ae-18357c1b342e\" style=\"font-weight: normal;\"><span style=\"color: black; font-size: 13pt; vertical-align: baseline;\"><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #d9ead3; vertical-align: top; width: 100%;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\">@SuppressWarnings(\"unchecked\") <span style=\"font-weight: bold;\">// The cast is safe because...</span></pre></td>  </tr></tbody></table></span></span></b></span></span></span></span></b></span></span></span></span></span></b></span></span></span></li></ul><span id=\"docs-internal-guid-80f4df1e-51a1-dc5e-5e8a-e7db89713222\"><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\">On the other hand, </span><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">avoid comments that just repeat what the code does</span></span><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">.</span> These are just noise:</span></span><br /><div style=\"overflow-x: auto;\"><table class=\"my-bordered-table\" style=\"width: 100%;\">  <tbody><tr>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Get all users.</span><br />userService.getAllUsers();</pre></td>    <td style=\"background-color: #f4cccc; vertical-align: top; width: 50%;\"><pre style=\"background-color: #f4cccc; border: 0px; color: black; margin-bottom: 0; margin-top: 0; padding-bottom: 0; padding-top: 0; padding-left: 0;\"><span style=\"font-weight: bold;\">// Check if the name is empty.</span><br />if (name.isEmpty()) { ... }</pre></td>  </tr></tbody></table></div></div><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=grfVWQaCJak:0--XfkbfS5s:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=grfVWQaCJak:0--XfkbfS5s:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=grfVWQaCJak:0--XfkbfS5s:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/grfVWQaCJak\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "This is another post in our Code Health series. A version of this post originally appeared in Google bathrooms worldwide as a Google Testing on the Toilet episode. You can download a printer-friendly version to display in your office. By Dori Reuveni and Kevin BourrillionWhile reading code, often there is nothing more helpful than a well-placed comment. However, comments are not always good. Sometimes the need for a comment can be a sign that the code should be refactored.Use a comment when it is infeasible to make your code self-explanatory. If you think you need a comment to explain what a piece of code does, first try one of the following:Introduce an explaining variable.      // Subtract discount from price.finalPrice = (numItems * itemPrice)    - min(5, numItems) * itemPrice * 0.1;    price = numItems * itemPrice;discount =    min(5, numItems) * itemPrice * 0.1;finalPrice = price - discount;  Extract a method.       // Filter offensive words.for (String word : words) { ... }    filterOffensiveWords(words);  Use a more descriptive identifier name.       int width = ...; // Width in pixels.    int widthInPixels = ...;  Add a check in case your code has assumptions.      // Safe since height is always > 0.return width / height;    checkArgument(height > 0);return width / height;  There are cases where a comment can be helpful:Reveal your intent: explain why the code does something (as opposed to what it does).       // Compute once because it’s expensive.  Protect a well-meaning future editor from mistakenly “fixing” your code.      // Create a new Foo instance because Foo is not thread-safe.  Clarification: a question that came up during code review or that readers of the code might have.      // Note that order matters because...  Explain your rationale for what looks like a bad software engineering practice.      @SuppressWarnings(\"unchecked\") // The cast is safe because...  On the other hand, avoid comments that just repeat what the code does. These are just noise:      // Get all users.userService.getAllUsers();    // Check if the name is empty.if (name.isEmpty()) { ... }",
        "id": "tag:blogger.com,1999:blog-15045980.post-5029073665609132906",
        "isoDate": "2017-07-17T20:54:00.000Z"
      },
      {
        "title": "Evolution of GTAC and Engineering Productivity",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/ptSfzdHRC4s/evolution-of-gtac-and-engineering.html",
        "pubDate": "2017-07-12T20:08:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<div dir=\"ltr\" style=\"line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">When Google first hosted GTAC in 2006, we didn’t know what to expect. We kicked off this conference with the intention to share our innovation in test automation, learn from others in the industry and connect with academia. Over the last decade we’ve had great participation and had the privilege to host GTAC in North America, Europe and Asia -- largely thanks to the many of you who spoke, participated and connected!</span></div><b id=\"docs-internal-guid-b2800243-380a-1c76-0e2a-b9be11aa8164\" style=\"font-weight: normal;\"><br /></b><div dir=\"ltr\" style=\"line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">In the recent months, we’ve been taking a hard look at the discipline of </span><a href=\"https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html\" style=\"text-decoration: none;\"><span style=\"background-color: transparent; color: #1155cc; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;\">Engineering Productivity</span></a><span style=\"background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> as a logical next step in the evolution of test automation. In that same vein, we’re going to rethink what an Engineering Productivity focused conference should look like today. &nbsp;As we pivot, we will be extending these changes to GTAC and because we expect changes in theme, content and format, we are canceling the upcoming event scheduled in London this November. We’ll be bringing the event back in 2018 with a fresh outlook and strategy. </span></div><b style=\"font-weight: normal;\"><br /></b><div dir=\"ltr\" style=\"line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">While we know this may be disappointing for many of the folks who were looking forward to GTAC, we’re excited to come back with a new format which will serve this conference well in today’s environment.</span></div><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ptSfzdHRC4s:5cACbMCS9fk:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ptSfzdHRC4s:5cACbMCS9fk:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=ptSfzdHRC4s:5cACbMCS9fk:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/ptSfzdHRC4s\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "When Google first hosted GTAC in 2006, we didn’t know what to expect. We kicked off this conference with the intention to share our innovation in test automation, learn from others in the industry and connect with academia. Over the last decade we’ve had great participation and had the privilege to host GTAC in North America, Europe and Asia -- largely thanks to the many of you who spoke, participated and connected!In the recent months, we’ve been taking a hard look at the discipline of Engineering Productivity as a logical next step in the evolution of test automation. In that same vein, we’re going to rethink what an Engineering Productivity focused conference should look like today. &nbsp;As we pivot, we will be extending these changes to GTAC and because we expect changes in theme, content and format, we are canceling the upcoming event scheduled in London this November. We’ll be bringing the event back in 2018 with a fresh outlook and strategy. While we know this may be disappointing for many of the folks who were looking forward to GTAC, we’re excited to come back with a new format which will serve this conference well in today’s environment.",
        "id": "tag:blogger.com,1999:blog-15045980.post-3343661551924942594",
        "isoDate": "2017-07-12T20:08:00.000Z"
      },
      {
        "title": "Code Health: Too Many Comments on Your Code Reviews?",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/5zk7lKKg5qo/code-health-too-many-comments-on-your.html",
        "pubDate": "2017-06-19T17:47:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<span id=\"docs-internal-guid-19e757bc-c15c-9682-a581-e80a1c1cb3fe\" style=\"font-family: inherit;\"><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">This is another post in our </span><a href=\"https://testing.googleblog.com/2017/04/code-health-googles-internal-code.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Code Health</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> series. A version of this post originally appeared in Google bathrooms worldwide as a Google </span><a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">Testing on the Toilet</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> episode. You can download a </span><a href=\"https://docs.google.com/document/d/1DnJl_Xt5xddfOU5JESq4Q5PvKKDMoDBxmrQFjColVpk/edit?usp=sharing\" style=\"text-decoration-line: none;\"><span style=\"background-color: white; color: #1155cc; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\">printer-friendly version</span></a><span style=\"background-color: white; color: #222222; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"> to display in your office. </span></span><br /><span style=\"background-color: white; color: #222222; font-size: 9.5pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"font-family: inherit;\"><br /></span></span><span style=\"font-family: inherit;\"><span style=\"color: #222222;\"><span style=\"background-color: white; white-space: pre-wrap;\">By </span></span><span style=\"color: #222222;\"><span style=\"white-space: pre-wrap;\">Tom O'Neill</span></span></span><br /><span style=\"font-family: inherit;\"><span style=\"color: #222222; font-family: inherit;\"><span style=\"white-space: pre-wrap;\"><br /></span></span></span><br /><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: inherit; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Code reviews can slow down an individual code change, but they’re also an opportunity to improve your code and learn from another intelligent, experienced engineer. How can you get the most out of them?</span></div><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Aim to get most of your changes approved in the first round of review, with only minor comments. </span><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">If your code reviews frequently require multiple rounds of comments, these tips can save you time.</span></span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span></span></div><span style=\"font-family: inherit;\"><span id=\"docs-internal-guid-baa04f06-c162-76ad-3d0e-c16973815eba\"></span><br /></span><br /><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Spend your reviewers’ time wisely—it’s a limited resource.</span></span><span style=\"background-color: transparent; color: #980000; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">If they’re catching issues that you could easily have caught yourself, you’re lowering the overall productivity of your team.</span></span></div><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: inherit; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Before you send out the code review:</span></div><ul style=\"margin-bottom: 0pt; margin-top: 0pt;\"><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Re-evaluate your code:</span></span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> Don’t just send the review out as soon as the tests pass. Step back and try to rethink the whole thing—can the design be cleaned up? Especially if it’s late in the day, see if a better approach occurs to you the next morning. Although this step might slow down an individual code change, it will result long-term in greater average throughput.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Consider an informal design discussion:</span></span><span style=\"background-color: transparent; color: #980000; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">If there’s something you’re not sure about, pair program, talk face-to-face, or send an early diff and ask for a “pre-review” of the overall design.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Self-review the change:</span></span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> Try to look at the code as critically as possible from the standpoint of someone who doesn’t know anything about it. Your code review tool can give you a radically different view of your code than the IDE. This can easily save you a round trip.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Make the diff easy to understand:</span></span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> Multiple changes at once make the code harder to review. When you self-review, look for simple changes that reduce the size of the diff. For example, save significant refactoring or formatting changes for another code review.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"font-size: 13pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Don’t hide important info in the submit message:</span></span><span style=\"font-size: 13pt; vertical-align: baseline; white-space: pre-wrap;\"> Put it in the code as well. Someone reading the code later is unlikely to look at the submit message.</span></span></div></li></ul><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"background-color: transparent; color: black; font-family: inherit; font-size: 13pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">When you’re addressing code review comments:</span></div><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span id=\"docs-internal-guid-baa04f06-c164-b2b4-fc66-88b0be942bfa\" style=\"font-family: inherit;\"></span></span></div><ul style=\"margin-bottom: 0pt; margin-top: 0pt;\"><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><span style=\"color: purple;\">Re-evaluate your code after addressing non-trivial comments:</span></span><span style=\"background-color: transparent; color: #980000; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Take a step back and really look at the code with fresh eyes. Once you’ve made one set of changes, you can often find additional improvements that are enabled or suggested by those changes. Just as with any refactoring, it may take several steps to reach the best design.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"color: purple;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Understand </span><span style=\"background-color: transparent; font-size: 13pt; font-style: italic; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">why</span><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> the reviewer made each comment:</span></span><span style=\"background-color: transparent; color: #980000; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">If you don’t understand the reasoning behind a comment, don’t just make the change—seek out the reviewer and learn something new.</span></span></div></li><li dir=\"ltr\" style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;\"><div dir=\"ltr\" style=\"line-height: 1.2; margin-bottom: 10pt; margin-top: 0pt;\"><span style=\"font-family: inherit;\"><span style=\"color: purple;\"><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Answer the reviewer’s questions in the </span><span style=\"background-color: transparent; font-size: 13pt; font-style: italic; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">code</span><span style=\"background-color: transparent; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">:</span></span><span style=\"background-color: transparent; color: #980000; font-size: 13pt; font-style: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"> </span><span style=\"background-color: transparent; color: black; font-size: 13pt; font-style: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\">Don’t just reply—make the code easier to understand (e.g., improve a variable name, change a boolean to an enum) or add a comment. Someone else is going to have the same question later on.</span></span></div></li></ul><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=5zk7lKKg5qo:m_ShSxZOQWI:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=5zk7lKKg5qo:m_ShSxZOQWI:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=5zk7lKKg5qo:m_ShSxZOQWI:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/5zk7lKKg5qo\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "This is another post in our Code Health series. A version of this post originally appeared in Google bathrooms worldwide as a Google Testing on the Toilet episode. You can download a printer-friendly version to display in your office. By Tom O'NeillCode reviews can slow down an individual code change, but they’re also an opportunity to improve your code and learn from another intelligent, experienced engineer. How can you get the most out of them?Aim to get most of your changes approved in the first round of review, with only minor comments. If your code reviews frequently require multiple rounds of comments, these tips can save you time. Spend your reviewers’ time wisely—it’s a limited resource. If they’re catching issues that you could easily have caught yourself, you’re lowering the overall productivity of your team.Before you send out the code review:Re-evaluate your code: Don’t just send the review out as soon as the tests pass. Step back and try to rethink the whole thing—can the design be cleaned up? Especially if it’s late in the day, see if a better approach occurs to you the next morning. Although this step might slow down an individual code change, it will result long-term in greater average throughput.Consider an informal design discussion: If there’s something you’re not sure about, pair program, talk face-to-face, or send an early diff and ask for a “pre-review” of the overall design.Self-review the change: Try to look at the code as critically as possible from the standpoint of someone who doesn’t know anything about it. Your code review tool can give you a radically different view of your code than the IDE. This can easily save you a round trip.Make the diff easy to understand: Multiple changes at once make the code harder to review. When you self-review, look for simple changes that reduce the size of the diff. For example, save significant refactoring or formatting changes for another code review.Don’t hide important info in the submit message: Put it in the code as well. Someone reading the code later is unlikely to look at the submit message.When you’re addressing code review comments:Re-evaluate your code after addressing non-trivial comments: Take a step back and really look at the code with fresh eyes. Once you’ve made one set of changes, you can often find additional improvements that are enabled or suggested by those changes. Just as with any refactoring, it may take several steps to reach the best design.Understand why the reviewer made each comment: If you don’t understand the reasoning behind a comment, don’t just make the change—seek out the reviewer and learn something new.Answer the reviewer’s questions in the code: Don’t just reply—make the code easier to understand (e.g., improve a variable name, change a boolean to an enum) or add a comment. Someone else is going to have the same question later on.",
        "id": "tag:blogger.com,1999:blog-15045980.post-841381449055719530",
        "isoDate": "2017-06-19T17:47:00.000Z"
      },
      {
        "title": "Code Health: Reduce Nesting, Reduce Complexity",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/21l7QzueXpc/code-health-reduce-nesting-reduce.html",
        "pubDate": "2017-06-05T17:01:00.001Z",
        "author": "Google Testing Bloggers",
        "content": "<i>This is another post in our <a href=\"https://testing.googleblog.com/2017/04/code-health-googles-internal-code.html\">Code Health</a> series. A version of this post originally appeared in Google bathrooms worldwide as a Google <a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\">Testing on the Toilet</a> episode. You can download a <a href=\"https://docs.google.com/document/d/12VE8zEDSp5s0WZSFY7eGn91WGbUE67ASvxZmmc-yFOM/edit?usp=sharing\">printer-friendly version</a> to display in your office.   </i><br /><br />By Elliott Karpilovsky <br /><br />Deeply nested code hurts readability and is error-prone.<strong> <span style=\"color: purple;\">Try spotting the bug</span></strong> in the two versions of this code: <br /><br /><table class=\"my-bordered-table\">  <tbody><tr>      <td style=\"text-align: center;\"><em>Code with too much nesting</em></td>      <td style=\"text-align: center;\"><em>Code with less nesting</em></td>    </tr><tr>    <td style=\"background-color: #cfe2f3;\"><pre style=\"background-color: #cfe2f3; border: 0px; color: black;\">response = server.Call(request)<br /> <br />if response.GetStatus() == RPC.OK:<br />  if response.GetAuthorizedUser():<br />    if response.GetEnc() == 'utf-8':<br />      if response.GetRows():<br />        vals = [ParseRow(r) for r in <br />                response.GetRows()]<br />        avg = sum(vals) / len(vals)<br />        return avg, vals<br />      else:<br />        raise EmptyError()<br />    else:<br />      raise AuthError('unauthorized')<br />  else:<br />    raise ValueError('wrong encoding')<br />else:<br />  raise RpcError(response.GetStatus())</pre></td>    <td style=\"background-color: #d9ead3;\"><pre style=\"background-color: #d9ead3; border: 0px; color: black;\">response = server.Call(request)<br /> <br />if response.GetStatus() != RPC.OK:<br />  raise RpcError(response.GetStatus())<br /><br />if not response.GetAuthorizedUser():<br />  raise ValueError('wrong encoding')<br /><br />if response.GetEnc() != 'utf-8':<br />  raise AuthError('unauthorized')<br /> <br />if not response.GetRows():<br />  raise EmptyError()<br /><br />vals = [ParseRow(r) for r in <br />        response.GetRows()]<br />avg = sum(vals) / len(vals)<br />return avg, vals</pre></td>  </tr></tbody></table><br /><br />Answer: the \"<em>wrong encoding\"</em> and \"<em>unauthorized\"</em> errors are swapped. <strong><span style=\"color: purple;\">This bug is easier to see in the refactored version, since the checks occur right as the errors are handled.</span></strong><br /><br />The refactoring technique shown above is known as <strong><em><span style=\"color: purple;\">guard clauses</span></em></strong>. A guard clause checks a criterion and fails fast if it is not met. It decouples the computational logic from the error logic. By removing the cognitive gap between error checking and handling, it frees up mental processing power. As a result, <strong><span style=\"color: purple;\">the refactored version is much easier to read and maintain</span></strong>. <br /><span style=\"color: purple;\"><strong><br /></strong></span><span style=\"color: purple;\"><strong>Here are some rules of thumb for reducing nesting in your code</strong>: </span><br /><ul><li>Keep conditional blocks short. It increases readability by keeping things local. </li><li>Consider refactoring when your loops and branches are more than 2 levels deep. </li><li>Think about moving nested logic into separate functions. For example, if you need to loop through a list of objects that each contain a list (such as a protocol buffer with repeated fields), you can define a function to process each object instead of using a double nested loop.</li></ul>Reducing nesting results in more readable code, which leads to discoverable bugs, faster developer iteration, and increased stability. When you can, simplify! <div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=21l7QzueXpc:whNUZ0rXesk:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=21l7QzueXpc:whNUZ0rXesk:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=21l7QzueXpc:whNUZ0rXesk:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/21l7QzueXpc\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "This is another post in our Code Health series. A version of this post originally appeared in Google bathrooms worldwide as a Google Testing on the Toilet episode. You can download a printer-friendly version to display in your office.   By Elliott Karpilovsky Deeply nested code hurts readability and is error-prone. Try spotting the bug in the two versions of this code:         Code with too much nesting      Code with less nesting        response = server.Call(request) if response.GetStatus() == RPC.OK:  if response.GetAuthorizedUser():    if response.GetEnc() == 'utf-8':      if response.GetRows():        vals = [ParseRow(r) for r in                 response.GetRows()]        avg = sum(vals) / len(vals)        return avg, vals      else:        raise EmptyError()    else:      raise AuthError('unauthorized')  else:    raise ValueError('wrong encoding')else:  raise RpcError(response.GetStatus())    response = server.Call(request) if response.GetStatus() != RPC.OK:  raise RpcError(response.GetStatus())if not response.GetAuthorizedUser():  raise ValueError('wrong encoding')if response.GetEnc() != 'utf-8':  raise AuthError('unauthorized') if not response.GetRows():  raise EmptyError()vals = [ParseRow(r) for r in         response.GetRows()]avg = sum(vals) / len(vals)return avg, vals  Answer: the \"wrong encoding\" and \"unauthorized\" errors are swapped. This bug is easier to see in the refactored version, since the checks occur right as the errors are handled.The refactoring technique shown above is known as guard clauses. A guard clause checks a criterion and fails fast if it is not met. It decouples the computational logic from the error logic. By removing the cognitive gap between error checking and handling, it frees up mental processing power. As a result, the refactored version is much easier to read and maintain. Here are some rules of thumb for reducing nesting in your code: Keep conditional blocks short. It increases readability by keeping things local. Consider refactoring when your loops and branches are more than 2 levels deep. Think about moving nested logic into separate functions. For example, if you need to loop through a list of objects that each contain a list (such as a protocol buffer with repeated fields), you can define a function to process each object instead of using a double nested loop.Reducing nesting results in more readable code, which leads to discoverable bugs, faster developer iteration, and increased stability. When you can, simplify!",
        "id": "tag:blogger.com,1999:blog-15045980.post-1593562688803890002",
        "isoDate": "2017-06-05T17:01:00.001Z"
      },
      {
        "title": "GTAC Diversity Scholarship",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/fr0rsUrGCGQ/gtac-diversity-scholarship.html",
        "pubDate": "2017-05-22T14:03:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<br /><br /><dt><em>by Lesley Katzen on behalf of the GTAC Diversity Committee</em></dt><br /><br />We are committed to increasing diversity at GTAC, and we believe the best way to do that is by making sure we have a diverse set of applicants to speak and attend. As part of that commitment, we are excited to announce that we will be offering travel scholarships again this year. <br />Travel scholarships will be available for selected applicants from traditionally underrepresented groups in technology. <br /><br />To be eligible for a grant to attend GTAC, applicants must: <br /><ul><li>Be 18 years of age or older. </li><li>Be from a traditionally underrepresented group in technology. </li><li>Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing. </li><li>Be able to attend core dates of GTAC, November 14th - 15th 2017 in London, England.</li></ul><strong>To apply:</strong><br />You must fill out the following scholarship <a href=\"https://docs.google.com/a/google.com/forms/d/1e7PrvRq5mshsS24lAeysHpAHspUL0XpU65W8xFkPnZ0\">form</a>and <a href=\"https://g.co/gtac/registrations/new\">register</a> for GTAC to be considered for a travel scholarship. <br />The deadline for submission is July 1st.  Scholarship recipients will be announced on August 15th. If you are selected, we will contact you with information on how to proceed with booking travel. <br /><strong><br /></strong><strong>What the scholarship covers:</strong><br />Google will pay for round-trip standard coach class airfare to London for selected scholarship recipients, and 3 nights of accommodations in a hotel near the Google King's Cross campus. Breakfast and lunch will be provided for GTAC attendees and speakers on both days of the conference. We will also provide a £75.00 gift card for other incidentals such as airport transportation or meals. You will need to provide your own credit card to cover any hotel incidentals. <br /><br />Google is dedicated to providing a harassment-free and inclusive conference experience for everyone. Our anti-harassment policy can be found at: <br /><a href=\"https://www.google.com/events/policy/anti-harassmentpolicy.html\">https://www.google.com/events/policy/anti-harassmentpolicy.html</a><br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=fr0rsUrGCGQ:cJyuR0piXEk:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=fr0rsUrGCGQ:cJyuR0piXEk:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=fr0rsUrGCGQ:cJyuR0piXEk:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/fr0rsUrGCGQ\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Lesley Katzen on behalf of the GTAC Diversity CommitteeWe are committed to increasing diversity at GTAC, and we believe the best way to do that is by making sure we have a diverse set of applicants to speak and attend. As part of that commitment, we are excited to announce that we will be offering travel scholarships again this year. Travel scholarships will be available for selected applicants from traditionally underrepresented groups in technology. To be eligible for a grant to attend GTAC, applicants must: Be 18 years of age or older. Be from a traditionally underrepresented group in technology. Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing. Be able to attend core dates of GTAC, November 14th - 15th 2017 in London, England.To apply:You must fill out the following scholarship formand register for GTAC to be considered for a travel scholarship. The deadline for submission is July 1st.  Scholarship recipients will be announced on August 15th. If you are selected, we will contact you with information on how to proceed with booking travel. What the scholarship covers:Google will pay for round-trip standard coach class airfare to London for selected scholarship recipients, and 3 nights of accommodations in a hotel near the Google King's Cross campus. Breakfast and lunch will be provided for GTAC attendees and speakers on both days of the conference. We will also provide a £75.00 gift card for other incidentals such as airport transportation or meals. You will need to provide your own credit card to cover any hotel incidentals. Google is dedicated to providing a harassment-free and inclusive conference experience for everyone. Our anti-harassment policy can be found at: https://www.google.com/events/policy/anti-harassmentpolicy.html",
        "id": "tag:blogger.com,1999:blog-15045980.post-1525358742210307660",
        "isoDate": "2017-05-22T14:03:00.000Z"
      },
      {
        "title": "GTAC 2017 - Registration is open!",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/mtAbaQ5OGVM/gtac-2017-registration-is-open.html",
        "pubDate": "2017-05-15T21:40:00.002Z",
        "author": "Google Testing Bloggers",
        "content": "<dl><dt>by Diego Cavalcanti on behalf of the GTAC 2017 Committee</dt></dl>The Google Test Automation Conference (GTAC) is an annual test automation conference hosted by Google. It brings together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It is a great opportunity to present, learn, and challenge modern testing technologies and strategies. <br /><br />We are pleased to announce that this year, GTAC will be held in Google's <a href=\"https://goo.gl/maps/TvYEZTQYxTP2\">London</a> office on <strong>November 14th and 15th, 2017</strong>. <br /><br />Registration is currently <strong>OPEN</strong> for attendees and speakers. See more information <a href=\"https://g.co/gtac/how-to-register/\">here</a>. <br /><br />The schedule for the upcoming months is as follows: <br /><ul><li><strong>May 15, 2017</strong> - Registration opens for <a href=\"https://g.co/gtac/registrations/new/\">speakers and attendees</a>, including applicants for the <a href=\"https://docs.google.com/a/google.com/forms/d/e/1FAIpQLSeatfvqleLqC5btPNCJvUICcQfL83D8bRG7z7m07VdHeUqzow/viewform\">diversity scholarship</a>. </li><li><strong>July 1, 2017</strong> - Registration closes for speaker submissions. </li><li><strong>July 15, 2017</strong> - Registration closes for attendee submissions. </li><li><strong>August 15, 2017</strong> - Selected speakers and attendees will be notified. </li><li><strong>November 13, 2017</strong> - Rehearsal day for speakers (not open for attendees). </li><li><strong>November 14-15, 2017</strong> - GTAC 2017!</li></ul>As part of our efforts to increase diversity of speakers and attendees at GTAC, we will again be offering travel scholarships for selected applicants from traditionally underrepresented groups in technology. Please find more information <a href=\"https://g.co/gtac/diversity-scholarship/\">here</a>. <br /><br />Please do not hesitate to contact <a href=\"mailto:gtac2017@google.com\">gtac2017@google.com</a> if you have any questions. We look forward to seeing you in London! <br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=mtAbaQ5OGVM:T6JV5W2B_HM:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=mtAbaQ5OGVM:T6JV5W2B_HM:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=mtAbaQ5OGVM:T6JV5W2B_HM:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/mtAbaQ5OGVM\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Diego Cavalcanti on behalf of the GTAC 2017 CommitteeThe Google Test Automation Conference (GTAC) is an annual test automation conference hosted by Google. It brings together engineers from industry and academia to discuss advances in test automation and the test engineering computer science field. It is a great opportunity to present, learn, and challenge modern testing technologies and strategies. We are pleased to announce that this year, GTAC will be held in Google's London office on November 14th and 15th, 2017. Registration is currently OPEN for attendees and speakers. See more information here. The schedule for the upcoming months is as follows: May 15, 2017 - Registration opens for speakers and attendees, including applicants for the diversity scholarship. July 1, 2017 - Registration closes for speaker submissions. July 15, 2017 - Registration closes for attendee submissions. August 15, 2017 - Selected speakers and attendees will be notified. November 13, 2017 - Rehearsal day for speakers (not open for attendees). November 14-15, 2017 - GTAC 2017!As part of our efforts to increase diversity of speakers and attendees at GTAC, we will again be offering travel scholarships for selected applicants from traditionally underrepresented groups in technology. Please find more information here. Please do not hesitate to contact gtac2017@google.com if you have any questions. We look forward to seeing you in London!",
        "id": "tag:blogger.com,1999:blog-15045980.post-1190634600492726801",
        "isoDate": "2017-05-15T21:40:00.002Z"
      },
      {
        "title": "OSS-Fuzz: Five Months Later, and Rewarding Projects",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/ZD3AXFoKqd8/oss-fuzz-five-months-later-and.html",
        "pubDate": "2017-05-08T16:00:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<em>By Oliver Chang, Abhishek Arya (Security Engineers, Chrome Security), Kostya Serebryany (Software Engineer, Dynamic Tools), and Josh Armour (Security Program Manager)</em><br /><br />Five months ago, we <a href=\"https://testing.googleblog.com/2016/12/announcing-oss-fuzz-continuous-fuzzing.html\">announced</a><a href=\"https://github.com/google/oss-fuzz/\">OSS-Fuzz</a>, Google's effort to help make open source software more secure and stable. Since then, our robot army has been working hard at <a href=\"https://en.wikipedia.org/wiki/Fuzzing\">fuzzing</a>, processing 10 trillion test inputs a day. Thanks to the efforts of the open source community who have integrated a total of <a href=\"https://github.com/google/oss-fuzz/tree/master/projects\">47</a> projects, we've found over <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=-component%3AInfra+status%3ANew%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">1,000</a>bugs (<a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=-component%3AInfra+status%3ANew%2CFixed%2CVerified+Type%3DBug-Security+&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">264</a>of which are potential security vulnerabilities).<br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-ZSpYiLN8VOU/WRCII1KSWmI/AAAAAAAAARk/uVk7QKXqHqQmbTIhgCeaJ4Z0q8-OeVQWQCLcB/s1600/Copy_of0.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"371\" src=\"https://2.bp.blogspot.com/-ZSpYiLN8VOU/WRCII1KSWmI/AAAAAAAAARk/uVk7QKXqHqQmbTIhgCeaJ4Z0q8-OeVQWQCLcB/s640/Copy_of0.png\" width=\"640\" /></a></div><div id=\"gdcalert1\" style=\"text-align: center;\">Breakdown of the types of bugs we're finding</div><dl><h2>Notable results</h2></dl>OSS-Fuzz has found numerous security vulnerabilities in several critical open source projects: <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dfreetype2+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">10</a>in FreeType2, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dffmpeg+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">17</a>in FFmpeg, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dlibreoffice+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">33</a>in LibreOffice, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dsqlite3+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">8</a>in SQLite 3, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dgnutls+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">10</a>in GnuTLS, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dpcre2+Type%3DBug-Security+status%3ANew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">25</a>in PCRE2, <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dgrpc+Type%3DBug-Security+status%3DNew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">9</a>in gRPC, and <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=Proj%3Dwireshark+Type%3DBug-Security+status%3DNew%2CAccepted%2CFixed%2CVerified&amp;colspec=ID+Type+Component+Status+Proj+Reported+Owner+Summary&amp;cells=ids\">7</a>in Wireshark. We've also had at least one bug collision with another independent security researcher (<a href=\"https://botan.randombit.net/security.html\">CVE-2017-2801</a>). (Some of the bugs are still view-restricted so links may show smaller numbers.) <br /><br />Once a project is integrated into OSS-Fuzz, the continuous and automated nature of OSS-Fuzz means that we often catch these issues just hours after the regression is introduced into the upstream repository, so that the chances of users being affected is reduced. <br /><br />Fuzzing not only finds memory safety related bugs, it can also find correctness or logic bugs. One example is a carry propagating bug in OpenSSL (<a href=\"https://www.openssl.org/news/secadv/20170126.txt\">CVE-2017-3732</a>). <br /><br />Finally, OSS-Fuzz has reported over 300 <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/faq.md#how-do-you-handle-timeouts-and-ooms\">timeout and out-of-memory failures</a> (~75% of which got fixed). Not every project treats these as bugs, but fixing them enables OSS-Fuzz to find more interesting bugs.<br /><h2>Announcing rewards for open source projects</h2>We believe that user and internet security as a whole can benefit greatly if more open source projects include fuzzing in their development process. To this end, we'd like to encourage more projects to participate and adopt the <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md\">ideal integration</a> guidelines that we've established. <br /><br />Combined with fixing all the issues that are found, this is often a significant amount of work for developers who may be working on an open source project in their spare time. To support these projects, we are expanding our existing <a href=\"https://www.google.com/about/appsecurity/patch-rewards/\">Patch Rewards</a>program to include rewards for the integration of <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/glossary.md#fuzz-target\">fuzz targets</a> into OSS-Fuzz. <br /><br />To qualify for these rewards, a project needs to have a large user base and/or be critical to global IT infrastructure. Eligible projects will receive $1,000 for initial integration, and up to $20,000 for ideal integration (the final amount is at our discretion). You have the option of donating these rewards to charity instead, and Google will double the amount. <br /><br />To qualify for the ideal integration reward, projects must show that: <br /><ul><li>Fuzz targets are checked into their upstream repository and integrated in the build system with <a href=\"https://github.com/google/sanitizers\">sanitizer</a> support (up to $5,000). </li><li>Fuzz targets are <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/clusterfuzz.md#performance-analyzer\">efficient</a>and provide good code coverage (&gt;80%) (up to $5,000). </li><li>Fuzz targets are part of the official upstream development and regression testing process, i.e. they are maintained, run against old known crashers and the periodically updated <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md#seed-corpus\">corpora</a>(up to $5,000). </li><li>The last $5,000 is a \"<a href=\"https://en.wikipedia.org/wiki/Leet\">l33t</a>\" bonus that we may reward at our discretion for projects that we feel have gone the extra mile or done something really awesome.</li></ul>We've already started to contact the first round of projects that are eligible for the initial reward. If you are the maintainer or point of contact for one of these projects, you may also <a href=\"https://goo.gl/YMLfG2\">reach out</a> to us in order to apply for our ideal integration rewards. <br /><h2>The future</h2>We'd like to thank the existing contributors who integrated their projects and fixed countless bugs. We hope to see more projects integrated into OSS-Fuzz, and greater adoption of fuzzing as standard practice when developing software. <br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ZD3AXFoKqd8:g1pamiscLMI:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ZD3AXFoKqd8:g1pamiscLMI:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=ZD3AXFoKqd8:g1pamiscLMI:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/ZD3AXFoKqd8\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By Oliver Chang, Abhishek Arya (Security Engineers, Chrome Security), Kostya Serebryany (Software Engineer, Dynamic Tools), and Josh Armour (Security Program Manager)Five months ago, we announcedOSS-Fuzz, Google's effort to help make open source software more secure and stable. Since then, our robot army has been working hard at fuzzing, processing 10 trillion test inputs a day. Thanks to the efforts of the open source community who have integrated a total of 47 projects, we've found over 1,000bugs (264of which are potential security vulnerabilities).Breakdown of the types of bugs we're findingNotable resultsOSS-Fuzz has found numerous security vulnerabilities in several critical open source projects: 10in FreeType2, 17in FFmpeg, 33in LibreOffice, 8in SQLite 3, 10in GnuTLS, 25in PCRE2, 9in gRPC, and 7in Wireshark. We've also had at least one bug collision with another independent security researcher (CVE-2017-2801). (Some of the bugs are still view-restricted so links may show smaller numbers.) Once a project is integrated into OSS-Fuzz, the continuous and automated nature of OSS-Fuzz means that we often catch these issues just hours after the regression is introduced into the upstream repository, so that the chances of users being affected is reduced. Fuzzing not only finds memory safety related bugs, it can also find correctness or logic bugs. One example is a carry propagating bug in OpenSSL (CVE-2017-3732). Finally, OSS-Fuzz has reported over 300 timeout and out-of-memory failures (~75% of which got fixed). Not every project treats these as bugs, but fixing them enables OSS-Fuzz to find more interesting bugs.Announcing rewards for open source projectsWe believe that user and internet security as a whole can benefit greatly if more open source projects include fuzzing in their development process. To this end, we'd like to encourage more projects to participate and adopt the ideal integration guidelines that we've established. Combined with fixing all the issues that are found, this is often a significant amount of work for developers who may be working on an open source project in their spare time. To support these projects, we are expanding our existing Patch Rewardsprogram to include rewards for the integration of fuzz targets into OSS-Fuzz. To qualify for these rewards, a project needs to have a large user base and/or be critical to global IT infrastructure. Eligible projects will receive $1,000 for initial integration, and up to $20,000 for ideal integration (the final amount is at our discretion). You have the option of donating these rewards to charity instead, and Google will double the amount. To qualify for the ideal integration reward, projects must show that: Fuzz targets are checked into their upstream repository and integrated in the build system with sanitizer support (up to $5,000). Fuzz targets are efficientand provide good code coverage (>80%) (up to $5,000). Fuzz targets are part of the official upstream development and regression testing process, i.e. they are maintained, run against old known crashers and the periodically updated corpora(up to $5,000). The last $5,000 is a \"l33t\" bonus that we may reward at our discretion for projects that we feel have gone the extra mile or done something really awesome.We've already started to contact the first round of projects that are eligible for the initial reward. If you are the maintainer or point of contact for one of these projects, you may also reach out to us in order to apply for our ideal integration rewards. The futureWe'd like to thank the existing contributors who integrated their projects and fixed countless bugs. We hope to see more projects integrated into OSS-Fuzz, and greater adoption of fuzzing as standard practice when developing software.",
        "id": "tag:blogger.com,1999:blog-15045980.post-1119552047263124056",
        "isoDate": "2017-05-08T16:00:00.000Z"
      },
      {
        "title": "Where do our flaky tests come from?",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/hGQd_Dvk3r0/where-do-our-flaky-tests-come-from.html",
        "pubDate": "2017-04-17T19:43:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>author: Jeff Listfield </i><br /><br />When tests fail on code that was previously tested, this is a strong signal that something is newly wrong with the code. Before, the tests passed and the code was correct; now the tests fail and the code is not working right. The goal of a good test suite is to make this signal as clear and directed as possible. <br /><br />Flaky (nondeterministic) tests, however, are different.  Flaky tests are tests that exhibit both a passing and a failing result with the same code. Given this, a test failure may or may not mean that there's a new problem. And trying to recreate the failure, by rerunning the test with the same version of code, may or may not result in a passing test. We start viewing these tests as unreliable and eventually they lose their value. If the root cause is nondeterminism in the production code, ignoring the test means ignoring a production bug. <br /><dl><dt><strong>Flaky Tests at Google</strong></dt><div id=\"gdcalert1\"><br /></div><div id=\"gdcalert1\">Google has around 4.2 million tests that run on our continuous integration system. Of these, around 63 thousand have a flaky run over the course of a week. While this represents less than 2% of our tests, it still causes significant drag on our engineers.</div></dl>If we want to fix our flaky tests (and avoid writing new ones) we need to understand them. At Google, we collect lots of data on our tests: execution times, test types, run flags, and consumed resources. I've studied how some of this data correlates with flaky tests and believe this research can lead us to better, more stable testing practices. Overwhelmingly, the larger the test (as measured by binary size, RAM use, or number of libraries built), the more likely it is to be flaky. The rest of this post will discuss some of my findings. <br />For a previous discussion of our flaky tests, see John Micco's <a href=\"https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html\">post</a>from May 2016. <br /><dl><dt><strong>Test size - Large tests are more likely to be flaky</strong></dt><div id=\"gdcalert2\"><br /></div><div id=\"gdcalert2\">We categorize our tests into three general sizes: small, medium and large. Every test has a size, but the choice of label is subjective. The engineer chooses the size when they initially write the test, and the size is not always updated as the test changes. For some tests it doesn't reflect the nature of the test anymore. Nonetheless, it has some predictive value. Over the course of a week, 0.5% of our small tests were flaky, 1.6% of our medium tests were flaky, and 14% of our large tests were flaky <sup>[1]</sup>. There's a clear increase in flakiness from small to medium and from medium to large. But this still leaves open a lot of questions. There's only so much we can learn looking at three sizes.</div></dl><strong>The larger the test, the more likely it will be flaky</strong><br /><br />There are some objective measures of size we collect: test binary size and RAM used when running the test <sup>[2]</sup>. For these two metrics, I grouped tests into equal-sized buckets <sup>[3]</sup> and calculated the percentage of tests in each bucket that were flaky. The numbers below are the r2 values of the linear best fit <sup>[4]</sup>.<br /><br /><table class=\"my-bordered-table\" style=\"white-space:nowrap;\">  <tbody><tr>   <td colspan=\"2\">Correlation between metric and likelihood of test being flaky    </td>  </tr><tr>   <td>Metric    </td>   <td>r2    </td>  </tr><tr>   <td>Binary size    </td>   <td>0.82    </td>  </tr><tr>   <td>RAM used    </td>   <td>0.76    </td>  </tr></tbody></table><br /><br />The tests that I'm looking at are (for the most part) hermetic tests that provide a pass/fail signal. Binary size and RAM use correlated quite well when looking across our tests and there's not much difference between them. So it's not just that large tests are likely to be flaky, it's that the larger the tests get, the more likely they are to be flaky. <br /><br />I have charted the full set of tests below for those two metrics. Flakiness increases with increases in binary size <sup>[5]</sup>, but we also see increasing linear fit residuals <sup>[6]</sup> at larger sizes. <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-weRP8mmxN0M/WPUZSHM-MzI/AAAAAAAAARA/gp9_OPT6nrw-kjF0-YwTvQLli5Y9R0PpwCLcB/s1600/Copy_of0.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://2.bp.blogspot.com/-weRP8mmxN0M/WPUZSHM-MzI/AAAAAAAAARA/gp9_OPT6nrw-kjF0-YwTvQLli5Y9R0PpwCLcB/s1600/Copy_of0.png\" /></a></div><br />The RAM use chart below has a clearer progression and only starts showing large residuals between the first and second vertical lines. <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://1.bp.blogspot.com/-uJ6ZH3HvAM4/WPUZfW1ofVI/AAAAAAAAARE/sWCNyksZKg4IsMA3j_vmDPjiSy-Rf18MgCLcB/s1600/Copy_of1.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://1.bp.blogspot.com/-uJ6ZH3HvAM4/WPUZfW1ofVI/AAAAAAAAARE/sWCNyksZKg4IsMA3j_vmDPjiSy-Rf18MgCLcB/s1600/Copy_of1.png\" /></a></div><br /><br />While the bucket sizes are constant, the number of tests in each bucket is different. The points on the right with larger residuals include much fewer tests than those on the left. If I take the smallest 96% of our tests (which ends just past the first vertical line) and then shrink the bucket size, I get a much stronger correlation (r2 is 0.94). It perhaps indicates that RAM and binary size are much better predictors than the overall charts show. <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://4.bp.blogspot.com/-5L4eq5CWlQ0/WPUZlvflMKI/AAAAAAAAARI/rG06pboI3FIievi_RGxWDnP3rFk1cv7wgCLcB/s1600/Copy_of2.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://4.bp.blogspot.com/-5L4eq5CWlQ0/WPUZlvflMKI/AAAAAAAAARI/rG06pboI3FIievi_RGxWDnP3rFk1cv7wgCLcB/s1600/Copy_of2.png\" /></a></div><br /><br /><div id=\"gdcalert5\"><strong>Certain tools correlate with a higher rate of flaky tests</strong></div><dl><div id=\"gdcalert6\">Some tools get blamed for being the cause of flaky tests. For example, <a href=\"http://www.seleniumhq.org/projects/webdriver/\">WebDriver</a> tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky <sup>[7]</sup>. For a few of our common testing tools, I determined the percentage of all the tests written with that tool that were flaky. Of note, all of these tools tend to be used with our larger tests. This is not an exhaustive list of all our testing tools, and represents around a third of our overall tests. The remainder of the tests use less common tools or have no readily identifiable tool.</div><div id=\"gdcalert6\"><br /></div></dl><table class=\"my-bordered-table\" style=\"white-space:nowrap;\">  <tbody><tr>   <td colspan=\"3\">Flakiness of tests using some of our common testing tools    </td>  </tr><tr>   <td>Category    </td>   <td>% of tests that are flaky    </td>   <td>% of all flaky tests    </td>  </tr><tr>   <td>All tests    </td>   <td><div style=\"text-align: right;\">1.65%</div></td>   <td><div style=\"text-align: right;\">100%</div></td>  </tr><tr>   <td>Java WebDriver    </td>   <td><div style=\"text-align: right;\">10.45%</div></td>   <td><div style=\"text-align: right;\">20.3%</div></td>  </tr><tr>   <td>Python WebDriver    </td>   <td><div style=\"text-align: right;\">18.72%</div></td>   <td><div style=\"text-align: right;\">4.0%</div></td>  </tr><tr>   <td>An internal integration tool    </td>   <td><div style=\"text-align: right;\">14.94%</div></td>   <td><div style=\"text-align: right;\">10.6%</div></td>  </tr><tr>   <td>Android emulator    </td>   <td><div style=\"text-align: right;\">25.46%</div></td>   <td><div style=\"text-align: right;\">11.9%</div></td>  </tr></tbody></table><br /><br />All of these tools have higher than average flakiness. And given that 1 in 5 of our flaky tests are Java WebDriver tests, I can understand why people complain about them. But correlation is not causation, and given our results from the previous section, there might be something other than the tool causing the increased rate of flakiness. <br /><dl><dt><strong>Size is more predictive than tool</strong></dt><div id=\"gdcalert7\"><br /></div><div id=\"gdcalert7\">We can combine tool choice and test size to see which is more important. For each tool above, I isolated tests that use the tool and bucketed those based on memory usage (RAM) and binary size, similar to my previous approach. I calculated the line of best fit and how well it correlated with the data (r2). I then computed the predicted likelihood a test would be flaky at the smallest bucket <sup>[8]</sup> (which is already the 48th percentile of all our tests) as well as the 90th and 95th percentile of RAM used.</div></dl><table class=\"my-bordered-table\" style=\"white-space:nowrap;\">  <tbody><tr>   <td colspan=\"5\">Predicted flaky likelihood by <strong>RAM </strong>and <strong>tool</strong>   </td>  </tr><tr>   <td>Category    </td>   <td>r2    </td>   <td>Smallest bucket<br>(48th percentile)    </td>   <td>90th percentile    </td>   <td>95th percentile    </td>  </tr><tr>   <td>All tests    </td>   <td>0.76    </td>   <td>1.5%    </td>   <td>5.3%    </td>   <td>9.2%    </td>  </tr><tr>   <td>Java WebDriver    </td>   <td>0.70    </td>   <td>2.6%    </td>   <td>6.8%    </td>   <td>11%    </td>  </tr><tr>   <td>Python WebDriver    </td>   <td>0.65    </td>   <td>-2.0%    </td>   <td>2.4%    </td>   <td>6.8%    </td>  </tr><tr>   <td>An internal integration tool    </td>   <td>0.80    </td>   <td>-1.9%    </td>   <td>3.1%    </td>   <td>8.1%    </td>  </tr><tr>   <td>Android emulator    </td>   <td>0.45    </td>   <td>7.1%    </td>   <td>12%    </td>   <td>17%    </td>  </tr></tbody></table><br /><br />This table shows the results of these calculations for RAM. The correlation is stronger for the tools other than Android emulator. If we ignore that tool, the difference in correlations between tools for similar RAM use are around 4-5%. The differences from the smallest test to the 95th percentile for the tests are 8-10%. This is one of the most useful outcomes from this research: tools have some impact, but RAM use accounts for larger deviations in flakiness.<br /><table class=\"my-bordered-table\" style=\"white-space:nowrap;\">  <tbody><tr>   <td colspan=\"5\">Predicted flaky likelihood by <strong>binary size</strong>and <strong>tool</strong>   <br /><strong><br /></strong></td>  </tr><tr>   <td>Category    </td>   <td>r2    </td>   <td>Smallest bucket<br>(33rd percentile)    </td>   <td>90th percentile    </td>   <td>95th percentile    </td>  </tr><tr>   <td>All tests    </td>   <td>0.82    </td>   <td>-4.4%    </td>   <td>4.5%    </td>   <td>9.0%    </td>  </tr><tr>   <td>Java WebDriver    </td>   <td>0.81    </td>   <td>-0.7%    </td>   <td>14%    </td>   <td>21%    </td>  </tr><tr>   <td>Python WebDriver    </td>   <td>0.61    </td>   <td>-0.9%    </td>   <td>11%    </td>   <td>17%    </td>  </tr><tr>   <td>An internal integration tool    </td>   <td>0.80    </td>   <td>-1.8%    </td>   <td>10%    </td>   <td>17%    </td>  </tr><tr>   <td>Android emulator    </td>   <td>0.05    </td>   <td>18%    </td>   <td>23%    </td>   <td>25%    </td>  </tr></tbody></table><br /><br />There's virtually no correlation between binary size and flakiness for Android emulator tests. For the other tools, you see greater variation in predicted flakiness between the small tests and large tests compared to RAM; up to 12% points. But you also see wider differences from the smallest size to the largest; 22% at the max. This is similar to what we saw with RAM use and another of the most useful outcomes of this research: binary size accounts for larger deviations in flakiness than the tool you use. <br /><dl><dt><strong>Conclusions</strong></dt><div id=\"gdcalert8\"><br /></div><div id=\"gdcalert8\">Engineer-selected test size correlates with flakiness, but within Google there are not enough test size options to be particularly useful.</div></dl>Objectively measured test binary size and RAM have strong correlations with whether a test is flaky. This is a continuous function rather than a step function. A step function would have sudden jumps and could indicate that we're transitioning from one type of test to another at those points (e.g. unit tests to system tests or system tests to integration tests). <br />Tests written with certain tools exhibit a higher rate of flakiness. But much of that can be explained by the generally larger size of these tests. The tool itself seems to contribute only a small amount to this difference. <br />We need to be more careful before we decide to write large tests. Think about what code you are testing and what a minimal test would look like. And we need to be careful as we write large tests. Without additional effort aimed at preventing flakiness, there's is a strong likelihood you will have flaky tests that require maintenance. <br /><dl><dt><strong>Footnotes</strong></dt></dl><ol><li>A test was flaky if it had at least one flaky run during the week. </li><li>I also considered number of libraries built to create the test. In a 1% sample of tests, binary size (0.39) and RAM use (0.34) had stronger correlations than number of libraries (0.27). I only studied binary size and RAM use moving forward. </li><li>I aimed for around 100 buckets for each metric. </li><li>r2 measures how closely the line of best fit matches the data. A value of 1 means the line matches the data exactly. </li><li>There are two interesting areas where the points actually reverse their upward slope. The first starts about halfway to the first vertical line and lasts for a few data points and the second goes from right before the first vertical line to right after. The sample size is large enough here that it's unlikely to just be random noise. There are clumps of tests around these points that are more or less flaky than I'd expect only considering binary size. This is an opportunity for further study. </li><li>Distance from the observed point and the line of best fit. </li><li>Other web testing tools get blamed as well, but WebDriver is our most commonly used one. </li><li>Some of the predicted flakiness percents for the smallest buckets end up being negative. While we can't have a negative percent of tests be flaky, it is a possible outcome using this type of prediction.</li></ol><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=hGQd_Dvk3r0:NeEVP1edE2I:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=hGQd_Dvk3r0:NeEVP1edE2I:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=hGQd_Dvk3r0:NeEVP1edE2I:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/hGQd_Dvk3r0\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "author: Jeff Listfield When tests fail on code that was previously tested, this is a strong signal that something is newly wrong with the code. Before, the tests passed and the code was correct; now the tests fail and the code is not working right. The goal of a good test suite is to make this signal as clear and directed as possible. Flaky (nondeterministic) tests, however, are different.  Flaky tests are tests that exhibit both a passing and a failing result with the same code. Given this, a test failure may or may not mean that there's a new problem. And trying to recreate the failure, by rerunning the test with the same version of code, may or may not result in a passing test. We start viewing these tests as unreliable and eventually they lose their value. If the root cause is nondeterminism in the production code, ignoring the test means ignoring a production bug. Flaky Tests at GoogleGoogle has around 4.2 million tests that run on our continuous integration system. Of these, around 63 thousand have a flaky run over the course of a week. While this represents less than 2% of our tests, it still causes significant drag on our engineers.If we want to fix our flaky tests (and avoid writing new ones) we need to understand them. At Google, we collect lots of data on our tests: execution times, test types, run flags, and consumed resources. I've studied how some of this data correlates with flaky tests and believe this research can lead us to better, more stable testing practices. Overwhelmingly, the larger the test (as measured by binary size, RAM use, or number of libraries built), the more likely it is to be flaky. The rest of this post will discuss some of my findings. For a previous discussion of our flaky tests, see John Micco's postfrom May 2016. Test size - Large tests are more likely to be flakyWe categorize our tests into three general sizes: small, medium and large. Every test has a size, but the choice of label is subjective. The engineer chooses the size when they initially write the test, and the size is not always updated as the test changes. For some tests it doesn't reflect the nature of the test anymore. Nonetheless, it has some predictive value. Over the course of a week, 0.5% of our small tests were flaky, 1.6% of our medium tests were flaky, and 14% of our large tests were flaky [1]. There's a clear increase in flakiness from small to medium and from medium to large. But this still leaves open a lot of questions. There's only so much we can learn looking at three sizes.The larger the test, the more likely it will be flakyThere are some objective measures of size we collect: test binary size and RAM used when running the test [2]. For these two metrics, I grouped tests into equal-sized buckets [3] and calculated the percentage of tests in each bucket that were flaky. The numbers below are the r2 values of the linear best fit [4].     Correlation between metric and likelihood of test being flaky         Metric       r2         Binary size       0.82         RAM used       0.76      The tests that I'm looking at are (for the most part) hermetic tests that provide a pass/fail signal. Binary size and RAM use correlated quite well when looking across our tests and there's not much difference between them. So it's not just that large tests are likely to be flaky, it's that the larger the tests get, the more likely they are to be flaky. I have charted the full set of tests below for those two metrics. Flakiness increases with increases in binary size [5], but we also see increasing linear fit residuals [6] at larger sizes. The RAM use chart below has a clearer progression and only starts showing large residuals between the first and second vertical lines. While the bucket sizes are constant, the number of tests in each bucket is different. The points on the right with larger residuals include much fewer tests than those on the left. If I take the smallest 96% of our tests (which ends just past the first vertical line) and then shrink the bucket size, I get a much stronger correlation (r2 is 0.94). It perhaps indicates that RAM and binary size are much better predictors than the overall charts show. Certain tools correlate with a higher rate of flaky testsSome tools get blamed for being the cause of flaky tests. For example, WebDriver tests (whether written in Java, Python, or JavaScript) have a reputation for being flaky [7]. For a few of our common testing tools, I determined the percentage of all the tests written with that tool that were flaky. Of note, all of these tools tend to be used with our larger tests. This is not an exhaustive list of all our testing tools, and represents around a third of our overall tests. The remainder of the tests use less common tools or have no readily identifiable tool.     Flakiness of tests using some of our common testing tools         Category       % of tests that are flaky       % of all flaky tests         All tests       1.65%   100%     Java WebDriver       10.45%   20.3%     Python WebDriver       18.72%   4.0%     An internal integration tool       14.94%   10.6%     Android emulator       25.46%   11.9%  All of these tools have higher than average flakiness. And given that 1 in 5 of our flaky tests are Java WebDriver tests, I can understand why people complain about them. But correlation is not causation, and given our results from the previous section, there might be something other than the tool causing the increased rate of flakiness. Size is more predictive than toolWe can combine tool choice and test size to see which is more important. For each tool above, I isolated tests that use the tool and bucketed those based on memory usage (RAM) and binary size, similar to my previous approach. I calculated the line of best fit and how well it correlated with the data (r2). I then computed the predicted likelihood a test would be flaky at the smallest bucket [8] (which is already the 48th percentile of all our tests) as well as the 90th and 95th percentile of RAM used.     Predicted flaky likelihood by RAM and tool        Category       r2       Smallest bucket(48th percentile)       90th percentile       95th percentile         All tests       0.76       1.5%       5.3%       9.2%         Java WebDriver       0.70       2.6%       6.8%       11%         Python WebDriver       0.65       -2.0%       2.4%       6.8%         An internal integration tool       0.80       -1.9%       3.1%       8.1%         Android emulator       0.45       7.1%       12%       17%      This table shows the results of these calculations for RAM. The correlation is stronger for the tools other than Android emulator. If we ignore that tool, the difference in correlations between tools for similar RAM use are around 4-5%. The differences from the smallest test to the 95th percentile for the tests are 8-10%. This is one of the most useful outcomes from this research: tools have some impact, but RAM use accounts for larger deviations in flakiness.     Predicted flaky likelihood by binary sizeand tool        Category       r2       Smallest bucket(33rd percentile)       90th percentile       95th percentile         All tests       0.82       -4.4%       4.5%       9.0%         Java WebDriver       0.81       -0.7%       14%       21%         Python WebDriver       0.61       -0.9%       11%       17%         An internal integration tool       0.80       -1.8%       10%       17%         Android emulator       0.05       18%       23%       25%      There's virtually no correlation between binary size and flakiness for Android emulator tests. For the other tools, you see greater variation in predicted flakiness between the small tests and large tests compared to RAM; up to 12% points. But you also see wider differences from the smallest size to the largest; 22% at the max. This is similar to what we saw with RAM use and another of the most useful outcomes of this research: binary size accounts for larger deviations in flakiness than the tool you use. ConclusionsEngineer-selected test size correlates with flakiness, but within Google there are not enough test size options to be particularly useful.Objectively measured test binary size and RAM have strong correlations with whether a test is flaky. This is a continuous function rather than a step function. A step function would have sudden jumps and could indicate that we're transitioning from one type of test to another at those points (e.g. unit tests to system tests or system tests to integration tests). Tests written with certain tools exhibit a higher rate of flakiness. But much of that can be explained by the generally larger size of these tests. The tool itself seems to contribute only a small amount to this difference. We need to be more careful before we decide to write large tests. Think about what code you are testing and what a minimal test would look like. And we need to be careful as we write large tests. Without additional effort aimed at preventing flakiness, there's is a strong likelihood you will have flaky tests that require maintenance. FootnotesA test was flaky if it had at least one flaky run during the week. I also considered number of libraries built to create the test. In a 1% sample of tests, binary size (0.39) and RAM use (0.34) had stronger correlations than number of libraries (0.27). I only studied binary size and RAM use moving forward. I aimed for around 100 buckets for each metric. r2 measures how closely the line of best fit matches the data. A value of 1 means the line matches the data exactly. There are two interesting areas where the points actually reverse their upward slope. The first starts about halfway to the first vertical line and lasts for a few data points and the second goes from right before the first vertical line to right after. The sample size is large enough here that it's unlikely to just be random noise. There are clumps of tests around these points that are more or less flaky than I'd expect only considering binary size. This is an opportunity for further study. Distance from the observed point and the line of best fit. Other web testing tools get blamed as well, but WebDriver is our most commonly used one. Some of the predicted flakiness percents for the smallest buckets end up being negative. While we can't have a negative percent of tests be flaky, it is a possible outcome using this type of prediction.",
        "id": "tag:blogger.com,1999:blog-15045980.post-5678277460708842788",
        "isoDate": "2017-04-17T19:43:00.000Z"
      },
      {
        "title": "Code Health: Google's Internal Code Quality Efforts",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/yQJaBVnLoWk/code-health-googles-internal-code.html",
        "pubDate": "2017-04-03T18:46:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>By <a href=\"https://twitter.com/mkanat\">Max Kanat-Alexander</a>, Tech Lead for Code Health and Author of <a href=\"http://www.codesimplicity.com/book/\">Code Simplicity</a></i><br /><br />There are many aspects of good coding practices that don't fall under the normal areas of testing and tooling that most Engineering Productivity groups focus on in the software industry. For example, having readable and maintainable code is about more than just writing good tests or having the right tools—it's about having code that can be easily understood and modified in the first place. But how do you make sure that engineers follow these practices while still allowing them the independence that they need to make sound engineering decisions? <br /><br />Many years ago, a group of Googlers came together to work on this problem, and they called themselves the \"Code Health\" group. Why \"Code Health\"? Well, many of the other terms used for this in the industry—engineering productivity, best practices, coding standards, code quality—have connotations that could lead somebody to think we were working on something other than what we wanted to focus on. What we cared about was the processes and practices of software engineering in full—any aspect of <em>how software was written</em> that could influence the readability, maintainability, stability, or simplicity of code. We liked the analogy of having \"healthy\" code as covering all of these areas. <br /><br />This is a field that many authors, theorists, and conference speakers touch on, but not an area that usually has dedicated resources within engineering organizations. Instead, in most software companies, these efforts are pushed by a few dedicated engineers in their extra time or led by the senior tech leads. However, every software engineer is actually involved in code health in some way. After all, we all write software, and most of us care deeply about doing it the \"right way.\" So why not start a group that helps engineers with that \"right way\" of doing things? <br /><br />This isn't to say that we are prescriptive about engineering practices at Google. We still let engineers make the decisions that are most sensible for their projects. What the Code Health group does is work on efforts that <em>universally</em> improve the lives of engineers and their ability to write products with shorter iteration time, decreased development effort, greater stability, and improved performance. Everybody appreciates their code getting easier to understand, their libraries getting simpler, etc. because we all know those things let us move faster and make better products. <br /><br />But how do we accomplish all of this? Well, at Google, Code Health efforts come in many forms. <br /><br />There is a Google-wide Code Health Group composed of <a href=\"https://abc.xyz/investor/founders-letters/2004/ipo-letter.html\">20%</a>contributors who work to make engineering at Google better for everyone. The members of this group maintain internal documents on best practices and act as a sounding board for teams and individuals who wonder how best to improve practices in their area. Once in a while, for critical projects, members of the group get directly involved in refactoring code, improving libraries, or making changes to tools that promote code health. <br /><br />For example, this central group maintains Google's code review guidelines, writes internal publications about best practices, organizes tech talks on productivity improvements, and generally fosters a culture of great software engineering at Google. <br /><br />Some of the senior members of the Code Health group also advise engineering executives and internal leadership groups on how to improve engineering practices in their areas. It's not always clear how to implement effective code health practices in an area—some people have more experience than others making this happen broadly in teams, and so we offer our consulting and experience to help make simple code and great developer experiences a reality. <br /><br />In addition to the central group, many products and teams at Google have their own Code Health group. These groups tend to work more closely on actual coding projects, such as addressing technical debt through refactoring, making tools that detect and prevent bad coding practices, creating <a href=\"https://github.com/google/yapf\">automated code formatters</a>, or making systems for automatically deleting unused code. Usually these groups coordinate and meet with the central Code Health group to make sure that we aren't duplicating efforts across the company and so that great new tools and systems can be shared with the rest of Google. <br /><br />Throughout the years, Google's Code Health teams have had a major impact on the ability of engineers to develop great products quickly at Google. But code complexity isn't an issue that only affects Google—it affects everybody who writes software, from one person writing software on their own time to the largest engineering teams in the world. So in order to help out everybody, we're planning to release articles in the coming weeks and months that detail specific practices that we encourage internally—practices that can be applied everywhere to help your company, your codebase, your team, and you. Stay tuned here on the Google Testing Blog for more Code Health articles coming soon! <br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=yQJaBVnLoWk:fkVkiiXEfoU:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=yQJaBVnLoWk:fkVkiiXEfoU:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=yQJaBVnLoWk:fkVkiiXEfoU:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/yQJaBVnLoWk\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By Max Kanat-Alexander, Tech Lead for Code Health and Author of Code SimplicityThere are many aspects of good coding practices that don't fall under the normal areas of testing and tooling that most Engineering Productivity groups focus on in the software industry. For example, having readable and maintainable code is about more than just writing good tests or having the right tools—it's about having code that can be easily understood and modified in the first place. But how do you make sure that engineers follow these practices while still allowing them the independence that they need to make sound engineering decisions? Many years ago, a group of Googlers came together to work on this problem, and they called themselves the \"Code Health\" group. Why \"Code Health\"? Well, many of the other terms used for this in the industry—engineering productivity, best practices, coding standards, code quality—have connotations that could lead somebody to think we were working on something other than what we wanted to focus on. What we cared about was the processes and practices of software engineering in full—any aspect of how software was written that could influence the readability, maintainability, stability, or simplicity of code. We liked the analogy of having \"healthy\" code as covering all of these areas. This is a field that many authors, theorists, and conference speakers touch on, but not an area that usually has dedicated resources within engineering organizations. Instead, in most software companies, these efforts are pushed by a few dedicated engineers in their extra time or led by the senior tech leads. However, every software engineer is actually involved in code health in some way. After all, we all write software, and most of us care deeply about doing it the \"right way.\" So why not start a group that helps engineers with that \"right way\" of doing things? This isn't to say that we are prescriptive about engineering practices at Google. We still let engineers make the decisions that are most sensible for their projects. What the Code Health group does is work on efforts that universally improve the lives of engineers and their ability to write products with shorter iteration time, decreased development effort, greater stability, and improved performance. Everybody appreciates their code getting easier to understand, their libraries getting simpler, etc. because we all know those things let us move faster and make better products. But how do we accomplish all of this? Well, at Google, Code Health efforts come in many forms. There is a Google-wide Code Health Group composed of 20%contributors who work to make engineering at Google better for everyone. The members of this group maintain internal documents on best practices and act as a sounding board for teams and individuals who wonder how best to improve practices in their area. Once in a while, for critical projects, members of the group get directly involved in refactoring code, improving libraries, or making changes to tools that promote code health. For example, this central group maintains Google's code review guidelines, writes internal publications about best practices, organizes tech talks on productivity improvements, and generally fosters a culture of great software engineering at Google. Some of the senior members of the Code Health group also advise engineering executives and internal leadership groups on how to improve engineering practices in their areas. It's not always clear how to implement effective code health practices in an area—some people have more experience than others making this happen broadly in teams, and so we offer our consulting and experience to help make simple code and great developer experiences a reality. In addition to the central group, many products and teams at Google have their own Code Health group. These groups tend to work more closely on actual coding projects, such as addressing technical debt through refactoring, making tools that detect and prevent bad coding practices, creating automated code formatters, or making systems for automatically deleting unused code. Usually these groups coordinate and meet with the central Code Health group to make sure that we aren't duplicating efforts across the company and so that great new tools and systems can be shared with the rest of Google. Throughout the years, Google's Code Health teams have had a major impact on the ability of engineers to develop great products quickly at Google. But code complexity isn't an issue that only affects Google—it affects everybody who writes software, from one person writing software on their own time to the largest engineering teams in the world. So in order to help out everybody, we're planning to release articles in the coming weeks and months that detail specific practices that we encourage internally—practices that can be applied everywhere to help your company, your codebase, your team, and you. Stay tuned here on the Google Testing Blog for more Code Health articles coming soon!",
        "id": "tag:blogger.com,1999:blog-15045980.post-1064181176957805666",
        "isoDate": "2017-04-03T18:46:00.000Z"
      },
      {
        "title": "Discomfort as a Tool for Change",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/xZUlRZu2VuQ/discomfort-as-tool-for-change.html",
        "pubDate": "2017-02-13T16:53:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by Dave Gladfelter (SETI, Google Drive)</i><br /><h2>Introduction</h2>The SETI (Software Engineer, Tools and Infrastructure) role at Google is a strange one in that there's no obvious reason why it should exist. The SWEs (Software Engineers) on a project understand its problems best, and understanding a problem is most of the way to fixing it. How can SETIs bring unique value to a project when SWEs have more on-the-ground experience with their impediments? <br /><br />The answer is scope. A SWE is rewarded for being an expert in their particular area and domain and is highly motivated to make optimizations to their carved-out space. SETIs (and <a href=\"https://testing.googleblog.com/2016/09/what-test-engineers-do-at-google.html\">Test Engineers</a> and <a href=\"https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html\">EngProd</a>in general) identify and solve product-wide problems. <br /><br />Product-wide problems frequently arise because local optimizations don't necessarily add up to product-wide optimizations. The reason may be the limits of attention, blind spots, or mis-aligned incentives, but a group of SWEs each optimizing for their own sub-projects will not achieve product-wide maxima. <br /><br />Often SETIs and Test Engineers (TEs) know what behavior they'd like to see, such as more integration tests. We may even have management's ear and convince them to mandate such tests. However, in the absence of incentives, it's unlikely that the decisions SWEs make in response to such mandates will add up to the behavior we desire. Mandates around methods/practices are often ineffective. For example, a mandate of documentation for each public method on an interface often results in \"method foo does foo.\" <br /><br />The best way to create product-wide efficiencies is to change the way the team or process works in ways that will (initially) be uncomfortable for the engineering team, but that pays dividends that can't be achieved any other way. SETIs and TEs must work to identify the blind spots and negative interactions between engineering teams and change the environment in ways that align engineering teams' incentives. When properly incentivized, SWEs will make optimal decisions enhanced by product-wide vision rather than micro-management. <br /><h2>Common Product-Wide Problems</h2><h3>Hard-to-use APIs</h3>One common example of local optimizations resulting in cross-team de-optimization is documentation and ease-of-use of internal APIs. The team that implements an internal API is not rewarded for making it easy to use except in the most oblique ways. Clients are compelled to use the internal APIs provided to them, so the API owner has a monopoly and will set the price of using it at \"you must read all the code and debug it yourself\" in the absence of incentives or (rare) heroes. <br /><h3>Big, slow releases</h3>Another example is large and slow releases. Without EngProd help or external pressure, teams will gravitate to the slowest, biggest release possible. <br /><br />This makes sense from the position of any individual SWE: releases are painful, you have to ensure that there are no UI and API regressions, watch traffic and error rates for some time, and re-learn and use tools and processes that are complex and specific to releases. <br /><br />Multiple teams will naturally gravitate to having one big release so that all of these costs can be bundled into one operation for \"efficiency.\" The result is that engineers don't get feedback on features for weeks and versioning of APIs and data stores is ignored (since all the parts of the system are bundled together into one big release). This greatly slows down developer and feature velocity and greatly increases risks of cascading failures when the release fails. <br /><h2>How EngProd fixes product-wide problems</h2>SETIs can nibble around the edges of these kinds of problems by writing tools and automation. TEs can create easy-to-use test environments that facilitate isolating and debugging faults in integration and ambiguities in APIs. We can use fancy technologies to sample live traffic and ensure that new versions of systems behave the same as previous versions. We can review design docs to ensure that they have an appropriate test plan. Often these actions do have real value. However, these are not the best way to align incentives to create a product-wide solution. Facilitating engineering teams' fruitful collaboration (and dis-incentivizing negative interactions) gives EngProd a multiplier that is hard to achieve with only tooling and automation. <br /><br />Heroes are few and far between so we must turn to incentives, which is where <em>discomfort</em> comes in. Continuity is comfortable and change is painful. EngProd looks at how to change the problem so that teams are incentivized to work together fruitfully and disincentivized (discomforted) to pursue local optimizations exclusively. <br /><br />So how does EngProd align incentives? Certainly there is a place for optimizing for optimal behaviors, such as easy-to-use integration environments. However, incentivizing optimal behaviors via negative feedback should not be overlooked. Each problem is different, so let's look at how to address the two examples above: <br /><h3>Incentivizing easy-to-use APIs</h3>Engineers will make the things they're incentivized to make. For APIs, make teams incentivized to provide integration help in the form of <em>fakes</em>. EngProd works with team leads to ensure there are explicit objectives to provide Fakes for their APIs as part of the rollout. <br /><br /><a href=\"https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html\">Fakes</a>are as-simple-as-possible implementations of a service that still can be used to do pre-submit testing of client interactions with the system. They don't replace integration tests, but they reduce the likelihood of finding errors in subsequent integration test runs by an order of magnitude. <br />Furthermore, have some subset of the same client-owned and server-owned tests run against the fakes (for quick presubmit testing) as well as the real implementation (for continuous integration testing) and work with management to make it the responsibility of the Fake owner to debug any discrepancies for either the client- or the server-owned tests. <br /><br /><strong>This reverses the pain!</strong> API owners, who are in a position to make APIs better, are now the ones experiencing negative incentives when APIs are not easy to use. Previously, when clients felt the pain, they had no recourse other than to file easily-ignored bugs (\"Closed: working as intended\") or contribute changes to the API owners' codebase, hurting their own performance with distractions. <br /><br />This will incentivize API owners to design APIs to be as simple as possible with as few side-effects as possible, and to provide high-quality fakes that make it easy for clients to integrate with the API. Some teams will certainly not like this change at first, but I have seen API teams come to the realization that this is the best choice for the larger effort and implement these practices despite their cost to the team in the short run. <br /><br />Helping management set engineering team objectives may not seem like a typical SETI responsibility, but although management is responsible for setting performance incentives and objectives, they are not well-positioned to understand how the low-level decisions of different teams create harmful interactions and lower cross-team performance, so they need SETI and TE guidance to create an environment that encourages optimal behaviors. <br /><h3>Fast, small releases</h3>Being forced to release more frequently than is required by feature deployment requirements has many beneficial side-effects that make release velocity a goal unto itself. SETIs and TEs faced with big, slow releases work with management to mandate a move to a set of smaller, more frequent releases. As release velocity is ratcheted up, negative behaviours such as too much manual testing or too much internal coupling become more painful, and many optimal behaviors are incentivized. <br /><h4>Less coupling between systems</h4>When software is released together, it is easy to treat the seams between different components as implementation details. Resulting systems becoming so intertwined (coupled) that responsibilities between them are completely and randomly mixed and their interactions are too complex for any one person to understand. When two components are released separately and at different times, different versions of them must be compatible with one another. Engineers who were previously complacent about this fragility will become fearful of failed releases due to implicit contract changes. They will change their behavior in beneficial ways such as defining the contract between components explicitly and creating regression testing for it. The result is a system composed of robust, self-contained, more easily understood components. <br /><h4>Better/More automated testing</h4>Manual testing becomes more painful as release velocity is ramped up. This will incentivize automated regression, UI and performance tests. This makes the team more agile and able to catch defects sooner and more cheaply. <br /><h4>Faster feedback</h4>When incremental feature changes can be released to dogfood or other beta channels more frequently, user interaction designers and product managers get much faster feedback about what paths lead to better user engagement and experience than in big, slow releases where an entire feature is deployed simultaneously. This results in a better product. <br /><h2>Conclusion</h2>The SETIs and TEs optimize interactions between teams and create fixes for product-wide, cross-team problems in order to improve engineering productivity and velocity. There are many worthwhile projects that EngProd can do using broad knowledge of the system and expertise in refactoring, automation and testing, such as creating test fixtures that enable continuous integration testing or identifying and combining duplicative tests or tools. <br /><br />That said, the biggest problem that EngProd is positioned to solve is to break the chain of local optimizations resulting in cross-team de-optimizations. To that end, discomfort is a tool that can incentivize engineers to find solutions that are optimal for the entire product. We should look for and advocate for these transformative changes. <div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=xZUlRZu2VuQ:1g-5rbI0tBQ:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=xZUlRZu2VuQ:1g-5rbI0tBQ:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=xZUlRZu2VuQ:1g-5rbI0tBQ:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/xZUlRZu2VuQ\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Dave Gladfelter (SETI, Google Drive)IntroductionThe SETI (Software Engineer, Tools and Infrastructure) role at Google is a strange one in that there's no obvious reason why it should exist. The SWEs (Software Engineers) on a project understand its problems best, and understanding a problem is most of the way to fixing it. How can SETIs bring unique value to a project when SWEs have more on-the-ground experience with their impediments? The answer is scope. A SWE is rewarded for being an expert in their particular area and domain and is highly motivated to make optimizations to their carved-out space. SETIs (and Test Engineers and EngProdin general) identify and solve product-wide problems. Product-wide problems frequently arise because local optimizations don't necessarily add up to product-wide optimizations. The reason may be the limits of attention, blind spots, or mis-aligned incentives, but a group of SWEs each optimizing for their own sub-projects will not achieve product-wide maxima. Often SETIs and Test Engineers (TEs) know what behavior they'd like to see, such as more integration tests. We may even have management's ear and convince them to mandate such tests. However, in the absence of incentives, it's unlikely that the decisions SWEs make in response to such mandates will add up to the behavior we desire. Mandates around methods/practices are often ineffective. For example, a mandate of documentation for each public method on an interface often results in \"method foo does foo.\" The best way to create product-wide efficiencies is to change the way the team or process works in ways that will (initially) be uncomfortable for the engineering team, but that pays dividends that can't be achieved any other way. SETIs and TEs must work to identify the blind spots and negative interactions between engineering teams and change the environment in ways that align engineering teams' incentives. When properly incentivized, SWEs will make optimal decisions enhanced by product-wide vision rather than micro-management. Common Product-Wide ProblemsHard-to-use APIsOne common example of local optimizations resulting in cross-team de-optimization is documentation and ease-of-use of internal APIs. The team that implements an internal API is not rewarded for making it easy to use except in the most oblique ways. Clients are compelled to use the internal APIs provided to them, so the API owner has a monopoly and will set the price of using it at \"you must read all the code and debug it yourself\" in the absence of incentives or (rare) heroes. Big, slow releasesAnother example is large and slow releases. Without EngProd help or external pressure, teams will gravitate to the slowest, biggest release possible. This makes sense from the position of any individual SWE: releases are painful, you have to ensure that there are no UI and API regressions, watch traffic and error rates for some time, and re-learn and use tools and processes that are complex and specific to releases. Multiple teams will naturally gravitate to having one big release so that all of these costs can be bundled into one operation for \"efficiency.\" The result is that engineers don't get feedback on features for weeks and versioning of APIs and data stores is ignored (since all the parts of the system are bundled together into one big release). This greatly slows down developer and feature velocity and greatly increases risks of cascading failures when the release fails. How EngProd fixes product-wide problemsSETIs can nibble around the edges of these kinds of problems by writing tools and automation. TEs can create easy-to-use test environments that facilitate isolating and debugging faults in integration and ambiguities in APIs. We can use fancy technologies to sample live traffic and ensure that new versions of systems behave the same as previous versions. We can review design docs to ensure that they have an appropriate test plan. Often these actions do have real value. However, these are not the best way to align incentives to create a product-wide solution. Facilitating engineering teams' fruitful collaboration (and dis-incentivizing negative interactions) gives EngProd a multiplier that is hard to achieve with only tooling and automation. Heroes are few and far between so we must turn to incentives, which is where discomfort comes in. Continuity is comfortable and change is painful. EngProd looks at how to change the problem so that teams are incentivized to work together fruitfully and disincentivized (discomforted) to pursue local optimizations exclusively. So how does EngProd align incentives? Certainly there is a place for optimizing for optimal behaviors, such as easy-to-use integration environments. However, incentivizing optimal behaviors via negative feedback should not be overlooked. Each problem is different, so let's look at how to address the two examples above: Incentivizing easy-to-use APIsEngineers will make the things they're incentivized to make. For APIs, make teams incentivized to provide integration help in the form of fakes. EngProd works with team leads to ensure there are explicit objectives to provide Fakes for their APIs as part of the rollout. Fakesare as-simple-as-possible implementations of a service that still can be used to do pre-submit testing of client interactions with the system. They don't replace integration tests, but they reduce the likelihood of finding errors in subsequent integration test runs by an order of magnitude. Furthermore, have some subset of the same client-owned and server-owned tests run against the fakes (for quick presubmit testing) as well as the real implementation (for continuous integration testing) and work with management to make it the responsibility of the Fake owner to debug any discrepancies for either the client- or the server-owned tests. This reverses the pain! API owners, who are in a position to make APIs better, are now the ones experiencing negative incentives when APIs are not easy to use. Previously, when clients felt the pain, they had no recourse other than to file easily-ignored bugs (\"Closed: working as intended\") or contribute changes to the API owners' codebase, hurting their own performance with distractions. This will incentivize API owners to design APIs to be as simple as possible with as few side-effects as possible, and to provide high-quality fakes that make it easy for clients to integrate with the API. Some teams will certainly not like this change at first, but I have seen API teams come to the realization that this is the best choice for the larger effort and implement these practices despite their cost to the team in the short run. Helping management set engineering team objectives may not seem like a typical SETI responsibility, but although management is responsible for setting performance incentives and objectives, they are not well-positioned to understand how the low-level decisions of different teams create harmful interactions and lower cross-team performance, so they need SETI and TE guidance to create an environment that encourages optimal behaviors. Fast, small releasesBeing forced to release more frequently than is required by feature deployment requirements has many beneficial side-effects that make release velocity a goal unto itself. SETIs and TEs faced with big, slow releases work with management to mandate a move to a set of smaller, more frequent releases. As release velocity is ratcheted up, negative behaviours such as too much manual testing or too much internal coupling become more painful, and many optimal behaviors are incentivized. Less coupling between systemsWhen software is released together, it is easy to treat the seams between different components as implementation details. Resulting systems becoming so intertwined (coupled) that responsibilities between them are completely and randomly mixed and their interactions are too complex for any one person to understand. When two components are released separately and at different times, different versions of them must be compatible with one another. Engineers who were previously complacent about this fragility will become fearful of failed releases due to implicit contract changes. They will change their behavior in beneficial ways such as defining the contract between components explicitly and creating regression testing for it. The result is a system composed of robust, self-contained, more easily understood components. Better/More automated testingManual testing becomes more painful as release velocity is ramped up. This will incentivize automated regression, UI and performance tests. This makes the team more agile and able to catch defects sooner and more cheaply. Faster feedbackWhen incremental feature changes can be released to dogfood or other beta channels more frequently, user interaction designers and product managers get much faster feedback about what paths lead to better user engagement and experience than in big, slow releases where an entire feature is deployed simultaneously. This results in a better product. ConclusionThe SETIs and TEs optimize interactions between teams and create fixes for product-wide, cross-team problems in order to improve engineering productivity and velocity. There are many worthwhile projects that EngProd can do using broad knowledge of the system and expertise in refactoring, automation and testing, such as creating test fixtures that enable continuous integration testing or identifying and combining duplicative tests or tools. That said, the biggest problem that EngProd is positioned to solve is to break the chain of local optimizations resulting in cross-team de-optimizations. To that end, discomfort is a tool that can incentivize engineers to find solutions that are optimal for the entire product. We should look for and advocate for these transformative changes.",
        "id": "tag:blogger.com,1999:blog-15045980.post-1842889840930552454",
        "isoDate": "2017-02-13T16:53:00.000Z"
      },
      {
        "title": "Testing on the Toilet: Keep Cause and Effect Clear",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/hA6X8_3SYu8/testing-on-toilet-keep-cause-and-effect.html",
        "pubDate": "2017-01-31T16:35:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by Ben Yu<br /><br />This article was adapted from a  <a href=\"http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html\">Google Testing on the Toilet</a> (TotT) episode. You can download a  <a href=\"https://docs.google.com/document/d/1UX93ZNV3Fy3w2eWo6iY3iqQHfMkElEoJmeK1TJNGesk/edit?usp=sharing\">printer-friendly version</a> of this TotT episode and post it in your office. </i><br /><br /><span style=\"color: purple;\"><b>Can you tell if this test is correct? </b></span><br /><pre style=\"background-color: #fff2cc; border: 1px solid #bbbbbb; color: black; padding: 5px 5px 5px 20px;\">208: @Test public void testIncrement_existingKey() {<br />209:   assertEquals(9, <span style=\"font-weight: bold;\">tally</span>.get(\"key1\"));<br />210: }</pre><br /><span style=\"color: purple;\"><b>It’s impossible to know</b></span> without seeing how the <span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">tally</span> object is set up: <br /><pre style=\"color: black; background-color: #fff2cc; border: 1px solid #bbbbbb; padding: 5px 5px 5px 20px;\">1:   private final Tally <span style=\"font-weight: bold;\">tally</span> = new Tally();<br />2:   @Before public void setUp() {<br />3:      <span style=\"font-weight: bold;\">tally.increment(\"key1\", 8);</span><br />4:      tally.increment(\"key2\", 100);<br />5:      <span style=\"font-weight: bold;\">tally.increment(\"key1\", 0);</span><br />6:      <span style=\"font-weight: bold;\">tally.increment(\"key1\", 1);</span><br />7:   }<br /><span style=\"color: #6aa84f;\">// 200 lines away</span><br />208: @Test public void testIncrement_existingKey() {<br />209:   assertEquals(9, <span style=\"font-weight: bold;\">tally</span>.get(\"key1\"));<br />210: }</pre><br />The problem is that the modification of <span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">key1</span>'s values <i>occurs 200+ lines away from the assertion</i>. Otherwise put, <span style=\"color: purple;\"><b>the <i>cause</i> is hidden far away from the <i>effect</i></b></span>.<br /><br />Instead, <span style=\"color: purple;\"><b>write tests where the effects immediately follow the causes</b></span>. It's how we speak in natural language: “If you drive over the speed limit (<i>cause</i>), you’ll get a traffic ticket (<i>effect</i>).” Once we group the two chunks of code, we easily see what’s going on: <br /><pre style=\"color: black; background-color: #cfe2f3; border: 1px solid #bbbbbb; padding: 5px 5px 5px 20px;\">1:   private final Tally tally = new Tally();<br />2:   @Test public void testIncrement_newKey() {<br />3:     <span style=\"font-weight: bold;\">tally.increment(\"key\", 100);</span><br />5:     assertEquals(100, <span style=\"font-weight: bold;\">tally</span>.get(\"key\"));<br />6:   }<br />7:   @Test public void testIncrement_existingKey() {<br />8:     <span style=\"font-weight: bold;\">tally.increment(\"key\", 8);</span><br />9:     <span style=\"font-weight: bold;\">tally.increment(\"key\", 1);</span><br />10:    assertEquals(9, <span style=\"font-weight: bold;\">tally</span>.get(\"key\"));<br />11:  }<br />12:  @Test public void testIncrement_incrementByZeroDoesNothing() {<br />13:    <span style=\"font-weight: bold;\">tally.increment(\"key\", 8);</span><br />14:    <span style=\"font-weight: bold;\">tally.increment(\"key\", 0);</span><br />15:    assertEquals(8, <span style=\"font-weight: bold;\">tally</span>.get(\"key\"));<br />16:  }</pre><br />This style may require a bit more code.  Each test sets its own input and verifies its own expected output.  <span style=\"color: purple;\"><b>The payback is in more readable code and lower maintenance costs. </b></span><br /><span style=\"color: purple;\"><b><br /></b></span><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=hA6X8_3SYu8:AOixGCq0cZw:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=hA6X8_3SYu8:AOixGCq0cZw:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=hA6X8_3SYu8:AOixGCq0cZw:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/hA6X8_3SYu8\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Ben YuThis article was adapted from a  Google Testing on the Toilet (TotT) episode. You can download a  printer-friendly version of this TotT episode and post it in your office. Can you tell if this test is correct? 208: @Test public void testIncrement_existingKey() {209:   assertEquals(9, tally.get(\"key1\"));210: }It’s impossible to know without seeing how the tally object is set up: 1:   private final Tally tally = new Tally();2:   @Before public void setUp() {3:      tally.increment(\"key1\", 8);4:      tally.increment(\"key2\", 100);5:      tally.increment(\"key1\", 0);6:      tally.increment(\"key1\", 1);7:   }// 200 lines away208: @Test public void testIncrement_existingKey() {209:   assertEquals(9, tally.get(\"key1\"));210: }The problem is that the modification of key1's values occurs 200+ lines away from the assertion. Otherwise put, the cause is hidden far away from the effect.Instead, write tests where the effects immediately follow the causes. It's how we speak in natural language: “If you drive over the speed limit (cause), you’ll get a traffic ticket (effect).” Once we group the two chunks of code, we easily see what’s going on: 1:   private final Tally tally = new Tally();2:   @Test public void testIncrement_newKey() {3:     tally.increment(\"key\", 100);5:     assertEquals(100, tally.get(\"key\"));6:   }7:   @Test public void testIncrement_existingKey() {8:     tally.increment(\"key\", 8);9:     tally.increment(\"key\", 1);10:    assertEquals(9, tally.get(\"key\"));11:  }12:  @Test public void testIncrement_incrementByZeroDoesNothing() {13:    tally.increment(\"key\", 8);14:    tally.increment(\"key\", 0);15:    assertEquals(8, tally.get(\"key\"));16:  }This style may require a bit more code.  Each test sets its own input and verifies its own expected output.  The payback is in more readable code and lower maintenance costs.",
        "id": "tag:blogger.com,1999:blog-15045980.post-6285892944359784274",
        "isoDate": "2017-01-31T16:35:00.000Z"
      },
      {
        "title": "Happy 10th Birthday Google Testing Blog!",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/gqP7o96IiVA/happy-10th-birthday-google-testing-blog.html",
        "pubDate": "2017-01-21T17:00:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by <a href=\"http://anthonyvallone.com/\">Anthony Vallone</a></i><br /><br />Ten years ago today, the <a href=\"https://testing.googleblog.com/2007/01/introducing-testing-on-toilet.html\">first Google Testing Blog article</a> was posted (<a href=\"https://testing.googleblog.com/2007/01/welcome-to-google-testing-blog.html\">official announcement</a> 2 days later). Over the years, Google engineers have used this blog to help advance the test engineering discipline. We have shared information about our testing technologies, strategies, and theories; discussed what code quality really means; described how our teams are organized for optimal productivity; announced new tooling; and invited readers to speak at and attend the annual <a href=\"https://developers.google.com/google-test-automation-conference/\">Google Test Automation Conference</a>.<br /><br /><div style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-hVhU3V32Js0/WH-YvGY1VDI/AAAAAAAAAQc/MaV9areIt_M7M7X2Kho1OQUax-mUyElmwCLcB/s1600/old-blog.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"70\" src=\"https://2.bp.blogspot.com/-hVhU3V32Js0/WH-YvGY1VDI/AAAAAAAAAQc/MaV9areIt_M7M7X2Kho1OQUax-mUyElmwCLcB/s400/old-blog.png\" width=\"400\" /></a></div><div style=\"text-align: center;\"><i><span style=\"font-size: x-small;\">Google Testing Blog banner in 2007</span></i></div><br /><br />The blog has enjoyed excellent readership. There have been over 10 million page views of the blog since it was created, and there are currently about 100 to 200 thousand views per month.<br /><br />This blog is made possible by many Google engineers who have volunteered time to author and review content on a regular basis in the interest of sharing. Thank you to all the contributors and our readers!<br /><br />Please leave a comment if you have a story to share about how this blog has helped you.  <br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=gqP7o96IiVA:bB3h-ODJdW8:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=gqP7o96IiVA:bB3h-ODJdW8:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=gqP7o96IiVA:bB3h-ODJdW8:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/gqP7o96IiVA\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Anthony ValloneTen years ago today, the first Google Testing Blog article was posted (official announcement 2 days later). Over the years, Google engineers have used this blog to help advance the test engineering discipline. We have shared information about our testing technologies, strategies, and theories; discussed what code quality really means; described how our teams are organized for optimal productivity; announced new tooling; and invited readers to speak at and attend the annual Google Test Automation Conference.Google Testing Blog banner in 2007The blog has enjoyed excellent readership. There have been over 10 million page views of the blog since it was created, and there are currently about 100 to 200 thousand views per month.This blog is made possible by many Google engineers who have volunteered time to author and review content on a regular basis in the interest of sharing. Thank you to all the contributors and our readers!Please leave a comment if you have a story to share about how this blog has helped you.",
        "id": "tag:blogger.com,1999:blog-15045980.post-1618931592231142334",
        "isoDate": "2017-01-21T17:00:00.000Z"
      },
      {
        "title": "Announcing OSS-Fuzz: Continuous Fuzzing for Open Source Software",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/qrtsgLPSaTc/announcing-oss-fuzz-continuous-fuzzing.html",
        "pubDate": "2016-12-01T17:00:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>By Mike Aizatsky, Kostya Serebryany (Software Engineers, Dynamic Tools); Oliver Chang, Abhishek Arya (Security Engineers, Google Chrome); and Meredith Whittaker (Open Research Lead).&nbsp;</i><br /><br /><strong>We are happy to announce <a href=\"https://github.com/google/oss-fuzz\">OSS-Fuzz</a>, a new Beta program developed over the past years with the <a href=\"https://www.coreinfrastructure.org/\">Core Infrastructure Initiative community</a>. This program will provide continuous fuzzing for select core open source software.</strong><br /><strong><br /></strong>Open source software is the backbone of the many apps, sites, services, and networked things that make up \"the internet.\" It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it. <br /><br /><a href=\"https://en.wikipedia.org/wiki/Heartbleed\">Recent</a> <a href=\"https://en.wikipedia.org/wiki/Stagefright_(bug)\">security</a> <a href=\"https://googlechromereleases.blogspot.com/2016/09/stable-channel-updates-for-chrome-os.html\">stories</a>confirm that errors like<a href=\"https://en.wikipedia.org/wiki/Buffer_overflow\">buffer overflow</a> and<a href=\"https://en.wikipedia.org/wiki/Dangling_pointer\">use-after-free</a> can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers. That's where<a href=\"https://en.wikipedia.org/wiki/Fuzz_testing\">fuzz testing</a> comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly. <br /><br />In recent years, several efficient general purpose fuzzing engines have been implemented (e.g. <a href=\"http://lcamtuf.coredump.cx/afl/\">AFL</a> and <a href=\"http://libfuzzer.info/\">libFuzzer</a>), and we use them to <a href=\"https://security.googleblog.com/2016/08/guided-in-process-fuzzing-of-chrome.html\">fuzz various components of the Chrome browser</a>. These fuzzers, when combined with <a href=\"https://github.com/google/sanitizers\">Sanitizers</a>, can help find security vulnerabilities (e.g. buffer overflows, use-after-free, bad casts, integer overflows, etc), stability bugs (e.g. null dereferences, memory leaks, out-of-memory, assertion failures, etc) and <a href=\"https://blog.fuzzing-project.org/31-Fuzzing-Math-miscalculations-in-OpenSSLs-BN_mod_exp-CVE-2015-3193.html\">sometimes</a>even logical bugs. <br /><br />OSS-Fuzz's goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution. OSS-Fuzz combines various fuzzing engines (initially, libFuzzer) with Sanitizers (initially, <a href=\"http://clang.llvm.org/docs/AddressSanitizer.html\">AddressSanitizer</a>) and provides a massive distributed execution environment powered by <a href=\"https://github.com/google/oss-fuzz/blob/master/docs/clusterfuzz.md\">ClusterFuzz</a>. <br /><h2>Early successes</h2>Our initial trials with OSS-Fuzz have had good results. An example is the <a href=\"https://www.freetype.org/\">FreeType</a> library, which is used on over a <a href=\"https://en.wikipedia.org/wiki/FreeType#Users\">billion devices</a> to display text (and which might even be rendering the characters you are reading now). It is important for FreeType to be stable and secure in an age when fonts are loaded over the Internet. Werner Lemberg, one of the FreeType developers, <a href=\"https://savannah.nongnu.org/search/?type_of_search=bugs&amp;words=LibFuzzer&amp;offset=0&amp;max_rows=100#results\">was</a>an early adopter of OSS-Fuzz. Recently the <a href=\"http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/src/tools/ftfuzzer/ftfuzzer.cc\">FreeType fuzzer</a> found a <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53\">new heap buffer overflow</a> only a few hours after the source change: <br /><br /><pre class=\"prettyprint\">ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa <br />READ of size 2 at 0x615000000ffa thread T0<br />SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)<br />   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31<br /></pre><br />OSS-Fuzz automatically<a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53\"> notified</a>the maintainer, who<a href=\"http://git.savannah.gnu.org/cgit/freetype/freetype2.git/commit/?id=7eeaf986b5ebb43cbbc8368c6e528ac311d87805\">fixed</a> the bug; then OSS-Fuzz automatically<a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53#c3\">confirmed</a> the fix. All in one day! You can see the <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=type=Bug-Security,Bug%20-component:Infra%20status:Fixed,Verified&amp;sort=-id&amp;colspec=ID%20Type%20Component%20Status%20Library%20Reported%20Owner%20Summary\">full list</a> of fixed and disclosed bugs found by OSS-Fuzz so far. <br /><h2>Contributions and feedback are welcome</h2>OSS-Fuzz has already found <strong><a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&amp;q=-component%3AInfra+-status%3ADuplicate%2CWontFix&amp;sort=-id&amp;colspec=ID+Type+Component+Status+Library+Reported+Owner+Summary&amp;cells=ids\">150 bugs</a></strong> in several widely used open source <a href=\"https://github.com/google/oss-fuzz/tree/master/projects\">projects</a> (and churns <strong>~4 trillion test cases</strong> a week). With your help, we can make fuzzing a standard part of open source development, and work with the broader community of developers and security testers to ensure that bugs in critical open source applications, libraries, and APIs are discovered and fixed. We believe that this approach to automated security testing will result in real improvements to the security and stability of open source software. <br /><br />OSS-Fuzz is launching in Beta right now, and will be accepting suggestions for candidate open source projects. In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure, a general heuristic that we are intentionally leaving open to interpretation at this early stage. See more details and instructions on how to apply <a href=\"https://github.com/google/oss-fuzz#accepting-new-projects\">here</a>. <br /><br />Once a project is signed up for OSS-Fuzz, it is automatically subject to the 90-day disclosure deadline for newly reported bugs in our <a href=\"https://bugs.chromium.org/p/oss-fuzz/issues/list\">tracker</a> (see details <a href=\"https://github.com/google/oss-fuzz#bug-disclosure-guidelines\">here</a>). This matches industry's <a href=\"https://googleprojectzero.blogspot.com/2015/02/feedback-and-data-driven-updates-to.html\">best practices</a> and improves end-user security and stability by getting patches to users faster. <br /><br />Help us ensure this program is truly serving the open source community and the internet which relies on this critical software, contribute and leave your feedback on <a href=\"https://github.com/google/oss-fuzz\">GitHub</a>.<div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=qrtsgLPSaTc:MCGyvj_AHeQ:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=qrtsgLPSaTc:MCGyvj_AHeQ:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=qrtsgLPSaTc:MCGyvj_AHeQ:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/qrtsgLPSaTc\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By Mike Aizatsky, Kostya Serebryany (Software Engineers, Dynamic Tools); Oliver Chang, Abhishek Arya (Security Engineers, Google Chrome); and Meredith Whittaker (Open Research Lead).&nbsp;We are happy to announce OSS-Fuzz, a new Beta program developed over the past years with the Core Infrastructure Initiative community. This program will provide continuous fuzzing for select core open source software.Open source software is the backbone of the many apps, sites, services, and networked things that make up \"the internet.\" It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it. Recent security storiesconfirm that errors likebuffer overflow anduse-after-free can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers. That's wherefuzz testing comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly. In recent years, several efficient general purpose fuzzing engines have been implemented (e.g. AFL and libFuzzer), and we use them to fuzz various components of the Chrome browser. These fuzzers, when combined with Sanitizers, can help find security vulnerabilities (e.g. buffer overflows, use-after-free, bad casts, integer overflows, etc), stability bugs (e.g. null dereferences, memory leaks, out-of-memory, assertion failures, etc) and sometimeseven logical bugs. OSS-Fuzz's goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution. OSS-Fuzz combines various fuzzing engines (initially, libFuzzer) with Sanitizers (initially, AddressSanitizer) and provides a massive distributed execution environment powered by ClusterFuzz. Early successesOur initial trials with OSS-Fuzz have had good results. An example is the FreeType library, which is used on over a billion devices to display text (and which might even be rendering the characters you are reading now). It is important for FreeType to be stable and secure in an age when fonts are loaded over the Internet. Werner Lemberg, one of the FreeType developers, wasan early adopter of OSS-Fuzz. Recently the FreeType fuzzer found a new heap buffer overflow only a few hours after the source change: ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa READ of size 2 at 0x615000000ffa thread T0SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31OSS-Fuzz automatically notifiedthe maintainer, whofixed the bug; then OSS-Fuzz automaticallyconfirmed the fix. All in one day! You can see the full list of fixed and disclosed bugs found by OSS-Fuzz so far. Contributions and feedback are welcomeOSS-Fuzz has already found 150 bugs in several widely used open source projects (and churns ~4 trillion test cases a week). With your help, we can make fuzzing a standard part of open source development, and work with the broader community of developers and security testers to ensure that bugs in critical open source applications, libraries, and APIs are discovered and fixed. We believe that this approach to automated security testing will result in real improvements to the security and stability of open source software. OSS-Fuzz is launching in Beta right now, and will be accepting suggestions for candidate open source projects. In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure, a general heuristic that we are intentionally leaving open to interpretation at this early stage. See more details and instructions on how to apply here. Once a project is signed up for OSS-Fuzz, it is automatically subject to the 90-day disclosure deadline for newly reported bugs in our tracker (see details here). This matches industry's best practices and improves end-user security and stability by getting patches to users faster. Help us ensure this program is truly serving the open source community and the internet which relies on this critical software, contribute and leave your feedback on GitHub.",
        "id": "tag:blogger.com,1999:blog-15045980.post-2730397717300250277",
        "isoDate": "2016-12-01T17:00:00.000Z"
      },
      {
        "title": "What Test Engineers do at Google: Building Test Infrastructure",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/rRmfQrYK4sU/what-test-engineers-do-at-google.html",
        "pubDate": "2016-11-18T17:10:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>Author: Jochen Wuttke</i><br /><br />In a recent post, we broadly talked about <a href=\"https://testing.googleblog.com/2016/09/what-test-engineers-do-at-google.html\">What Test Engineers do at Google</a>. In this post, I talk about one aspect of the work TEs may do: building and improving test infrastructure to make engineers more productive. <br /><br /><strong>Refurbishing legacy systems makes new tools necessary</strong><br />A few years ago, I joined an engineering team that was working on replacing a legacy system with a new implementation. Because building the replacement would take several years, we had to keep the legacy system operational and even add features, while building the replacement so there would be no impact on our external users. <br /><br />The legacy system was so complex and brittle that the engineers spent most of their time triaging and fixing bugs and <a href=\"https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html\">flaky tests</a>, but had little time to implement new features. The goal for the rewrite was to learn from the legacy system and to build something that was easier to maintain and extend. As the team's TE, my job was to understand what caused the high maintenance cost and how to improve on it. I found two main causes: <br /><ul><li>Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code. </li><li>The infrastructure used for the end-to-end tests had no good way to create and inject fakes or mocks for these services.  As a result, the tests had to run the large number of servers for all these external dependencies. This led to very large and brittle tests that our <a href=\"https://testing.googleblog.com/2011/06/testing-at-speed-and-scale-of-google.html\">existing test execution infrastructure</a> was not able to handle reliably.</li></ul><strong>Exploring solutions</strong><br />At first, I explored if I could split the large tests into smaller ones that would test specific functionality and depend on fewer external services. This proved impossible, because of the poorly structured legacy code. Making this approach work would have required refactoring the entire system and its dependencies, not just the parts my team owned. <br /><br />In my second approach, I also focussed on large tests and tried to mock services that were not required for the functionality under test. This also proved very difficult, because dependencies changed often and individual dependencies were hard to trace in a graph of over 200 services. Ultimately, this approach just shifted the required effort from maintaining test code to maintaining test dependencies and mocks. <br /><br />My third and final approach, illustrated in the figure below, made small tests more powerful. In the typical end-to-end test we faced, the client made <strong><a href=\"https://en.wikipedia.org/wiki/Remote_procedure_call\">RPC</a>calls</strong> to several services, which in turn made RPC calls to other services. Together the client and the transitive closure over all backend services formed a large <a href=\"https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)\">graph</a> (not <a href=\"https://en.wikipedia.org/wiki/Tree_(graph_theory)\">tree</a>!) of dependencies, which all had to be up and running for the end-to-end test. The new model changes how we test client and service integration. Instead of running the client on inputs that will somehow trigger RPC calls, we write unit tests for the code making <strong>method calls</strong> to the RPC stub. The stub itself is mocked with a common mocking framework like Mockito in Java. For each such test, a second test verifies that the data used to drive that mock \"makes sense\" to the actual service. This is also done with a unit test, where a replay client uses the same data the RPC mock uses to call the RPC handler method of the service. <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-XV6lUxNAMgs/WC81WN0TtHI/AAAAAAAAAQA/MBWOlMRo73MhXNfDF4E6LkPy7YuukVEXwCLcB/s1600/img1.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"296\" src=\"https://2.bp.blogspot.com/-XV6lUxNAMgs/WC81WN0TtHI/AAAAAAAAAQA/MBWOlMRo73MhXNfDF4E6LkPy7YuukVEXwCLcB/s640/img1.png\" width=\"640\" /></a></div><br />This pattern of integration testing applies to any RPC call, so the RPC calls made by a backend server to another backend can be tested just as well as front-end client calls. When we apply this approach consistently, we benefit from smaller tests that still test correct integration behavior, and make sure that the behavior we are testing is \"real\". <br /><br />To arrive at this solution, I had to build, evaluate, and discard several prototypes. While it took a day to build a proof-of-concept for this approach, it took me and another engineer a year to implement a finished tool developers could use. <br /><br /><strong>Adoption</strong><br />The engineers embraced the new solution very quickly when they saw that the new framework removes large amounts of boilerplate code from their tests. To further drive its adoption, I organized multi-day events with the engineering team where we focussed on migrating test cases. It took a few months to migrate all existing unit tests to the new framework, close gaps in coverage, and create the new tests that validate the mocks. Once we converted about 80% of the tests, we started comparing the efficacy of the new tests and the existing end-to-end tests. <br /><br />The results are very good: <br /><ul><li>The new tests are as effective in finding bugs as the end-to-end tests are. </li><li>The new tests run in about 3 minutes instead of 30 minutes for the end-to-end tests. </li><li>The client side tests are 0% flaky. The verification tests are usually less flaky than the end-to-end tests, and never more.</li></ul>Additionally, the new tests are unit tests, so you can run them in your IDE and step through them to debug. These results allowed us to run the end-to-end tests very rarely, only to detect misconfigurations of the interacting services, but not as functional tests. <br /><br />Building and improving test infrastructure to help engineers be more productive is one of the many things test engineers do at Google. Running this project from requirements gathering all the way to a finished product gave me the opportunity to design and implement several prototypes, drive the full implementation of one solution, lead engineering teams to adoption of the new framework, and integrate feedback from engineers and actual measurements into the continuous refinement of the tool. <div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=rRmfQrYK4sU:-2oaDPyZQEA:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=rRmfQrYK4sU:-2oaDPyZQEA:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=rRmfQrYK4sU:-2oaDPyZQEA:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/rRmfQrYK4sU\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "Author: Jochen WuttkeIn a recent post, we broadly talked about What Test Engineers do at Google. In this post, I talk about one aspect of the work TEs may do: building and improving test infrastructure to make engineers more productive. Refurbishing legacy systems makes new tools necessaryA few years ago, I joined an engineering team that was working on replacing a legacy system with a new implementation. Because building the replacement would take several years, we had to keep the legacy system operational and even add features, while building the replacement so there would be no impact on our external users. The legacy system was so complex and brittle that the engineers spent most of their time triaging and fixing bugs and flaky tests, but had little time to implement new features. The goal for the rewrite was to learn from the legacy system and to build something that was easier to maintain and extend. As the team's TE, my job was to understand what caused the high maintenance cost and how to improve on it. I found two main causes: Tight coupling and insufficient abstraction made unit testing very hard, and as a consequence, a lot of end-to-end tests served as functional tests of that code. The infrastructure used for the end-to-end tests had no good way to create and inject fakes or mocks for these services.  As a result, the tests had to run the large number of servers for all these external dependencies. This led to very large and brittle tests that our existing test execution infrastructure was not able to handle reliably.Exploring solutionsAt first, I explored if I could split the large tests into smaller ones that would test specific functionality and depend on fewer external services. This proved impossible, because of the poorly structured legacy code. Making this approach work would have required refactoring the entire system and its dependencies, not just the parts my team owned. In my second approach, I also focussed on large tests and tried to mock services that were not required for the functionality under test. This also proved very difficult, because dependencies changed often and individual dependencies were hard to trace in a graph of over 200 services. Ultimately, this approach just shifted the required effort from maintaining test code to maintaining test dependencies and mocks. My third and final approach, illustrated in the figure below, made small tests more powerful. In the typical end-to-end test we faced, the client made RPCcalls to several services, which in turn made RPC calls to other services. Together the client and the transitive closure over all backend services formed a large graph (not tree!) of dependencies, which all had to be up and running for the end-to-end test. The new model changes how we test client and service integration. Instead of running the client on inputs that will somehow trigger RPC calls, we write unit tests for the code making method calls to the RPC stub. The stub itself is mocked with a common mocking framework like Mockito in Java. For each such test, a second test verifies that the data used to drive that mock \"makes sense\" to the actual service. This is also done with a unit test, where a replay client uses the same data the RPC mock uses to call the RPC handler method of the service. This pattern of integration testing applies to any RPC call, so the RPC calls made by a backend server to another backend can be tested just as well as front-end client calls. When we apply this approach consistently, we benefit from smaller tests that still test correct integration behavior, and make sure that the behavior we are testing is \"real\". To arrive at this solution, I had to build, evaluate, and discard several prototypes. While it took a day to build a proof-of-concept for this approach, it took me and another engineer a year to implement a finished tool developers could use. AdoptionThe engineers embraced the new solution very quickly when they saw that the new framework removes large amounts of boilerplate code from their tests. To further drive its adoption, I organized multi-day events with the engineering team where we focussed on migrating test cases. It took a few months to migrate all existing unit tests to the new framework, close gaps in coverage, and create the new tests that validate the mocks. Once we converted about 80% of the tests, we started comparing the efficacy of the new tests and the existing end-to-end tests. The results are very good: The new tests are as effective in finding bugs as the end-to-end tests are. The new tests run in about 3 minutes instead of 30 minutes for the end-to-end tests. The client side tests are 0% flaky. The verification tests are usually less flaky than the end-to-end tests, and never more.Additionally, the new tests are unit tests, so you can run them in your IDE and step through them to debug. These results allowed us to run the end-to-end tests very rarely, only to detect misconfigurations of the interacting services, but not as functional tests. Building and improving test infrastructure to help engineers be more productive is one of the many things test engineers do at Google. Running this project from requirements gathering all the way to a finished product gave me the opportunity to design and implement several prototypes, drive the full implementation of one solution, lead engineering teams to adoption of the new framework, and integrate feedback from engineers and actual measurements into the continuous refinement of the tool.",
        "id": "tag:blogger.com,1999:blog-15045980.post-4609725131224908918",
        "isoDate": "2016-11-18T17:10:00.000Z"
      },
      {
        "title": "Hackable Projects - Pillar 3: Infrastructure",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/Mm1LcM6IYyI/hackable-projects-pillar-3.html",
        "pubDate": "2016-11-10T19:07:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>By: Patrik Höglund</i><br /><i><br /></i><i>This is the third article in our series on Hackability; also see <a href=\"https://testing.googleblog.com/2016/08/hackable-projects.html\">the first</a> and <a href=\"https://testing.googleblog.com/2016/10/hackable-projects-pillar-2-debuggability.html\">second article</a>.</i><br /><br /><br />We have seen in our previous articles how <i>Code Health</i> and <i>Debuggability</i> can make a project much easier to work on. The third pillar is a solid infrastructure that gets accurate feedback to your developers as fast as possible. Speed is going to be major theme in this article, and we’ll look at a number of things you can do to make your project easier to hack on.<br /><br /><h2>Build Systems Speed</h2><div><br /></div><div style=\"border: 1px solid black; padding: 10px;\">Question: What’s a change you’d really like to see in our development tools?<br /><br />“I feel like this question gets asked almost every time, and I always give the same answer: <br />&nbsp;I would like them to be faster.” <br />&nbsp; &nbsp; &nbsp; &nbsp; -- <a href=\"http://research.google.com/pubs/author37504.html\">Ian Lance Taylor</a></div><br /><br />Replace make with <a href=\"https://ninja-build.org/\">ninja</a>. Use the <a href=\"https://en.wikipedia.org/wiki/Gold_(linker)\">gold</a> linker instead of ld. Detect and delete dead code in your project (perhaps using coverage tools). Reduce the number of dependencies, and enforce dependency rules so new ones are not added lightly. Give the developers faster machines. Use distributed build, which is available with many open-source continuous integration systems (or use Google’s system, <a href=\"http://www.bazel.io/\">Bazel</a>!). You should do everything you can to make the build faster.<br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-olHylBpyDck/WCTBuNiPfQI/AAAAAAAAAPM/vqGFMN1aMkUe7ZyYLO4dH0JGjerFPOtvQCLcB/s1600/image04.jpg\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"226\" src=\"https://2.bp.blogspot.com/-olHylBpyDck/WCTBuNiPfQI/AAAAAAAAAPM/vqGFMN1aMkUe7ZyYLO4dH0JGjerFPOtvQCLcB/s400/image04.jpg\" width=\"400\" /></a></div><div style=\"text-align: center;\"><i>Figure 1: “Cheetah chasing its prey” by Marlene Thyssen.</i></div><br /><br />Why is that? There’s a tremendous difference in hackability if it takes 5 seconds to build and test versus one minute, or even 20 minutes, to build and test. Slow feedback cycles kill hackability, for many reasons:<br /><ul><li>Build and test times longer than a handful of seconds cause many developers’ minds to wander, taking them out of the zone.</li><li>Excessive build or release times* makes tinkering and refactoring much harder. All developers have a threshold when they start hedging (e.g. “I’d like to remove this branch, but I don’t know if I’ll break the iOS build”) which causes refactoring to not happen.</li></ul><br /><div><div>* The worst I ever heard of was an OS that took 24 hours to build!</div><div><br /><br /></div><div>How do you actually make fast build systems? There are some suggestions in the first paragraph above, but the best general suggestion I can make is to have a few engineers on the project who deeply understand the build systems and have the time to continuously improve them. The main axes of improvement are:</div><div><ol><li>Reduce the amount of code being compiled.</li><li>Replace tools with faster counterparts.</li><li>Increase processing power, maybe through parallelization or distributed systems.</li></ol></div><div>Note that there is a big difference between <i>full builds</i> and <i>incremental builds</i>. Both should be as fast as possible, but incremental builds are by far the most important to optimize. The way you tackle the two is different. For instance, reducing the total number of source files will make a full build faster, but it may not make an incremental build faster.&nbsp;</div><div><br /></div><div>To get faster incremental builds, in general, you need to make each source file as decoupled as possible from the rest of the code base. The less a change ripples through the codebase, the less work to do, right? See <a href=\"https://testing.googleblog.com/2016/08/hackable-projects.html\">“Loose Coupling and Testability” in Pillar 1</a> for more on this subject. The exact mechanics of dependencies and interfaces depends on programming language - one of the hardest to get right is unsurprisingly C++, where you need to be <a href=\"https://google.github.io/styleguide/cppguide.html#Forward_Declarations\">disciplined with includes and forward declarations</a> to get any kind of incremental build performance.&nbsp;</div><div><br /></div><div>Build scripts and makefiles should be held to standards as high as the code itself. Technical debt and unnecessary dependencies have a tendency to accumulate in build scripts, because no one has the time to understand and fix them. Avoid this by addressing the technical debt as you go.<br /><br /></div><div><h2>Continuous Integration and Presubmit Queues</h2><div><br /></div>You should build and run tests on all platforms you release on. For instance, if you release on all the major desktop platforms, but all your developers are on Linux, this becomes extra important. It’s bad for hackability to update the repo, build on Windows, and find that lots of stuff is broken. It’s even worse if broken changes start to stack on top of each other. I think we all know that terrible feeling: when you’re not sure your change is the one that broke things.</div><div><br /></div><div>At a minimum, you should build and test on all platforms, but it’s even better if you do it in presubmit. The <a href=\"https://dev.chromium.org/developers/testing/commit-queue\">Chromium submit queue</a> does this. It has developed over the years so that a normal patch builds and tests on about 30 different build configurations before commit. This is necessary for the 400-patches-per-day velocity of the Chrome project. Most projects obviously don’t have to go that far. Chromium’s infrastructure is based off <a href=\"http://buildbot.net/\">BuildBot</a>, but there are many other job scheduling systems depending on your needs.</div></div><div><br /></div><div><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://3.bp.blogspot.com/-24eTelErgpc/WCTCuC0wDsI/AAAAAAAAAPU/it0o56UJybMZaZPBOz-MDLhbe-ljyqAtwCLcB/s1600/image02.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://3.bp.blogspot.com/-24eTelErgpc/WCTCuC0wDsI/AAAAAAAAAPU/it0o56UJybMZaZPBOz-MDLhbe-ljyqAtwCLcB/s1600/image02.png\" /></a></div><div style=\"text-align: center;\"><i>Figure 2: How a Chromium patch is tested.</i></div></div><div><div><br /></div><div>As we discussed in Build Systems, speed and correctness are critical here. It takes a lot of ongoing work to keep build, tests, and presubmits fast and free of flakes. You should never accept flakes, since developers very quickly lose trust in flaky tests and systems. Tooling can help a bit with this; for instance, see the <a href=\"http://chromium-try-flakes.appspot.com/\">Chromium flakiness dashboard</a>.<br /><br /></div><div><h2>Test Speed</h2></div><div><br /></div><div>Speed is a feature, and this is particularly true for developer infrastructure. In general, the longer a test takes to execute, the less valuable it is. My rule of thumb is: if it takes more than a minute to execute, its value is greatly diminished. There are of course some exceptions, such as soak tests or certain performance tests.&nbsp;</div></div><div><br /></div><div><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://4.bp.blogspot.com/-TnsfrijE-HQ/WCTDKNXmWeI/AAAAAAAAAPc/3J0OEpWIMegHrhD4XCMnwaRoUd6qUILQwCLcB/s1600/image07.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://4.bp.blogspot.com/-TnsfrijE-HQ/WCTDKNXmWeI/AAAAAAAAAPc/3J0OEpWIMegHrhD4XCMnwaRoUd6qUILQwCLcB/s1600/image07.png\" /></a></div><div style=\"text-align: center;\"><i>Figure 3. Test value.</i></div></div><div><div><br /></div><div>If you have tests that are slower than 60 seconds, they better be incredibly reliable and easily debuggable. A flaky test that takes several minutes to execute often has negative value because it slows down all work in the code it covers. You probably want to build better integration tests on lower levels instead, so you can make them faster and more reliable.</div><div><br /></div><div>If you have many engineers on a project, reducing the time to run the tests can have a big impact. This is one reason why it’s great to have <a href=\"https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html\">SETIs</a> or the equivalent. There are many things you can do to improve test speed:</div><div><ul><li><a href=\"https://github.com/google/gtest-parallel\">Sharding and parallelization</a>. Add more machines to your continuous build as your test set or number of developers grows.</li><li>Continuously measure how long it takes to run one build+test cycle in your continuous build, and have someone take action when it gets slower.</li><li>Remove tests that don’t pull their weight. If a test is really slow, it’s often because of <a href=\"http://googletesting.blogspot.se/2008/10/gui-testing-dont-sleep-without.html\">poorly written wait conditions</a> or because the test bites off more than it can chew (maybe that unit test doesn’t have to process 15000 audio frames, maybe 50 is enough).</li><li>If you have tests that bring up a local server stack, for instance inter-server integration tests, making your servers boot faster is going to make the tests faster as well. Faster production code is faster to test! See Running on Localhost, in <a href=\"https://testing.googleblog.com/2016/10/hackable-projects-pillar-2-debuggability.html\">Pillar 2</a> for more on local server stacks.</li></ul></div><div><br /><h2>Workflow Speed</h2></div><div><br />We’ve talked about fast builds and tests, but the core developer workflows are also important, of course. Chromium undertook multi-year project to switch from Subversion to Git, partly because Subversion was becoming too slow. You need to keep track of your core workflows as your project grows. Your version control system may work fine for years, but become too slow once the project becomes big enough. Bug search and management must also be robust and fast, since this is generally systems developers touch every day.<br /><br /></div><div><h2>Release Often</h2></div><div><br />It aids hackability to <a href=\"https://en.wikipedia.org/wiki/Release_early,_release_often\">deploy to real users as fast as possible</a>. No matter how good your product's tests are, there's always a risk that there's something you haven't thought of. If you’re building a service or web site, you should aim to deploy multiple times per week. For client projects, <a href=\"https://blog.chromium.org/2010/07/release-early-release-often.html\">Chrome’s six-weeks cycle</a> is a good goal to aim for.</div><div><br /></div><div>You should invest in infrastructure and tests that give you the confidence to do this - you don’t want to push something that’s broken. Your developers will thank you for it, since it makes their jobs so much easier. By releasing often, you mitigate risk, and developers will rush less to hack late changes in the release (since they know the next release isn’t far off).<br /><br /></div><div><h2>Easy Reverts</h2></div><div><br />If you look in the commit log for the Chromium project, you will see that a significant percentage of the commits are reverts of a previous commits. In Chromium, bad commits quickly become costly because they impede other engineers, and the high velocity can cause good changes to stack onto bad changes.&nbsp;</div></div><div><br /></div><div><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://4.bp.blogspot.com/-AV5gYtJ0eXY/WCTD6w-yzjI/AAAAAAAAAPg/XrnpxKHkF88qkzjLV28E1KmoKz4tvvfUwCLcB/s1600/image08.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://4.bp.blogspot.com/-AV5gYtJ0eXY/WCTD6w-yzjI/AAAAAAAAAPg/XrnpxKHkF88qkzjLV28E1KmoKz4tvvfUwCLcB/s1600/image08.png\" /></a></div></div><div><div><div style=\"text-align: center;\"><i>Figure 4: Chromium’s revert button.</i></div></div><div><div style=\"text-align: center;\"><br /></div></div><div><br />This is why the policy is “revert first, ask questions later”. I believe a revert-first policy is good for small projects as well, since it creates a clear expectation that the product/tools/dev environment should be <i>working</i> at all times (and if it doesn’t, a recent change should probably be reverted).</div><div><br /></div><div>It has a wonderful effect when a revert is simple to make. You can suddenly make <i>speculative reverts</i> if a test went flaky or a performance test regressed. It follows that if a patch is easy to revert, so is the inverse (i.e. reverting the revert or relanding the patch). So if you were wrong and that patch wasn’t guilty after all, it’s simple to re-land it again and try reverting another patch. Developers might initially balk at this (because it can’t possibly be their patch, right?), but they usually come around when they realize the benefits.</div><div><br /></div><div>For many projects, a revert can simply be</div><div><span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">git revert 9fbadbeef</span></div><div><span style=\"font-family: &quot;courier new&quot; , &quot;courier&quot; , monospace;\">git push origin master</span></div><div><br /></div><div><br />If your project (wisely) involves code review, it will behoove you to add something like Chromium’s revert button that I mentioned above. The revert button will create a special patch that bypasses review and tests (since we can assume a clean revert takes us back to a more stable state rather than the opposite). See <a href=\"https://testing.googleblog.com/2016/08/hackable-projects.html\">Pillar 1</a> for more on code review and its benefits.</div><div><br /></div><div>For some projects, reverts are going to be harder, especially if you have a slow or laborious release process. Even if you release often, you could still have problems if a revert involves state migrations in your live services (for instance when rolling back a database schema change). You need to have a strategy to deal with such state changes.&nbsp;</div><div><br /></div><div><br /></div><div>Reverts must always put you back to safer ground, and everyone must be confident they can safely revert. If not, you run the risk of massive fire drills and lost user trust if a bad patch makes it through the tests and you can’t revert it.<br /><br /></div><div><h2>Performance Tests: Measure Everything</h2></div><div><br />Is it critical that your app starts up within a second? Should your app always render at 60 fps when it’s scrolled up or down? Should your web server always serve a response within 100 ms? Should your mobile app be smaller than 8 MB? If so, you should make a performance test for that. Performance tests aid hackability since developers can quickly see how their change affects performance and thus prevent performance regressions from making it into the wild.</div><div><br /></div><div>You should run your automated performance tests on the same device; all devices are different and this will reflect in the numbers. This is fairly straightforward if you have a decent continuous integration system that runs tests sequentially on a known set of worker machines. It’s harder if you need to run on physical phones or tablets, but it can be done.&nbsp;</div><div><br /></div><div>A test can be as simple as invoking a particular algorithm and measuring the time it takes to execute it (median and 90th percentile, say, over N runs).</div></div><div><br /></div><div><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://1.bp.blogspot.com/-wOxZrXJCrOw/WCTEakhXI4I/AAAAAAAAAPk/EfuFAdV-3YkW-IOLO3Lgv2zursRFmP00wCLcB/s1600/image03.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"172\" src=\"https://1.bp.blogspot.com/-wOxZrXJCrOw/WCTEakhXI4I/AAAAAAAAAPk/EfuFAdV-3YkW-IOLO3Lgv2zursRFmP00wCLcB/s400/image03.png\" width=\"400\" /></a></div><div style=\"text-align: center;\"><i>Figure 5: A VP8-in-WebRTC regression (<a href=\"https://bugs.chromium.org/p/chromium/issues/detail?id=597705\">bug</a>) and its fix displayed in a Catapult dashboard.</i></div></div><div><div><div style=\"text-align: center;\"><br /></div></div><div><br />Write your test so it outputs performance numbers you care about. But what to do with those numbers? Fortunately, Chrome’s performance test framework <a href=\"https://github.com/catapult-project/catapult\">has been open-sourced</a>, which means you can set up a dashboard, with automatic regression monitoring, with minimal effort. The test framework also includes the powerful <a href=\"https://github.com/catapult-project/catapult/blob/master/telemetry/README.md\">Telemetry</a> framework which can run actions on web pages and Android apps and report performance results. Telemetry and Catapult are battle-tested by use in the Chromium project and are capable of running on a wide set of devices, while getting the maximum of usable performance data out of the devices.<br /><br /></div><div><h2>Sources</h2></div><div><br />Figure 1: By Malene Thyssen (Own work) [CC BY-SA 3.0 (<a href=\"http://creativecommons.org/licenses/by-sa/3.0\">http://creativecommons.org/licenses/by-sa/3.0</a>)], via Wikimedia Commons</div></div><div><br /></div><div><br /></div><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=Mm1LcM6IYyI:qRoBDTINoTM:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=Mm1LcM6IYyI:qRoBDTINoTM:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=Mm1LcM6IYyI:qRoBDTINoTM:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/Mm1LcM6IYyI\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By: Patrik HöglundThis is the third article in our series on Hackability; also see the first and second article.We have seen in our previous articles how Code Health and Debuggability can make a project much easier to work on. The third pillar is a solid infrastructure that gets accurate feedback to your developers as fast as possible. Speed is going to be major theme in this article, and we’ll look at a number of things you can do to make your project easier to hack on.Build Systems SpeedQuestion: What’s a change you’d really like to see in our development tools?“I feel like this question gets asked almost every time, and I always give the same answer: &nbsp;I would like them to be faster.” &nbsp; &nbsp; &nbsp; &nbsp; -- Ian Lance TaylorReplace make with ninja. Use the gold linker instead of ld. Detect and delete dead code in your project (perhaps using coverage tools). Reduce the number of dependencies, and enforce dependency rules so new ones are not added lightly. Give the developers faster machines. Use distributed build, which is available with many open-source continuous integration systems (or use Google’s system, Bazel!). You should do everything you can to make the build faster.Figure 1: “Cheetah chasing its prey” by Marlene Thyssen.Why is that? There’s a tremendous difference in hackability if it takes 5 seconds to build and test versus one minute, or even 20 minutes, to build and test. Slow feedback cycles kill hackability, for many reasons:Build and test times longer than a handful of seconds cause many developers’ minds to wander, taking them out of the zone.Excessive build or release times* makes tinkering and refactoring much harder. All developers have a threshold when they start hedging (e.g. “I’d like to remove this branch, but I don’t know if I’ll break the iOS build”) which causes refactoring to not happen.* The worst I ever heard of was an OS that took 24 hours to build!How do you actually make fast build systems? There are some suggestions in the first paragraph above, but the best general suggestion I can make is to have a few engineers on the project who deeply understand the build systems and have the time to continuously improve them. The main axes of improvement are:Reduce the amount of code being compiled.Replace tools with faster counterparts.Increase processing power, maybe through parallelization or distributed systems.Note that there is a big difference between full builds and incremental builds. Both should be as fast as possible, but incremental builds are by far the most important to optimize. The way you tackle the two is different. For instance, reducing the total number of source files will make a full build faster, but it may not make an incremental build faster.&nbsp;To get faster incremental builds, in general, you need to make each source file as decoupled as possible from the rest of the code base. The less a change ripples through the codebase, the less work to do, right? See “Loose Coupling and Testability” in Pillar 1 for more on this subject. The exact mechanics of dependencies and interfaces depends on programming language - one of the hardest to get right is unsurprisingly C++, where you need to be disciplined with includes and forward declarations to get any kind of incremental build performance.&nbsp;Build scripts and makefiles should be held to standards as high as the code itself. Technical debt and unnecessary dependencies have a tendency to accumulate in build scripts, because no one has the time to understand and fix them. Avoid this by addressing the technical debt as you go.Continuous Integration and Presubmit QueuesYou should build and run tests on all platforms you release on. For instance, if you release on all the major desktop platforms, but all your developers are on Linux, this becomes extra important. It’s bad for hackability to update the repo, build on Windows, and find that lots of stuff is broken. It’s even worse if broken changes start to stack on top of each other. I think we all know that terrible feeling: when you’re not sure your change is the one that broke things.At a minimum, you should build and test on all platforms, but it’s even better if you do it in presubmit. The Chromium submit queue does this. It has developed over the years so that a normal patch builds and tests on about 30 different build configurations before commit. This is necessary for the 400-patches-per-day velocity of the Chrome project. Most projects obviously don’t have to go that far. Chromium’s infrastructure is based off BuildBot, but there are many other job scheduling systems depending on your needs.Figure 2: How a Chromium patch is tested.As we discussed in Build Systems, speed and correctness are critical here. It takes a lot of ongoing work to keep build, tests, and presubmits fast and free of flakes. You should never accept flakes, since developers very quickly lose trust in flaky tests and systems. Tooling can help a bit with this; for instance, see the Chromium flakiness dashboard.Test SpeedSpeed is a feature, and this is particularly true for developer infrastructure. In general, the longer a test takes to execute, the less valuable it is. My rule of thumb is: if it takes more than a minute to execute, its value is greatly diminished. There are of course some exceptions, such as soak tests or certain performance tests.&nbsp;Figure 3. Test value.If you have tests that are slower than 60 seconds, they better be incredibly reliable and easily debuggable. A flaky test that takes several minutes to execute often has negative value because it slows down all work in the code it covers. You probably want to build better integration tests on lower levels instead, so you can make them faster and more reliable.If you have many engineers on a project, reducing the time to run the tests can have a big impact. This is one reason why it’s great to have SETIs or the equivalent. There are many things you can do to improve test speed:Sharding and parallelization. Add more machines to your continuous build as your test set or number of developers grows.Continuously measure how long it takes to run one build+test cycle in your continuous build, and have someone take action when it gets slower.Remove tests that don’t pull their weight. If a test is really slow, it’s often because of poorly written wait conditions or because the test bites off more than it can chew (maybe that unit test doesn’t have to process 15000 audio frames, maybe 50 is enough).If you have tests that bring up a local server stack, for instance inter-server integration tests, making your servers boot faster is going to make the tests faster as well. Faster production code is faster to test! See Running on Localhost, in Pillar 2 for more on local server stacks.Workflow SpeedWe’ve talked about fast builds and tests, but the core developer workflows are also important, of course. Chromium undertook multi-year project to switch from Subversion to Git, partly because Subversion was becoming too slow. You need to keep track of your core workflows as your project grows. Your version control system may work fine for years, but become too slow once the project becomes big enough. Bug search and management must also be robust and fast, since this is generally systems developers touch every day.Release OftenIt aids hackability to deploy to real users as fast as possible. No matter how good your product's tests are, there's always a risk that there's something you haven't thought of. If you’re building a service or web site, you should aim to deploy multiple times per week. For client projects, Chrome’s six-weeks cycle is a good goal to aim for.You should invest in infrastructure and tests that give you the confidence to do this - you don’t want to push something that’s broken. Your developers will thank you for it, since it makes their jobs so much easier. By releasing often, you mitigate risk, and developers will rush less to hack late changes in the release (since they know the next release isn’t far off).Easy RevertsIf you look in the commit log for the Chromium project, you will see that a significant percentage of the commits are reverts of a previous commits. In Chromium, bad commits quickly become costly because they impede other engineers, and the high velocity can cause good changes to stack onto bad changes.&nbsp;Figure 4: Chromium’s revert button.This is why the policy is “revert first, ask questions later”. I believe a revert-first policy is good for small projects as well, since it creates a clear expectation that the product/tools/dev environment should be working at all times (and if it doesn’t, a recent change should probably be reverted).It has a wonderful effect when a revert is simple to make. You can suddenly make speculative reverts if a test went flaky or a performance test regressed. It follows that if a patch is easy to revert, so is the inverse (i.e. reverting the revert or relanding the patch). So if you were wrong and that patch wasn’t guilty after all, it’s simple to re-land it again and try reverting another patch. Developers might initially balk at this (because it can’t possibly be their patch, right?), but they usually come around when they realize the benefits.For many projects, a revert can simply begit revert 9fbadbeefgit push origin masterIf your project (wisely) involves code review, it will behoove you to add something like Chromium’s revert button that I mentioned above. The revert button will create a special patch that bypasses review and tests (since we can assume a clean revert takes us back to a more stable state rather than the opposite). See Pillar 1 for more on code review and its benefits.For some projects, reverts are going to be harder, especially if you have a slow or laborious release process. Even if you release often, you could still have problems if a revert involves state migrations in your live services (for instance when rolling back a database schema change). You need to have a strategy to deal with such state changes.&nbsp;Reverts must always put you back to safer ground, and everyone must be confident they can safely revert. If not, you run the risk of massive fire drills and lost user trust if a bad patch makes it through the tests and you can’t revert it.Performance Tests: Measure EverythingIs it critical that your app starts up within a second? Should your app always render at 60 fps when it’s scrolled up or down? Should your web server always serve a response within 100 ms? Should your mobile app be smaller than 8 MB? If so, you should make a performance test for that. Performance tests aid hackability since developers can quickly see how their change affects performance and thus prevent performance regressions from making it into the wild.You should run your automated performance tests on the same device; all devices are different and this will reflect in the numbers. This is fairly straightforward if you have a decent continuous integration system that runs tests sequentially on a known set of worker machines. It’s harder if you need to run on physical phones or tablets, but it can be done.&nbsp;A test can be as simple as invoking a particular algorithm and measuring the time it takes to execute it (median and 90th percentile, say, over N runs).Figure 5: A VP8-in-WebRTC regression (bug) and its fix displayed in a Catapult dashboard.Write your test so it outputs performance numbers you care about. But what to do with those numbers? Fortunately, Chrome’s performance test framework has been open-sourced, which means you can set up a dashboard, with automatic regression monitoring, with minimal effort. The test framework also includes the powerful Telemetry framework which can run actions on web pages and Android apps and report performance results. Telemetry and Catapult are battle-tested by use in the Chromium project and are capable of running on a wide set of devices, while getting the maximum of usable performance data out of the devices.SourcesFigure 1: By Malene Thyssen (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons",
        "id": "tag:blogger.com,1999:blog-15045980.post-7608417273263191906",
        "isoDate": "2016-11-10T19:07:00.000Z"
      },
      {
        "title": "Hackable Projects - Pillar 2: Debuggability",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/fRCQcpX9oNU/hackable-projects-pillar-2-debuggability.html",
        "pubDate": "2016-10-11T16:04:00.001Z",
        "author": "Google Testing Bloggers",
        "content": "<i>By: Patrik Höglund </i><br /><i><br /></i><em>This is the second article in our series on Hackability; also see <a href=\"https://testing.googleblog.com/2016/08/hackable-projects.html\">the first article</a>.</em><br /><em><br /></em><br /><div style=\"border: 1px solid black; padding: 10px;\">“Deep into that darkness peering, long I stood there, wondering, fearing, doubting, dreaming dreams no mortal ever dared to dream before.”   -- Edgar Allan Poe</div><br /><br />Debuggability can mean being able to use a debugger, but here we’re interested in a broader meaning. Debuggability means being able to easily find what’s wrong with a piece of software, whether it’s through logs, statistics or debugger tools. Debuggability doesn’t happen by accident: you need to design it into your product. The amount of work it takes will vary depending on your product, programming language(s) and development environment. <br /><br />In this article, I am going to walk through a few examples of how we have aided debuggability for our developers. If you do the same analysis and implementation for your project, perhaps you can help your developers illuminate the dark corners of the codebase and learn what truly goes on there.<br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://1.bp.blogspot.com/-1WOvm87QI-U/V_0IEnBp4pI/AAAAAAAAAOs/mCppX2KO2jUWmKpXgNdJlepqKPBvRy0eQCLcB/s1600/image00.jpg\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"315\" src=\"https://1.bp.blogspot.com/-1WOvm87QI-U/V_0IEnBp4pI/AAAAAAAAAOs/mCppX2KO2jUWmKpXgNdJlepqKPBvRy0eQCLcB/s400/image00.jpg\" width=\"400\" /></a></div><div style=\"text-align: center;\"><em>Figure 1: computer log entry from the </em>Mark II<em>, with a moth taped to the page.</em></div><br /><h2>Running on Localhost </h2><div style=\"border: 1px solid black; padding: 10px;\">Read more on the Testing Blog: <a href=\"http://testing.googleblog.com/2012/10/hermetic-servers.html\">Hermetic Servers</a> by Chaitali Narla and Diego Salas </div><br /><br />Suppose you’re developing a service with a mobile app that connects to that service. You’re working on a new feature in the app that requires changes in the backend. Do you develop in production? That’s a really bad idea, as you must push unfinished code to production to work on your change. Don’t do that: it could break your service for your existing users. Instead, you need some kind of script that brings up your <a href=\"http://testing.googleblog.com/2012/10/hermetic-servers.html\">server stack on localhost</a>. <br /><br />You can probably run your servers by hand, but that quickly gets tedious. In Google, we usually use fancy python scripts that invoke the server binaries with flags. Why do we need those flags? Suppose, for instance, that you have a server A that depends on a server B and C. The default behavior when the server boots should be to connect to B and C in production. When booting on localhost, we want to connect to our local B and C though. For instance: <br /><br /><pre class=\"prettyprint\">b_serv --port=1234 --db=/tmp/fakedb<br />c_serv --port=1235<br />a_serv --b_spec=localhost:1234 --c_spec=localhost:1235<br /></pre><br />That makes it a whole lot easier to develop and debug your server. Make sure the logs and stdout/stderr end up in some well-defined directory on localhost so you don’t waste time looking for them. You may want to write a basic debug client that sends HTTP requests or RPCs or whatever your server handles. It’s painful to have to boot the real app on a mobile phone just to test something. <br /><br />A localhost setup is also a prerequisite for making <em>hermetic tests,</em>where the test invokes the above script to bring up the server stack. The test can then run, say, integration tests among the servers or even client-server integration tests. Such integration tests can catch protocol drift bugs between client and server, while being super stable by not talking to external or shared services. <br /><h2>Debugging Mobile Apps</h2>First, mobile is hard. The tooling is generally less mature than for desktop, although things are steadily improving. Again, unit tests are great for hackability here. It’s really painful to always load your app on a phone connected to your workstation to see if a change worked. <a href=\"http://robolectric.org/\">Robolectric unit tests</a> and <a href=\"https://google.github.io/android-testing-support-library/docs/espresso/basics/index.html\">Espresso functional tests</a>, for instance, run on your workstation and do not require a real phone. <a href=\"https://developer.apple.com/reference/xctest\">xcTests</a>and <a href=\"https://github.com/google/EarlGrey\">Earl Grey</a> give you the same on iOS. <br /><br />Debuggers ship with Xcode and Android Studio. If your Android app ships JNI code, it’s a bit trickier, but you can attach GDB to running processes on your phone. It’s worth spending the time figuring this out early in the project, so you don’t have to guess what your code is doing. Debugging unit tests is even better and can be done straightforwardly on your workstation. <br /><h2>When Debugging gets Tricky</h2>Some products are harder to debug than others. One example is <a href=\"https://en.wikipedia.org/wiki/Real-time_computing\">hard real-time systems</a>, since their behavior is so dependent on timing (and you better not be hooked up to a real industrial controller or rocket engine when you hit a breakpoint!). One possible solution is to run the software on a fake clock instead of a hardware clock, so the clock stops when the program stops. <br /><br />Another example is multi-process sandboxed programs such as Chromium. Since the browser spawns one renderer process per tab, how do you even attach a debugger to it? The developers have made it quite a lot easier with <a href=\"https://chromium.googlesource.com/chromium/src/+/master/docs/linux_debugging.md#Multiprocess-Tricks\">debugging flags and instructions</a>. For instance, this wraps gdb around each renderer process as it starts up: <br /><br /><pre class=\"prettyprint\">chrome --renderer-cmd-prefix='xterm -title renderer -e gdb --args'<br /></pre><br />The point is, you need to build these kinds of things into your product; this greatly aids hackability. <br /><h2>Proper Logging</h2><div style=\"border: 1px solid black; padding: 10px;\">Read more on the Testing Blog: <a href=\"http://testing.googleblog.com/2013/06/optimal-logging.html\">Optimal Logging</a> by Anthony Vallone </div><br /><br />It’s hackability to get the right logs when you need them. It’s easy to fix a crash if you get a stack trace from the error location. It’s far from guaranteed you’ll get such a stack trace, for instance in C++ programs, but this is something you should not stand for. For instance, Chromium had a problem where renderer process crashes didn’t print in test logs, because the test was running in a separate process. This was later fixed, and this kind of investment is worthwhile to make. A clean stack trace is worth a lot more than a “renderer crashed” message. <br /><br />Logs are also useful for development. It’s an art to determine how much logging is appropriate for a given piece of code, but it is a good idea to keep the default level of logging conservative and give developers the option to turn on more logging for the parts they’re working on (example: <a href=\"https://www.chromium.org/for-testers/enable-logging\">Chromium</a>). Too much logging isn’t hackability. <a href=\"https://googletesting.blogspot.se/2013/06/optimal-logging.html\">This article</a> elaborates further on this topic. <br /><br />Logs should also be properly symbolized for C/C++ projects; a naked list of addresses in a stack trace isn’t very helpful. This is easy if you build for development (e.g. with -g), but if the crash happens in a release build it’s a bit trickier. You then need to build the same binary with the same flags and use addr2line / ndk-stack / etc to symbolize the stack trace. It’s a good idea to build tools and scripts for this so it’s as easy as possible. <br /><h2>Monitoring and Statistics</h2>It aids hackability if developers can quickly understand what effect their changes have in the real world. For this, monitoring tools such as <a href=\"https://cloud.google.com/stackdriver/\">Stackdriver for Google Cloud</a>are excellent. If you’re running a service, such tools can help you keep track of request volumes and error rates. This way you can quickly detect that 30% increase in request errors, and roll back that bad code change, before it does too much damage. It also makes it possible to debug your service in production without disrupting it. <br /><h2>System Under Test (SUT) Size</h2>Tests and debugging go hand in hand: it’s a lot easier to target a piece of code in a test than in the whole application. Small and focused tests aid debuggability, because when a test breaks there isn’t an enormous SUT to look for errors in. These tests will also be less flaky. <a href=\"http://googletesting.blogspot.se/2015/04/just-say-no-to-more-end-to-end-tests.html\">This article</a> discusses this fact at length. <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"></div><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://3.bp.blogspot.com/-oBuMTeOYhww/V_0IswR5HKI/AAAAAAAAAO0/8jxV5oWkr-ErKGmavjvlRlkVpcGbhnWTACLcB/s1600/img3.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://3.bp.blogspot.com/-oBuMTeOYhww/V_0IswR5HKI/AAAAAAAAAO0/8jxV5oWkr-ErKGmavjvlRlkVpcGbhnWTACLcB/s1600/img3.png\" /></a></div><div style=\"text-align: center;\"><em><br /></em><em>Figure 2. The smaller the SUT, the more valuable the test.</em></div><br />You should try to keep the above in mind, particularly when writing integration tests. If you’re testing a mobile app with a server, what bugs are you actually trying to catch? If you’re trying to ensure the app can still talk to the server (i.e. catching protocol drift bugs), you should not involve the UI of the app. That’s not what you’re testing here. Instead, break out the signaling part of the app into a library, test that directly against your local server stack, and write separate tests for the UI that only test the UI. <br /><br />Smaller SUTs also greatly aids test speed, since there’s less to build, less to bring up and less to keep running. In general, strive to keep the SUT as small as possible through whatever means necessary. It will keep the tests smaller, faster and more focused. <br /><h2>Sources</h2>Figure 1: By Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988. - U.S. Naval Historical Center Online Library Photograph NH 96566-KN, Public Domain, https://commons.wikimedia.org/w/index.php?curid=165211  <br /><br /><br /><div style=\"text-align: center;\">(Continue to <a href=\"https://testing.googleblog.com/2016/11/hackable-projects-pillar-3.html\">Pillar 3: Infrastructure</a>)</div><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=fRCQcpX9oNU:C3n9JH-fm5I:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=fRCQcpX9oNU:C3n9JH-fm5I:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=fRCQcpX9oNU:C3n9JH-fm5I:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/fRCQcpX9oNU\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By: Patrik Höglund This is the second article in our series on Hackability; also see the first article.“Deep into that darkness peering, long I stood there, wondering, fearing, doubting, dreaming dreams no mortal ever dared to dream before.”   -- Edgar Allan PoeDebuggability can mean being able to use a debugger, but here we’re interested in a broader meaning. Debuggability means being able to easily find what’s wrong with a piece of software, whether it’s through logs, statistics or debugger tools. Debuggability doesn’t happen by accident: you need to design it into your product. The amount of work it takes will vary depending on your product, programming language(s) and development environment. In this article, I am going to walk through a few examples of how we have aided debuggability for our developers. If you do the same analysis and implementation for your project, perhaps you can help your developers illuminate the dark corners of the codebase and learn what truly goes on there.Figure 1: computer log entry from the Mark II, with a moth taped to the page.Running on Localhost Read more on the Testing Blog: Hermetic Servers by Chaitali Narla and Diego Salas Suppose you’re developing a service with a mobile app that connects to that service. You’re working on a new feature in the app that requires changes in the backend. Do you develop in production? That’s a really bad idea, as you must push unfinished code to production to work on your change. Don’t do that: it could break your service for your existing users. Instead, you need some kind of script that brings up your server stack on localhost. You can probably run your servers by hand, but that quickly gets tedious. In Google, we usually use fancy python scripts that invoke the server binaries with flags. Why do we need those flags? Suppose, for instance, that you have a server A that depends on a server B and C. The default behavior when the server boots should be to connect to B and C in production. When booting on localhost, we want to connect to our local B and C though. For instance: b_serv --port=1234 --db=/tmp/fakedbc_serv --port=1235a_serv --b_spec=localhost:1234 --c_spec=localhost:1235That makes it a whole lot easier to develop and debug your server. Make sure the logs and stdout/stderr end up in some well-defined directory on localhost so you don’t waste time looking for them. You may want to write a basic debug client that sends HTTP requests or RPCs or whatever your server handles. It’s painful to have to boot the real app on a mobile phone just to test something. A localhost setup is also a prerequisite for making hermetic tests,where the test invokes the above script to bring up the server stack. The test can then run, say, integration tests among the servers or even client-server integration tests. Such integration tests can catch protocol drift bugs between client and server, while being super stable by not talking to external or shared services. Debugging Mobile AppsFirst, mobile is hard. The tooling is generally less mature than for desktop, although things are steadily improving. Again, unit tests are great for hackability here. It’s really painful to always load your app on a phone connected to your workstation to see if a change worked. Robolectric unit tests and Espresso functional tests, for instance, run on your workstation and do not require a real phone. xcTestsand Earl Grey give you the same on iOS. Debuggers ship with Xcode and Android Studio. If your Android app ships JNI code, it’s a bit trickier, but you can attach GDB to running processes on your phone. It’s worth spending the time figuring this out early in the project, so you don’t have to guess what your code is doing. Debugging unit tests is even better and can be done straightforwardly on your workstation. When Debugging gets TrickySome products are harder to debug than others. One example is hard real-time systems, since their behavior is so dependent on timing (and you better not be hooked up to a real industrial controller or rocket engine when you hit a breakpoint!). One possible solution is to run the software on a fake clock instead of a hardware clock, so the clock stops when the program stops. Another example is multi-process sandboxed programs such as Chromium. Since the browser spawns one renderer process per tab, how do you even attach a debugger to it? The developers have made it quite a lot easier with debugging flags and instructions. For instance, this wraps gdb around each renderer process as it starts up: chrome --renderer-cmd-prefix='xterm -title renderer -e gdb --args'The point is, you need to build these kinds of things into your product; this greatly aids hackability. Proper LoggingRead more on the Testing Blog: Optimal Logging by Anthony Vallone It’s hackability to get the right logs when you need them. It’s easy to fix a crash if you get a stack trace from the error location. It’s far from guaranteed you’ll get such a stack trace, for instance in C++ programs, but this is something you should not stand for. For instance, Chromium had a problem where renderer process crashes didn’t print in test logs, because the test was running in a separate process. This was later fixed, and this kind of investment is worthwhile to make. A clean stack trace is worth a lot more than a “renderer crashed” message. Logs are also useful for development. It’s an art to determine how much logging is appropriate for a given piece of code, but it is a good idea to keep the default level of logging conservative and give developers the option to turn on more logging for the parts they’re working on (example: Chromium). Too much logging isn’t hackability. This article elaborates further on this topic. Logs should also be properly symbolized for C/C++ projects; a naked list of addresses in a stack trace isn’t very helpful. This is easy if you build for development (e.g. with -g), but if the crash happens in a release build it’s a bit trickier. You then need to build the same binary with the same flags and use addr2line / ndk-stack / etc to symbolize the stack trace. It’s a good idea to build tools and scripts for this so it’s as easy as possible. Monitoring and StatisticsIt aids hackability if developers can quickly understand what effect their changes have in the real world. For this, monitoring tools such as Stackdriver for Google Cloudare excellent. If you’re running a service, such tools can help you keep track of request volumes and error rates. This way you can quickly detect that 30% increase in request errors, and roll back that bad code change, before it does too much damage. It also makes it possible to debug your service in production without disrupting it. System Under Test (SUT) SizeTests and debugging go hand in hand: it’s a lot easier to target a piece of code in a test than in the whole application. Small and focused tests aid debuggability, because when a test breaks there isn’t an enormous SUT to look for errors in. These tests will also be less flaky. This article discusses this fact at length. Figure 2. The smaller the SUT, the more valuable the test.You should try to keep the above in mind, particularly when writing integration tests. If you’re testing a mobile app with a server, what bugs are you actually trying to catch? If you’re trying to ensure the app can still talk to the server (i.e. catching protocol drift bugs), you should not involve the UI of the app. That’s not what you’re testing here. Instead, break out the signaling part of the app into a library, test that directly against your local server stack, and write separate tests for the UI that only test the UI. Smaller SUTs also greatly aids test speed, since there’s less to build, less to bring up and less to keep running. In general, strive to keep the SUT as small as possible through whatever means necessary. It will keep the tests smaller, faster and more focused. SourcesFigure 1: By Courtesy of the Naval Surface Warfare Center, Dahlgren, VA., 1988. - U.S. Naval Historical Center Online Library Photograph NH 96566-KN, Public Domain, https://commons.wikimedia.org/w/index.php?curid=165211  (Continue to Pillar 3: Infrastructure)",
        "id": "tag:blogger.com,1999:blog-15045980.post-5362239673396458775",
        "isoDate": "2016-10-11T16:04:00.001Z"
      },
      {
        "title": "Testing on the Toilet: What Makes a Good End-to-End Test?",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/j7YYsvHlZkc/testing-on-toilet-what-makes-good-end.html",
        "pubDate": "2016-09-21T17:23:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<em>by Adam Bender</em><br /><em><br /></em><em>This article was adapted from a <a href=\"http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html\">Google Testing on the Toilet</a> (TotT) episode. You can download a <a href=\"https://docs.google.com/document/d/19aUiPozqFYJ7EqTrzP69ityJ_X1ZCYMmf6Z8nBfuDQo/edit?usp=sharing\">printer-friendly version</a> of this TotT episode and post it in your office.</em><br /><br />An end-to-end test tests your entire system from one end to the other, treating everything in between as a black box. <strong><span style=\"color: #990000;\">End-to-end tests can catch bugs that manifest <em>across </em>your entire system</span></strong>. In addition to unit and integration tests, they are a critical part of a balanced testing diet, providing confidence about the health of your system in a near production state. Unfortunately, end-to-end tests are <strong><span style=\"color: #990000;\">slower, more flaky, and more expensive to maintain</span></strong> than unit or integration tests. Consider carefully whether an end-to-end test is warranted, and if so, how best to write one. <br /><br />Let's consider how an end-to-end test might work for the following \"login flow\":<br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://1.bp.blogspot.com/-XpDz-avnNhA/V-LBGc9cvtI/AAAAAAAAAOQ/FuzdVujJW8EOTzSUZ0rbUPeg2U2VJh2QgCLcB/s1600/image00.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" src=\"https://1.bp.blogspot.com/-XpDz-avnNhA/V-LBGc9cvtI/AAAAAAAAAOQ/FuzdVujJW8EOTzSUZ0rbUPeg2U2VJh2QgCLcB/s1600/image00.png\" /></a></div><br /><br />In order to be cost effective, an end-to-end test should <strong><span style=\"color: #990000;\">focus on aspects of your system that cannot be reliably evaluated with smaller tests</span></strong>, such as resource allocation, concurrency issues and API compatibility. More specifically:<br /><ul><li><strong><span style=\"color: #990000;\">For each important use case, there should be one corresponding end-to-end test</span>.</strong> This should include one test for each important class of error. The goal is the keep your total end-to-end count low. </li><li>Be prepared to <strong><span style=\"color: #990000;\">allocate at least one week a quarter per test to keep your end-to-end tests stable</span> </strong>in the face of issues like slow and flaky dependencies or minor UI changes. </li><li><strong><span style=\"color: #990000;\">Focus your efforts on verifying overall system behavior instead of specific implementation details</span></strong>; for example, when testing login behavior, verify that the process succeeds independent of the exact messages or visual layouts, which may change frequently. </li><li><strong><span style=\"color: #990000;\">Make your end-to-end test easy to debug</span></strong> by providing an overview-level log file, documenting common test failure modes, and preserving all relevant system state information (e.g.: screenshots, database snapshots, etc.).</li></ul>End-to-end tests also come with some important caveats: <br /><ul><li>System components that are owned by other teams may change unexpectedly, and break your tests. This increases overall maintenance cost, but can highlight incompatible changes </li><li><strong><span style=\"color: #990000;\">It may be more difficult to make an end-to-end test fully hermetic</span></strong>; leftover test data may alter future tests and/or production systems. Where possible keep your test data ephemeral. </li><li>An end-to-end test often necessitates multiple test doubles (<a href=\"https://testing.googleblog.com/2013/07/testing-on-toilet-know-your-test-doubles.html\">fakes or stubs</a>) for underlying dependencies; they can, however, have a high maintenance burden as they drift from the real implementations over time.</li></ul><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=j7YYsvHlZkc:RyVoT81k8jE:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=j7YYsvHlZkc:RyVoT81k8jE:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=j7YYsvHlZkc:RyVoT81k8jE:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/j7YYsvHlZkc\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Adam BenderThis article was adapted from a Google Testing on the Toilet (TotT) episode. You can download a printer-friendly version of this TotT episode and post it in your office.An end-to-end test tests your entire system from one end to the other, treating everything in between as a black box. End-to-end tests can catch bugs that manifest across your entire system. In addition to unit and integration tests, they are a critical part of a balanced testing diet, providing confidence about the health of your system in a near production state. Unfortunately, end-to-end tests are slower, more flaky, and more expensive to maintain than unit or integration tests. Consider carefully whether an end-to-end test is warranted, and if so, how best to write one. Let's consider how an end-to-end test might work for the following \"login flow\":In order to be cost effective, an end-to-end test should focus on aspects of your system that cannot be reliably evaluated with smaller tests, such as resource allocation, concurrency issues and API compatibility. More specifically:For each important use case, there should be one corresponding end-to-end test. This should include one test for each important class of error. The goal is the keep your total end-to-end count low. Be prepared to allocate at least one week a quarter per test to keep your end-to-end tests stable in the face of issues like slow and flaky dependencies or minor UI changes. Focus your efforts on verifying overall system behavior instead of specific implementation details; for example, when testing login behavior, verify that the process succeeds independent of the exact messages or visual layouts, which may change frequently. Make your end-to-end test easy to debug by providing an overview-level log file, documenting common test failure modes, and preserving all relevant system state information (e.g.: screenshots, database snapshots, etc.).End-to-end tests also come with some important caveats: System components that are owned by other teams may change unexpectedly, and break your tests. This increases overall maintenance cost, but can highlight incompatible changes It may be more difficult to make an end-to-end test fully hermetic; leftover test data may alter future tests and/or production systems. Where possible keep your test data ephemeral. An end-to-end test often necessitates multiple test doubles (fakes or stubs) for underlying dependencies; they can, however, have a high maintenance burden as they drift from the real implementations over time.",
        "id": "tag:blogger.com,1999:blog-15045980.post-6772169526811686848",
        "isoDate": "2016-09-21T17:23:00.000Z"
      },
      {
        "title": "What Test Engineers do at Google",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/ajLgQE0Jn3M/what-test-engineers-do-at-google.html",
        "pubDate": "2016-09-12T15:00:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by Matt Lowrie, Manjusha Parvathaneni, Benjamin Pick,  and Jochen Wuttke </i><br /><br />Test engineers (TEs) at Google are a dedicated group of engineers who use proven testing practices to foster excellence in our products. We orchestrate the rapid testing and releasing of products and features our users rely on. Achieving this velocity requires creative and diverse engineering skills that allow us to advocate for our users. By building testable user journeys into the process, we ensure reliable products. TEs are also the glue that bring together feature stakeholders (product managers, development teams, UX designers, release engineers, beta testers, end users, etc.) to confirm successful product launches. Essentially, every day we ask ourselves, “How can we make our software development process more efficient to deliver products that make our users happy?”. <br /><br />The TE role grew out of the desire to make Google’s early free products, like Search, Gmail and Docs, better than similar paid products on the market at the time. Early on in Google’s history, a small group of engineers believed that the company’s “launch and iterate” approach to software deployment could be improved with continuous automated testing. They took it upon themselves to promote good testing practices to every team throughout the company, via some programs you may have heard about: <a href=\"http://googletesting.blogspot.com/2007/01/introducing-testing-on-toilet.html\">Testing on the Toilet</a>, the <a href=\"https://mike-bland.com/2011/10/18/test-certified.html\">Test Certified Program</a>, and the <a href=\"https://developers.google.com/google-test-automation-conference/\">Google Test Automation Conference (GTAC)</a>. These efforts resulted in every project taking ownership of all aspects of testing, such as <a href=\"http://googletesting.blogspot.com/2014/07/measuring-coverage-at-google.html\">code coverage</a> and <a href=\"http://googletesting.blogspot.com/2007/10/performance-testing.html\">performance testing</a>. Testing practices quickly became commonplace throughout the company and engineers writing tests for their own code became the standard. Today, TEs carry on this tradition of setting the standard of quality which all products should achieve. <br /><br />Historically, Google has sustained two separate job titles related to product testing and test infrastructure, which has caused confusion. We often get asked what the difference is between the two. The rebranding of the Software engineer, tools and infrastructure (SETI) role, which now concentrates on engineering productivity, has been addressed in a <a href=\"http://googletesting.blogspot.com/2016/03/from-qa-to-engineering-productivity.html\">previous blog post</a>.  What this means for test engineers at Google, is an enhanced responsibility of being the authority on product excellence. We are expected to uphold testing standards company-wide, both programmatically and persuasively. <br /><br />Test engineer is a unique role at Google. As TEs, we define and organize our own engineering projects, bridging gaps between engineering output and end-user satisfaction. To give you an idea of what TEs do, here are some examples of challenges we need to solve on any particular day: <br /><ul><li>Automate a manual verification process for product release candidates so developers have more time to respond to potential release-blocking issues. </li><li>Design and implement an automated way to track and surface Android battery usage to developers, so that they know immediately when a new feature will cause users drained batteries. </li><li>Quantify if a regenerated data set used by a product, which contains a billion entities, is better quality than the data set currently live in production. </li><li>Write an automated test suite that validates if content presented to a user is of an acceptable quality level based on their interests. </li><li>Read an engineering design proposal for a new feature and provide suggestions about how and where to build in testability. </li><li>Investigate correlated stack traces submitted by users through our feedback tracking system, and search the code base to find the correct owner for escalation. </li><li>Collaborate on determining the root cause of a production outage, then pinpoint tests that need to be added to prevent similar outages in the future. </li><li>Organize a task force to advise teams across the company about best practices when testing for accessibility.</li></ul>Over the next few weeks leading up to <a href=\"https://developers.google.com/google-test-automation-conference/\">GTAC</a>, we will also post vignettes of actual TEs working on different projects at Google, to showcase the diversity of the Google Test Engineer role. Stay tuned! <div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ajLgQE0Jn3M:v3zsf8-lJ7k:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ajLgQE0Jn3M:v3zsf8-lJ7k:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=ajLgQE0Jn3M:v3zsf8-lJ7k:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/ajLgQE0Jn3M\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Matt Lowrie, Manjusha Parvathaneni, Benjamin Pick,  and Jochen Wuttke Test engineers (TEs) at Google are a dedicated group of engineers who use proven testing practices to foster excellence in our products. We orchestrate the rapid testing and releasing of products and features our users rely on. Achieving this velocity requires creative and diverse engineering skills that allow us to advocate for our users. By building testable user journeys into the process, we ensure reliable products. TEs are also the glue that bring together feature stakeholders (product managers, development teams, UX designers, release engineers, beta testers, end users, etc.) to confirm successful product launches. Essentially, every day we ask ourselves, “How can we make our software development process more efficient to deliver products that make our users happy?”. The TE role grew out of the desire to make Google’s early free products, like Search, Gmail and Docs, better than similar paid products on the market at the time. Early on in Google’s history, a small group of engineers believed that the company’s “launch and iterate” approach to software deployment could be improved with continuous automated testing. They took it upon themselves to promote good testing practices to every team throughout the company, via some programs you may have heard about: Testing on the Toilet, the Test Certified Program, and the Google Test Automation Conference (GTAC). These efforts resulted in every project taking ownership of all aspects of testing, such as code coverage and performance testing. Testing practices quickly became commonplace throughout the company and engineers writing tests for their own code became the standard. Today, TEs carry on this tradition of setting the standard of quality which all products should achieve. Historically, Google has sustained two separate job titles related to product testing and test infrastructure, which has caused confusion. We often get asked what the difference is between the two. The rebranding of the Software engineer, tools and infrastructure (SETI) role, which now concentrates on engineering productivity, has been addressed in a previous blog post.  What this means for test engineers at Google, is an enhanced responsibility of being the authority on product excellence. We are expected to uphold testing standards company-wide, both programmatically and persuasively. Test engineer is a unique role at Google. As TEs, we define and organize our own engineering projects, bridging gaps between engineering output and end-user satisfaction. To give you an idea of what TEs do, here are some examples of challenges we need to solve on any particular day: Automate a manual verification process for product release candidates so developers have more time to respond to potential release-blocking issues. Design and implement an automated way to track and surface Android battery usage to developers, so that they know immediately when a new feature will cause users drained batteries. Quantify if a regenerated data set used by a product, which contains a billion entities, is better quality than the data set currently live in production. Write an automated test suite that validates if content presented to a user is of an acceptable quality level based on their interests. Read an engineering design proposal for a new feature and provide suggestions about how and where to build in testability. Investigate correlated stack traces submitted by users through our feedback tracking system, and search the code base to find the correct owner for escalation. Collaborate on determining the root cause of a production outage, then pinpoint tests that need to be added to prevent similar outages in the future. Organize a task force to advise teams across the company about best practices when testing for accessibility.Over the next few weeks leading up to GTAC, we will also post vignettes of actual TEs working on different projects at Google, to showcase the diversity of the Google Test Engineer role. Stay tuned!",
        "id": "tag:blogger.com,1999:blog-15045980.post-7987865843515328086",
        "isoDate": "2016-09-12T15:00:00.000Z"
      },
      {
        "title": "Hackable Projects - Pillar 1: Code Health",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/JbKGw935ngA/hackable-projects.html",
        "pubDate": "2016-08-18T18:14:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<em>By: Patrik Höglund </em><br /><h1>Introduction</h1>Software development is difficult. Projects often evolve over several years, under changing requirements and shifting market conditions, impacting developer tools and infrastructure. Technical debt, slow build systems, poor debuggability, and increasing numbers of dependencies can weigh down a project The developers get weary, and cobwebs accumulate in dusty corners of the code base. <br /><br />Fighting these issues can be taxing and feel like a quixotic undertaking, but don’t worry — the Google Testing Blog is riding to the rescue! This is the first article of a series on “hackability” that identifies some of the issues that hinder software projects and outlines what Google SETIs usually do about them. <br /><br /><a href=\"https://en.wiktionary.org/wiki/hackable\">According to Wiktionary</a>, hackable is defined as: <br /><div style=\"border: 1px solid black; padding: 5px;\"><b>Adjective</b><br /><b>hackable</b> ‎(comparative <b>more hackable</b>, superlative <b>most hackable</b>) <br /><ol><li>(computing) That can be hacked or broken into; insecure, vulnerable.&nbsp;</li><li>That lends itself to hacking (technical tinkering and modification); moddable.</li></ol></div><br />Obviously, we’re not going to talk about making your product more vulnerable (by, say, rolling your own crypto or something equally unwise); instead, we will focus on the second definition, which essentially means “something that is easy to work on.” This has become the <a href=\"http://googletesting.blogspot.com/2016/03/from-qa-to-engineering-productivity.html\">mainfocus for SETIs at Google</a> as the role has evolved over the years.<br /><h2>In Practice</h2>In a hackable project, it’s easy to try things and hard to break things. Hackability means fast feedback cycles that offer useful information to the developer. <br /><br />This is hackability: <br /><ul><li>Developing is easy </li><li>Fast build </li><li>Good, fast tests </li><li>Clean code </li><li>Easy running + debugging </li><li>One-click rollbacks</li></ul>In contrast, what is not hackability? <br /><ul><li>Broken HEAD (tip-of-tree) </li><li>Slow presubmit (i.e. <a href=\"https://www.chromium.org/developers/how-tos/depottools/presubmit-scripts\">checks running before submit</a>) </li><li>Builds take hours </li><li>Incremental build/link &gt; 30s </li><li><a href=\"http://googletesting.blogspot.com/2016/05/flaky-tests-at-google-and-how-we.html\">Flakytests</a></li><li>Can’t attach debugger </li><li>Logs full of uninteresting information</li></ul><h2>The Three Pillars of Hackability</h2>There are a number of tools and practices that foster hackability. When everything is in place, it feels great to work on the product. Basically no time is spent on figuring out why things are broken, and all time is spent on what matters, which is understanding and working with the code. I believe there are three main pillars that support hackability. If one of them is absent, hackability will suffer. They are: <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://3.bp.blogspot.com/-esLisB8th9M/V7XlS7pViVI/AAAAAAAAANs/lSJgkuiOF5w2Eqf1Yt_yaAbHqdwCYlg-gCLcB/s1600/image03.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"203\" src=\"https://3.bp.blogspot.com/-esLisB8th9M/V7XlS7pViVI/AAAAAAAAANs/lSJgkuiOF5w2Eqf1Yt_yaAbHqdwCYlg-gCLcB/s320/image03.png\" width=\"320\" /></a></div><br /><h1>Pillar 1: Code Health</h1><div style=\"border: 1px solid black; padding: 5px;\">“I found Rome a city of bricks, and left it a city of marble.”<br />&nbsp; &nbsp;-- Augustus</div><br />Keeping the code in good shape is critical for hackability. It’s a lot harder to tinker and modify something if you don’t understand what it does (or if it’s full of hidden traps, for that matter). <br /><h2>Tests</h2>Unit and small integration tests are probably the best things you can do for hackability. They’re a support you can lean on while making your changes, and they contain lots of good information on what the code does. It isn’t hackability to boot a slow UI and click buttons on every iteration to verify your change worked - it is hackability to run a sub-second set of unit tests! In contrast, end-to-end (E2E) tests generally help hackability much less (and can <a href=\"http://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html\">evenbe a hindrance</a> if they, or the product, are in sufficiently bad shape). <br /><br /><div class=\"separator\" style=\"clear: both; text-align: center;\"><a href=\"https://2.bp.blogspot.com/-H0OCdokbaVw/V7XlkhRcJcI/AAAAAAAAANw/z-v9qXyVig8dspmHHnRu6FQscqSOtvrTgCLcB/s1600/image01.png\" imageanchor=\"1\" style=\"margin-left: 1em; margin-right: 1em;\"><img border=\"0\" height=\"276\" src=\"https://2.bp.blogspot.com/-H0OCdokbaVw/V7XlkhRcJcI/AAAAAAAAANw/z-v9qXyVig8dspmHHnRu6FQscqSOtvrTgCLcB/s320/image01.png\" width=\"320\" /></a></div><div style=\"text-align: center;\"><em>Figure 1: the Testing Pyramid.</em></div><br />I’ve always been interested in how you actually make unit tests happen in a team. It’s about education. Writing a product such that it has good unit tests is actually a hard problem. It requires knowledge of dependency injection, testing/mocking frameworks, language idioms and refactoring. The difficulty varies by language as well. Writing unit tests in Go or Java is quite easy and natural, whereas in C++ it can be very difficult (and it isn’t exactly ingrained in C++ culture to write unit tests). <br /><br />It’s important to educate your developers about unit tests. Sometimes, it is appropriate to lead by example and help review unit tests as well. You can have a large impact on a project by establishing a pattern of unit testing early. If tons of code gets written without unit tests, it will be much harder to add unit tests later. <br /><br />What if you already have tons of poorly tested legacy code? The answer is refactoring and adding tests as you go. It’s hard work, but each line you add a test for is one more line that is easier to hack on. <br /><h2>Readable Code and Code Review</h2>At Google, “readability” is a special committer status that is granted per language (C++, Go, Java and so on). It means that a person not only knows the language and its culture and idioms well, but also can write clean, well tested and well structured code. Readability literally means that you’re a guardian of Google’s code base and should push back on hacky and ugly code. The use of a <a href=\"https://github.com/google/styleguide\">style guide</a> enforces consistency, and <a href=\"https://en.wikipedia.org/wiki/Code_review\">code review</a> (where at least one person with readability must approve) ensures the code upholds high quality. Engineers must take care to not depend too much on “review buddies” here but really make sure to pull in the person that can give the best feedback. <br /><br />Requiring code reviews naturally results in small changes, as reviewers often get grumpy if you dump huge changelists in their lap (at least if reviewers are somewhat fast to respond, which they should be). This is a good thing, since small changes are less risky and are easy to roll back. Furthermore, code review is good for knowledge sharing. You can also do pair programming if your team prefers that (a pair-programmed change is considered reviewed and can be submitted when both engineers are happy). There are multiple open-source review tools out there, such as <a href=\"https://www.gerritcodereview.com/\">Gerrit</a>. <br /><br />Nice, clean code is great for hackability, since you don’t need to spend time to unwind that nasty pointer hack in your head before making your changes. How do you make all this happen in practice? Put together workshops on, say, the <a href=\"https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)\">SOLID principles</a>, unit testing, or concurrency to encourage developers to learn. Spread knowledge through code review, pair programming and mentoring (such as with the Readability concept). You can’t just mandate higher code quality; it takes a lot of work, effort and consistency. <br /><h2>Presubmit Testing and Lint</h2>Consistently formatted source code aids hackability. You can scan code faster if its formatting is consistent. Automated tooling also aids hackability. It really doesn’t make sense to waste any time on formatting source code by hand. You should be using tools like <a href=\"https://golang.org/cmd/gofmt/\">gofmt</a>, <a href=\"http://clang.llvm.org/docs/ClangFormat.html\">clang-format</a>, etc. If the patch isn’t formatted properly, you should see something like this (<a href=\"https://cs.chromium.org/chromium/tools/depot_tools/presubmit_canned_checks.py?q=%22requires+formatting%22&amp;sq=package:chromium&amp;l=1151\">example from Chrome</a>): <br /><br /><pre style=\"background: #f0f0f0; border-color: #7b7b7b; border-style: solid; border-width: 1px; color: black; overflow: auto; padding: 10px;\">$ git cl upload<br />Error: the media/audio directory requires formatting. Please run <br />    git cl format media/audio.</pre><br />Source formatting isn’t the only thing to check. In fact, you should check pretty much anything you have as a rule in your project. Should other modules not depend on the internals of your modules? <a href=\"https://cs.chromium.org/chromium/src/buildtools/checkdeps/builddeps.py?q=check_deps&amp;sq=package:chromium&amp;dr=C\">Enforce it with a check</a>. Are there already inappropriate dependencies in your project? Whitelist the existing ones for now, but at least block new bad dependencies from forming. Should our app work on Android 16 phones and newer? <a href=\"https://bugs.chromium.org/p/webrtc/issues/detail?id=5063\">Add linting</a>, so we don’t use level 17+ APIs without gating at runtime. Should your project’s <a href=\"https://en.wikipedia.org/wiki/VHDL\">VHDL</a> code always place-and-route cleanly on a particular brand of <a href=\"https://en.wikipedia.org/wiki/Field-programmable_gate_array\">FPGA</a>? Invoke the layout tool in your presubmit and and stop submit if the layout process fails. <br /><br />Presubmit is the most valuable real estate for aiding hackability. You have limited space in your presubmit, but you can get tremendous value out of it if you put the right things there. You should stop all obvious errors here. <br /><br />It aids hackability to have all this tooling so you don’t have to waste time going back and breaking things for other developers. Remember you need to maintain the presubmit well; it’s not hackability to have a slow, overbearing or buggy presubmit. Having a good presubmit can make it tremendously more pleasant to work on a project. We’re going to talk more in later articles on how to build infrastructure for submit queues and presubmit. <br /><h2>Single Branch And Reducing Risk</h2>Having a single branch for everything, and putting risky new changes behind <a href=\"https://en.wikipedia.org/wiki/Feature_toggle\">feature flags</a>, aids hackability since branches and forks often amass tremendous risk when it’s time to merge them. Single branches smooth out the risk. Furthermore, running all your tests on many branches is expensive. However, a single branch can have negative effects on hackability if Team A depends on a library from Team B and gets broken by Team B a lot. Having some kind of stabilization on Team B’s software might be a good idea there. <a href=\"https://googletesting.blogspot.com/2015/05/multi-repository-development.html\">Thisarticle</a> covers such situations, and how to integrate often with your dependencies to reduce the risk that one of them will break you. <br /><h2>Loose Coupling and Testability</h2>Tightly coupled code is terrible for hackability. To take the most ridiculous example I know: I once heard of a computer game where a developer changed a ballistics algorithm and broke the game’s <em>chat</em>. That’s hilarious, but hardly intuitive for the poor developer that made the change. A hallmark of loosely coupled code is that it’s upfront about its dependencies and behavior and is easy to modify and move around. <br /><br />Loose coupling, coherence and so on is really about design and architecture and is notoriously hard to measure. It really takes experience. One of the best ways to convey such experience is through code review, which we’ve already mentioned. Education on the SOLID principles, rules of thumb such as tell-don’t-ask, discussions about anti-patterns and code smells are all good here. Again, it’s hard to build tooling for this. You could write a presubmit check that forbids methods longer than 20 lines or cyclomatic complexity over 30, but that’s probably shooting yourself in the foot. Developers would consider that overbearing rather than a helpful assist. <br /><br />SETIs at Google are expected to give input on a product’s testability. A few well-placed test hooks in your product can enable tremendously powerful testing, such as serving mock content for apps (this enables you to meaningfully test app UI without contacting your real servers, for instance). Testability can also have an influence on architecture. For instance, it’s a testability problem if your servers are built like a huge monolith that is slow to build and start, or if it can’t boot on localhost without calling external services. We’ll cover this in the next article. <br /><h2>Aggressively Reduce Technical Debt</h2>It’s quite easy to add a lot of code and dependencies and call it a day when the software works. New projects can do this without many problems, but as the project becomes older it becomes a “legacy” project, weighed down by dependencies and excess code. Don’t end up there. It’s bad for hackability to have a slew of bug fixes stacked on top of unwise and obsolete decisions, and understanding and untangling the software becomes more difficult. <br /><br />What constitutes technical debt varies by project and is something you need to learn from experience. It simply means the software isn’t in optimal form. Some types of technical debt are easy to classify, such as dead code and barely-used dependencies. Some types are harder to identify, such as when the architecture of the project has grown unfit to the task from changing requirements. We can’t use tooling to help with the latter, but we can with the former. <br /><br />I already mentioned that <a href=\"https://cs.chromium.org/chromium/src/buildtools/checkdeps/builddeps.py?q=check_deps&amp;sq=package:chromium&amp;dr=C\">dependency enforcement</a> can go a long way toward keeping people honest. It helps make sure people are making the appropriate trade-offs instead of just slapping on a new dependency, and it requires them to explain to a fellow engineer when they want to override a dependency rule. This can prevent unhealthy dependencies like circular dependencies, abstract modules depending on concrete modules, or modules depending on the internals of other modules. <br /><br />There are various tools available for visualizing dependency graphs as well. You can use these to get a grip on your current situation and start cleaning up dependencies. If you have a huge dependency you only use a small part of, maybe you can replace it with something simpler. If an old part of your app has inappropriate dependencies and other problems, maybe it’s time to rewrite that part. <br /><br /><div style=\"text-align: center;\">(Continue to&nbsp;<a href=\"https://testing.googleblog.com/2016/10/hackable-projects-pillar-2-debuggability.html\">Pillar 2: Debuggability</a>)</div><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=JbKGw935ngA:3_938gBE5Gw:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=JbKGw935ngA:3_938gBE5Gw:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=JbKGw935ngA:3_938gBE5Gw:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/JbKGw935ngA\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "By: Patrik Höglund IntroductionSoftware development is difficult. Projects often evolve over several years, under changing requirements and shifting market conditions, impacting developer tools and infrastructure. Technical debt, slow build systems, poor debuggability, and increasing numbers of dependencies can weigh down a project The developers get weary, and cobwebs accumulate in dusty corners of the code base. Fighting these issues can be taxing and feel like a quixotic undertaking, but don’t worry — the Google Testing Blog is riding to the rescue! This is the first article of a series on “hackability” that identifies some of the issues that hinder software projects and outlines what Google SETIs usually do about them. According to Wiktionary, hackable is defined as: Adjectivehackable ‎(comparative more hackable, superlative most hackable) (computing) That can be hacked or broken into; insecure, vulnerable.&nbsp;That lends itself to hacking (technical tinkering and modification); moddable.Obviously, we’re not going to talk about making your product more vulnerable (by, say, rolling your own crypto or something equally unwise); instead, we will focus on the second definition, which essentially means “something that is easy to work on.” This has become the mainfocus for SETIs at Google as the role has evolved over the years.In PracticeIn a hackable project, it’s easy to try things and hard to break things. Hackability means fast feedback cycles that offer useful information to the developer. This is hackability: Developing is easy Fast build Good, fast tests Clean code Easy running + debugging One-click rollbacksIn contrast, what is not hackability? Broken HEAD (tip-of-tree) Slow presubmit (i.e. checks running before submit) Builds take hours Incremental build/link > 30s FlakytestsCan’t attach debugger Logs full of uninteresting informationThe Three Pillars of HackabilityThere are a number of tools and practices that foster hackability. When everything is in place, it feels great to work on the product. Basically no time is spent on figuring out why things are broken, and all time is spent on what matters, which is understanding and working with the code. I believe there are three main pillars that support hackability. If one of them is absent, hackability will suffer. They are: Pillar 1: Code Health“I found Rome a city of bricks, and left it a city of marble.”&nbsp; &nbsp;-- AugustusKeeping the code in good shape is critical for hackability. It’s a lot harder to tinker and modify something if you don’t understand what it does (or if it’s full of hidden traps, for that matter). TestsUnit and small integration tests are probably the best things you can do for hackability. They’re a support you can lean on while making your changes, and they contain lots of good information on what the code does. It isn’t hackability to boot a slow UI and click buttons on every iteration to verify your change worked - it is hackability to run a sub-second set of unit tests! In contrast, end-to-end (E2E) tests generally help hackability much less (and can evenbe a hindrance if they, or the product, are in sufficiently bad shape). Figure 1: the Testing Pyramid.I’ve always been interested in how you actually make unit tests happen in a team. It’s about education. Writing a product such that it has good unit tests is actually a hard problem. It requires knowledge of dependency injection, testing/mocking frameworks, language idioms and refactoring. The difficulty varies by language as well. Writing unit tests in Go or Java is quite easy and natural, whereas in C++ it can be very difficult (and it isn’t exactly ingrained in C++ culture to write unit tests). It’s important to educate your developers about unit tests. Sometimes, it is appropriate to lead by example and help review unit tests as well. You can have a large impact on a project by establishing a pattern of unit testing early. If tons of code gets written without unit tests, it will be much harder to add unit tests later. What if you already have tons of poorly tested legacy code? The answer is refactoring and adding tests as you go. It’s hard work, but each line you add a test for is one more line that is easier to hack on. Readable Code and Code ReviewAt Google, “readability” is a special committer status that is granted per language (C++, Go, Java and so on). It means that a person not only knows the language and its culture and idioms well, but also can write clean, well tested and well structured code. Readability literally means that you’re a guardian of Google’s code base and should push back on hacky and ugly code. The use of a style guide enforces consistency, and code review (where at least one person with readability must approve) ensures the code upholds high quality. Engineers must take care to not depend too much on “review buddies” here but really make sure to pull in the person that can give the best feedback. Requiring code reviews naturally results in small changes, as reviewers often get grumpy if you dump huge changelists in their lap (at least if reviewers are somewhat fast to respond, which they should be). This is a good thing, since small changes are less risky and are easy to roll back. Furthermore, code review is good for knowledge sharing. You can also do pair programming if your team prefers that (a pair-programmed change is considered reviewed and can be submitted when both engineers are happy). There are multiple open-source review tools out there, such as Gerrit. Nice, clean code is great for hackability, since you don’t need to spend time to unwind that nasty pointer hack in your head before making your changes. How do you make all this happen in practice? Put together workshops on, say, the SOLID principles, unit testing, or concurrency to encourage developers to learn. Spread knowledge through code review, pair programming and mentoring (such as with the Readability concept). You can’t just mandate higher code quality; it takes a lot of work, effort and consistency. Presubmit Testing and LintConsistently formatted source code aids hackability. You can scan code faster if its formatting is consistent. Automated tooling also aids hackability. It really doesn’t make sense to waste any time on formatting source code by hand. You should be using tools like gofmt, clang-format, etc. If the patch isn’t formatted properly, you should see something like this (example from Chrome): $ git cl uploadError: the media/audio directory requires formatting. Please run     git cl format media/audio.Source formatting isn’t the only thing to check. In fact, you should check pretty much anything you have as a rule in your project. Should other modules not depend on the internals of your modules? Enforce it with a check. Are there already inappropriate dependencies in your project? Whitelist the existing ones for now, but at least block new bad dependencies from forming. Should our app work on Android 16 phones and newer? Add linting, so we don’t use level 17+ APIs without gating at runtime. Should your project’s VHDL code always place-and-route cleanly on a particular brand of FPGA? Invoke the layout tool in your presubmit and and stop submit if the layout process fails. Presubmit is the most valuable real estate for aiding hackability. You have limited space in your presubmit, but you can get tremendous value out of it if you put the right things there. You should stop all obvious errors here. It aids hackability to have all this tooling so you don’t have to waste time going back and breaking things for other developers. Remember you need to maintain the presubmit well; it’s not hackability to have a slow, overbearing or buggy presubmit. Having a good presubmit can make it tremendously more pleasant to work on a project. We’re going to talk more in later articles on how to build infrastructure for submit queues and presubmit. Single Branch And Reducing RiskHaving a single branch for everything, and putting risky new changes behind feature flags, aids hackability since branches and forks often amass tremendous risk when it’s time to merge them. Single branches smooth out the risk. Furthermore, running all your tests on many branches is expensive. However, a single branch can have negative effects on hackability if Team A depends on a library from Team B and gets broken by Team B a lot. Having some kind of stabilization on Team B’s software might be a good idea there. Thisarticle covers such situations, and how to integrate often with your dependencies to reduce the risk that one of them will break you. Loose Coupling and TestabilityTightly coupled code is terrible for hackability. To take the most ridiculous example I know: I once heard of a computer game where a developer changed a ballistics algorithm and broke the game’s chat. That’s hilarious, but hardly intuitive for the poor developer that made the change. A hallmark of loosely coupled code is that it’s upfront about its dependencies and behavior and is easy to modify and move around. Loose coupling, coherence and so on is really about design and architecture and is notoriously hard to measure. It really takes experience. One of the best ways to convey such experience is through code review, which we’ve already mentioned. Education on the SOLID principles, rules of thumb such as tell-don’t-ask, discussions about anti-patterns and code smells are all good here. Again, it’s hard to build tooling for this. You could write a presubmit check that forbids methods longer than 20 lines or cyclomatic complexity over 30, but that’s probably shooting yourself in the foot. Developers would consider that overbearing rather than a helpful assist. SETIs at Google are expected to give input on a product’s testability. A few well-placed test hooks in your product can enable tremendously powerful testing, such as serving mock content for apps (this enables you to meaningfully test app UI without contacting your real servers, for instance). Testability can also have an influence on architecture. For instance, it’s a testability problem if your servers are built like a huge monolith that is slow to build and start, or if it can’t boot on localhost without calling external services. We’ll cover this in the next article. Aggressively Reduce Technical DebtIt’s quite easy to add a lot of code and dependencies and call it a day when the software works. New projects can do this without many problems, but as the project becomes older it becomes a “legacy” project, weighed down by dependencies and excess code. Don’t end up there. It’s bad for hackability to have a slew of bug fixes stacked on top of unwise and obsolete decisions, and understanding and untangling the software becomes more difficult. What constitutes technical debt varies by project and is something you need to learn from experience. It simply means the software isn’t in optimal form. Some types of technical debt are easy to classify, such as dead code and barely-used dependencies. Some types are harder to identify, such as when the architecture of the project has grown unfit to the task from changing requirements. We can’t use tooling to help with the latter, but we can with the former. I already mentioned that dependency enforcement can go a long way toward keeping people honest. It helps make sure people are making the appropriate trade-offs instead of just slapping on a new dependency, and it requires them to explain to a fellow engineer when they want to override a dependency rule. This can prevent unhealthy dependencies like circular dependencies, abstract modules depending on concrete modules, or modules depending on the internals of other modules. There are various tools available for visualizing dependency graphs as well. You can use these to get a grip on your current situation and start cleaning up dependencies. If you have a huge dependency you only use a small part of, maybe you can replace it with something simpler. If an old part of your app has inappropriate dependencies and other problems, maybe it’s time to rewrite that part. (Continue to&nbsp;Pillar 2: Debuggability)",
        "id": "tag:blogger.com,1999:blog-15045980.post-5905231631174895803",
        "isoDate": "2016-08-18T18:14:00.000Z"
      },
      {
        "title": "The Inquiry Method for Test Planning",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/A8l6jMC3smM/the-inquiry-method-for-test-planning.html",
        "pubDate": "2016-06-06T13:05:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by <a href=\"http://anthonyvallone.com/\">Anthony Vallone</a></i><br /><div style=\"text-align: center;\"><i>updated: July 2016</i></div><br /><br /><br /><br /><div>Creating a test plan is often a complex undertaking. An ideal test plan is accomplished by applying basic principles of <a href=\"https://en.wikipedia.org/wiki/Cost%E2%80%93benefit_analysis\">cost-benefit analysis</a> and <a href=\"https://en.wikipedia.org/wiki/Risk_analysis\">risk analysis</a>, optimally balancing these software development factors:</div><div><ul><li><b>Implementation cost</b>: The time and complexity of implementing testable features and automated tests for specific scenarios will vary, and this affects short-term development cost.</li><li><b>Maintenance cost</b>: Some tests or test plans may vary from easy to difficult to maintain, and this affects long-term development cost. When manual testing is chosen, this also adds to long-term cost.</li><li><b>Monetary cost</b>: Some test approaches may require billed resources.</li><li><b>Benefit</b>: Tests are capable of preventing issues and aiding productivity by varying degrees. Also, the earlier they can catch problems in the development life-cycle, the greater the benefit.</li><li><b>Risk</b>: The probability of failure scenarios may vary from rare to likely, and their consequences may vary from minor nuisance to catastrophic.</li></ul></div><div>Effectively balancing these factors in a plan depends heavily on project criticality, implementation details, resources available, and team opinions. Many projects can achieve outstanding coverage with high-benefit, low-cost unit tests, but they may need to weigh options for larger tests and complex corner cases. Mission critical projects must minimize risk as much as possible, so they will accept higher costs and invest heavily in rigorous testing at all levels.</div><div><br /></div><div>This guide puts the onus on the reader to find the right balance for their project. Also, it does not provide a test plan template, because templates are often too generic or too specific and quickly become outdated. Instead, it focuses on selecting the best content when writing a test plan.</div><div><br /><br /></div><div><h3>Test plan vs. strategy</h3><div><br />Before proceeding, two common methods for defining test plans need to be clarified:</div><div><ul><li><b>Single test plan</b>: Some projects have a single \"test plan\" that describes all implemented and planned testing for the project.</li><li><b>Single test strategy and many plans</b>: Some projects have a \"test strategy\" document as well as many smaller \"test plan\" documents. Strategies typically cover the overall test approach and goals, while plans cover specific features or project updates.</li></ul></div><div>Either of these may be embedded in and integrated with project design documents. Both of these methods work well, so choose whichever makes sense for your project. Generally speaking, stable projects benefit from a single plan, whereas rapidly changing projects are best served by infrequently changed strategies and frequently added plans.</div><div><br /></div><div>For the purpose of this guide, I will refer to both test document types simply as \"test plans”. If you have multiple documents, just apply the advice below to your document aggregation.</div></div><div><br /><br /></div><div><h3>Content selection</h3><div><br />A good approach to creating content for your test plan is to start by listing all questions that need answers. The lists below provide a comprehensive collection of important questions that may or may not apply to your project. Go through the lists and select all that apply. By answering these questions, you will form the contents for your test plan, and you should structure your plan around the chosen content in any format your team prefers. Be sure to balance the factors as mentioned above when making decisions.</div></div><div><br /><br /></div><div><h3>Prerequisites</h3><div><br /></div><div><ul><li><b>Do you need a test plan?</b> If there is no project design document or a clear vision for the product, it may be too early to write a test plan.</li><li><b>Has testability been considered in the project design?</b> Before a project gets too far into implementation, all scenarios must be designed as testable, preferably via automation. Both project design documents and test plans should comment on testability as needed.</li><li><b>Will you keep the plan up-to-date?</b> If so, be careful about adding too much detail, otherwise it may be difficult to maintain the plan.</li><li><b>Does this quality effort overlap with other teams?</b> If so, how have you deduplicated the work?</li></ul></div></div><div><br /></div><div><h3>Risk</h3><div><br /></div><div><ul><li><b>Are there any significant project risks, and how will you mitigate them?</b> Consider: <ul><li>Injury to people or animals</li><li>Security and integrity of user data</li><li>User privacy</li><li>Security of company systems</li><li>Hardware or property damage</li><li>Legal and compliance issues</li><li>Exposure of confidential or sensitive data</li><li>Data loss or corruption</li><li>Revenue loss</li><li>Unrecoverable scenarios</li><li>SLAs</li><li>Performance requirements</li><li>Misinforming users</li><li>Impact to other projects</li><li>Impact from other projects</li><li>Impact to company’s public image</li><li>Loss of productivity</li></ul></li><li><b>What are the project’s technical vulnerabilities?</b> Consider: <ul><li>Features or components known to be hacky, fragile, or in great need of refactoring</li><li>Dependencies or platforms that frequently cause issues</li><li>Possibility for users to cause harm to the system</li><li>Trends seen in past issues</li></ul></li></ul></div></div><div><br /><h3>Coverage</h3><div><br /></div><ul><li><b>What does the test surface look like?</b> Is it a simple library with one method, or a multi-platform client-server stateful system with a combinatorial explosion of use cases? Describe the design and architecture of the system in a way that highlights possible points of failure.</li><li><b>What platforms are supported?</b> Consider listing supported operating systems, hardware, devices, etc. Also describe how testing will be performed and reported for each platform.</li><li><b>What are the features?</b> Consider making a summary list of all features and describe how certain categories of features will be tested.</li><li><b>What will not be tested?</b> No test suite covers every possibility. It’s best to be up-front about this and provide rationale for not testing certain cases. Examples: low risk areas that are a low priority, complex cases that are a low priority, areas covered by other teams, features not ready for testing, etc.&nbsp;</li><li><b>What is covered by unit (small), integration (medium), and system (large) tests?</b> Always test as much as possible in smaller tests, leaving fewer cases for larger tests. Describe how certain categories of test cases are best tested by each test size and provide rationale.</li><li><b>What will be tested manually vs. automated?</b> When feasible and cost-effective, automation is usually best. Many projects can automate all testing. However, there may be good reasons to choose manual testing. Describe the types of cases that will be tested manually and provide rationale.</li><li><b>How are you covering each test category?</b> Consider: <ul><li><a href=\"http://www.w3.org/wiki/Accessibility_testing\">accessibility</a></li><li><a href=\"http://en.wikipedia.org/wiki/Functional_testing\">functional</a></li><li><a href=\"http://en.wikipedia.org/wiki/Fuzz_testing\">fuzz</a></li><li>internationalization and localization</li><li><a href=\"http://en.wikipedia.org/wiki/Software_performance_testing\">performance</a>, <a href=\"http://en.wikipedia.org/wiki/Load_testing\">load</a>, <a href=\"http://en.wikipedia.org/wiki/Stress_testing\">stress</a>, and <a href=\"https://en.wikipedia.org/wiki/Soak_testing\">endurance</a> (aka soak)</li><li>privacy</li><li><a href=\"http://en.wikipedia.org/wiki/Security_testing\">security</a></li><li><a href=\"http://en.wikipedia.org/wiki/Smoke_testing_(software)\">smoke</a></li><li><a href=\"http://en.wikipedia.org/wiki/Stability_testing\">stability</a></li><li><a href=\"http://en.wikipedia.org/wiki/Usability_testing\">usability</a></li></ul></li><li><b>Will you use static and/or dynamic analysis tools?</b> Both <a href=\"https://en.wikipedia.org/wiki/Static_program_analysis\">static analysis tools</a> and <a href=\"https://en.wikipedia.org/wiki/Dynamic_program_analysis\">dynamic analysis tools</a> can find problems that are hard to catch in reviews and testing, so consider using them.</li><li><b>How will system components and dependencies be stubbed, mocked, faked, staged, or used normally during testing?</b> There are good reasons to do each of these, and they each have a unique impact on coverage.</li><li><b>What builds are your tests running against?</b> Are tests running against a build from HEAD (aka tip), a staged build, and/or a release candidate? If only from HEAD, how will you test release build cherry picks (selection of individual changelists for a release) and system configuration changes not normally seen by builds from HEAD?</li><li><b>What kind of testing will be done outside of your team?</b> Examples: <ul><li><a href=\"https://en.wikipedia.org/wiki/Eating_your_own_dog_food\">Dogfooding</a></li><li>External crowdsource testing</li><li>Public alpha/beta versions (how will they be tested before releasing?)</li><li>External trusted testers</li></ul></li><li><b>How are data migrations tested?</b> You may need special testing to compare before and after migration results.</li><li><b>Do you need to be concerned with backward compatibility?</b> You may own previously distributed clients or there may be other systems that depend on your system’s protocol, configuration, features, and behavior.</li><li><b>Do you need to test upgrade scenarios for server/client/device software or dependencies/platforms/APIs that the software utilizes?</b></li><li><b>Do you have line coverage goals?</b></li></ul></div><div><br /><h3>Tooling and Infrastructure</h3><div><br /></div><ul><li><b>Do you need new test frameworks?</b> If so, describe these or add design links in the plan.</li><li><b>Do you need a new test lab setup?</b> If so, describe these or add design links in the plan.</li><li><b>If your project offers a service to other projects, are you providing test tools to those users?</b> Consider providing mocks, fakes, and/or reliable staged servers for users trying to test their integration with your system.</li><li><b>For end-to-end testing, how will test infrastructure, systems under test, and other dependencies be managed?</b> How will they be deployed? How will persistence be set-up/torn-down? How will you handle required migrations from one datacenter to another?</li><li><b>Do you need tools to help debug system or test failures? </b>You may be able to use existing tools, or you may need to develop new ones.</li></ul><br /><h3>Process</h3><div><br /></div><ul><li><b>Are there test schedule requirements? </b>What time commitments have been made, which tests will be in place (or test feedback provided) by what dates? Are some tests important to deliver before others?</li><li><b>How are builds and tests run continuously?</b> Most small tests will be run by <a href=\"https://en.wikipedia.org/wiki/Continuous_integration\">continuous integration</a> tools, but large tests may need a different approach. Alternatively, you may opt for running large tests as-needed.&nbsp;</li><li><b>How will build and test results be reported and monitored? </b><ul><li>Do you have a team rotation to monitor continuous integration?</li><li>Large tests might require monitoring by someone with expertise.</li><li>Do you need a dashboard for test results and other project health indicators?</li><li>Who will get email alerts and how?</li><li>Will the person monitoring tests simply use verbal communication to the team?</li></ul></li><li><b>How are tests used when releasing? </b><ul><li>Are they run explicitly against the release candidate, or does the release process depend only on continuous test results?&nbsp;</li><li>If system components and dependencies are released independently, are tests run for each type of release?&nbsp;</li><li>Will a \"release blocker\" bug stop the release manager(s) from actually releasing? Is there an agreement on what are the release blocking criteria?</li><li>When performing canary releases (aka % rollouts), how will progress be monitored and tested?</li></ul></li><li><b>How will external users report bugs?</b> Consider feedback links or other similar tools to collect and cluster reports.</li><li><b>How does bug triage work? </b>Consider labels or categories for bugs in order for them to land in a triage bucket. Also make sure the teams responsible for filing and or creating the bug report template are aware of this. Are you using one bug tracker or do you need to setup some automatic or manual import routine?</li><li><b>Do you have a policy for submitting new tests before closing bugs that could have been caught?</b></li><li><b>How are tests used for unsubmitted changes?</b> If anyone can run all tests against any experimental build (a good thing), consider providing a howto.</li><li><b>How can team members create and/or debug tests?</b> Consider providing a howto.</li></ul><br /><h3>Utility</h3><div><br /></div><ul><li><b>Who are the test plan readers?</b> Some test plans are only read by a few people, while others are read by many. At a minimum, you should consider getting a review from all stakeholders (project managers, tech leads, feature owners). When writing the plan, be sure to understand the expected readers, provide them with enough background to understand the plan, and answer all questions you think they will have - even if your answer is that you don’t have an answer yet. Also consider adding contacts for the test plan, so any reader can get more information.</li><li><b>How can readers review the actual test cases?</b> Manual cases might be in a test case management tool, in a separate document, or included in the test plan. Consider providing links to directories containing automated test cases.</li><li><b>Do you need traceability between requirements, features, and tests?</b></li><li><b>Do you have any general product health or quality goals and how will you measure success?</b> Consider: <ul><li>Release cadence</li><li>Number of bugs caught by users in production</li><li>Number of bugs caught in release testing</li><li>Number of open bugs over time</li><li>Code coverage</li><li>Cost of manual testing</li><li>Difficulty of creating new tests</li></ul></li></ul></div><br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=A8l6jMC3smM:l_oRTDHa9fw:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=A8l6jMC3smM:l_oRTDHa9fw:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=A8l6jMC3smM:l_oRTDHa9fw:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/A8l6jMC3smM\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Anthony Valloneupdated: July 2016Creating a test plan is often a complex undertaking. An ideal test plan is accomplished by applying basic principles of cost-benefit analysis and risk analysis, optimally balancing these software development factors:Implementation cost: The time and complexity of implementing testable features and automated tests for specific scenarios will vary, and this affects short-term development cost.Maintenance cost: Some tests or test plans may vary from easy to difficult to maintain, and this affects long-term development cost. When manual testing is chosen, this also adds to long-term cost.Monetary cost: Some test approaches may require billed resources.Benefit: Tests are capable of preventing issues and aiding productivity by varying degrees. Also, the earlier they can catch problems in the development life-cycle, the greater the benefit.Risk: The probability of failure scenarios may vary from rare to likely, and their consequences may vary from minor nuisance to catastrophic.Effectively balancing these factors in a plan depends heavily on project criticality, implementation details, resources available, and team opinions. Many projects can achieve outstanding coverage with high-benefit, low-cost unit tests, but they may need to weigh options for larger tests and complex corner cases. Mission critical projects must minimize risk as much as possible, so they will accept higher costs and invest heavily in rigorous testing at all levels.This guide puts the onus on the reader to find the right balance for their project. Also, it does not provide a test plan template, because templates are often too generic or too specific and quickly become outdated. Instead, it focuses on selecting the best content when writing a test plan.Test plan vs. strategyBefore proceeding, two common methods for defining test plans need to be clarified:Single test plan: Some projects have a single \"test plan\" that describes all implemented and planned testing for the project.Single test strategy and many plans: Some projects have a \"test strategy\" document as well as many smaller \"test plan\" documents. Strategies typically cover the overall test approach and goals, while plans cover specific features or project updates.Either of these may be embedded in and integrated with project design documents. Both of these methods work well, so choose whichever makes sense for your project. Generally speaking, stable projects benefit from a single plan, whereas rapidly changing projects are best served by infrequently changed strategies and frequently added plans.For the purpose of this guide, I will refer to both test document types simply as \"test plans”. If you have multiple documents, just apply the advice below to your document aggregation.Content selectionA good approach to creating content for your test plan is to start by listing all questions that need answers. The lists below provide a comprehensive collection of important questions that may or may not apply to your project. Go through the lists and select all that apply. By answering these questions, you will form the contents for your test plan, and you should structure your plan around the chosen content in any format your team prefers. Be sure to balance the factors as mentioned above when making decisions.PrerequisitesDo you need a test plan? If there is no project design document or a clear vision for the product, it may be too early to write a test plan.Has testability been considered in the project design? Before a project gets too far into implementation, all scenarios must be designed as testable, preferably via automation. Both project design documents and test plans should comment on testability as needed.Will you keep the plan up-to-date? If so, be careful about adding too much detail, otherwise it may be difficult to maintain the plan.Does this quality effort overlap with other teams? If so, how have you deduplicated the work?RiskAre there any significant project risks, and how will you mitigate them? Consider: Injury to people or animalsSecurity and integrity of user dataUser privacySecurity of company systemsHardware or property damageLegal and compliance issuesExposure of confidential or sensitive dataData loss or corruptionRevenue lossUnrecoverable scenariosSLAsPerformance requirementsMisinforming usersImpact to other projectsImpact from other projectsImpact to company’s public imageLoss of productivityWhat are the project’s technical vulnerabilities? Consider: Features or components known to be hacky, fragile, or in great need of refactoringDependencies or platforms that frequently cause issuesPossibility for users to cause harm to the systemTrends seen in past issuesCoverageWhat does the test surface look like? Is it a simple library with one method, or a multi-platform client-server stateful system with a combinatorial explosion of use cases? Describe the design and architecture of the system in a way that highlights possible points of failure.What platforms are supported? Consider listing supported operating systems, hardware, devices, etc. Also describe how testing will be performed and reported for each platform.What are the features? Consider making a summary list of all features and describe how certain categories of features will be tested.What will not be tested? No test suite covers every possibility. It’s best to be up-front about this and provide rationale for not testing certain cases. Examples: low risk areas that are a low priority, complex cases that are a low priority, areas covered by other teams, features not ready for testing, etc.&nbsp;What is covered by unit (small), integration (medium), and system (large) tests? Always test as much as possible in smaller tests, leaving fewer cases for larger tests. Describe how certain categories of test cases are best tested by each test size and provide rationale.What will be tested manually vs. automated? When feasible and cost-effective, automation is usually best. Many projects can automate all testing. However, there may be good reasons to choose manual testing. Describe the types of cases that will be tested manually and provide rationale.How are you covering each test category? Consider: accessibilityfunctionalfuzzinternationalization and localizationperformance, load, stress, and endurance (aka soak)privacysecuritysmokestabilityusabilityWill you use static and/or dynamic analysis tools? Both static analysis tools and dynamic analysis tools can find problems that are hard to catch in reviews and testing, so consider using them.How will system components and dependencies be stubbed, mocked, faked, staged, or used normally during testing? There are good reasons to do each of these, and they each have a unique impact on coverage.What builds are your tests running against? Are tests running against a build from HEAD (aka tip), a staged build, and/or a release candidate? If only from HEAD, how will you test release build cherry picks (selection of individual changelists for a release) and system configuration changes not normally seen by builds from HEAD?What kind of testing will be done outside of your team? Examples: DogfoodingExternal crowdsource testingPublic alpha/beta versions (how will they be tested before releasing?)External trusted testersHow are data migrations tested? You may need special testing to compare before and after migration results.Do you need to be concerned with backward compatibility? You may own previously distributed clients or there may be other systems that depend on your system’s protocol, configuration, features, and behavior.Do you need to test upgrade scenarios for server/client/device software or dependencies/platforms/APIs that the software utilizes?Do you have line coverage goals?Tooling and InfrastructureDo you need new test frameworks? If so, describe these or add design links in the plan.Do you need a new test lab setup? If so, describe these or add design links in the plan.If your project offers a service to other projects, are you providing test tools to those users? Consider providing mocks, fakes, and/or reliable staged servers for users trying to test their integration with your system.For end-to-end testing, how will test infrastructure, systems under test, and other dependencies be managed? How will they be deployed? How will persistence be set-up/torn-down? How will you handle required migrations from one datacenter to another?Do you need tools to help debug system or test failures? You may be able to use existing tools, or you may need to develop new ones.ProcessAre there test schedule requirements? What time commitments have been made, which tests will be in place (or test feedback provided) by what dates? Are some tests important to deliver before others?How are builds and tests run continuously? Most small tests will be run by continuous integration tools, but large tests may need a different approach. Alternatively, you may opt for running large tests as-needed.&nbsp;How will build and test results be reported and monitored? Do you have a team rotation to monitor continuous integration?Large tests might require monitoring by someone with expertise.Do you need a dashboard for test results and other project health indicators?Who will get email alerts and how?Will the person monitoring tests simply use verbal communication to the team?How are tests used when releasing? Are they run explicitly against the release candidate, or does the release process depend only on continuous test results?&nbsp;If system components and dependencies are released independently, are tests run for each type of release?&nbsp;Will a \"release blocker\" bug stop the release manager(s) from actually releasing? Is there an agreement on what are the release blocking criteria?When performing canary releases (aka % rollouts), how will progress be monitored and tested?How will external users report bugs? Consider feedback links or other similar tools to collect and cluster reports.How does bug triage work? Consider labels or categories for bugs in order for them to land in a triage bucket. Also make sure the teams responsible for filing and or creating the bug report template are aware of this. Are you using one bug tracker or do you need to setup some automatic or manual import routine?Do you have a policy for submitting new tests before closing bugs that could have been caught?How are tests used for unsubmitted changes? If anyone can run all tests against any experimental build (a good thing), consider providing a howto.How can team members create and/or debug tests? Consider providing a howto.UtilityWho are the test plan readers? Some test plans are only read by a few people, while others are read by many. At a minimum, you should consider getting a review from all stakeholders (project managers, tech leads, feature owners). When writing the plan, be sure to understand the expected readers, provide them with enough background to understand the plan, and answer all questions you think they will have - even if your answer is that you don’t have an answer yet. Also consider adding contacts for the test plan, so any reader can get more information.How can readers review the actual test cases? Manual cases might be in a test case management tool, in a separate document, or included in the test plan. Consider providing links to directories containing automated test cases.Do you need traceability between requirements, features, and tests?Do you have any general product health or quality goals and how will you measure success? Consider: Release cadenceNumber of bugs caught by users in productionNumber of bugs caught in release testingNumber of open bugs over timeCode coverageCost of manual testingDifficulty of creating new tests",
        "id": "tag:blogger.com,1999:blog-15045980.post-3445090868451415297",
        "isoDate": "2016-06-06T13:05:00.000Z"
      },
      {
        "title": "GTAC 2016 Registration Deadline Extended",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/ubmtWki7vUU/gtac-2016-registration-deadline-extended.html",
        "pubDate": "2016-06-03T03:58:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by Sonal Shah on behalf of the GTAC Committee</i><br /><br />Our goal in organizing GTAC each year is to make it a first-class conference, dedicated to presenting leading edge industry practices. The quality of submissions we've received for GTAC 2016 so far has been overwhelming. In order to include the best talks possible, we are extending the deadline for speaker and attendee submissions by 15 days. The new timelines are as follows:<br /><br /><strike>June 1, 2016</strike> <b>June 15, 2016</b> - Last day for speaker, attendee and diversity scholarship submissions.<br /><strike>June 15, 2016</strike> <b>July 15, 2016</b> - Attendees and scholarship awardees will be notified of selection/rejection/waitlist status. Those on the waitlist will be notified as space becomes available.<br /><strike>August 15, 2016</strike> <b>August 29, 2016</b> - Selected speakers will be notified.<br /><br />To register, please fill out <a href=\"https://docs.google.com/a/google.com/forms/d/1gT3jZtP03a38zaJ6gFUXb74PW6MugustSem938UWv3c/viewform\">this</a> form.<br />To apply for diversity scholarship, please fill out <a href=\"https://docs.google.com/a/google.com/forms/d/1PozpfkbSH3YX_BVbXRhju5jkZNy-gGcs45dnKIvIEsE/edit?usp=sharing\">this</a> form.<br /><br />The <a href=\"https://developers.google.com/google-test-automation-conference/2016/faq-conference\">GTAC website</a> has a list of frequently asked questions. Please do not hesitate to contact <a href=\"mailto:gtac2016@google.com\">gtac2016@google.com</a> if you still have any questions.<br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ubmtWki7vUU:aGYxyHEr7gs:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=ubmtWki7vUU:aGYxyHEr7gs:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=ubmtWki7vUU:aGYxyHEr7gs:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/ubmtWki7vUU\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Sonal Shah on behalf of the GTAC CommitteeOur goal in organizing GTAC each year is to make it a first-class conference, dedicated to presenting leading edge industry practices. The quality of submissions we've received for GTAC 2016 so far has been overwhelming. In order to include the best talks possible, we are extending the deadline for speaker and attendee submissions by 15 days. The new timelines are as follows:June 1, 2016 June 15, 2016 - Last day for speaker, attendee and diversity scholarship submissions.June 15, 2016 July 15, 2016 - Attendees and scholarship awardees will be notified of selection/rejection/waitlist status. Those on the waitlist will be notified as space becomes available.August 15, 2016 August 29, 2016 - Selected speakers will be notified.To register, please fill out this form.To apply for diversity scholarship, please fill out this form.The GTAC website has a list of frequently asked questions. Please do not hesitate to contact gtac2016@google.com if you still have any questions.",
        "id": "tag:blogger.com,1999:blog-15045980.post-5031427020801457462",
        "isoDate": "2016-06-03T03:58:00.000Z"
      },
      {
        "title": "Flaky Tests at Google and How We Mitigate Them",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/MuFb9AlOInw/flaky-tests-at-google-and-how-we.html",
        "pubDate": "2016-05-28T00:31:00.000Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by John Micco</i><br /><br />At Google, we run a very large corpus of tests continuously to validate our code submissions. Everyone from developers to project managers rely on the results of these tests to make decisions about whether the system is ready for deployment or whether code changes are OK to submit. Productivity for developers at Google relies on the ability of the tests to find real problems with the code being changed or developed in a timely and reliable fashion. <br /><br />Tests are run before submission (pre-submit testing) which gates submission and verifies that changes are acceptable, and again after submission (post-submit testing) to decide whether the project is ready to be released. In both cases, all of the tests for a particular project must report a passing result before submitting code or releasing a project.<br /><br />Unfortunately, across our entire corpus of tests, we see a continual rate of about 1.5% of all test runs reporting a \"flaky\" result. We define a \"flaky\" test result as a test that exhibits both a passing and a failing result with the same code. There are many <a href=\"http://www.cs.umd.edu/~atif/pubs/gao-icse15.pdf\">root causes</a> why tests return flaky results, including concurrency, relying on non-deterministic or undefined behaviors, flaky third party code, infrastructure problems, etc. We have invested a lot of effort in removing flakiness from tests, but overall the insertion rate is about the same as the fix rate, meaning we are stuck with a certain rate of tests that provide value, but occasionally produce a flaky result. Almost 16% of our tests have some level of flakiness associated with them! This is a staggering number; it means that more than 1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests.<br /><br />When doing post-submit testing, our Continuous Integration (CI) system identifies when a passing test transitions to failing, so that we can investigate the code submission that caused the failure. What we find in practice is that about 84% of the transitions we observe from pass to fail involve a flaky test! This causes extra repetitive work to determine whether a new failure is a flaky result or a legitimate failure. It is quite common to ignore legitimate failures in flaky tests due to the high number of false-positives. At the very least, build monitors typically wait for additional CI cycles to run this test again to determine whether or not the test has been broken by a submission adding to the delay of identifying real problems and increasing the pool of changes that could contribute.<br /><br />In addition to the cost of build monitoring, consider that the average project contains 1000 or so individual tests. To release a project, we require that all these tests pass with the latest code changes. If 1.5% of test results are flaky, 15 tests will likely fail, requiring expensive investigation by a build cop or developer. In some cases, developers dismiss a failing result as flaky only to later realize that it was a legitimate failure caused by the code. It is human nature to ignore alarms when there is a history of false signals coming from a system. For example, see <a href=\"http://www.travelweekly.com/Travel-News/Airline-News/Analysis-shows-pilots-often-ignore-Boeing-737-cockpit-alarm\">this article</a> about airline pilots ignoring an alarm on 737s. The same phenomenon occurs with pre-submit testing.  The same 15 or so failing tests block submission and introduce costly delays into the core development process. Ignoring legitimate failures at this stage results in the submission of broken code.<br /><br />We have several mitigation strategies for flaky tests during presubmit testing, including the ability to re-run only failing tests, and an option to re-run tests automatically when they fail. We even have a way to denote a test as flaky - causing it to report a failure only if it fails 3 times in a row. This reduces false positives, but encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row, which is hardly a perfect solution.<br />Imagine a 15 minute integration test marked as flaky that is broken by my code submission. The breakage will not be discovered until 3 executions of the test complete, or 45 minutes, after which it will need to be determined if the test is broken (and needs to be fixed) or if the test just flaked three times in a row.<br /><br />Other mitigation strategies include:<br /><ul><li>A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness. This prevents it from becoming a problem for developers, but could easily mask a real race condition or some other bug in the code being tested.</li><li>Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.</li></ul><br />In summary, test flakiness is an important problem, and Google is continuing to invest in detecting, mitigating, tracking, and fixing test flakiness throughout our code base. For example:<br /><ul><li>We have a new team dedicated to providing accurate and timely information about test flakiness to help developers and build monitors so that they know whether they are being harmed by test flakiness.</li><li>As we analyze the data from flaky test executions, we are seeing promising correlations with features that should enable us to identify a flaky result accurately without re-running the test.</li></ul><br /><br />By continually advancing the state of the art for teams at Google, we aim to remove the friction caused by test flakiness from the core developer workflows.<br /><br /><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=MuFb9AlOInw:m50BW0gKpt4:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=MuFb9AlOInw:m50BW0gKpt4:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=MuFb9AlOInw:m50BW0gKpt4:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/MuFb9AlOInw\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by John MiccoAt Google, we run a very large corpus of tests continuously to validate our code submissions. Everyone from developers to project managers rely on the results of these tests to make decisions about whether the system is ready for deployment or whether code changes are OK to submit. Productivity for developers at Google relies on the ability of the tests to find real problems with the code being changed or developed in a timely and reliable fashion. Tests are run before submission (pre-submit testing) which gates submission and verifies that changes are acceptable, and again after submission (post-submit testing) to decide whether the project is ready to be released. In both cases, all of the tests for a particular project must report a passing result before submitting code or releasing a project.Unfortunately, across our entire corpus of tests, we see a continual rate of about 1.5% of all test runs reporting a \"flaky\" result. We define a \"flaky\" test result as a test that exhibits both a passing and a failing result with the same code. There are many root causes why tests return flaky results, including concurrency, relying on non-deterministic or undefined behaviors, flaky third party code, infrastructure problems, etc. We have invested a lot of effort in removing flakiness from tests, but overall the insertion rate is about the same as the fix rate, meaning we are stuck with a certain rate of tests that provide value, but occasionally produce a flaky result. Almost 16% of our tests have some level of flakiness associated with them! This is a staggering number; it means that more than 1 in 7 of the tests written by our world-class engineers occasionally fail in a way not caused by changes to the code or tests.When doing post-submit testing, our Continuous Integration (CI) system identifies when a passing test transitions to failing, so that we can investigate the code submission that caused the failure. What we find in practice is that about 84% of the transitions we observe from pass to fail involve a flaky test! This causes extra repetitive work to determine whether a new failure is a flaky result or a legitimate failure. It is quite common to ignore legitimate failures in flaky tests due to the high number of false-positives. At the very least, build monitors typically wait for additional CI cycles to run this test again to determine whether or not the test has been broken by a submission adding to the delay of identifying real problems and increasing the pool of changes that could contribute.In addition to the cost of build monitoring, consider that the average project contains 1000 or so individual tests. To release a project, we require that all these tests pass with the latest code changes. If 1.5% of test results are flaky, 15 tests will likely fail, requiring expensive investigation by a build cop or developer. In some cases, developers dismiss a failing result as flaky only to later realize that it was a legitimate failure caused by the code. It is human nature to ignore alarms when there is a history of false signals coming from a system. For example, see this article about airline pilots ignoring an alarm on 737s. The same phenomenon occurs with pre-submit testing.  The same 15 or so failing tests block submission and introduce costly delays into the core development process. Ignoring legitimate failures at this stage results in the submission of broken code.We have several mitigation strategies for flaky tests during presubmit testing, including the ability to re-run only failing tests, and an option to re-run tests automatically when they fail. We even have a way to denote a test as flaky - causing it to report a failure only if it fails 3 times in a row. This reduces false positives, but encourages developers to ignore flakiness in their own tests unless their tests start failing 3 times in a row, which is hardly a perfect solution.Imagine a 15 minute integration test marked as flaky that is broken by my code submission. The breakage will not be discovered until 3 executions of the test complete, or 45 minutes, after which it will need to be determined if the test is broken (and needs to be fixed) or if the test just flaked three times in a row.Other mitigation strategies include:A tool that monitors the flakiness of tests and if the flakiness is too high, it automatically quarantines the test. Quarantining removes the test from the critical path and files a bug for developers to reduce the flakiness. This prevents it from becoming a problem for developers, but could easily mask a real race condition or some other bug in the code being tested.Another tool detects changes in the flakiness level of tests and works to identify the change that caused the test to change the level of flakiness.In summary, test flakiness is an important problem, and Google is continuing to invest in detecting, mitigating, tracking, and fixing test flakiness throughout our code base. For example:We have a new team dedicated to providing accurate and timely information about test flakiness to help developers and build monitors so that they know whether they are being harmed by test flakiness.As we analyze the data from flaky test executions, we are seeing promising correlations with features that should enable us to identify a flaky result accurately without re-running the test.By continually advancing the state of the art for teams at Google, we aim to remove the friction caused by test flakiness from the core developer workflows.",
        "id": "tag:blogger.com,1999:blog-15045980.post-6277555988217606236",
        "isoDate": "2016-05-28T00:31:00.000Z"
      },
      {
        "title": "GTAC Diversity Scholarship",
        "link": "http://feedproxy.google.com/~r/blogspot/RLXA/~3/aXFut9UQVhE/gtac-diversity-scholarship.html",
        "pubDate": "2016-05-04T14:32:00.001Z",
        "author": "Google Testing Bloggers",
        "content": "<i>by Lesley Katzen on behalf of the GTAC Diversity Committee</i><br /><br />We are committed to increasing diversity at GTAC, and we believe the best way to do that is by making sure we have a diverse set of applicants to speak and attend. As part of that commitment, we are excited to announce that we will be offering travel scholarships this year.<br />Travel scholarships will be available for selected applicants from traditionally underrepresented groups in technology.<br /><br />To be eligible for a grant to attend GTAC, applicants must:<br /><br /><ul><li>Be 18 years of age or older.</li><li>Be from a traditionally underrepresented group in technology.</li><li>Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing.</li><li>Be able to attend core dates of GTAC, November 15th - 16th 2016 in Sunnyvale, CA.</li></ul><br /><br /><b>To apply:</b><br />Please fill out the following <a href=\"https://docs.google.com/a/google.com/forms/d/1PozpfkbSH3YX_BVbXRhju5jkZNy-gGcs45dnKIvIEsE/edit?usp=sharing\">form</a> to be considered for a travel scholarship.<br />The deadline for submission is <strike>June 1st</strike> June 15th. &nbsp;Scholarship recipients will be announced on <strike>June 30th</strike> July 15th. If you are selected, we will contact you with information on how to proceed with booking travel.<br /><br /><br /><b>What the scholarship covers:</b><br />Google will pay for standard coach class airfare for selected scholarship recipients to San Francisco or San Jose, and 3 nights of accommodations in a hotel near the Sunnyvale campus. Breakfast and lunch will be provided for GTAC attendees and speakers on both days of the conference. We will also provide a $50.00 gift card for other incidentals such as airport transportation or meals. You will need to provide your own credit card to cover any hotel incidentals.<br /><br /><br />Google is dedicated to providing a harassment-free and inclusive conference experience for everyone. Our anti-harassment policy can be found at:<br /><a href=\"https://www.google.com/events/policy/anti-harassmentpolicy.html\">https://www.google.com/events/policy/anti-harassmentpolicy.html</a><div class=\"feedflare\">\r\n<a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=aXFut9UQVhE:xOkTBcVdW-A:yIl2AUoC8zA\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?d=yIl2AUoC8zA\" border=\"0\"></img></a> <a href=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?a=aXFut9UQVhE:xOkTBcVdW-A:V_sGLiPBpWU\"><img src=\"http://feeds.feedburner.com/~ff/blogspot/RLXA?i=aXFut9UQVhE:xOkTBcVdW-A:V_sGLiPBpWU\" border=\"0\"></img></a>\r\n</div><img src=\"http://feeds.feedburner.com/~r/blogspot/RLXA/~4/aXFut9UQVhE\" height=\"1\" width=\"1\" alt=\"\"/>",
        "contentSnippet": "by Lesley Katzen on behalf of the GTAC Diversity CommitteeWe are committed to increasing diversity at GTAC, and we believe the best way to do that is by making sure we have a diverse set of applicants to speak and attend. As part of that commitment, we are excited to announce that we will be offering travel scholarships this year.Travel scholarships will be available for selected applicants from traditionally underrepresented groups in technology.To be eligible for a grant to attend GTAC, applicants must:Be 18 years of age or older.Be from a traditionally underrepresented group in technology.Work or study in Computer Science, Computer Engineering, Information Technology, or a technical field related to software testing.Be able to attend core dates of GTAC, November 15th - 16th 2016 in Sunnyvale, CA.To apply:Please fill out the following form to be considered for a travel scholarship.The deadline for submission is June 1st June 15th. &nbsp;Scholarship recipients will be announced on June 30th July 15th. If you are selected, we will contact you with information on how to proceed with booking travel.What the scholarship covers:Google will pay for standard coach class airfare for selected scholarship recipients to San Francisco or San Jose, and 3 nights of accommodations in a hotel near the Sunnyvale campus. Breakfast and lunch will be provided for GTAC attendees and speakers on both days of the conference. We will also provide a $50.00 gift card for other incidentals such as airport transportation or meals. You will need to provide your own credit card to cover any hotel incidentals.Google is dedicated to providing a harassment-free and inclusive conference experience for everyone. Our anti-harassment policy can be found at:https://www.google.com/events/policy/anti-harassmentpolicy.html",
        "id": "tag:blogger.com,1999:blog-15045980.post-6877285779470090432",
        "isoDate": "2016-05-04T14:32:00.001Z"
      }
    ],
    "link": "http://testing.googleblog.com/",
    "feedUrl": "http://www.blogger.com/feeds/15045980/posts/default?start-index=26&max-results=25&redirect=false",
    "title": "Google Testing Blog",
    "lastBuildDate": "2017-10-18T03:40:38.732-07:00"
  }
}