How Microsoft went from Test to No Test to Test again
I was lucky enough (or “lucky” depending on who you ask) to be working at Microsoft at the time when something they called “combined engineering” happened around 2014/2015. What this meant in reality was that all the testers were fired or turned into developers.
But the story, like Microsoft itself goes back a lot further. So here is a play in 4 acts.
Act 1 - Microsoft in the 1990s
Although Microsoft was founded already in 1975, there weren’t any major or interesting changes in the company until 1990s when software development processes had had the time to evolve and MS execs realised testing needs to evolve as well.
At this time there was a rough ratio of 1 developer to 1 tester to 0.5 PM in any given engineering team.
There was actually more than a single testing role - it had been broken up into two very distinct roles - Software Design Engineer in Test and Software Test engineer.
SDEs were essentially developers, who set up and maintained all the test infrastructure and developed automation.
SDETs were functional testers who ran the manual and automated tests, reported bugs and tried to push developers to fix them.
This approach worked quite well given how well Windows OS and Office suite products were received, not to mention their commercial success.
But the world was evolving and waterfall approach was starting to fall out of favour, as software rollout speed was starting to be an issue.
There were actually a number of other issues as well:
testing org kept growing at a high rate, including outsourced testers
SDETs threw automation over the wall to SDEs
Devs threw code over the wall to SDETs
Test automation was starting to be expensive to maintain
Testing (at the end of the cycle) became a bottleneck, causing release delays (well, actually bugs caused delays not testing)
There was growing realization that these issues are not going to get better, they’re going to get worse. Something had to change. And so it did.
Act Two - The New Millennium
The year 2000 was here and so was agile and so were new ways of thinking about testing.
It started with removing the STE role. Many of them were let go, others were turned into SDETs and a few took a “regular” developer career path.
But it wasn’t a major blow to just STEs, developers suffered equally having found themselves in a situation where quality assurance was more in their own hands than ever before.
More than anything this meant that the majority of quality work fell on SDETs shoulders, now being the only test role at Microsoft. Being mostly on par with regular developers, their approach was to solve the problem of “more things to test with the same amount of time” with automation.
In general this approach worked, as most of the projects were still using waterfall. There was also an attempt to establish something called Milestone Quality. In reality these were tech debt and test debt focused sprints. On paper it seemed like a great idea, but what happened was that all the testing work was being pushed forward to MQ sprints and quality suffered. Fortunately everyone noticed and Milestone Quality was quickly a thing of the past.
As 2010 was approaching, so was the prevalence of agile and the push to release more often.
Act 3 - Microsoft in 2010
The proliferation of agile had surfaced the same problem that Microsoft had already battled in the 90s - too many stakeholders and people throwing stuff over the wall to each other without too much accountability. When in the 90s the fix for this was to combine the two existing testing roles, then now the idea was more radical - to combine the roles of dev and test.
This would cause a number of problems at first, but would solve a number of major issues in the long run.
The first problem, as mentioned, was accountability. The idea was that developers should be able to detect the bugs they or their teammates had caused, and fix them. Not only did this mean that developers would still do everything they had been doing so far, now they had to learn how to do test automation, triaging, manual testing - and most importantly to change their mindset, to find issues in the code they had just written. Yikes!
It wasn’t any easier for testers. They had to start shipping code, writing features and everything else that developers had been doing. Although this wasn’t supposed to be an overnight process - in fact the transition was planned to take place over one year period, still many testers decided to leave, daunted by this shift.
The original plan was that over this year developers would shift closer to what testers were doing day-to-day and vice versa, and in a years time there would be “combined engineering”.
In reality there was a lot that testers focused on based on their experience and hunch, which turned out to be not easily teachable. At the same time it became very clear that you can’t turn every non-technical person into a capable developer, so a number of people had to be let go for not meeting their targets.
The year was over and dust had settled. In general the transition had worked, surprisingly the velocity hadn’t dropped, teams were still shipping features at a similar and slightly increasing rate. But also once the dust did settle we could observe that ex-testers were still doing roughly 70% testing work and 30% developer work. The same ratios but the other way around were also true for ex-developers.
This isn’t the whole story however. Microsoft didn’t actually fully get rid of testers, and the probably never will - a large number of testing is outsourced, this way teams can still have traditional testers and personally I believe these outsourced contributors were the key to the success of the whole transition, although my own experience with outsourced testing while at Microsoft has been extremely mixed.
Another contributor to the success of (or at least partial success) had been what are called “v-teams”. These “virtual teams” consisted of mostly experienced developers whose job was to figure out the testing infrastructure, which other teams could use. Its quite common for FAANG-sized companies to have extensive in-house tooling teams, and the test infrastructure v-team definitely played an important role in the transition.
📧 Join our over 17 000 subscribers from top tech companies for more insights to quality engineering. Also we’re giving away free courses and books to new subscribers each month!
Act 4 - The Future and the return of The Tester
There are a number of quality initiatives inside Microsoft, which officially are already implemented, but in reality based on what I’ve seen is still work in progress. But once they do succeed, they’re definitely a step in the right direction.
Shift Left - increase the emphasis on unit tests and reducing test flakiness. The idea is that this way the need for manual testing is reduced and it helps with shipping faster (potentially several times an hour)
Keep master shippable - an initiative where master branch should be in such a state at all times that we could build and release a new version to public. This means a lot of feature work in kept in branches for longer, which creates its own issues, like messy merge conflicts once they do get merged to master
Testing in production - nope, this isn’t a joke. A topic warranting its own article (or a book), but for now you can read more about it here
Summary
A lot has changed over the last 30 years, but there are some overarching themes. The main one being accountability.
Personally I don’t believe that the tester role is disappearing. Actually while combined engineering is still in place now, Microsoft is actively hiring new people for purely testing roles, which gives me hope.
The idea of accountability has probably been taken too far and the notion that developer has to be a good developer and a good tester is probably not realistic.