Windows 95 Boosts Productivity

A test conducted by International Data Corp. shows Windows 95 users were able to complete a series of business computing tasks 19% faster than Mac users and 50% faster than OS/2 users. The tasks included managing and printing local and networked files, managing documents and software programs, checking system resources, creating shortcuts and customizing the desktop. (Investor's Business Daily 27 Nov 95 A6)

Apple's Response
IBM's Response
Microsoft's Press Release
My Comments

Apple's Response:

It was a disappointment when Apple was briefed a couple of weeks ago on a new computer usage study sponsored by Microsoft. It's basically a flawed imitation of the Arthur D. Little productivity study that was sponsored by Apple a couple of years ago.

This study has some serious technical and methodological flaws and in our professional opinion was not implemented well. We wanted to share our concerns about the study.

Background on the study

About two years ago, Apple sponsored a study in which Arthur D. Little tested the productivity of Windows 3 and Macintosh users performing 24 common tasks, in ten categories, on their computers. The study was designed to document something that virtually all industry analysts knew was true but couldn't quantify: That people using Macs could do work faster and better.

Microsoft's study appears to partially duplicate seven of the ten categories, and adds three new ones (we say "partially" because Microsoft changed many of the individual tests, and because we can't get access to all the details of the testing; they're a secret). The study also tests OS/2 in addition to Mac and Windows 95. Microsoft claims its tests show that Windows 95 users are 19% faster than Mac users, and somewhat more accurate in their work.

That's the headline I expect you to hear when the study is released. What they won't tell you is that the methodology of the tests is severely flawed and appears to contain some serious biases. You should insist on reading their full report very carefully.

Concerns about the study

We feel that the methodology of the tests was severely flawed and there appear to have been some serious biases. Here are some examples:

  • The user sample was biased. The Microsoft users in the study were participants in the Windows 95 Preview Program, suggesting that they were in the category of early adopters or technology enthusiasts. At least 20% of the Mac users in the study were selected from a list provided by Microsoft.

  • The tests are biased in favor of Windows. Tests were suggested by Microsoft that unfairly highlighted Windows 95 features. A "find file" test used Windows bias (find a file "between" a date) that handicapped Mac users (which can find a file "around" a date). Because the test is timed, any hesitation or fumbling around changes the results.

  • The tests are biased against Mac users. One test asked the user to install print drivers off a floppy for a Postscript printer that Mac users would not ordinarily have to do since the HP printer involved in the test is automatically supported by the standard Macintosh PostScript driver. Another test asked users to determine the amount of "free disk space" and "capacity" of the hard drive. Windows 95 uses those exact labels on screen; Macintosh uses the labels "MB in disk" and "MB available." The network test was also not fair in that the Macs and PCs were set up with different protocols and services.

  • The tests are biased in what they don't cover:

  • The tests cover only a limited number of "operating system" operations, representing only a fraction of what is involved in defining overall personal computer productivity.

  • The tests did not include many common system level functions in which Macintosh excels but that Windows 95 cannot handle well:
  • Moving folders around (which breaks links between applications)
  • Installing or uninstalling applications (in which a Mac has a clear advantage)
  • Renaming or moving shortcuts (which can easily be confused in Windows 95)
  • Dealing with short and long file name issues (a major problem for Windows 95)
  • Learning about the file system (Windows 95 still has C:\ pathnames)
  • Renaming documents anything the user wants (Windows 95 has restrictions on characters and length of "pathname")
  • Changing number of colors (which the test warned Windows users explicitly not to do since doing so would require a reboot for Windows 95).
  • Installing the operating system itself.
  • The tests don't cover many important areas that can be the real driving factors in user productivity:
  • System troubleshooting. When something goes wrong, Windows 95's complexity makes troubleshooting much more difficult than Mac troubleshooting. Huge WIN.INI & SYSTEM.INI text files, CONFIG.SYS & AUTOEXEC.BAT for DOS and Win 3.1 apps, and hundreds of mysterious system files are overwhelming to many users trying to resolve a problem.

  • Learning the system. The test only samples experienced users. Macintosh ease of learning would likely give Macintosh a strong edge if such a test were included.

  • Integrating new hardware. The Mac's seamless integration of hardware and software obviously gives Apple a big advantage. Plug & play is just beginning in the Windows world and users must replace all of their hardware and peripherals before they experience the benefits Mac users have had for 11 years.

  • Utilizing advanced technology. This is, of course, where the Mac truly shines. The Mac has many technology advantages that can make an even bigger impact on overall user productivity and effectiveness - AV, 3D, VR, Videoconferencing, Internet publishing, telephony, advanced built-in hardware, active assistance, speech, RISC speed, universal mailbox, GX, multimedia authoring, and more.

  • Conclusions

    For the record, Apple believes that the basic usability of Windows 95 does come closer to a Mac than Windows 3.1 did. We think there are still some major usability advantages in the Mac, though -- things like built-in scripting and a much easier to use file system. With the next version of the Mac OS, we expect to widen that lead. But the most important point is that the industry is moving beyond basic ease of use to other issues, such as the ease of working with multimedia and the Internet. Those are the areas in which today's Macs really shine. And they're not even addressed by this study.

    Mac Platform Marketing
    Apple Computer, Inc.

    IBM's Response

    Microsoft recently released results from a study which indicated that users of Windows 95 outperformed users of OS/2. The study raises more questions than it answers.

    Question: Outperformed doing what?

    This study was conducted, by IDC, at the request of Microsoft and was fully funded and directed by Microsoft. IBM has attempted to review the study, in more detail, but has been denied access to the actual test. All descriptions of tasks were written by Microsoft. IDC did read, over the phone, descriptions of the tasks users were asked to perform. Surprisingly, there were no multitasking questions, no measuring of one's ability to access the Internet through popular service providers, update a spreadsheet, or fax and file documents. The study apparently was based on an Apple study done three years ago. Not very representative of today's environment and not designed to take advantage of true 32 bit technology. Had the study really examined how many steps it takes to accomplish specific tasks using Windows applications versus doing the same tasks using OS/2 Warp applications, the study might have shown that OS/2 Warp's drag and drop interface saves the users many steps when faxing, reaching a favorite web page, or launching an application from the toolbar.

    Question: How was the study conducted?

    Microsoft indicates that the study measured speed and accuracy on a variety of tasks The firm who executed the tests, Andre Associates, did group evaluation with one timer to several participants. It is not clear to us that the evaluators knew what the pass/fail criteria was. There were no intervention protocols which means that if a person was on the wrong track they could stay there until they failed. One failure on a sub task could cause the entire task family to be counted as a failure. A precarious position for an OS/2 user since they did not have the same starting point as a Windows user, yet they were required to be a user of Microsoft applications. There was no customization of the desktop to allow for migration from the workplace shell. Thus, IBM has no way of knowing whether or not the systems were set up to allow exploitation of productivity enhancing features like the desktop toolbar to allow single-click application launching and task switching, Fast Boot for quick loading of Windows applications, or even launching of Windows applications directly from the Workplace Shell desktop. Since the study dictates the use of mostly Microsoft applications, Microsoft controlled how the applications were installed and presented to the user on each platform. Unless Microsoft went back to the machines to ensure that the Windows applications had been properly migrated to the OS/2 desktop and that they had been setup properly on the desktop toolbar - a set of operations that takes only seconds to do on OS/2 and is well documented - then it is likely that many users simply used Program Manager to launch the Office applications rather than exploiting the user interface advantages of OS/2 Warp.

    Of interesting note also, Microsoft monitored the testing by having staff members on hand to answer questions. We have no way of knowing what "helpful hints" they may have given the OS/2 testers.

    Question: Who were the test subjects?

    The study indicated that IDC had difficulty finding OS/2 users with networking experience. Microsoft provided a list of OS/2 users to IDC. Microsoft's directive was that they only wanted to evaluate OS/2 users with Netware experience. IDC screened their lists and eliminated all OS/2 users with LAN Server experience. This eliminated a very substantial portion of the OS/2 user base from which to draw since many OS/2 installations rely on IBM LAN Server as their server of choice along with its intuitive drag and drop based user interface consistent with the rest of OS/2. Limiting the tests to OS/2 users who primarily rely on Windows applications and Netware servers, while NOT limiting the selection of Windows or Apple users to those with Netware but also including Windows users with Windows NT as a server or Apple users with AppleTalk, puts OS/2 at a marked disadvantage. Users of OS/2 Warp Connect peer functions would have been excluded while users of peer functions between Windows 95 and Windows NT as a server would not have been. Selection of the Windows 95 users from the Preview Program user base virtually guaranteed Microsoft a fairly advanced user base - those wishing to participate in a beta program as an early adopter - giving them an advantage neither IBM nor Apple were provided in the tests.

    It seems clear that Microsoft carefully crafted this study to portray Windows in the best light.

    We believe, as Apple has stated, that the study is flawed and unfair. The tests performed have been biased towards a Microsoft user and therefore would result in an improved performance for their operating system.

    This study simply lacks credibility.

    Walter W. Casey Director of Marketing Personal Software Products Division International Business Machines Corporation

    Microsoft's Press Release

    REDMOND, Wash. - In a test aimed at measuring the productivity of personal computer users in terms of speed and accuracy on a variety of common tasks, International Data Corp. (IDC) found that users of the Microsoft(R) Windows(R) 95 operating system outperformed Macintosh(R) and OS/2(R) users. Highlights of the study include the following:

  • Users of Windows 95 finished the overall test in an average of 58 minutes, 19 percent faster than Mac(tm) users, who completed it in 72 minutes, and 50 percent faster than OS/2 users, who finished in 116 minutes.

  • Seventy-six percent of the users of Windows 95 completed eight or more of the task groups successfully, compared to 58 percent of the Mac users and 31 percent of the OS/2 users.

  • In a key measure of productivity, users of Windows 95 who completed eight or more of the 10 task groups successfully did so in 22 percent less time than Mac users and in 51 percent less time than OS/2 users. Eighty-five percent of that group of users of Windows 95 completed the test in less than one hour, compared to 47 percent of the Mac users.
  • "The IDC test indicates that users of Windows 95 completed common tasks in 19 percent less time than Mac users, and with greater overall accuracy," said David Card, author of the report and director of PC software research at IDC.

    In July, IDC teamed with consulting firm Andre' Associates of Oakland, Calif., to execute a series of tests and focus groups for Microsoft that measure user productivity under different operating systems. The tests compared 54 Windows 95 Preview Program participants, 55 Mac operating system users and 52 OS/2 users with comparable skills on identical business-computing tasks and on comparably configured systems. To IDC's knowledge, this was the largest such study ever undertaken.

    The tests comprised 10 families of tasks covering local and networked file management and printing, document and application management, checking system resources, creating an alias or shortcut, and customizing the desktop. The test also required the users to attach an external CD-ROM drive.

    The study's intent was to evaluate the productivity of users of the operating system itself, rather than to measure application usage. The minimal interaction with applications was controlled through identical applications in each environment. Likewise, this test was designed to measure a mix of everyday tasks, such as manipulating files, as well as things business users do less frequently. IDC weighted all task groups equally.

    My Comments

    This is, in my opinion, typical of the Microsoft marketing machine. In nearly every review of Windows 95 it was acknowledged that, while better than Windows 3.11, Windows 95 was still not as good as the Macintosh. They are now trying to obfuscate that fact with carefully doctored studies.

    Look at this claim in particular:

    In a key measure of productivity, users of Windows 95 who completed eight or more of the 10 task groups successfully did so in 22 percent less time than Mac users and in 51 percent less time than OS/2 users. Eighty-five percent of that group of users of Windows 95 completed the test in less than one hour, compared to 47 percent of the Mac users.

    Note that they are careful to distinguish the group of "users of Windows 95 who completed eight or more of the 10 task groups successfully" while they do not do so for the Macintosh or OS/2 users. This implies that they counted the entire groups of these users, including the ones who did not accomplish the tasks successfully. They did this again with "that group of users of Windows 95" versus simply "Mac users".

    Only by comparing their successes to our failures can they come out ahead. It's pretty sad when they have to resort to this sort of thing.

    Remember, Mark Twain said it best: "There are three types of lies. Lies, Damn Lies, and Statistics." I think this study fits all three of these categories.

    What do you think?

    Return to my Macintosh Advocacy page
    Return to opposing the Microsoft juggernaut
    Return to my home page

    And just in case you wondered, MacOS Rules!

    Created on Sun, Nov 26, 1995 and last modified on Wed, Feb 25, 1998.